Manual Data Retrieval from HTML Code

Recently, I wanted a full database of every single planet and star in the entire system. However, although I searched, I didn’t find any database. I tried data scraping with python which was unsuccessful because I needed to run JavaScript, and then I realized that I wasn’t allowed to in the guidelines which I really should have read first. Then I found a GitHub post which had all of the stars, so I got all the names using python, and started manually copying the data in a dictionary. I got the exact values for the amounts of resources by inspecting the graph and reading the value from html. However, things were going slow, and after about 5 hours, I was only about 2.9827% done. Also, I timed myself for 1 hour going as fast as possible, and I found it would take me 100 hours. 100 hours is doable, but I don’t want to spend that much time gathering data that might change or be increased in a new update.

Then I came up with my new idea: I found I could copy large amounts of the html code by selecting a div further back, and then I would paste in it (somewhere, idk yet), and then run a program to extract the data from the html! Even if I work really fast, like 15 seconds per copy/paste, it would still take me at least 16.6666… hours to complete, so I consider it quite manual - and that’s not even considering figuring out how to extract the data from it! I assume html classes and ids will be useful there, but I have no idea.

Although the last strategy also used reading from html, this strategy involves literally copying and pasting the code directly, which is more similar to data scraping, so I was concerned about whether or not it was allowed. My argument is that it is still very manual, and would take me many hours to finish.

Uses for the data:
I would not release any of the data unless I have permission since you guys directly stated that you would not add a map search feature, and searching for planets is one thing I would be doing. I think I would have fun compiling data about different resources being on what types of planets, habitability vs. resource content, etc. Also, one major accomplishment I would like to achieve would be a program to generate transportation routes (one for gasses, one for dense material) that could find clusters of resource-filled planets.

Thanks for reading!

  • Jet_Mouse

Here is a searchable database of resource deposits: FIO

Here is our discord dedicated to the development of tools and spreadsheets to assist with playing Prun: PrUn Community Tools

Thank you! I found a google doc that has everything I need and more! I don’t even know what I’ll do with magnetic field data but I will have magnetic field data!!! I’m sure there’s a way to grab all that data and stick it in python. Maybe I’ll find some correlations!

@Jet_Mouse: :slight_smile:

Just finished getting everything I needed from different pages in the 2 google sheets files, and I put all the planetary data in this json file. Feel free to share it around - json files are really useful for programming.

Here’s where I got the data:
Sheet 1
and I would put sheet 2 here except that I can only put 2 links in each post, so I will add sheet 2 in a reply

Sheet 2