Recently, I wanted a full database of every single planet and star in the entire system. However, although I searched, I didn’t find any database. I tried data scraping with python which was unsuccessful because I needed to run JavaScript, and then I realized that I wasn’t allowed to in the guidelines which I really should have read first. Then I found a GitHub post which had all of the stars, so I got all the names using python, and started manually copying the data in a dictionary. I got the exact values for the amounts of resources by inspecting the graph and reading the value from html. However, things were going slow, and after about 5 hours, I was only about 2.9827% done. Also, I timed myself for 1 hour going as fast as possible, and I found it would take me 100 hours. 100 hours is doable, but I don’t want to spend that much time gathering data that might change or be increased in a new update.
Then I came up with my new idea: I found I could copy large amounts of the html code by selecting a div further back, and then I would paste in it (somewhere, idk yet), and then run a program to extract the data from the html! Even if I work really fast, like 15 seconds per copy/paste, it would still take me at least 16.6666… hours to complete, so I consider it quite manual - and that’s not even considering figuring out how to extract the data from it! I assume html classes and ids will be useful there, but I have no idea.
Although the last strategy also used reading from html, this strategy involves literally copying and pasting the code directly, which is more similar to data scraping, so I was concerned about whether or not it was allowed. My argument is that it is still very manual, and would take me many hours to finish.
Uses for the data:
I would not release any of the data unless I have permission since you guys directly stated that you would not add a map search feature, and searching for planets is one thing I would be doing. I think I would have fun compiling data about different resources being on what types of planets, habitability vs. resource content, etc. Also, one major accomplishment I would like to achieve would be a program to generate transportation routes (one for gasses, one for dense material) that could find clusters of resource-filled planets.
Thanks for reading!
- Jet_Mouse
