New batch
Stranded
Crash landing
Crash landing #2
Colony swallowed by the sand
Desert Landscape
New batch. Getting closer to desired settings:
Word spaceship does not work as intended, for the rest, im pretty happy with the results. Trying to get rid of artifacts and we’re good to go for Desert Motives.
While exploring why spaceships are quite badly painted, decided to do an inbetweener run with slightly different prompt depicting desert town.
Not the best results when looking at the overall painting, but fine tuning details is pretty much done now. Will start exploring other styles and artists, as well as prompt engineering to avoid results which have very few matches (such as spaceships).
Settings:
{
"text_prompts": {
"0": [
"A beautiful painting of sci-fi settlement in the desert landscape, shuttle is approaching it from the sky, dust storm on the horizon, sun is dimmed by dust clouds, by greg rutkowski and thomas kinkade, Trending on artstation.",
"brown color scheme"
],
"100": [
"",
""
]
},
"image_prompts": {},
"clip_guidance_scale": 80000,
"tv_scale": 0,
"range_scale": 300,
"sat_scale": 0,
"cutn_batches": 8,
"max_frames": 10000,
"interp_spline": "Linear",
"init_image": null,
"init_scale": 1000,
"skip_steps": 10,
"frames_scale": 1500,
"frames_skip_steps": "60%",
"perlin_init": false,
"perlin_mode": "mixed",
"skip_augs": false,
"randomize_class": true,
"clip_denoised": false,
"clamp_grad": true,
"clamp_max": 0.05,
"seed": 549612405,
"fuzzy_prompt": false,
"rand_mag": 0.05,
"eta": 0.3,
"width": 1024,
"height": 1024,
"diffusion_model": "512x512_diffusion_uncond_finetune_008100",
"use_secondary_model": true,
"steps": 100,
"diffusion_steps": 1000,
"diffusion_sampling_mode": "ddim",
"ViTB32": true,
"ViTB16": true,
"ViTL14": false,
"ViTL14_336px": false,
"RN101": false,
"RN50": true,
"RN50x4": false,
"RN50x16": false,
"RN50x64": false,
"cut_overview": "[12]*400+[4]*600",
"cut_innercut": "[4]*400+[12]*600",
"cut_ic_pow": 2,
"cut_icgray_p": "[0.2]*400+[0]*600",
"key_frames": true,
"angle": "0:(0)",
"zoom": "0: (1), 10: (1.05)",
"translation_x": "0: (0)",
"translation_y": "0: (0)",
"translation_z": "0: (10.0)",
"rotation_3d_x": "0: (0)",
"rotation_3d_y": "0: (0)",
"rotation_3d_z": "0: (0)",
"midas_depth_model": "dpt_large",
"midas_weight": 0.3,
"near_plane": 200,
"far_plane": 10000,
"fov": 40,
"padding_mode": "border",
"sampling_mode": "bicubic",
"video_init_path": "/content/drive/MyDrive/init.mp4",
"extract_nth_frame": 2,
"video_init_seed_continuity": false,
"turbo_mode": false,
"turbo_steps": "3",
"turbo_preroll": 10,
"use_horizontal_symmetry": false,
"use_vertical_symmetry": false,
"transformation_percent": [
0.09
],
"video_init_steps": 100,
"video_init_clip_guidance_scale": 1000,
"video_init_tv_scale": 0.1,
"video_init_range_scale": 150,
"video_init_sat_scale": 300,
"video_init_cutn_batches": 4,
"video_init_skip_steps": 50,
"video_init_frames_scale": 15000,
"video_init_frames_skip_steps": "70%",
"video_init_flow_warp": true,
"video_init_flow_blend": 0.999,
"video_init_check_consistency": false,
"video_init_blend_mode": "optical flow"
}
Night batch. Selected the 3 out of 10 with fewest artifacts.
Aivazovskiy and John Harris style.
Unfortunatelly i believe Aivazovskiy is adding too many clouds, paired with ‘clouds’ in the prompt.
{
"text_prompts": {
"0": [
"A beautiful painting of dystopian desert city background sci fi, tiny GX-799 Starship Anthera image 7 in the sky, dust and smoke clouds, dim sun, by Ivan Aivazovsky and John Harris, Trending on artstation.",
"brown color scheme"
],
"100": [
"",
""
]
},
"image_prompts": {},
"clip_guidance_scale": 85000,
"tv_scale": 0,
"range_scale": 300,
"sat_scale": 0,
"cutn_batches": 8,
"max_frames": 10000,
"interp_spline": "Linear",
"init_image": null,
"init_scale": 1000,
"skip_steps": 10,
"frames_scale": 1500,
"frames_skip_steps": "60%",
"perlin_init": false,
"perlin_mode": "mixed",
"skip_augs": false,
"randomize_class": true,
"clip_denoised": false,
"clamp_grad": true,
"clamp_max": 0.05,
"seed": 2874621777,
"fuzzy_prompt": false,
"rand_mag": 0.05,
"eta": 0.3,
"width": 256,
"height": 256,
"diffusion_model": "512x512_diffusion_uncond_finetune_008100",
"use_secondary_model": true,
"steps": 250,
"diffusion_steps": 1000,
"diffusion_sampling_mode": "ddim",
"ViTB32": true,
"ViTB16": true,
"ViTL14": false,
"ViTL14_336px": false,
"RN101": false,
"RN50": true,
"RN50x4": false,
"RN50x16": false,
"RN50x64": false,
"cut_overview": "[12]*400+[4]*600",
"cut_innercut": "[4]*400+[12]*600",
"cut_ic_pow": 20,
"cut_icgray_p": "[0.2]*400+[0]*600",
"key_frames": true,
"angle": "0:(0)",
"zoom": "0: (1), 10: (1.05)",
"translation_x": "0: (0)",
"translation_y": "0: (0)",
"translation_z": "0: (10.0)",
"rotation_3d_x": "0: (0)",
"rotation_3d_y": "0: (0)",
"rotation_3d_z": "0: (0)",
"midas_depth_model": "dpt_large",
"midas_weight": 0.3,
"near_plane": 200,
"far_plane": 10000,
"fov": 40,
"padding_mode": "border",
"sampling_mode": "bicubic",
"video_init_path": "/content/drive/MyDrive/init.mp4",
"extract_nth_frame": 2,
"video_init_seed_continuity": false,
"turbo_mode": false,
"turbo_steps": "3",
"turbo_preroll": 10,
"use_horizontal_symmetry": false,
"use_vertical_symmetry": false,
"transformation_percent": [
0.09
],
"video_init_steps": 100,
"video_init_clip_guidance_scale": 1000,
"video_init_tv_scale": 0.1,
"video_init_range_scale": 150,
"video_init_sat_scale": 300,
"video_init_cutn_batches": 1,
"video_init_skip_steps": 50,
"video_init_frames_scale": 15000,
"video_init_frames_skip_steps": "70%",
"video_init_flow_warp": true,
"video_init_flow_blend": 0.999,
"video_init_check_consistency": false,
"video_init_blend_mode": "optical flow"
}
It’s not a question of adding more adjectives to the game, we need to convert the existing planet data into adjectives so that appropriate images can be generated. Something like this:
Fertility (low, medium, high) => “a barren appearance”, “sparse vegetation”, “lush greenery”
Temperature (low, medium, high) => “freezing”, “temperate”, “sweltering”
Pressure (low, medium, high) => “clear”, “cloudy”, “oppressive”
Gravity (low, medium, high) => “tall, average height, short”
Metal resources => “lots of rocks”
Mineral resources => “crystalline outcroppings”
Liquid resources => “large lakes”
Gas resources => “a dense atmosphere”
You then generate a painting prompt from the planet data, e.g. Montem would come out as “A temperate landscape with sparse vegetation, clear skies, average height features, and lots of rocks.”
We may then also want some kind of way to turn the planet name into a colour scheme. e.g. convert it to a hash and then take a few bits as an index into a big list of pastel colours, so two planets with the same description still end up looking different - “a mauve colour scheme”. Or even have two colour lookups, one for the sky and one for the ground.
In addition, bits from the hashed name could be used to add other strings, such as “space craft landing” or “a human settlement” or “people in spacesuits”, to generate more variety.
The main challenge with this would be ensuring that the resulting pseudo-random string is always a reasonable prompt for the painting algorithm. I don’t know much leeway there is in the interpretation.
1 Like
I totally agree. Do you think you can create such adjectives for Montem and Valis? Lets try those two first. Im almost done ironing out artifacts during generation.
Also take into account:
Population and occupied slots: Dense populated metropolis or Lonely Outpost
The color pallete should be governed by: surface color (lots of sulfur = yellow, lots of iron = orange, green for forests, blue for water etc) + distance to the star (take solar system as reference, dim light or bright light) + type of the star (class B, A etc, if this is missing we come up with our own)
Taking into consideration of what you said with prompt engineering i would also add these as the primary parameters for planets:
Color palette (what was said above)
Shape of features (sharp rocks, flat grasslands, curved dunes)
Density of features (big mountain ranges, or deserted savannah, or lonely outpost)
I can generate various results for the same prompt with these 3 parameters, so the style can remain the same say for Vallis and Montem (because they are in the same star system and presumably are constructed by same people residing there i.e. local culture)
Does the prompt need to be a well formed English sentence, or can we just tack a load of adjectives onto the end of it?
If we want the colour to derive from the mineral contents, then we may just have a mapping from resource type to a string. Montem and Vallis are pretty similar planets, so we may want to pick a third one that is quite different (e.g. Avalon), to show variety. The only significant differences between Montem and Vallis are that Montem is slightly closer to the star, and has limestone. So the string descriptions might be:
Montem: A temperate, brightly lit landscape with sparse vegetation, clear skies, average height features, red ground, light gray rocks.
Vallis: A temperate landscape with sparse vegetation, clear skies, average height features, red ground
I’m assuming the images would be generated once for each planet, and then set as the permanent image for it. So I don’t think that basing the image on plot count is a good idea in general, since population can change over time. Having said that, I guess it would be pretty safe to have the current starting planets show lots of people. So maybe those could have a “busy human settlement” string added, while all other planets have a pseudo random selection from the generic “spacecraft landing”, “humans in space suits” list.
Star distance adjectives: bright daylight, sunset, twilight, night time
FEO: red ground
LST: light gray rocks
SIO: white sand
SCR: yellow crystals
HAL: white crystals
CUO: green rocks
etc.
It does not have to be a well formed english sentence. An in any case i will check the words against CLIP database to ensure the desired objects exist in the model.
Yes lets do Montem and Avalon
Yes i agree on pop growing over time, i overlooked this.
I will launch Montem now with:
A temperate , brightly lit landscape with sparse vegetation , clear skies, average height features, red ground , light gray rocks .
Are we after more details, more abstraction, or smoother overall image? See above images, let me know which one fancies your eye.
I’d say it’s good to have variety. But ultimately we will need a set of parameters that work well with all the possible inputs. So I think we will have to come up with a set of example planet strings, and then see which style produces good results with all those strings.
Perhaps we can do some tricks like use “landscape” for temperate Earth-like worlds, and “planet surface” for more hostile ones.
Montem: A sci-fi painting of a landscape, bright daylight, sparse vegetation, clear skies, red ground, light gray rocks, busy human settlement
Avalon: A sci-fi painting of a planet surface, night time, barren, clear skies, dark gray rocks, gray dusty ground, yellow crystals, humans in spacesuits
okay for historical reasons im gonna post the evolution over last 24hrs
So initially i checked your suggested prompt and tweaked it abit, to have relevant CLIP database matches:
A sci-fi painting of a landscape, bright daylight, sparse vegetation, clear blue skies, dark orange and red soil, light gray rocks, busy village, by John Harris and Hubert Robert, Trending on artstation.
Here are unfiltered results. I was not too happy because of too many artifacts.
I stopped the generation because i was un happy with other settings, so went in to tweak them afterwards
Then i tweaked the settings and consulted documentation to refine the results. Prompt was producing almost desirable results, except for busy village. This is still absent. The color pallete is awesome.
So here are the tweaked results. I added additional artist and tweaked stuff to prevent cloud castles, and refines filters to achieve better detalization. This is a night run with 20 generations:
In my opinion this batch looks amazing, and the current tweaked settings are perfect in terms of style. I continued adjusting artists, to achieve variations in architecture that is built and in detalization.
This is Batch13 and its settings are going to be the baseline:
{
"text_prompts": {
"0": [
"A sci-fi painting of a landscape, bright daylight, sparse vegetation, clear blue skies, dark orange and red soil, light gray rocks, busy village, by Hubert Robert and greg rutkowski, Trending on artstation."
],
"100": [
"",
""
]
},
"image_prompts": {},
"clip_guidance_scale": 5000,
"tv_scale": 0,
"range_scale": 300,
"sat_scale": 0,
"cutn_batches": 8,
"max_frames": 10000,
"interp_spline": "Linear",
"init_image": null,
"init_scale": 1000,
"skip_steps": 10,
"frames_scale": 1500,
"frames_skip_steps": "60%",
"perlin_init": false,
"perlin_mode": "mixed",
"skip_augs": false,
"randomize_class": true,
"clip_denoised": false,
"clamp_grad": true,
"clamp_max": 0.08,
"seed": 642245076,
"fuzzy_prompt": false,
"rand_mag": 0.05,
"eta": 0,
"width": 1366,
"height": 768,
"diffusion_model": "512x512_diffusion_uncond_finetune_008100",
"use_secondary_model": true,
"steps": 250,
"diffusion_steps": 1000,
"diffusion_sampling_mode": "ddim",
"ViTB32": true,
"ViTB16": true,
"ViTL14": false,
"ViTL14_336px": false,
"RN101": false,
"RN50": true,
"RN50x4": false,
"RN50x16": false,
"RN50x64": false,
"cut_overview": "[4]*1000",
"cut_innercut": "[4]*1000",
"cut_ic_pow": 1,
"cut_icgray_p": "[0.2]*400+[0]*600",
"key_frames": true,
"angle": "0:(0)",
"zoom": "0: (1), 10: (1.05)",
"translation_x": "0: (0)",
"translation_y": "0: (0)",
"translation_z": "0: (10.0)",
"rotation_3d_x": "0: (0)",
"rotation_3d_y": "0: (0)",
"rotation_3d_z": "0: (0)",
"midas_depth_model": "dpt_large",
"midas_weight": 0.3,
"near_plane": 200,
"far_plane": 10000,
"fov": 40,
"padding_mode": "border",
"sampling_mode": "bicubic",
"video_init_path": "/content/drive/MyDrive/init.mp4",
"extract_nth_frame": 2,
"video_init_seed_continuity": false,
"turbo_mode": false,
"turbo_steps": "3",
"turbo_preroll": 10,
"use_horizontal_symmetry": false,
"use_vertical_symmetry": false,
"transformation_percent": [
0.09
],
"video_init_steps": 100,
"video_init_clip_guidance_scale": 1000,
"video_init_tv_scale": 0.1,
"video_init_range_scale": 150,
"video_init_sat_scale": 300,
"video_init_cutn_batches": 1,
"video_init_skip_steps": 50,
"video_init_frames_scale": 15000,
"video_init_frames_skip_steps": "70%",
"video_init_flow_warp": true,
"video_init_flow_blend": 0.999,
"video_init_check_consistency": false,
"video_init_blend_mode": "optical flow"
}
Finally i attempted to achieve detailed brushstrokes by adding artists which preffered tiny details. There are currently 3 runs in the pipeline but here are the initial results:
Batch 14 FINISHED
Greg Rutkowski and Hubert Robert and Marc Simonetti
By looking at these i believe i may need to reduce tv_scale on all of them, as i noticed there are some grains on the images.
Started 3 new processes with the following changes:
tv_scale 300/600/900
cut_innercut [1]*400+[4]*400+[12]*200
to achieve finer details.
Im killing off Batch 15
by Albert Bierdstat and Hubert Robert and Marc Simonetti and Dan Mumford
Has too many weird round shapes and artifacts with light.
Will replace it by Batch 17 tomorrow.
I like Batch 14 the best. I think the round shapes in Batch 15 are because it’s trying to insert a moon or sun in the sky, but doesn’t know how.
I’m not sure that “busy village” is the right prompt - it seems to give it a Mediterranean stone cottage look. Perhaps “settlement” wasn’t in the AI vocabulary, which is why you changed it? We could try something else that is more sci-fi, which the AI understands. Does “futuristic city” or “busy outpost” work better? Maybe we can try “city dome” for Avalon. I would search for applicable terms myself but I don’t know where to look.
Also, I’m not sure that you’ll be able to come up with a single set of parameters which will generate something good for every possible planet string input. You may have to have one set for Earth-like worlds, and then a different set for Moon-like worlds, and so on. Gaseous planets will likely be their own thing in any case.
Here are some more planets to try though. I don’t know if you need 20 images per run, it seems 4 or 5 is enough to tell whether the settings are working. So maybe you could try 5 of each of these, and see?
Montem: A sci-fi painting of a landscape, bright daylight, sparse vegetation, clear blue skies, dark orange and red soil, light gray rocks, futuristic city
Avalon: A sci-fi painting of a planet surface, night time, barren, clear skies, dark gray rocks, gray dusty ground, yellow crystals, city dome
Promitor: A sci-fi painting of a landscape, bright daylight, lush farmland, clear blue skies, white crystalline rocks, futuristic city
Umbra: A sci-fi painting of a planet surface, sunset, barren, thick clouds, white crystalline rocks, spaceship landing
Boucher: A sci-fi painting of a planet surface, bright daylight, barren, clear blue skies, large lakes, humans in spacesuits
Artifacts have a lot to do when your prompt does not match artist style, say you are asking a person that was painting strict blocky buildings, to paint a nice round face. It causes contradictions, and if you are trying to achieve realistic painting, it doesnt work well.
settlement had no matches, hence i replaced it to busy village, but then i discovered ‘industrial buildings’ with batch 17 and 18 (which mostly failed) and will use that on batch 14 settings.
Futuristic city works, yes. Busy outpost, not. City dome does not work either. You can check approximate matches here:
https://rom1504.github.io/clip-retrieval/?back=https%3A%2F%2Fknn5.laion.ai&index=laion5B&useMclip=true&query=citydome
Its not an exact match database, so not all prompts actually show up on that site, but still construct okay in actual neural net, but still its a good indicator.
Yeah having several sets of settings for various planets, is no problem. i keep all of them with reference images and settings
I usually run 10 because out of 10 theres usually 3-4 with major artifacts, while others can be just perfect. I dont always post all 10. Batch 17 and 18 mostly failed, will filter those out.
I will tune the prompt for Montem and launch it with Batch14 settings, as well as a few tweaks for details, and then go through the prompts you offered. Thanks.
I looked on that site. Instead of “futuristic city” you could try “distant city” or “factories”.
Instead of “city dome” you could try “underwater city”.
Instead of “humans in space suits” you could try “astronauts”.
1 Like