I've proof of concept-ed the same workflow and it just doesn't hold up in practice. Even so, there are some benefits that truly do help a solo dev.
It's actually easier to de-bake yourself than use SD-derived assets. What I mean is that img2depth is severely lacking at the moment. The geometry it produces it unsuitable for compositing with other game assets. You'll notice compositing was avoided in the demo.
As such it's easier to just block out geometry in Blender, and either project image as textures to it or repaint them by hand using diffusion output as a reference (or both if you're into that kind of thing). Even so, that's still incredibly useful.
One of the most common things holding back a solo dev are art assets. I might not be able to create concept art, models, and textures ex novo, but I can pretty easily block out some geometry, repaint textures, and throw in some basic lighting.
Looking forward, ideally img2depth+normal+diffuse+lighting would be a thing, but until that's something it's easier to take SD output as advice and suggestion than tangible assets and that's still worth a lot.
CN is a total game changer for generative image models, it solves so many things that were problematic before (proper depth, pose, sensible text and much more). This along with LoRa[0] and other improvements from the SD community really turn this into a super capable toolchain.
ControlNet is where it's at - the depth model can completely reskin an image [1] with a bit of prompting, and you can compose them (multiple controlnets feeding on each other)
It's actually easier to de-bake yourself than use SD-derived assets. What I mean is that img2depth is severely lacking at the moment. The geometry it produces it unsuitable for compositing with other game assets. You'll notice compositing was avoided in the demo.
As such it's easier to just block out geometry in Blender, and either project image as textures to it or repaint them by hand using diffusion output as a reference (or both if you're into that kind of thing). Even so, that's still incredibly useful.
One of the most common things holding back a solo dev are art assets. I might not be able to create concept art, models, and textures ex novo, but I can pretty easily block out some geometry, repaint textures, and throw in some basic lighting.
Looking forward, ideally img2depth+normal+diffuse+lighting would be a thing, but until that's something it's easier to take SD output as advice and suggestion than tangible assets and that's still worth a lot.