yea so as of now, it is a fairly loose integration between threejs "rendering" a 2d image of a scanned product at some orientation, and then Stable diffusion inpainting a background around it.
However, there's a number of extensions here, that makes the integration of 2D and 3D more interesting going forward
1. 3D models means you can relight the model before running it through in-painting. adding lights in 3D around the product, plus using a more raycasted rendering system (which is now possible in-browser) means you can control the input to SD really well.
2. This one is the most interesting piece. You can create a 3D editor to allow very simple low poly style scene creation, lets say a pedestal and a vase or something from a product photography standpoint - then pass the depth map or canny edges as conditioning through something like control net and you have a super controllable scene design tool that you can finely control - both in camera angle and perspective.
However, there's a number of extensions here, that makes the integration of 2D and 3D more interesting going forward
1. 3D models means you can relight the model before running it through in-painting. adding lights in 3D around the product, plus using a more raycasted rendering system (which is now possible in-browser) means you can control the input to SD really well.
2. This one is the most interesting piece. You can create a 3D editor to allow very simple low poly style scene creation, lets say a pedestal and a vase or something from a product photography standpoint - then pass the depth map or canny edges as conditioning through something like control net and you have a super controllable scene design tool that you can finely control - both in camera angle and perspective.