I am a complete layman in the world of complex computer graphics programming and research, but I have an interest in it as someone who enjoys playing computer games.
Not sure how well it fits what you're looking for, but if you're not already aware of tom7/suckerpinch, you might get a kick out of some of his videos -- e.g. https://www.youtube.com/watch?v=5TFDG-y-EHs
Splatting various kinds of blobs is actually an old idea. It’s a branch of point cloud rendering that researchers have been talking about for a couple decades now. What’s new is the technique to create the splats from photos.
NeRFs are made of a dense grid of tiny AIs that each can answer what the inside of that grid cube looks like from any angle. You march through the grid to see how a whole scene looks from a particular angle.
Gaussian Splats target the same goal with a different approach. A splat is just a fuzzy oriented blob. You render them by sorting them front to back then marching through them per pixel until it gets opaque enough.
The Gaussian aspect is important because Gaussians are differentiable. With that you can start with a bad estimation of the splats, render the bad estimation, compare it to the photos and back-propagate the difference between actual vs desired to iteratively improve the bad estimate to a better one.
Triangles are great because you can make solid, closed surfaces from them. But, you have to maintain the connectivity between the vertices. That makes adding and removing triangles from a mesh a major challenge.
SDFs are great because you can easily make crazy changes to the topology without ever thinking about connectivity. But, you have to be able to describe the shape of the whole scene with math. For realistically complicated shapes, that adds up to a lot of math per marching step.
Splats are loose, independent blobs. They are easy to individually manipulate. Adding and removing them is trivial. But, it’s difficult to get the appearance of a solid surface from them.
> The Gaussian aspect is important because Gaussians are differentiable.
Sure, but why Gaussians and not any of the many other well-known smooth functions? Is it because of lack of imagination (since the Gaussian is a sort of "default" choice)? Or is there some reason to pick gaussians, specifically?
One additional reason is that the projections of a Gaussian are also Gaussian (more generally any linear transform of a Gaussian is Gaussian). So if you have the parameters of a 3D Gaussian ellipsoid and a viewing angle, you can quite easily get the parameters of the corresponding 2D ellipse. So the ray marching algorithm turns into just blending a stack of 2D ellipses. For many other non Gaussian functions projection is trickier.
Ah, that is a very good reason indeed! At least for projecting from 2D to 1D the gaussian is the only distribution with this property. The functional equation g(x)g(y)=g(√(x^2+y^2))g(0) determines the gaussian up to a constant.
NeRFs were cool, but it turns out you get just-as-good results by fitting these Gaussian blobs. Plus it's faster. Plus the representation of gaussians is much easier to edit than the opaque neural representation.
It's better than other representations because...no one has been able to reconstruct scenes with other representations this well.
Gaussian splatting is not popular because it's a good rendering method. It's popular because there's now a good technique for generating great 3d gaussian splats from 2d images. And now that you have gaussian splats, it's nice to render them quickly.
Inspired by NeRFs (renders radiance fields as well) but is faster. It's a point cloud with fuzzy points that have size, shape, and direction, not just color/alpha. It has the same niche and up/downsides as point clouds, and the recent wave of interest is due to the easy reconstruction and rendering of photorealistic scenes from a few photos.
I'm probably wrong and I honestly hope I'm wrong but AFAICT this is a dead end. even a small scene takes 30-50meg and is only viewable withim a short distance of the center of the data.
If you wanted the inside of an apartment or house like matterport it would easily be 500mb to 1gig of data and forget it if you try an outdoor scene
You’re correct in that, i could have picked different wording. it can be used to create radiance fields, but in a way that is completely different than NeRFs.
Would be interesting if researchers would start using Blender/Godot instead of Unity/Unreal since they are not truly open source. But I guess you might need a particular skillset for this to work with, which might come from the pro game development community which might be more familiar with these studio tools.
Most of actual research I've seen (i.e. papers) are not Unity/Unreal, they rather are CUDA/Python.
The "Unity Gaussian Splatting" is not a research per se, just integration of that existing technique into Unity. As for why Unity, well yeah that's because I have experience with it, and am comfortable using it.
There's many other people experimenting with in in web ecosystem (WebGL, WebGPU, three.js, aframe etc.), that you can say is properly open source.
Blender as is right now is lacking certain functionalities to efficiently do gaussian splatting. However, for upcoming Blender 4.1 several pieces are landing - I have extended PLY file importer to be able to import custom attributes (as needed by gaussian splats), someone else contributed APIs to be able to write compute shaders from a Python addon. Once various pieces like that are in place, someone could indeed make a decently performant gaussian splat add-on or something.
Thanks for your response and incredible work, and also to learn people are using different toolsets. Its great to hear Blender 4.1 is also shaping up to be a landmark release. Compute shaders are a new interest of mine so hopefully I can do some playing around with it.
Do you think we will see Gaussian splatting integrated into a game in the (near or not near) future?
The author "Aras P" worked for Unity close to 2 decades (retired beginning of this year). Doesn't invalidate what you said, just some background on why Unity :)
I really enjoyed this succinct YouTube video on the topic, for those interested in a primer: https://youtu.be/HVv_IQKlafQ?si=y9gdS-1UkmdTX2Ik