Splatting various kinds of blobs is actually an old idea. It’s a branch of point cloud rendering that researchers have been talking about for a couple decades now. What’s new is the technique to create the splats from photos.
NeRFs are made of a dense grid of tiny AIs that each can answer what the inside of that grid cube looks like from any angle. You march through the grid to see how a whole scene looks from a particular angle.
Gaussian Splats target the same goal with a different approach. A splat is just a fuzzy oriented blob. You render them by sorting them front to back then marching through them per pixel until it gets opaque enough.
The Gaussian aspect is important because Gaussians are differentiable. With that you can start with a bad estimation of the splats, render the bad estimation, compare it to the photos and back-propagate the difference between actual vs desired to iteratively improve the bad estimate to a better one.
Triangles are great because you can make solid, closed surfaces from them. But, you have to maintain the connectivity between the vertices. That makes adding and removing triangles from a mesh a major challenge.
SDFs are great because you can easily make crazy changes to the topology without ever thinking about connectivity. But, you have to be able to describe the shape of the whole scene with math. For realistically complicated shapes, that adds up to a lot of math per marching step.
Splats are loose, independent blobs. They are easy to individually manipulate. Adding and removing them is trivial. But, it’s difficult to get the appearance of a solid surface from them.
> The Gaussian aspect is important because Gaussians are differentiable.
Sure, but why Gaussians and not any of the many other well-known smooth functions? Is it because of lack of imagination (since the Gaussian is a sort of "default" choice)? Or is there some reason to pick gaussians, specifically?
One additional reason is that the projections of a Gaussian are also Gaussian (more generally any linear transform of a Gaussian is Gaussian). So if you have the parameters of a 3D Gaussian ellipsoid and a viewing angle, you can quite easily get the parameters of the corresponding 2D ellipse. So the ray marching algorithm turns into just blending a stack of 2D ellipses. For many other non Gaussian functions projection is trickier.
Ah, that is a very good reason indeed! At least for projecting from 2D to 1D the gaussian is the only distribution with this property. The functional equation g(x)g(y)=g(√(x^2+y^2))g(0) determines the gaussian up to a constant.
NeRFs are made of a dense grid of tiny AIs that each can answer what the inside of that grid cube looks like from any angle. You march through the grid to see how a whole scene looks from a particular angle.
Gaussian Splats target the same goal with a different approach. A splat is just a fuzzy oriented blob. You render them by sorting them front to back then marching through them per pixel until it gets opaque enough.
The Gaussian aspect is important because Gaussians are differentiable. With that you can start with a bad estimation of the splats, render the bad estimation, compare it to the photos and back-propagate the difference between actual vs desired to iteratively improve the bad estimate to a better one.
Triangles are great because you can make solid, closed surfaces from them. But, you have to maintain the connectivity between the vertices. That makes adding and removing triangles from a mesh a major challenge.
SDFs are great because you can easily make crazy changes to the topology without ever thinking about connectivity. But, you have to be able to describe the shape of the whole scene with math. For realistically complicated shapes, that adds up to a lot of math per marching step.
Splats are loose, independent blobs. They are easy to individually manipulate. Adding and removing them is trivial. But, it’s difficult to get the appearance of a solid surface from them.