AKA: real-time repair tool. Impressive results overall, and a fun use for AR for sure, though it seems they're not using previous frames' covering textures as hints (or not overly well). Especially visible with a heavy-noise background like where the drainage grate is, at around 1:46.
Next up: augmented reality headgear, so you don't have to see billboards while driving / hunting zombies in the real world. Or your annoying coworkers.
Well great. Now I can't even trust live video not to be censored.
I wonder how it works on non-static objects, like people? Seeing the drain removed from the sink made me think of removing faces from people. Advance it enough, and you could probably get a visual Autotune for people to make them generically attractive by removing wrinkles and whatnot. Quite useful for politics, news, etc.
Given that the much simpler problem of autotune can't be solved without giving the voice an artificial sound, I doubt that auto-removal of wrinkles is going to work very well.
Of course you can also just remove wrinkles with a blur. I guess this gets more and more difficult in the age of HDTV.
Also: I'm less concerned about the fact that I can't trust live video to be uncensored, and more concerned about the fact that conspiracy theorists now can't trust live video to be uncensored. Now that they've been given the idea, I look forward to the next round of conspiracy theories, in which pixels are analyzed to prove that [favourite boogeyman] was really behind [next major event].
The OP is a more advanced technology. But as to whether you've already seen this on live video, if you've ever watched a baseball game, live coverage of home plate usually replaces the advertisement behind the batter. You'll see that newer stadiums facilitate this by painting blocks of solid color in the spots where the ads appear on TV.
When I was in college I wrote some software that did something sorta like this. My target was to post-process video content from TV shows with the channel logos in the corners.
My approach was to detect the logos based on rudimentary pattern-detection (mostly finding parts of the screen the didn't change over a period of time) then calculate the replacement pixel-color based on the other pixels around the edges of this logo, weighting their impact on the replacement pixels by their proximity to the pixel being calculated.
It worked reasonably well ... albeit much slower than the demo in this video ... and transparent logos were then introduced complicating the calculations. Also, sitcoms worked much better than sports. Their cameras were more stationary making detection and pixel-color replacement-calculation easier.
It's re-synthesize but then for the number of frames per second, there are already re-synthesize plugins for Photoshop and Gimp so this is only a logical step up.
I'm a little bit skeptical of the process. How do you "improve" the image and still get the details back? When you downsample/blur, you lose information. Does he keep the original and splice parts of it back? If so, how does he decide which to splice where? And if it's just using pixel color statistics when you up sample, that would work on most of the single color textures in the video, but how would that work for the brick road with the drain? The seams between the bricks are directional, so there'd have to be some way to account for that.
Anyone got a good guess? (or a paper for that matter?)
I'm thinking this won't work as well on a background that's not plain. I want to see the same thing except with a checker board pattern in the background.
I've often wondered how long it will be before TV companies edit the video of sporting events to replace the adverts in the stadium with different ones they've sold themselves. Given the adverts are in fixed places it would be easy to do you'd think.
Great for those videos you took on vacation last year -- with the girlfriend that didn't work out. Now she's not there any more.
I would imagine the next logical step (assuming people removal works, it was not demonstrated) would be to _add_ people to videos in realtime. Uncle Joe dead? No problem. We'll just take those videos that we had of him last Christmas and put him in our videos from this year.
While the tech is nowhere being new, the real-time nature of this really makes you think about the possibilities.
Next up: augmented reality headgear, so you don't have to see billboards while driving / hunting zombies in the real world. Or your annoying coworkers.