I could actually make this, and it sounds like a cool idea. But I'm wary about putting the time into it.
My concern is how many recruiters would actually download and run a 1 or 2 GB application simply to see my resume, when they instead could visit my website and watch some quick videos.
This is a great example of how we (collectively) are using virtual worlds as "dreams" for Deep learning.
In the Deep Vision world we as a group are trying to segment, classify and reinforce our NN training on labeled real world data. The challenge is, it's a very manual process to label data - specifically images. The more that we can do inside the computer, for example automatically labeling pixels inside an image, without having to acquire and label real world data (or making it easier to do with real world data) the easier and faster training becomes that can be applied to real world use cases.
The trick is making the virtual world match the real world as closely as possible so that the nets we make are accurate representations of real world scenarios.
In order to transfer learnings from virtual worlds to the real world more efficiently, I imagine a module that translates real world footed to an abstract representation that is the same as for rendered footage.
In other words, you would train a NN on input from a footage of a walkthrough of a real building to output rendered footage from the same path (3D data from architects is already there). The middle layer of this quasi autoencoder then is the basis to train fully simulated tasks, e.g. autonomous vehicles. In a way, it would be similar to colorization of b/w footage. Would that scale training data?
Yea, I mean that is basically what we do with our home furnishings app - except we have to SFM to build the models.
The challenge is labeling - or autolabeling pixels.
One thing we are trying to work out is how do you label and build nets on volumes - rather than just pixels? I'm thinking it's going to be a magnitude harder.
I was surprised, since I didn't think it would be realistic enough for this purpose, but Grand Theft Auto V is being used in this manner to train self-driving cars.
Nothing, actually, it's a quality library, just old.
I can think of a few algorithms that have changed or been created since the 1990s so I have to assume there's something that can be done to optimize it.
There's also a severe lack of "in-browser" CV solutions (which could be fun), instead opting for binding OpenCV in the server-side.
Same here. I was thinking of how long it would take us before the inflation in fancy CVs and resumés would force recruitment agencies to invest in rendering farms and full-body VR suits in order to read job applications.
I don't mean to criticise you personally, because I have done the same thing, but: Comments about what someone thought the title meant are very common and not interesting in any way.
I disagree. Most of the reason that people click on anything on this site is directly because of the title, so if I have particular expectations that the title fails to meet, then it's fair game for commenting. I don't think it's fair to assert that what you find common and uninteresting is irrelevant to someone else.
You could use it to say, auto create large labeled data sets of images and their corresponding depth maps. Or generate large labeled data sets of objects and their segmentation masks. Or for training an autonomous agent (say self driving car) in a simulation environment
Many thanks for your work, will be putting it to use! Please consider integrating it with https://gym.openai.com/ for easy reinforcement learning tasks.
Some similar efforts (though not with open source engines)
I thought about something like this for autonomous cars. The game engine could simulate all the sensors in a car. Then it would be possible to use this as training data. Would be fun to create a racing game out of it to let different algorithm compete against each other.
Does this already exist? Google should have something like this, but probably not freely available.
This is a brilliant idea, IMO, and a real game changer (in principle, not necessarily tied to the implementation). Without this, CV problems are largely only limited by how much data you can get. If you can simulate arbitrary scene configurations it's now possible to solve almost any non-human CV problem, even ones with sparse data.
For example, in security applications you have basically zero data on the sorts of anomalous events you want to trigger on. If you can simulate example cases of those things that means you can actually start applying supervised machine learning.
Many researchers have done interesting work using synthetic data (a lot of them are mentioned in this post). This tool just tries to make the creation of virtual world (which can be used by computer vision) easier.
The reason I ask is that I have a large dataset that I would like to visualize.
The ideal situation would be:
I could use python to create a giant flat plane, and then iterate over the dataset to generate the level (which initially would just be quads on the 2d plane).
The dataset would be too large to do in-engine in realtime, and the quads are static - I would like to generate the level as a 1 off process, and then have Unreal do any optimisations that it can by "compiling" the level (e.g. frustum culling, quad tree calculation etc) I might have some of those optimizations wrong since my knowledge of engines is still stuck in the Quake 1 era, but you get the gist :)
Do you think this is possible? If this tech doesn't exist, I would pay someone to develop it for me....
How many total datapoints are you trying to represent at once, ballpark? My first instinct is that it would probably be most efficient to feed the data into Unreal and build the physical representation there VS importing a large monolithic 3D model, because then you could make use of instancing, distance culling, etc. This would also mean you could animate the objects separately as well, and manipulate scale, color and the like in realtime.
Again, probably depends heavily on the specifics. In general though, modern gaming GPU's can handle just an insane number of polygons so either way, you should be able to represent a ton of datapoints.
I believe there's also a SQL lite support module for the engine as well, if you want to deal with really massive sets of data, though the documentation there is sparse.
My contact info is in my profile, feel free to hit me up if you have any other questions, happy to help.
Anyone know of something similar for Unity? Unity has a bit of a steep learning curve of you want to do stuff like importing external data and working with outside programs
The CVC lab and Virtual KITTI use Unity in their projects. They might have some internal tools. The advantage of UE4 is open-source, so that we can fully understand what the engine is doing.
I, like many others most likely, thought this was gonna be a cool way to present your resume, or someone with a demo of theirs in Unreal Engine... Still, I was not disappointed by the plugin.
(nevermind that the last Unreal I played was in 1998.)