I actually wrote my honours thesis on this topic. In academia it's called Alpha Matting/Image Matting.
There's a great website http://alphamatting.com/ that provides a benchmark dataset and a comparison of a lot of influential methods.
I applaud your work in making a real time and interactive matting algorithm! One day it might be also as accurate as state of the art methods. You guys should write a paper about your method.
Thanks for the link! Real time is a pretty big constraint, and the input for those results is a much more complete labeling than we expect (only 22 to 44 pixels around the ground truth boundary are left uncertain). But I'll definitely read the paper and hopefully we can gain some insights from it!
Have a look at KNN Matting http://ihome.ust.hk/~dli/projects/knn/. Their MATLAB implementation uses a similar style to yours but is not as real time. They take single mouse clicks and spread them over a much larger window/bounding box. Only takes a few clicks on foreground/background to get excellent results.
Actually Shared Matting is far more applicable than KNN Matting. It's designed for real time use and has a post processing step which will help you deal with fine detail like hair.
You should read Jue Wang's 2007 survey paper which groups and puts all the techniques in context, then a handful of the top performing papers from recent years on alphamatting.com
A bit off topic, but I'm working on a video interviewing startup and one of the things we'd looove to offer is (semi-)automatic Chrome Keying / green screening to show just a person's torso and face.
Might (one of) you know what's the state of the art in research on this topic as well?
This is really slick. I work at a newspaper where we do a lot of cutouts and I've looked at several automatic tools to save people time, and I think this is the best I have seen in terms of balancing ease of use with accuracy.
The only thing I might suggest is that for photos with softer edges, you might offer some degree of "feathering" on the edges of the cut out. I don't know how hard this is with your algorithm, but it seems like it might be an easy fix.
Congrats! I had a similar idea a while ago so I really appreciate that somebody accomplished it in so high level! I already signed up and already waiting for the post alpha version :)
Did you guys consider in the future to add multiple object segmentation? That looks like a natural next step. Keep it up the good work! :)
This is beautifully done -- great UX, truly impressive tech. My wife loves it and is using it to repurpose some shots from our wedding for e.g. Christmas cards.
Completely agree! I actually tried it out with one of our pictures [1] from my proposal in Rome. I'm just having a bit of trouble with the small tuft of her hair that's forcing the algorithm to keep some of the background. I suppose I could probably fix this by using the full resolution picture, instead of this scaled down one.
A pretty busy background, soft/furry edges between subject and background, less-than-obvious clear color boundaries.
The real-time feedback makes it easy to give it the markings that it needs in less than 20 seconds for a near perfect crop (or as good as one would expect with fluffy hair as a boundary). I was pleasantly surprised.
Really neat. I was really itching to use CTRL+Z though.
I screwed up the first image I tried to clip. Maybe have a walkthrough of the features when the user first uses it? I didnt notice the help button until it was too late. I thought if I closed out of the editor and clicked on the image again, it would let me start over but it just takes you back to the editor. It would be nice if there was a way to redo editing the image from the start.
We only auto-show it during that time. The hope is that it triggers you to click the button if you need it, but lets you ignore it if you already know the drill...
(all these wonderful tradeoffs between "first visit" vs "repeat visit" - and what if it was missed the first time, blabla... )
Yeah, same thing for me. The action was too quick for me to notice that it was intentional. I would just leave the help window open on first time use until the user decides to close it. Very nicely done site though!
I get "Your Browser Is Out Of Date... Terribly sorry, but your browser appears to be missing key feature(s) required to use this website: Data URL Support Please try the latest version of one of these browsers instead: Google Chrome, Firefox", however I'm using Safari 5.1: data URL support has been in Safari since 3.1 (according to http://caniuse.com/datauri).
I get the same message for FF 21.0 on my phone ... it's the current beta release, so I'm not sure how to run anything newer without building it myself... 😓
GIMP doesn't give you a realtime boundary like this does, nor a preview image. I've done a lot of alpha matting in The GIMP and this seems far quicker to get similar results (except the up/download for me).
For quick and dirty work I'd use this for sure but revert to a desktop tool for precision work I think.
Does anyone know other Linux tools, or online for that, that do this comparatively well?
I use The GIMP for these sorts of things, the boundary is not realtime in the same way. As you select using the foreground or bg paint, the boundary doesn't change immediately. Instead it waits until you've finished painting, un-click and then it processes - it's a minor but significant difference in UX.
Very cool! I've seen this technique before in research papers and their videos, but not as actual software.
The one suggestion I would make is to follow Photoshop's shortcuts. Space-Drag to move the image, [ or ] to change the brush size, scroll wheel to zoom (already there), and letters to select the various brushes.
Pretty neat! http://imgur.com/p1B9Hng But sharing the shift key as the hotkey for erase & panning made Windows's "Sticky Keys" dialog appear a couple times.
Ah. Good point. We recently changed the shift key from "hold to pan" to toggle between pan / erase. Should probably change it to another key now that the operation is different.
Seems that the same tech could be reapplied to giving depth of feel to images by giving an option to unfocus the background instead of just deleting it.
Filters could also be a consideration for future (ie. popular things like black and white background with a color foreground)
It's doing the weirdest thing for me, I get this[1] for the preview, but I get this[2] when I download it. Other than that it works better than I expected for non-real life stuff.
We are having trouble connecting to the server. The following tasks have been unable to complete:
Connecting to worker "Unable to connect to the worker." Waiting"
I first thought that there is still the same problem but then I saw that the web socket worker is using 9006. Our company network is blocking unusual ports. Any chance to move that to 80, 443 or 8080?
This is seriously great and very intuitive to use. Two small improvements : it would be great to be able to change brushes (at least size). The workaround is zoom/draw/unzoom which is a bit awkward. The second thing is to add an example on the landing page. I didn't take the time to try it on a picture of mine until I read the comments (and saw screenshots) here. HTH, keep up the good work!
Pros: Doesn't require the fg/bg coloring step needed here, just figures out your FG without intervention.
Cons: Have to register for an account to access it currently, but can use FB or Google to register
Thats cool, but still it needs a lot of work/features. I tried removing the background using the purse picture in the pictorial, and it wasn't as easy as it was made out to be and I wasn't able to achieve the same results. Maybe I'm missing something, but the fixed brush size makes it pretty useless at this point.
Also, one of our usability headaches is that people draw OUTSIDE of the lines - any pixel marked green WILL BE IN THE RESULT.
Now, that's not how people actually use it at first - they "just scribble" (which our example might exacerbate), and then accidentally go outside the actual lines => not good.
Will need to figure out a way to make that more clear.
I copied the marks in the example and besides needing to tell it to include the cream color under the strap, it worked exactly as shown on the homepage, so maybe just add that extra red there in the example.
Unfortunately it is freezing up on me using an image with a gradient at about 90% of the initialization.
Could be related to the image color profile. Noticed the same issue when I uploaded a jpeg image with a CMYK color profile, with RGB images it worked normally. I've had the same trouble when using Java's ImageIO to read images. There were some workarounds, but I didn't find a robust one. Ended up using a preprocessor to normalize images before reading them with Java. Would love to know if someone knows a better solution.
Really nice. First time I tried it I carefully shaded the red and green areas and it worked ok. After looking through the comments here though I clicked green once on the part I wanted to save and then 3-5 red clicks in other areas and it worked flawlessly. Great job.
Interesting, but shadows, brown and blacks, tend to trip it up. Was playing with a photo of my car that is really difficult to break out because the rear bumper is pretty dark in shadow and blends to the driveway.
I've been trying to use the demo for the last... 30 minutes or so, but after I upload the image the "Initializing" progress bar in the "Booting" window always freezes around 90%.
Hmmm. That's not ideal. I added some more servers, so hopefully it can at least connect to the worker now. Sounds like I have some JavaScript debugging to do. (I'm one of the devs)
Really nice work! I tried a fairly complicated image and it worked quite well.
Couple bugs:
1) Chrome on Win8 consistently gives me the Network Error dialog when trying to upload
2) Was able to get it working fine on IE10, but sometimes the background still shows up in the preview, even though it is erased correctly in the download. http://i.imgur.com/fSB4n4r.jpg
We all know these things never work as well as the demos. Well... this worked as well as the demo. Better, actually--I was literally finished before I knew it because I didn't see the real-time update on the right at first. A deceptively simple UI and beautifully implemented. Useful, well conceived--I'm quite impressed.
This is really nice, it would be cool to add a rotate button - the image I picked at random to upload I think was rotated via EXIF.
Also, it'd be really cool to have a "simplify lines" type feature, I was using a fairly dark photo and the lines were a bit jagged - but generally the foreground item will have smooth edges.
Adding a "step-back" button (even if it was only a few states), and the ability to increase or decrease brush size would greatly enhance this. Great start though, and something like this is much needed for a lot of people who may not know an image editor, or don't want to open it for something simple.
Tried removing a background from a stock car photo. It involved quite a bit of precise mouse wiggling and the end result was still pretty sloppy looking. I assume there are cases where this works well, but it would probably help clarifying what those are before inviting people to try the tool out.
Cars can be very tricky - the lines need to be very sharp, so there's no tolerance for blur. I used to mask and recolour car bodywork in Photoshop as a freelance job, and hand-drawn spline paths with the pen tool were the only acceptable solution for professional use.
That's a very neat solution! Really like the UX too.
A long time ago I had a crack at it because I was working on an app that requires image background removal.
Bookmark manager, speed, GUI configurability. It's pretty much the difference between a yacht and a tricycle.
I won't even let Chrome on my sytem directly (i.e. outside of a VM), because I'm tired of it downloading updates with some super needy background service, keeping 100s of megabytes around for no good reason, and no way to turn it off. At least Mozilla products ask. For that reason alone Opera and Firefox play in an entirely different league than Chrome or Safari. Fuck features, let me see your heart... Firefox I use and love, it's just not my main browser because only Opera has bookmark management worth calling that.
I got a bunch of "Failed to get a worker assigned: undefined" for a bit, but once that stopped happening and I figured out the zoom, it was very impressive: http://i.imgur.com/WT4bH5M.jpg
Wow! That's amazingly easy for good-enough results. With some general UI slickening up (good suggestions in this thread), this could save a lot of people a lot of time. How have Adobe not wrapped this into Photoshop yet?
Just re-deployed the backend. It should reconnect automatically, but that's not been tested to 100%. Reloading the page should resolve it if it doesn't auto-recover.
(some memory woes on the backend as well, so restarting is having some hickups. Well, well)
Another vote for space-to-drag. Though, I guess us photoshoppers probably will continue to use photoshop for this sort of thing so our muscle-memory shortcuts might not be so important :-D
Thanks!
Colors aren't a good way to portray information accessibly.
I just helped dto implement color blind support in his game 2x0ng[0][1] and we settled on the solid colors being hashed[2] to allow people with all forms of color blindness access the game.
You can get a smaller brush by zooming in using the mouse wheel. That way the brush is always an appropriate size (small when you are zoomed in for detail work, and large when you are zoomed out for the broad strokes).
Worked really well for me. One issue though: after doing a few of these and returning the main page, my uploaded images look broken. Anyway, great job!
Eh, maybe you don't use Photoshop much but this barely similar to the Magic Wand in any way other than selecting a part of an image. Magic Wand doesn't give you a nice live preview of the alpha channel for starters.
Yeah, it actually works really well on vector art. If you're into that sort of thing, check out the sick bg color removal and alpha channel "recovery" by zooming way in on the boundaries.
Regarding the segmentation phase, we were not satisfied with any existing algorithms because we wanted the tool to be as real-time as possible (for up to 4 megapixel images), so we rolled our own segmentation algorithm that has some similarities to GrabCuts (not GraphCuts).
But the binary segmentation is really only a small part of the technical challenge. You also need to figure out how to anti-alias the boundary, and remove the background halo. Simply feathering doesn't do the job unless you also erode the foreground first, which we wanted to avoid. So we developed a custom sub-pixel background-removal and anti-aliasing algorithm.
I think you guys did a great job. This is a very tough problem (speaking from experience) particularly depending on the complexity of the image. One problem we had to solve along the way for our startup is similar, but focused only on fashion related images. Quite tricky particularly if you want something automatic. Anyway cheers! :)
There's a great website http://alphamatting.com/ that provides a benchmark dataset and a comparison of a lot of influential methods.
I applaud your work in making a real time and interactive matting algorithm! One day it might be also as accurate as state of the art methods. You guys should write a paper about your method.