So I've actually tried to implement Eulerian video magnification myself, albeit using Python, because I believe when I first looked into this, ~5 year ago, the code was MatLab only.
I was really excited about it, because, as a cardiologist, I immediately thought we could use this to identify irregular heart beats in the patient waiting room, for example. This was pre Apple watch etc. I was thinking people could briefly sit in the chair in front of the camera if they are willing to be recorded, as a quick screening whilst they wait for the doc.
Unfortunately, I found it incredibly difficult to reproduce their results, even using similar data. I remember videoing my foot as I have a relatively prominent posterior tibial pulse, which in good light I can actually make out visually (but not in the video I took, as intended). The Eulerian magnification didn't seem to do anything.
Has anyone else had better luck? I did wonder if maybe I had to use specific video capture equipment, certain frame rates or filters, but if so that wasn't clear from the documentation?
It's been 8 years since this paper came out, and yet I still haven't seen it being used in the 'real world', so I am slightly suspicious that I am not alone in my experiences.
I played with it 7 years ago for a hackathon -- not to transform the video, but just to extract pulse data from a face cam. For that purpose, even without optimization, it did work fairly well, though when we went to demo it live, the frequency of the projector lamp interfered with detection.
https://github.com/MegaByte/web-webcam-pulse-detector/tree/r...
I did run across this company recently, which seems to be doing something similar: https://www.binah.ai/
A colleague and me implemented it for a university lab course a while ago using a combination of C++ and Python. (Used C++ for the processing and Python for the UI and filter design)
One important thing we learned is that the filter design is a very important part of the whole thing. We used scipy.signal[1] for filter design. One thing that was very noticeable was that even small IIR filters performed significantly better than FIR.
I thought we put our code online a while ago but looks like we didn't. If you're still interested I might have a look if I can put my stuff online later today. Not that it's all that fancy anyways. The C++ part is pure processing and the python interface just puts a gui around it for filter design and calling the application.
My friend and I hacked on something like this for a while, and eventually got something pretty accurate for measuring heart rate. Shipped on iOS as [0].
It did indeed require a lot of additional tweaks and benchmarking. Honestly don't think any part of our original algorithm or the tricks in the paper made it into the final version. Just doing a Fourier type thing works rarely (~20%), but the principle does work; you can get it working ~90% of the time. That said, I'd be surprised irregular heartbeat were doable in a short recording period. But I'd love to be surprised!
I did a Masters project on this sort of technology and yes, I think it makes fantastic demos but hasn't found a killer usecase in the real-world (publicly, anyway.)
Their techniques are at least patent-pending if not fully patented by now, which is going to limit adoption.
A friend reimplementated it in python and it worked pretty well for analyzing cardiac liver movements[1]!
I don't think he is willing to share the code, but my guess is that a python implementation should work just fine, with very nice results :) (I saw the results applied in his research on videos with impressive results). It took him something like several months to have a code without major bugs.
[1] Hahn, S., Absil, J., Debeir, O., & Metens, T. (2019). Assessment of cardiac‐driven liver movements with filtered harmonic phase image representation, optical flow quantification, and motion amplification. Magnetic resonance in medicine, 81(4), 2788-2798.
You are free to contact him! I'm not sure he is not willing to share.
Sharing code to the public in research depends on several factors, it can get open sourced when the thesis is done, it may depends on who is paying the research (eg: some industrial PhD are not public), long term plan exist to publish the code but there are only 24h a day and a whole lot of work to do, the author/uni may want to make a business after the thesis, and a lot of other parametres. It gets even worse when the code is used in medical applications as a validation process with all the relevant authorities has to be done. This can take a lot of time and freeze the code in the waiting for a certification. Also sharing code between researchers is very usual, it doesn't mean the code has to go public. In this case I think it is part because the original matlab code is patented, research is ongoing and as it is research code it is not necessarily public ready.
It doesn't mean he will never release it.
If you desperately need it, the matlab code is available, and has the same functionality, you are free to adapt it or use it directly (for research purposes).
I must say I don't like that the original code is patented, but I see hundreds of reasons why the authors patented it even if I don't particularly agree :)
For detecting a pulse from the face, I was able to get good results using pixel tracking and spatial averaging with no fancy math. However if illumination at all changed due to motion, it threw things off considerably without a more work or band passing around the heart rate frequency. (assuming too much in my book)
I once tried to reproduce a webcam PPG paper, but not using Eulerian video magnification.
The pipeline started with face detection, then found the mean colour value of a rectangle of forehead. It then did some DSP and returned the pulse frequency.
It worked, but only when the person held extremely still very close to the camera, with auto exposure turned off.
Disappointed with how badly mine worked, I took a look at some open source webcam PPG projects to find out how they were working. It turns out they were pretty similar, except most of them were aggressively filtering the pulse signal to be in the correct range, without really checking there was already a signal there. Of course by taking white noise signal and filtering it to a certain frequency range, you'll get a signal in that frequency range.
I now believe that to make video PPG work, you need some of (1) a very good camera, i.e. not just a low end webcam (2) quite a long recording, i.e. 10s-1m (3) a controlled environment
This or similar tech would definitely be integrated into high end baby monitors if it worked. State of the art is putting a wrist or ankle band onto your baby though AFAICT.
This sort of video motion magnification seems like it could have a use in cinema movies, in order to accentuate motion that the director wants the audience to notice.
Even in little videos on Instagram it could be a cool and eye-catching effect.
For the unfamiliar, 'Eulerian' and 'Lagrangian' are basically just names for two different frames of reference.
Eulerian sampling is like dotting the countryside with weather stations. Your measurement data shows the weather at each of those fixed points over time and from that you can understand what was happening.
Lagrangian sampling is like sending up weather balloons. Your measurement data shows the weather around the balloons as they float along with the wind. That will give you all the same information, but from a different perspective.
The Eulerian perspective for weather makes it easier to understand what's happening at a specific location, but harder to understand what's happening to a specific particle of air. The Lagrangian perspective is the opposite.
There is an explanation of this in the paper, but IMO, it's a little more difficult to follow than the classic weather analogy. Just wanted to demystify the name a bit.
For those interested in the pulse rate extraction, there is actually a branch of biomedical engineering/computer science called "remote photoplethysmography" (rPPG).
I worked on it a bit at some point, it does work pretty well assuming you don't move that much and the light isn't too bad. As you can imagine, it's more difficult to estimate on non-white faces, particularly very dark skin tones. All the algorithms (at least until 2016) differ in the way of combining the RGB signals into a pulse signal by making different assumptions.
If you are interested more, Philips in the Netherlands are particularly active in this research domain, one of the main application being non-contact pulse estimation in hospitals for example.
What a toxic comment. He is simply referring to the technical difficulty associated with darker skin tones. Of course we should strive for making it applicable for everyone. Sometimes physiological variations get into play; sometimes this means a method isn't adequate, or you can use different methods for different persons.
The technique of analysing light which penetrates the skin, then reflects, then penetrates the skin again on the way out? Easier with skin that allows more light through.
You can turn up the light source to make up for the loss due to skin pigmentation, though this can require a very bright lamp for particularly dark skin.
William Freeman's group does lots of cool things in computational imaging and sensing, as well as ML. Fun fact: his group contributed to the formation of the first black hole image. Dr. Katie Bouman was his PhD student.
The whole field of remote photoplethysmography (rPPG) is quite interesting. I have implemented a couple algorithms, but they are quite "brittle" if you will, and only work well with certain videos. Many recent papers have introduced deep learning techniques. Hopefully these techniques are more robust to movement, skin color, lighting, and other natural conditions (unlike the usual computer vision/signal processing algorithms previously used).
I’d read that a blow to the chest during a specific phase of the heart waveform can stop the heart. In theory, this video processing algorithm, coupled to a computer fired beanbag type firearm could result in heart stoppage that would look like a heart attack.
I'm shocked this hasn't surfaced as a lie detector app on the iphone. I get that it doesn't work very well, but it seems like it would sell. Maybe offer some calibration steps.
I was really excited about it, because, as a cardiologist, I immediately thought we could use this to identify irregular heart beats in the patient waiting room, for example. This was pre Apple watch etc. I was thinking people could briefly sit in the chair in front of the camera if they are willing to be recorded, as a quick screening whilst they wait for the doc.
Unfortunately, I found it incredibly difficult to reproduce their results, even using similar data. I remember videoing my foot as I have a relatively prominent posterior tibial pulse, which in good light I can actually make out visually (but not in the video I took, as intended). The Eulerian magnification didn't seem to do anything.
Has anyone else had better luck? I did wonder if maybe I had to use specific video capture equipment, certain frame rates or filters, but if so that wasn't clear from the documentation?
It's been 8 years since this paper came out, and yet I still haven't seen it being used in the 'real world', so I am slightly suspicious that I am not alone in my experiences.