I was going to interview at LandingAI. I was asked before the interview to install a spyware browser extension to monitor my traffic to detect if I was cheating during the interview. I respectfully declined and didn't have that interview.
Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad.
> Wow if you can “cheat” during an interview - meaning either that they’re asking trivial, google-able stuff or that they’re so bad at interviewing that they can’t tell if you actually know your stuff - then their hiring process is pretty bad.
Not necessarily, at least on the first point. Someone could be getting coached.
A few years ago, a coworker of mine hired a contractor onto his team and was convinced the person who actually showed up was not the person who he interviewed (over the phone). He also thought the guy who did show up was getting a lot of help day-to-day from somewhere. The guy was a contractor, so it wasn't a huge problem because we could drop him quickly, but I would have never expected someone would do anything like that. However, it kind of makes sense as a scam: be a decent developer, get a stable of unhirable incompetents, and rotate them through companies while taking a cut of their salary.
You cannot really prevent those kinds of cheats. Even if you use the most insidious spyware a coach can advice the interviewee from a different device.
The only way to prevent those kind of scams is to put all employees in probation for the first months of work and fire them if they don't perform, like it's common in the UK.
I once heard where a dev literally offshored his work and had entire teams working on his tasks. He was employed by multiple companies at once and paid a small fraction of his combined pay to offshore team.
Eventually he got caught trying to manage all this
~80hrs on topic A squeezes the available time for being acquainted with the rest. [Edited because there was little way not to make the former formulation read, unwillingly, nasty]
Some of us believe instead on the advantage of being a polymath, (also) to be able to export wisdom from other contexts into the current work.
Also in terms of the proper ground to facilitate innovation.
It's literally our job to not just assume the possible solution that rolls off the top of our heads might not be the most up to date / best practice and to research it
Agreed. A decent interviewer can also determine a person’s understanding of a topic by simply talking to them about it. IE why did you build a model like this? What diagnostics did you use? Have you tried ____ before in your career?
I'd just note that if pushed by circumstances (if one was willing to be interviewed in spite of their ways), the interview environment could be (would be) on a throwaway virtual machine...
Possibility which, by the way, makes the interviewer's cautionary move generally useless.
It's fairly easy to augment a video stream to paste on eyes that always look in the direction of the camera. It's the digital equivalent of glasses with eyes on them[1].
I suppose if you’re clever enough to set up a VM in order to evade detection, that’s a pretty positive aptitude signal in its own right (though pretty negative on the behavioral/ethics side).
My understanding is that they are trying to automate the data preparation steps that seasoned ML practitioners are doing anyway today.
The fact that he tries this in manufacturing makes the case stronger. In most manufacturing companies you do not have access to top ML talent.
You have Greg who knows python and recently visualized some production metrics.
If we could empower Greg with automated ML libraries that guide him in the data preparation steps in combination with precooked networks like autogluon, then manufacturing could become a huge beneficiary of the ML revolution.
Greg probably also knows SAS and AMPL, and has a good knowledge of ops research, which is within stone-tossing distance of whatever ML is pretending to be this week.
After 15 years of experience with SAS this sounds to me like saying "knowing how to write and having a pen makes you to a poet". But it depends on how far you can toss a stone...
OR is perfect when you can describe explicitly what the decision space is and what the restrictions are.
ML is great fit when you want to identify and use patterns. Quality control with machine vision is a good application for ML. NLP for PDF documents is a huge field for manufacturing as well. Companies have so much data in email attachments that they do not currently take advantage of.
> OR is perfect when you can describe explicitly what the decision space is and what the restrictions are.
As opposed to having to figure it out later from the outputs of a black box?
> Quality control with machine vision is a good application for ML.
I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc.
> NLP for PDF documents is a huge field for manufacturing as well.
NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes?
I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
> As opposed to having to figure it out later from the outputs of a black box?
Not all problems can be formulated as a set of explicit equalities, constraints and variables (e.g. machine vision). If explicit modeling is an option, of course you should do it. I am seeing efforts to try reinforcement learning on systems that we know how to describe with equations, and of course the results are laughable compared to the traditional methods.
> I can't imagine CV could be an actual replacement for actual SPC in many industries. There's a reason we need to take samples and stress test, analyze composition, etc.
In one big manufacturing company they were using Machine vision and a cheap web camera to control flaring. Could they do it with fancy sensors instead? Of course, but it would be more expensive, and they never did in the past.
Another manufacturing company is using machine vision to raise an alarm if the door of a cargo car of a train is not closed after loading. Could they install sensors in all of the doors of the train instead? Sure, but it would be cost prohibitive.
>NPL could be big everywhere... if it provides actual value, which is not a given. ML has a lot of tangential applications (you could also say, better forecasting), but how will directly improve manufacturing processes?
In manufacturing we have multiple people opening pdfs from emails to copy contract numbers to excel spreadsheets. Others are getting orders in emails and then type them in SAP manually. I think that these tasks can be automated specially with the recent versions of NLP networks.
>I apologize for being abrasive, but I'm so tired of cs people descending upon all industries, plugging shit data into pytorch and doing shitty ML like it will automatically add value. Even more so in industrial engineering, which in my experience is full of people way better at math than computer scientists and requires a deep understanding of the product and the manufacturing process.
All is good :) There has been a lot of unsubstantiated hype in ML, made even worse by big consulting companies and cloud providers who just sell the hype.
There is a significant amount of research from the field of computer vision before ML even existed. It was quite robust as well within certain constraints. Those techniques simply did not generalize anywhere even close to as well as deep learning.
However, that said, in a tightly controlled environment such as a manufacturing line trying to spot defects I would imagine they would have a good chance at performing a lot better than deep learning.
A lot of the advancements in deep learning have also come out of ideas from that research. While they didn't use the techniques directly, there is a lot of knowledge that we'd be lost without.
This is one thing that scares me about ML. We are losing research into the fundamental physics/science to deeply understand these things and instead just throwing models at them.
I work in this space: most "cool" ML is useless, and stakeholder are very skeptical of new modelling techniques. It is a long slog of EDA and finding actionable causality. Deep learning, modern reinforcement learning... are not the best fit here.
However I have seen CV and NLP useful here and there... but it is not the bread and butter.
A tangent, if you have time: where would I go for a primer on operations research and/or discrete event simulation?
My thought is that Goldratt's "The Goal" / theory of constraints is a useful way of thinking about optimizing throughput in a computer system. http://www.qdpma.com/Arch_files/RWT_Nehalem-5.gif plus an instruction latency table is something like a well modeled factory. (The Phoenix Project applies these principles to project management, which I think is a somewhat less useful analogy!)
I'm curious about applying existing tools to modeling things like: how will this multi-tiered application behave when it gets a thundering herd of requests? What if I tweak these timeouts, adjust this queue, make a particular system process requests on a last-in-first-out basis? Can I get a pretty visualization of what would happen?
Visual inspection in manufacturing is a very solved problem, especially in the AI field. The big bucks are in pattern matching anyway... it's a dumb company.
That is the problem with generalization and cop outs like these. It's no good to people in the field doing actual work where the devil is in the detail.
Big data is fairly important to a lot of things, for example I was listening to Tesla's use of Deep net models where they mentioned that there were literally so many variations of Stop Signs that they needed to learn what was really in the "tail" of the distribution of Stop Sign types to construct reliable AI
That is called transfer learning. You might only need to see one photo of a sign to identify it in real life (although arguably learner drivers take a while to notice signs) but that is only because you have been training on identifying generic objects since you left the womb.
You brain already knows how to select the most important features of a sign. The shape, the size and the color. You have also learned how to understand the text on the sign.
A new born baby does not have that ability.
This is applied in ANN as well. Transfer learning is using a pre-trained neural network, which has already learned identifying objects, and then using it to train on identifying a new, usually smaller, set of objects using, usually, a lot less training data. That is what Andrew is talking about in the article.
And the context. For example, self-driving cars need to account for "Pizza Stop" restaurant signage, placards stuck to telephone poles that say things like "Stop Cancer", stop signs retracted into the sides of school buses, signs with additional instructions like "Stop when lights flashing", road workers with handheld stop signs, and the unconventional stop signs you see in parking lots.
You can probably get pretty far by checking the proximity to the road, height, dimensions, orientation, what it's mounted on, and if the sign incorporates any other text. But you can't just scan some pixels for "red octagon with STOP on it".
> because you have been training on identifying generic objects since you left the womb.
We can go back even further - your genes carry information about the structure and function of your brain and this has been refined by natural selection over the course of human evolution. Humans don't start from scratch with randomly initialised weights.
I'm not sure "training" is the right way to think about it. Children don't train to identify objects, they quickly develop the ability to recognize objects and are able to correlate them with prior information that was retained and learned. Case in point: if you take a child born blind, give them the ability to see, they are immediately able to recognize and correlate objects around them.
I have to disagree. They spend an inordinate amount of time trying to understand what they see, taste and hear.
> able to correlate them with prior information that was retained and learned
This is what we call inference.
> Case in point: if you take a child born blind, give them the ability to see, they are immediately able to recognize and correlate objects around them.
Not everyone who is legally blind can see absolutely nothing, but people who have recovered from complete vision loss [1] have problems. Mike May [2] lost vision as 3 year old child and regained it in his 40s. Despite seeing for the first three years of his life, years after regaining vision he was unable to see in 3D or recognize people from faces alone.
Blind people do not lack spatial awareness, so being able to recognize objects with context if they regained sight with would not surprise me. There are blind people that can "see" with echo location using parts of the brain associated with visual processing [3] But for example in Mike's case, he was unable to recognize close family by their faces years after regaining vision, he needed additional context.
Many things we take for granted as being innate to the human experience, are in fact learned (trained) behavior.
> I have to disagree. They spend an inordinate amount of time trying to understand what they see, taste and hear.
To understand yes, however I meant "identify" in the sense of recognizing an object. I show a picture of an apple to a child a bit over a year old, hide it, put it in a basket of other things and present it back to them, they will be able to identify it. Or if you show an object to a child, then hide the object, they can realize that the object is no longer there, regardless of their understanding of what happened to the object.
> Many things we take for granted as being innate to the human experience, are in fact learned (trained) behavior.
I don't think it's that simple, I would say it's both. I don't question learning plays a big role in recognition, just pointing out that a large amount intrinsic knowledge also exists from early child development. Many of those cases where someone regains eyesight happens much later in life, at a time when their brains have largely matured to the point that neuroplasticity is pretty much over for them. Having someone's brain develop with almost no visual input at a young age is bound to mean that their visual cortex and its connections to everything else doesn't develop as it should.
From a quick search, it appears that to some degree children born categorically blind can recover all the way up to teenage life. [1] But indeed it's likely less effective than a younger child undergoing a similar procedure (which I don't think is really that rare: it's hard to diagnose vision problems at young age, and a lot of children who get necessary corrective surgery at young age turn out fine).
Is there some underlying point to this statement? It comes off as a passive dismissal of something but I'm not sure what. It might be helpful to directly state what you're trying to say so that other people can engage with it.
It doesn't seem like the other replies are having trouble engaging with it. Since we're giving each other advice, you should use your down votes instead of pontificating that other people's comments are "passive dismissals" when you don't like or understand them.
when you learn to drive you need [approximately 16 years training your visual system hardware that took 6 million years to evolve and] to see approximately one example and you're able to identify them all.
FTFY.
Yet Tesla have been working on both the hardware and software for 10 years? Amazing progress right?
I’ve been wondering about the limits of data-centric approach – there seems to be this implicit notion that more data equals better performing ML or AI. I think it would be interesting to imagine a point of diminishing return on additional data if we consider that our ability to perceive is probably largely based on two parts - sensory input and knowledge. Note that I’m making an explicit distinction here on the difference between data and knowledge.
For instance, an English speaker and a non-English speaker may listen to someone speaking English and while the auditory signals received by both are the same, the meaning of the speech will only be perceived by the English speaker. When we’re learning a new language, it’s this ‘knowledge’ aspect that we’re enhancing in our brain, however that is encoded.
This knowledge part is what allows us to see what’s not there but should be (e.g. the curious incident of the dog in the night) and when the data is inconsistent (e.g. all the nuclear close calls). I’m really not sure how this ‘knowledge’ part will be approached by the AI community but feel like we’re already close to having squeezed out as much as we can from just the data side of things.
Somewhat related, we have a saying in Korean – ‘you see as much as you know’.
It does in general, but what is elaborated and how? Structuring patterns is not the same as "knowledge" (there are missing subsystems), and that fed data is not fed efficiently, with ideal efficiency - compare with the realm in which "told one notion you acquire it" (this while CS is one of the disciplines focusing on optimization, so it would be a crucial point).
I have a feeling that too much knowledge might slow learning process as it's harder to spot/test observe steepest gradient. At least that's how it feels intuitively from human PoC. From computation that would be just little more computation but I guess would mean slower convergence also. Taking math as more extreme example it's hard to understand something complex unless you understand basic algebra.
Anyone knows if this might be true mathematically speaking? Does order of data matters?
Can't you consider that knowledge is a function of previous data? In your example, the 2 individuals actually didn't receive the same amount of data because the English speakers received data previously that allowed him to build some kind of "knowledge" that allows him to solve specific related tasks (understanding a spoken sentence). This would be the equivalent of transfer learning where "knowledge" is a model trained on previous, more general, data.
Nope, it's never a function of data -- because data is always ambiguous. It is never possible just to infer the conceptual model of the data from the data alone.
Animals solve this problem by having bodies and moving around. It is that we take the bent stick out of the water which allows us to impart a theory to the "data" we receive... a theory implicit in our actions.
Since we are causally active in the world, sequenced in time, and directly changing it -- our bodies enable us to resolve this problem. The motor system is the heart of intelligence, not the frontal lobe -- which is merely book-keeping and accounting for what our bodies are doing.
Yep. Perhaps what I should have written is that much our knowledge is tacit in nature. We implicitly understand that things we can sense are limited projections of the real world and are able to derive mental model of what that reality might be more or less. Chomsky also talks about this notion of separating mind and body as an incorrect approach to understanding human intelligence which I think aligns with your view.
The notion that 'all we need is data and data is all there is' seems to summarize a lot of ML sentiments.
Right. Such as, extras in movies. Because, "350 million public faces" does not seem "cognitively digestible", but it could just be «350 million images» including large variations of the same faces, consistently with the "frames of video" idea.
data quality is important. every ai project i've worked on has started with visualizing the data and thinking about it.
it's easy to get complacent and focus on building big datasets. in practice, looking at the data often reveals issues sometimes in data quality and sometimes scope of what's in there (if you're missing key examples, it's simply not going to work).
Glad to see the term ML being used more often than AI in the comments as it looks like most "AI" models are trained for image classification.
Having said that, the idea of "doing more with less" sounds interesting and I wonder what it means exactly. Does it mean taking a dataset of 50 images and to create 1000s of synthetic images from it?
Yeah I was very interested about that point in particular. I think synthetic data is one of the ideas, but I got the sense that he also means helping to identify what makes a data set good, even if small. It looks like Andrew Ng is developing a platform for automatically detecting whether a dataset is suitable and, if not, what are the steps to improve it. A sort of automated ML consultant, allowing you to sell capabilities much cheaper than if you needed to consult an actual expert.
Pretty interesting. Mr. Ng claims that for some applications having a small set of quality data can be as good as using huge set of noisy data.
I wonder if, assuming the data is of highest quality, with minimal noise, having more data will matter for training or not. And if it matters, on what degree?
In general you want to add more variants of data but not so much that the network doesn't get trained by them. Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference) and prefer more of those.
Now, why not simply add everything? Well in general it takes too long to train.
> Typical practice is to find images whose inclusion causes high variation in final accuracy (under k-fold validation, aka removing/adding the image causes a big difference)
How do you identify these images? It sounds like I'd need to build small models to see the variance but I'm hoping that there's a more scientific way?
For industrial application, there are already mature systems based on CV. For majority of those applications, there is no need for deep learning or multilayer CNN. Shocked to see Andrew Ng talking like a marketing guy.
What's the role of these tools? Can't a developer just write the code to get those data points?
At a first glance it seems like the hassle of integrating such a product into an existing ML codebase/pipeline is larger than solving the problem by hand.
What I mean is an annotation tool that interacts with the model itself in such a way that it will present to the user exactly those training examples next that will have the greatest impact in helping the model learn. So an annotation tool that provides a user interface for annotating data quickly (with keyboard shortcuts etc.). And looped into inference through the model to be trained, so you always get presented with the very training example that, out of the ones available, the model currently would be most unsure about.
> simply saying "we should make this better thing" isn't enough.
Besides the references to his company which has customers and a product that already works on these principles the literature currently shows that this is very much possible if you dig into the correct niches. Besides the SOTA in few-shot and meta-learning it is possible to smartly choose the correct few samples for the network that yield the same results.
It has also been my primary focus for the past 5 years and the core of the company I founded.
They might get new SOTA because the metric is accuracy, but if the metric was accuracy weighted by sample efficiency, then SOTA would look a lot less impressive.
Simplest way to weigh by sample efficiency: multiply accuracy by ratio of test set to training set sizes. Everyone's training/testing on 80/20 splits, so everybody's SOTA would go down by 3/4s.
