The common saying is "more data usually beats a better algorithm". However, this is only for short term progress. Long term progress in the field of AI clearly requires better algorithms, and doing more with less data is exactly the kind of problem that a startup in the field could solve with a clever idea.
That said, these startups will have to be extremely research heavy — how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? And what kind of product could theoretical research produce in the short term that would keep a startup afloat long enough to come up with ground breaking ideas?
Vanilla deep learning is so far ahead of what most companies use for analytics, that very little research is needed to offer a product with significant advantages over what's in practice now. In fact, I'd say most ML/DL/AI startups don't need to do research at all. They just need to spread more widely what exists now.
I disagree with Andreessen that Google's Tensorflow propaganda on Udacity is "opening the kimono" or having a large effect on AI. TF isn't better than pre-existing tools in many ways. And the ways that Google uses it aren't really transferable in an online course, since Google does have the data, architectures, infrastructure and talent -- some of which it can't share by definition, and some of which it won't share, for obvious reasons.
The smartest startups outsource research to universities and government funded bodies. It's a way of derisking themselves as investments. That's no different for AI. Theoretical research and product development are almost (but not quite) totally different things. Which is something that graduate students who think they want to do a startup should remember, because if they love research, they probably won't have much time for it.
[Disclosure: I'm associated with a rival DL library that I won't plug here.]
> Vanilla deep learning is so far ahead of what most companies use for analytics, that very little research is needed to offer a product with significant advantages over what's in practice now
But by "vanilla deep learning", I think you are referring to this vs research into new models and algorithms. I think "research" in the context of "research-heavy start-ups" is not about new algorithms, but about finding ways to use even "vanilla deep learning" on their available datasets. Quite a lot of papers at conferences, for example, are not about new algorithms, but about new ways to use old algorithms.
You're right on many points. My observation is, when it comes to deeper Ai that trends towards AGI, the private sector seems lost in terms of the big ideas and applications. The VCs are also lost and have no idea on how to appreciate fundamentally groundbreaking R&D development ventures.
To be honest, most of the funding and deeply interesting work for ground breaking Ai research lies in the public and defense sector. They have more of an idea and vision for what Ai can be applied to than the private sector. Leaps and bounds beyond.
In the private sector, I get asked questions like 'Can it be used to predict if someone will click on an ad'. In the public sector, I get a 40 page technology acquisition pdf outlining a forward thinking application of Ai.
People forget that self driving cars arose from Darpa research. The latest innovations in computer security utilizing Ai came from a Darpa competition. IBM's neuromorphic chip : darpa funding. Darpa and other defense programs also have a slew of Ai development projects open to researchers.
Ultimately I find this a bit ridiculous given how vocal some individuals in the tech field are about the 'dangers of Ai'. They harp and harp about what could happen if more capable Ai gets developed but you don't see their money anywhere near the groups doing deeper development in Ai.
Lastly.. Yes, better solutions and more deeply inspired approaches to Ai will trump hoards of data and compute power any-day. Thank goodness for the public sector pushing forward fundamental research and development in Ai.
> Ultimately I find this a bit ridiculous given how vocal some individuals in the tech field are about the 'dangers of Ai'. They harp and harp about what could happen if more capable Ai gets developed but you don't see their money anywhere near the groups doing deeper development in Ai.
...because if you think AGI is dangerous, you should fund groups trying to develop it? I am puzzled by the presentation of this proposition as if it is an absolutely obvious sequitur requiring no defense, such that failure to follow through on it is "ridiculous".
Yep.
"The best way to predict the future is to create it"
Don't stretch your mind to far on this one. If someone or various groups develop AGI and you have no stake in them, you will have no say in how it gets used. For once, people need to be honest with themselves about this.
You're not going to be able to dictate from a third party consortium of PhDs how someone should run their company or develop their software.
I agree, but it's the same problem with all basic research. There are no guarantees of results and it can take a long time. So much of it gets done with public resources.
In fact much of the deep learning advances we (including Google) enjoy today were done with Canadian (University of Montreal) and European (IDSIA) taxpayers money over the last couple of decades
More should be done to highlight this. The persistent fanboy like culture, among those even in tech, which centers on only a handful of popular corporate names who serve as system integrators after the fact is dishonest and disingenuous in my opinion.
Given that the public funding of basic research most always results in the ushering in of new technological paradigms that make corporations tens of billions if not hundreds of billions of dollars, more should be done to make sure they pay back into the system that ensures their constant success.
>You're right on many points. My observation is, when it comes to deeper Ai that trends towards AGI, the private sector seems lost in terms of the big ideas and applications. The VCs are also lost and have no idea on how to appreciate fundamentally groundbreaking R&D development ventures.
DeepMind was VC funded before Google acquired them. A number of AI/ML startups would like to pursue AGI in the long-term, but why not make revenue along the way? Still, there is at least one startup - Vicarious - focusing purely on long-term research towards AGI that VCs poured plenty of money into, and there are probably others.
Deep Mind was funded by Elon Musk as well as others in its earlier stages. Elon Musk is one of the individuals who speaks out as if he is wildly fearful of where Ai is going. Fundamentally, an individual has no control over what an AGI development group does unless they're invested in them. There are many notable groups beyond the handful that get the spotlight. They are for instance : working on a Conscious Artificial Intelligence. That is not a 'future goal' of theirs. That is the one and only goal of their efforts. Who do you think will get their first?
>A number of AI/ML startups would like to pursue AGI in the long-term, but why not make revenue along the way?
That's exactly the dilemma that many capable researchers faced. Some chose to make money along the way. It shapes the way you then go about solving AGI down the road. It saps up your resources, imagination, and potential and it redirects it to shorter term thinking. Being in the space, there should be an understanding of Short-term vs. long-term thinking based on a reward system that targets short-term success.
> Still, there is at least one startup - Vicarious - focusing purely on long-term research towards AGI that VCs poured plenty of money into, and there are probably others.
I'm familiar with Vicarious. You are correct and they notably publish far less details than others. They are funded by top names and are probably the least mentioned when people mention Ai. Why, given the impact that AGI can have, aren't more groups funded? A solution is seemingly right around the corner... Do people not see this?
>> outlining a forward thinking application of Ai.
As someone miles away from the field, what are those forward thinking applications (ie things that are within the reach of MI today but are basically ignored by private sector)
Yes, do you know of the number of groups who are actively developing AGI? who have made steady progress over the years? I see no big names invested in them. Why would someone try to create your own group having no background in development of AGI? Seems a bit off the mark no?
Most AGI focused ventures are not even in the U.S. A prominent one is in China. Another is in Europe. The ones in the U.S are privately funded beyond the big names. Furthermore, where is the U.S's equivalent of :
https://www.humanbrainproject.eu/ ?
Where are the seasoned software engineers in the ranks at these highly funded silicon valley Ai companies? I typically see a roster of 'big names' and PhDs... Whereas, I look at the companies specifically pursuing AGI and I see a whole range of individuals.. PHD neuroscience, Senior game developer, Robotics Engineer, Senior firmware developer, Guy from down the hall who can code circles around silicon valley's most decorated engineer, etc etc.
Look at the Geohot story, that's the kind of individual and company who ushers in a new paradigm ....
If the solution requires you to 'think different', how do you expect to achieve it by padding your company with a bunch of individuals who are all centered on the same techniques from academia?
>do you know of the number of groups who are actively developing AGI? who have made steady progress over the years?
I do not. It seems like researchers who call themselves AGI researchers have made no more significant progress than researchers developing specific analysis techniques. Do you have a list of these groups? I would be very interested to read about their approach and progress.
On the human brain initiative it seems like their is such a huge gap between AGI and current techniques that these HBI projects (and the US response[0]) are a bit misguided into giant neural simulations.
> ... companies specifically pursuing AGI and I see a whole range of individuals ...
I completely agree this is the necessary approach to pursuing AGI. Who are these companies? They sound awesome and I would like to learn more. Are you talking OpenAI? Some other companies?
Geohot. Will check him out.
EDIT: oh, George Hotz. He is using deep learning[1]
Agreed deep learning will be a tool, but not the end of AGI. Companies padding with deep learning PhDs are looking more at specific tasks that can be tackled with deep learning. Also, I would argue that deep learning has become an umbrella term for all neural network based approaches (great marketing) and there are still great advances to be made with DL building blocks.
I'm not attempting to be a smart arse. However, I really think you should google it :). It is too common a trend for one to latch on to popular names as if they're the only ones doing anything important in a space. There are many cases whereby they're doing the least important work.
I mentioned George Hotz to highlight a capable individual with vision, passion, and capability who 'thinks different'. With many individuals, the capability is there. However, often times the vision, passion, or capability to think different isn't. Funny then that there becomes this hiring norm which attempts to find the 'most capable' individual and ignores the other more important characteristics.
Time will tell. I'm personally looking elsewhere beyond the names everyone mentions when it comes to development in this space.
I'm not attempting to be a smart ass, but you should back up your speculation with facts and references. Is the best research a secret?
Your post implied that you have some special insight into research groups that are doing AGI research and are making steady progress. I have done plenty of Google searches and not run across any notable AGI progress.
The one reference you provided was for a deep learning implementation, which is what you seemed to be dismissing.
> I'm not attempting to be a smart ass, but you should back up your speculation with facts and references. Is the best research a secret?
Yes. Given the attitudes of people in the space and beyond and lack of funding why shouldn't the more valuable and fundamental research be kept secret? They're taking all of the risk to pursue something that everyone is saying is impossible. Why would they publish details? If more funding and support would come through maybe they would. Otherwise, they're seemingly developing a fundamental AGI that has the ability to operate under its own free-will and mechanisms. It is being said that it's impossible... Once it's completed no one will be able to deny it at that point. Maybe the money will come then... Maybe the openness will too.
> Your post implied that you have some special insight into research groups that are doing AGI research and are making steady progress. I have done plenty of Google searches and not run across any notable AGI progress.
Yes and I will not disclose where my insights derive from. I'm sure people have their ways of discovering it nonetheless. Some groups and individuals are developing conscious self-willed software solutions. I'm referencing them. Is that not what 'life' which intelligence springs from fundamentally is? As will be true for AGI... Not programs that mimic human behavior or optimization algorithms.. Software solutions that are truly aware.
>Given the attitudes of people in the space and beyond and lack of funding why shouldn't the more valuable and fundamental research be kept secret?
If it's secret, how do you know about it? If you don't have specific knowledge, how can you claim there is "fundamental AGI" research done in secret, as opposed to it just not being done, as all appearances indicate?
>It is being said that it's impossible...
Nobody who understands recent neuroscience claims any such thing is impossible. We just claim it hasn't been done in the lab yet.
Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible - but if small data cannot give it confidence to confirm the existence of some subtile pattern, on what grounds is even an ideal learner supposed to believe in it?
For example, no smarts would ever confirm the Higgs in the tevatron in its 10 year run even though the machine produced thousands of Higges; its a subtle signal and you just don't have the sample size to be more than 3 sigma sure there's something there (nor where exactly).
What is key, however, is being able to leverage more than just explicit training sets, like with transfer learning and unsupervised learning, as well as making algorithms scale well with more data. The latter seems promising with deep learning, the former still a research thing.
"more data beats better algorithm" is obviously false when the algorithms don't scale with data as well, as deep learning has convincingly demonstrated, being just as mediocre as anything else on small datasets, but being leaps above everything else on sufficiently large ones.
> Truly doing much with little data may simply be impossible; its not a magical black box after all, but just as ideal a learner as possible
This is interesting, and something that needs further explored IMO. I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics.
We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.
>I've been doing research on extracting mutual information from noisy, shifted copies of a ground truth signal, and it turns out there is a crossover point where recovering the ground truth essentially becomes impossible (in an information theoretic sense). What's interesting is that this crossover point is sharp and it looks a lot like a phase transition that one might see in statistical physics.
That's awesome! Can you link me to your research? I've been writing a similar paper on the information theory behind probabilistic programming and (possibly) deep learning.
>We need more research that provides limits on what we are capable of predicting from a given dataset — an upper bound, in other words. This would let us know if it's worth it to spend time trying to get more predictivity out of a dataset, or if the data just simply doesn't contain sufficient enough information.
Definitely! Not only is the Shannon entropy of large or high-dimensional datasets computationally obscene to estimate nonparametrically, what we actually care about is the conditional entropy given an untrained model, and the degree to which we can reduce it by training the model.
The concept of an upper bound on prediction for a given data set does exist, I've seen it discussed explicitly in some mid-1990s universal sequence prediction literature. I haven't seen the term (or concept) used in modern predictive algorithm literature but I believe the term of art is "prediction error complexity" which is conceptually similar to concepts like Kolmogorov complexity in algorithmic information theory.
A lot of AI startups' best assets are their data which google also has plenty of. Major advancements in AI will come from less glamorous things like labeled data available for models to learn from.
The code without the data is useless. The algorithms are only part of the equation here...and giving them away without the requisite data is useful, but not the critical path from profiting from AI. It's also not going to advance AI much.
These AI labs publish subsets of the research they actually do (even if it is still a generous amount which is great).
We can even see this from OpenAI's gym efforts. These environments are creating fundamental infrastructure for pushing the boundaries on reinforcement learning.
That being said, research is a component of the problem, but even most "AI" startups just git clone some open source code and run something pretrained (say: opencv, kaldi for audio,..) and then wrap it in a nice gui.
The main things these startups focus on is delivery of the product just like the rest of these startups, very few are actually building novel algorithms. A lot of it is just them collecting data.
That being said, this is also why a lot of the novel research happens in the major for profit ad tech companies.
They have the data to do research on and they can choose what to publish and what to profit from in products.
> how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? And what kind of product could theoretical research produce in the short term that would keep a startup afloat long enough to come up with ground breaking ideas?
I think the short answer is, you don't. Between Google, OpenAI, Facebook, Baidu, Microsoft, IBM, and every other startup that is putting "deep learning" in their pitch deck to see what happens, there's enough money being pumped into the area. 90% of the results of that money being spent will be made available to everyone in the world for free. IE, if you're investing for competitive advantage, you will lose. If you're not already on one of those teams or studying the subject, I don't see any opportunity for at least a year, maybe 2.
To be clear, I'm talking about opportunities around starting a new company. If your business is consulting or you're working as an engineer for an existing company, there are many applications for deep learning, but none that stand on their own.
In 2 years, content generation will be a big thing. Like pandora with purely algorithmically generated music, or games with assets that create themselves in a way far superior to what we see in the current generation of "procedural generation". The big cos will have very good versions of Siri, but no startup is going to be able to touch that near term.
> how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need?
Tons of resources dont matter if they aren't looking in the right direction. A small shop can make huge progress just need to think "okay they're all doing it this way", and then do the opposite most crazy thing ever. Honestly we dont have the hardware yet. AI needs something groundbreaking before it turns into more than a clever analytical tool.
Yes, the data and the algorithms don't really matter unless you have the right product. Otto apparently was a group of Googlers who thought the product was wrong, and went for the narrower self-driving trucks market (before being acquired).
I imaginne that the winner of the self-driving car race will be determined by the product and not the technology per se. Tesla, Google, Uber, comma.ai, etc. all have quite different conceptions of what the product is.
>> doing more with less data is exactly the kind of problem that a startup in the field could solve with a clever idea.
It will take much more than a "clever idea" to overcome the reliance on data of the current AI state of the art- that is to say, machine learning.
If you think about it, machine learning algorithms are essentially clever search procedures for some optimum in a heap of data (that's optimisation algorithms, so about 90% of the field). You can always get smarter in the way you search, but unless you search far enough what you find won't generalise very well to unseen data. And if "unseen data" is the real world, you need immense amounts of data to get anywhere- hence, the reliance on big data.
What we need is much more than a clever idea: it's a paradigm shift. There's probably some sort of way to make a system that can learn from very few examples and at a very short time, like we do ourselves. Machine learning is certainly not that way. We need to find another way.
I'm not sure startups with clever ideas have a very good chance to just stumble on such an other way while looking for a way to do "Facebook with deep learning".
There's a lot of ongoing work in machine learning focusing on learning from few examples, online learning, more biologically plausible learning and so on. Most of it is done in the academia, and some of it is done at large corporations.
Startups generally have a short runway before they run out of cash, which isn't long enough to sustain this kind of exploratory work.
>What we need is much more than a clever idea: it's a paradigm shift. There's probably some sort of way to make a system that can learn from very few examples and at a very short time, like we do ourselves. Machine learning is certainly not that way. We need to find another way.
That's more like probabilistic methods, which also fight overfitting by regularizing (with a prior distribution) and allocating probability density in a smear around the training data rather than concentrating it as hard as it can (as optimization methods usually do, so to speak). And then hierarchical Bayes is what's used in many of the neat one-shot learning demos, and in theoretical neuroscience these days.
> There's probably some sort of way to make a system that can learn from very few examples and at a very short time, like we do ourselves.
Maybe. I wonder if we can pick up identifying platypus with only a couple of examples, because we spent a long time identifying bird or cat as a toddler. We have a complicated model of the world, and we can take short cuts, because we just identify the differences compared to stuff we already know.
You may be right, no one knows for sure. I think it's one of those P = NP kinda problems. Lots of clever people have thought about it for years. Maybe there's a trick. It seems unlikely there's a trick though. Absence of evidence starts to become evidence of absence.
>how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need?
You mean how does one draw people to graduate school?
That said, these startups will have to be extremely research heavy — how does one draw the most talented researchers to work for a small group of people with minimal pay over working for a large company with a high salary and all the computational resources you might need? And what kind of product could theoretical research produce in the short term that would keep a startup afloat long enough to come up with ground breaking ideas?