I'm one of the recipients of the AI grants, to support my work at fast.ai. I'm extremely grateful to a16z for their support. Here's some additional details based on questions I see in the comments:
My grant covered the purchase of a Scalar server from Lambda Labs, which allowed me to configure a system with 8 previous-gen A6000 GPUs, partly also thanks to NVIDIA who has recently given me 3 of those cards, and Lambda Labs who offered to provide everything at cost.
a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant. They couldn't have been more helpful through the process - I didn't have to fill out any forms (other than sign a single one page agreement), the contract was clear and totally fair (even explicitly saying that a16z wasn't going to receive any kind of rights to anything), and they wired me the money for buying the server promptly.
I'm sure you get this a lot, but thank you for teaching me ML - I am a hardware engineer/manufacturing person who learned to code in the early 2010s and it's been immensely helpful in my career. Learning ML feels like V2 of that - picking up a new skill that is going to end up being useful in so many places. You made that journey much easier and more accessible and I very much appreciate it.
(No need to respond at all - just wanted to pass along the gratitude!)
Just in case anyone hasn't surveyed the landscape lately, the distinction being made here is between "RTX A6000" (Ampere architecture; GeForce 30-series equivalent) and "RTX 6000 Ada Generation" (Ada architecture; GeForce 40-series equivalent) products. Nvidia is apparently convinced that it can get away with being almost as terrible at naming technology generations as the USB Implementers Forum.
Thanks for all the work Jeremy! There's a large community of folks that have been learning from you, the Fast.ai community and your work. Super excited for this grant!
> a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant.
... the contract was clear and totally fair (even explicitly saying that a16z wasn't going to receive any kind of rights to anything)...
Thanks for sharing, that is generous, and a pretty solid sign of being honest businessmen.
It makes me very happy to read about things like this -- support where it's needed and appreciated AND absolutely no strings attached. Wow. Wish there were more cases like this.
Thanks for providing context Jeremy. Fast AI has done an admirable job in communicating its commitment to an ethical pursuit of AI advancement. Do you feel such a commitment aligns with the bellicose rhetoric found in a16z's vision? Here is Marc Andreessen in a recent essay...
"There is one final, and real, AI risk that is probably the scariest at all:
AI isn’t just being developed in the relatively free societies of the West, it is also being developed by the Communist Party of the People’s Republic of China.
China has a vastly different vision for AI than we do – they view it as a mechanism for authoritarian population control, full stop. They are not even being secretive about this, they are very clear about it, and they are already pursuing their agenda. And they do not intend to limit their AI strategy to China – they intend to proliferate it all across the world, everywhere they are powering 5G networks, everywhere they are loaning Belt And Road money, everywhere they are providing friendly consumer apps like Tiktok that serve as front ends to their centralized command and control AI.
The single greatest risk of AI is that China wins global AI dominance and we – the United States and the West – do not.
I propose a simple strategy for what to do about this – in fact, the same strategy President Ronald Reagan used to win the first Cold War with the Soviet Union.
'We win, they lose.'
Rather than allowing ungrounded panics around killer AI, “harmful” AI, job-destroying AI, and inequality-generating AI to put us on our back feet, we in the United States and the West should lean into AI as hard as we possibly can.
We should seek to win the race to global AI technological superiority and ensure that China does not."
I don't think one needs to think too hard about the grave consequences a mission of "We win, they lose" may have for human survival.
Hi HN, we are super excited to announce this new initiative!!
Please note, this program was designed to support individuals, teams, and hackers who are not pursuing commercial companies. Nevertheless, these projects push the state of the art in open source AI and help provide us with a more robust and comprehensive understanding of the technology as it is developed.
We are really proud to be contributing in this small fashion and grateful to the first cohort and all others contributing in this space!
If an individual is looking to contribute to the field with different training data ideas - would they need to first establish themselves and get your attention, or would there be a way to submit a proposal?
For myself compute is rather expensive, I could likely afford a few test runs for proofs of concept but beyond that it would be difficult. I've got a single 4090 so I can't run llama70b faster then 1it/s.
Thank you so much for your work and enthusiasm here!! Unfortunately, that’s roughly the best way to approach it and how most of these have worked so far, demonstrations on smaller models (e.g. 7/13B), with slightly smaller datasets, catching the eye of the community, etc. in general it’s not a bad approach to research too, prove out concepts at smaller scale before scaling up!
Definitely. This was kicked off in the last month or two when there just seemed to be a bit more happening in OSS LLMs, but we are prioritizing diversifying this for the next cohort!
A lot of people are using RunPod for experimental/small-scale workloads. They have good network and disk speeds and you can generally find availability for a latest-gen GPU like an L40 or 4090 if your workload can fit on a single GPU. One GPU is plenty for fine-tuning Llama 2 7B or 13B with LoRA or QLoRA. They also sometimes have availability for multi-GPU servers like 8xA100s, but that's more hit-or-miss.
If you want to go even cheaper vast.ai is a popular option. It's a P2P marketplace for individuals to rent out their GPUs. You can generally get a ~20-30% discount vs RunPod prices by using Vast, but network speeds and perf are much more variable and there's always the possibility that the host will just shut you off without warning. I also wouldn't recommend using it if you're training with proprietary data since they can't guarantee the host isn't logging it, but most of the OSS fine-tuning community publishes their datasets anyway.
1. I've updated the section now: https://gpus.llm-utils.org/cloud-gpu-guide/#so-which-gpus-sh... - that should answer it. Basically 1x 3090 or 1x 4090 is an ideal set up for stable diffusion, 1x A100 80GB is an ideal setup for llama 2 70b GPTQ (and you can use much smaller GPUs or even CPUs if needed, for smaller llama 2 models)
2. No, they don't have access to the internet unless you build something that gives them access
Interesting, hadn’t thought of that, thank you! If you want to host end points Replicate is a great option, they also have a newer fine tuning api and solution! for raw VMs with GPUs right now it’s a bit situational and you have to try multiple different vendors tbh, also really depends on the capacity you need and which machines!!
Very happy to see oobabooga on the list. Everyone's contributing in different ways, and ooba serves as a fantastic portal to try out various models which I appreciate. Its readme even says it wants to be like the automatic 1111 of text generation. For beginners like me who are curious about this new world, ooba is a great entry point.
I want to congratulate you based on your current grant recipients. They are all very impactful individuals/teams. It shows that you clearly did your research and put the money in the right places.
Is there a way we can apply? I am building https://atomictessellator.com, and since I am 100% technical I am not sure about commercialising or open sourcing it, I definitly would like to open source some of the components (e.g. turnkey dockerized scientific computing environments, the molecular force viewer, and also open up a hub for the environments and models) - when I was in Stanford meeting with some comp chemists a few months ago they got very excited for this.
I negotiated with my day job to work 2AM to 10AM so that I can work on this passion project in the afternoons, some grant funding would really help me focus on this more. There's quite a bit of interesting stuff not on the website in catalyst discovery and molecular dynamics. Contact me if you'd like to chat / demo.
In my ideal world I would work on this full time and totally open source it.
feel free to DM on twitter or linkedin. In a nutshell, if you want to pursue a company now in or in the near future, we would likely need to consider you through our normal investing process for equity funding. If you are considering just working on it OSS, we would encourage taking first steps in that direct,
and testing community-project fit validation before quite going full or near-full time!
25K will either get you money for renting servers, or allow you to jerry rig yourself something like a 8x4090 machine.
The servers that most people would like to run (8xA100, 8xH100) are over 200k, and even if you had the money they're probably very hard to get right now.
8x4090 machines should be a lot cheaper than that. 1599*8=12,792. I.e. closer to 16x4090 machines. Sure labor is worth something but he has it in cash, and also there's barely a lot of people (yet) paying high prices for research (non ML ops) labor hours for this yet so the salary wouldn't be that high anyway.
Just a pet peeve when ppl under exaggerate how many gpus/flops a dollar can buy (if you're slighly smart with your money).
What about PSU to power all of this, GPU motherboards, SSDs, CPU to not bottleneck this and sufficient RAM? You could argue it's more than 8, but definitely not 16. To utilize those GPUs to the max you need memory bandwidth to even feed it enough data.
Yeh I built a dual 3090 machine and just to have those 2 cards run well cost a lot for all the other stuff. Workstation spec mobo, top tier PSU, CPU and memory with headroom for the system, not to mention storage and case and cooling. And this is just a hobbyist build. Cost would have easily doubled if I actually got server grade CPU and memory etc.
Meh, I was just ballparking. But add a threadripper, compatible motherboard, lots of RAM, fast storage, a few beast PSUs, and tax, and you're probably around 18k. So you ain't getting 2 of these machines for 25k.
Plus you'll need to set aside some money for the power bill lol
You won't be able to limit it to a particular set of clients who want to use it, but you're basically supporting the "GPU Poor" by doing this, as I think it's mostly smaller companies, individuals and researchers using vast.ai, rather than huge companies.
honestly would recommend using spot/rented instances on that budget vs. buying! A100s run around ~$2/hour, the hard part is getting a large allocation... training workloads are quite spikey anyways, usually running lots of smaller experiments before actually doing a larger single run...
No intention to attribute malice, but wouldn't this be more like buying a condo and renting it out to pay your mortgage rather than donating resources?
Whoa this is honestly awesome, a lot of tools I see on the list are ones I’ve been using so it’s great to see they’re getting financial support. Really hoping Ollama joins the list soon. I have some fun ideas I’ve been wanting to test out but probably requires a couple hours on 8X A100s + EvolveInstruct style data that cost upward of $1K+. Gonna start preparing some code to see if I can join grant #2!
Even knowing this is partly motivated by branding/marketing, it's great to see a16z getting more aligned with solving real pain points (vs crypto and churning out shallow media in recent years). Hope they can keep it up and hopefully more "thought leaders"/VCs follow suit. Best of luck.
Of course - there will always be investors/influencers pushing narratives to hype their bets. IMO everyone is better off when the hype is grounded in solving real pain points, which the AI projects I've seen seem to be closer to than most of the crypto projects I've seen.
Certainly there are plenty of grifters in AI too (as with any gold rush) and many AI efforts will fizzle out. But it seems there is more real value being created here than in crypto, which is the main thing I'm excited about and hope to see more of
Agreed, but IMO it's their job to do stuff like this (and prob unrealistic to expect a world where it doesn't exist). I'd rather see a higher percentage of marketing for things that have more real value vs. less
I agree this initiative helps solve real pain points, and I'm not trying to defend "Crypto" with my question here but isn't solving a "real pain" kinda subjective? For example, many people/companies have the real pain/need to improve ad click-through-rates and reduce crypto/gas fees but maybe these aren't problems you care about.
Agreed, "real pain" is a subjective/vague term and even crypto is a broad term. My intended use here is to describe solutions that are more grounded/integrated into reality vs. more grounded in a fantasy of how the world works. It is hard to describe well though and if you keep unrolling most perceptions of reality are subjective
I guess sometimes a pain can exist without a profit, there's only a small number of people willing to solve those pains without a profit incentive though (pro bono lawyers, doctor's without borders, etc).
Makes me very happy that this is being undertaken.
LLMs will be a very valuable tool. Having censorship built into these tools makes me very nervous. Imagine if Word refused to allow you to type any “bad word” in the name of safety and ethics.
No deeper reason. I think there just a lot in LLMs happening right now which skewed it towards them. We would love to do something in the SD ecosystem.
It's interesting to note that there's particular emphasis on "removing censorship from LLM outputs" given a16z's investments outside the AI space in some ideological social media companies.
Hi Robin Hood. I wonder about the "continue their work without the pressure to generate financial returns" part. I don't see an expectation to work on non-commercial, or socially conscious, applications, just to commit to open source (what level of commitment also being unclear). It would actually be great if support programs along these lines were being offered by governments and public service institutions, with some serious governance and transparency around the process.
Surely there are also some interesting econometrics on the "GPU poverty line" - someone has literature to share?
This reminds when a random chinese user finetuned the leaked NovelAI Stable Diffusion model and released it online, the Anything V3 model, tech company like Tencent started using it in their products.
"We believe artificial intelligence has the power to save the world"
I don't agree with the side that says AI is about to destroy everything, it seems very hyperbolic... but neither do I agree with this sentiment that it's going to save the world.
Leftfield thought, imagining a tax (vaguely like carbon credits) on energy/cpu cycles/modelling (just my vague idea). With the proceeds going to other projects to diversify the outcomes.
Always seems to me our economy is being less based on money and more on energy and productivity, basically being rounded down to physics as a healthy market economy should.
if the work is OSS/public and you aren't trying to start a VC-backed company, it could be! We usually want to see some early work already published, community reception and validation, but feel free to DM Matt or myself and we can take a look!
> if the work is OSS/public and you aren't trying to start a VC-backed company, it could be!
Yes, it is a company that often applies and receives grants. For example, we are working on adding AI to security tools such as https://github.com/CoinFabrik/scout
I'm still not sure what I'm supposed to be using AI for. I already know how to code, tried for a decade to make ACR (auto content recognition) work with CNNs, and as best I can tell, chatGPT can write a news article for you if you write it first and then hire someone else to edit it.
This is great, but I'm not seeing anything about how to apply, or really anything about the application and selection process at all. Is this by design?
My guess is that they'll prefer to get nominations.
Having once done a grant application process - I think their approach is smart.
If they opened grant applications, they'd get thousands of (sadly mostly very low quality) applications. It's extremely hard to vet.
It's a much safer bet to wait until projects are already popular in the community, already being relied on by people, and where those devs would love to spend more time on their open source projects but they're limited by money.
Yes, the downside of this is that some people could make things happen if they were given money first. But, if I was them, I'd still prefer to do what they're doing and pick projects that are already successful and give grants to those.
(This isn't a comment about your project specifically, for all I know your project might be a popular open source LLM project that lots of people already rely on, in which case they probably already know about your project and will probably reach out to you for the second round, is my guess)
Well, at least this time a16z is giving money to actually useful technologies instead of bloviating about the world changing potential of cryptocurrencies. I imagine they've written down most of their cryptocurrency investments at this point.
That being said, these are some great grants for local LLM builders, local computation for such powerful tools is needed, and will continue to be needed, as industry incumbents like OpenAI continue to not release their models.
Don't VCs basically chase everything because they only rely on 10% or less to succeed but, that 10% needs to have at least 10x or more profit and then they do another round?
I had become honestly cynical of their actions in the past, as they felt hype-driven and externalities-blind. Yet I don’t see how this grant could become rapacious. Hopefully it is an attempt at cleaning the brand which can work as making amends.
“We are disguising this round of seed funding as a grant, accomplishing all of the below:
1) brand-washing for people who don’t do the math
2) avoiding angering our LPs and/or violating our fund theses and
3) getting in on the ground floor of things that might follow OpenAI through the non-profit-to-VC-darling door
Thanks for reading our press release, feel free to try to apply using your personal network (somehow) because we don’t care as we’ve already landed our positions.”
This is insurance for their already huge investments (and future rounds) in AI businesses. It's a pittance relative to everything else they have at risk. It's prudent since failure of small projects like these can delay work on profitable corporate efforts.
They state the nature of their support in very clear terms:
"We’ll support a small group of open source developers through grant funding (not an investment or SAFE note), giving them the opportunity to continue their work without the pressure to generate financial returns."
Some of the recipients of financial support have described the nature of the agreement as well.
> a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant.
You could choose to interpret it as marketing spend or "buying relationships", sure. For me, there's enough negativity in the world - I don't want to spend time imagining more.
And why would I? A16Z have already explicitly stated their intent: "a thriving open source ecosystem is essential to building [ai's] future" and "We’ll support a small group of open source developers through grant funding (not an investment or SAFE note), giving them the opportunity to continue their work without the pressure to generate financial returns."
There's not a lot of spin on this. They want to be a leader in AI so they're investing in the people who build the tools that they use in pursuit of their commercial goals.
> We believe artificial intelligence has the power to save the world—and that a thriving open source ecosystem is essential to building this future.
Thank you for understanding the fundamental flaws of the World and human consciousness, and thank you for spending multiple lifetimes as Buddhist monks (not to mention lamas) to understand the intricacies of the human mind from a scientific perspective. This is exactly what we need right now and this initiative looks to be the answer to all of our problems.
A16z is getting 'mad respect' from me for taking this line of the 'ai wars' and generally their attitude on all this. Really cool that they're going straight into 2010 era startup energy on this new wave. Idk I just find them really cool and inspirational.
It's all relative. Compared to the huge companies spending thousands per month/week/day on GPU usage, having two 4090s (which you could acquire for like ~$3000) could be considered on the lower end for sure.
I interpret "GPU poor" the same way as "house poor", i.e. you're poor (lack cash) because you've use it to purchase GPUs, not that you don't have enough GPUs.
My grant covered the purchase of a Scalar server from Lambda Labs, which allowed me to configure a system with 8 previous-gen A6000 GPUs, partly also thanks to NVIDIA who has recently given me 3 of those cards, and Lambda Labs who offered to provide everything at cost.
a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant. They couldn't have been more helpful through the process - I didn't have to fill out any forms (other than sign a single one page agreement), the contract was clear and totally fair (even explicitly saying that a16z wasn't going to receive any kind of rights to anything), and they wired me the money for buying the server promptly.