I've also not seen great results on three.js or A-Frame questions from any of these models. I'm guessing it's simply because there's a limited corpus of text from which to learn, however I wonder if LLM's lack of inherent spatial awareness contributes. Admittedly three.js and related concepts can be confusing for humans too
I suspect it could soon be feasible to fine-tune on code with limited open data through a bootstrapping approach.
Give it the source code, a test library and access to a dev environment and with some prompting it could start to experiment with increasingly complex use cases, learning from successful attempts. This would depend on the model’s ability to understand what a successful outcome is so it can define test cases, which might be harder if the output isn’t text but not impossible.
Being able to give a model expert knowledge on an undocumented library or language seems like it could help accelerate adoption of new technologies that might otherwise suffer from network effects as users get used to AI-assisted development. Not to mention automated testing, finding edge cases and making pull requests to fix them, security, etc.
A human taking time to experiment with their assumptions and gain experience with an unfamiliar subject is in a sense creating their own training data as well.
