This is really fascinating to me, as this same problem drove an down a collection of side projects, ranging from a scraping pipeline[0] to a WIP full-stack application[1][2] for quick-swiping through results (also to better enable other use cases like scaling out to different data sources, but less about the auto apply). Development pace isn't at all breakneck, since I keep getting rabbit-holed into other ancillary stuff -- periodic bot detection issues, upstream pull requests (like user agent switching in Astral, for the bot detection issues), so on -- and have more broadly prioritized using the thing I've been tinkering on over throwing all of my energy into building it... but it's incidentally been a lot of different stuff to tinker on. I've been more intentional about DIYing over LLMs -- for the experiment of it, and to have stuff I can show off -- and it's gotten me into TypeScript for the first time, Deno KV, browser automation, and as an added bonus my understanding of selector logic has made enough of a jump that so far I've been building this entirely in vanilla CSS.
My other consideration is that I'm running on CPU and don't love depending on cloud services, so I've also been mostly getting stuff out of a DOM where I can... but it's occurred to me that there are scoped reasons to consider this, like deeper parsing from job descriptions, and I occasionally toy with it on stuff like HN posts. The prompt format here is a lot more thorough than what I've tried, and I might have to go back and experiment with this some more. I haven't gotten great consistency with this yet myself, and I'm not sure how much that's my prompting and how much it's that I'm using smaller (mostly ~7B) models. Which LLMs are you using for this -- ChatGPT throughout, or are there others?
I've been trying to avoid too much prior art while working on mine, but I'm definitely interested in hearing more about what you've been building around this.
[0] bhmt.dev/blog/scraping (warning: this is 10000 words + code samples, as it goes from browser console to browser automation and covers a few different side projects)
My other consideration is that I'm running on CPU and don't love depending on cloud services, so I've also been mostly getting stuff out of a DOM where I can... but it's occurred to me that there are scoped reasons to consider this, like deeper parsing from job descriptions, and I occasionally toy with it on stuff like HN posts. The prompt format here is a lot more thorough than what I've tried, and I might have to go back and experiment with this some more. I haven't gotten great consistency with this yet myself, and I'm not sure how much that's my prompting and how much it's that I'm using smaller (mostly ~7B) models. Which LLMs are you using for this -- ChatGPT throughout, or are there others?
I've been trying to avoid too much prior art while working on mine, but I'm definitely interested in hearing more about what you've been building around this.
[0] bhmt.dev/blog/scraping (warning: this is 10000 words + code samples, as it goes from browser console to browser automation and covers a few different side projects)
[1] github.com/chaosharmonic/escape-rope
[2] github.com/chaosharmonic/escape-rope-ui