When a material shrinks, it can create immense internal stresses equivalent to what it would experience at extreme pressures. Compare to say a Prince Rupert's drop.
Is it known to anyone how OpenAI (and others) are extending the context windows of things like ChatGPT so far? E.g. if you exceed 2048/8192 (subword) tokens, does the model just chunk the inputs and evaluate separately on the chunks? Is context/state maintained across chunks? I've never seen anyone actually explain this.
> While ChatGPT is able to remember what the user has said earlier in the conversation, there is a limit to how much information it can retain. The model is able to reference up to approximately 3000 words (or 4000 tokens) from the current conversation - any information beyond that is not stored.
This implies ChatGPT has a 4000 token maximum prompt and prior prompts in a given web session are inserted into the current prompt, most recent to oldest (probably with some sort of time context like "previously, user asked:"), up to 4000 tokens.
I've had longer discussions but I'm realising that I often ask for a summary, which would mean the model has a summary of the conversation so far in the window.
I have been playing with their completions API. I just keep track of previous conversational lines, and when I approach a configurable token threshold ( input+output must fit within the given number of tokens, and it returns it each time ), I just instruct chatgpt to summarize the conversation thus far with additional specific instructions to help it keep useful bits of context. I then make that summary part of the context I send in along with kept and future conversational lines.
Their API calls on the site have references to previous message ids, which makes me expect they're doing something similar.
I wonder if they tack on a neural network with a larger input set as preprocessing with convolutions or something beforehand, and pass those into the inputs of the large model. It’s something I’d try but I have no idea if that’s what they’re doing.
As a 2019 Insight alum, I can't say more good things about the program. As someone who has now been on the other side of hiring, I can say that the fellows from the program really differentiate it from every other 'bootcamp' (I wouldn't call Insight a bootcamp as there are no real tests/homework/etc, but of course lots of expectations set by the program directors).
Unfortunately, Insight kind of imploded like most other talent recruiting did during the pandemic, laying off basically the entire staff and not having a spring 2021 cohort (and I have no clue what ended up happening with the fall 2020 cohort). I believe they are trying to revive it, albeit at a smaller size and likely with the ISA. I truly hope they succeed as it was a fantastic program. I also hope they stay small.
I would say i) having a relatable project which I came up with/collected data for/built out/productionized, so I could talk about a real data science project during interviews that didn't require a ton of background ii) having a cohort of very motivated people to learning with/from iii) interview prep with experts, both technical and not.
