This take really misses a key part of implementation of these LLMs and I’ve been struggling to put my finger on it.
In every LLM thread someone chimes in with “it’s just a statistical token predictor”.
I feel this misses the point and I think it dismisses attention heads and transformers, and that’s what sits weird with me every time I see this kind of take.
There _is_ an assumption being made within the model at runtime. Assumption, confusion, uncertainty - one camp might argue that none of these exist in the LLM.
But doesn’t the implementation constantly make assumptions? And what even IS your definition of “assumption” that’s not being met here?
Edit: I guess my point, overall, is: what’s even the purpose of making this distinction anymore? It derails the discussion in a way that’s not insightful or productive.
> I feel this misses the point and I think it dismisses attention heads and transformers
Those just makes it better at completing the text, but for very common riddles those tools still gets easily overruled by pretty simple text completion logic since the weights for those will be so extremely strong.
The point is that if you understand its a text completer then its easy to understand why it fails at these. To fix these properly you need to make it no longer try to complete text, and that is hard to do without breaking it.
This is part of why I don’t understand the argument for vim/emacs.
If you _like_ those then keep using them by all means.
But when people tell me “it’s so much more efficient if you never have to take your hands off the keyboard”, I don’t buy that.
Sure, it is more efficient to know shortcuts for things you do commonly in your IDE. And yeah, editing text is a common task in the IDE.
But (for me) 90% of an engineering task is gathering requirements, testing, waiting for Jenkins or something, chatting with a colleague about the context of the task. Fumbling around.
Writing new code or editing existing code is really a small part of completing the task, and being able to do it as quickly as possible isn’t that big of a benefit (again, for me specifically).
So, again I’ll reiterate, these editors work great for some (I know enough in vi/vim to get stuff done), but don’t try to sell me on the efficiency argument. Just use it because you like it!
The efficiency argument isn't really about macro-efficiency, it's about micro-efficiency. It's not, "with my lightning fast editing speed, I will drop 10k lines today", it's about writing this method faster so I can run tests and see if anything changes, getting me to the next thinking step that much quicker while the problem is still fully loaded into my head. It's about keeping flow state flowing. It's about reducing the iteration time between working code states, so your cycle time gets tightened.
IME, without good muscle memory (touch or hybrid typing) of keys, consciously searching for each key is going to seriously disrupt the flow and thought process behind the code being written.
It may not make you a rockstar programmer, but it will definitely make coding more enjoyable.
I think there was a recent HN discussion on this topic.
10% of my time is 4-8 hours a week. And I'm writing up those requirements, tests, etc. someplace, so I've got hands on keyboard far more than 10% of my time (usually in a vi). Seems like a good opportunity to optimize. I made my choice...not because I 'like it', but because I've been doing it both ways since the 80s and this works.
In the end, tho, you don't understand the argument, don't know the tools (know enough to get stuff done doesn't count), haven't walked in the shoes, but you're absolutely certain all us vi/emacs users are wrong. I mean, I've heard this pissing contest since the Win3 days, so I don't expect much intellectual rigor, but most people who feel the need to "well ackshually" keyboard jockeys at lest take the time to google up something like AskTogs infamous "I spent $50m to prove mice are faster" (for Apple, trying to push the Macintosh, with none of the research ever published AFAIK) to 'prove' how deluded we are. There is actual published research out there covering not just speed but ergonomics and accuracy; it's pretty inconclusive all things considered, and very dependent on the use case the researchers decide to examine, so really easy to cherry pick so you get to win arguments on the internet.
I had a PSP and also loved it, but I don’t feel like it was that mind blowing. The tricks they pulled made it less impressive IMO. Most games felt like cheap pantomime versions of what you wanted them to be, you’d buy something and then go “oh, never mind”.
The homebrew scene was much more impressive than anything official, if you ask me.
God of war chains of olympus, metal gear solid peace walker, crisis core ff7, test drive unlimited, motor storm artic edge... I could go on.
I don't see a world where a handheld console released in 2005 being able to play these games isn't impressive. And I wonder what homebrew could offer to match these games too.
Homebrew made the PSP a killer emulation machine way ahead of its time. You could emulate the original PlayStation, which was how I got around to playing the original Final Fantasy 7 (although I think this also had an official release, my memory is fuzzy). And it could emulate the SNES, which was how I got around to playing Chrono Trigger.
Funnily enough, I remember some rumors going around about some people trying to develop an N64 emulator which never got released or went anywhere, AFAIK. I have a vague memory of a guy going by the ID PSDonkey or something like that.
PS1 emulation on the PSP is actually mostly a native Sony feature! Sony has always been very interested in backwards compatibility; initially to solve the chicken-and-egg problem of a new console not having many titles available at launch, but later largely focusing on additional revenue by selling people digital downloads for games they might well already own (because they removed native media compatibility...)
For the PSP, media compatibility was obviously never a possibility, but as I understand it, the native PS1 emulator on the PSP essentially works with completely unmodified PS1 ISOs in some sort of container file format, which homebrew developers have then quickly reverse engineered.
The emulator itself is technically quite impressive, and apparently largely runs unmodified PS1 MIPS R3000 code on the PSP's MIPS R4000, although the GPU is emulated [1].
FF7 did get an official PSN release playable on PSP. You could play FF1-9 on PSP by the end of it's life once the port of the DS version of FFIII arrived. There was also a PSN release of the PS1 version of Chrono Trigger, which is actually how I played through it (although perhaps on Vita).
I could see this as an adult, but as a kid, this thing was killer.
I remember thinking: "Whoa, burnout 3 is something I play on the big screen in the living room, and now it looks the same and I can play it on a handheld device?" (it definitely didn't look the same, but I was about 5 when it came out, so I never paid much attention to the differences. Also sorry if I'm making you feel old).
I also remember it being the first to do a joystick on a handheld. That was pretty cool.
I remember being so excited for Bloodlines because I had no way to play AC2. It was really impressive what you and your team managed to accomplish on a handheld. Now knowing that you built it from the ground up makes it even more amazing.
Can you share more about the experience, the development process, and any other PSP titles you worked on?
I loved my PSP and the homebrew scene during that era. The biggest area where the PSP felt like it was lacking was with multiplayer games. I remember playing some fighting and puzzle games with friends on the PSP, but the most fun wireless multiplayer experiences were on the DS; and primarily it was Mario Kart DS. The real killer feature was being able to play with friends without requiring them to own the game; I don't think PSP games were capable of this.
They were. You could play Tekken against a friend without him having to own the game. It was great.
But in general not many games did this. Since even Tekken had pretty high load times, I suspect that the much higher graphical fidelity of the PSP made most games just too big for this to work well.
There were some games that had a feature called game sharing. This let you send a smaller version of the game, essentially a demo, to another PSP over wifi, and this let you at least try the experience without owning the game. Burnout Legends definitely did this, but I also remember it on some NFS and sports titles. I just found a list of games that did this.[0]
Also, there were some good shooters as well, like Medal of Honor or Battlefront, with pretty nice maps, especially for a handheld. And for racing games, Gran Turismo was incredible with almost all GT5 tracks and 900+ cars present.
For some reason the DS didn't take off at all in the area I grew up (neither did the original Pokémon games despite seemingly being massive everywhere and the cartoon being a big hit in my school).
it was only like ten years later when I got to realise how great those download play titles must have been for some kids. PSP games would have been way too large to pull off similar stuff, DS games were often only a few megabytes
Maybe it's just because it was high school and not elementary school, but I remember basically everyone who was into games even a bit having a DS, but nobody used them as a daily carry portable. It was something you'd just notice in people's rooms when you went over.
I recall Bloodlines as not being a bad game, but not particularly good either. I think the biggest problem with PSP is that lots of games were just diet ports of PS2 games. Why play them if you can play better versions of same games on PS2? Having said that, there were absolute gems in the library, and it breaks my heart that in some alternative universe I haven't played Patapon or LocoRoco.
Bloodlines was not a diet port. It was an original. Given what we had to do and the timeframe we had (9 months) we had to go from zero to finished game and it was a terrific challenge. We had no original source code so everything you see, including the mobile version of parkour, all the game systems, complete pipeline from Maya had to be designed and built as well as getting a game finished. I truly believe the project was a AAA endeavour but with a AA budget and timeframe. Assassin's Creed, the original game has a much larger scope but also a much larger team and a four year development time. Bloodlines if only given an additional months to a year could have been outstanding. I'm still very proud of the accomplishment of our team given our constraints.
As someone involved in the game production, you saw all the hard work and miracles that happened. It's your child and you love it even if it's imperfect. As audience, I compare this game to a different one, without any regard to the constrains you had. Heck, I'll give your game a bad score simply because I don't like the genre, in which case there's absolutely nothing you could've done to improve my experience.
I just grabbed my PSP and took a look at the save file of Bloodlines. It says "Percentage completed: 0", which is super strange, because I distinctly remember playing the game as a kid when visiting my aunt. Maybe I actually enjoyed the game, but the savefile got fucked, I got angry, and never played it again? My memories are hazy, but now that I focus, I do remember enjoying it, but not much beyond that.
BTW when looking at the save files, what strikes me is how few games I actually played. To me it seems like there were lots of them, but actually no, I just spent lots of time playing same games.
I'm impressed! Bloodlines was the only AC I've ever finished completely, and even though the story was odd at some points, I found the gameplay and the graphics to be spot on for the platform. Very enjoyable for teenager me back then, and still holds up quite well. Doing this in 9 months... Just wow.
been long time but I recall playing it on PSP, ( before playing the AC game before though ) I have pretty good memories, and it was one of the best games I played on PSP for sure.
I think maybe Sony created too high expectations with PSP.
I absolutely loved that game. I remember playing it on my white Star Wars PSP with a decal of Darth Vader on the back. I spent so many hours sitting on my bean bag chair playing it.
Tell us more!
I still beleieve that the PSP is a killer platform, those games are beautiful with 60 fps mods at high resolutions. I never really stopped playing them, just moved them to my phone.
I can tell you how surprised I was to find out that index triangles were slower than triangle lists on the PSP due to (I conjecture) a broken vertex cache in the PSP graphics hardware (guessing it was rushed out with this bug). The huge challenge of clipping triangles on the PSP - they had to be clipped on the CPU. I loved working on the PSP and wished we could have done more PSP projects.
Despite the limitations, many games pulled off amazing graphics. Many in the thread compare it to the PS2, but that's unfair. Doing better than PS1 graphics in a handheld was quite an accomplishment back then.
Interestingly, the quality of the assets in these games didn't have a chance to show on the 480x272 screen of the PSP. However, many games, including Bloodlines, Gran Turismo, both GTA's, just to name a few, look very close to PS2 when upscaled. So the work was very much put into these titles, and it shows, even after two decades.
Personally back in the days I had Nintendo DSi, but just recently I bought gamepad for phone just to check what I have missed by not owning PSP at that time :)
The PSP was probably the most hyped up gadget in my teenage years. It was launched only after a long delay in Europe, but even after all that time of waiting and the highest of expectations, it delivered. So thank you for being a part of that :)
I had a PSP and also loved it, but I don’t feel like it was that mind blowing.
Perhaps you came to the PSP late in its cycle.
I happened to be in Tokyo when the PSP was released. I was on a subway and saw a girl playing Lumines and it completely blew my mind.
As soon as I was done with my daytime obligations, I went straight to Yodobashi Camera and bought one. When I got back to the U.S., showing it to people always blew their minds, also.
Until then, the notion of "handheld gaming" was mostly the early GameBoy series, and maybe Lynx. The PSP was a whole different category.
I got a PSP-3000 during high school, and loved it. In addition to playing native and PS1 games, it was a very powerful media player as well. Much more versatile than most phones at the time, especially with its nice, though not very hi-res, screen.
The PSP wasn't very common in our area, but I managed to get some friends to get one, and we had a lot of fun with multiplayer too.
It was niche outside of Japan but Monster Hunter Freedom/Portable was incredible and wildly more successful than the console edition - the PSP and social monhun parties in Japan validated a brand new IP for Capcom
Admittedly with homebrew it was even more fun when the community patched MHP3 translations and we would have AdHoc wifi parties proxied over the net using a dongle. Good times
Some of the games were awesome. I spent a lot of time playing the star wars battlefront game on there. And armored core formula front, wipeout, a few others.
You’re not wrong that a lot of them felt hollow. Like that assassin’s creed game that had virtually nobody walking around on the street and felt like a dead world as a result.
I love the AC franchise and have been playing it* since the beginning. I know it was because of the limited power of the platform that you couldn't have the usual crowds of bystanders etc. I'm sure it was a great game in general, it just felt like a bit of a ghost town :/
(* on and off, I took a break around 3/black flag after a bit of burnout)
As I mentioned in other comments we only had 9 months from zero to finished original game with no source code or help from a very busy Ubisoft other than some reference assets (which were completely redone for the PSP). If we had more time, another year, bigger crowds would have been a high priority!
It wasn't until 2009 - which is also when my family got WiFi (and switched from dial-up) - that I got mine: a Nintendo DSi.
Objectively, that browser was terrible, even at the time. But the ability to just read text on my own (and not have to ask to use the family computer - and to stay up all night reading under the covers, which was much harder with books) was amazing then! Of course, it would seem so quaint to kids today.
That 2.0 update was mind-blowing on its own. Smallest device you could browse the web with, I think? Pretty sure it was still the time of PDAs, before smartphones.
EDIT: Yeah, July 2005, 2 years before the iPhone came out.
> Smallest device you could browse the web with, I think?
I'd contest that the Blackberry devices were smaller and earlier, e.g the 7230 from 2003[0]. Pretty sure some of the iPAQs (and other CE devices) arrived with wifi and a browser before 2005 as well since Pocket PC 2000 shipped with a version of IE 3.1.
Yeah. As a high schooler I just wanted to use AIM on it, but all the web-based AIM clients required Flash at the time, which wasn't supported in the PSP browser.
So I ended up hacking together a site called AIMonPSP with PHP and MySQL that did all the AIM communication on the server instead of client-side. Made a decent amount of money (for a high schooler) on AdSense banners, and learned a lot about software design the hard way.
Started on a VPS and then scaled to a single larger dedicated server through a hosting company. It was never doing crazy amounts of traffic, maybe 1-2k concurrent users at peak, and the frontend was just doing slow polling every few seconds.
It was mind blowing at that time because it felt solid and you can even watch movies on it. Too bad Sony abandoned it. The form factor was amazing even Nintendo and Steam copied it.
To be fair, Sony released an UMD-less model themselves! Many titles, especially later in the system's life, got a digital/PSN release in addition to the UMD one, and some were even digital exclusive.
I just never felt like there were games worth buying for the PSP. I only ever got three - Crisis Core, FFT, and God of War Chains of Olympus. The system collected dust mostly. My DS got a lot more use, because Nintendo is just better at making games imo.
Personally I don't trust a NAS as far as I can throw it, or rather I'd use one the way I use Amazon S3.
My home server has a ZFS array, the media server and some other programs access it directly. If I want to move files to or from it I use SFTP. If I want to back files up to it I use rsync. I have Lightroom running pretty good on an external HDD, no way I'd take my chances running it on a NAS.
An appliance with the primary role of storage, and the ability to share files over the network are the distinguishing features for a NAS. Network support for iSCSI, SMB, NFS makes a NAS; sharing data exclusively over the media protocols (http, rtsp, etc) makes it a media server
By "NAS" most people mean "box they bought from someone to share files on a network" - think Qnap or Synology. They'll call a "NAS" that is home-built a server, even if they do basically the same thing.
To make it more fun, you'll have people refer to "I don't have a NAS, I run FreeNAS on my server."
This is gaslighting. Even if it’s a coincidence, people have numerous examples of situations where they or anyone with them did not search for a thing. I’m not saying the only possible explanation is that the phone is spying on you - but I’m saying, don’t gaslight people and say “you probably searched about it and forgot”.
I think it’s wild that your counter argument is basically just gaslighting everyone. There are tons of (possibly coincidental) examples of this type of thing happening. Even if it’s a coincidence, it’s not explained away by “maybe everyone just forgot or is not paying attention”.
well you'll need integrations to have the WASMised doom tell you when a monster gets hit, or when you died. you need to worry about which keyboard events to pass in, etc. i know it's not super hard but it's definitely the more exciting part of this, vs all the "make the title text bigger" stuff that's in the chat
Sure but if you think about making this project I think a lot is already know how to layout a box on a webpage and probably don't know how to hook WASM-ism Doom into the page logic.
Wait you wanna live monkey patch the WASM bitcode to hack that hook in? There's gotta be easier ways. This is the kind of stuff I'd hope an AI tool like Vercel's v0 could help with but instead this demo just shows it as a layouter, and an annoyingly bad one at that.
I think it's mainly a millennial and gen z thing-- older generations still answer all calls, at least those that aren't into tech. I think it's just easier to realize that anyone not in your contacts will either leave a voicemail or text you if it's that important.
I'm mid Gen X, and I can't imagine wasting my time answering all calls. I have my phone set to silence any unknown numbers. I'm not going to answer any call that isn't in my contact list. Voicemail a coherent message and I'll call you back and add you to my contacts.
We ignore it too. But I can tell the ones who do answer. They get extremely irate if you do not pick up when they call. As if it is their personal line to you and you should drop everything for them. I dump them into voicemail too.
In every LLM thread someone chimes in with “it’s just a statistical token predictor”.
I feel this misses the point and I think it dismisses attention heads and transformers, and that’s what sits weird with me every time I see this kind of take.
There _is_ an assumption being made within the model at runtime. Assumption, confusion, uncertainty - one camp might argue that none of these exist in the LLM.
But doesn’t the implementation constantly make assumptions? And what even IS your definition of “assumption” that’s not being met here?
Edit: I guess my point, overall, is: what’s even the purpose of making this distinction anymore? It derails the discussion in a way that’s not insightful or productive.
reply