> And yes, I also hate when I misplace my billions. Especially since I have yet to relocate them...
I think you're joking, but I'm not sure. I work in trading, and I've been on the receiving end of that phone call. As I recall, it was around 9PM in the US, my work phone rang and could see from the caller ID that it was from our London office. There were no greetings, first words I hear were "We're missing over a billion dollars. You need to find it...NOW."
When I received that call, it was in the middle of the 2008 financial crisis. The daily PnL swings were wild, and it wasn't always clear on the cause. FX volatility was insane. We did all of our PNL reporting in USD, but held a lot of foreign assets.
There was no malfeasance; I'd just taken ownership of the system a week or two before, and a nightly job had silently failed. Perl job on Windows, extracting data from a 3rd party trading system that wasn't built w/ integrations in mind, feeding it into in-house systems. It was a very flimsy house of cards. A gentle breeze in the night would knock it over. Rewrote the integration in Python, hooked everything up into our monitored job scheduler. Had to do some janky UI automations in Python until we got the vendor to add a proper CLI-based reporting mechanisms. It was a "fun" ride, but I eventually got my evenings back. Did cause the end of a relationship, though, so there's that.
A lot less money involved, but I remember my boss at the time (we were a small mortgage broker in Eastern Europe) asking me to write a quick Python script that would automatically get the daily Libor number and save it into our DB.
Seeing as I was hearing about Libor all day, every day (almost all of our clients had their mortgages computed on that piece of info), I had expected it to be something “automatic” (like at least an XML thingie) and well documented. Instead I had to parse some html on a page somewhere (I remember some yellow background) and hope that the HTML structure around that Libor figure would remain unchanged.
This was all happening around 2007 - early 2008, suffice is to say that when all the Libor scandal happened a little later on I was not at all surprised.
I remember some internet arguments on Libor when I researched how it was determined and everyone disagreed with me because of how much was based on it.
It's just survey data, and no verification of whether a loan can happen at that rate? And you throw away the lowest rates, which should be the market-clearing rate all-the-things-being-equal if lenders are fungible? And if lenders aren't fungible, then isn't it all apples and oranges?
High finance even fairly recently (80s) was based on handshakes and trust. The value of contracts tied to LIBOR grew by an order of magnitude or two while the definition of LIBOR wasn’t adjusted.
Why wasn’t it fixed? Because replacing it would require an enormous amount of coordination and there was no clear evidence that it was broken. When that changed it finally got replaced by SOFR.
LIBOR was developed in '86. The LIBOR scandal broke in 2012, with evidence of bad conduct going back to '08 or so. The LIBOR to SOFR transition was officially kicked off around 2016. Due to the complexity, the transition took 5 years.
The LIBOR rate-setting mechanism was a reasonable design for the world of 1986 [1], when finance was a smaller club and trust among bankers was higher. Also, it wasn't clear back then that the volume of contracts referencing LIBOR would grow to become many trillions of dollars just a short couple decades later.
When problems finally became very clear, we replaced it.
[1] One criticism often levied at LIBOR is why it sufficed to take a windsorized bank poll instead of looking at something more reliable, such as market transactions. But that's ahistoric: in 1986, banks didn't always do unsecured 3m lending on a daily basis. And later on, banks mostly stopped doing unsecured lending to one another entirely. So critics who suggest that LIBOR should've looked at market transactions entirely miss that finance, actually practiced, is a constantly moving target and it's hard to predict the future when designing benchmarks. SOFR, LIBOR's replacement, looks at secured interbank lending — a practice that was rare when LIBOR was created.
I was joking, the max I ever "lost" was a couple 100k of inventory, the majority of it was recovered and the reminder, as far as I know, covered by insurance. Also a, surprisingly similar, fun story involving just slightly different interpetation and handling of messages between our and the service providers WMS, which screwed up things in ways I never thought possible. And almost went unnoticed, after all even with top notch metrics and my borderline paranoia the issue went on for almost three weeks before we caught it.
I can only imagine so the slight shock you had after that phone call so! I love those stories from the trenches so you hear on HN, thanks for sharing!
I think you're joking, but I'm not sure. I work in trading, and I've been on the receiving end of that phone call. As I recall, it was around 9PM in the US, my work phone rang and could see from the caller ID that it was from our London office. There were no greetings, first words I hear were "We're missing over a billion dollars. You need to find it...NOW."
When I received that call, it was in the middle of the 2008 financial crisis. The daily PnL swings were wild, and it wasn't always clear on the cause. FX volatility was insane. We did all of our PNL reporting in USD, but held a lot of foreign assets.
There was no malfeasance; I'd just taken ownership of the system a week or two before, and a nightly job had silently failed. Perl job on Windows, extracting data from a 3rd party trading system that wasn't built w/ integrations in mind, feeding it into in-house systems. It was a very flimsy house of cards. A gentle breeze in the night would knock it over. Rewrote the integration in Python, hooked everything up into our monitored job scheduler. Had to do some janky UI automations in Python until we got the vendor to add a proper CLI-based reporting mechanisms. It was a "fun" ride, but I eventually got my evenings back. Did cause the end of a relationship, though, so there's that.