Years ago in a competitive programming contest, there was an interesting problem to calculate the number of days a bus runs in a given year and a given bus schedule. The bus schedule could contain things like, the bus only runs on weekends, or the bus only runs on weekdays that aren't holidays, or the bus only runs on holidays but including the observed holiday which is the weekday after the holiday if the original holiday falls on a weekend. You know things are intense when you are programming a computer to perform the Computus (calculation of the date of Easter).
Technologists thought, how easy it is. A year has only a few hundred days, and there are only a few dozen possible rules for buses. Brute force is enough.
Nobody got the problem right. It gave me a great appreciation for how complex when technology interacts with culture and customs, long before my professional career.
How we solve that problem is a Calendar table in the DB, and some human somewhere is required every year to update the calendar with the various holidays and what not. Then we just look up if it's in the list or not. It's a human problem anyway, and then when someone decides, wouldn't it be spiffy if we did a day of staff training on X day and did a reduced service? Sure, whatever, just put it in the calendar and blamo, accounted for properly.
I literally saw this at my first work 20 years ago. You generate most by script and just add those few odd public holidays manually, much easier to have a person spend 5 mins once every year rather than some smart script with 20 not-so-well-handled corner cases and 7 bugs.
Oh goodness no, we don't generate ANY of it, we require the HR dept to own the calendar data. They can copy from the previous years calendar if they want as a button in the UI, but I honestly have no idea if they use that button.
Like I mentioned in a diff. comment, we generate all the official calendars of the org from this data. We print them, we generate .ics, etc.
I've seen this done multiple times too. It works until the human forgets about it, or knowledge is lost through collective organisational forgetfulness or churn. And the outages still happen because the lookup fails at some point.
What we do to counteract that problem, is we generate organization wide calendars from this list. We provide .ics feeds for our organization, etc. ALL of it comes from this calendar, so if you want to know when a payday happens, you are just looking at a view of our calendar. If you want to know what the official org holidays are, you are looking at a view of our calendar. I mean it might display in Google calendar or Apple Calendar or whatever, but we are the data source.
So when a mistake happens, every employee that cares about that date will find someone to complain about it to, and ultimately someone somewhere goes, oh bother, we forgot to update the Calendar, and they promptly fix it.
Plus we avoid all blame, as it's very clear that we don't invent the calendar data. For us the HR dept is in charge of maintaining the calendar.
Does it end up better or worse in practice than the pure computational approach? A system with the same faults as the system it replaces could still be an improvement if they occur less often.
This is also how we do it, but it’s the “obvious” solution the moment you realize that holidays don’t fall on the same days each year. It’s simple enough to predict the day they’ll fall on, but why bother with that when it takes someone 2 minutes every year to just enter these days in?
And then some king dies in some country and it's a national holiday the next week, and the data entry guy is in sick leaves and your table isn't updated to close trading... happened to the bank I work in, wasn't funny.
Well-designed human-involved processes try to reduce “once in a blue moon” edge-cases, sometimes at the cost of overall optimisation, in large part because people will forget how to do them, people will forget to keep building in redundancy, etc.
How would this surprise one-off holiday be accounted for programmatically? Does the government provide an API you can ping every day? Or you scrape the news feeds and feed it through ChatGPT?
Bunting just means decorations, the value is true when the government recommends decorating for the holiday. It's reserved for celebrations, so events like the queens funeral have it set to false.
Stuff like this is often officially communicated through some kind of government newsletter. Other times it's just custom and doesn't have a formal notification.
How we actually solve that problem (at least over here in Poland) is by excluding the weirder holidays from the electronic schedule altogether and deactivating all the arrival time boards and such. If you're lucky, there are cardboard signs explaining all the holiday rules at each stop, there may also be a PDF on the operator's website.
It needs to be like this. There is an underlying computation. You can cache the results of this computation in a DB table.
Sometimes, ad hoc things happen in our culture. Spurious bank holidays get added on a day’s notice in exceptional situations. These can be inserted into the cached DB. There is no calculation to back them.
This is a big one. And you don't get much notice, maybe a day or two. Typical ones in the US are death of a President or prominent Senator (I _think_ we had one for John McCain). Much easier if you can insert the holiday in a lookup table and have all your systems adapt.
I can’t prove it, but suspect a lot of business processes would be overall more efficient with less automation. Once you factor in the cost of automation, the cost of maintenance, the cost of un-fucking the computerized system when it manages to make a mistake “at scale”, et c.
Unexpected bank holidays (happened to me in 2002 during a financial crisis), official days of mourning, etc. Nothing to compute there: a government declares a holiday and you just enter it into the table.
Sure, you can compute Easter (you can just look up Knuth’s explanation of the algorithm for this…), but you can’t compute government decrees.
I work with finance data and we ended up building just that + and API. Almost everything is done automatically though. Some languages have a function to calculate Easter
We 100% refuse to do that, no way. The data in the calendar table is owned by HR. They update it. If they farm it out to their receptionist, fine with me. It's not a complicated task, but it's just complicated enough that a computer shouldn't be doing it.
We do generate all the official calendars though, so we have print options to print various views of it(paydays, etc) and we also provide .ics feeds for Google/Apple Calendar, etc.
Dates are a nightmare even before you start accounting for public holidays. There’s a whole bunch of calendar and Timezone changes, and valid dates and times that simply don’t exist. Early in my career I had a bug that involved some date in February 1923 not occurring in Greece. I think I eventually fixed it, but I was never 100% sure.
Oohh interesting. From the Wikipedia article on February, 1923:
February 28, 1923 (Wednesday)
The nation of Greece used the Julian calendar for the last time before adopting the Gregorian calendar, used by most of the world, the next day. In that the Julian calendar was 13 days behind the Gregorian, the day was noted as "February 15". The next day was March 1 rather than February 16.
At least in North America we get every observed holiday on every year. There are places (hi France) where if it falls on Sat/Sun and if you're a M-F employee, you just don't get that holiday that year. Of course you have a pile of vacation time to use if you want.
>At least in North America we get every observed holiday on every year.
Ah, but that's not so. For example, if New Year's Day falls on a Saturday, then the NYSE doesn't observe the following Monday as a holiday - happened in 2022, for example.
Look up NYSE rule 7.2 - a public holiday that occurs on a Saturday is typically observed on the Friday prior to the holiday, except in cases where that would interfere with quarter-ends and other unusual circumstances. This is why the New Years Day observance is sometimes skipped - December 31 is a quarter-end.
They also avoid being closed more than 3 consecutive days, which is why they are open for trading on Black Friday (but they do close early!).
ah, thanks for reminding me. I had some Canadian investments to move that day, and I could see if it made sense to move it at the "frozen" price that day (Canadian markets were closed), or better to wait for them to drop the next day. Saved me tens of tax dollars!
When you guys quote salaries in monthly figures.. Is the hourly rate fluctuating month to month based on the number of weekends and holidays? Or it vis a vis some standard amount of work days?
Also in Finland your effective hourly rate can fluctuate by year. That is if your salary is monthly. Depends on sector, but the hours worked depend on things like if you do shift work.
I could've expanded but for senior engineers or managers in europe it's generally easier to get better vacation packages even before adding in public holidays. I know I value extra 10 days of vacation more than like $50k/year extra on my salary.
> Technologists thought, how easy it is. A year has only a few hundred days, and there are only a few dozen possible rules for buses. Brute force is enough.
> Nobody got the problem right. It gave me a great appreciation for how complex when technology interacts with culture and customs, long before my professional career.
For those of us who weren't there, could you let us share in that appreciation: why wasn't brute force enough?
> You know things are intense when you are programming a computer to perform the Computus (calculation of the date of Easter).
Seems like that is a red herring since Easter is always on a Sunday, which, it seems to me, is all that could possibly matter given the description of the problem.
But Good Friday has the same property as Easter, namely, that it's always on a Friday (duh!). Just like Mardi Gras is always on a Tuesday and Thanksgiving is always on a Thursday.
Sure, the Sunday it's on moves around, but why would that matter? You're just trying to count the number of days the bus runs during a year under conditions like it runs/doesn't run on holidays/weekends. Easter is always a holiday, and always on a weekend. Under what circumstances would it matter which weekend?
> Financial systems are inseparably computer systems. Most similarly important computer systems don’t take holidays. Google doesn’t take holidays… or doesn’t seem to, from the perspective of a typical user, at any rate.
Wouldn't it be better for users if our financial systems worked like the Google example? Why can't I as a regular retail investor trade a stock on Saturday? Why can't I move money from one bank account to another on Sunday? Imagine not being able to send E-mail on Easter because nobody at your ISP was there to push bits around or whatever these banks' excuse is?
> “Why can't I as a regular retail investor trade a stock on Saturday?”
Stock trading on exchanges traditionally has delayed settlement. You don’t need to have the money and stock at hand when you make the trade, but you’re expected to provide them on the settlement date. For a long time this was two days later, but is now being changed to only one day in the U.S.
Trading on Saturdays would either mean complications for settlement, or that settlement must happen on Saturday too. Surely not impossible, but these things have ripple effects that are not immediately obvious.
> “Why can't I move money from one bank account to another on Sunday?”
This exists in Europe with 24/7 SEPA instant credit across banks and Eurozone countries. It doesn’t cost anything extra.
But getting to that point needed a serious mandate from the European Union, years of planning, and cooperation by the various governments. The effort was worth it because it’s a far better solution for consumers than e.g. cryptocurrencies or the American proprietary payment solutions like Venmo.
>> “Why can't I move money from one bank account to another on Sunday?”
> This exists in Europe with 24/7 SEPA instant credit across banks and Eurozone countries. It doesn’t cost anything extra.
> But getting to that point needed a serious mandate from the European Union, years of planning, and cooperation by the various governments.
Hm, it also exists in the USA with credit "within a few minutes" across banks (the one time I've used this, it resolved within a second), and doesn't cost anything extra. Getting to that point did not require a mandate from anyone, or government involvement of any kind.
It does seem to have been a four-year cooperative project between several banks, though it is not immediately clear to me that instant settlement was a founding goal of the project so much as something that got implemented after four years.
Ah. While Zelle works fine, it’s fundamentally very different from SEPA instant credit.
It’s not available with all banks. Both sender and receiver have to actively sign up to Zelle. And the transfer limits are quite low.
SEPA instant credit works with all existing bank accounts in the Eurozone. You send money to someone’s IBAN which they’ve already had for decades and are accustomed to using for bank transfers. If the receiving bank doesn’t yet support instant credit, it becomes a regular bank transfer that usually clears the next banking day. (It’s like batched email delivery back in the dial-up days. As a sender, you don’t need to care if their email system does immediate or batched delivery. And of course nowadays everyone has immediate.)
The transfer limit is 100k EUR, so the same system is useful to both businesses and consumers.
I think Zelle is limited at $5k last I tried? (When I lived in New York, I tried figuring out if I can pay rent without the absurdity of paper checks. But Zelle’s limit was so low that you can’t pay a Manhattan family apartment rent with it.)
> ... But getting to that point needed a serious mandate from the European Union, years of planning, and cooperation by the various governments. The effort was worth it
I'd take my EU strong from an economic point of view and without "yay sunday SEPA transfer" any day over the EU we have today. The EU is a massive failure and it's going to fall into oblivion versus the US and China: especially now that the EU voted to kill its main economy, Germany, by destroying german automakers (with the mandatory shift to EVs).
The EU produces metric shittons of rules and legislation which brings absolutely nothing of value to its citizens.
If anything I don't think the EU is something to strive for: on the contrary, it should serve as an example of what not to do. Even if this means no "yay, instant sunday payments".
Sure, they gave us 100 years of peace on the European continent, and hassle-free commerce, travel and communication across borders, and the freedom for any of its citizens to study and work anywhere within its borders, and modern mobile phone networks [1], and universal phone chargers, and strong protections of our personal data, etc., etc., etc.
…but other than that, what has the EU ever done for us?
OK, I admit to being (more than) a bit overzealous on that point. Still, peace has existed within the borders of the EU and its predecessors since the formation of the ECSC (one of the EU’s predecessors) in 1951.
Even if you disregard that point, though, the overarching point of my comment still stands.
Technically speaking, you're still incorrect. Algerian War happened inside the ECSC borders, however outside the European continent. Then, there were much smaller Basques-Spain and IRA-England conflicts.
The right wording might be "there were not yet a war between internationally recognized acting EU member states". That may still be an achievement, because you cannot say the same for a lot of other alliances, e. g. for NATO.
OK, I admit to being (more than) a bit overzealous on that point. Still, peace has existed within the borders of the EU and its predecessors since the formation of the ECSC (one of the EU’s predecessors) in 1951.
Even if you disregard that point, though, the overarching point of my comment still stands.
After WWII Europe was occupied by the allies on one side and the Soviet Union on the other side. It was called the cold war and peace was guaranteed by mutually assured nuclear destruction. Not by the European Union.
> The EU is a massive failure and it's going to fall into oblivion versus the US and China: especially now that the EU voted to kill its main economy, Germany, by destroying german automakers (with the mandatory shift to EVs).
German auto makers did that all by themselves. The first Tesla Roadster was released in 2011, it took a whole goddamn decade for the German industry to react.
And now the Chinese are churning out cars at a ridiculous speed.
I remember being at a robotics conference in 2014, chatting with an engineer from some German car company. He was saying that EV would never break through, because "who doesn't love the sound and feel of a combustion engine."
Of course the engineers that work at German car manufacturers love combustion engines. That is why they chose to go work there in the first place. That is why they had (and have) such a disconnect from the consumer market. Cars are built by people that love cars, not by people that want to get you from A to B. They're people that love the solution, not the problem.
The inertia of big companies starts even in people who are not part of the company yet. It is the people who love finicking with the intricacies of combustion engines that end up at car manufacturers, not the other way around.
> Why can't I as a regular retail investor trade a stock on Saturday?
Markets have chosen to concentrate trading to specific hours so as to concentrate liquidity and to help out the professional traders. If everyone's trading 24/7 then it means you need to be prepared to trade 24/7. Having defined market hours means defined hours in which important things can happen. And since the people who work in the market can also define the hours, they've chosen to give themselves reasonable working hours.
In a world where the major markets are trading 24/7, traders need to be at their desk 24/7.
> Why can't I move money from one bank account to another on Sunday?
Instant money transfer means instantly draining someone's bank account with no recourse. Checks have plenty of opportunities to stop the money, so you can write a check on a Sunday as easily as a Wednesday. Wires are more irreversible and so you can only send those when the bank is open and there's someone to double check any suspicious behavior.
Crypto trades 24/7 and I don't see anyone complaining or liquidity problems. Lots of people can't trade in the public markets because they are at work those hours. Let's just admit it s all a legacy antiquated system instead of finding excuses for its shortcomings
Except sophisticated traders are allowed to trade outside of market hours (after hours markets are a thing). When news drops, price is absolutely affected in the after-hours market.
It's only the regular Joe's that are not allowed the priviledge to trade at all hours.
Anyone can trade after hours. It’s just a shit illiquid market so you shouldn’t be doing it unless you are absolutely certain on the price you want to pay. It’s not a special privilege.
> Instant money transfer means instantly draining someone's bank account with no recourse. Checks have plenty of opportunities to stop the money, so you can write a check on a Sunday as easily as a Wednesday. Wires are more irreversible and so you can only send those when the bank is open and there's someone to double check any suspicious behavior.
We have instant 24/7 money transfer in EU/Eurozone using SEPA Instant Credit Transfer (well technically it has "up to 10 second" delay) and it hasn't really been an issue.
I can do an instant payment at the bank I work at, on a Sunday. We just warn clients that there’s no recourse but we give them the option. This is a choice, not an act of god. Our clients (including me) love it.
Liquidity had been a concern historically. Traders couldn't necessarily get good execution outside normal exchange hours, especially for lower volume securities. But now there are so many HFT firms willing to take the other side of any trade that I think the liquidity concern has become moot, at least for regular stocks and bonds.
Even during the current trading hours liquidity isn't uniform but more concentrated around certain times of day - that might "smear out" over more time, not move to the additional days, or block trades start smaller or ...
Changes to market structure should at least be analyzed thoroughly as we don't want any "parasitic" strategies to gain from changes.
> Instant money transfer means instantly draining someone's bank account with no recourse. Checks have plenty of opportunities to stop the money, so you can write a check on a Sunday as easily as a Wednesday.
This argument doesn't actually work; the period during which you can recover a check that shouldn't have been valid is (much) longer than the settlement period. This immediately shows that instant transfer does not mean instantly draining someone's account with no recourse. The recourse period isn't related to the settlement period.
You're going a bit extreme, but 5 auctions per week strike me as reasonable.
If the markets are trading continuously, benefits of parsing new information go to those who parse it the fastest. If the markets trade in occasional auctions, everyone shares the burden (or gift) of new information equally, and no resources are wasted in, say, building a somewhat more straight glass fibre connection between Chicago and NY, then building a sequence of microwave towers, because light moves faster in air than in fibre.
Some people have seriously suggested shortening trading hours, on the grounds that all the interesting trading happens at the open and especially at the close, closing prices are the ones that are reported and used for risk analysis etc.. If everyone wants to trade at the close, why not just have one "opening" auction every day, and the price from that auction is the "closing" price for the day? That also eliminates the opportunity for trying to "pick off" or "front-run" other people's orders, or "quote fade" on the other side.
> Why can't I as a regular retail investor trade a stock on Saturday?
Having worked in the industry for some years: because it's always been that way, and because the people who make financial systems are lazy and bad at their jobs and don't want to admit they were wrong.
It's happening, very slowly, just yesterday I was reading about "the first 24/7 stock exchange" making another filing or something. But the financial industry is conservative and most users are happy for it to be - sure they'll talk about how they want their bank to have a better online service, but they'll keep banking at the big four names they recognise rather than the upstarts that offer better service.
> Imagine not being able to send E-mail on Easter because nobody at your ISP was there to push bits around or whatever these banks' excuse is?
Here in Japan a lot of government systems work like that.
In the last 25 years I have seen the expansion of trading hours as well the reduction of trading hours in some markets. It really comes down to demand. Currently, the demand for 24x7 is more retail than institutional which shapes whatever will be on offer (some instis have interest, but for at least some that have interest I could imagine they are eyeing new inefficiencies/opportunities from that).
Sorry to hear you've worked in such a depressing part of the industry. I've worked with some incredibly motivated and talented people and this is not my impression at all.
Still, it's an old, large and hugely complex interconnected system made up of literally thousands of banks employing millions of people and several regulatory bodies run by different federal and state administrations.
Such interconnected systems/networks are notoriously hard to evolve, especially given that, as you also noted, people are often extremely conservative when it comes to their life savings. I don't think it's fair to blame the people working in the industry for all of that or even call them lazy.
> I've worked with some incredibly motivated and talented people and this is not my impression at all.
I worked with some incredibly motivated and talented people too - and saw many of them ground down as basic improvements took literally years to make it through in watered-down form, if they even saw the light of day at all. I was actually amazed by how few people in the financial industry are incompetent in the "couldn't write fizzbuzz" sense, but there seems to be a big-picture insularity where stuff that would be basic and common sense in every other industry (e.g. automated CI) is just not done.
Because there is a lot more plumbing involved in buying and selling equities or moving money than there is to send and receive email.
Specifically, a lot of measures to prevent something going wrong, because the the damages are far greater when moving things of great value than an email.
You technically can trade stock on a Saturday, just not using all the aforementioned plumbing of a popular stock exchange and brokerage. And you can move money on the weekends, just maybe not with the help of banks because they consider it not worth their while to offer the service on weekends.
But why? What makes the plumbing stop on weekends and holidays? Is this some kind of law of nature, or can it be fixed?
It seems silly that bank transactions, stock trades, purchases of insurance or anything like that requires human intervention. It's like an HTTP request requiring a human to put it through. Bonkers.
I don’t know all the reasons, it could involve regulators not wanting to be on call, or more likely, the businesses not wanting to be on call. I know that for lots of accounting related things, there used to be a need for downtime time to tally things up, and maybe that schedule carried over.
But overall, I would bet it’s just a low demand thing that isn’t worth it. Suppose you opened an exchange that was 24/7. Would the cost to trade on your exchange be sufficiently low to compete with the current exchanges with limited hours?
Technically, if you had a paper stock certificate, you could sell it or buy it, and if you had a bag of money, you could move it. But whether or not other people will help you is probably dependent on how much you can pay, which for 99.99% of people, is probably not worth it to just wait a couple days.
Think about real estate. It’s possible that you have a paper deed and draw up some documents and buy or sell it in 30 minutes, but the cost of something going wrong is so high that it makes sense to take a little time and use lawyers to double and triple check things.
Then I guess we have to question why those .01% of transactions need human intervention at all. It seem like .01% of requests to Google Search don't need human interaction, why does money, now that it's mostly just an entry in a database somewhere (or on its way to be)? If it's a matter of approvals, then those should be codified in rules that can be expressed in code. That may be difficult for laws, the general refrain being 'laws aren't code', but money transfer is just a list of rules, transactions, settlement dates and so on.
As a general guideline I have deleted the word “just” from my vocabulary when talking about anything engineering related. Doubly so when it’s a system used directly by humans. And double again when there are legacy systems involved that weren’t designed to have human judgement deleted from their operation.
Here’s a fun example of the complexity even when there are live humans in the loop. I’m Canadian. My client is Canadian. I sent him an invoice using QuickBooks with the option to pay using QuickBooks Payments. He pays the invoice sitting at the table with me and a transaction shows up in my books but no money in my bank account. That’s normal, it usually takes a few days to clear. A week passes and still no cash. I call QuickBooks and they provide me with an ACH tracing number; they tell me that they released the funds days ago. I call my own bank and provide them the tracing number, my bank tells me they have no record of that transaction ever coming to them. I call QuickBooks back and escalate. They insist that my bank screwed up and lost it and provide me with the same tracing number. At least after escalating I now have a direct line to a case manager at QB. Back and forth, back and forth, and eventually I get an email from a VP of Compliance at Chase Manhattan asking that I call them.
Now… I don’t do business with Chase. My client doesn’t do business with Chase. QuickBooks does business with Chase and Chase wants to talk to me about financing Iranian terrorists and FINCEN. Chase happily accepted the money from QB and gave them a receipt for it and then the money disappeared into a mysterious compliance department who, apparently, can’t legally tell me why I was flagged or what I can do to ensure this doesn’t happen again. But after a quick chat about my business and relationship with my client the money magically showed up in my bank account.
Because with transactions things have to match and sometimes data errors happen.
Take something really simple like a bank cash transfer. Do you really think that no one ever accidentally puts in a typo in an account number or an amount?
If that happens then the transaction either fails or doesn't do as desired. Either way someone has to manually resolve it.
Take that and put it on steroids for securities transactions.
In a lower reply of mine, I still have to ask why this is the case. If it's that big of a thing, then there should be rules defined to catch things. We have exception handlers for code and steps that go awry, why not the same for money, so the rest of the money flow can keep going, and only the weird things need to be checked out (and further rules then be refined).
That's what happens. Almost all transactions are Straight Through Processed. It's only when things go wrong (or very exceptional cases) that a human gets involved.
The banks can't afford for humans to be involved in too many transactions.
> Wouldn't it be better for users if our financial systems worked like the Google example?
Yes, I guess there are a few precious hours of our lives that aren't crushed under the perversity of the Protestant work cult, we should seek them out and grind them under the heel.
If anything I think it would go in the opposite direction, freeing people to do things on their own time, and not the banks'. Available when when people need them to be, not when the institutions want to be, but that would require effort on their part and maybe cost them a few basis points in profit.
> Why can't I as a regular retail investor trade a stock on Saturday?
Because stock exchanges don't run themselves. There's all kinds of services with humans in the loop, and without these, you'd have a market with much less guardrails than the current public stock exchanges.
You can make a case for such a market being good or bad for society, but it currently simply wouldn't be legal in most countries.
> Why can't I move money from one bank account to another on Sunday?
You can in many countries! In the EU, SEPA Instant is going to be mandatory soon. In the US, I can already do this between two of my bank accounts; most banks unfortunately don't participate yet.
The larger problem seems to be that the US simply does not have customer-initiated account transfers (to third parties) in the first place. In a way, the reason why you can't do bank transfers on the weekends is that you can't even do them on weekdays either.
> Why can't I as a regular retail investor trade a stock on Saturday?
Because for you it means spending few minutes to open your app and send and order, for other market participants it means organizing holiday shifts, night shifts, and working long hours. You just cannot keep the strategies running without a human eye, the lack of oversight could be too expensive. Exchanges and banks and big players can afford that (though still, you're making some people more miserable for unclear reason), but can the small prop shop with tens of people trading on multiple exchanges do the same?
Wouldn't exchanges and market makers (i.e. the big players) be all that's needed to clear orders at reasonable spreads? Others can always choose to what scope they'd participate.
Wouldn't it be better if Google worked like financial systems? Perhaps people would learn to read maps if they weren't misled into unrealistic assumptions that Google Maps will always be available, always responsive, always correct.
What is the cost, in money time, attention, engineering effort of trying to build for five 9s of reliability?
I operate a Service Bureau, which is a firm that facilitates transactions for brokers and banks. There are two calendars for US Equity Trading that we need to maintain: Trading calendar and Settlement calendar. The difference is that trading happens at exchanges/brokers and settlement happens at banks. Thus, a bank holiday may be trading day -- for example Veteran's Day. We're small, so I personally manage these two calendars (and as others say, we use DB tables).
There's a settlement concept called T+2 (soon to be T+1) which is Trade date plus 2 days; this is the day when the cash and shares must be exchanged.
On the Friday morning before last Veteran's Day, we get operations emails from one of the Banks we service that our Settlement dates for the previous day's trades are all incorrect and causing issues for them. I was very confused and very worried that I somehow forgot Veteran's Day.
But... then I realized that it was a Canadian Bank settling US Equities trades!
In their business practice they would settle the trades on a day all the US Banks would not. T+2 for the Canadian Bank was different than T+2 for US Banks. So we resubmitted all the trades with a settlement day of Veteran's Day.
As OP says, managing business calendars is hard for many reasons. I also got really jacked up by a Leap Second, a story for another day.
Oh man, I had a Bloomberg terminal, and got data from their Data License product.
Even if you have all the calendars, it’s still non-trivial to apply the calendars correctly for different counterparties based on instrument and function.
And corporate actions are awful. The Bloomberg Data license data sometimes doesn’t match what the terminal has. Bloomberg said they updated it in the terminal but would not fix their bulk data sources since the errors were for actions that happened a while ago.
To unorientalize the Japan thing here: there’s a national list of holidays. It is updated once a year. You put it in your code base. In the code base there’s a comment pointing to the page (hopefully one on a .go.jp page)
Your vendor says thing does not happen on holidays or weekends. There might be some other special day where things also do not happen. Keyword is “training day for staff”. Also end of year holidays. They will send you an email about it. Maybe 3. You read the email (important!), and put it in the vendor-specific list of holidays in your code (also important!)
And of course because you like your sanity you use whatever iCal date relativity respecting library to do logic like “the next Wednesday from today” or whatever.
Some things take 1 step. Some take 5. Being able to do something that takes 5 steps without writing a 3000 word screed about the world being broken is the sign of a successful person. Document it, do it, maybe improve it a bit, move on with life.
"The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man." ― George Bernard Shaw, Man and Superman
This doesn’t actually solve the problem because the problem is that the list of holidays depends not only on the country but also on the particular company you’re interacting with. So, is_holiday is something like a function of the parties involved in a particular contract.
There is no general problem. There is the act of doing business between parties. You do this across N relationships. You have N vendor contacts. You document them. You deal with them. This is what working with someone is like! Of course after doing the same thing 3 or 4 times you might understand some patterns, but even if you don't totally factor it out you'll at least have a general pattern.
The point is you don't write the is_holiday function, you might write "is_jp_govt_holiday" or "is_mufj_bank_holiday" (though really it might be more "is_mufj_bank_transfer_csv_acceptance_day" or whatever), that might look at "is_jp_govt_holiday". You write functions to their contexts, in a specific way. This lets you write "hacks" that aren't hacks when you get the specific email. A bunch of tiny, specific functions calling other tiny, specific functions.
5-word functions might look gnarly but they will be specific, and you got the magic of autocomplete to save you some typing.
Not at all unique to Japan either, I was confused by the Japan-centric focus of the discussion. I've worked for two different US corporations that observed a company-specific holiday, either founding day or IPO day.
It feels to me like the article sets up the confusion it must resolve by starting from national holidays... I don't think any of my employers have had a holiday schedule that matched another. There's a lot of variation on what holidays companies do and do not recognize, often with compromises around giving longer runs off around Christmas. If you really need to know when another company is doing business I wouldn't have ever thought there was a solution other than "collect a list of holidays each of your business partners observes."
Hell, a lot of US states don't match the federal holidays.
If anything the banking industry is unusually consistent for practical reasons. But even then, I worked for a major US bank and discovered that our holidays didn't quite match another major US bank we did business with!
The Japan focus is likely because patio11 lived in Japan for several decades, was a salaryman for many years, and is exceedingly familiar with their business and banking practices.
To add to that: In several jurisdictions, the calendars of when shares can trade (T, with a closing price) and when shares can settle (T+2) is distinct (say, shares can trade on NYE, but not settle). So, one definitely needs to maintain (potentially) multiple calendars per jurisdiction. One calendar per counterparty was news to me, but fair enough.
The next complication then is that these calendars can change. In Hong Kong, we have typhoons quite occasionally between around August and October, and if the observatory declares a T8 or above, several businesses close, including the stock exchange. It then depends on the exact timing of the raising and lowering of the signal whether there is the usual trading, or half day, or no trading, which implies no closing price, thus an additional holiday for the trading calendar.
To treat holidays properly, one must then take into account the day it was declared for some purposes.
So every company in Japan has their own public holiday, and they've written it down _somewhere_. Sounds easy compared to what the middle east has to offer on that account:
- When Dubai was awarded the 2020 World Expo the following day was declared public holiday (yes, less than 24 hours notice)
- In the UAE, the death of a Sheik is often followed by 3 days of mourning where government offices are closed (this can include a lot of office workers one might not think of as "government", as the respective emirates runs a lot of companies)
- When Qatar scored a goal at the world cup, a public holiday was immediately announced for the following day.
- Even planned public holidays, such as Ramadan (not a public holiday as such, but work does grind to a near halt) and Eid are determined by sightings of the moon, and the start and length of these are often announced days in advance.
In my country on a public holiday everyone is free to spend money so all the stores and restaurants are open all day. What are people supposed to do on a holiday if not consume product?
Same here in ME. Every business in retail and hospitality works on weekends and holidays.
For Dubai at least, they have something like 90% foreign workforce, so it is easy to find people who do not observe Eid.
But in the context of the article, we are talking banks and offices and particularly KYC requirements. I’m a bit unclear how banks are affected by these “tomorrow is an advocate holiday!” announcements, particularly the banking software.
> Technologists describe their systems as having “uptime” and measure it in “nines”, such as “We have five nines of uptime”, which means that a system has 99.999% uptime or, equivalently, about five minutes of downtime per year. Five nines is admirable in many circumstances and would be considered _disastrously_ below expectations for e.g. Google Search.
This seems wrong? 5 9s is probably a reasonable benchmark or even unattained goal for Google Search, right?
> [4] Readers of a certain age might sensibly ask what “cyber” means. Consider it a way to gesture broadly at technology used almost exclusively by people who both do not understand technology and feel some amount of pride in that. Teams at large retailers, believing online commerce was doomed to be a tiny sideline like catalogs and only worth tens of billions of dollars, were involved in naming Cyber Monday. The other place you’re likely to hear it frequently is American national security circles, which exist in a superposition of understanding that technology can certainly be used to kill people and break things while also believing that it’s not a real way to kill people and break things if it is the sort of technology built by people who look like pre-juice Steve Rogers.
> We (the financial industry) teleport value through time and across space and make this look easy.
I don’t know if it’s possible to describe finance better in less words?
Anyway, if this quote piqued your interest, you should familiarize yourself with this author’s still incomplete work: https://open.substack.com/pub/concoda
I love undefined holidays, such as there is a holiday in this or that state of this or that country whenever the final of a play-off series is to happen.
I like patio11’s writing, but this has so much joking commentary compared to his usual articles. It felt like a discworld novel. If patio11 sees this: I like your writing, and your jokes are funny, but I’d like them more if you just kept the half that were funniest.
... and then you discover that the problem also extends to the supranational level. European countries have holidays, but Europe-as-a-whole has holidays too (from the banking perspective). e.g. the holiday calendar for TARGET2, which is certainly neither the union nor the intersection of the holiday calendars of its participants.
Also: I guarantee you that there’s a team at every major U.S. bank with a crisis management plan ready to kick in when Jimmy Carter finally passes: U.S. markets are closed by custom on the day of a Presidential funeral.
There’s little worse for an ops team than a holiday that you can’t predict.
Good news on this specific issue: the current year will be both the final year of the old era and the 元年 (first year) of the new one. The following calendar year is the 2nd of the new era. So in most cases you have a window of time to get ready to update systems to accommodate e.g. Japanese counterparties whose systems want you to specify dates of birth using eras in preference to the Western calendar.
One of the relatively few bits of ultimately consequential engineering I did in the last few years was assisting a colleague in grepping for comments we had both left identifying code to update in the event of a new era. (We are currently, of course, in Reiwa 6, and Heisei ended in what most HNers consider 2019.)
This is a great article. Everything in it is 100℅ true if you've ever worked in finance.
The absolute most nightmarish situation you could possibly imagine is the overnight batch cycle running late or even worse, having some portion of it not complete. I've been involved in the cleanup of those sorts of issues many times, they tend to be incredibly complicated chains of hundreds of programs from all different departments, and there is not a modern concept of a single cohesive "application" that runs. Its a very crazy world that even web developers will inevitably encounter a bit of if they ever take a job at a bank for instance.
It was an interesting blog (as usual), though I'm really surprised by the assertion that most Americans get two holidays for Thanksgiving (Thanksgiving plus black Friday). I have almost never in my life had a job where black Friday was a holiday, nor has anyone else I know. Sometimes I take the day off (because it's nice to not have to go back to work for just one day), but I certainly wouldn't say it's a holiday for most people here.
I find it fascinating that pg's and patio11's writings are popular on HN at the same time. The contrast is fascinating not just in topics and writing-style but with the identification of engineers. The first guy gave engineers the 'hacker' identity, the second guy speaks about and partly identifies himself as 'technologist'. What a contrast! :)
> In an increasingly interconnected world where decisions are increasingly made by people who count and value nines, you can reasonably expect financial systems to partially close the gap between historical practice and contemporary practice of e.g. Google Search.
this chap has clearly never worked for a bank
in the 2020s I had to write code to handle files in EBCDIC
I suspect that system producing it will outlive IP, TCP, HTTP and JavaScript
Never thought about the fact that holidays are both when there are surges of consumer spending but also when money doesn't settle.
Also quite funny that black friday was translated as a sale on black things in at least one instance. When I grew up a decades ago in east/SE Asia, black friday was not really a thing, but I guess times change.
Interesting, but honestly not fond of the style of writing of the author.
I will add another common source of problems in Italy: every single city or village has its saint, and if it is that saint day on the calendar that city or village shuts down.
But here lies an issue: if you are employed in a different city then you reside (e.g. you work for a company in Milan but you live in Rome), you get the worst combination. You don't work on Sant'Ambrogio's day (Milan's saint patron), but you get to work in your city's holiday, so when everyone is enjoying and having fun you're working.
And yes, this has technological implications of the sort that author describes.
I once lived and worked in the US. To my surprise (as a Brit), each year we had a public holiday from work but where my kids still went to school and many other places were still open.
I'd probably just create a config file, where I'd store all known Easter dates for the next 50 years. If software still works 50 years later, they can extend the config file.
This was incredibly fun to read and a wonderful rabbit hole.
Technologists are often defeated by their assumptions of how easy it is to solve a problem, but it’s also how we get ourselves in to solving impossibly hard ones.
Of course the natural conclusion is we repeat the same architecture in the name of new ones.
One of the unaddressed problems with a hypothetical 4-day workweek is it means financial transactions, mail, processing etc. will be even slower and productivity may suffer. Wires and will take forever to land, checks take forever to clear, mail takes forever to be received
If we do T + N -> T + (N - 1) settlement at the same time we declare Wednesday a bank holiday, settlement that used to fall on Thursday still will, and settlment that used to fall on Wednesday will fall on Tuesday instead.
Easy peasy, as long as N is currently >= 2. Changing from T + 1 to T + 0 is a big switch. T + 0 settlement isn't really effected by Wednesday being a holiday in that settling transactions isn't delayed, with the large caveat that entering transactions is.
“ Financial systems are inseparably computer systems”
I think this is where he goes wrong. The fact that things stop working overnight means that there is some human intervention required some of the time such that the risk or cost is too great to run continuously.
Same thing happens with porting mobile numbers between carriers. Sure telecommunications systems are computer systems but some percentage of cases must require oversight or intervention enough that it’s not viable to run overnight otherwise they’d do it.
Assuming banks, and other business or subsystems, are open by default is a bit silly when you have to interact with more than a handful.
Why not just assume they are not available on any given day unless proven otherwise?
A positive list of what's actually confirmed is much better than trying to find every obscure holiday everywhere on Earth and then adding it to a negative list.
From working at places that do electronic trading, most have some sort of central db, as well as procedures that mostly work for those times when you realize it's a trading holiday or a special session Right Now.
What's worse is when timezones/DST changes at short notice, because that involves rolling tzdb, or shudder the JVM.
>Five nines is admirable in many circumstances and would be considered disastrously below expectations for e.g. Google Search.
This is BS. 5 nines is great for pretty much any system especially if extrapolated over 5+ years. I stopped reading right after this statement as the rest is probably BS.
Technologists thought, how easy it is. A year has only a few hundred days, and there are only a few dozen possible rules for buses. Brute force is enough.
Nobody got the problem right. It gave me a great appreciation for how complex when technology interacts with culture and customs, long before my professional career.