> Raj thought that checking the "principal" checkbox and entering the number of a Citibank wash account would ensure that the principal payment would stay at Citibank. He was wrong. To prevent payment of the principal, Raj actually needed to set the "front" and "fund" fields to the wash account as well as "principal." Raj didn't do that.
I don't even have anything to add. The paragraph speaks for itself... You can't make this up
It is also worth noting that Raj wasn't alone in the confusion on how to perform this task. Before he could complete the transfer his boss in India also had to look at what Raj did. Then their boss in Deleware also approved the transaction.
ALL THREE PEOPLE INVOLVED IN THIS TASK WERE CONFUSED by the interface and all three thought that this was the correct way to do this.
This wasn't the case of one person checked the wrong box. Everyone who looked at this was confused. This problem was not individual, but systemic.
Totally NOT SURPRISED that this was an awful Oracle enterprise app.
These things were put together in haste in the 90's and Oracle keeps pushing them and their corporate customers keep these things going long past when the interface becomes utterly unfamiliar.
The blame has to be shared, however, with the battle-axe personalities of the people that use this stuff. They would like nothing more than to retire while still using the _exact_ same applications and dated interfaces they've used for 20+ years. Literally, these are default "grey-theme" java applications, with all kinds of inconsistent UI quirks and absolutely no shits given for discoverability or even being able to create hyperlinks so that people can share stuff without screenshots.
It's fine if it's the same people keep using these apps forever, but the problems start when new people get on-boarded. If it's like most places, the newbies will be given no training and the only help they will get would be from some dour in-house oracle-application jockey whose been doing the same thing for decades. That's when people make very, very expensive mistakes. We hear about this one because the sum was vast, but people make $10,000 mistakes on shit software from Oracle all the time-- that's partly why Oracle has a very deep bench of retained lawyers, I suppose.
> These things were put together in haste in the 90's and Oracle keeps pushing them and their corporate customers keep these things going long past when the interface becomes utterly unfamiliar.
Actually the history of this app is a little more complicated. In the early 1990s, Citibank decided to establish an outsourced software development group in India to rewrite their banking software from scratch–they established it as a subsidiary company called iFlex. From 2005 onwards, Citibank progressively sold a majority of its stake in iFlex to Oracle, who then renamed the company Oracle Financial Services Software (OFSS). OFSS is listed on the Mumbai stock exchange, but Oracle owns the majority of its stock.
So, rather than being a product which Oracle developed themselves and sold to Citibank, this is actually in some ways the opposite – a product Citibank developed themselves and sold to Oracle.
I used to work for Oracle. I never worked on OFSS software directly, but I did work on an implementation project for some of it. I never saw the internal banking UI, all I ever saw was backend stuff – LDAP, BPML, JVMs, databases, load balancers, firewalls, etc. Indeed this article is the first time I've seen that particular UI in my life. But I can tell from the visual appearance that it is not one of Oracle's more recent UI development frameworks. (I don't know if this is because Citibank is running an older version, or if there are parts of the product still running on a legacy UI framework.)
That scroll bar is an Oracle scroll bar - you can see it in the GUI installer for the RDBMS (and in all Oracle Java apps). The Borland-like checkmark button can be something Citi-specific or just a developer who liked Borland enough to import the graphic into the app's UI.
When the 3D look became the norm for Windows apps, introduced by Office and, later, by Windows 3.1 (or 3.11), I used the DLL to 3D-fy my Windows app dialogs automagically. I also took some liberties with using Office graphic elements reasoning that a Windows or Office user would know the 3.5" floppy saves things.
Yes, it looks like OWL framework which was bundled with Borland C++. Google Image Search for "borland c++ owl" yields more screenshots in the similar visual style.
Oh wow - gotta wonder how this "arms length transaction" got past any internal auditors or technology officers, or if no one was looking. (Then again, did Citibank have a CTO in the 90s?)
Citibank developing in-house software then selling it to Oracle so that they can charge recurring licensing and support fees for eternity is incredibly short-sighted if you're Citibank or Manna from heaven if you're Oracle.
My guess though is the Citibank executives who signed off on this got promoted up and out of the department that has to budget for these licensing fees and support costs, and on both sides they received tidy bonuses for "increasing revenues".
The software fees are likely reduced/free.
I've worked at companies that have sold in house software to other companies to make them public. It's usually resulted in lowering in house maintenance costs and the licensing agreements have a sweetheart deal for the original company.
Usually it's a win win, unless you value keeping control (which can be quite valuable)
> Usually it's a win win, unless you value keeping control (which can be quite valuable)
My experience is that it's usually overvalued. An external, focused company will usually do a better job than an internal project. It's emotionally hard to give up control, but successful companies tend to focus on a few things, do them really well (areas of competitive advantage), and outsource everything else.
A best-of-breed external solution almost always beats an in-house solution.
I'd just never do something like this with Oracle. Their business model seems to consist of locking companies in, and then milking them.
> My guess though is the Citibank executives who signed off on this got promoted up and out of the department that has to budget for these licensing fees and support costs, and on both sides they received tidy bonuses for "increasing revenues".
Oracle paid Citibank over US$500 million for this company. There is no way a transaction that big isn't being approved by the CEO and board of directors of both firms. It wouldn't be under the control of the software licensing department. It would be controlled by the business development group in charge of M&As and divestments.
> Citibank developing in-house software then selling it to Oracle so that they can charge recurring licensing and support fees for eternity is incredibly short-sighted if you're Citibank or Manna from heaven if you're Oracle.
I have no internal info on the details of the Citibank-Oracle relationship (my role at Oracle was technical, not the kind of role where I would know that kind of business relationship stuff, and I never worked on the Citibank account either.) But, as @dagmx has already pointed out, I think it is highly likely due to the history of this software that Citibank has some kind of special licensing deal with Oracle which protects them against that sort of thing.
Even if AaronFriel's assertion is true and they received no special deal, it's a naïve view of corporate governance and SAAS v.s. in-house devleopment to say that (as an example) if their license fee was high enough that they'd start incurring a net loss in 10-15 years that they made the wrong move.
For Citibank that was US$500 million then, which they could either invest directly or return to investors. The opportunity cost of that cash has to be factored in.
It's also an a large ongoing liability to have a sizable in-house development effort when it's not your main business. Departments need to be staffed and run, business plans made etc.
For all of Oracle's flaws they're presumably going to have an incentive to improve the software, and more capital with which to do so from clients other than Citibank. For obvious reasons other banks would be more inclined to buy from Oracle than a direct competitor.
There's also often tax reasons for why it's preferable to buy a service v.s. maintain in-house software, and naïve back of the envelope math usually doesn't account for that.
> It's also an a large ongoing liability to have a sizable in-house development effort when it's not your main business. Departments need to be staffed and run, business plans made etc.
This is the same ideology that led to companies outsourcing everything from facility management and cleaning services as the first victims to stuff as critical as IT operations.
Yes, it is a liability and likely also a higher cost (e.g. due to collective wage agreements) to do that stuff yourself. On the other side, you have the large liability of having no direct control over staff and work quality any more - you're entirely at the mercy of your contractors (who are incentivized to find not the best people, but the people willing to be paid the least).
> ...no direct control over...work quality any more...
This is important. Just a few days ago here on HN, there was a good discussion on how deliverables quality on seemingly "mundane" device chargers (in this example, one that was arguably for early Kindles) can vary wildly, while maintaining the same price point, with the sub-standard quality yielding more profits to the seller, and higher-quality units yielding less profits.
This is a impedance mismatch between what the buyer knows and values and what the seller knows and values. With a time slippage between the time the buyer makes a decision and when the buyer's organization figuring out the real ramifications. By the time an organization has outsourced enough to lose competency to not even know the impedance mismatch and time slippage exists or scope it if they're aware of its existence, sometimes they will find might be cheaper to in-source in the first place. My personal rule of thumb going in to evaluate these decisions is if the procedural complexity surrounding such software artifacts rises above a certain level, it is likely better to keep it in-house. Where that level is lays the art of business, it is largely experience-based.
On the other hand, they also sold one of their core competencies away (managing complex financial transactions) and it just cost them half a billion dollars and who knows how much they've lost since to other errors, let alone lost productivity.
Why would the arms length principle be a concern here? It’s generally only a concern in regards to transfer pricing of transactions between entities that have common ownership/control.
I can confirm it is OF. However versions 7 through 10(maybe 6 as well) all look like that so I can't say which one it is. I used to work in a company using OF for the UI and I can say that while it is horrific to look at, it had its strengths. However it was appropriate 20 years ago when monitors had standard aspect ratio and resolutions. Nowadays with scaling and all kinds of aspect ratio it just doesn't work. Also Java... Ironically for one client our software was interfacing with Flexcube.
However the OF technology is still supported by Oracle and while they themselves are trying to kill it, nobody is willing to migrate their core systems to something new.
It blows my mind that Oracle gets paid huge sums of money for the garbage they put out. I went to a CUNY school and they rolled out a "new" administration system called CUNYfirst. Literally a several $100 million dollar contract and Oracle provided a slightly modified version of Peoplesoft. The UI looks like something that should have never existed on this earth. Every student gets an Employee ID, you choose classes for a semester by adding then to a shopping cart and checking out. Minimal effort from them for maximum cost.
That’s aggressive sales department, make no mistake about it.
It’s actually easy to sell shit software. The average person knows a lot more about cars, for example, thank they do software. I’m sure Oracle makes huge money on name alone. The IT manager doesn’t want to take a chance with some lesser name. Just get Oracle, then nobody can blame them when shit hits the fan.
As shitty as it is, do you think that the CUNY school has the ability to write a spec for an administration system that handles all of the business needs and compliance requirements? Or do that and manage the delivery of those requirements?
The answer is of course no.
The Oracle pitch is "We do this for Michigan State and can do X, Y and Z". That is true.
Oracle gets alot of shit for this stuff because they own the market for enterprise financial software. Their software is gross, they bought out the competition and generally suck, but that is what the market demands.
My uncle is a locomotive engineer. His railroad makes a conscious decision to remove any creature comforts from the cab, like a chair cushion. It's needlessly uncomfortable and is awful, but that doesn't mean that GE (or whomever) sucks. They delivered what they are asked to do.
To be fair, comfort can be a mixed bag for system safety, especially when humans are alone, safety critical and on weird schedules which can all be the case for train drivers (which is what I understand "locomotive engineer" to mean in this context) - because comfort is conducive to sleep.
One of the setups I come to almost expect in maritime accident reports is an officer taking a night watch, alone, while somewhat fatigued. Warm, dark, quiet. After a few hours the ship has drifted significantly off its intended course. For unknown reasons (the officer was probably asleep) no attempt is made to correct... When they do wake up, they often don't have good situational awareness, and their focus is on pretending nothing is wrong, even if they are in fact already in serious trouble and need all the help they can get.
Night-time railway accidents are less common in my country, perhaps because the relevant unions insist on proper "if you're too tired just go home you aren't fired" type rules to reduce fatigue errors, but I do recall a case where a freight driver was dipping in and out of consciousness. When he was conscious he could see a red lamp out of the window, signalling "Danger" ahead so he'd reduce the traction power. If he'd been less sleepy he'd have noticed those red lamps were not getting closer each time, they were moving gradually away from him. His train was on a steep climb, and he'd reduced power so much it was gently rolling backwards. Oops. Nobody died, that time.
> do you think that the CUNY school has the ability to write a spec for an administration system that handles all of the business needs and compliance requirements? Or do that and manage the delivery of those requirements?
WTF are the administrators paid for if not that? Isn't that literally their whole job?
I went to QCC from 98-01 and you'd sit in the admin building on an IBM green screen terminal and setup your schedule. There was little to no prerequisite checking so I made all sorts of goofy schedules and took classes out of order. I don't remember having any issues with it as the interface was pretty strait forward.
I wound up going back several years later when I was thinking of a different field and they had that trashcan CUNYfirst system. By then the UI was already outdated and clunky. I wound up with the wrong class the first time around because of some issue and have to have it changed post payment which was time consuming. I would have rather used the IBM's.
But in the case of university classes, you are in fact buying them, it ain't free. So the shopping cart actually does make sense even if its a terrible use of another use cases' interface.
In college, we had a two-week stretch at the beginning of a new semester when you could audition classes and add/drop them at will. This was called the "shopping period", so in my view, the shopping cart analogy on the registrar's web site doesn't sound too out of place here. Of course, the logic behind it would need to be different from an off-the-shelf e-commerce cart.
Yes, but to me, a “shopping cart” implies a purchase. It’s what I use at a grocery store to put my food and stuff in. A “cart” would be better term IMO despite just being a shorter form of “shopping cart”. But idk.
But you signing up for classes is you buying something.
I used a class registration system that wasn't this and didn't mention a "shopping cart" metaphor anywhere, but the system was exactly the same. Why object to calling it a "shopping cart?
The University of Wisconsin used (and still uses AFAIK) this system as well. "Shopping" for classes was horrendous, and there were at least three interfaces to search for classes.
I can only imagine how hard it was to populate the available classes. I wouldn't be surprised if part of the huge administrative bloat in the various departments was people who's only job was to force data into and out of PeopleSoft.
10 years ago I did exactly this for the University of Wisconsin. My job was entirely to facilitate data extraction out of oracle databases / PeopleSoft for longitudinal studies done by faculty or other interested parties.
There were a few dozen of us spread across the UW System who's sole job it was to extract and massage data from these databases... One, very large, central database that handled data for UW and every satellite college they had.
I have to use an Oracle product for timesheets and besides the UI which is as you can imagine reading the previous comments, contains javascript code for a 'finite state machine for controlling the notification popup component'...
[edit: typo]
Oracle? I didn't realize that from the article. The entire situation makes A LOT more sense now.
I once was deposed in a lawsuit regarding Oracle, and part of it was basically a UI issue. We had a paid customized interface that allowed users to upload basically a blob text or document (word, etc). When we went to UAT, we didn't see it in the UI in the app itself (peoplesoft):
Us: "Okay, we can upload stuff, where do we find it?"
Oracle: "Oh, that wasn't part of the project"
Us: "Yes it was, see spec #<insert #>."
Oracle: "... Well, it's in the database. A DBA can access it, so we met the spec."
Us: "That is not a rational interpretation of '... and users will access it within the application...' "
Oracle: "We're happy to build that customization for another <insert massive $$$ sum>"
This was only one of a few material issue that led to the lawsuit. We probably wouldn't have sued over it alone, which should give you a hint at just how bad some of the other problems were.
As an aside, the $/hour billed for customizations was $600, but the work was also outsourced to India. I asked a PM from Oracle how this was justified, and they gave a vague answer about multiple layers of management needed to oversee the developers. I pointed out that this price tag would allow for 3 managers per developer, with each person in the chain making ~$200k/year, and still allow Oracle $200/hour profit. I got a ¯\_(ツ)_/¯ response. That I have to blame on the organization & not Oracle though. I had the detailed ~500+ page RFP response & implementation plan, which specified 1,000 hours of custom work. But I didn't have the actual contract to see what additional work would cost. The organization either didn't negotiate that in advance (so their fault) or they knew in advance & agreed to it (also their fault)
The thing was, when it rolled out (2007), everyone was thrilled with it because it replaced a mainframe that you had to telnet into. That mainframe had hours of operation, i.e. you couldn't look at the course catalog at night because they shut down the mainframe to save power.
Oracle makes a ton of money because what they're competing with is just incredibly terrible.
The last college I attended also shut down their servers from 11pm-7am, so you couldn't make payments, check your balance, view your transcript, drop a course...
I personally despise Oracle for this. I've seen it time and time again and every enterprise I go into, Oracle is my first target for death. Almost all say "But it's critical" until it isn't.
The problem is partly a lack of choice in off the shelf products. In some industries the choice is basically Oracle, or a 3rd party app that also runs on an Oracle database. The alternative is developing your own system, which really isn't an option for anything but large organizations.
Things are a little better these days where you can often find apps (usually SaaS) that fill a specific need very well, but then you're usually stuck managing a dozen different products and a massively more difficult integration between all of them.
If you're using Oracle to power a home-grown system though, yeah: With a little effort you can swap the backend for something where the vendor won't show up with a surprise multi-$million invoice along with pre-made lawsuit papers if you don't sign it.
Absolutely. In trying to save time business wants an off the shelf product that Oracle claims they have. This isn’t a cost saving move (though I have encountered a naive VP who thought so), it’s strictly to reduce go-to-market time.
What really happens is entirely different. Oracle takes the request, foots the bill for an army of contractors to build it, if they haven’t already, and pass it off as we had it the whole time, just like we said.
Then up charge. Shake down street. Lock in.
I’m sure not all of Oracle operates this way, but that’s been my experience.
Homegrown systems are an easy low-hanging fruit to cost reduction.
Software to tackle the BPaaS on top of an Oracle add-on service, requires a pitch deck.
Oracle takes the request, foots the bill for an army of contractors to build it, if they haven’t already, and pass it off as we had it the whole time, just like we said.
That must be what happened in my case! I didn't realize it was a standard practice. A large part of the lawsuit revolved around functionality the RFP required, Oracle said they had, and that Oracle even showed in what may have been faked screenshots during our review of vendors. Only in our case, the army of developers never actually finished it and Oracle came back to us with another massive bill to complete the project.
I'm not sure I've ever heard someone who's worked with Oracle who didn't have a bad experience. But if you have and you're reading this, just drop a quick comment to say "yeah, they were okay"
I've had nothing but amazing experiences with Oracle. I can basically eat at whatever restaurant I want with our rep at any time, no matter the expense. I can get catered box tickets to any game. I get gift baskets that actually kick ass.
There was a complaining faction in my org, mostly disgruntled hold-outs from before my tenure. Whatever complaint they thought they had, I could bring it to our account manager, and it was pretty easy to see the root of the problem came from personal issues in staff, and insecurity with changing to better technology that they were not experts in.
Now that I've got mostly oracle certified engineers and oracle familiar managers on board I don't hear any complaints. It's a great product, and in all my future roles, transitioning to Oracle is one of the first things I'll do when I get on board.
This sounds like this job was done by consultants (could have been Oracle consultants too) and this is standard behavior across consultants (that doesn't make it right though).
Oracle as a product company (applies to all Enterprise Product companies) will not charge separately for a customization.
I've worked with other consulting groups-- such as on the successor project after the dust settled on this one-- that were not nearly as bad, so I guess sleaziness varies significantly.
In any case, these were from the the actual consulting arm of Oracle inc. And IIRC, our Associate VP for IT said we shouldn't use both Oracle & Oracle Consulting, but that person was overruled by the VP for IT. I the VP was kept around until the lawsuit settled because it might have been taken as a sign or admission of culpability to fire the VP before that, but shortly after the settlement (which we won, incidentally) that person moved on for "other opportunities".
Not a browser-based app. It was an extremely customized peoplesoft. And the whole project ended in a lawsuit. We had learned a bit, so we took our time to get our duck in a row, and it was about 4 years before we started fresh with a different vendor.
Was that project worth it? We didn't have much choice. Our old ERP was close to 35 years old, and no longer supported, so we were paying large sums of money just to get annual compliance updates for state & federal requirement. The new system is fine, does what it needs to do, but it's a monolithic ERP system: It does just about everything, but it's mediocre at all of it with most of it's tech perennially 4 to 7 years out of date with the best of breed. So, in the areas where we really need best of breed, we bought that and integrated it into the ERP, which is considered the "system of record". So it's fine, it works, I guess. The real kicker though is that the system we ultimately chose, while not an Oracle product, still required an Oracle DB. Fortunately we don't actually deal with Oracle as a vendor though. Our vendor is basically a VAR.
> The blame has to be shared, however, with the battle-axe personalities of the people that use this stuff. They would like nothing more than to retire while still using the _exact_ same applications and dated interfaces they've used for 20+ years.
Are you sure they wouldn't like a new interface if it was actually a real improvement?
I'm reminded of a story[0] about Norwegian doctors refusing to use the new GUI system and stubbornly continuing to use the obsolete text-based system. Were they luddites? No. The old system let them quickly navigate with the keyboard in seconds while talking to the patient. The new system required them to use the mouse, so they had to spend a lot of time focussing on using the GUI, rather than helping the patient.
Wife is a pharmacist, can confirm this is true. In school she worked for a chain where everything IT related was done on an awful MSDOS program- she would get calls from other pharmacists asking how to do something and I would hear her walking them through "hit ctrl-F12 then hit tab three times now enter the furble and then return then tab twice more and enter the gurble" and it sounded terrible. But if you had built up the muscle memory, you could do whatever needed to be done while being polite on the phone to an irate doctor who was super pissed that you were questioning their judgement in an effort to save the patient $300. Their new system was an internal web app that didn't do any of that.
Sometimes, a core job of IT is eating all the pain of using the old technology that the users already know so they can go do their job and the company can achieve its objectives.
Well, there's "green-screen" VT100 terminal apps, and then there's 90's Vintage Java/oracle-forms UI's and then there's modern applications designed with informed UX approaches.
The terminal apps, at least, have some history behind them and properly trained folks can be very fast with them.
But we're talking here about shitty early Swing-apps (or Oracle forms). If you've ever used one it can be incredibly frustrating. Moreover, people are usually just tossed into a role where they have to use these M-F-ing interfaces with no training. Who makes a table with cells too small to fit their content and where you can't resize the table? Oracle. That's who. And they're paid for it like you wouldn't believe!
> The old system let them quickly navigate with the keyboard in seconds while talking to the patient.
Encountered a similar issue many years ago at a company trying to sell a new ad booking system to Titan - they had crufty old VT100 based systems, the new one was fancy AJAX HTML whizzbangs. Exactly the same result - they could handle 5 bookings on the old system in the time it took to do one on the new (which was also terrible and ugly besides slow.)
> The blame has to be shared, however, with the battle-axe personalities of the people that use this stuff. They would like nothing more than to retire while still using the _exact_ same applications and dated interfaces they've used for 20+ years. Literally, these are default "grey-theme" java applications, with all kinds of inconsistent UI quirks and absolutely no shits given for discoverability or even being able to create hyperlinks so that people can share stuff without screenshots.
Here is the thing, the reason these people would want to keep using the software is... The replacement will be broken and they'll spend years with bugfixes and what not to get to the exact same position they're in just now.
Using old UIs isn't fun but if it works it's good enough. This is a case of everyone getting used to the workarounds until no one knew about the workarounds.
To me it looks like a victim of cost saving and overoptimization.
Somebody requested that feature, and then someone discovered that if you hack around your system and set precise combination of fields, it will do the trick. So man hours were saved, some memo added to docs, and then the knowledge was lost.
And all of that instead of adding separate form for specific case. I've seen it more than once.
This is precisely the story with the reporting software at the company my partner works for.
Everybody who was there when the software was created is long gone. It hasn’t been updated in decades. Nobody really knows how it works, and training is entirely folk wisdom and cargo cult practices handed down over generations.
Yep. We recently retired the Oracle reporting application at my employer. It was called "Discoverer" <-- not just "Discover" but really with an extra "er" at the end. Massive pile of shit that only a few people had the patience (or stubborn desperation) to master.
The people that did know how to use it got constant phone calls (yeah, PHONE CALLS, because that's how these people roll) to do ad-hoc queries for management. I guess I can only fault them so far. This skill put food on their table and kept them employed even though their end product was endless dowdy bar-charts and badly formatted tables.
Now we're using a new product. It's a "BI" tool.
So many corps are doomed to repeat the bad parts of computer application history over and over and over again. In the case of "BI" or "Business Intelligence", it got it's start when middle-management got tired of begging for reports that weren't baked-in by the application vendor years ago. They started with getting permission to directly query the database (a profound coup by their standards). When they found that it was _actually_ possible to get useful information from the database, it looked like "intelligence" to them-- hence the term "business intelligence".
Yep, I had never heard of it but recognized it as an Oracle app immediately. Not surprising at all.
Oracle will audit them and find that someone accidentally enabled an extra, unused feature during an install one time. So, there goes another 10 million.
Keeping this in mind for the next time someone asks why we need to reinvent UIs every so often, or why technical debt needs to be paid off. Because they tend to cause financial damage, that's why.
Government uses Oracle based financial software too.
I have a soft spot for Citi because where else can you get 2% back on your credit card? They're not bad, they just like giving money away to their customers...
These 'double checking' things are like a worst case scenario of code reviews.
There's a psychological defect in the human brain called Anchoring. If you are first presented with the wrong answer to a problem, it affects your ability to objectively come up with the 'right' answer. You will not catch all of the garbage I threw at you because I threw it. "Double" checking is a misnomer because it's not really doubling the number of brains.
Some things should work more like nuclear launch codes. I don't approve your action, I perform the exact same action, and if we both did exactly the same thing, then something happens. If we don't, then nothing happens or you get an error.
Might this problem still have happened? Yes. But the first time it happened would probably have been taken more seriously, instead of what we usually do: deflect, deflect, deflect.
Doctor 1 gathers evidence and makes a preliminary diagnosis and writes it down. Doctor 1 then presents only the evidence to Doctor 2. Doctor 2 comes up with an independent diagnosis and writes it down. They then see if they match.
Selecting which evidence to present leaves this process subject to some of the same errors. The evidence that leads doctor 1 to conclude the correct diagnosis is X is going to be passed along and other evidence may not be, so doctor 2 will diagnose the same. (This doesn't even have to be intentional. If I think I'm right, I'll be naturally inclined to provide evidence to that effect.)
Looking at the UI, anyone would be confused. I feel bad for the employee put into the spotlight. The spotlight should be on whomever designed and approved this software and UI.
Unfortunately because of the court case, instead, people associated are the users.
That UI is depressingly awful, and not having a confirmation screen with a summary of the transaction about to be initiated is so very very bad that it should get people fired.
Pretty much every modern financial app I've used has one of those confirmations - some better than others but all pretty clearly state "you are going to transfer $XXXX.XX to ACCOUNT_YYYY -- click here to approve transfer".
Even venmo's quick prompt to make sure you really mean to send your pet sitter $125.00 before initiating a transfer would have avoided this problem.
The thing is, this is where the difference is between internal and external tools. Your client facing app has to confirm every action because the client isn't expected to have a deep understanding of the process. Bank employees are expected to have that understanding, so I'm not surprised there is no confirmation window. It's like with Linux - certain completely destructive commands have no confirmation before running, because clearly the user knows what they are doing - otherwise why did they type in that command in the first place.
> Bank employees are expected to have that understanding, so I'm not surprised there is no confirmation window. It's like with Linux - certain completely destructive commands have no confirmation before running, because clearly the user knows what they are doing
Well, no, it's not like that at all. There IS a confirmation window; it's just not embedded in the software. Three different people -- the flunky, his boss, and his boss's boss -- were all required to confirm the validity of the transaction before making it.
And there was a confirmation window in the software, it just didn't distinguish between some and all of the funds being wired out of the bank.
> Raj then proceeded with the final steps to approve the transfers, which prompted a warning on his computer screen — referred to as a “stop sign” — stating: “Account used is Wire Account and Funds will be sent out of the bank. Do you want to continue?” But “[t]he ‘stop sign’ did not indicate the amount that would be ‘sent out of the bank,’ or whether it constituted an amount equal to the intended interest payment, an amount equal to the outstanding principal on the loan, or a total of both.”
(from the judge's opinion, via Matt Levine and Bloomberg)
In this case it may be the best option you have, but the UX you should be aiming for is always undo.
Humans are always sure this is what they wanted to do right up until they understand the consequences. Then they experience regret. If possible design your software so that regret has a natural response in the interface in the form of an "Undo" option.
A confirmation step doesn't trigger that "understand the consequences" outcome, and so users will mostly be annoyed by the confirmation, and pick "Confirm" even when in fact they'll immediately regret that once they do it. They may even ask you to add another "Confirm" step. You want "Undo".
Notice the law here reflects this preference for Undo. If Citibank wired $500M to me the law says they get to Undo that, because they didn't owe me $500M and so that's just a mistake. They can't undo this because it looks exactly like a legitimate payment to a creditor, and if you could always undo those it opens a real Pandora's box. Normally you can "undo" paying a willing creditor because they'll just lend you the money again. But of course not everybody is a willing creditor, as in this case. Boo hoo for Citibank.
Right, but the desire to undo is incompatible with the desire to have fast transfers. Yet another solution to this problem would be to have transfers take 24 hours during which they can be "un" sent by the sender, but I imagine banks will have a very strong preference towards instant or very very fast transfers.
I saw a great Linux quote recently. I'm paraphrasing, but it goes like:
"Linux gives you enough rope to shoot yourself in the foot. If you didn't think you could shoot yourself with a rope, you should read the man page first."
That said, as Linux has gotten more popular, it's put more safeguards in. For instance, you can no longer "rm -rf / file.txt".
I know someone that works for a (smaller) bank regularly processing loans in the mid six figure range. When I think about all the steps she has to go through around verification on each loan, It's just crazy to me how they're able to accidentally send an amount over 1000x times what she has to deal with.
Not saying the Linux analogy isn't good, but this is more like if entering the flags for `tar` in the wrong order could send you and your immediate family into irreversible ruin forever.
It simply should not be possible, this is about literal tons of money, they shouldn't rely on something this banal.
Simple safeguards can be put in place to prevent this. They don't even need to be very fancy:
Fairly sure a simple trained-human-readable description of what was about to happen (think: this money will go to this acct, this money here and this other money over there) would have saved this people a whole lot of grief.
>ALL THREE PEOPLE INVOLVED IN THIS TASK WERE CONFUSED
I know you didn't mean it this literally but the issue is worse than that. They weren't confused. If they were confused, they would have sought clarification. Instead, they were sure they were doing the right thing.
I did a quick google search and looked at the Oracle documentation for Flexcube. While I'm not 100% sure it is for the same version of software, the documentation appears to discuss this point.
Overwrite default settlement instruction
Select the ‘Overwrite default settlement instruction’ check box to confirm
that the liquidation entries should be posted into the Internal GL account.
System posts the entries to the Internal GLs only if you check this check box.
Otherwise, system posts the entries as per the settlement instructions specified
for the component of the contract.
It seems to me that the docs say when the user doesn't check the overwrite default settlement instruction checkboxes, the software will xfer the money according to the settlement instructions. Either way, for approving almost $900,000,000 I'd take a second look before clicking the approval button on a subcontractor's work. Heck, I'd even check when approving this transaction for the VP down the hall.
I'd think they could, but that would bring up the question of how many transactions Citibank had properly run through the software.
I doubt Citi would want to pursue that sort of a suit, since it would lkely highlight they were at fault. Sure the software is probably difficult to use, but if they already knew how to use it, then using it incorrectly is not the fault of the UI.
> Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta. All three believed that setting the "principal" field to an internal wash account number would prevent payment of the principal. As he approved the transaction, Fratta wrote: "looks good, please proceed. Principal is going to wash."
Somewhere in there, one of the SREs said something that's stuck with me. Roughly, he said that the concept of human error is useless. Every time a human error escalates into a real problem, the deeper problem is a design flaw. At scale, human errors are a certainty, and, while documentation and training is far from useless, no amount of it can change that fact. It's therefore a critical responsibility of systems designers to design their systems to be resilient to mistakes.
This is also the approach taking in aviation safety, and similar safety-critical systems. The pilot is treated as part of the system. Just like any other component, it's a part that can fail, and its error states can be modeled. Performance and reliability can be affected by training and certification, but just like anything else you're shooting for a threshold of tolerance for errors, not a complete removal of errors.
Your comment reminds me of a hidden brain episode that focused on the use of checklists in aviation and surgery. Now I often think of checklists from the perspective of various stakeholders at various levels when designing systems. This reframe has given me a more systemic view of parts that I'd once considered discrete. I'm much more careful now in applying the concepts of "blame" and "personal responsibility".
It’s also the approach the large e-retailer I worked for took. If a human made an error, it’s not the human’s fault. Find the process and design failures that led up to it.
This concept is elemental to any competent manager. Which is funny, because I've had very, very few managers who've ever understood it.
Oddly enough, the ones who utilized it most were in the military, using the "Swiss Cheese" method of failure analysis. In order for a failure to actually occur, there had to be a complete path all the way through the "cheese", where the problem wasn't caught by any "slice". So a failure at the lowest level was also a failure at each level all the way to the top, and each level was required to figure out why it wasn't caught at their level. Very effective system.
For all its failings, the military has tons of good management practices. After all, leading people is their core business and they do it in some pretty severe circumstances. They also have extremely good incentives to get it right.
Surprise, surprise, the military has been around a few years/centuries. Some of that experience has stuck...
I remember an ex-colleague saying that as an RAF person, he knew he could go on to a totally new airfield and ask for "XYZ Role" officer - and there would be an answer and direction to appropriate person.
Military knows to design also for "deceased personnel" (redundancies etc)...
My mother (civilian MOD employee working at an "airbase" which was in fact just a lot of office buildings, no airfield, although it did have machine guns and a nuclear bunker unlike most office parks) used to have an email address which basically said what her job was [I don't remember the exact address, something involving REG twice because she worked for the REGistry of the RAF's REGiment], and yup, the day after she retired that address still worked it just dropped in somebody else's lap.
So many IT outfits have problems that come back to "Oh, that was going to Bob and Bob left" and it infuriates me every time I see it, because we've known for decades how to avoid this problem.
What’s odd is that the MOD does that and the rest of the civil service carries on using name-based emails, despite people moving round every year or two.
There is a huge problem in the civil service of information being local to teams. Important information sots in inboxes or a shared drive if you’re lucky.
Finding out who to contact in another department on a given topic can take days and usually requires asking your private office to check (because they at least have the other department’s private office contact details). Otherwise people just look for emails in published documents and email out in the hope that at least one of them still works in the right department.
This could all be so easily solved with permanent, shared mailboxes per team. I suspect the reluctance is in part due to how much easier it would make FOIs.
Well then you need at least two email addresses per person. One role-based, one personal - promotions happen, you may want to keep being able to contact the human, not the role.
Yes, you do. But for the military this does not really matter, one of their defining characteristics is that they care MUCH more about "getting the job done" than about cost. This leads both to massive inefficiencies and to their extreme resilience to "bus factor" type events. It's a decent tradeoff for the military, it probably wouldn't be for commercial entities.
In Sweden we used to have a conscription army. It was said that they got really good at leadership and training since they could afford to experiment. If they screwed up one year they could just scrap the whole batch since they anyway got a new cohort every year.
The Israeli claim a lot of their startup culture comes from the high degree of responsibility their (very young) conscripts get in during their mandatory military service. I can imagine that someone who's been an 18-year old tank gunner in actual tank-to-tank combat has a different outlook on acceptable levels of risk than most people.
> For all its failings, the military has tons of good management practices
Military needs consistency, predictability, and uniformity. It needs a lot of good management practices to achieve that with people and parts that are not consistent, predictable, or uniform.
The military has to account for leadership moving up or out, a constant influx of new people, and the operational challenges of what they do, and a different perspective on money. Military dysfunction is different than private sector
Companies tend to focus on chiseling pennies and hope for the best.
In critical systems I agree with this philosophy. However, there are trade-offs to building process and checklists and peer reviews to every step in a human system.
Yet when this same retailer takes down half the internet with a bad config change to a large storage system you are likely asking why they didn't have better processes to prevent this dangerous action
Great example: Cirrus has a new personal jet, the Vision Jet. It has a feature called Safe Return Emergency Land [1]. If the pilot is incapacitated, a passenger can press the Big Red Button to activate the system, at which point the aircraft switches to autonomous mode to select the closest airport to navigate to and auto land at. Even the pilot can fail and the system will attempt to recover from this failure mode.
Pretty much all new commercial jets have an autoland mode. If something happens to the pilots get into the cockpit (this is the hard part!) fast put on a headset, find the push to talk button and yell help. You need to be fast enough that the radio frequency to use hasn't changed (this is almost as hard as getting in!) as the plane moved on, but if you get a response whoever it is will get in contact with control for you and from there on they will tell you what buttons to push.
Of course if you are in an old airplane (private plane, or legally retired but still in use in some poor country) all bets are off.
"Autoland" in an airliner is nothing like this. It performs the actual landing (and in some cases part of roll out) only. The airliner's designers assume that since their planes are designed to be used only with at least two fully trained pilots there is no reason to worry about scenarios in which a non-pilot is now in charge. Those simply don't happen.
Is it possible that they could talk a civilian down in an otherwise perfectly functional jet airliner given the pilots both are totally unavailable? Yes, but it'd be really hard and I would not give them great odds for it working.
Plenty of random non-pilots won't even successfully work an airliner's radio, if you end up talking to yourself, or the passenger cabin, that doesn't help you.
In contrast what Safe Return Emergency does is really one button "Oops, the pilot is unconscious/ dead, fix it". It has the same reassuring "Don't touch the controls" you get from Waymo cars. The SRE system is flying the plane, you can best help by staying calm and maybe trying to fix the pilot if that possible. It will figure out where there's a viable landing strip it can reach, broadcast its intentions to nearby humans, fly to its chosen strip, set up, perform the landing, roll out, stop and shut down the plane.
I agree that it isn't easy. However in fair weather gentel adjustments to the controls (preferably the auto pilot ) will get you close enough. Control towers will clear your path so you having a fighting chance of walking away. Mostly the training is for small planes where there is one pilot.
Pushing a button is or course a lot easier. I don't want to diminishing that. However most of us can't afford the type of plane ticket where it is needed, first class on a commercial flight is an order of magnitude cheaper. Thus the odds of both pilots dead is much higher, and so knowing there is still hope may be useful.
The revolution in this product is that it's available to general aviation users and the UX is "push button to land". It'll even squawk 7700 and announce intentions on 121.5 [1].
The first think that occurred to me is that the app could list the transactions this would cause and present that list for confirmation, along with a total and how much of the whole it was.
It would make the error much easier to spot.
OTOH, the people approving may not have the right to see values, just the process part they are involved in.
This and the aerospace mention reminded me of John Glenn: "two million parts, all built by the lowest bidder".
And the approach to natural disasters. There's no way to avoid them, we just need to be prepared to minimize the damage and recover as gracefully as possible.
That's a fundamental thesis of systems safety work. "Human error" is a convenient, but fairly useless, designation for the cause behind something. Why was the human able to perform the erroneous act?
If the person went out of their way to perform the act, for instance they manually turned off all alarms because they wanted to take a nap, then sure, it's (largely) human error. But then you can still ask, why were they even able to turn off critical alarms? Why was there a single person involved in this apparently crucial position? What allowed us to hire someone like Homer Simpson into the nuclear power plant in the first place? And do we really want to rely on eeny-meeny-miny-moe to solve our problems? Maybe training needs to be improved, including evaluation of training results through simulations.
I think the only cases of major system failures that could be blamed on human error, are ones where people INTENTIONALLY did wrong stuff, sadly, these cases do exist and were very lethal.
1. First nuclear accident: the theory is one operator was upset that another operator was screwing his wife, and removed the control rods 100% when he was scheduled to work on them, causing a sudden spike in temperature and then a water hammer that killed everyone in the building. (and made it JUMP!!!)
2. Chernobyl, although many people (specially anti-nuclear) blame the accident on the design, it was obvious deliberate error, even if the intentions weren't malicious, the operators ignored alarms, then DISABLED the alarms, and intentionally pushed the reactor past safety limits, even if it didn't had the design flaw it had, it was possible they would still find a way to blow it up if they kept their behaviour.
3. Depressed airline pilot, locked co-pilot out of the cockpit and crashed on the ground on purpose.
There are probably other situations out there... thing is, those are situations no matter how well the system is designed, it will still happen, it is impossible to make a "malicious-proof" system, after all, even a rock, with zero moving parts, can be used in the "wrong way" and kill someone.
There's a good argument that these are still system failures. If a single individual can intentionally blow up a nuclear reactor, that's a massive point of failure. Humans are often irrational and prone to anger, depression, malice, and greed. Any of those impulses could be motivation for destructive action. Nuclear submarines avoid this by requiring multiple personnel to simultaneously coordinate in activating a launch.
On Chernobyl, my take away from the excellent Netflix mini-series was that there was a massive cultural failure, where everyone abdicated responsibility, and there no ability for employees to counter poor decisions from superiors. Culture and organizational design are also important components of complex systems
I don't have any insight into why the Chernobyl's operators disabled the alarms, but there is a very common effect known as "alarm fatigue" [1] whereby poorly-tuned alarm thresholds are ignored or actively sabotaged by operators so that they can perform their regular work. For legal & CYA reasons system designers often set alarm thresholds so low that false alarms are extremely common. When operators naturally start to tune out the constant background alarms then the designer can either fix the issue by making better alarm thresholds or they can make the alarms more insistent which sets off an arms race for operator attention.
If you're interested in specific stories I'd recommend "Set Phases on Stun" by Steven Casey. (It shows up in the syllabus of a number of Human-Computer Interaction classes.)
But as a general heuristic: If one operator has trouble dealing with the number of false alarms you can just hire a new operator, but if most of the operators find it problematic then the problem clearly isn't with the operators.
The notable example of alarm fatigue is the designated pillow on board of the Tu-144 (the Russian copy of the Concorde) to stuff into the alarm horn. The wikipedia page is itself a “bestof” bad engineering/operating practices, which are worth a good laugh now that it is not flying anymore. See chapter “Reasons for cancellation” - https://en.wikipedia.org/wiki/Tupolev_Tu-144
It blows my mind that this is still a problem; Western Electric was already making allowances for it in what I believe to be the 1930s:
Most telephone hardware has some form of self-checking and fault detection. Alarms are classified into "minor" and "major", sometimes also "power" or "critical". Each piece of equipment has its own alarm indicator lamps, and contacts for external alarms as well. Typically the external contacts are wired to both an "aisle light" which aggregates all the equipment in that lineup, and a floor-wide or section-wide "alarm sounder". When the sounder goes off, you look down the aisle to see which lineup has the trouble, then walk down the lineup to find the individual unit and address the problem.
But first, you push the ACO button. The alarm cutoff button silences the sounder, which won't come back on unless something new happens. ACO also tells the other workers that someone is on the case.
Crucially, it preserves attention. If the sounder was going the whole time someone was working on a repair, nothing would ever get done, because the workers would be down at the corner bar trying to get some sanity back.
In my experience it's sometimes a result of mismatched incentives resulting in a prisoner's dilemma rather than a lack of awareness or engineering ability.
The designer responsible for an alarm wants to avoid being blamed for the operator missing that specific alarm so they're incentivized to make their alarm as prominent as they can get away with. Of course once most designers are doing that then they have to keep doing it or risk get drowned out.
One way to fix that is by having someone who is responsible for the entire experience who can dictate shared prioritization, requirements, alerting frameworks, rate limiters, etc. In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard which is why there's a cacophony of noise even though everyone involved acknowledges that it's a problem.
> In cases where the operator uses multiple products from different companies (e.g. in a hospital) that's especially hard
Somehow this works just fine in telecom. I was handling equipment from DSC, Alcatel, Fujitsu, Marconi, Rockwell, Cerent, Nortel, Pirelli, CarrierAccess, Tellabs, Cisco, ADC, Lucent, and more. There are Telcordia and NEBS standards for alarm severity and wiring, and everything just works when you hook it up. Does the medical industry not have standards?
I've heard from colleagues in the (US) medical industry that there are guidelines from the likes of AAMI and ECRI but few real "requirements". My second-hand understanding is that it's largely dependent on how much leverage the end-users (doctors and nurses) have to dictate system requirements and how much leverage the hospital system has with respect to negotiating with suppliers.
Alarm fatigue might have played a minor role in Chernobyl but the main cause was authoritarian management trying to meet a scheduled deadline to run a test that the reactor was incapable of running safely in its current operational state. They purposely ignored the alarms in order to run the test. There was also a design flaw nobody knew about
(positive void coefficient) that caused the scram make things worse, but they'd never have needed to scram if the deputy chief engineer on duty had cared more about safety than meeting his milestone.
I rediscovered this effect myself and it’s why I harp on people for flaky tests. Eventually you discover the build has been broken for hours, everyone assumes they’re just the same old errors, and pushes rebuild without even looking at the error messages.
Everyone does it. I’ve caught myself doing it. So I move to the previous domino and work on that.
Same goes for log output - if you get thousands of lines every day, no-one ever looks at the logs and things get missed. If you only get log messages when things are really broken, people actually notice.
I mean, this is what log levels are for. My personal standard is log level error is for things raising an alert over that you need a human to look at, warning is for stuff that can raise an alert if it goes over a threshold, info is only for looking at when a problem has already been identified, so that you can try understand the lead up to the problem, and debug is incredibly spammy stuff only turned on when the above gives insufficient details to recreate the problem locally.
Stats are good for things that may be a problem or may not. It’s hard to get a sense of rate with log analysis. You can, but comparing several stats to each other at once is painful.
My car dashboards (and most cars as far as I know) are actually a good example of avoiding alarm fatigue.
Red lights mean that the sensors have detected a state that may damage the car or put you in danger. You should stop immediately. Examples: low oil pressure, overheat, emergency brake still on, low brake fluid. You can safely tell a new driver that if they are driving and they see a red light on the dash it means stop right now.
Yellow means that something is operating outside of nominal ranges and you should try and fix it soon. Examples: low tire pressure, low washer fluid, check engine light.
One thing it discusses is how the alerts inside the control hub were so noisy, and were logged via a reasonably slow printer, that the alerts which suggested the reactor was in serious trouble only printed hours after you could do anything.
And the alert light that said "shits fucked" was placed right next to the light about an elevator being out of order.
Human error is rarely intentional, and even if it is, a system that fails in response has missed the mark. The talk explains the amazing finds made by the government committee hired to investigate the incident, once they left the idea of 'human error' at the door.
I do suggest you watch the video, but as a teaser: it turned out that at the time, most US nuclear engineers had been sourced from nuclear subs. Subs have totally different rules on how to manage the reactor, and train their staff inappropriately for a land-based reactor. Pretty significant cultural find...
> the theory is one operator was upset that another operator was screwing his wife
That's _one_ theory and it's not even the most likely one, it's just the only one they like to trot out whenever a History Channel producer needs to fill time between ads.
The control rods in SL-1 were prone to getting stuck in their channels for a variety of reasons. They were apparently not light either, as part of the investigation for the accident, they built a mockup control rod assembly that was weighted at 84 pounds. For military-age adult male, this is right in the range being able to move, without a lot of control over how _much_ it is moved.
It is, however, clearly a design failure that the removal of a single rod allowed the reactor to go critical, and/or that the rod was able to be removed by a human to the extent it was.
For 1, it should be noted that that is only one theory. Another less lurid theory is that while attempting to pull the rod a small distance out as part of a routine maintenance procedure, the rod stuck. The operator gave it a yank to unstick it and accidentially withdrew it the whole way.
I took a design course in grad school and one of the things they emphasized was that it's impossible for everyone to be wrong about something. If everyone is using something the wrong way, then its the design that's wrong, not the people.
I’m looking at the front door of my apartment as I type this, and I can see sixteen states for it quite easily. It’s got a lock on the doorknob, a deadbolt, and a big window with a blind. All of these can be open or closed independently of each other; add on the basic “is it open or closed” and that’s four binary choices for 16 states.
Also there is that fun state of “open but will lock behind you because the doorknob lock does not prevent you from closing the door, hope you have your keys on you or someone inside” though that’s maybe more a special case in the state transition table than an actual state?
Another fun state: open the door, then turn the deadbolt. Now the door can't be closed! Attempt to close it hard enough and you can bend the deadbolt shaft. Now the deadbolt can't be retracted, and the door is entirely disabled! An arcane edge case, but something a clumsy child could do in less than a minute.
How could we tolerate such an awful design? Who is responsible for this catastrophe?
My personal philosophy for design has always been "if one person used it wrong, maybe they're an idiot. If two people used it wrong, you're definitely the idiot."
Since properly learning a programming language is such a huge time investment for most people, once they do it the language becomes part of their identity. Many people will, in professional setting, introduce themselves as "Java/GoLang/Python Developer". When such developer hears a critique of their language they hear personal insult.
This is also not a new idea that even comes from the software world. I'm not entirely sure of the original of such ideas, but the first time I heard about was in conjunction with airplanes and their UX. Basically, crashes and other issues from flying is never attributed to the operator, but the UX of the controls as if it was done better, would have prevented the operator of committing the error in the first place.
Indeed. One of the really core things you learn when you dig deeper into aviation is that essentially every major disaster is a chained combination of multiple extremely unlikely failures in sequence. Which is a testament to both the incredible success this approach has had, as well as the impossibility of perfection.
I once saw a detailed breakdown of a well-known Concorde crash
and it's staggering how many coincidences came together to mess
up those people's day. Iirc a spare part for another plane was
faulty, fell off onto the runway, the Concorde's wheels hit it
and exploded due to the strain, one of the rubber chunks hit the
fuel tanks causing a leak and another shredded some live wires,
leading to ignition of the leaking fuel.
It wasn't the first time the Concorde had exploded tires. There already was several "close calls". The last one was for real, with the power of hindsight, they should have addressed the problem long before the deadly crash.
For me, the best illustration is the Tenerife airport disaster, the deadliest accident in the history of aviation. Two Boeing 747 crashing into each other on the runway during takeoff.
Circumstances included fog, problems in the control tower, nonstandard phraseology, radio interference and overworked personnel.
Where do you draw the line between blaming the UX and blaming the training process, or the inevitable cases where an irreversible decision made under uncertainty turned out to be the wrong one? Some level of training is certainly needed, so in my naive view anyway, it makes no sense to attribute every error to UX.
For example, let's say a pilot belly-lands because they failed to lower their landing gear, possibly because they were distracted by some other equipment failure, or possibly because sometimes qualified and experienced pilots do this. This error could have been avoided with, say, an automated spoken "gear up" warning instead of a nonverbal warning horn (I have no idea if some aircraft already have this). But it could also have been avoided by longer and more extensive training, right?
> But it could also have been avoided by longer and more extensive training, right?
Sure, it maybe could, but why risk it? Humans are also humans, and even pilots with endless amount of flight time does mistakes sometimes, because we're all humans. The idea is to work with the idea that sometimes we fail, even if we are well trained, so when failures happen, it's easy to recover.
> Where do you draw the line between blaming the UX and blaming the training process, or the inevitable cases where an irreversible decision made under uncertainty turned out to be the wrong one?
Basically, it's always the UX. If there is a case where the pilot made the wrong decision, something needs to change so if a pilot with similar experience, wouldn't have made that decision with a different UX. Basically force the pilots to do the right decision, always.
I'm not 100% on the history of all of this, so if someone is more familiar with it, please correct me. But I think it started with "Crew resource management" to ensure proper training of the crew. It has gone through many iterations where less and less blame is being made on the crew and more on the equipment. I think the point of the current iteration basically boils down to "Human-error will happen, let's plan for it and if human-error lead to a plane going down, let's fix it so the same error would be prevented".
>Basically, it's always the UX. If there is a case where the pilot made the wrong decision, something needs to change so if a pilot with similar experience, wouldn't have made that decision with a different UX. Basically force the pilots to do the right decision, always.
That's not realistic. If you think it is realistic, please take a look at the investigation report for PK8303. Some pilots will disregard alerts, override safeties and ignore procedures: there's only so much a system can do to promote good decision-making.
> If there is a case where the pilot made the wrong decision, something needs to change so if a pilot with similar experience, wouldn't have made that decision with a different UX. Basically force the pilots to do the right decision, always.
If it were possible to do this, we wouldn't have pilots at all.
> But it could also have been avoided by longer and more extensive training, right?
Generally in this sort of risk analysis there is a hierarchy of mitigation, and the least of them is training. e.g. when you find a potentially hazardous situation, the best thing you can do is design the system so that it can't happen. Failing that, you design in an procedural or workflow aspect that naturally avoids it. Failing that, you put in a system to force the use to confirm or review the action. Failing that, you train around it.
A common sign of a flawed design process is when things get pushed into the "train around it" bucket late in the process, not because they were unavoidable, but because they were looked at too late in the design to effectively make changes.
> Failing that, you design in an procedural or workflow aspect that naturally avoids it. Failing that, you put in a system to force the use to confirm or review the action. Failing that, you train around it.
I view the procedural or workflow aspect as a subset of training. As an example, I always lower the gear and leave my hand on the gear selector until I see three green (gear safe) lights prior to selecting any flaps.
The operating handbook and factory checklists do not require this, but my planning and training has elected this optional workflow/procedure. Why? Because it's incredibly hard to get my airplane to slow down and come down for landing while completely clean (no flaps, no gear). If I use the factory accepted procedure to select approach flaps, I can get it slow enough to land. I've literally never seen a "pilot forgot the gear" airplane sitting on the runway with no flaps deployed.
I wasn't very articulate there, but by a procedural or workflow aspect I mean something like: the workflow will lead you through a series of actions that will make it physically impossible or unlikely to [do the dangerous thing]. This is different than [system can't do the dangerous thing]. It's also different, and safer, than relying on user actions outside your control.
Example: I have a mechanical system moving fast with enough mass to hurt the operator, so I put the control on the other side of the machine (i.e. machine body between person and the dangerous part), and require the operator to hold a switch "on" for the entire time the machine is moving.
This doesn't make it impossible for the machine to hurt the operator, of course, but it makes it a lot less likely - and it doesn't rely on training at all.
While you can also rely on training aspects like you describe (a good idea!) it's not the same thing.
I don't know aircraft, so I don't know if there is something equivalent but I wouldn't be surprised. Anything you simply have to do with two hands on disparate controls?
My “gear before flaps” is an example of just such a workflow. If I always do that, it’s extremely hard to get the airplane anywhere near the runway without realizing something’s wrong. (Both the gear and the flaps add considerable drag, helping the airplane go down and/or slow down.)
The only thing that I can think of that directly work this way are some of the gust locks (to prevent control surfaces from banging about in the wind while parked) are required to be designed so the airplane can’t be started and taxied out with them in place. (That’s arguably a poka-yoke as much as a procedure, but probably still fits.) Mine simultaneously locks the yoke and has a plastic flag that blocks the starter switch when installed.
If the airplane offered a mechanism to inhibit flap operation unless the gear was down and locked (a poka-yoke enforcement of my procedure), I’d decline it/argue against it. In the event of a gear failure or a forced water landing, I want to have the flaps without the gear being safe and, especially for water, I don’t want to have to look up the abnormal/emergency flap extension without gear in a QRH while doing everything else needed while descending for a forced landing.
There are configuration warning systems (most are aural tones, though there are available spoken systems). There are two challenges with a gear warning system: When to trigger it. How to present it.
In my aircraft, it's triggered by ((no gear safe indication) and ((full flaps) or (very low throttle setting))). The presentation is a loud, alternating warning tone and a red "Gear Unsafe" light directly in my field of view on the glareshield. These are not silenceable by button as is common on training aircraft.
Even though a handful are inadvertently geared up every year, I'm not convinced that better warning systems would significantly reduce the rate. Better training would. If I ever do the deed, I'll be the first to admit that I was the primary cause and that I failed not only to follow the checklist, but my own SOPs and flew the aircraft onto the runway with a large red light right in front of me and a blaring horn. Making it a voice seems not likely to help.
If I gear it up after an electrical failure [which some of the inadvertent ones are], that's probably slightly more understandable (and a scenario in which an additional, electrically powered, warning system would presumably also be INOP).
> At scale, human errors are a certainty, while documentation and training is far from useless, no amount of it can change that fact. It's therefore a critical responsibility of systems designers to design their systems to be resilient to mistakes.
If only the people who insist on using unsafe languages understood this. Their position is that you should get better at not making mistakes.
I work with C++ on a daily basis and the position is actually closer to "sandbox anything tricky, and run everything through fuzzers and sanitizers."
Anyone who thinks they can overcome 100% of memory-unsafety bugs through looking at the code real hard is fooling themselves. But I'm not convinced there's a lot of people who actually think that (maybe there's some people who just don't care or never considered it)
It's written by someone that does airliner crash investigations. His central point is that "human error" as a term functions to redirect blame away from the people who establish systems and procedures. It blames the last domino vs the people who stacked them.
It's a quick breezy read, and you'll get the main points within the first 30 min or so of reading. I've found it useful for getting these ideas across to people though, especially more generic business types where "no blame post mortem" strikes them as some care bear nonsense rather than being an absolutely essential tool to reduce future incidents.
This is also a big concept in the medical world. There's an influential report on this from the 90s called "To Err is Human." https://pubmed.ncbi.nlm.nih.gov/25077248/
I forget which book exactly, but one of Scott Adams's has a story from a reader about a process consultant who comes from a background in aviation into some typical Fortune 500 company and tells them the trick to improvement is just to never make a mistake, ever.
And then proceeds to use reductio ad absurdum to demonstrate that if you can be perfect for 5 seconds, you can string those together to be perfect for 5 minutes ... 5 hours ... 5 days .. your whole life!
And when someone in the crowd pushed back on this idea, the consultant said something like, "If the airlines took that same approach to flying, we'd have 1,000 accidents a year!"
And then maybe the reader looked it up later and it turned out there were about 1,000 flying accidents a year.
Given enough eggs, a chef will eventually throw the eggs in the trash and put the shells in the pan. Given enough orders, someone will do that every single day.
That sounds like you're trying to propose a solution that's as far as possible from addressing the root cause and any "human error". NASA didn't need a shuttle that would refuse to take off in cold weather. They needed engineering assessments that realistically quantified and clearly communicated the risks of launching in cold weather. Given accurate information about the capabilities of the shuttle, mission control would have waited for warmer weather or implemented countermeasures.
The Challenger explosion was entirely political. It was the result of the structure of human relationships and not any engineering discipline.
Had the social structure been politically different on that day they would have acted on the available engineering information by refusing to launch. It was the political environment that goaded them into taking more risk than they had previously decided was acceptable.
Then I think you're still missing the point of the comment you replied to. Chalking something up to "human error" is not a particularly useful conclusion to come to; it doesn't really solve or prevent any problems. The takeaway should be that a system did indeed fail from a design flaw. That system was not the space shuttle as a mechanical device. The system that failed and needed to be re-designed was the process of spacecraft engineering: NASA was lacking in structural incentives and protections necessary to ensure that predictable catastrophes would actually be predicted, and that such predictions would be communicated through the organization before anyone got killed.
That certainly seems like a reasonable thing to me, now that we know that to be an issue. Why wouldn't you design a system that prevents it from taking off in conditions that are known to cause a crash?
Obviously they may not know these conditions ahead of time. But preventing user error for known problems seems obvious.
Thing is, even with a UI as shitty as this, you would still have caught this if before actually committing the transaction, the software had a "before" and "after" view to show the effect of the change. Raj caught the error during a routine inspection the next morning, so the same view he used for that, presented before the change had been committed, would have allowed him to avoid the error.
I've worked with Citi as a partner, as a client and as a service provider in various other banks and fintech companies, and I'm so not surprised. In fact none of my colleagues are.
The problem is Citi itself. It's always an absurd amount of indian subcontractors who aren't paid to care but just to look good on a cost cutting plan (so they do their job), a disorganized / panicked organization who seem to never know what's happening when and follow mega-processes rather than solve problems.
My bank has those, but we can take shortcut if we must - not Citi, can make you wait 2 months for a csv file via sftp, and then pretend to be ready when you finally discover it's a frigging indian contractor writing the csv rather than an automated process... we were like "guys seriously why are you even lying about it" when it all came crashing down with crazy errors never seen in the UAT. Turns out the automation guys were late and they made an effing human do the prod version meanwhile ...
In another company, working with Citi on a completely different sector/group/vertical, I spent nights with their tech guys in India figuring out their own system and why they couldn't work... and they had less clue than me, it's absurd.
They seem to work in very split team where americans will do the high level design and then outsource to dev + support + deployment teams, ofc isolated from each other, in India. So nobody US cares about any actual implementation detail of anything, the 3 indian teams don't know each other, and nobody feels responsible of anything. It's absurd I've had multiple separate experience of the same thing, with me and my colleagues doing the connection layer between various Citi stakeholder who had no idea about each other.
There was even a time where a Citi country team in Hong Kong begged us to introduce them to the Citi country team in Singapore, and show them their product and processes...
Your experience with Citi is very similar to mine. It is funny that as an organization Citi seems to believe that its technology is brilliant and that any software that it uses must be built by Citi. Tremendous hubris combined with total incompetence. Not a winning combination.
Well I ll be honest we all do. My own bank, another giga multinational, also has this hubris, fine. But at Citi they went so far in the outsourcing nobody knows anything anymore. The people you talk to to organize stuff are timezones away from the guys doing stuff. And me I m close to the Indian time in HK, work with our own outsourcing in India who are fine for what they re here for etc and we have pains. But never to the extent of Citi where there s like internal conflict or total lack of communication that makes the contractors lie more than usual to the managers... I dont know I dont get why each time we have to deal with Citi it's the same story. It s organisational though, not tech (their tech I ve dealt with is okay)
At work we have several modules which each have their "output control". When the user hits "submit", we run several checks and outputs a list of potential issues.
We've got four levels: info, warning, severe warning and error. Warnings are highlighted, severe warnings must be explicitly accepted (and we log this) and errors prevent submission.
When clicking on an item in the list, the cursor moves to the relevant control, so they don't have to hunt around.
We add the checks we can think of, and keep on adding checks based on user mistakes and feedback.
I'm not familiar with the financial world, so I don't know how common it would be to do what they wanted to do, but at the very least it sounds like the principal account number being filled out but not the two others is something we'd made an info item from, to highlight what would happen. If what they ended up doing is unusual, it would be a warning and likely severe warning due to the amount.
That would, by itself, make the UI less shitty. Also, if you're the sort of person who thinks in those terms, you're also likely the sort of person who'd have come up with a better design overall anyway.
This sounds like one of those situations where devs tasked with programming something that they have little to no understanding of what they are programming does. Yes, the dev programmed button A to perform action B, but doesn't truly understand how/why/what/when/where action B. It's just a check box of the requirements that has now been completed.
However, a dev that does understand action B would be able to anticipate various forms of input from button A and how that could cause undesired results from action B.
This isn't a knock on the dev that is less knowledgable about the full thing. It is a knock on the people upstream that should be more aware of issues. Also, the people most knowledgeable about action B tend to have little to no understanding of programming. While they might understand that providing bogus data to action B can have bad results, they are not aware enough to tell the devs that sometimes valid looking data is just not sane.
I wouldn’t even assume lack of understanding as much as being locked into a work-to-order situation where they’re not rewarded, possibly punished, for doing anything which isn’t on the assignment. This is super common in outsourced hellscapes where the MBAs like to LARP as architects and UI designers and there’s no effective feedback loop from the actual users.
I assume this system is designed to handle manually adjusted payments for syndicated lending. So you've got one borrower with potentially multiple credit facilities, with each credit facility will be funded by multiple lenders, each with potentially multiple accounts. The events you are dealing with are fees, underpayment/overpayment of interest or principal, late or early payments of principal, transfers of balances from one lender to another, funding transactions and plenty I haven't thought of or considered but are represented by check boxes on that UI.
It's complicated. The reason the system exists is because it's complicated, and prone to human error, and the before and after might generate dozens of events in dozens of accounts.
What seems weird is that accounting software typically adds these transactions immediately with a future date, so in this case whenever the batch was executed. You don't need to wait to see the impact, just for it to take effect. This UI has to work extra hard to obscure this visibility.
I don't know how it's in this particular system, but I have also seen banking core software which would add these transactions during end of day batch procesing, by which time the original operators wouldn't be looking at them anymore, and anything that should be settled 'today' would also get executed during that batch process.
I had exactly the same thought. Whenever I write a tool that manages a complex process, I try to add a dry run mode, and more often than not, I use it.
Unrelated, but is it a good idea make the names of those employees public? Wondering how this will end up affecting their lives, when in fact it was caused by a UI issue and lack of proper training on how to use it.
It speaks even worse of that interface. It means that the design was not only bad enough to confuse a single individual but also the two others meant to act as safety mechanism.
It looks like they were all under-qualified. I wonder how much the decision to hire them all will end up costing anyone other than them, and the people who hired them.
Maybe I misunderstood, but all of them either "thought" that was the right thing to do or signed off on it. Even if the software should have warned them, shouldn't they have known the correct transactions to make?
That's issue here with this specific software's UX.
They all thought they were making the right transaction. There was no warning that they were not.
It would be like if I hit the reply button here on Hacker News and instead of the comment I typed out it my reply contained my bank account information. And I was not told something I do not want to happen would happen by doing so.
I think I understand now: the software was doing the wrong thing. They thought they were making one transaction, but they made a different one. One wonders how long that had been going on for, and how it could have been going on for so long.
Thanks for the explanation. It saddens me how brutal people here are in response to a misunderstanding, judging by the downvotes, but oh well.
It saddens you? You made a scathing misjudgement about the involved people without having read the article, or even the rest of the thread here. I don't think it's the downvoters that are "brutal" here.
They knew the correct transactions to make; the problem is the software was unbelievably counter-intuitive. At some point the user can't be blamed if the UX design is that bad.
These issues are everywhere if you bother to notice. My in-laws had a coffee maker with a built-in grinder. The grind function was default enabled, so if you went to brew some coffee from pre-ground beans and just turned it on, the grinder would kick on with an awful sound.
The solution they came up with was a "Grinder Off" button that lit up to indicate the grinder was turned off. It was...confusing to say the least. And as an infrequent user of this coffee maker it was all too easy to screw up. To add insult to injury, that off light turned off (i.e. enabled grinding) every time you brewed a new pot. You had to be sure to press it every time you brewed from pre-ground beans, which was always for us.
My digital kitchen scale turns off pretty quickly after use. This is terrible, because it also doesn't remember tare (i.e. the weight of the empty container you put onto the scale prior to filling it), so when you turn it back on it zeroes on whatever's on the scale. Did not read the measurement immediately, or forgot the value? You now have to empty the content into another container and try again. Annoying if you're measuring uncooked rice. Infuriating if it's sticky, ultra-viscous fat.
My dishwasher has a "COMPLETE" light that tells the household that a washing cycle finished after the door had last been closed, a signal that the contents of the dishwasher are likely clean (unless you leave the light on after emptying, but we rarely forget). But if you close the door accidentally just once after that, the light stays off, and there is absolutely no way to put it back on without a washing cycle. I read the manual, tried various logical and illogical key combinations, nothing. Now you have to tell your household "I didn't have time yet to put all the clean stuff out but the light is off, please remember that".
My TV likes to switch automatically to its internal speakers for various reasons, which I never want to use because they are terrible in this particular TV, and switching back requires going through multiple laggy submenus. But worse, if I turn the TV on and notice it's on internal speakers again, I have to wait for it to finish booting completely before I'm allowed to navigate the laggy submenus.
To leave this rant on a more positive note, though: The digital interfaces of my coffee maker, my microwave, and my water cooker (multiple temperatures, auto-reheat, auto-shutoff) seem to do exactly what they should, under many and sometimes non-trivial circumstances, so I'm glad good UX design still exists.
> But if you close the door accidentally just once after that, the light stays off, and there is absolutely no way to put it back on without a washing cycle.
To be honest, that setup would make more sense for me; I always brew from freshly ground whole beans. It sounds like they flat out bought the wrong coffee maker. They should have gotten one that doesn't come with an integrated grinder at all.
That coffee maker is well designed for its intended happy path use case, which is not the same as you can say for this Oracle software.
No doubt it was the wrong coffee maker for them. I don't think it was well designed though. I think it was "eh, ok" designed for the happy path, and very poor for the unhappy (but probably pretty normal) path.
I think what threw me is the "grinder off" button with a red light. The red light made sense in the context. But, if I were designing this, I would have a "grinder" button with a green light that was default "on", indicating that grinding was imminent. There was no green light on this part. It was red or off.
That would have been consistent with the rest of the interface, where a green light on the "brew" dial meant the timer was set and a red light meant the coffee had brewed.
Yeah, something like this would be much better off with a rocker switch with Grinder: ON, OFF. Then it saves state between runs and it's always clear if the grinder is on or off.
That does seem like the ideal solution, but sadly a mechanical rocker switch would cost some number of pennies more to implement, hence they haven't.
I have a hot water stand-by boiler in my kitchen a $15 outlet switch because the darn thing didn't come with a simple power button. Hardly anything these days does. All of which is to say, the ideal design often conflicts with the consumer's other expressed preference of "lowest price possible, it doesn't matter how many corners are cut and how quickly it dies".
I used to have this exact coffee maker and always thought about how bad the UI was for grinding. My proposal would be to have a "Brew" button and a "Grind & Brew" button.
On top of that you're also supposed to clean and dry the grinder parts after every use or it eventually gets clogged. I just switched to pre-ground coffee and a simpler coffee maker.
You look at the screenshot,then read the summary on what happened and then wonder how on earth the bank still has any money left on their accounts with systems like this.
Looking at the screen tells you nothing about the quality of this system. The screens look a bit dated because the system is 20 years old. It is also a back office system used by a small number of what should be trained professionals. Pretty html and javascript would not have helped. This was a failure of the whole system. The software and the organization.
This is not about ugly vs beautiful. Good UI is not primarily about being nice to look at, it is to be usable. The UI has to be really bad to be able to screw up this bad without even knowing it. Horrendously bad.
I didn't mean that it looks ugly- I take functionality over pretty design any time. The screenshot itself doesn't provide much info,but when combined with comment from the article,the who picture is pretty bleak tbh..
Well one could add, that other possible options are "COLLAT", "COMPINTSF", "DEFAUL" and "DFLFTC". Hard to believe that anyone could possibly be confused by that or miss the clearly necessary "FRONT" and "FUND".
Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta. All three believed that setting the "principal" field to an internal wash account number would prevent payment of the principal. As he approved the transaction, Fratta wrote: "looks good, please proceed. Principal is going to wash."
I've used Citizens Bank before for business reasons and their UI is probably one of the worst I have ever seen. I don't understand why consumers get such good stuff and enterprises get such crap.
It comes down to the difference between b2c and b2b business. In the case of b2c, you’re selling to consumers, who themselves use your UI and will leave if it’s bad.
In the case of b2b, it’s more complicated. You’re selling to office administrators, whose goal is to score the best deal for the company. Since they aren’t the primary users of the product, stuff like UX doesn’t matter as much as cost.
Best illustrated by Slack, which thought they could sell to businesses based on a beautiful UI, but their market was dominated by Microsoft Teams, which could sell better to enterprises.
Slack thought right; they did sell to businesses. Microsoft came later to the game, leveraging their size to both undercut them (with a feature rich free plan), and offer an integrated suite with the productivity tools that enterprises are already using (with Microsoft 365).
Which is probably one reason why Slack is looking to sell to Salesforce.
My point being I don't think Slack's positioning was "chat but prettier". The fact Teams is more dominant now seems more to do with bundling.
As a forcéd user of Teams, I have to say this, a thousand times over: it _sucks_, I -- and everyone I know -- _hates_ it and it seems to have been rushed out to crush the competition and integrate into Office online products. I also hate these and don't use them, but unfortunately some of my colleagues do.
But, you know what? Nobody cares about my opinion! You are exactly right: the people who make these decisions about what software the university "runs on" are at a far higher level, and have far more clout. Their incentives are totally misaligned with mine -- I use linux as a daily driver, and write highly mathematical software. I am exactly _not_ the target audience of these things, because, frankly, I can support myself: if something's broken, I fix it -- and don't consume IT's time. The people who _do_ however, are all of the admin staff, many of whom have professional backgrounds elsewhere, and think this is the way the world works. And they get promoted, and end up in managerial roles that make purchasing decisions at a high level. Hence, Teams.
Sure but until the UI is so bad that you end up accidentally sending out money you didn't intend to, it is important.
Separately, In the business UI I used previously with Citizens it was so easy to get locked out of your account (and I had no time for phone calls, which was the only way to unlock it after that) that so many payments were made late because I had no other choice; my schedule was already full and the payment would only be made when I could schedule time for that stupid phone call.
That led to delays and late fees of sorts, and that is money as well, all because of a bad UX.
At a previous very well-known employer, they had also overpaid Oracle 5+ orders of magnitude for the value of their software and had an entire floor of developers, analysts, and other support roles building what they needed into purportedly “off the shelf” software.
One day, a novice-level Java error in Oracle’s code caused a multi-day outage requiring data to be restored from backups and reloaded to get any functionality back and several months of that group deploying duct tape to get everything working.
For a consumer app, people would leave in droves as one star reviews poured in. In this case, they solved this by inviting the relevant VP to a football game in the corporate box with the account rep, both of the same fraternity, and starting that Monday nobody in management talked about the outage publicly again.
Nowhere in the article (which I read in its entirety) does it answer the above reply's question. "Do companies consider good UI a competitive advantage when looking for a commercial bank or lender?" I don't know. But that's not addressed in the article.
My guess is that it probably isn't a huge consideration. Very few people interact with these interfaces. Maybe one or two controllers at a company would ever log into this. Even CEOs and top management would never actually log into these accounts directly. They are going to get their information from internal reports and BI dashboards. So I bet no one really cares. If Citi offers a $400M loan at 1.1% interest and JP Morgan offers the same loan at 1.3% interest, my guess is that that company is going with Citi. I doubt UI/UX makes a difference here. Although in the consumer world it does have an impact which is why we have seen many banks improving their apps and websites lately to compete in the consumer banking space.
You hit on the head with the difference in interest. At that sort of scale you're looking at a difference of almost a million dollars in just a year, even if I were doing that for my own personal account I don't think I'd consider a better UI to be worth a million dollars a year. Anyone in a corporate setting suggesting that amount should be swallowed so that someone interacting with the account a few times a month can do it a bit easier would be laughed out of the room.
Yeah, the corporate measure for "good/bad UX" is pretty much "how many full time employees will we have to hire to deal with the worse UX or how many employees will we be able to save due to the better UX?". Convenience matters for productivity, but usually not that much - I have seen some UX'es so bad that you'd have extra dedicated employees just to deal with their horribleness, but even then it's cheaper just to hire one or two extra people than spend many millions and person-years on system migration.
I am in such a position and I've done exactly what you think i should do to alleviate the problem.
Ive gone so far as to open 4 accounts with different banks to do actual tests
The biggest issue i found is that no bank offers a "free trial" of its own to facilitate discovery.
I have filled at least a dozen firms twice becuase the bank wants "wet ink" signatures
I had to overlook those instances just so i could test the UI, but honestly, its an exercise in supreme patience and i am afraid my whole effort will be pointless. I suspect all 5 banks will be equally bad.
Did you read the context of my reply? I know the article is about bad internal UI/UX. But my understanding of what the person I am replying to said was that businesses have to put up with terrible UI/UX for their banking portals. Which I anecdotally know to be true, at least with Canadian banks.
The business banking portal is often much worse than the consumer one.
There's generally no relationship whatsoever between the customer-facing UI and the internal UI used by bank employees as in this article, those tend to be separate systems with minimal, 'arms-length' integration.
I've used credit unions that managed to have worse UIs than Citizens Bank. Can't say I've used Citibank though. Ally seems to have it figured out pretty well, which they kind of have to.
I work on a large financial transaction system. A dumb backbone for bank the same size as citi.
This UI is pretty good ! Haha.
No seriously today we had a one hour conversation on what happen when a user cancel a trade twice. I don’t think we’re gonna find a elegant solution within the contraints of the clusterfuck of sub-system we maintaining. Kinda disheartening.
Not defending this horrible UI, but this is a case of a non-standard operation and one should be extra careful when trying to override default behavior. Matt Levine has more details.[1]
If I am an approver of a $900M transfer using an edge case I will follow instructions:
"Citibank’s internal Fund Sighting Manual provides instructions for suppressing Flexcube’s default. When entering a payment, the employee is presented with a menu with several “boxes” that can be “checked” along with an associated field in which an account number can be input. The Fund Sighting Manual explains that, in order to suppress payment of a principal amount, “ALL of the below field[s] must be set to the wash account: FRONT[;] FUND[; and] PRINCIPAL” — meaning that the employee had to check all three of those boxes and input the wash account number into the relevant fields." [1]
Having previously worked at "Megabank", this does not surprise me at all.
Conway's Law applies here. It's a hugely dysfunctional bank, saddled with bureaucracy, fragile software (company change control ran on a IBM mainframe as late as 2015, requiring IE6), teams that are 'silo-ed', and talking with other teams are often the equivalent of a divorce lawyer's communication between spouses.
Sandy Weill's crowning glory of mashing together Salomon Brothers, Smith Barney, Primerica, Travellers, Citicorp, and Diner's Club.
Amusing and mildly racist use of the name Raj in the comment, not from the original article as far as I can see. Please note that John of Delaware, his boss confirmed that principal goes to wash and said to “please proceed”
Interesting. My gut reaction was that they must have initially published his name and silently redacted it. Your gut reaction is that HN is seething with many racist commenters who use R__ as an epithet for any Indian contractor.
Wayback Machine if you haven't already figured out which one of us is right.
Hopefully he just gets moved to a different project. At least according to the story and what was discussed in court, he was not singled out for blame.
I can understand if you’re venmoing your buddy $20, but if your are transferring millions of dollars, and faced with that horrendous interface, you might think to consult the standard operating procedures.
I also wonder what the action of setting an account number in the "PRINCIPAL" field does if the system basically ignores it unless other fields are set. Why is this configuration even possible?
The issue is that programmatic checks probably wouldn't work in this particular instance, considering sometimes you do want to repay the principal.
The biggest problem is this whole notion of "fake-paying" off the principal as the means to calculate accrued interest by essentially piping the actual repayment to /dev/null but actually forcing the user to type /dev/null 3 times
> The issue is that programmatic checks probably wouldn't work in this particular instance, considering sometimes you do want to repay the principal.
Of course it would - simply have a different top of the decision tree that would hard code "PRINCIPAL REPAYMENT" in one and "NO PRINCIPAL REPAYMENT" in another. Do not allow filling out details on the screen that selects the flow.
Or, as I've seen in consumer bill-paying software, a confirmation dialog with extra info (given that the erroneous payment was two orders of magnitude unexpected): "Your last payment to Verizon was $78.98. Are you sure you wish to pay $7,898.00?"
These are good, and even better when they require confirmation by re-entering the value exactly as it was previously entered; however in this case, I don’t think it would have helped.
The number typed was correct, and it was even in the correct location. It was that it was not also included in two other locations that caused the payment to go through in the incorrect manner.
Given the assumptions presented by the UI, I’m not sure anything other than an extremely detailed confirmation dialog showing exactly where the amounts were going would have solved the problem.
Even then it’s not clear to me that that info would even be available given that the software never seemed to have been designed for the use case of a partial payment in the first place.
FLEXCUBE code base is two decades' worth of hard coded decision trees. Fitting yet another decision tree in there might break something else for every single loan.
The entire story is the cost of not recognizing you're an engineering firm that handles money, versus a bank that bolts on IT at the lowest cost possible.
Citi learned the hard way, which is good, as it's the only way for such an org to learn.
Surely the cost of upgrading their user interfaces is less than the $500M lost here? What about next time? What about previous losses that were small enough not to warrant press attention but still add up over time?
To be clear, the entire $500 million isn't lost yet. Citi will need to carry the loan itself for Revlon, so the money didn't evaporate. The cost will be the admin overhead, the reputation loss from the event, and the cost for having to carry the loan into the future (or not, if Citi can find a way to securitize it out).
If Revlon defaults on the loan entirely, then the money evaporates.
Those are unrealized paper losses. The final accounting can end up being a loss of anything in the range of $0-500M depending on how it all shakes out (whether Revlon goes bankrupt, whether Citibank sells the loans on at their reduced valuations now or holds on, and how the appeal goes).
In all my previous companies whenever we earned more than expected from a given project, we'd routinely analyze it and it was our job to make sure we can somehow repeat or benefit from this lesson for the future. In the same way, whenever there was a significant drop, it was our job to make sure we know the reasons behind it and implement preventive measures if applicable. The only factor that changed was the level of 'why?'s asked.
Having worked for companies the size of small startups up through megacorps, my experience has been that megacorps can be unfathomably foolish in these kinds of things.
I'd expect Citibank to be less likely to learn (and apply) the right lessons from this experience, than a small or medium-sized business would.
To me the key part was that they couldn't create a new loan. The mistake wouldn't have mattered much if Citi/Revlon had been on good terms with their creditors.
Well the underlying issue was that they accidentally sent the money in the first place. The fact that they were on bad terms with their creditors meant that they couldn't correct the mistake after it happened.
> The mistake wouldn't have mattered much...
Accidentally sending nearly 1 Billion dollars is a pretty big mistake and it matters a lot. You can't just be throwing hundreds of millions of dollars into people's accounts and think its not a big deal because they will "probably" give it back.
Maybe. A billion here, a billion there. Judging from that UI and overview of their process/controls, I can't imagine this was the first time something like this happened.
It seems noteworthy because they didn't quickly restructure the debt. They weren't throwing money into random accounts, they accidentally sent payments early.
As was mentioned in the article, Citibank has a "six-eyes" policy. They had three people review and approve this transaction. They were all confused by the interface.
(I am not sure what this policy means if one of the three people has lost an eye; presumably it requires a fourth approval.)
as many pointed out, it still isn't uncommon to repurpose or "Extend" and existing UI to do stuff it was never meant to do and then never go back and clean up the efforts when such use is common place.
resulting in issues like Citi got. Seriously who thought that UI, requirements, text, and all, was a long term solution for handling any amount of money?
> The actual work of entering this transaction into Flexcube fell to a subcontractor in India named Arokia Raj.
> Citibank's procedures require that three people sign off on a transaction of this size. In this case, that was Raj, a colleague of his in India, and a senior Citibank official in Delaware named Vincent Fratta.
The names of these individuals seems like an unnecessary detail, especially since the article names the software interface as the culprit. I can't help but think about the recent NYT/SSC incident.
Agreed, especially considering the article's title blames UI.
They aren't naming the designers of the software, the devs who implemented it, or any number of other middle managers and other folks involved in building it and maintaining it. Just these 3 unfortunate individuals who failed to use it properly...
A comment on HN the other day referred to this as the 'moral crumple zone:' how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system.
The names are public knowledge anyway given they were listed in the lawsuit. Anybody who blames Arokia Raj, his boss, or Vincent Fratta is either lazy or a fool.
The blame rests on the software vendor who built such an atrocious product and decision makers at Citibank who chose that software, in my opinion.
> The names are public knowledge anyway given they were listed in the lawsuit.
Just because their names are relevant to the court case doesn't mean they are relevant to readers of an article reporting on that court case - especially when that article doesn't even bother to identify the court case itself or the software vendor implied by the article to be responsible for the root cause of the matter.
That is a very good point. I'm not necessarily opposed to their names being mentioned personally, but you're right that had they not been mentioned, those peoples lives might be quite a bit less stressful.
This made me re-check the article. I am consistently getting served a redacted version of this article that does not contain names. The only way for me to see the version with names is through the wayback machine. This makes it look like there are different versions of this article for different regions.
I think it drew attention to the fact that it was a real human that was involved, and made the story a bit more personal, as if to say this could happen to anyone. But I do think that they could have used a pseudonym to the same effect, maybe something like "fell to a subcontractor, let's call him Raj." blah blah blah.
If the intent is to help readers verify the story, surely an identifier for the court case would be more useful. However, the article does not explicitly identify the case (Citibank NA v Brigade Capital Management, 20-CV-6539). It does at least name the Judge (Jesse Furman) and includes a link to a "Findings of Fact and Conclusions of Law" PDF document from courtlistener.com which identifies the court case as well as the identities of the two individuals named in the article. I hope that link stays operational for the sake of anyone who intends to actually verify the article.
If the intent is to just provide the reader with more relevant information, surely the identity of the development firm behind Flexcube would be more relevant. Again, that information is conspicuously absent. It seems Flexcube was developed and distributed by Oracle.
If this sort of behavior is the result of "journalistic standards", then maybe it's time for those standards to be revised.
The names are already public record in filings. My gut says that if they left the names out, people would be yelling that there's a conspiracy to protect individuals.
It's a strange new world we are living in where people don't understand why journalists report the names and details of notable and newsworthy things that people do.
The information is drawn from public court documents and the incident is something with nearly a billion dollars in consequences that affects multiple publicly traded companies.
What do people think the word "news" means if not this?
If a journalists writes "Vincent Fratta did X", this is something that can be verified or disputed by Fratta or his colleagues. If the journalists writes "A low-level employee did X", then it's much, much more difficult to verify or dispute.
Similarly, if a journalist writes "A judge ruled X for court case Y", this is something that can be verified or disputed using public records for the court case... but that court case is never identified in the article except for a link to a PDF document hosted by a third-party organization. The public records for that court case (including the linked document) identify those individuals - along with many more useful details - for anyone seeking further validation.
Anonymous contributors where all the craze during the Trump admin, I'm not sure how dedicated the papers are to 3rd party verification.
Also, it's possible to reach out to Fratta and ask for comment beforehand - then you can either include that response, or lack of response, without naming them.
If Vincent Fratta is alleged to have urinated in a public park and gets arrested for it that's public information, and those kind of things often end up in police blotters, even if the charges are later dropped. If Vincent Fratta is overwhelmed by medical debt and files for bankruptcy that's in the public record as well.
But we should concern ourselves with Vincent Fratta's privacy when he botches a financial transaction in a way that has hundreds of millions of dollars in consequences for public companies and involves an interesting and newsworthy legal precedent?
> Are you the judge to rule whether the individual has lost his right of privacy?
I mean, there is an actual judge. This is a major legal case, and they've placed the info in the public record. The judge has in fact made a ruling.
I'm not saying it's always a perfect system, but it's definitely a very well established system in the U.S. that all court proceedings are very public. It's one of the core founding principles of the judicial branch, and it's also been a staple of American journalism for hundreds of years.
I personally think the European approach is much better on this specific issue, and that right to be forgotten laws and similar are justified.
But the reason I mentioned it in this context is that the idea that journalists naming people who are clearly notable in the context of reporting news only seems to generate a critical response when it's someone that the average HN reader identifies with.
I don’t think those first two examples should be done either. In many countries in central and Northern Europe, it’s common journalistic practice not to print the names of those accused of crimes. Innocent until proven guilty, right? I think we’ll look back at the NYT et al publishing mugshots and full names of the accused as downright barbaric.
Undoubtedly this mistake and the media coverage has ruined this employee’s life, for something that’s surely a very poorly run operation. Anyone who has ever worked in operations knows this type of error happens daily and repeatedly.
Who is the relevant person to name here? The one who screwed up designing the UI so much? The one who decided to enter into the contract with oracle? There are much more relevant people to this story, but this is presumably just reporting on public records without doing any legwork. If the more relevant names aren’t in there, the reporter isn’t going to go find them and will instead just go with what’s easy.
You don’t actually say what this thing we’re supposed to understand is, but I think you’re implying that it’s in the public’s interest to know it? If that’s what you meant, I’d disagree that these names are in the public interest.
The risks involved of naming someone have gone up quite dramatically. Today, anyone can take a random person and, from anywhere in the world, cause them direct pain. Swatting, slandering them as a pedophile, bank fraud, etc.
Sure, their names are “news” but how does the name of someone who made an innocent mistake help me understand the news? What’s the value-add? The “ROI” seems in strongly negative territory here.
It means "The people's names are irrelevant to the actual event that happened. Especially when they're rank-and-file engineers and not people in positions of power."
There is no public interest in naming these people since they're ordinary employees and there is no particular blame on them since it's indeed clearly a UI issue. For a high-up manager, this would be different.
Naming the developers who wrote this and signed off on it, now that might be more relevant.
US journalists (and bloggers - hello Brian) tend not think about this much. They'll publish names in their articles just because they can.
When pressed they'll blab some indefensible nonsense about their duty as a journalist, which apparently includes throwing innocent people under the bus with a flourish.
That doesn’t make it okay to amplify that information.
I’ve got no sympathy for Citi and the litany of penny-pinching that created the situation. And I loathe that their first reaction was probably to require ANOTHER layer of bureaucratic approval. But I don’t find any value in embarrassing the people who pressed the buttons.
I find that the real names improves the emotional impact of the story. Like when you watch a movie, it is just nice to know that "these are real events that occured to real people". Invented names hurt the story-telling, since it reminds you of the fact that "oh, this isn't totally real".
This is the same reason that NTSB reports are fun to read: you know that you're not just wasting your time with someone's imagination.
The public knowledge of their names also has positive impact in their circle of friends. Like, for example, now all their friends know that Oracle is a trigger to them and shouldn't be mentioned. It's a win-win-win: Society is entertained, the people named avoid getting triggered, and the friends avoid mentioning uncomfortable topics, triggering them.
1st, this isn't a movie. getting entertainment from a real life person's drama can be inappropriate and dystopian in certain contexts, and this very well looks like it could be one of those contexts.
2nd, if an individual chooses they can share this story with their friends. an individual could feel very ashamed/embarrassed, and this comes across like the media airing their dirty laundry. it could really injure someone psychologically.
3rd, they only mentioned the Indian man's name but not the guy from Delaware which comes across as borderline racist.
Anyway it appears they took the names down from the article.
Well, it looks like Ars has censored their article of information that is part of the public record. Not exactly a great accomplishment for free speech.
How exactly do you "censor" something information that is part of the public record? It's part of the public record.
Nobody is saying they aren't allowed to publish the information. Tabloids exist and can say whatever they want, but a serious outlet like Ars is held to higher standards by their customers. I assume Ars removed the names because they agreed those details were unnecessary and only served as a potential liability for those named.
Free speech means the freedom to say or not say what you want. Ars chose to not say something of their own volition.
Self censorship is nonsense. Freedom of speech entails the right to not say things you don't want to. People tend to avoid saying things others don't like - particularly businesses whose profits depend on their reputation.
> Freedom of speech entails the right to not say things you don't want to
Ars clearly wanted to say it, which is why they published what they originally did. However, because of the chilling effect, they felt the need to go back and retroactively censor themselves even if they didn't want to.
This really is an awesome example of a bad UI. These are exactly the kind of UIs you see in the enterprise all the time, with the article containing the totally crazy description of what was expected plus the screenshots to show that there was absolutely no way to get this, if not being the programmer who wrote it (and even then, no guarantee at all to get it right). To have a somewhat exact approach of detecting and fixing issues like this, that is what usability is all about.
So it's really the design of the software and not just the style of it that was wrong and had to be improved. A usability professional would have caught that immediately - and would have been way cheaper.
I somehow get the feeling that this wasn't so much designed as evolved. There probably was (maybe still is) some kludgy program on a mainframe somewhere, written in Fortran or Cobol, and they just slapped a web interface inbetween.
Slight nitpick anecdata: one of the best utility program I ever had the pleasure to use was an AS400 terminal based program for national tax office. I don't want to oversell but it was the nicest software tool I ever used in a job period.
Those old things were so lean, so fast, so cheap, zero fancy feature, zero presentation, zero confusion. And the software used every cycle to either make you go on, or then check or even prepare fixes (think live typecheck suggestion in your IDE but for accounting input operators).
I was brutally shocked by the contrast between everything I've been fed and seen in college (that said these were the j2ee 4 days .. so peak sadness). Of course by the time I left there was some grand plan to replace it with modern html/css/js evil reincarnation that was slower, and less useful.. basically going back from your old makita impact driver to the shiny new bluetooth connected released yesterday that will fail and require the v2 before being barely useful.
tl;dr: glory to old terminal application made with skills.
That's true in general, but not in this case. The UI shown in the screenshot violates every single usability principle for dialogues as defined in DIN ISO 9241-110 - and that's just one part of the screen! I'm 100% you put that in front on a random usability professional and he will immediately see that the UI is wrong.
Explain on top of that the scheme for the interest payments and he will notice that the UI does not fit to that purpose at all. Why would paying interests involve a wash account and a real money transfer at all? If that's the job to do the UI has to facilitate it directly, not with crazy schemes. That's Aufgabenangemessenheit (to be fit for the purpose, not sure about the official translation) and it's the highest dialogue principle. That got completely ignored here. This was never usability tested, I'm certain - or if it was the results were ignored.
Sorry, I think my post above was confusing. Please let me clarify:
There seems to be a huge number of fields-of-expertise that potentially have bearing on my life or work. E.g., urologists, transmission repair techs, indoor air quality experts, sleep therapists, house painters, early childhood education consultants, therapists/counselors, sports trainers, physical therapists, housecleaners, etc.
In my experience, most of those fields have practitioners who are sure they could help you avoid future problems; you just have to hire them to see the benefit! Or at the very least, invest time in helping them understand your particular situation so they can give more detailed advice.
I could easily believe that most of them are correct, too. But each of us, even in a business setting, has limited funds and time/bandwidth to deal with such specialists.
In my post above, I was trying to say that knowing which of these many specialists one should have engaged with isn't always clear ahead of time. E.g., maybe I didn't need to bring my car to a transmission repair shop, when the only real problem I'd have in the near year is back pain from poor sitting posture.
What you say is true, but a little bit off base I think. A team designed and developed this software. So this isn't shopping around for specialists, this is basic engineering practice to do risk analysis (by whatever name you call it). Two possibilities 1) there was no such process, or 2) the process didn't evaluate UI and workflows well.
In the first case, the teams basic competence is definitely in question. In the second, hopefully this sort of thing causes an improvement in the approach. Point being, this isn't really one of the vast number of possible things that could be looked at, rather it is a fundamental part of the bread-and-butter work of the team that shipped this product.
That being said, a lot of E2E front end stuff is done really poorly because of the way that these systems are sold and serviced. Sometimes they end up in bucket 1 (team unable or not allowed to do a competent job) by design.
I made a different assumption about what happened here. The tool as developed fit requirements very well. Over time, the tool started to be used for things that were progressively further and further away from its intended design; I imagine that sending to a wash account was (for example) some employee’s clever idea of how to make use of what they already had when their overbearing boss told them to “just make it work.” Then, having been shown to “work”, the process became reified.
Fair point, but part of risk analysis is looking at what can happen when a tool is not used as designed. Obviously you can't cover all eventualities, but one thing you should do is focus on the worst possible outcomes and work them backwards.
Ah, indeed I did not get that that's the point you wanted to make :) Reading it again I really jumped to a misinterpretation there. Sorry.
You are right, it's not as obvious that this is a half a billion dollar issue that usability work would have avoided as it is obvious that the UI is flawed. I jumped to "A usability professional would have noted that it's the wrong approach" and from there simplified to "see how flawed it is".
> These are exactly the kind of UIs you see in the enterprise all the time
It really is the norm. I've worked a handful of different jobs over the years and always got an extensive onboarding guide because the section for the software is just so bloated. Software with good UI could've easily reduced the guide size by 3/4.
I think that it is a mistake to call this a UI issue. Its not like a mistake was made because text was difficult to read, or because to check boxes were too close together and the wrong one was checked. Those would be UI issues.
This was an issue of an operations team that just did not know how their system worked. Perhaps the system design could have been better. Most likely it could have been, but if Citi had a team of people using a system responsible for billions of dollars without the competence to use it correctly, that failing goes way beyond the UI.
I have to half agree with you, but in essence disagree. You are absolutely right in that this is not a UI issue where a checkbox was too small. But it was an usability issue that was caused by an inadequate UI. Plus an inadequate process, but that also manifested in the UI. UI issues can cover a lot :)
The user thought that setting the target account in one line would be enough to have the money be sent there instead of where it ended up being sent to. There the UI was misleading, that's a violation of the self describality a dialogue has to fulfill (I'll freely translate the principles, I would have to look up the official english translation); in that the UI has to communicate by itself in which state it is, what every button does, every icon means, what even is clickable etc.
Then the system did not clearly communicate what would happen. That's not only an issue of describing itself, it's also an issue of fault tolerance. The system did not prevent the user from a mistake he was about to make. This could have happened by clearly describing what would happen at the end of the process and offering an undo operation. They tried to implement this outside of the UI by requiring 2 additional set of eyes, clearly that was not a sufficient solution.
But most importantly, the process the UI was offering was the wrong one for the job. This was about sending parts of a debt to creditors. Instead of offering a clear way to do that, they had a convoluted process that involved a wash account plus a real money transfer to that account (how crazy is that!) instead of supporting exactly the work that was to be done. That violates the primary dialogue principle: Be the right tool for the job (Aufgabenangemessenheit). Admittedly, I miss context to know for sure what would be the right tool for the job and why they ended with this solution, maybe it's a bit less crazy than it sounds.
Yes, all of that together goes far above simple UI issues, but it's nonetheless an issue of the UI. It's just also a complete usability fail. It's really the job of usability professionals to analyze software and scenarios like this and to provide better solutions. That was and partly still is my job :)
The headline ideally should have called it an usability issue. That would have covered the UI part and included the other aspects.
I would think that even programmers don't know how this works.
They likely just got document that when you receive message with these fields filled this should happen. And then other team of programmers had implement UI with these fields and then make it send message here.
Scenario from article would be solved by UI if the requirement would be "SEND MONEY [YES/NO]; AMOUNT [XXXX]; APPROVALS [JANE/JACK/JILL]" but that is not what that window is doing and at that level there is no "SIMPLY SEND $7.8M; MAKE IT NOW [YES/NO];". I don't even know what it takes to move $7.8M to different accounts but I expect it is quite involved process that has to trigger multiple different things.
To be sure, this is an ugly UI but this is not necessarily a pure UI issue at root.
Internal applications have to be more complex that external applications - that's why you sometimes have to call a company to do something the consumer frontend doesn't support. The employees are expected to operate a more powerful/flexible system than the customer, and I think there's always a risk inherent there.
In this case, it's likely that it's not just a dumb UI thing that the employee "needed to set the "front" and "fund" fields to the wash account as well as "principal." I suspect thank "front" and "back" are actually real business concepts in how the bank models transfers which the employee/reviewer did not understand well. Instead they expressed a different model of a transfer (an external one) which is a totally legitimate use case, just not the intended one.
That's kinda one level of empathy, I suspect this really happened because these users think about these operations as "I check this box, then I check that box" rather than "I get how transfers work and I am going to express my intention using that understanding." So it's probably much more of a training issue, because to give these people a very narrow and polished consumer-style frontend would take away the flexibility they likely need to actually execute their roles.
A ten pound sledge hammer can be used to drive finishing nails in fine furniture but it would be absurd to claim that all the dented furniture coming out of your shop is a result of poorly trained workers rather than inadequate tooling.
A generic hammer, like this UI (at least, as much as is apparent from the screenshots in the article) can drive basically any nail. But specialized hammers exist for good reasons. This is equally applicable in software tools.
A good tool should enable you to do things you couldn't do without it but the best tools make it easier to do what you want and harder to do what you don't want.
I would ordinarily agree, but to be honest the UI in the screenshot doesn't really look all that terrible. Not saying it couldn't improve to clarify effects, incorporate documentation and nudge the user away from making mistakes - but overall I agree with the grandparent post. Clearly the operator is supposed to have business knowledge of what those GL fields mean. I'd bet there are more experienced operators who would be frustrated and less productive if they were forced to use a dumbed-down UI where you couldn't just put the amounts you want in the fields they should go.
My decade-old Quickbooks software similarly lets me create GL entries and assign amounts to the accounts involved. If I let my $12/hr intern who doesn't know what they're doing use that feature and they post to the wrong accounts, then of course bad things will happen. Instead, he uses other more specialized interfaces (like you suggested) to enter things like invoices, bills, etc. We give children safety scissors, and expect adults to be able to handle the real thing.
I really do have sympathy for the employees involved, and strongly agree with your sentiments that software should be engineered for safety. But I encourage anyone else reading this and forming an opinion to click into the article and have a glance at the screenshot.
The form looks like it can accomplish all sorts of things and business logic, just like a SQL query can accomplish all sorts of things and business logic in a database.
There's nothing inherently wrong with that. But like you say, the issue is with training.
The fact that 3 people who were presumably trained in how to use this tool, used it wrong, means it's absolutely a failure of training.
Business logic very often isn't "intuitive", because tools are used in all sorts of ways. When it comes to business logic, what's often most important is documented procedures and checklists.
The real question is, since this was a procedure that had been performed regularly and correctly in the past, why did it fail this particular time? Why were these 3 employees left to assume they were using the tool correctly, rather than referring to a definitive documented procedure that would give them the answer for sure?
Oftentimes it's an employee turnover/transfer/vacation issue that exposes the lack of documentation, and the real issue is too much knowledge in employee's heads that isn't committed to paper in the right places -- pure sloppiness of business procedures.
> The Fund Sighting Manual explains that, in order to suppress payment of a principal amount, “ALL of the below field[s] must be set to the wash account: FRONT[;] FUND[; and] PRINCIPAL”
and checklists:
> Raj then emailed Fratta, seeking final approval under the six-eye review process, explaining “NOTE: Principal set to Wash and Interest Notice released to Investors.”
and confirmations:
> which prompted a warning on his computer screen — referred to as a “stop sign” — stating: “Account used is Wire Account and Funds will be sent out of the bank. Do you want to continue?” But “The ‘stop sign’ did not indicate the amount that would be ‘sent out of the bank,’
To answer your question "Why were these 3 employees left to assume they were using the tool correctly, rather than referring to a definitive documented procedure that would give them the answer for sure?" My guess is that the manual is a million pages long and incredibly confusing.
My takeaway from this is that you can't train and document your way out of a core human factors issue. Your point about SQL queries is apt; that's why you don't just give humans access to prod.
That's a good point, besides not following the documentation (or they weren't aware they were supposed to "suppress payment of principal amount). They UI "stop sign" prompt should show the # of on the confirmation machine. It should accurately summarize everything they are doing, not just a simple "yes/no" they've seen a bunch of times.
Just like in Hawaii when they sent the wrong cell phone warning of an incoming NK missile attack, it could easily have been prevented if there was a dialog which said "You're sending this 'prompt' as a live alert to millions of cellphones (yes/no)".
Software always assumes people are paying attention but for serious usecases like this (which can't be undone for example, or sending hundreds of millions) there should be a sanity check dialog with real information.
Hierarchy of controls! Training is supposed to reinforce technical measures not replace. Have you never screwed up something you've done hundreds of times perfectly before?
I think that your comment mostly proves the opposite point. I wouldn’t say that it’s only a ui/ux issue. However, I’d say that ui/ux is the best solution in this case (and most other cases)
The fact that three people independently verified the UI (all highly experienced with this exact GUI) before issuing the transaction betrays the notion this was a lack of training.
Training cannot substitute for Engineering controls aka physical safety features that make it impossible to explode the entire lab or send 1 billion dollars by mistake.
This is no more user error than Boeing’s recent sensor disaster. Sure if the pilots had the proper training none of that would have happened, and Boeing didn’t make sure they had it. But it wasn’t a training failure that was the issue with with the Max 8. Training was only a second order cause.
Looking at the evidence, I would say this is a pure UX/UI issue at root: three people got confused about what they needed to do, and they all thought they completed their task successfully when they didn't.
The first UI issue is the obvious lack of intuitiveness. There is ancient software that is resonably intuitive to use, because the people who led development took responsibility to do the right thing: invest in UX/UI so that their software is a success. Others however cut corners to get more short-term profits, or dismiss design as if it was decoration (it's not, good design is form + function and most serious designers pay more attention to the latter than the former).
The second UI issue is (an assumed) lack of feedback in the output of the transaction. After doing the transaction, if there was a feedback screen confirming to the user what they've done, the error could be caught much more easily.
For example, before actually sending any transaction there could be a screen saying:
- Summary of transaction
- X$ will be sent to account XYZ
- X$ will be sent to account ABC
This looks like obvious stuff, but is often obvious stuff and small details what turn a problematic user experience into a succesful one.
I am the original poster you're responding to. On the high level I agree with what good looks like - I'd strive to have all of these things in a system if I were the designer.
I am also recognizing that both you and I may not understand the complexity. Saying "X is being sent to ABC" may be a gross-oversimplification of the transaction and could obscure lots of badness. It's possible that the level of complexity of the screen IS what's needed to express it.
Analogy:
Would you expect to walk into the cockpit of an airliner and see a big dumb screen that tells the pilot "you are landing the plane?" or would you accept that it's a complicated serious of operations which the pilot understands by putting together the various bits of info on various instruments. Hope the parallel is clear.
Again, by no means am I insisting this is a great application (I don't know anything about it) but that you can only simplify the display of very complex state so much.
> The first UI issue is the obvious lack of intuitiveness.
You can't judge intuitiveness without knowledge of the intended userbase and usage pattern. A sovereign app for specialists in a process can be intuitive with a design which would be radically unintuitive for an occasional-use app for people only tangentially aware of the same process.
> Looking at the evidence, I would say this is a pure UX/UI issue at root: three people got confused about what they needed to do, and they all thought they completed their task successfully when they didn't.
It's definitely a UI/UX alignment issue, where the understanding/knowledge of t(at least some of) the current users is not aligned with the assumptions of the UI.
But without knowing a lot more, it's hard to tell if the real problem is UI/UX design or processes and procedures for maintaining, assessing, and addressing (whether through system changes or otherwise) the current understanding and evolving needs of the current staff.
We aren't at the point where it is legitimate to expect software adapt automatically to changes in the userbase.
> three people got confused about what they needed to do, and they all thought they completed their task successfully when they didn't.
> For example, before actually sending any transaction there could be a screen saying: - Summary of transaction - X$ will be sent to account XYZ - X$ will be sent to account ABC
These sums up what I think too. I get that internal software is more complex, and bank transaction are also complex, but when three people checked it, though it was good to go and didn't realize a mistake until the next day, that's a problem.
Showing a summary of how much is being sent to where, and how much is going to stay in the account is the minimum I would expect for a finance software that's used to transfer millions of dollars in a single action.
Yeah, I'm familiar with that. My old boss used to say that a dog that has 2 owners die of hunger. When we started having 2 reviewers for PRs, he was adamant that there will be ONE that would be the person actually responsible for it. So the main reviewer always had the same responsibility as if they were the sole reviewer.
> Honestly with names like "front" and "wash" the whole thing should be checked by tax officials. The words probably mean something else, but ... wow
It's funny how worked up and self righteous one could get even while recognizing that they don't actually understand the concept. It's particularly amusing that you were alarmed by "front" (what, like a mafia front?) when there's also a "back".
I have never worked at Citi but I can make reasonable guesses as to what these accounts are.
Wash is a term that means "funds did not leave the firm" (you may have heard the expression "it's a wash" to mean "things net out to 0."
Front and back may have to do with direction of funds flow, or perhaps a hierarchy of accounts (like "we are pulling from account X on the back-end and into account Y on the front-end")
Not saying these are exactly right, point is that as someone who's been in finance for a long time these are totally innocent.
worked up? self righteous? alarmed? You seem to read with a lot of -bad-faith- (edit: no that's the wrong idiom.. ähm read while assuming the worst?). Take a step back and relax. It's obviously not about mafia fronts and money laundry ^^ The whole story is about the user interface being unclear and not being understood by those who use it. Could have made it clearer i am playing with that theme ... maybe an /S would have helped, or one of those smiley things i keep hearing about that are so uncommon on this page.
Not surprising that Oracle's promotional video for Flexcube is all corporate-speak and buzz-words, and doesn't actually show the product or even explain what it does. https://videohub.oracle.com/media/t/1_mxpp4dyv
All enterprise software is terrible like this. It's because the people ordering the software are disconnected from the people who will have to use it
> The system is embedded with a patented blockchain adapter that enables Oracle FLEXCUBE to interface with any blockchain system. The adapter enables a seamless interchange of information between Oracle FLEXCUBE and external blockchain data setscan work with any version of Oracle FLEXCUBE with minimal changes. The easy configurability of the adapter enables banks to leverage blockchain technologies to solve business problems,improve process efficiency, reduce risk and enhance straight through processing.
This is golden. It's spoken like a voice synth. "cut operating costs, deliver high volumes, and transfer these benefits to its customers", then "meanwhile, at the back office, ... Oracle's new Machine Learning adapter unlocks intelligence ingrained in indivudual operational patterns".
I had to share this with my team today. We produce an app for banking that is used at the front line (i.e. in the branches). We have explicitly acknowledged the fact that the bulk of our users are going to be new to the business & are going to be paid peanuts while simultaneously being expected to produce consistent business outcomes. As a result, we have tailored our interfaces for self-discoverability and intuitiveness. Making a UI intuitive is a book all by itself, but suffice to say that we spend a lot of time agonizing over how UI elements are laid out so that users are less likely to make errors in judgement.
Our application is not immune to exposing exceptions to the business. But, we have added measures throughout to minimize this as much as possible. In areas where there is a lot of jargon which might confuse new employees, we put help buttons which allow a user to quickly review important terms & other documentation in-line with the actual functionality. This makes our application almost a training course in and of itself.
Additionally, in cases where we identify that poor choices could have broader impacts to other parts of the business (i.e. lighting 500mm on fire), we add explicit validations with hard cutoff limits to prevent insane things from happening. In this specific case, we would probably walk the user through a decision tree to force them down a valid path and ensure the funds were being tagged to the correct account(s). We also have approval loops in our application for more sensitive operations, but even these have integral validations & other measures to ensure that "experienced" employees don't screw up either.
This is a great lesson. However, I'm afraid for you're designing your product to be good for the people who are being paid peanuts and are new to the business, but those are not the people who are buying your software. They're just the users of the software. Oracle put basically no thought into their UI but put a lot of money into promotional videos and sales, and they sold their software to giant international banks because the people buying the software experience it only through videos and salespeople.
I'm terribly afraid that you're going to be punished for focusing on the right things.
The entire point of our software is to be intuitive enough so our clients can hire cheap labor. Its a first class feature. Why would we be punished for this?
I would like to point out that Flexcube is built by Oracle. Citi's mistake here was in tying themselves to what is likely the "safest", "market leading" solution.
If you all could see how frustrating Ariba (by SAP) is, you'd have a great laugh.
Ah, Ariba. I am currently getting an email every day from Ariba stating that I need to approve the purchase of my company laptop (that was purchased 3 months ago, currently sitting on my desk). The email also threatens me that if I don't approve the purchase, it will be escalated to my manager. As a lowly software dev I don't have the ability to approve anything in Ariba. Because it is assigned to me, apparently no one else can approve it either.
Does Citibank have a case against Oracle? OSS licenses all have that "this software is not fit for any purpose at all, don't even think about suing us if it goes wrong" clause, which would seem to imply that licensing poorly-made software can be grounds for a lawsuit.
Ha - whatever OSS licences say, when you're dealing with terms written by Oracle lawyers, you're getting even less chances of successfully suing them if it goes wrong. You can argue about their technology, but the legal part of Oracle is quite capable, so much that I would assert that Citibank does not have a case against Oracle no matter what the software did. Pretty much all commercial software contracts includes similar idemnification clauses - they can't include the "not fit for any purpose" but the "if it goes wrong, it's not our problem" part definitely is there; and if it turns out to be not fit for purpose (which is ridiculously unlikely to prove for any actual product that's not totally made-up-nonexistent-fraudulent thing) then you might get back some of what you paid for it, but not for the losses you had due to it failing, assuming the licence/contract wasn't written by incompetent idiots.
It could just as easily imply that OSS looked at commercial software and said "holy crap that's a lot of legalese; we better have some of that to make sure we don't get sued, since we don't even get to decide who uses what we put out there"
> which would seem to imply that poorly-made software can be grounds for a lawsuit.
That doesn't necessarily follow, lawyers have a long tradition of including "cover your ass" clauses even if they don't think it's likely that they would be liable without it.
> Ordinarily, the law would be on Citibank's side here. Under New York law, someone who sends out an erroneous wire transfer—for example, sending a payment to the wrong account—is entitled to get the money back.
> But the law makes an exception when a debtor accidentally wires money to a creditor. In that case, if the creditor doesn't have prior knowledge the payment was a mistake, it's free to treat it as a repayment of the loan.
There's a major UI issue here, no doubt about it.
But, it's important to also point out how bizzare this exception is, how did this even get included as the one and only exception?
I think the law envisions a case where you owe money to someone, and the due date for repayment is already in the past.
So, if I borrow $1,000 from you and am supposed to pay it back next month, and then the due date arrives, and I accidentally transfer $1,000 to you, then yeah, it makes sense that you should be able to keep the money you are owed, even if it was sent mistakenly. It would be bizarre to demand that the creditor return the money to a debtor who is at risk of stiffing them.
But in this case, the due date for repayment was far in the future, and it makes far less sense for the exception to come into play.
That said, I don't understand why the creditor would want to hold onto the money. If they made a loan, they did so to make money off of interest payments, and if the loan is prepaid, they make less.
EDIT:
I read more about the lawsuit. In most cases, it doesn't make sense for the creditor to want to hold onto the money.
But in this case, the creditors were on bad terms with the debtor and were worried that if they returned the erroneously transferred money, they might never see it paid. So the exception to the law, which seems unnecessary unless the loan is past due, ended up benefiting them, because they got their at-risk loan paid back.
> That said, I don't understand why the creditor would want to hold onto the money. If they made a loan, they did so to make money off of interest payments, and if the loan is prepaid, they make less.
The borrower in this case is in financial decline, so they would not get the loan on those terms day. Imagine if you had a loan, lost your job, and then accidentally repaid the loan. The lender would be breathing a sigh of relief.
> Earlier in the year, as the pandemic was accelerating, Revlon experienced financial difficulties and sought to borrow more money. To do that, Revlon convinced a majority of its previous creditors to allow it to transfer collateral from its old loan to a new one.
> The strong-arm tactic angered the other creditors, who felt that the reduced collateral could leave them holding the bag if Revlon ran out of money. That's more than a theoretical concern: Bloomberg's Matt Levine reports that Revlon's debt is "trading at around 42 cents on the dollar." But under the terms of the loan, the minority lenders didn't have a way to force early repayment.
Creditor doesn't necessarily mean bank lender. If you're Revlon, and I supply you raw materials (like beeswax used in lipstick) with standard 30 days payment terms, I am also considered as your creditor because you owe me money. There is no interest expected, rather this is typical of doing business and how firms expect their suppliers to give them payment terms of 30 days, 60 days, or in many cases I've seen, 6 to 9 months. And yes, in Revlon's creditors' case, or in the case of any business who supplies goods on credit, the pandemic absolutely made you panic over whether you could depend on receiving all the money that is owed to you.
In any case, I doubt this ruling will stand, they are a huge bank and this is a decent chunk of change. They've got their ways to making sure the appeal goes in their favour.
I think it could easily stand for this specific case.
The loan was supposed to be repaid by 2023 if I read the article correctly.
It's already 2021, the case could very easily drag for another 2 years, then there is no reason to return any money because it was supposed to be paid.
The other party are other banks and hedge funds could can hold the money and can also afford lawyers.
In this case - the creditors all expected it was likely that Revlon would default on the loans and never pay them back in full (let alone interest). And there were active lawsuits between the creditors and Revlon regarding financial maneuvering Revlon was doing around these loans that would make it even less likely.
> If they made a loan, they did so to make money off of interest payments, and if the loan is prepaid, they make less.
The article mentions the creditor's impression of the company soured over time, and further their debt was trading for 42c on the dollar. So they got a nice little >2x return on expected value.
Think about it from the side of the creditor (the person to whom the debt is owed).
The point is, if the creditor is owed a debt, and receives payment for it, and A) He has no reason to think the payment was a mistake at the time he received it, B) He did not falsely induce the debtor to make the payment, then the creditor should not have to worry that the debtor will come to him the next day and say “Uh, that was a mistake, I want the money I owe you back.”
This is not the only class of accidental-but-plausible transactions. Why not just use "reason to believe the payment is valid" directly rather than singling out creditors?
What circumstance do you have in mind in which the person receiving the transaction would A) have reason to believe the payment is valid, B) not be considered a “creditor”?
One could have a reasonable belief that the payment was intended as a gift, in which case the recipient would not necessarily be a "creditor" but there is still no reason for them to believe that the payment was invalid.
Here's the distinction though: $400 out of the $900 million was returned. Thus, it seems like we can't get past your part A) in this case, as hedge funds had enough reason to think the payment was a mistake that they returned it. I doubt the creditors in this case are so heterogeneous such that one creditor has no reason at all to think it was a mistake while another does.
The question the law asks is whether they knew it was a mistake at the time they received it. Not whether anyone later decided to be a nice guy upon learning it was a mistake.
In this case, what happened was, Citibank sent out these payments on August 11. Each creditor received a payment in the exact amount (down to the cent) of what they would be owed if Revlon had actually intended to pay down the debt in full, with interest.
On August 12 they told all the hedge funds “oops, send it back.” Some of them said “Hey, no problem, it happens to the best of us, here you go!”, and returned the money. Some of them said “Uh, we’re not going to do that.” And this latter group are the ones who just won in court.
> But, it's important to also point out how bizzare this exception is, how did this even get included as the one and only exception?
It is question how exactly the law is written. I would not bet on legal details to be exactly right in non-law-expert journal.
Here, a receiver or an errorneous wire transfer is not required to send them back based on the fact that wire transfer was unintentional on the sender side, but is required send them back based on the fact that the receiver do not have valid claim to that money.
If the law in NY has been formulated this way, the 'bizarre exception' would be just natural result of the law and not explicit exception. If a debtor unintentionally sends money to the creditor while the debt is due, then creditor has valid and legal claim on that money.
It seems pretty straightforward and logical if you view the payment as the second payment of a series.
If you look at this sequence of events you can see why it might be problematic:
1) Bob has money, which he lends to Dave. Dave has possession but it's still Bob's money.
2) Dave returns the money to Bob. It's Bob's money and he has possession of his own money.
3) Nobody owes anyone anything and there's no obligation from either Dave or Bob to do anything as any agreement that existed has been fulfilled.
4) Court orders Bob to give Bob's money to Dave.
The action contemplated in point 4 here is a pretty extraordinary remedy. Even though it's Bob's money, and there's no contract or agreement outstanding that obligates him to do anything, and even though Bob has done nothing wrong and is not in breach of any obligation or agreement he's going to be ordered to give money, which is his and he has every right to have, to someone else.
It's pretty easy to see why the legal system might decide that's not something it wants to do.
> 1) Bob has money, which he lends to Dave. Dave has possession but it's still Bob's money.
That's not the way a loan is structured legally. Legally, a loan is a bargained-for exchange (a contract) in which the lender gives the borrower money and the borrower promises to make payments according to a schedule. (Note: It can be more complicated than that.) When the borrower receives the money, it becomes his money. If the borrower doesn't make the payments, then the lender can sue for breach of contract or to foreclose on a security interest if there is one.
> ... there's no contract or agreement outstanding that obligates him to do anything ...
By default, a lender is entitled to receive the stream of payments specified in the loan contract and is not required to accept prepayment if he doesn't want to (e.g., if the lender is receiving interest at 10% and the current rate is 5%, the lender does not have to allow the borrower to pay the principal all at once). Prepayment rules can be modified by the loan contract terms (e.g., imposing a penalty for prepayment) or by statute (e.g., a state law requiring housing mortgages to allow prepayment without penalty after some amount of time).
Broadly, the New York default is a departure from the historical norm, and the debtor/creditor "exception" is a return to it. The historical view is that the error is the payor's; they're the ones that should bear the burden.
There's also a value to finality. Particularly given that we didn't have instant or even next-day transactions when these laws were written, allowing a creditor to move on with their lives once the debt seemed to have been repaid makes sense. And not having an exception would allow debtors to pay their debt and then reconstitute it days later if they changed their mind. Having this rule reduces the prevalence of that and the corresponding costs.
Just from a 30k foot view, the lender has a valid legal claim to that money. If you accidentally paid your mortgage or rent without meaning to, should the lender have an obligation to return it?
For an unintentional payment, I agree. For an accidental overpayment, an argument could be made that the payer has a legal claim to a refund of the overpaid amount.
If you take out a $200,000 home loan with $200 minimum payments and accidentally make a $2000 payment ($1800 over the required minimum), I can see a request for an $1800 return being legally valid - granted you aren't behind on payments.
If you are behind, I would argue that including the total past-due, whatever is still overpaid, can legally be requested for return. So if you owe 3 payments, a request for $1400 returned would be legally valid.
However, if you are behind on payments, it could also be argued that you've breached trust with the lender and any money they receive from you is now theirs until the balance is $0. This is something that should be explicitly discussed in a loan contract prior to agreement.
Comparing an bank-to-individual loan and a bank-to-bank loan is a bit apples to oranges, but a massive unintentional overpayment can have serious ramifications, and I wouldn't write it off as simply as, "Well, you owe them a grand total of X, and even though you only owe them Y per month and overpaid, you don't get any of what you overpaid back".
> For an unintentional payment, I agree. For an accidental overpayment,
From the article:
> The defendants noted that the amounts they received matched the amounts Revlon owed down to the penny, making it reasonable for them to assume it was an early repayment of the loan.
So they were safe to assume they just got paid early, at least in this case
Feels like a super odd edge-case where someone accidentally overpays a loan exactly in the amount that is due in total, but I agree. It's a fair assumption.
I think it makes sense. If you are already owing money the creditor is expecting the payment and has no reason to believe it is a mistake. If you then afterwards come and say "hey I was not supposed to pay the money i owe you, please pay it back" things become sort of complicated.
The weirdest part seems to me that the judge applied it despite the "doesn't have prior knowledge the payment was a mistake" part. The dept traded for not even half its value, nobody actually expected Revlon to ever pay it back. I think anyone on the receiving end claiming that there was nothing questionable about it suddenly getting repaid in full would have problems keeping a straight face.
> The UMB Complaint was the culmination of a bitter dispute between Revlon and Citibank, on the one hand, and a group of Lenders holding a large interest in the 2016 Term Loan, led by or including most Defendants here, on the other. The group of Lenders alleged that, in the May 2020 Transaction, Citibank and Revlon had improperly manipulated the voting provisions of the 2016 Loan Agreement to gain approval of an amendment that permitted Revlon to strip collateral backing the loans.
> ...
> In that heated context, it was reasonable for the Lenders to believe — and several did believe — that Revlon and Citibank, perhaps with the help of Perelman and MacAndrews & Forbes, had figured out a creative way to pay down either enough of the Lenders to prevent them from filing the UMB Bank lawsuit or the 2016 Term Loan altogether.
> But, it's important to also point out how bizzare this exception is, how did this even get included as the one and only exception?
It's not really 'an exception'. The article most likely written by a non business person confuses a mistake (money sent to the wrong account) with money sent to the right account. The law almost certainly doesn't allow a wire transfer sent to be recovered for that reason and type of mistake. It does (by design) allow a transfer sent to a mistake (like a typo) to be recovered. This actually makes sense and here is why:
A wire transfer is payment for 'goods or services' let's say. Often that 'goods or services' is released (or relied upon) when money is received.
Wire transfers are by design 'final'. ACH is not. An ACH can be reverse (for a number of reasons).
If you could claw back wire transfers all sorts of things (that depend on them being final) would break down and you'd have all sorts of problems down the transaction line.
It's like giving someone money in a sense. You give someone money they give you something in return.
That law does seem to make it a pretty straightforward case.
I'd assume that got included as an exception due to lobbying by lenders/creditors. :) If a hundred years ago or whatever, it might require some historical digging to ascertain the details.
If we're looking for rational reasons (and laws aren't always motivated by rational reasons), we could imagine that the creditor, assuming that it was a debt repayment, has already spent the money or whatever, it's not fair to make them give it back, they might not even have it anymore. They weren't just a random person with no reason to expect the wire transfer so should have looked into it more before just spending what wasn't their money -- they had all the reason in the world to assume it was their money, and go forward accordingly. And the money was owed to them after all, so they aren't exactly keeping what didn't belong to them, they just got what belonged to them back a bit early.
My guess is since over payment is a thing 99% of overpayments are intentional and require no intervention but 1% would require an entire department to figure out the correct refund so they went to their local politicians and asked if they could skip that whole thing and that’s what the politicians agreed with.
If the clause didn't exist, I could pay off my 250K Lamborghini loan, get the title and then ask for the money back because it was a "mistake".
For similar reasons, items at Sherrif's auctions can't be returned to the original owner even if they win the original court case that led to the auction.
Everyone here seems to focus on the bad UI and blames those who made the software because this particular operation was complicated to perform and failed. But all this is missing the points: the UI reflect the business, the UI has likely been designed to perform many very complex banking operations and works well for that, we just don't
know how powerful it is from the screenshot and are not experts in the field. It's as if we were shown a shell and people were like "wow look at that horrible UI, no wonder they deleted the production database".
Sometimes a basic UI is the most efficient at some tasks, but one has to be trained to use it properly and processes have to be in place to prevent horrible issues. This is the problem here and this is where they failed.
This problem seems like it could have been fixed by a better UI though, specifically one which included a confirmation step that showed the implications of the decision i.e. the amounts that would be transferred and who they would be transferred to.
If there was a screen that said "the transaction you have selected will result in a total of $900 million being sent to the following parties, broken down as follows: ...; Please confirm you wish to proceed" - then the outcome would have been very different.
OK, so they payed $500M about two years early, right?
That's very bad for your cash flow, and you lose 2 years worth of interest from that, but in the long run, citibank isn't $500M poorer, right?
I'm trying to understand what the real financial impact is. Interest rates aren't very high right now, so can maybe a few percent of the $500M per year?
Regarding the UI issue: Not only is it bad that the user interface is weird, but that approval seems to use the same bad UI as entering the data. The approval stage really needs a line like "this will result in a $ X being sent out on $Date to $Recipient".
Citi accidentally transferred the very real risk of default from the creditor to themselves. Revlon bonds trading at 42 cents on the dollar... So, you could say that Citi is out 58% of $500M today.
Edit: The 'creditor' in this case is composed of 10 investment advisory firms, who, at one point, thought the interest payments were a good deal for them. But, rest assured that they are quite pleased to be made whole on their bad investment. Source: https://www.cnn.com/2021/02/16/business/citibank-revlon-laws...
Not quite. They paid on behalf of of Revlon which is unable to pay.
Effectively, Citibank bought a 500m loan at par when its trading at 43c to the dollar. So Citi just gave the lenders a little over $250m and now is stuck with a non performing loan.
The sort of financial gymnastics all these groups participating in hoping to leech money off gambles in the structural system vs creating real value makes it difficult for me to have sympathy for any of the parties involved gambling on what will happen.
It's like playing poker and you decided to flip one of accidentally showed your hand. Sorry, but you already joined passing the risk game and shamefully faulted out on the low risk scenario due to negligence.
But the law makes an exception when a debtor accidentally wires money to a creditor
Okay, but Citibank wasn't the debtor here, they were simply the middleman. And:
"To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake... would have been borderline irrational"
Sure, if you hear hoofbeats, the likely guess is horses, not zebras. But if someone calls you up and says "Hey, I got some nice pics of zebras running by your house" then you know the probable guess was wrong, and should act accordingly.
Absolutely the mistake was Citibank's fault. Maybe you think that means, inherently, that they should lose the case. But if you look at the actual laws on the books, I really don't understand how the interpretation of the laws comes down against Citi here. Though a half $billion mistake on UI design might still be a very beneficial lesson for the entire UX design community to remember.
I see rulings like this and lose even more faith in the law. Clearly the spirit of the law should have protected Citibank here, but the letter of it fell short. I imagine a bunch of self-important lawbook-warriors patting themselves on the back for how they manage to miss the forest for the trees on a daily basis.
I have a contrarian view that this is not a UI design issue - instead a user training issue.
Oracle and bad UI design often comes up on HN and every time the debate is that Oracle software is very bad and ripe for disruption.
However, most of what Oracle has built or bought is complex because the purposes it serves is complex.
You cannot expect a user to just sit in front of this screen (or any other Oracle screen for that matter) and figure it out. There is a lot of tribal knowledge involved and most of the times it is a user training issue.
If the user and his manager was unsure, they should have tested it on a dev system or with a smaller amount. Blaming bad software is easy. But having worked with complex ERP systems for more than a decade, I can easily say tat most of these systems are designed keeping user feedback in mind. In fact a lot of modules that Oracle sells were systems that their users built as add-ons to Oracle and were finally embraced by Oracle.
Everybody blames bad ERP systems from Oracle (PeopleSoft, JD Edwards, Oracle EBS) and SAP. Yet noboody is building competing systems at a mass scale. Because they are complex and needs lots of integrations.
That is why, you see lots of small business ERP systems but very few complex full scale ERP systems -the kind Oracle dabbles in.
I tend to agree. It seems likely that Citibank handles transactions of this magnitude with some regularity, so even if the UI is crappy I would expect the employees involved are used to the quirks and know how to work with the system. Given that my company uses similar shitty Oracle software, I'm sympathetic, but we have people at the company whose only job is to work with this software and make sure things are done correctly.
Yes, for these things you can expect to figure it out - and/or it should be obvious what you are doing.
There's no need for that level of ambiguity.
You're dealing with large sums of money.
The parameters described in the video are superlatively ridiculous.
It's their fault.
FYI verticals of ERP are usually not that complex. And there are other reasons people don't compete, it's because of the inherent flexibility needed in these systems and their vast incumbency.
Imagine working on the complexities of financial software for a major financial institution. I don’t have to imagine this as I work for a much larger financial institution with many more financial products.
Now take that complexity that is in your imagination now and couple it with internal software. Think about the last megacorp you worked at and how awesome the internal tools are.
Now imagine writing the applications and processes that govern and that internal software. The internal end product doesn’t get the product attention it deserves, because well... it’s just internal and not seen by customers. The stuff you write to support that end product gets even less product attention. It’s an act of God to get a few more lines in on top of the millions of redundant lines already present. Now couple that with the inane process of financial management internal to the financial industry.
Shitty enough yet knowing the product management requirements to make this good user facing software? Try imagining then testing it with automation.
The complexities are vast enough that converging, in my area, to git for version control from various other tools was a multi year effort to avoid completely halting the business requirements.
Used to work in banking innovation and have plenty of friends that work/have worked at banks. Bad UIs cause errors and inability to help customers all the time.
I have worked for tier 1 banks for 6+ years in total and concur with the aforementioned statement about bad UI. My experience though is that no one wants to be bad obviously, and there’s plenty of effort to improve things, but big banks move slowly and corporate structures often inhibit swift innovation.
Edit: regulation doesn’t help either. Very often not enough time and effort is spent translation legal requirements into usable UI, and you have be both a legal and software expert to understand what you’re doing. If you get it wrong, you may well have just signed away all your money. Case in point: power of attorney features, pretty much everywhere.
A lot of people are pointing to the UI here but I think there’s a more fundamental problem of using a system in a way it wasn’t designed to in the first place.
The need for transferring principal to your own “wash” account in order to make an interest payment to other parties seems unconventional.
It’s violating the problem domain. Similar to using special int values like -1 or -2 in an orderId DB field to indicate that order was cancelled or in progress instead of creating an explicit orderStatus field.
Oh, it's worse than that. They wanted to roll the portion of the loan with one specific creditor into a new loan, in a sweetheart deal that was going to screw over the other creditors.
To do that they needed to make an interim interest payment to that one creditor, and mark the loan as paid off to that creditor; but the system wouldn't let them mark the loan as paid off without making payment of the principal in full, and it wouldn't let them pay creditors individually.
If they'd just been "repaying" the principal they'd have caught the issue when it said (paraphrasing) "Money actually leaving, are you sure [Y/N]", but they were expecting some money to go out as interim interest (they'd got the OK to pay the relatively small amount of interim interest to the other creditors).
And at that point I would expect them to run the transactions to some people who are exactly trained or capable to verify that this transactions is correctly recorded.
If you are doing something really really specific against usual procedures no UI can save you...
This should be top comment. I didn’t really understand the article but you’ve explained it perfectly. Did you read a more thorough explanation somewhere?
Yes, I read about it in Matt Levine's newsletter (on Bloomberg), and then read the judgement itself - it's a surprisingly enjoyable read and the judge goes into some length to explain both the (crazy) situation and how they were compelled to interpret the law and precedent.
So they put 3 guys who seem to never had done such a thing in charge of a $1B transaction, that requires special parameters, and they didn't even checked if their assumptions were right?
And here I am, tearing out my hair, sleepless nights, wondering if using iodash instead of pure JS incurs in needless overhead for people I don't know from anywhere visiting my website.
(probably someone got a nice check out of it though)
It was not a 1B transaction. It was supposed to be a 7.8M transaction. Not trivial, I admit. But from what is written in the article, it seems that even if they had transacted just 1$, they might have ended up in the same situation of losing 900M. With 1$, there might not have even been the added layer of 3 people checking the transaction.
Yeah, that UI looks like it could perhaps use some improvement, it's a bit on the 80's side (which isn't a bad thing in itself, I mean, I like my emacs, but I only use keyboard there unless I forget the bindings, which is not totally everyday).
Maybe they can re-write it in rust and add like help tooltips and some confirmation popups? Like "You're about to do this X with Y to W. Confirm?"
Oracle Flexcube is a packaged banking product. While Citibank did make the choice to purchase the product, the UI design flaw is really inherent in the product itself.
on the link they say "A rich and intuitive user experience that helps bankers enrich customer guidance,
advice and product recommendations to customers".
And to be fair to Oracle, it might be, now. And Citi is just using a twenty+ year old version of it. Because Oracle charges more for upgrades than support.
Is it? Or is it like most Oracle enterprise products (E-Business Suite, PeopleSoft, yadda yadda) – and the actual UX is bespoke constructed by an army of contractors across the world?
Wasn't actually about the UI, here's the buried lede:
> Ordinarily, paying back a loan early wouldn't be a big deal, since the parties could simply negotiate a new loan on similar terms. But in this case, some of the lenders were not on good terms with Revlon and Citibank.
That's not actually the point of this article. Many many lenders are not on good terms with their counterparties. Distressed debt and bankruptcy exist. And yet, I've never heard of another story of a bank accidentally wiring nearly a billion dollars to creditors.
Source: spent several years working in the loan and high yield space at a well-known fund.
The amount is unusually large for this type of mistake, making it newsworthy, but it isn't particularly unique. I have personally seen a couple cases of various mistakes in banking (both on customer and bank's side) causing accidental repayment of debt that otherwise would be unrecoverable as the payer became insolvent. Mistakes happen. Human processes and technical processes can help make a bit less mistakes, UX is part of that.
Obviously Citibank was at fault here for not understanding how the software was supposed to be used, but they did not write the software Flexcube, Oracle did. And if you look at some of the manuals for the software, it is all very poorly designed:
However, Citibank's real mistake was trying to use the software for something it probably was not really designed for. It seems they created a "hack" to make this type of interest-only and rollover payment possible so that they didn't have to bother trying to figure out the correct interest payments for the loan holders.
Isn't this a core design failure, rather than merely a UI failure?
<<
On Flexcube, the easiest (or perhaps only) way to execute the transaction was to enter it in the system as if paying off the loan in its entirety, but to direct the principal portion of the payment to a "wash account" (an internal Citibank account) to help ensure that money does not leave the bank.
>>
Flexcube was developed by i-Flex Solutions which was originally called Citicorp Information Technology Industries Ltd (CITIL) and was actually spun off Citibank. Citibank invested about USD 400,000 in it and though I don't have direct details of how much they paid to use it, it wouldn't be a bad guess to assume that as the original/founding investor in the firm, they probably got a very sweet deal. Oracle buying Flexcube from i-Flex happened much later.
I remember using Infosys's Finacle in 1998 (it was called Bancs2000 then). It had a TUI interface and was very snappy until they moved to a web based version and called it Finacle. We hated it because it couldn't keep up with muscle memory everyone had developed on the TUI. Flexcube came a bit later and started to gain momentum around 2005-2006.
PS: also wanted to add that most of the issues I've seen in these banking software are due to never-ending customizations demanded by the banks. The core products are usually well designed. There is basically an entire industry around core banking customizations and a lot of people working in that are basically old-school bankers who have migrated to technology (nothing wrong with that) - a good portion of them have excellent business knowledge but not much idea of UI/UX.
One of the most fascinating parts of this story, to me, is that Revlon's creditors were able to use a mistake made by Citi and a legal loophole in New York to recover debt that had increased in risk due to some allegedly-shady asset management by Revlon. It's like a perfect storm formed to repay bad debt.
Exactly. Revlon's creditors received a windfall because they were paid back 100% on a risky debt investment. Citibank, on the other hand, is now an involuntary creditor to Revlon and has taken on a lot of risk because of the mistake.
This looks like one of those delightful UIs where basically some database field are thrown onto the screen, with very little though as to the U and the I
When faced with UI as shitty as this, they could’ve (and in hindsight obviously should’ve) instituted a two step procedure where not just the principal gets put into an internal wash account, but the interest also goes to an internal account. Then the next day they can pay out from the interest only account. That way a mistake would be caught before the money leaves the bank. But hey, I don’t feel too sorry for them if they still haven’t figured out this whole banking thing after 208 years in the business.
> Ordinarily, the law would be on Citibank's side here. Under New York law, someone who sends out an erroneous wire transfer—for example, sending a payment to the wrong account—is entitled to get the money back. But the law makes an exception when a debtor accidentally wires money to a creditor. In that case, if the creditor doesn't have prior knowledge the payment was a mistake, it's free to treat it as a repayment of the loan. Judge Furman ruled that that principle applies here, even though Citibank notified its creditors of the mistake the very next day.
Key point 'wrong account' is not 'right account but for the wrong reason'.
The article most likely written by a non business person confuses a mistake (money sent to the wrong account) with money sent to the right account. The law almost certainly doesn't allow a wire transfer sent to be recovered for that reason and type of mistake. Importantly though it does (by design) allow a transfer sent to a mistake (like a typo) to be recovered. This actually makes sense and here is why:
A wire transfer is payment for 'goods or services' let's say. Often that 'goods or services' is released (or relied upon) when money is received. In the sense that it is not reversable under any and all circumstances. A wrong account is 'going to the wrong place'. The right account is the right place.
Wire transfers are by design 'final'. ACH is not. An ACH can be reversed (for a number of reasons).
If you could claw back wire transfers all sorts of things (that depend on them being final) would break down and you'd have many problems down the transaction line.
Any time I see a "dry run" option in internal tools I tread very, very carefully :) Often they don't work due to lack of exercise, or worse are not dry at all.
Which is very sad, because I think first-class support for `--dry-run` is a great way to architect a tool! Anything I write for which it's meaningful is divided into a "work out and represent in memory what I'm going to do" stage and a "do it" stage. Then `--dry-run` is a simple matter of `print` instead of `execute`.
This paid off really well for me recently, where the `execute` stage turned out not to be fit for purpose, but the statement of intent was all still correct. The rewrite didn't have to touch the "gather" stage at all. Huge rewrite becomes naturally much more limited in scope. Just another tool in the modular-design toolbox: you create a strongly-typed API boundary down the middle of the program.
You are unfairly downvoted, while giving the proper answer.
For complex actions, the UI should summarize the future result of the comment - not so little as to be useless (ex: "some of the money will go to an external account" = how much money? on which external account? only one? more?) and not much as to be a description of the underlying software process (as a debugging output would)
This requires both domain knowledge and software literacy, so it is hardly found, if ever.
Hmm, online stores can do this ("here's the product you're about to buy, the quantity and its price, shipping, total sum, here's the address it'll be shipped to, hit 'Purchase' to confirm."), but I guess no one at the bank wanted to pay for that feature on their ancient app.
You forget the first law of UI: nobody reads messages.
And the second law of UI: if the user wants to see porn^H^H^H^Hdancing bunnies, they will do everything they can until they get to the dancing bunnies.
Matt Levine's article, which has some greater detail, pointed out that there was an "Some of this money is going to an external account. Are you sure this is correct?" confirmation pop-up. The problem was they intended to actually send the interest, so there was an assumption that that's what the dialog was referring to.
What I read in parent post is that if instead of a generic message it would tell what exactly it is doing ($xxx is going to AAA and $yyy is going to BBB) then that would have helped.
I find this somewhat hilarious, but on the plus side it allows Citibank to put a $ figure on what it could cost to outsource key software infrastructure.
> To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion—would have been borderline irrational
This seems weird. There are all sorts of novel mistakes all the time, even by very competent groups. Even if it’s not a mistake, it would be a surprising action to take.
But most of all, it seems really weird to say it would be irrational to expect what happened would happen. Like people who profited off of 2008. Were they irrational or prescient? It’s impossible to tell a lot of the time, so “it would be irrational to be correct” seems like a very over strong statement.
Maybe I’m just taking offense to an irrelevant point then. Conclusion A is rational and conclusion B is too (and in this case correct). Judge says B is irrational. But all that matters is that conclusion A is rational.
I am baffled that most developers seem to see (even in banks) thousands separators as a useless gimmick, and somehow expect users to mentally count the number of digits when they are dealing with amounts in billions (or even more in JPY).
Great point. Luckily my bank interface writes out the amount in words before asking me to commit. Once while entering the amount on a mobile, I had an extra 0 by mistake. Caught it just because I had to read the amount in words before confirming.
There is no amount in the entire screenshot - the number in the "principal" field is an account number, not a monetary amount. And nobody would put thousand separators into account numbers.
Everything about banking is devoid of aesthetic. Banks don't care how things look. My nearest BoA has fake plants and dinged-up furniture. This is the kind of culture that would put up with horrendously bad UI/UX until it bit them in the ass and then they would probably just fix the problem with lawyers instead of finding or demanding a better software. Design matters.
Now let's talk about Flexcube. Oh, it's an Oracle app! Guess who else is rich, has no interest in aesthetics, and loves lawyers. I'm seeing a pattern here. Design. Matters.
If you're involved in designing critical systems (whatever critical may mean to you and your customers), I can recommend this book for a collection of case studies of where things went wrong:
This seems like the kind of thing that was originally a paper form that got coded 1-1 to avoid friction, and then decades later someone was not trained properly on how to use.
More about training. Walk into any shop, heck a title company, and you'll see UIs that have no explanations or anything just abbreviations that are understood by internal people. Look at the screen they use in Grease Monkey. There's probably people at your insurance company that still logs into a Telnet session to do things with your account.
There's more software out there like that than your Twitter UI for posting a message than you can imagine.
This isn't Citibanks UI design. This is about Oracle's core banking software Flexcube. It's way deeper than UI. The loan entity didn't support the update they needed to make so they tried delete and re-create - which involved paying off the first loan - but the counterparty didn't agree with the "re-create" bit. Take the "U" part of CRUD seriously people.
subcontractor in India named Arokia Raj... Citibank official in Delaware named Vincent Fratta
Raj probably still has a job working on non-Citibank accounts. I doubt Vincent Fratta is still employed by Citi. I guess this is fair? Raj made a mistake for one client where the client-- the experts-- signed off on it.
Human error is a terrible concept, a crutch for bad engineering. Take the story of the VSS Enterprise (SpaceShip Two) that broke up back in 2014. It broke apart because the co-pilot hit a button four seconds early[1]. But if hitting one button four seconds early is the difference between nominal flight and falling from 15 kilometers up without a pressure suit that's really the fault of the person who designed and built the thing[2].
[1]: Outside analysis, sync up a video from an earlier flight with the failed flight, sync'd up to T-0 ignition. The successful flight the co-pilot Alsbury hit the button at T+13s, the fatal flight at T+9s. The fatal flight was the first test of a new engine, so there is some error in this calculation. Wikipedia claims it was 14 seconds too early, but I'm not sure how much I trust the source it cites.
[2]: The feathering device was a clever idea that the designer, Burt Rutan, had: if you fold the entire spacecraft in half while you are up in space then it will pretty much always be in a safe attitude for reentry and you don't need to have much maneuvering authority in space. But below Mach 1.4 there isn't enough aerodynamic pressure to keep the spacecraft from folding in half when you don't want it to (and then the spacecraft gets torn apart by all of the aerodynamic pressure that is there- not enough to keep it from folding in half, but too much to survive if it does fold in half). So they put in these big honking locks to prevent it from folding when you don't want it to. But now you have a new problem: if those locks prevent it from folding while you are up in space now you burn up in the atmosphere because you don't have enough control to keep the spacecraft oriented in a safe manner. So you have to undo the locks above mach 1.4 but before you have burned enough of your rocket fuel that you are committed to going into space: if you try and undo the locks and they don't go, you immediately stop the rocket engine and glide back down to a safe landing. So you have a very narrow window that the button has to be hit, and if you are out of that window you can either accidentally feather and die or fail to feather and die. At a certain point you just have to give up on your clever ideas and admit that they are not helping you.
The next generation of software is going to include check boxes and text fields to confirm to enter data in "front", "fund", "but*" and check boxes to confirm "I am not smoking", "Yes, I understand the software cryptic codes 110%", "I acknowledge that the workflow is super dumb down" and "We all thank Oracle consultants' support for super intuitiveness". It is a question of time Oracle gets dodo dna. The one trick pony has outlived itself..
Under New York law, someone who sends out an erroneous wire transfer—for example, sending a payment to the wrong account—is entitled to get the money back.
But the law makes an exception when a debtor accidentally wires money to a creditor. In that case, if the creditor doesn't have prior knowledge the payment was a mistake, it's free to treat it as a repayment of the loan.
I wonder if folks here on HN agree with this law or not. It sounds like it could be construed as an unfair rule put in due to lobbying by lenders. Thoughts?
General case: I receive an unexpected wire transfer from someone I wouldn't expect to pay me. It would be unreasonable for me to expect that the money is fairly mine.
Exception: I loan out money to someone and ask them to pay me back no later than one year from now. Six months in I receive a wire transfer for the entire amount back. It's reasonable for me to expect that the money is mine and I'm free to spend it as I like. If now the debtor comes and asks for the money back because their wire transfer was a mistake, I may not even have the money anymore and would be put in a bad spot through no fault of my own.
> If now the debtor comes and asks for the money back because their wire transfer was a mistake...
One of the questions here was at what point in time is notice of mistake required. The defendants argued that notice was required prior to funds actually being received. Citibank argued that notice prior to the debt being discharged should be sufficient. The court held that notice should be required prior to the funds being received.
In the case of a wire transfer that happens instantaneously, that interpretation essentially eliminates the possibility of giving notice of a mistake.
I think it's reasonable. If I were lender got repayment early, found another lendee now sometime later original lendee would want the money back. I don't have it anymore and thus can't return it. Should I now be forced myself to lend the sum to cover it? Could that cost more than I make from either transaction?
Well, why just the blame on the UI unless it is a glitch ? In general, It is the blame on the process, trainings and Users not participating in the Development life cycle.
Why would someone not want to refer to some true documented procedures while processing such large transactions ? Firstly, is there such a documentation or a procedure or a checklist in place ? Documentation and process is a key to every unique tool or solution.
This is widespread at Citibank. Citibank's website will show you the correct minimum credit card payment on the summary screen, but when you click through, it will show you a different lower payment. If you only pay that one, your account will be flagged as not paying your minimum payment every month. This has been reported to Citibank multiple times over the last few years. It won't be fixed.
The genius business folks at Citi can get their $500,000,000 out of the money they saved outsourcing to India.
I've always had a great experience with my onshore and offshore Indian colleagues, I have family of east Indian origin. But all of the offshore call centers and IT should be banned or hit hard with high penalties by the US government given the COVID pandemic.
One of the major detractors to Bitcoin et. al. That you hear about on HN is that transactions in USD has a legal recourse to recoup funds in these types of oopsie situations. Does this court decision make anyone feel more bullish about Bitcoin seeing that there isn’t always a legal recourse in USD? Does it at least temper that discussion slightly?
I wonder if that weird rule if you accidentally send the wrong amount to a creditor comes from credit card industry lobbying. Say if a consumer accidentally sends $10,000 to his credit card company instead of $1,000. Consumer can’t claw back the money. Citibank has probably benefited from the law.
Right, this exception seems predatory; and I think it should only apply if a loan is delinquent. That seems to be the intention of the law, but it's probably broad thanks to a lobbyist.
It's worth noting that in many (perhaps most?) cases of such payments "creditor" does not imply a loan, but rather debt caused by unpaid obligations for goods or services - e.g. there's a shipment of coal or iPhones and there's an invoice that's due to be paid by date X, and it gets paid on date Y.
In many cases Y<X - perhaps there's some discount or other negotiations involved, perhaps some people or companies don't leave payment to the absolute last day.
This clause means that if someone is paid for their goods or services, the debtor can't come back afterwards and say "ahhh, this was a mistake, give us back our money, we weren't delinquent and still had 20 days to pay it, we intended to use that cash for other things like payroll and we pinky-swear that we won't go insolvent during that time" - if you got paid, you got paid, period.
Even if this exception to the law didn't exist which it doesn't in many states. I am sure no judge is going to let you float your company by claiming it was a loan payment a "mistake". You need to prove it was mistake like Citibank did and show that you attempted to immediately fix the mistake like Citibank did when they noticed the next day.
> "To believe that Citibank, one of the most sophisticated financial institutions in the world, had made a mistake that had never happened before, to the tune of nearly $1 billion—would have been borderline irrational,"
Really? People make mistakes. Institutions make mistakes.
There is a big issue with UX with Oracle products, I have been telling it for ages, no changes. Even Thomas Kurian said “I want to make sure that the entire HCM dev organization understands what a disgrace your UI [user interface] is and stop living in denial on that.”
I guess they need some tectonic shift to really to start listening to the customers. Another example, I get long list of options in a LOV and guess what, it is not sorted! I have to go through each value until I find the needed one.
That UI looks like a very old version of the software (kind of like when you still had stuff like Oracle Forms or whatever it was called). Oracle UIs (since they launched Fusion) look different (I think they are now all web-based)
Why so surprised? 80% of 'enterprise software' is similarly terrible. Nobody involved in the business of creating this crap gives two shits about users or quality except for a few developers that nobody listens to.
Their UI sucks! Poorly designed form, no tooltip explaining the fields, bad contrast ratio, and a button with just an icon. A bank will spend $$$$$ on consumer facing UI, but spare almost no thought on UI used by employees.
But this is definitely not as bad as some of the other UI I've seen in SAP, Oracle, etc. It's certain that nobody who ever uses this UI actually approved this UI. Even if you do use it, hate it, you have no way of asking for a change.
Shameless plug, I run an open source project to build internal and enterprise apps. We've built it so you can set up warnings and confirmations to prevent such errors.
Our pitch: Use Appsmith and save at least $500M https://github.com/appsmithorg/appsmith
With our local bank, a developer made a mistake and some amount went into wrong customers. They made him to visit all those companies, ask them to return money. Eventually they forced him to pay the transaction commission.
Wow, what an interesting piece of UI to manage such a critical piece of financial process. Banks are probably one of the worst offenders of UI and UX design, and half the times, I can't even reason why.
I truly wonder if this chsnge at least something in the industry. Like, you know, it is loud news, and huge numbers. Will the competitors look into their systems, involve some ux'ers to sleep calm?
Oracle Forms ftw. so much software built just to make it run, without any UX design put to work. well.. there's so much more of these forms running worldwide, so - just get used to it..
There's these web browser extensions that let you disable JavaScript for that reason.
Websites can do way more than just hijack your browser button if your browser just runs any and all JS it receives. See for example https://theannoyingsite.com/ but at your own risk.
My takeaway is that by now every company is a "tech company" whether they want to be or not, and needs a bench of in-house software engineers, NOT contractors.
As a product designer who works in the enterprise space, it's good to know I'll have a long career barring any personal failures due to programs like this.
This UI resembles a register transfer language; it's what programming in microcode looks like. It doesn't even rise to the abstraction level of assembly language. Expecting humans not to make errors at this level of abstraction with high consequences for failure is moronic. It's sad that the only motivation for improving outsourced lowest-bidder enterprise crapware like this is when it costs the company $$$.
Inept judge, or certainly one not steeped in the ways of finance. People make wire errors all the time. Fraud occurs, things happen. I haven’t read the statute but as written the exception doesn’t even apply (Citibank being an agent and not the lender.)
From the 105 page opinion [0], here is the relevant statement of law:
> A creditor of another or one having a lien on another’s property who has received from a third person any benefit in discharge of the debt or lien, is under no duty to make restitution therefor, although the discharge was given by mistake of the transferor as to his interests or duties, if the transferee made no misrepresentation and did not have notice of the transferor’s mistake. RESTATEMENT (FIRST) OF RESTITUTION § 14(1) (Am. Law Inst. 1937).
(Note: The ALI Restatements are not law per se. This section, however, was explicitly adopted by the New York Court of Appeals in a prior case. Here, the federal court applies this law because it is the relevant law of New York.)
I know they’ve outsourced customer service agents to India but didn’t think they also outsourced the execution of billion dollar transfers. Hope the wage arbitrage was worth it boys!
How could they have trusted some Indian subcontractor to do something so important with their money? Guess it's also a lesson in "you get what you pay for".
A lot of finance software is really subpar. I'm a tiny tiny bit eager to know how the defi crowd will do there, they come with a different culture and might find cute ipodsy ways to design financial UIs.
The CITI event was a real disaster, likely enough for the importance of UI design to be taken more seriously around the world. IMHO, there are some quite good contributions in this thread with several excellent ones.
I will chip in my 2 cents from my experience on both sides of user interfaces:
In parts of user interface (UI) design, there has long been a principle that to make good use of the UI the user should have accumulated experience with the UI where they ran essentially experiments to discover how the UI and associated system worked.
So, with this principle, the CITI people just needed more experience with experiments, at ~$900 million per experiment!
So, right, the principle is flawed and for some applications expensive/dangerous.
While this principle of users getting experience from experiments has worked for the UIs of a range of applications, to be careful a principle should be that a user should be able to use an application as intended the first time and with no experiments.
Case 1. Last week I went to the Web site of my bank and used its UI to transfer some money from one account to another. The experience was Excedrin headache #394,325,757,110: I entered the data, and nothing happened.
So, I had to start running experiments. I hit Enter and left/right single/double clicked on everything in the window, and nothing happened. It appeared I was seeing all of the window horizontally, so I checked vertically, and to do that I looked for a vertical scroll bars. There were none. So, I converted the application to full screen and still saw no vertical scroll bars or way to make the money transfer happen. So, I did some more study and finally discovered that the window was so tall that the bottom of the screen was hidden by my tilted keyboard, and the part I could not see had the HTML push button for approving the transfer.
So, the UI designers just assumed, insisted, never told anyone, that, naturally, of course, all the users will be using their Web site with the windows expanded to full screen. And for some reason, whether their window is full screen or not, the UI designers don't like vertical scroll bars. The screen for the transfer had only a few lines of text and didn't need so much vertical screen space, but the designers seem to like using as much screen area as they can.
Okay, I learned how to use their UI. Still not so good: (a) The bank keeps changing their UI, and in this case made it worse. So, with that bank I will have to go through such mud wrestling nonsense several times a year. (b) To me, the original HTML controls were nicely designed, including the scroll bars, and one result was that nearly all the many millions of Web pages worked somewhat the same. Then somehow many UI designers wanted their screens to work in some unique way until they changed to another unique way a few months later. The power of JavaScript made this problem much worse.
Case 2. Also last week, in a weak moment, I decided to get a user id and password (PW) at one of the social media Web sites. They stated some rules for passwords; I followed those and typed in a password in a file where I keep such things; copied that PW to the system clipboard,
pasted it into their HTML single line text box for passwords, and nothing happened. I hit Enter, clicked around, etc. and nothing happened. I pasted the password in again, reloaded the page, closed the browser and tried again, etc. and nothing happened -- no messages, nothing. I still don't know what is wrong except it isn't me.
As I've mentioned at Hacker News before, for my startup I have 100,000 lines of typing of software for a Web site ready to go live. In that code I used eight design principles in UI: (i) No icons. Instead all the links are words in the English language that clearly describe the function of the link. For icons, I usually am not sure what they mean, can't look them up in a dictionary, and can't pronounce them, spell them, or type them -- IMHO, bummer. (ii) There are no acronyms. None. (iii) The only controls used are standard HTML. I wrote no JavaScript at all although Microsoft's ASP.NET wrote a little for me; apparently it has to do with cursor positioning but is optional. (iv) Each page (screen) has a link "Help" to explain the screen in detail. (v) Each screen has both vertical and horizontal scroll bars. (vi) The screens are designed for a window 700 pixels wide and are still usable in a screen 300 pixels wide. (vii) All the fonts are large and have high contrast. (viii) All HTML control bounding boxes are bold and black with high contrast. IMHO (i) - (viii) help UI design.
You are dealing with wire-transfers here. So there isn't really a way to undo a wire transfer. The recipient would need to voluntarily create another wire transfer back to you of the same amount.
So an interface like this only controls the money on Citi's end. Once the wire transfer occurs, there wouldn't be any way for this software to undo that transaction.
You could create a psuedo-undo in this software tool, which is how Gmail creates an "undo send" feature. Like a wire-transfer, once you send an email, there is no way to undo it. The email is sent to the recipients servers, there is no way to reverse that. Gmail creates a Psuedo-undo which means that they basically wait after you click "send" and they don't send the email right away. They wait 30 seconds or so before actually sending the email. So if you click "undo" during that artificial waiting period, then it simply cancels the send that hasn't occured yet. It feels like "undo" to the sender, but in reality an artificial waiting period was created, which allows you time to cancel it before sending. Not a proper "undo", but it acts similarly. But it is worth noting that once that artificial waiting period is up and the email is actually sent, then there is no undo anymore.
In the case of this story, you could potentially create a psuedo-undo. The software could create an artificial waiting period before initiating the wire transfer, which would allow time to undo. But the problem is that no one would have noticed it until the wire transfer was completed anyway. Someone noticed that $900M left the account when they thought money was staying in the bank. Only then did they realize that the mistake was made. Three people signed off on this transaction before it was sent and all three thought it was correct. So a psuedo-undo wouldn't have helped in this case.
I thought so too, but the Matt Levine article says the debt was trading at 42c in the dollar, so likely people didn't expect the debt to be repaid. This makes the loss $300mm instead of $500mm, but is worse than your (and mine) initial take.
3 people were required to okay the transaction and none of them saw anything wrong with it. This is not just bad UI but bad user trainings as well. Mistakes happen and in systems as complex as these with very few visual cues this was an accident waiting to happen and when it did, it was an expensive one.
I was moaning when i had to use SAP for a little while
I don't even have anything to add. The paragraph speaks for itself... You can't make this up