Microsoft had run huge ads on how LSE was using .NET and Windows in critical financial applications. Within a year LSE had suffered crashes in the same critical areas.
In fairness, though, it may not have been MS' fault - crappy programming is crappy, no matter the underlying OS. And Linux fanboys should wait to see if the new system has roll-out issues...
I agree. At these speeds it has less to with OS and/or language and has more to do with smart programming decisions. PS - Singapore Stock Exchange quotes a faster latency than Nasdaq and LSE:
Can you elaborate on this? Are they developing a fork of the Linux kernel? A new Linux distro? Or could it be that they are developing the trading "platform", using a commodity Linux distro?
World record? Not quite. Let's see full details on the numbers. NASDAQ publishes their latency numbers weekly, including average, 99th percentile, and 99.9th percentile: http://www.nasdaqtrader.com/trader.aspx?id=inet. Their average latency on order entry is 98 microseconds and their average latency on market data is 92 microseconds. Perhaps by "world record" they mean "european record".
Go read the website I linked to. NASDAQ publishes their methodology. I'll quote it for you here:
OUCH Measurements of Order Latency
* The time it takes to accept, process, and
acknowledge or fill an order.
* Measured as the complete round trip for an order
or cancel transaction leaving a customer network
port, through the co-lo network and application
firewalls to the matching engine and back.
* Measured in the production environment during a
full trading day including opening and close.
As you can see, they are including the time it takes to acknowledge and fill the order. It is the complete round trip time, and does include transit latency through the co-located network (which I thought was excluded).
They are clustered together because it takes roughly the same amount of time to determine that an order can be matched (filled) or that it can't be matched (acknowledged, on the book.) Either way, it has to be processed by the matching engine.
True but this is like the "World Series" for stock trading latency. Who really plays baseball other than the US and small group of other countries like Cuba ;-)
If I were a critical systems operator like say, a stock exchange, I would have to have my head examined by a team of head examiners for even considering Windows as an acceptable OS platform.
Honestly, is the reason Microsoft can even get consideration for these projects because there are so many tech executives at major corporations who used to be Microsofties?
Have you experience with Windows Server in a properly set up production environment? I know of at least one high-performance shop that has used Windows servers (we are not talking onesies or twosies here--more than hundreds) and has higher performance requirements than exchanges.
That makes as much sense as blaming US developers for Accenture failing at the LSE and claiming that Asian programmers are the best because they made the LSE work.
Accenture developed .NET software for the LSE and a company in Sri Lanka made the new LSE software for Linux
It's fascinating that these latencies are an order of magnitude faster than most 'web services'. Can anyone provide more insight into what is specifically involved in executing a single trade? Are they just enqueueing a trade message in this latency or actually executing the trade? Are they including network latency in these numbers? Also, does anyone have any insights into the hardware and customizations?
I'm a software developer at an exchange in the US. The LSE has not released details of how it is measuring "latency", but FWIW, we measure latency as the time it takes to submit an order and receive an order acknowledgment or fill, with both the sending and receiving outside our firewall. The two biggest SW components involved are the FIX handler (the component used to process orders; FIX is an industry standard protocol) and the matching engine (the component used to match buy and sell orders). I could of course spend hours talking about our hardware and software customizations, but it's a very competitive industry. Our software is C++ running on Linux, and that's about as much detail as I'm willing to go into.
"we measure latency as the time it takes to submit an order and receive an order acknowledgment or fill, with both the sending and receiving outside our firewall." This isn't normally the metric exchanges are referring to when they say 'latency'. They mean the internal processing time, within their network, to process, fill and generate a response to an incoming order. Anything else depends on the individual clients connection, some may be co-located, some may not etc etc... exactly as lrm242 describes.
I thought what lrm242 said and what I said were pretty much the same thing. But there is no standard definition of 'latency', so it's hard to say what "they mean" unless they publish a definition on their website. But I know of at least three major exchanges (one of which I work for) that consider the starting and endpoints for latency measurements to be outside the firewall. As for what Turquoise means, I'm not sure but would love to see their definition to know if their numbers are comparable. Also would like to know if these measurements were made under load because that can also make a significant difference.
Right, the only metric I want to see is the roundtrip latency between when I submit my order to when I get my execution report. I just aggregate the timestamps in my FIX logs and usually quote the min, median, max when trying to compare exchanges.
Every exchange we negotiate with quotes us their ping-test results. I always have to remind them that there's a lot more to getting an order into the book (especially via FIX) than just TCP.
The latencies are typically quoted as round trip through the matching engine. This does not include transit latency to the matching engine but will include any internal transit latency once the matching engine has the order. So you can assume it is from when the order message reaches the network interface on the order processor to when the response leaves the network interface towards the market participant.
The amount of work the matching engine has to do is non-trivial but straightforward. Take order, look at type, if market order look for a match on the limit order book, if not handle appropriately (this bit is not included in the latency as it might involve routing to other venues, etc). If it is a limit order it must find the right place to position that order within the order book. Building an order book data structure that is cache friendly and thus very fast is not rocket science but it certainly isn't easy.
I agree, but with one exception. I think it is measured from the point of view of a co-located trading firm. So from the time that a trading engine that is a customer of the exchange fires an order to the time it receives either an order acknowledgement (this would result in a resting order) or a fill (completion of a by sell) at the co-located machine is how this would be measured.
That is certainly hard to say. It was, as I understand it, built by avanade, the joint venture between Microsoft and Accenture.
The speed of a system as complex as an exchange is influenced by many factors--network infrastructure, language of choice, software architecture, OS tuning.
In fact, garbage collected languages are known to perform well in serious low-latency systems.
Some firms have been known to rewrite the network stack for performance.
So presuming that it is an OS issue, or a language issue, or a vendor issue alone is not likely to arrive at a useful answer.
I would be really surprised if it was the fault of the OS. I mean, Windows can be really fast, in the right hands, especially since they've introduced HTTP.SYS (parts of the HTTP server now run in kernel level). However, .NET being a garbage-collected, managed and JITted environemnt, it is maybe not the best tool for achieving these kind of response times reliably. Any serious and competent .NET developer would admit that.
In general, I would blame the incompetence of Accenture for the failure: bad decisions, bad architecture, bad management, using the wrong tools, crappy code.
.Net and newer windows have some really cool real-time features -- like deterministic garbage collection options, various per-thread priorities for scheduling, preemption and so on, making it a decent real-time platform. (for some definitions of real-time).
I don't know why it didn't work out for Red Hat but it works quite well on Windows. And honestly I don't know whether they use(d) HTTP or not, but given that HTTP is dead simple and as lightweight as a protocol could get, maybe it wouldn't be such a bad idea to use it for exchanging messages in this scenario.
I'm sure they will notice much more benefits than just improved trading times.
For example: improved infraestructure management, overall performance increase, fast patches for security bugs, and a long etcetera.
Of course, as JoachimSchipper says, if the applications on top of the OS ar crap, a different OS won't solve the problem, but to have a good underlying OS is a good start.
This makes me wonder how, in mission critical and real time systems like LSE, they maintain a balance between being ACID and fast. Does any of these transactions ever touch the disk ? If not, how do they handle machine failures ?
No, the core of a stock exchange resides entirely in RAM. Everything is reset once a day (overnight for stocks, immediately after regular hours for futures), and brokerages are responsible for resubmitting long-lived "good 'til cancelled" orders each day before the session begins.
Despite being RAM bound it's easy to parallelize because each stock is its own isolated exchange, so they can be distributed across hardware in proportion to the average volume in each stock. In other words, it's localized but embarrassingly parallel.
The only traditional databases are for reporting purposes e.g. who traded with whom and are append-only as far as the core exchange code is concerned. The reporting database is coupled through the same firehose feed that you can get as an exchange member, albeit with more redundancy. There are also access control systems that only come into play when you first open a session, and lots of other moving parts each with their own task. For example, if you lose your connection to the exchange you can ask a certain (non-core) server to replay all the events that happened from a given point in time in order to catch up. These, too, would be listening to the main event stream and spooling to disk, but the central exchange processes are RAM-only.
How do they handle machine failures? That's harder to speculate on from the outside, but if I were them I'd be running the same exchange in parallel on 2-3 machines. I don't know how it could be done without serializing the incoming order flow to make sure that each machine sees the same order of events, but it's not impossible.
I think systems like this use a lot of RAM. You can afford it when you're a stock exchange. You can get high availability from a RAM store if you replicate it over multiple machines and react quickly to machine failures. Then you keep a commit log on disk that's append only and avoids seeking.
Encouraging ever faster trading times might be unwise, leading to instability and "hunting". There needs to be some damping in the system to guard against large uncontrolled fluctuations.
The profit motive should drive the use, not the implementation of a trading system that effects the world economy so directly.
I've maintained that the standard increment for trades should be hours or days, not microseconds. This way liquidity is maintained on a macro scale, and parasitic orders based on micro manipulations of the system are eliminated.
Well it's a complicated system. Whilst individual traders might benefit from faster trading if you consider the health of the system as a whole, which favours steady predictable controlled growth, faster trading may introduce instabilities which result in reduced profitability.
They replaced one software package (a custom solution by Accenture) with another (a company they bought, as an aside, for a solution that they now sell: Ergo, take their claims with a grain of salt).
Drawing assumptions about underlying systems is speculative and somewhat ignorant.
Microsoft had run huge ads on how LSE was using .NET and Windows in critical financial applications. Within a year LSE had suffered crashes in the same critical areas.