This is not a battle between cipherpunks and the NSA. It's a battle between BitTorrent and cable Internet providers.
I assert without evidence or significant analysis, but with some past experience: the products WaPo is talking about are used almost exclusively for two purposes:
* To enable ISPs to build "fast-path" offerings for "premium" Internet access for streaming media.
* To ratchet down the incredibly painful impact that BitTorrent and other "always-on" file sharing applications are having on ISP networks.
The irony is, the high-end products WaPo is talking about are the least applicable to NSA-style spying. The code handling the packets that AT&T shunted to NSA is crappy vanilla pcap; any of us could write it.
Doesn't the ISP open themselves up to more lawsuits from the RIAA and the like when they start inspecting traffic because they are no longer just pipe?
The leaders in the space are Sandvine and Cisco, and Arbor (my old employer) just bought their way in by nabbing Ellacoya.
The answer in the general case is, "stuff that looks substantially like libpcap". It's all written in C, surprise!
On the very high end, or in very specialized cases, you'll find:
* FPGA regexers, OEM'd from a couple common vendors, that compile DFAs into gates.
* Multicore "network processors" with MIPS cores and fast custom memory busses.
* Blades that connect into the backplanes of (ubiquitous) Cat6k switches to intercept traffic.
By and large, this is a systems design problem, not an algorithms challenge. To appreciate that, you have to get the context for this WaPo story, which includes the fact that the story is totally overblown and ISPs are not reading your email or web traffic, but rather trying to figure out how to commercialize a "fast-path" network product for bulk P2P/streaming media customers.
edit: also, packet classification is quite cool from research perspective. for example, check out the following : "packet classification on multiple fields, gupta, mckewon". also, network algorithmics devotes an entire chapter on various strategies for efficient packet-classfication.
Network Algorithmics is an awesome book; it's been recommended here before.
But I don't think most of the DPI products use advanced packet classification algorithms. No product I've worked on has; it's pretty much, "that's port 80, so use the HTTP decode". There's classification done for binning and accounting, but it's pretty brute force.
Narus is a DPI vendor, but not of the type WaPo is talking about. They provide "lawful" (read, "unlawful") intercept for traffic that has already been classified and diverted.
I assert without evidence or significant analysis, but with some past experience: the products WaPo is talking about are used almost exclusively for two purposes:
* To enable ISPs to build "fast-path" offerings for "premium" Internet access for streaming media.
* To ratchet down the incredibly painful impact that BitTorrent and other "always-on" file sharing applications are having on ISP networks.
The irony is, the high-end products WaPo is talking about are the least applicable to NSA-style spying. The code handling the packets that AT&T shunted to NSA is crappy vanilla pcap; any of us could write it.