Tcpdump is amazing. The quickness that you can have with it over Wireshark is awesome. I love that it is a command line tool instead of a GUI tool, since I needed to analyze TCP packets for quick debugging purposes.
But for the people who are new(ish) to tcpdump have you heard of libnet and libpcap? You can basically build your own tcpdump! :D
I was amazed at the speed packets fire when you program it yourself in C.
Wireshark also comes with "tshark" which is a CLI tool that gives you access to display filters and some of the GUI features. In particular, it's great for scripting if you're trying to pull some stats while troubleshooting.
Tcpdump is cool, tshark is cool, but nothing beats raw libpcap for grabbing raw data off the wire. It is simply so stable at high throughput it's often the only option.
Wireshark starts to break down at a certain point. When that happens, I've found scapy (Python pcap parser) very useful. Once that breaks down, a good option reading the hex directly in C.
Got to add dpkt to this list for parsing pcap streams - allows you to write some awesome python analysis tools - e.g. For this 5GB packet capture find me all the streams where tcp packets were retransmitted more than twice.
I use tshark for capturing a lot, it's great. One of the big advantages for me is that it allows you to use the more expressive (and familiar) wireshark filter syntax, e.g.
tshark -i eth0 -Y ip.addr==8.8.8.8
Instead of having to remember tcpdump's own fiddly syntax. Given that most of my protocol debugging work is done inside Wireshark, I'm much more fluent in that filter language.
Note that the above snippet is a display filter, so it's capturing all the packets on the wire, doing a full dissection, and then running them through a filter before printing them; if you're trying to capture on a saturated 1G link you might need to fall back to the capture filters which are simpler and faster (and I believe also use tcpdump's syntax).
Wireshark syntax is also fiddly. It's just more familiar for some folks. The first several years of my experience troubleshooting network problems happened when Wireshark didn't exist, but tcpdump did. I still have better recollection and comprehension of tcpdump syntax than Wireshark. I have to read the docs to use either these days (since I don't work at that layer of the network much anymore), but I have to read longer to make Wireshark do what I want.
Sometimes "ease of use" is really just "what I'm used to". (Not saying Wireshark isn't more expressive or more powerful...it probably is. But, I usually have a very basic use case, and tcpdump can do it with a couple of flags and an address or port.)
If you use VM's for this sort of thing, having images laying around like: https://security-onion-solutions.github.io/security-onion/ gives you most of the tools you'd need (wireshark, scapy, etc. in one place). Seems to be the best way to go for me (Mac laptop user).
> I was amazed at the speed packets fire when you program it yourself in C.
For sure. There's no better way of learning the protocols than writing your own network sniffer. Even writing a crappy one like the one that a few folks and I put together in college will teach you a lot about how each protocol behaves and what to look for: https://github.com/carlosonunez/nbfm-sniffer
>> Also if you understand how to reason about the overhead of using tcpdump ("below 2 MB/s is always ok"?), I would REALLY REALLY LOVE TO KNOW. Please tell me.
I think when Dick Sites from Google says he has a rule of "stay below 1% of overhead" when analyzing traffic on datacenter nodes he wants you to select the right tool for the job. In the context of tcpdump you can run it with so many options that makes it very powerful. But that power is dangerous in the hand of a novice user. A simple error in how you run it (maybe missing filters or too wide an address space) can cause you to shoot well above the theoretical 1% limit. But that's not the tools fault IMO.
Anyway this seems more theoretical, because in practice I'd prefer a hardware based network tap and analyze that without creating any risk to the live traffic
Using something like "time tcpdump -c 1000 -w file.pcap ..." which stops after 1000 packets is always a good idea. It helps catch a too greedy BPF expression (PCAP filter), before using to many resources. Also always output to a file first, and then you can check the output afterwards with "tcpdump -nr file.pcap | less".
However, one of the worst things that tcpdump does is to put the NIC into promisc mode. On a physical NIC, this can be VERY expensive and may involve bouncing the link (behind your back) and dropping packets. At the very least, it can wreak havoc with steering filters on some NICs. To prevent this, use the -p option to prevent tcpdump from putting the NIC into promisc mode.
Another issue with tcpdump on an endstation is caused by stateless offloads like checksum offload and offloads like TSO on the send side, and GRO / LRO on the receive side. Because the BPF filters are applied between the network stack and the device driver, you may noticed tcpdump / wireshark complaining about bad checksums on transmit -- this is likely due to checksum offload. And you may see gigantic (way larger than MTU) sized frames. This is due to GRO/LRO on receive, and TSO on transmit. You can disable stateless offloads (ethtool -K on linux, ifconfig on bsd), but that will slow the entire system down.
> Anyway this seems more theoretical, because in practice I'd prefer a hardware based network tap and analyze that without creating any risk to the live traffic
I'll ask amazon to install it when I need it...? They probably won't mind.
Unless I misunderstood her ... I think the argument she makes is analyzing traffic in a datacenter and she talks about analyzing network traffic on a scale. Also she mentions Dick Sites who in his talk mentions datacenter technologies and not how to debug as an end user in some cloud.
Sure. I'm just pointing out that it's becoming increasingly rare to be able to replace a given use case of tcpdump with a physical tap, so you can't always sidestep the nastiness of calculating overhead—especially on a live system.
thinking about it further I wouldn't want even a 1% fluctuation caused by analysis since it distorts the data captured in statistics. And as a datacenter guy who hasn't written the software I might not know the sensitivity level that some of my software is depending upon (timeout, max-latency, etc) which could spin it well above 1% not because tcpdump but because of a domino effect.
So if it would be my datacenter I'd have a rule of no analysis of non-tap data.
A hardware tap becomes more and more meaningless in modern SDN networks behind dynamic virtualisation platforms.
Sure, you can find out where your VM is currently physically hosted, then tap the 4x10Gbit that come out of that hypervisor. Your network stream is probably somewhere in there.
But it is mixed with a lot of other data, and you could miss the bits that go directly to other VMs on the same hypervisor. Also it is encapsulated, split up in multiple streams, multiple virtual networks etc. If the VMs is moved (for load-balancing or maintenance or whatever reason) your tap becomes useless and you have to chase your VM.
This is a ridiculous rule, you are cutting off your nose to spite your face. The saying "perfect is the enemy of good" comes to mind.
To your above point
>maybe missing filters or too wide an address space
There is merit in just capturing a shitload more than you need (provided of course, you're not trying to cap a full 10Gbit) because it is often the filters that are the cause of any "performance" issues, however you define that.
the rule depends on context which I failed to mention. I agree with you in context of non critical infrastructure. Though I'm not talking about web apps in AWS but places where you still want to have your own datacenter like in a banking or telecoms, or insurance or medical environments.
EDIT: whether you allow any access to a node for whatever purpose other than the software that was meant to run on that node would probably depend on what damage is done if that node goes down. If the damage is a blip in statistic and you can live with that fine but that's not always the case
You're still overthinking things and dealing in hypotheticals it sounds like to me.
Almost the only time you would want a physical tap is if you need a permanent tap capturing everything over a long term - often for the purpose of running through an IDS/IPS, an even then SPAN/RSPAN/ERSPAN works pretty well.
Even in those industries you mention, most of the other time you are doing a capture is to troubleshoot something, so you don't need to run things for a long time, nor does any "perf issue", which is overstated, probably matter, since things are possibly already half way to fucked. And the compliance argument doesn't hold either vis-a-vis installing tcpdump - your processes and policies would be written as such to allow for debugging (or should be)
If you had 100 computers, <1% overhead could mean "capture every packet going over a 1 Gbit network interface going to only 1 computer, even if it has a 100% overhead on that computer".
As a network engineer, tcpdump and friends are on my list of most frequently used applications.
If you're not a network engineer, you'd be amazed at how many times we get issues escalated to us exclaiming that "it's the firewall" and demanding that we fix it.
It's usually not the firewall, though, and unfortunately it falls on us to prove that that's the case. It's not always easy but, luckily, tcpdump and friends allow me to show that and I can punt the issues back to where they came from.
(Several years ago, I was able to prove it was a customer's on-premise firewall -- managed by them -- and not our firewall based upon the packet timestamps and this little thing known as "the speed of light".)
I'm not a network engineer and oftentimes have to use tcpdump and mtr on behalf of network engineers combined with Chrome devtools to show application developers that there's nothing wrong with AWS or even the internal enterprise network and that it's closer to them. The hard part is trying to make these diagnostic workflows scale for tens of thousands of application developers when you have maybe a handful of people that can help them in this manner.
No, BPF is the extremely simple and limited virtual machine. The optimized compiler is in libpcap and does some neat stuff to eliminate redundant code. There are implementations of the pcap filtering language that do not compile to BPF, and there are other languages compiling to BPF which do not benefit from the compiler in libpcap.
It's worth noting that optimizer is also absolutely necessary since the design of the pcap language is such that very simple filters are easy, anything even remotely complicated becomes very verbose and repetitive.
I'm pretty sure the three things (BPF, tcpdump, and the pcap compiler) were developed in tandem, or, at least, the latter two were.
I had a job, about 15 years ago, hacking on a customized version of libpcap (mostly to do filter merges). There is a surprising amount of stuff going on there.
Indeed. There's a great talk about those early days at https://www.youtube.com/watch?v=XHlqIqPvKw8 ; I'd recommend anyone who is interested in the design and implementation of little languages to watch that. Few things will make a language designer throw out everything and restart from scratch, but I guess having Van Jacobson dismiss the initial design as unusable would do it!
But after that common starting point they quickly developed lives of their own, and should not be treated as a single unit. You'll get things like pflua as a completely distinct implementation of the filtering language compiling to lua instead of BPF, pfmatch with language extensions that are not really compilable to BPF, seccomp and other uses of BPF in the Linux kernel for things that have nothing at all to do with packet processing, a (e)BPF code generator backend in LLVM, and so on.
Yes it is. Going from 2 to 10 registers and getting map data structures and compilation to machine code doesn't change that BPF is intentionally simple and even disallows loops. "simple and limited" is a good thing.
It's simple, sure. And it's limited, in that there are sane limits in place, because you don't want an application running in the kernel to loop for forever, but 'extremely simple and limited' I don't think is fair.
You can do a hell of a lot with BPF, and it's not like work to extend it's functionality is slowing down, either. We've had tons of features implemented in the 4.x family, stack walking is likely to be implemented in the future, etc.
When you have people like Alexei Starovoitov and Brendan Gregg saying it can do all sorts of things including "crazy stuff", I think we've moved beyond 'extremely simple and limited'
You're reading a negativity into that phrase that isn't really there. It's harshly limited, but in a careful way. Despite having few parts and not being turing complete it has a massive number of use cases.
Actually, you can decode a stream as HTTP traffic in Wireshark. It assembles related TCP packets and puts together the conversation from the viewpoint of the application layer.
Nitpicky, but: this isn't MITM, this is the client side of the connection writing out the keys necessary to decrypt the wire traffic. This is important because the process doesn't change alter the network data in any way, whereas a MITM proxy would.
You can use netsh on Windows without need to install anything external (beside capture file viewer - Microsoft's Message Analyzer - but that can be done on your workstation rather than servers)
What's really irritating about the windows stack is how difficult it is to dump local host traffic compared to the same problem on Linux. I believe it's because the loopback interface isn't as completely implemented.
I guess the most obvious thing is what it not has, which is a massive GUI. It is much faster to work with when you know what you are looking for in my opinion. In most cases running wireshark without tcpdump first is really inadequate.
My usage or Wireshark is rather sporadic, so I appreciate the traffic drill down I can do w/o any knowledge in advance about the protocols I have captured.
Wireshark is great, but it's fundamentally an interactive tool; you fire it up when you have a problem to look into and close it down afterwards. Running wireshark/tcpdump all the time seems like it would generate too much data. I'm not sure how well Wireshark handles dumps larger than available RAM, either.
The author mentions the pcap filter language, but one of the misfeatures of Wireshark is that it has a different filter language for the GUI filter box.
The capture filters and display filters are interfaces to two sort of unrelated pieces. Maybe the misfeature is that wireshark doesn't do good job of communicating that.
BTW, If you want constant monitoring for your applications, you can buy a shark appliance that you can always go back to to investigate issues.
There are commercial products that you can leave running all the time to generate data from your packets on the fly, such as ExtraHop (http://extrahop.com). There are also continuous PCAP tools, but they need massive amounts of storage and in larger environments the lookback you get is limited.
Some Linux distros (Debian family) separate tshark from wrireshark which is great because that means you can install tshark without X-Windowsetc... I wish the other predominant distro family (Fedora) did the same.
I have not used tcpdump a lot, but I believe there is nothing it can do that tshark cannot.
Tshark can capture only the portion of traffic you want via filters, which can reduce the size of your pcap files considerably, and possibly haves less of an impact on performance.
When using tshark, make sure you capture the traffics to a file too, so you can go back to look at something that happened x seconds or minutes ago.
I understand the security concerns, but forms being able to debug with tshark (or an equivalent tool) is a very good reason for not using HTTPS internally.
> I understand the security concerns, but forms being able to debug with tshark (or an equivalent tool) is a very good reason for not using HTTPS internally.
One of the nice things about HTTPS is that it's just a SSL connection and the HTTP goes over it. It's not hard to use socat to create something which listens to TCP connections on one port, and wraps it in a SSL layer and sends that on to another host:port. This allows you to remove the S from HTTPS, and you then speak HTTP to it, and hence can use tcpdump (etc) to dump.
Tcpdump use the BPF syntax to filter packages. In the current
linux kernel,BPF was implemented and extended as a kernel virtual machine,when cooperated with the perf module,they can be used for collect trace info of the system. see https://lwn.net/Articles/599755/
This is useful for troubleshooting outbound requests that your backends are making. I've had the interesting logic explained to me but can't remember the details.
It's good to have things like that in your toolbox, but it's probably also a good idea to figure out how they work and what their limitations are going to be, otherwise you'll may hit situations where they don't work and you don't understand why.
So, start with the magik number. If you look up those 8-bit ASCII codes you'll see that it spells out GET followed by a space, which should give a clue as to how it's working. So it will capture a lot of HTTP requests, but it may not be getting them all.
Aren't tcpdump and wireshark some pretty interfaces to libpcap? Also linking the home page of tcpdump/libpcap [1], and check the documentation about the packet capturing.
They're intelligent interfaces to libpcap. If you have a pretty interface, you get nice opaque data you'll have to understand yourself. With intelligent interface you can do high-level processing like "I only want to see GET requests".
But they're still fairly stateless... without some extra scripting you cannot do a query for "TCP connections with more than 3 restarts".
What I meant with pretty, is pretty for human eyes, unless you are able to read TCP/IP headers ( for example ) directly in hexa.
Regarding ther other topic, nothing stops someone to capture the packets to a generic DB ( sqlite or berkleydb ) to operates such queries. Looks like a weekend project!
I started really using tcpdump when I discovered "-X" switch. It displays all the traffic as it happens in hexadecimal, so you can then use other standard unix tools (grep, less, redirection...) to check the traffic. For inspection recording is still better, but nothing beats "tcpdump -X" when you just want to know if the packages are arriving at some port.
tcpflow is another useful tool. It's similar to tcpdump, but reassembles connections by sequence number. I've found it easier to use for application-layer analysis, where you care more about the data being sent than what literally appears on the wire.
One thing that you have to watch out for with tcpdump in the field (and especially in production) is that it doesn't rotate its capture files by default, and so you'll eventually end up with very large capture files which aren't loadable in Wireshark (which struggles to load capture files of size ~= avaliable RAM).
Note that tcpdump's default behaviour here is better than Wireshark's; last I checked Wireshark just crashes when your capture file exceeds the available RAM. Again, you can enable file rotation, but many don't realize this until they have been bitten by this attempting to do an overnight capture of a production issue...
I have used tcpdump in the past to capture traffic when I had physical access and ability to implement a hardware tap and analyze packets after feeding in packets with tcpreplay from another network's pcap file (IIRC). The point was to configure an IDS like Snorby using rules I derived from the packets I analyzed in Wireshark (from the resulting pcap file).
However, I haven't seen a need to use tcpdump in awhile since my problem domains have been quite different in that my focus back then was primarily network monitoring. Usually performance problems where I have worked have been easy enough to identify at a higher layer (e.g. n+1 select issues with SQL).
Tcpdump and wireshark are built on top of libpcap. If you are interested in learning how to write programs in C or Go using libpcap check out these posts.
Just yesterday tcpdump saved me an indeterminate amount of time working with a Horribly Problematic vendor's support team by allowing me to sniff the loopback interface to pinpoint which of two (closed-source) components was introducing a sizeable processing delay in our real-time network event management system.
If you are interested in tcpdump and use it for debugging, you might potentially also be interested in the Bro network monitoring system (http://bro.org).
It gives you very deep visibility in the supported protocols, dumps easy to parse log-files by default (see e.g. https://www.bro.org/sphinx-git/httpmonitor/index.html for HTTP information) - and it is fully scriptable.
We use tcpdump all the time to capture traffic that we later analyse with Wireshark, so I'm a bit surprised by some comments trying to confront the two.
I apologize if someone already mentioned it, but you can capture packets from Wireshark directly as well, without the need for tcpdump or tshark. However, this is only useful if you want to capture packets on the machine you are running Wireshark, obviously.
true, for machines without a monitor you normally use tcpdump and tshark then do local analysis with wireshark, plus wireshark is a little heavy for real time capturing sometimes.
With tshark used to be possible to capture packets using display filters, which was pretty handy in lots of situations. When privilege separation was implemented, the feature was lost(bug 2234).
I understand the importance of privilege separation, but I miss the feature.
Inaccurate. Most browsers will now be using ephemeral key exchange. You pretty much have to configure one of the end points to dump session keys to a log file, then load that in to wireshark alongside the packet dump.
tcpdump is fantastic; I often use it in lieu of Wireshark if I can. It's also a bit faster, which kind-of doesn't matter for me since I usually have it output the trace to a file and then use less to go through it.
I haven't used it yet but it's libpcap based so I can't imagine it being too different. It has to be at least 2000x better than the piece of shit Microsoft Network Monitor (it's like Wireshark, except so much worse...oh, and it doesn't do promiscuous mode)
What do you see as the advantage of using CloudShark over Wireshark? I see CloudShark costs between $200-$9000 a year and Wireshark is, of course, free so I'm wondering why CloudShark is preferable. Is it the collaborative aspect?
love this, I always thought tshark is similar to tcpdump for CLI, did not realize it knows more protocol than tcpdump, learned something new today. Thanks!
tcpdump is just a way of capturing the traffic, it's not a great analysis tool. It's simple, command-line, and part of the base install for many distributions. Generally very useful when you're remote debugging where you don't have a GUI: the workflow is tcpdump -> download .pcap to local machine -> analyze offline with wireshark. Alternately, tcpdump -> analyze in situ with tshark. (You can also use tshark directly, but then you don't get a do-over if you want to change your display filters and such). Also, easily scriptable.
But for the people who are new(ish) to tcpdump have you heard of libnet and libpcap? You can basically build your own tcpdump! :D
I was amazed at the speed packets fire when you program it yourself in C.
See: https://github.com/the-tcpdump-group/libpcap <-- CAPturing packets https://github.com/sam-github/libnet <-- sending packets
Libnet tutorial that I used religiously: https://repolinux.wordpress.com/2011/09/18/libnet-1-1-tutori...