[Logo]

 

IPsumdump

The ipsumdump program summarizes TCP/IP dump files into a self-describing ASCII format easily readable by humans and programs. Here's a few lines from an example summary:

!IPSummaryDump 1.2
!creator "ipsumdump -sd /home/kohler/largedump.gz"
!host max.lcdf.org
!runtime 1092201225.541624 (Tue Aug 10 22:13:45 2004)
!data ip_src ip_dst
64.55.139.202 209.247.204.242
18.26.4.9 64.55.139.202
 

Ipsumdump can read packets from network interfaces, from tcpdump files, and from existing ipsumdump files. It will transparently uncompress tcpdump or ipsumdump files when necessary. It can randomly sample traffic, filter traffic based on its contents, anonymize IP addresses, and sort packets from multiple dumps by timestamp. Also, it can optionally create a tcpdump file containing actual packet data.

Options supplied by the user determine what goes in the dump. Possibilities include:

  • Timestamp
  • Source IP address
  • Destination IP address
  • TCP/UDP source port
  • TCP/UDP destination port
  • Length
  • Protocol
  • IP ID
  • Fragment flag
  • Fragment offset
  • TCP sequence number
  • TCP acknowledgement number
  • TCP flags
  • Payload length
  • Payload (as a quoted string)

It's easy to add similar properties.


Ipsumdump and Click

Ipsumdump is the first standalone program to use the Click modular router. (You do not need Click to compile ipsumdump.) The ipsumdump program simply constructs a Click configuration based on options provided by the user, then runs that configuration. All packet-related tasks, such as reading and writing tcpdump and ipsumdump files, sampling, filtering, and anonymization, are handled by Click elements designed for the purpose. Most of those elements existed already and required only modest changes, which made ipsumdump pretty easy to write. The elements we wrote from scratch have been fed back into Click, making them available to other projects. For example, if you'd like to read an ipsumdump file and synthesize packets with the same characteristics, use the FromIPSummaryDump element.

I hope people start using Click for similar analysis projects and programs. It's more flexible than programming to libpcap directly, and pretty easy, too.

Here's the Click configuration that corresponds to a modestly complex ipsumdump invocation, "ipsumdump ~/largedump.gz -sdSDp --sample 0.5 --anonymize". Use ipsumdump's --config option to see similar configurations.

shunt :: { input -> output };
src0 :: FromDump("/home/kohler/largedump.gz", FORCE_IP true, STOP true, SAMPLE 0.5) -> shunt;
shunt
  -> anon :: AnonymizeIPAddr(CLASS 4, SEED false)
  -> to_dump :: ToIPSummaryDump(-, CONTENTS "ip_src" "ip_dst" "sport" "dport" "ip_proto",
          CAREFUL_TRUNC false, VERBOSE true, BAD_PACKETS false, 
          BANNER "ipsumdump /home/kohler/largedump.gz -sdSDp --sample 0.5 --anonymize");
progress :: ProgressBar(src0.filepos, src0.filesize, UPDATE 0.1, DELAY 2s, CHECK_STDOUT true);
DriverManager(, wait_stop 1, write_skip progress.mark_done);

Ipsumdump configurations can include the following elements:



News

Version 1.42: Add support for PPP-encapsulated DAG dumps, via the ’--dag-ppp’ option, and for PPP-encapsulated tcpdump files. Requested by Pedro Torres.

Version 1.41: Add support for IEEE 802.11/Prism2 encapsulated dumps. Requested by David Wetherall. Use ’C’ for CWR in tcp_flags dumps, since that’s what tcpdump does. Bump IPSummaryDump file format version number to 1.2 as a result.

Version 1.40: Fix ERF/DAG timestamps. Again, reported by Holger Dreger.

Version 1.39: Support new-style ERF/DAG dumps. Reported by Holger Dreger.

Version 1.38: Print information from partially-captured headers. For example, ipsumdump will print the TCP source ports of packets whose TCP options were not captured (as long as the source ports were captured). Previously the source ports would be printed as ’-’, since the whole TCP header was not captured. Requested by Vern Paxson. ’--bad-packets’ now prints ’!bad’ lines IN ADDITION TO normal output, not instead of normal output.

Version 1.37: Support HDLC link layers. Requested by Vern Paxson.

Version 1.36: Support 64-bit-long systems (patch directly from Click). Reported by Kirill Ponomarew.

Version 1.35: Add ’--capture-length’ option, requested by Vern Paxson. Add ’--dag’ and ’--nlanr’ options, for DAG and NLANR dumps.

Version 1.34: Fix configure check for machines where int64_t and long are the same type. Reported by Kirill Ponomarew. Add ’--tcpdump-text’ option.

Version 1.33: Document segmentation-fault behavior when mmaping corrupt files, and add ’--no-mmap’ option (requested by Vern Paxson). Allow progress bars when there’s no IP summary dump output.

Version 1.32: Support large files (bug reported by David Loose).

Version 1.31: Include Unqueue element (bug reported by David Loose).

Version 1.30: Set UDP and IP lengths correctly, hopefully addressing problems reported by Andrew White. In ’--multipacket’, the sum of the individual packet lengths should equal the total packet length.

Version 1.29: Fix build problems reported by Andrew White.

Version 1.28: Fix ’--netflow-summary’ problems reported by Andrew White: FromNetFlowSummaryDump didn’t set IP length correctly, causing packets to be treated as corrupt. ’--netflow-summary’ pays attention to flow end timestamp as well as flow timestamp, and to TCP flags. ’--multipacket’ spreads out packets between flow-begin and flow-end timestamps.

Version 1.27: Change build process to put dependency flags in their own variable (avoid ./configure problems reported by Anestis Karasaridis).

Version 1.26: Update to newer version of Click (reduces ./configure problems).

Version 1.25: Add ‘--binary’ option and support for binary IPSummaryDump files.

Version 1.24: Speed up interaction of ‘-r’ (read from tcpdump(1) files) and ‘-A’ (anonymize). Previously this would uselessly copy 4MB of data per packet! Reported by Vern. Add ‘--limit-packets’ option.

Version 1.23: Documentation updates. TCP flag bits 6 and 7 are printed as E and W, for ECE and CWR, based on the ECN Proposed Standard. Fixed bug with interaction between ‘--interval’ and ‘--interface’. Under ‘--tcpdump’, truncated IP packets are now printed as normal packets. Some tcpdump files don’t correctly record the caplen.

Version 1.22: Fragment offset fields, produced by -G, formerly were expressed in 8-byte units. Starting with this version, fragment offset fields are expressed in bytes. Updated the ‘!IPSummaryDump’ file version number to ‘1.1’ to represent this change. Fixed bug with reading ipsumdump files: incorrect IP lengths caused many packets to be ignored. Update to Click-1.2.4.

Version 1.21: Update endianness test (Click failed to compile on Solaris). Reported by Ahmed Aslam <aaslam@csee.usf.edu>.

Version 1.20: Add ‘--interval’ option.

Version 1.19: Add some sanity checks on ip_len to the TCP and UDP header checks, and improve payload length calculation. Reported by Vern.

Version 1.18: Fix bug with --payload: ipsumdump could output a payload string containing too much or too little data. Reported by Vern.

Version 1.17: Reading from network interfaces was broken by version 1.14: unless -w was given, ToIPSummaryDump treated partially-read packets (low SNAPLEN) as erroneous. Reported by Brecht Vermeulen <brecht.vermeulen@rug.ac.be>.

Version 1.16: Progress bar improvements: The progress bar will not appear if normal output is to the terminal, or if the invocation doesn’t take very long.

Version 1.15: --length, --payload-length, and --payload deal with IP length, ignoring any link-level padding at the end. Problem reported by Vern.

Version 1.14: Don’t generate mistaken output for bad IP, TCP, or UDP headers (print dashes instead). Bad headers include IP versions other than 4, bad IP header lengths, bad TCP header lengths, and TCP/UDP headers not included in a single fragment. Suggested by Vern. Add --bad-packets option. When supplied, the IP summary dump contains messages like ‘!bad IP header length 2’ on packets with bad IP, TCP, or UDP headers, instead of normal output (with dashes). Add --payload option.

Version 1.13: Speed improvements in --tu-summary/--ipsumdump.

Version 1.12: Remove default dump content options. If you don’t supply any dump contents, ipsumdump won’t create a summary dump. It still will create any --write-tcpdump file, though. For Vern. Catch fewer signals.

Version 1.11: Add progress bar.

Version 1.10: Add --fragment and --fragoff options. Fix behavior with fragments. Reported by Vern Paxson <vern@icir.org>.

Version 1.9: Add --record-counts option.

Version 1.8: Fix --filter option, and add --no-promisc.

Version 1.7: Bug fixes. Signal handling. Sampling works correctly with MULTIPACKET.

Version 1.6: The --write-tcpdump output file includes link-level headers.

Version 1.5: Add --random-seed option.

Version 1.4: Add --sample and --collate options.

Version 1.3: Add FromDevice.kernel_drops handler, and use that handler to report any kernel packet drops.


Return to [Eddie Kohler].