Duplicate Multicast Packets

Duplicate multicast packets are symptoms of several things:

An ethernet-level device (e.g. bridge) that doesn't understand multicast so reforwards it. This is easy to track to the right subnet using mtrace's packet statistics, but hard to track to the final device because it doesn't change the ethernet source address.
The design of the PIM protocol accepts multiple copies periodically; when there are multiple paths to a multi-access link the default state is that all routers on the link forward; PIM Assert messages elect a single router as the forwarder but periodically time out.
Forwarding loops. This is often due to a misconfigured router.

NOTE: data collection is occurring but the data is not being processed at this time; the link below should just be taken as an example The data collection filters out packet pairs spaced far apart, attributing it to PIM Assert messages. The remainder of the results are presented in order of the number of extra packets the events placed on the network; this is a combination of the number of duplication events and the number of extra copies that each event caused.

Data Collection Details

The data collection is happening on parc.cairn.net, a PC router in the CAIRN testbed. This router sees some amount of transit traffic along with all MBone traffic entering PARC. The data collection process is a perl script which runs tcpdump to monitor all multicast traffic. (The actual command line is tcpdump -s 60 -tt -n -l -v udp and ip multicast). Output from this command looks like

894416496.029801 128.223.83.26.1026 > 224.2.246.107.22130: udp 332 (ttl 118, id 64839)
894416496.029977 128.102.84.134.11304 > 224.2.200.165.27166: udp 172 (ttl 117, id 18292)
894416496.031001 205.226.12.250.1032 > 224.2.210.122.62752: udp 469 (ttl 114, id 6459)
894416496.034636 198.164.125.5.47864 > 224.2.184.26.65044: udp 1023 (ttl 117, id 55516)
894416496.035247 13.1.101.97.1506 > 224.2.154.30.26810: udp 336 (ttl 125, id 50517)
894416496.036620 128.102.84.134.11302 > 224.2.200.165.60560: udp 1023 (ttl 117, id 18293)
894416496.038021 128.223.83.27.1029 > 224.2.163.188.23824: udp 332 (ttl 118, id 25675)
894416496.039931 128.173.12.161.32849 > 224.2.171.219.24958: udp 332 (ttl 115, id 62680)
894416496.042039 128.102.84.134.11302 > 224.2.200.165.60560: udp 1013 (ttl 117, id 18294)

The data collection script seperates out the timestamps, IP source and destination, TTL and ID. Any packet with the same IP source, destination, and ID that occurs within 2 seconds of another with these same values is considered to be a duplicate.

In order to reduce the amount of data collected, the script does not consider packet pairs that occur more than 300 seconds apart to be duplicates, since these are most likely explained by PIM Asserts. For other packets, the script simply writes the 5 pieces of information to a log file, e.g.

894416586.668814 144.212.100.12 227.6.5.4 120 60229
894416586.686640 144.212.100.12 227.6.5.4 117 60229
894416586.687773 144.212.100.12 227.6.5.4 117 60229
894416586.717201 144.212.100.12 227.6.5.4 115 60229
894416586.719388 144.212.100.12 227.6.5.4 115 60229
894416586.751066 144.212.100.12 227.6.5.4 113 60229
894416586.751898 144.212.100.12 227.6.5.4 113 60229
894416586.783171 144.212.100.12 227.6.5.4 111 60229
894416586.785961 144.212.100.12 227.6.5.4 111 60229

By sorting by IP addresses and IP ID, you can isolate "duplication events". (Here we ignore the possibility of two different packets with the same ID seperated temporally; this is sufficiently unlikely that we accept the possibility of misinterpretation of this event for the gain of being able to process the data with standard UNIX tools). Within a duplication event, sorting by timestamp and TTL can both provide interesting results. Sorting by TTL can help to determine the diameter of the loop or loops, and sorting by time can help to determine the gross topology of the loop.

Finding the source of the duplicates

just notes for now

Find the hop on which the duplication is occurring
- mtrace, negative loss
- -but bursty dups require mtrace -l
- and could be caused by state instantiation, meaning might not be able to be caught by mtrace at all
- mtrace only traces the proper path, so for loops it can be hard to track down the rest of the loop without sniffing one of the networks involved.
Then figure out which part of that hop is causing the duplication
- mdups can help with tcpdump-able networks
- cisco's show ip mpacket can help
- also use cisco's debug ip mpacket <group> detail
- If the source MAC address changes, the MAC source points to where to investigate next
- If the source MAC address changes, either the duplication is not on this link or the duplication is being caused by a MAC-layer device
Some reasons for duplicates:
- Routing loops (TTL decrements)
- Shared medium (i.e. bridges,switches) that routers don't know about
  - e.g. router A has interface on 10.0.1/24
  - router B has interface on 10.0.2/24
  - These 2 subnets are in the same multicast logical subnet, due to e.g. a bridge
  - Packets from host 10.0.1.1 will be picked up by A, forwarded to B, and then B will forward onto 10.0.2/24
  - layer 2 will forward from B to A, and A will pick the packet up again and forward it again
- Layer 2 weirdness (TTL doesn't decrement)

Using `show ip mpacket`

show ip mpacket can be useful in discovering the existence of duplicate packets but can only give circumstantial evidence about what device is responsible. debug ip mpacket <group> detail can give a better pointer to the device causing the problem but generally overloads the router unless you set up a special low-rate source with which to debug.

In order to use show ip mpacket, you must enable the global command ip multicast cache-headers.

The output of show ip mpacket looks like:

router#show ip mpacket
IP Multicast Header Cache
14664134 packets received over 15:31:03, cache size: 1024 entries
Key: id/ttl timestamp (name) source group

B39E/8   55870.720 (?) 192.187.8.42 224.2.218.180
FB66/118 55870.736 (?) 128.102.84.134 224.2.188.70
FB67/118 55870.736 (?) 128.102.84.134 224.2.145.44
FB67/117 55870.740 (?) 128.102.84.134 224.2.145.44
FB68/118 55870.740 (?) 128.102.84.134 224.2.188.70

The tuple that makes an IP packet unique are the ID, source and group. The two highlighted entries are for a duplicate packet.

Now that you know there are duplicate packets occurring, use mtrace and mstat (or the UNIX mtrace client, which combines both functions) to determine at what hop the extra packets occur, and try to find the reason (including using the debug ip mpacket <group> detail command at that router).

Using `debug ip mpacket <group> detail`

Here's one way to track this problem down:

Start a slow source inside your domain (maybe 1 packet per second) and ensure with mtrace that the packets are being replicated. (If the source is transmitting too quickly, the debug messages will overwhelm the router).

Turn on "debug ip mpacket <group> detail"

Look at the detailed debug entries. Each entry should look something like this:

Jan 13 11:09:48.243: IP: MAC sa=00a0.c96e.210f (Ethernet1/0), IP last-hop=204.162.228.1
Jan 13 11:09:48.243: IP: IP tos=0x0, len=124, id=0xCA7, ttl=116, prot=17
Jan 13 11:09:48.243: IP: s=128.111.52.10 (Ethernet1/0) d=224.2.1.1 (Serial5/0) len

You can tell that two entries are for the same packet because they will have the same IP source (s=), destination (d=), and ID (id=). Then, look at the first line - there should be two different MAC sources. Translate those MAC addresses to the routers they belong to. Based upon your topology and configuration, one router should be forwarding, and the other router shouldn't. Figure out which is which, and then figure out why the router that shouldn't be forwarding is. Bill Fenner - <fenner@research.att.com> This material is based upon work supported by the National Science Foundation under Grant No. 9729498. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation (NSF).

Duplicate Multicast Packets

Data Collection Details

Finding the source of the duplicates

Using show ip mpacket

Using debug ip mpacket <group> detail

Using `show ip mpacket`

Using `debug ip mpacket <group> detail`