Observer GigaStor : Gen3 Capture Card : Understanding duplicate packets
   
Understanding duplicate packets
 
Page Contents
Understanding packet deduplication
How to remove duplicate packets in real time
How to remove duplicate packets from saved captures
Created: 2013-05-20   Revised: 2016-09-14
Duplicate packets lower the statistical accuracy of analysis, increase network link saturation, and can interfere with tools. Packet deduplication removes duplicate packets and helps you avoid those situations.
 
A duplicate packet is any packet that is identical to another packet. The packet header is inspected and all fields must be identical for it to be a duplicate. However, there are some situations where the header has been modified slightly during the packet's journey. These situations require some fine-tuning of the deduplication settings to ignore those fields that were modified before the duplicate packet is received.
 
Understanding packet deduplication
Created: 2014-04-15   Revised: 2016-09-14
Deduplication is useful when multiple copies of the same packet are received, but only a single copy should be seen.
 
Duplicate traffic is part of any network environment and is unavoidable. However, reducing duplicate packets as much as possible helps ensure your network is more efficient. It also allows your tools to be more accurate. Duplicate packets reduce statistical accuracy, which leads to higher perceived levels of traffic or network connections. If you experience duplicate packets, consider your analytical needs and network topology when deciding whether deduplication should be used. You most often encounter them when packets are traversing multiple routers and those routers are copying their traffic to the SPAN/mirror port.
Removing duplicates from a saved packet capture can be more accurate than deduplication with the capture card. Observer has several more options than the capture card for ignoring packet header fields. These are header fields you choose to not examine (ignore) when determining if a packet is a duplicate. When all packet header fields are used as criteria (none are ignored) the capture card-based deduplication and Observer deduplication produce nearly the same results.
In some cases you may want to retain the duplicate packets. For example, when packets are being looped or when multiple VLANs are used with your hardware, you may want to keep the packets. Retaining a copy of duplicate packets and their traversal through both VLANs may be necessary when verifying whether the traffic was routed properly.
If you are attempting to find the source of duplicate packets in real time, do not deduplicate packets. Removing duplicate packets before they reach Observer or the GigaStor system lessens your ability to find the source of duplicates—if that is your goal. Instead, you can allow all duplicate packets and make changes to you monitored switches or SPANs and see if that resolves the duplicates coming in or helps locate the source.
 
How to remove duplicate packets in real time
Created: 2015-08-07   Revised: 2015-08-11
Use hardware acceleration on the Gen3 capture card to remove duplicate packets before they are captured to disk or acknowledged by Observer. This is called packet deduplication.
Hardware acceleration must be enabled on your Gen3 capture card for this feature to function. Hardware acceleration for your virtual adapter is enabled by default.
To use hardware acceleration to skip duplicate packets:
1. Right-click the Gen3 capture card-equipped probe instance and choose Probe or Device Properties.
2. Click the Virtual Adapters tab.
3. Click Edit Adapter.
4. In the Assign Ports to Virtual Adapter window, select Skip duplicate packets and click OK.
5. (Optional) Click Skip Duplicate Packets Configuration.
The Skip Duplicate Packets Using Hardware Accelerated Capture Adapter dialog appears.
Figure 68: Skip Duplicate Packets Using Hardware Accelerated Capture Adapter
Screenshot Gen2 Skip Duplicate PacketsScreenshot Gen2 Skip Duplicate Packets
6. (Optional) Select how duplicate packets are recognized by the Gen3 capture card and click OK.
Example: (Optional) For example, by selecting Examine IP time to live (TTL), the packet time to live is considered when determining a duplicate packet. If the option is cleared, TTL is ignored for all consideration of what is, and what is not, a duplicate packet.
7. Click OK to enable packet deduplication.
 
The Gen3 capture card now skips duplicate packets that it receives on the active instance. The duplicate packets will not be saved to disk or acknowledged by Observer.
 
Deduplication details for the capture card
Created: 2016-09-14
The packet deduplication engine of the capture card is controlled by features in Observer. The features must be in use for packet deduplication to work.
 
For capture card packet deduplication to work:
Hardware acceleration must be enabled and in use.
Virtual adapters must enabled and in use.
 
 
These settings can be used to ignore certain packet headers when determining if a packet is a duplicate:
IP time to live (TTL)
If selected, the time to live (TTL) value in packets would not be examined when determining if a packet is a duplicate. This is useful to select when the same packets makes multiple hops through routers.
TCP sequence and acknowledgement numbers, and TCP options
If selected, TCP sequence and acknowledgement numbers, and also TCP options, are not would not be examined when determining if a packet is a duplicate. Overall, this ignores the ordering of the packets and the values of optional packet fields.
 
How to remove duplicate packets from saved captures
Duplicate packets are packets that are captured twice or multiple times by Observer. Typically, duplicates are a result of how data is sent to Observer .
For a switch, the use of a SPAN/mirror port and/or trunk is required to capture data. Knowing this, the following scenarios may produce duplicate packets—which are then seen by Observer:
If a SPAN/mirror port is configured to send both ingress (in) and egress (out) data from multiple ports, any communication between any two ports being monitored results in a duplicate packet.
If a trunk is monitoring multiple VLANS, data flowing between VLANS is seen as duplicate packets.
If Observer is monitoring data pre- and post-route. Meaning, a single packet is seen at one location pre-route and again post-route. The post-route packet is considered by Observer as a duplicate packet.
While this is harmless as it pertains to your network working correctly, Observer identifies these as duplicate packets. There are two ways of dealing with this situation:
Configure the SPAN/mirror port or trunk to show only ingress or egress traffic, but not both.
Use Observer to remove duplicate packets from an existing capture file.
Observer includes a feature that removes the “noise” caused by duplicate packets without affecting the underlying packet capture data. This feature is a special version of the standard capture buffer file-loader. To remove duplicate packets (i.e. skip them) while loading a capture buffer file, complete the following steps:
2. Type, or navigate to, the capture file you want to load.
3. Select your criteria for how duplicate packets are handled.
Skip duplicate packets only when packet time difference is less than
During evaluation, packets are only compared against packets that arrived at nearly the same time, or specifically during a time difference less than N-milliseconds. Setting this can help avoid some false-positive results, but you may need to experiment with the value.
Data link layer
If selected, layer 2 of the OSI Model is ignored when determining duplicate packets. For example, MAC addresses would not be examined when determining if a packet is a duplicate. They are ignored, but the rest of the packet is not.
IP time to live (TTL)
If selected, the time to live (TTL) value in packets would not be examined when determining if a packet is a duplicate. This is useful to select when the same packets makes multiple hops through routers.
IPv4 type of service or IPv6 traffic class
If selected, type of service (ToS) and traffic class (for IPv6) would not be examined when determining if a packet is a duplicate. The option is most useful when network hardware or software is changing these quality of service fields.
TCP sequence and acknowledgement numbers, and TCP options
If selected, TCP sequence and acknowledgement numbers, and also TCP options, are not would not be examined when determining if a packet is a duplicate. Overall, this ignores the ordering of the packets and the values of optional packet fields.
4. Click OK.
The capture file loads into Observer and you arrive at the Decode and Analysis tool.
5. (Optional) If duplicate packets are still visible, repeat the process and select different duplicate packet handling criteria.
 
Duplicate packets should now be skipped/ignored in your capture file. No permanent changes are made to your loaded capture file.
 
 
If you want to make your changes permanent, save your results as a new capture file.