Observer Analyzer : Expert Probe Software : How a probe uses RAM
   
How a probe uses RAM
 
Page Contents
Packet capture buffer and statistics buffer
Running Observer without reserved memory
Running Observer with reserved memory
How packet capture affects RAM
A Windows computer uses Random Access Memory (RAM) as a form of temporary data storage. Windows separates all available memory into three sections: protected memory, user memory, and reserved memory. An Observer probe, depending on how it is configured, uses these types of memory differently.
The protected memory is used to load critical operating system files, such as device drivers. If any of this RAM is dedicated to a driver or some other critical file, it cannot be used by another program. However, after Windows finishes loading its drivers, the memory is freed and any program may access the remaining protected memory.
User memory is all available memory beyond the protected memory. It is available to any application at any time. The probe uses this memory to temporarily store statistical information, such as Top Talkers data.
Reserved memory is user memory that you have specifically set aside for use by the Observer probe. Only the probe may use that portion of RAM. When the RAM is reserved for the probe not even the operating system may access it—even when Observer is closed.
By having RAM reserved specifically for the Observer probe, you ensure that the probe has the memory necessary to capture packets and store these packets for statistical processing. If Observer runs without any reserved memory, it requests and uses the operating system’s protected memory for capturing packets. There is no adverse effect of running an Observer probe without reserved memory, but it is not the most efficient way to run the probe. By default, the probe uses no reserved memory. Our recommendation is that you reserve memory for Observer so that the probe runs efficiently and leaves the protected memory for the operating system and other programs to use.
Packet captures are always written sequentially from the first open byte of RAM in reserved memory or in Windows protected memory. They are written until all available space is used. If you are using a circular buffer, then the first packet is overwritten with the newest packet. This is first-in, first out (FIFO). With Windows protected memory, your capture space is limited to about 50 to 80 MB, but with reserved memory you have the potential to store many gigabytes in memory. Figure 66 describes the two different ways that Observer runs.
Figure 66: Windows protected memory, user memory, and reserved memory
Whether using protected memory or reserved memory, Observer uses the RAM to store data for things such as (and creates a section within the RAM dedicated to):
Packet capture
Statistics queue buffer
Collected statistical memory
Network packets seen by Observer are passed to both the packet capture memory and to the statistics queue buffer. After a packet is processed by the statistics queue buffer, the statistical information is passed to the statistical memory. All packets in both the packet capture memory and the statistical queue buffer stay in memory until the buffer is full and the oldest packets are replaced by newer packets (using FIFO).
Figure 67 shows what options in Observer control the size of various portions of memory.
Figure 67: How to resize various memory options
Packet capture buffer and statistics buffer
There are two kinds of buffers that a probe uses to store data in real-time: capture buffers and statistical buffers. The capture buffer stores the raw data captured from the network while the statistical buffer stores data entries that are snapshots of a given statistical data point.
Selecting an appropriate capture buffer size given system resources is all most users need to worry about; the default settings for the statistical buffers work perfectly fine in the vast majority of circumstances.
However, if you are pushing the limits of your probe system by creating many probe instances, you may be able to avoid some performance problems by fine-tuning the memory allocation for each probe instance.
For example, suppose you want to give a number of remote administrators access to Top Talkers data from a given probe. You will be able to add more probe instances within a given system’s memory constraints if you set up the statistics buffers to only allocate memory for tracking Top Talkers and to not allocate memory for statistics that no one will be looking at.
Observer has no limitations on the amount of RAM that can be used for a buffer.
Note that when run on a 64-bit Windows, there is no 4 GB limitation for the capture buffer; you are limited only by the amount of physical memory installed on the probe.
In all cases, the actual buffer size (Max Buffer Size) is also reduced by 7% for memory management purposes. Should you try and exceed the Max Buffer Size an error dialog will be displayed indicating the minimum and maximum buffer size for your Observer (or probe) buffer.
For passive probe instances, which are most often used for troubleshooting, the default settings should be sufficient. If you are creating an active probe instance (one that writes to disk and not just reads from it), then you may want to use the following formula as a rough guideline to determine how much RAM to reserve for the probe instance when doing a packet capture. (This formula does not apply when doing a GigaStor capture to disk. It is only for probe instances doing packet captures.)
Use this formula to determine your RAM buffer size:
Network Speed
×
Average Throughput (MB/second)
Seconds of data storable in RAM
Tip! You want a buffer that will handle your largest, worst case unfiltered burst.
Use this formula to determine how much hard drive space a capture requires (in GB) and Observer’s write-to-disk capability. There is no limitation to the amount data Observer can write to disk other than the disk size itself.
(Traffic Level / 8 bit) × 3600 Seconds
÷
1024 bytes
Gigabytes per hour
For instance a fully utilized 1 Gb port (1 Gbps is 125 MBps):
(125 MBps / 8 bit) × 3600 Seconds
÷
1024 bytes
~54.93 GB per hour
Running Observer without reserved memory
Revised: 2017-10-24
Single probes cannot use reserved memory. By default, no memory is reserved for Observer if you install it on your own system.
All versions of Observer Expert, Observer Suite, Expert Probe software, and Multi Probe software installed on your own hardware, unless modified.
Single Probe software at all times
NetFlow probes
Observer without reserved memory is the default, but not recommended, configuration. It is the default because each network is unique and you must determine how you want Observer to be configured for your system.
Note: This section does not apply to the GigaStor or other hardware products from VIAVI. They are properly configured at the factory.
Tip! If you need more RAM for the statistics queue buffer, you may need to lower the amount of RAM dedicated to packet capture so that it is freed and available to add to the statistics queue.
After you install Observer and first open the program it does not have any reserved memory. Observer allocated a portion of the available protected memory for its use. This creates a “Windows memory pool” for Observer of about 50 to 80 MB (depending on the amount available from Windows, and cannot be increased). This is a limitation of the Windows memory pool and the Windows operating system.
Single Probes, unlike Multi-Probes and Expert Probes, cannot use reserved memory because of their design. By default, 16 megabytes is available for the Packet Capture and Statistics Queue buffer. Single Probes have a maximum of 52 megabytes that can be assigned from the Windows memory pool. Because of the Windows memory pool constraint, Single Probes are limited to a Packet capture buffer maximum of 72 megabytes, assuming you set the Statistics queue buffer to its minimum (12 megabytes). The Statistics queue buffer maximum is 76 megabytes if the Packet capture buffer is set to 0 megabytes.
1. Click the Memory Management tab to display the list of probe instances and their buffer sizes.
2. Click the Configure Memory button at the top of the window to view and modify how Observer uses the protected memory for this probe instance. The Edit Probe Instance window opens.
On the Edit Probe Instance window, you can see how memory is allocated for:
Packet capture
Statistics queue buffer
You can also see how much protected memory is still available in the Windows memory pool.
Figure 68: Edit Probe Instance
3. Use the arrows to the right of the Packet capture and Statistics queue buffer to increase or decrease the amount of RAM you want dedicated to each. See How to allocate the reserved RAM to help determine how to divide the memory.
 
 
Running Observer with reserved memory
Reserved memory helps Observer run more efficiently by dedicating memory for its exclusive use.
Observer Expert
Observer Suite
Expert Probe software
Multi Probe software
Observer uses reserved memory for packet capture and the statistics queue buffer. It is highly-recommended that you use reserved memory. (GigaStor appliances running Observer are preconfigured this way.) You must determine how you want Observer to be configured for your system.
Caution: Never change the reserved memory settings of VIAVI hardware unless VIAVI instructs you do so. Reserved memory settings should only be modified on non-VIAVI hardware, such as a desktop computer running Observer.
Tip! If you need more RAM for the statistics queue buffer, you may need to lower the amount of RAM dedicated to packet capture so that it is freed and available to add to the statistics queue.
Reserving memory allows Observer to allocate RAM for its exclusive use. This ensures that Observer has the necessary memory to store packets for statistical analysis, or for capturing large amounts of data for decoding. The more memory you reserve for Observer, the larger the packet capture and statistical queue buffers can be.
If the memory buffer for the statistics queue buffer is too small, you may end up with inaccurate statistical data because some data may get pushed out before it can be processed. Observer processes packets on a first-in, first out (FIFO) basis, so it is important that the buffer be large enough to allow for processing.
When reserving RAM for Observer you are taking RAM away from the operating system. Table 35 shows how much memory is required by the operating system. Anything beyond this amount may be reserved for Observer.
Table 35. Reserved memory requirements
Operating System
RAM required for the operating system
64-bit with less than 4 GB RAM
800 MB
64-bit with 4 GB RAM
4 GB 1
64-bit with 6+ GB RAM
4 GB
32-bit2
256 MB (although 400+ MB is recommended)

1 Because of how 64-bit Windows loads its drivers when 4 GB of RAM is installed all 4 GB is used by Windows. This is sometimes referred to as the BIOS memory hole and means you cannot reserve any memory for Observer. To capture packets on 64-bit Windows install either more than or less than 4 GB of RAM.

2 32-bit operating systems do not support more than 4 GB of RAM. Observer cannot use any RAM above 4 GB.

1. To see how much protected memory the probe has, click the Memory Management tab.
2. Click the Configure Memory button at the top of the window to view and modify how Observer uses the protected memory for this probe instance. The Edit Probe Instance window opens.
On the Edit Probe Instance window, you can see how memory is allocated for:
Packet capture
Statistics queue buffer
You can also see how much protected memory is still available in the Windows memory pool.
3. Use the arrows to the right of the Packet capture and Statistics queue buffer to increase or decrease the amount of RAM you want dedicated to each. See How to allocate the reserved RAM to help determine how to divide the memory.
4. After reserving memory for Observer you must restart the system for the changes to take affect. After you restart the system you can allocate the memory to the different probe instances.
 
 
How packet capture affects RAM
When you start a packet capture (Capture > Packet Capture and click Start), all packets that Observer sees are placed into the packet capture buffer (a specific portion of the protected memory). The packets stay in this protected memory until the buffer is cleared. If you are using a circular packet buffer, new packets overwrite old ones after the buffer is full.
Figure 69 shows how Observer receives a packet and distributes it throughout RAM, and how it is written to disk for packet capture and GigaStor capture.
Packets received by the network card are passed to Observer, where Observer puts each packet into RAM, specifically in the packet capture memory buffer and the statistical queue buffer. If a packet must be written to disk for either a GigaStor capture or a Packet Capture, it is copied from the RAM and written to the disk.
Figure 69: How packets move through Observer’s memory
The capture card receives data off the network.
The capture card passes data into RAM. In the RAM it goes into the packet capture buffer and the statistics queue buffer.
The statistics queue buffer passes the information to the statistics memory configuration.
The statistics memory configuration passes the data to the real-time graphs.
The Network Trending Files receive data from the statistics queue buffer through the NI trending service, where they are written to disk.
The following steps occur only if you are writing the data to disk through a packet capture to disk or a GigaStor capture.
If you are using packet capture to disk, the packet capture buffer passes the data to the operating system’s disk.
If you are using GigaStor capture, the statistics queue buffer and the packet capture buffer passes the information to the RAID.
A few notes about how some buffers are used:
Packets received by the statistics queue buffer are processed and put in the collected statistics buffer.
Data for network trending comes from the statistics queue buffer, then it is written to disk, and finally flushed from the buffer every collection period.
The collected statistical buffer does not use first-in, first-out to determine statistics. Therefore, after the statistic limit is reached the remaining data is no longer counted; however, data for known stations continue to be updated indefinitely.
Regardless of whether Observer is using reserved memory, the statistics memory, statistics queue buffer, and packet capture buffer function the same. The storage space available for storing packets in memory increases though when you reserve memory.