These applications can be network analyzers (also known as network monitors) or an intrusion prevention/detection system. Such common open source applications are tcpdump [1], snort [2], wireshark [3] (previously known as ethereal) , ntop [4] etc.

As the packet propagates from Network Interface Controller (NIC) to the kernel and then to the userspace application, it creates some overhead. Under heavy traffic conditions the percentage of the captured packets over the total number can decrease.

The size of the frame does play a significant factor, as the smaller the packet size the higher the negative impact in the packet capture percentage. The reason for this is that for same throughput the amount of smaller packets is greater then for bigger packet sizes, having as result more need for processing power.

In this article we will describe how one can improve lossless network packet capturing with libpcap by using the PF_RING kernel patch. Libpcap[1] is one of the more vastly open source library for packet capturing and uses by default PF_PACKET protocol in order to transfer the packets from the driver to the userspace.

It is the de facto library that facilitates the packet transition from kernel onto the userspace is libpcap. It provides an API for the programmer to select the capturing interface (device) and gives the ability to compile Linux Packet Filters (LPF) into the kernel for selective packet capturing based on the 5 tuple (protocol, source/destination IP address and source/destination port).

PF_RING is a replacement for PF_PACKET that not only uses memory mapping instead of processing expensive buffer copies from kernel space to userspace, but it also uses ring buffers making to transportation in a more efficient way.

In this article we will also describe an installation guide and a comparison of the libpcap, tested for lossless packet capturing with and without the PF_RING applied.

We will show that for 64 byte packets there was a 17.01% improvement, for 128 byte packets 3.49%, for 256 byte packets 57.82, for 512 byte packets 20.14%, for 1024 byte packets 19.19% and finally for 1518 byte packets, the improvement was 11.15%.

These improvements reveal ability for the existing hardware to allow lossless packet capture at higher data speed rates making more efficient open source IDS and IPS systems that use libpcap. Also any other applications such as protocol analyzers that use libpcap can work more efficiently.

Optimization using PF_RING
PF_RING is a mechanism conceived by Luca Deri [5] for accelerating libpcap. It comprises of a kernel patch and a modified libpcap. This modified libpcap provides exactly the same API to the user but underneath it is using the ring buffers provided by the kernel patch to read packets. The patch copies the packets into the ring straight from the driver. Figure 1 below shows the architecture of PF_RING.

Figure 1 - PF_RING and legacy architecture

Obtaining the PF_RING Source
PF_RING can be obtained using subversion [6]:

    svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/

The PF_RING package must be moved to /usr/src/kernels/ and the PF_RING environment variable needs to point to that directory. The mkpatch.sh file needs to be edited in order for the kernel version to be specified. Here is an example for the 2.6.24 kernel:

    VERSION=${VERSION:-2}
    PATCHLEVEL=${PATCHLEVEL:-6}
    SUBLEVEL=${SUBLEVEL:-24}

When the above script gets executed it will automatically download the specified kernel source and apply the PF_RING patch. A new directory with the specified and patched kernel will be available.

The specifier "PF_RING" will appear at the end of the kernel version. For the example of the 2.6.24 kernel, the directory /usr/src/kernels/ the directory $PF_RING/workspace/linux-2.6.24PF_RING will be created.

    VERSION = 2
    PATCHLEVEL = 6
    SUBLEVEL = 24
    EXTRAVERSION = -PF_RING

It was noticed also that the file "net/ring/Kconfig" could give some errors during the kernel configuration and line 12 (blank line) needed to be deleted. Following this, the kernel needs to be configured. The user can do it their preferred way, the authors executed:

    make menuconfig

PF_RING needs to be also compiled as a module. The option in the .config file is

    CONFIG_RING=m

The kernel then can be built and installed. The user should not forget to build and install the modules along with the (preferably NAPI enabled) network driver.

Compiling the modified libpcap
The PF_RING package contains a modified libpcap as explained above. It can be found in

    /usr/src/pf_ring/PF_RING/userland/libpcap-X.Y.Z-ring

where X.Y.Z is the version of the libpcap the downloaded PF_RING version was built to support. In this directory the pcapint. h and pcap-linux.c files can be found.

They need to replace the files with the same name in the original libpcap version X.Y.Z. At the time this work was conducted, libpcap could be downloaded from http://www.tcpdump.org/

After it has been downloaded and unzipped/untarred the two files from the PF_RING version need to replace the original ones as mentioned at the beginning of this section. Also in the pcap-linux.c file, the lines

    #define HAVE_PF_RING
    //#undef HAVE_PF_RING

need to be uncommented/commented accordingly so HAVE_PF_RING is defined, in order for the PF_RING to be enabled. Following that the library needs to be configured and built.

In order to configure it, one can execute:

    ./configure CPPFLAGS="-I/usr/local/include"
    LDFLAGS="-L/usr/local/lib" CFLAGS="-
    D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -
    D_FILE_OFFSET_BITS=64"

And then it can be built by executing:

    make && gcc -shared -Wl,-soname -
    Wl,libpcap.so.`cat VERSION` -o
    libpcap.so.`cat VERSION` *.o "lc

followed by

    make install && cp libpcap.so.X.Y.Z
    /usr/local/lib

in order to install it.

Compilation of the PF_RING shared library
Compiling and testing the PF_RING library several is a multi-step process. Firstly, the header files need to be copied over to the standard include directory:

    cp $PF_RING/workspace/linux-2.6.24-1-686-
    smp-PF_RING/include/linux/ring.h
    /usr/include/linux

Then a shared library (.so) can be generated:

    cp $PF_RING/userland/lib
    make clean
    make all
    gcc -shared -Wl,-soname -
    Wl,libpfring.so.0.9.4 "o
    libpfring.so.0.9.4 *.o "lc
    gcc -shared -Wl,-soname "
    W1,libpfring.so.0.9.4 "o
    libpfring.so.0.9.4 *.o "lc

The files can now be copied to a common system directory:

    cp libpfring.a libpfring.so.0.9.4
    /usr/local/lib

    cp pfring.h /usr/local/include

    ln -s /usr/local/lib/libpfring.so.0.9.4
    /usr/local/lib/libpfring.so

Next, the directory can be included in the path that the dynamic loader can search:

    echo "/usr/local/lib" >> /etc/ld.so.conf

    Ldconfig

One should verify the dynamic loader can use the library by running the following:

    ldconfig -v | grep pfring

which should produce the following output:

    libpfring.so.0.9.4->libpfring.so.0.9.4

    cd $PF_RING/userland/examples
    make clean
    make all
    ./pfcount -v -I ethX

where ethX is the relative interface. To verify the kernel module is being used, run the following:

    lsmod | grep ring

Executing:

    Dmesg

should yield presence of PF_RING by an output that should look similar to the following:

    RING: succesfully allocated 128 KB
    [tot_mem=26509372][order=5]
    RING: allocated 80 slots
    [slot_len=1618][tot_mem=131072]
    device eth1 entered promiscuous mode

Compilation of user space application (tcpdump)
Packet capture applications that use libpcap can now use the modified PF_RING libpcap. In this work tcpdump-4.0.0 is described:

    make clean
    ./configure CPPFLAGS="-
    I/usr/local/include"
    LDFLAGS="-L/usr/local/lib -
    lpfring "lpcap -lpthread"
    Make && make install

Statistics interface provided by PF_RING patch
The PF_RING patch provides a statistics interface to the system user in

    /proc/net/pf_ring/info

With root privileges one can inspect the contents of the above file to get information similar to the following:

    Version : 3.2.1
    Bucket length : 128 bytes
    Ring slots : 4096
    Sample rate : 1 [1=no sampling]
    Capture TX : No [RX only]
    Total rings : 0

Also in /proc/net/pf_ring/x can provide the following information:

    Bound Device : eth0
    Version : 6
    Sampling Rate : 0
    Cluster Id : 0
    Tot Slots : 7181
    Slot Len : 146
    Data Len : 128
    Tot Memory : 1048576
    Tot Packets : 62206634

Comparison Between PF_RING And PF-PACKET
Following are the comparison tests we did with a vanilla libpcap (using by default PF_PACKET) and a PF_RING enabled libpcap. Figure 2 below shows the test setup.

Figure 2 - Test Setup

The packet generator was the Spirent Smartbits 6000C. Table 1 and Table 2, below show the hardware and software configuration on the PC:

Table 1 - Hardware Configuration

Table 2 - Software Configuration

Test procedure
The NIC was set to promiscuous mode from tcpdump and using SmartApps, 10,000,000 (10 Million) 64 byte IP packets initially were injected at line rate.

The amount of captured packets by tcpdump was observed: if it matched the number of injected packets (10 Millions), and tcpdump/libpcap did not report any packets dropped by the kernel, the frame rate was recorded, otherwise the rate was reduced and this procedure would be reiterated.

When a lossless capture would occur the packet size would increase and again the above procedure would be repeated for 128, 256, 512, 1024 and 1518 byte packets.

The above procedure has been repeated for captures with PF_PACKET and with PF_RING.

Figure 3 - Lossless packet capture comparison

Results of the comparison tests
The results can be found in Table 3 and Table 4, below and Figure 3 above shows graphically the comparison of the two methods.

Table 3 - Lossless packet capturing with PF_PACKET

Table 4 - Lossless packet capturing with PF_RING

Figure 4 below shows the percentage improvement in the lossless capture frame rate that PF_RING offered compared to PF_PACKET.

Figure 4 - % Improvement with PF_RING

As you can see, for all the packet sizes, PF_RING can provide lossless packet capture at higher frame rates. In this test case the least improvement it offered was 3.49% at 128 bytes sized packets.

The best improvement however was for 256 byte packets where it offered 57.82% improvement compared to PF_PACKET. In average across the spectrum of the packet sizes that have been tested, the improvement in the frame rate for lossless capture was 21.47%.

It is worth mentioning that the above results are directly affected by the kernel and NIC driver versions and configurations, and also the presence of NAPI [7].

Conclusions
In this article we have tried to show how one can achieve improved lossless packet capturing using the PF_RING. Our results show clearly that for all sizes PF_RING was a more efficient mechanism as it can provide lossless packet capture at higher data rates.

At 256 byte packets it provided improvement of as much as 57.82% compared to PF_PACKET, where in average across all packet sizes, the improvement was 21.47%.

These improvements allow existing hardware that runs open source applications such IDS, IPS and protocol analyzers that use libpcap to achieve lossless packet capture at higher data speed rates making more efficient.

Joseph Gasparakis is a senior software engineer working for Intel Corp. in Shannon, Ireland for the Performance Product Division of the Embedded Computing Group. He has been in telecom industry since 2000 working on several proprietary and open source software stacks. He has also worked as system administrator, and he is a Certified Ethical Hacker (CEH). Joseph is currently specializing in optimizations of multicore and network related technologies.

James Chapman is a software engineer with Intel Shannon, Ireland, working for the Performance Product Division of the Embedded Computing Group. He is currently working on pattern match and cryptographic technologies.

References:
[1] www.tcpdump.org/>
[2] www.snort.org/
[3] www.wireshark.org/
[4] www.ntop.org/overview.html
[5] Deri Luca - Proceedings of SANE 2004, October 2004 - Improving Passive Packet Capture: Beyond Device Polling, 2004
[6] subversion.tigris.org/
[7] http://en.wikipedia.org/wiki/NAPI

Disclaimer: Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, go to the web link on performance limitations.