DPDK 1.7.1 on Linux

DPDK has had many updates from the last time I wrote about it in January this year. Once of the main issues in the earlier build of DPDK (1.7.0), was SSE4 when building onc processors that did not support it.

The prep and build is exactly as my old notes, with some minor changes. These are my notes on building DPDK on x86_64 Debian 7.3. DPDK has many configurable options like UIO support, HUGETLBFS, PROC_PAGE_MONITOR, HPET etc. I only talk about how to quickly build DPDK, for more details regarding the above options, refer to the DPDK guide.

Dependencies

Install necessary packages

sudo apt-get install make coreutils gcc binutils

Install the kernel headers

sudo apt-get install linux-headers-`uname -r`

apt sometimes does not create proper symlinks for the kernel headers.

Make sure that you have the following in place:

user@kerneldev:~/dpdk-1.5.2r1$ ls -l /usr/src/linux-headers-`uname -r`
total 764
drwxr-xr-x 3 root root   4096 Jan 24 06:16 arch
drwxr-xr-x 5 root root   4096 Jan 24 06:16 include
-rw-r--r-- 1 root root    295 Sep 19 23:46 Makefile
-rw-r--r-- 1 root root 766891 Sep 19 23:25 Module.symvers
lrwxrwxrwx 1 root root     34 Sep 19 23:46 scripts -> ../../lib/linux-kbuild-3.2/scripts

Also, reset your shell or logoff/login.

Building DPDK

Once the dependencies are all taken care of, building DPDK should be fairly easy.

Extract:

tar xf dpdk.tar.gz
cd dpdk

Build:

make config T=x86_64-native-linuxapp-gcc
make

That should complete smooth without any issues.

Issues

Sometimes, you will see issues like this:

== Build lib/librte_eal/linuxapp/igb_uio
make[8]: *** No targets specified and no makefile found.  Stop.
make[7]: *** [igb_uio.ko] Error 2
make[6]: *** [igb_uio] Error 2
make[5]: *** [linuxapp] Error 2
make[4]: *** [librte_eal] Error 2
make[3]: *** [lib] Error 2
make[2]: *** [all] Error 2
make[1]: *** [x86_64-default-linuxapp-gcc_install] Error 2
make: *** [install] Error 2

This happens either because you dont have the proper kernel headers installed (version mismatch) or if apt could not create proper symlinks for you.

Also, i would recommend you repeat your install with the exact instructions as above

Compatible devices

This is straight from the Intel documentation:

So 82545EM, would be using the e1000 driver

Testing

The kernel modules built are available in the build/kmod directory.

Loading the kernel module

Load the kernel module:

modprobe uio
insmod build/kmod/igb_uio.ko

Binding

To make this all work, we need to unbind the kernel driver for the interface and bind it to the igb_uio driver. DPDK provides a python script (tools/dpdk_nic_bind.py) to bind and unbind interfaces to drivers. You need to be root to run this script.

Check the current binding status:

root@kerneldev:/home/kbandla/dpdk-1.5.2r1# python tools/dpdk_nic_bind.py --status

Network devices using IGB_UIO driver
====================================
<none>

Network devices using kernel driver
===================================
0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth0 drv=e1000 unused=igb_uio *Active*

Other network devices
=====================
<none>

Bind the eth0 interface to the IGB_UIO driver:

root@kerneldev:/home/kbandla/dpdk-1.5.2r1# python tools/dpdk_nic_bind.py --bind=igb_uio 0000:02:01.0
Routing table indicates that interface 0000:02:01.0 is active. Not modifying

The interface was not bound to the driver because the interface is active. Bring down the interface and re-try and everything should be okay

Set hugepages

mkdir -p /mnt/huge
mount -t hugetlbfs nodev /mnt/huge
echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

Install cpufrequtils

To manage cpu-scaling, you need to install the cpufrequtils package. Follow the instructions on the Debian wiki here

The available options are userspace conservative powersave ondemand performance. DPDK will use the 'performance' option and set the processor at the highest frequency. DPDK documentation uses the following way to do it:

for gov in /sys/devices/system/cpu/*/cpufreq/scaling_governor ; do echo performance >$gov ; done

You can use other ways to clock your CPU to the highest frequency.

Testpmd and Test

Once you have set hugepages, you can test your build using the testmpd program.

You might see this error:

EAL: pthread_setaffinity_np failed
PANIC in eal_thread_loop():
cannot set affinity

This happens because you might not have sufficient number of cores. You might have to set the core mask appropriately.

Configure Hugepages

Hugepages is an important part of the DPDK setup. The perfect values for hugepages really depends on various things (CPU, memory etc). You can setup hugepages atleast two different ways.

Make hugepages available to DPDK

mkdir /mnt/huge
mount -t hugetlbfs nodev /mnt/huge

Add this to /etc/fstab:

nodev /mnt/huge hugetlbfs defaults 0 0

For 1GB pages:

nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0

References

1. Debian Linux Kernel Handbook
2. Intel DPDK getting started guide
3. DPDK Quick start
4. CPU frequency scaling on Debian