DPDK 1.7.1 on Linux
DPDK has had many updates from the last time I wrote about it in January this year. Once of the main issues in the earlier build of DPDK (1.7.0), was SSE4 when building onc processors that did not support it.
The prep and build is exactly as my old notes, with some minor changes. These are my notes on building DPDK on x86_64 Debian 7.3. DPDK has many configurable options like UIO support, HUGETLBFS, PROC_PAGE_MONITOR, HPET etc. I only talk about how to quickly build DPDK, for more details regarding the above options, refer to the DPDK guide.
Install necessary packages
sudo apt-get install make coreutils gcc binutils
Install the kernel headers
sudo apt-get install linux-headers-`uname -r`
apt sometimes does not create proper symlinks for the kernel headers.
Make sure that you have the following in place:
user@kerneldev:~/dpdk-1.5.2r1$ ls -l /usr/src/linux-headers-`uname -r` total 764 drwxr-xr-x 3 root root 4096 Jan 24 06:16 arch drwxr-xr-x 5 root root 4096 Jan 24 06:16 include -rw-r--r-- 1 root root 295 Sep 19 23:46 Makefile -rw-r--r-- 1 root root 766891 Sep 19 23:25 Module.symvers lrwxrwxrwx 1 root root 34 Sep 19 23:46 scripts -> ../../lib/linux-kbuild-3.2/scripts
Also, reset your shell or logoff/login.
Once the dependencies are all taken care of, building DPDK should be fairly easy.
tar xf dpdk.tar.gz cd dpdk
make config T=x86_64-native-linuxapp-gcc make
That should complete smooth without any issues.
Sometimes, you will see issues like this:
== Build lib/librte_eal/linuxapp/igb_uio make: *** No targets specified and no makefile found. Stop. make: *** [igb_uio.ko] Error 2 make: *** [igb_uio] Error 2 make: *** [linuxapp] Error 2 make: *** [librte_eal] Error 2 make: *** [lib] Error 2 make: *** [all] Error 2 make: *** [x86_64-default-linuxapp-gcc_install] Error 2 make: *** [install] Error 2
This happens either because you dont have the proper kernel headers installed (version mismatch) or if apt could not create proper symlinks for you.
Also, i would recommend you repeat your install with the exact instructions as above
This is straight from the Intel documentation:
|e1000e||82571..82574, 82583, ich8..ich10, pch..pch2|
|igb||82575..82576, 82580, i210, i211, i350, i354, dh89xxcc|
So 82545EM, would be using the e1000 driver
The kernel modules built are available in the build/kmod directory.
Loading the kernel module
Load the kernel module:
modprobe uio insmod build/kmod/igb_uio.ko
To make this all work, we need to unbind the kernel driver for the interface and bind it to the igb_uio driver. DPDK provides a python script (tools/dpdk_nic_bind.py) to bind and unbind interfaces to drivers. You need to be root to run this script.
Check the current binding status:
root@kerneldev:/home/kbandla/dpdk-1.5.2r1# python tools/dpdk_nic_bind.py --status Network devices using IGB_UIO driver ==================================== <none> Network devices using kernel driver =================================== 0000:02:01.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth0 drv=e1000 unused=igb_uio *Active* Other network devices ===================== <none>
Bind the eth0 interface to the IGB_UIO driver:
root@kerneldev:/home/kbandla/dpdk-1.5.2r1# python tools/dpdk_nic_bind.py --bind=igb_uio 0000:02:01.0 Routing table indicates that interface 0000:02:01.0 is active. Not modifying
The interface was not bound to the driver because the interface is active. Bring down the interface and re-try and everything should be okay
mkdir -p /mnt/huge mount -t hugetlbfs nodev /mnt/huge echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
To manage cpu-scaling, you need to install the cpufrequtils package. Follow the instructions on the Debian wiki here
The available options are userspace conservative powersave ondemand performance. DPDK will use the 'performance' option and set the processor at the highest frequency. DPDK documentation uses the following way to do it:
for gov in /sys/devices/system/cpu/*/cpufreq/scaling_governor ; do echo performance >$gov ; done
You can use other ways to clock your CPU to the highest frequency.
Testpmd and Test
Once you have set hugepages, you can test your build using the testmpd program.
You might see this error:
EAL: pthread_setaffinity_np failed PANIC in eal_thread_loop(): cannot set affinity
This happens because you might not have sufficient number of cores. You might have to set the core mask appropriately.
Hugepages is an important part of the DPDK setup. The perfect values for hugepages really depends on various things (CPU, memory etc). You can setup hugepages atleast two different ways.
Make hugepages available to DPDK
mkdir /mnt/huge mount -t hugetlbfs nodev /mnt/huge
Add this to /etc/fstab:
nodev /mnt/huge hugetlbfs defaults 0 0
For 1GB pages:
nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0