ibarnea

How TRex SR-IOV support can help you?

Blog Post created by ibarnea on Feb 1, 2017

Since version 2.15, TRex supports SR-IOV support for XL710 and X710. I would like to describe how we tested this, and the performance we have seen.

 

What is SR-IOV

SR-IOV (Single root IO virtualization) is a specification that allows a PCIe device to appear to be multiple separate physical PCIe devices. For network cards, this means the ability of one card to appear as if it was few separate cards called virtual functions (VFs), each with its own PCI address. This is particularly useful on virtual machines environments, where you have one NIC, and you want each VM to view it as if it is the only consumer.

There are many resources about SR-IOV in the web. The below link for example gives very good explanation:

FAQ for Intel® Ethernet Server Adapters with SR-IOV

4.png



5.png

 

How we used SR-IOV

Good tutorial about SR-IOV configuration on Linux can be found in the following link:

SR-IOV and PCI device passthrough traps & tricks | Linux kernel ramblings and more

We used Centos 7. The steps we had to do to make SR-IOV work were:

  1. Load the latest i40e driver source from here: Download Intel® Network Adapter Driver for PCIe* Intel® 40 Gigabit Network Connections Under Linux*
  2. Compile the driver, and install on your system
  3. Add following line: “options i40e max_vfs=x,y” to some file in “/etc/modprobe.d/”. x and y should be replaced with the number of VFs you want create for each port of the NIC (each PF). For example, putting 2,3 will create 2 VFs on one port, and 3 on the second.
  4. Add the following file /etc/sysconfig/modules/i40e.modules, with content as described below:
    #!/bin/sh
    rmmod i40e >/dev/null 2>&1
    exec /sbin/modprobe i40e >/dev/null 2>&1
  5. Reboot the system.


Now, when running:

sudo ./dpdk_setup_ports.py -s

You should see lines like this:

0000:04:02.0 'XL710/X710 Virtual Function' drv=igb_uio unused=i40evf,vfio-pci,uio_pci_generic

These are the ports that should be used in TRex config file.



TRex stateful performance

Using the following command, running on x710 card with VF driver, we can see that TRex can reach 30GBps, using only one core. We can also see that the average latency is around 20 usec, which is pretty much the same value we get on loopback ports with x710 physical function without VF.

sudo ./t-rex-64 -f cap2/http_simple.yaml -m 40000 -l 100 -c 1 –p


-Per port stats table

      ports |               0 |               1

-----------------------------------------------------------------------------------------

   opackets |       106573954 |       107433792

     obytes |     99570878833 |    100374254956

   ipackets |       107413075 |       106594490

     ibytes | 100354899813    |     99590070585

    ierrors |            1038 |            1027

    oerrors |               0 |               0

      Tx Bw |      15.33 Gbps |      15.45 Gbps

 

-Global stats enabled

Cpu Utilization : 91.5  %  67.3 Gb/core

Platform_factor : 1.0

Total-Tx :      30.79 Gbps

Total-Rx :      30.79 Gbps

Total-PPS :       4.12 Mpps

Total-CPS :     111.32 Kcps

 

Expected-PPS :       4.11 Mpps

Expected-CPS :     111.04 Kcps

Expected-BPS :      30.71 Gbps

 

Active-flows :    14651  Clients : 255   Socket-util : 0.0912 %

Open-flows :  5795073  Servers : 65535   Socket :    14652 Socket/Clients :  57.5

drop-rate :       0.00 bps

current time : 53.9 sec

test duration : 3546.1 sec

 

-Latency stats enabled

Cpu Utilization : 23.4 %

if| tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max window

   | ,        ,          , ,   average   , max  ,    (usec)

----------------------------------------------------------------------------------------------------------------

0 | 5233,    5233,         0, 0,         19  , 580,       5      | 37  37  37 40  39  38 36  26  37 38  36  37  27

1 | 5233,    5233,         0, 0,         22  , 577,       5      | 38  40  39 38  39  39 40  29  39 39  40  38  27

TRex stateless performance
Using stateless with one core with x710 VF:
sudo ./t-rex-64 -i -c 1

trex>portattr

Port Status

 

     port |          0           |          1

-------------------------------------------------------------

driver          | net_i40e_vf      |     net_i40e_vf

description     | XL710/X710 Virtual  |  XL710/X710 Virtual

 

With the console command:

start -f stl/imix.py -m 8mpps --force --port 0

We can see, that we can reach 8M packet per second, which in this case is around 24.28 Gbit/second.

 

Global Statistics

 

connection   : localhost, Port 4501                  total_tx_L2  : 24.28 Gb/sec

version      : v2.15 total_tx_L1  : 25.55 Gb/sec

cpu_util.    : 80.6% @ 1 cores (1 per port)          total_rx     : 24.28 Gb/sec

rx_cpu_util. : 66.8%                                 total_pps    : 7.99 Mpkt/sec

async_util.  : 0.18% / 1.84 KB/sec                   drop_rate    : 0.00 b/sec

queue_full   : 3,467 pkts

 

Port Statistics

 

   port    |         0         |         1         | total

----------------------------------------------------------------------

owner      |           ibarnea |           ibarnea |

link       |                UP |                UP |

state      | TRANSMITTING      |              IDLE |

speed      |           40 Gb/s |           40 Gb/s |

CPU util.  | 80.6%             |              0.0% |

--         |                   |                   |

Tx bps L2  | 24.28 Gbps        |          0.00 bps |        24.28 Gbps

Tx bps L1  | 25.55 Gbps        |             0 bps |        25.55 Gbps

Tx pps     | 7.99 Mpps         |          0.00 pps |         7.99 Mpps

Line Util. |           63.89 % |            0.00 % |

---        |                   |                   |

Rx bps     | 0.00 bps          |        24.28 Gbps |        24.28 Gbps

Rx pps     | 0.00 pps          |         7.99 Mpps |         7.99 Mpps

----       |                   |                   |

opackets   | 658532501         |                 0 |         658532501

ipackets   |                 0 |         658612569 |         658612569

obytes     | 250039721918      |                 0 |      250039721918

ibytes     |                 0 |      250070124150 |      250070124150

tx-bytes   | 250.04 GB         |               0 B |         250.04 GB

rx-bytes   |               0 B |         250.07 GB |         250.07 GB

tx-pkts    | 658.53 Mpkts      |            0 pkts |      658.53 Mpkts

rx-pkts    | 0 pkts            |      658.61 Mpkts |      658.61 Mpkts

-----      |                   |                   |

oerrors    |                 0 |                 0 |                 0

ierrors    |                 0 |            15,539 |            15,539

 

status:  /

 

Press 'ESC' for navigation panel...

 

For any issues, or questions, please respond here, or email the TRex dev team at our Google Groups


Thanks,

Ido

Outcomes