Skip navigation
All Places > DevNet > TRex > Blog


8 posts

Since version 2.15, TRex supports SR-IOV support for XL710 and X710. I would like to describe how we tested this, and the performance we have seen.


What is SR-IOV

SR-IOV (Single root IO virtualization) is a specification that allows a PCIe device to appear to be multiple separate physical PCIe devices. For network cards, this means the ability of one card to appear as if it was few separate cards called virtual functions (VFs), each with its own PCI address. This is particularly useful on virtual machines environments, where you have one NIC, and you want each VM to view it as if it is the only consumer.

There are many resources about SR-IOV in the web. The below link for example gives very good explanation:

FAQ for Intel® Ethernet Server Adapters with SR-IOV




How we used SR-IOV

Good tutorial about SR-IOV configuration on Linux can be found in the following link:

SR-IOV and PCI device passthrough traps & tricks | Linux kernel ramblings and more

We used Centos 7. The steps we had to do to make SR-IOV work were:

  1. Load the latest i40e driver source from here: Download Intel® Network Adapter Driver for PCIe* Intel® 40 Gigabit Network Connections Under Linux*
  2. Compile the driver, and install on your system
  3. Add following line: “options i40e max_vfs=x,y” to some file in “/etc/modprobe.d/”. x and y should be replaced with the number of VFs you want create for each port of the NIC (each PF). For example, putting 2,3 will create 2 VFs on one port, and 3 on the second.
  4. Add the following file /etc/sysconfig/modules/i40e.modules, with content as described below:
    rmmod i40e >/dev/null 2>&1
    exec /sbin/modprobe i40e >/dev/null 2>&1
  5. Reboot the system.

Now, when running:

sudo ./ -s

You should see lines like this:

0000:04:02.0 'XL710/X710 Virtual Function' drv=igb_uio unused=i40evf,vfio-pci,uio_pci_generic

These are the ports that should be used in TRex config file.

TRex stateful performance

Using the following command, running on x710 card with VF driver, we can see that TRex can reach 30GBps, using only one core. We can also see that the average latency is around 20 usec, which is pretty much the same value we get on loopback ports with x710 physical function without VF.

sudo ./t-rex-64 -f cap2/http_simple.yaml -m 40000 -l 100 -c 1 –p

-Per port stats table

      ports |               0 |               1


   opackets |       106573954 |       107433792

     obytes |     99570878833 |    100374254956

   ipackets |       107413075 |       106594490

     ibytes | 100354899813    |     99590070585

    ierrors |            1038 |            1027

    oerrors |               0 |               0

      Tx Bw |      15.33 Gbps |      15.45 Gbps


-Global stats enabled

Cpu Utilization : 91.5  %  67.3 Gb/core

Platform_factor : 1.0

Total-Tx :      30.79 Gbps

Total-Rx :      30.79 Gbps

Total-PPS :       4.12 Mpps

Total-CPS :     111.32 Kcps


Expected-PPS :       4.11 Mpps

Expected-CPS :     111.04 Kcps

Expected-BPS :      30.71 Gbps


Active-flows :    14651  Clients : 255   Socket-util : 0.0912 %

Open-flows :  5795073  Servers : 65535   Socket :    14652 Socket/Clients :  57.5

drop-rate :       0.00 bps

current time : 53.9 sec

test duration : 3546.1 sec


-Latency stats enabled

Cpu Utilization : 23.4 %

if| tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max window

   | ,        ,          , ,   average   , max  ,    (usec)


0 | 5233,    5233,         0, 0,         19  , 580,       5      | 37  37  37 40  39  38 36  26  37 38  36  37  27

1 | 5233,    5233,         0, 0,         22  , 577,       5      | 38  40  39 38  39  39 40  29  39 39  40  38  27

TRex stateless performance
Using stateless with one core with x710 VF:
sudo ./t-rex-64 -i -c 1


Port Status


     port |          0           |          1


driver          | net_i40e_vf      |     net_i40e_vf

description     | XL710/X710 Virtual  |  XL710/X710 Virtual


With the console command:

start -f stl/ -m 8mpps --force --port 0

We can see, that we can reach 8M packet per second, which in this case is around 24.28 Gbit/second.


Global Statistics


connection   : localhost, Port 4501                  total_tx_L2  : 24.28 Gb/sec

version      : v2.15 total_tx_L1  : 25.55 Gb/sec

cpu_util.    : 80.6% @ 1 cores (1 per port)          total_rx     : 24.28 Gb/sec

rx_cpu_util. : 66.8%                                 total_pps    : 7.99 Mpkt/sec

async_util.  : 0.18% / 1.84 KB/sec                   drop_rate    : 0.00 b/sec

queue_full   : 3,467 pkts


Port Statistics


   port    |         0         |         1         | total


owner      |           ibarnea |           ibarnea |

link       |                UP |                UP |

state      | TRANSMITTING      |              IDLE |

speed      |           40 Gb/s |           40 Gb/s |

CPU util.  | 80.6%             |              0.0% |

--         |                   |                   |

Tx bps L2  | 24.28 Gbps        |          0.00 bps |        24.28 Gbps

Tx bps L1  | 25.55 Gbps        |             0 bps |        25.55 Gbps

Tx pps     | 7.99 Mpps         |          0.00 pps |         7.99 Mpps

Line Util. |           63.89 % |            0.00 % |

---        |                   |                   |

Rx bps     | 0.00 bps          |        24.28 Gbps |        24.28 Gbps

Rx pps     | 0.00 pps          |         7.99 Mpps |         7.99 Mpps

----       |                   |                   |

opackets   | 658532501         |                 0 |         658532501

ipackets   |                 0 |         658612569 |         658612569

obytes     | 250039721918      |                 0 |      250039721918

ibytes     |                 0 |      250070124150 |      250070124150

tx-bytes   | 250.04 GB         |               0 B |         250.04 GB

rx-bytes   |               0 B |         250.07 GB |         250.07 GB

tx-pkts    | 658.53 Mpkts      |            0 pkts |      658.53 Mpkts

rx-pkts    | 0 pkts            |      658.61 Mpkts |      658.61 Mpkts

-----      |                   |                   |

oerrors    |                 0 |                 0 |                 0

ierrors    |                 0 |            15,539 |            15,539


status:  /


Press 'ESC' for navigation panel...


For any issues, or questions, please respond here, or email the TRex dev team at our Google Groups




TRex Stateless GUI v3.1

Posted by vogai Feb 1, 2017

TRex Stateless GUI application provides a graphical user interface for

GUI application provides capabilities to manage TRex, create different traffic profiles, run them it in TRex, create VM instructions. Also it has built-in packet crafting tool which allows to create any type of packet(it has the same limitation as Scapy python library).

TRex Port Management

GUI provides capabilities to acquire ports, run a traffic on a port, display port status and different statistic.

Screen Shot 2017-01-31 at 1.48.38 PM.png

Dashboard with global statistics provides detailed information for each port. Such as CPU Utilization, Input/Output traffic and so on.

Screen Shot 2017-01-31 at 3.44.58 PM.png

Profile management

GUI provides:

  • Import existing YAML profiles / export them to JSON or YAML formats.
  • Create a profile from scratch.
  • Create one or more streams for a given profile.
  • Specify profiles stream order.
  • Edit existing stream properties or create new ones.
  • Build a stream from existing PCAP file or from scratch (using advanced Stream builder).
  • Export a stream to PCAP format.

Screen Shot 2017-01-31 at 2.21.03 PM.png


Advanced mode with Packet Crafting Tool

Packet Crafting Tool has two main capabilities. Packet Editor and Field Engine. Packet editor allows user create and/or modify any type of packet which is supported by Scapy

Screen Shot 2017-01-31 at 2.30.54 PM.png

Packet editor's key features:

  • Specify only necessary fields
  • Generate random field values
  • "Lengths" and "checksums" could be automatically generated or explicitly specified.
  • Ability to create TRex VM instruction from predefined template.

Field Engine

Field Engine tab provide to user an easy way to create TRex VM instructions in current stream.

Screen Shot 2017-01-31 at 2.49.25 PM.png

Field Engine's key features:

  • Ability to create predefined instructions from Packet Builder
  • Smart variable name and packet offsets suggester.
  • Integrated help for each instruction

More features and theirs description you can find on project Wiki page

Videos instructions for basic Stateless GUI features

You can find some videos about basic features in this TRex Stateless GUI - YouTube playlist.


For next release we plan to support following features:

  • Per stream statistic
  • Neighboring Protocol
  • Traffic capturing
  • Debug Hardware counters



First of all you need configured TRex core and the easiest way to get it is install TRex VM image into VirtualBox locally. Here is the manual how to install TRex VM.


  • Configure VM:
    • Open VM Settings -> Network -> Advanced -> Port forwarding
    • Add VM port forwarding for ports: 4501(async), 4507(scapy-server)
  • Update trex:


To run TRex use following cli commands: "cd /opt/trex/v2.* ; sudo ./t-rex-64 -i"

You can download Stateless GUI v3.1 released build from here.


TRex Dev team

TRex is an open source, low cost, stateful and stateless traffic generator fuelled by DPDK. It is all about scale. It can scale up to 200-400Gbps,160MPPS and millions of flows using one Cisco UCS (or any COTS server).

This release contains many new features, fixes and improvements. Please find below a detailed description of what’s new in v2.15

  • Cisco VIC support
  • Mellanox ConnectX-4
  • SR-IOV support
  • Stateless GUI  with Packet builder
  • Stateful scale improvement

Cisco VIC support

  • Supported on both UCS C-series and B-series (blade server)
  • PCIe16 - 2x40GbE ports
  • Can be shared by VMs using vNIC (non SR-IOV)
  • Only 13xx series Cisco adapters are supported

see more here vic support


Mellanox ConnectX-4

  • Supports 25/50/100 GbE speeds
  • PCIe16 - 2x100GbE ports with one NIC
  • Require OFED kernel drivers
  • Small footprint
  • SR-IOV support
  • The Linux kernel driver is up while DPDK works
  • Performance numbers
    • Tx only - 64B, maximum 90MPPS
    • Tx and Rx - 64B, maximum ~50MPPS
    • IMIX  - maximum 90Gbps
    • CPU cycles per packet is up to 50% higher relative to Intel XL710 when checked with 64B packet size
    • In the average stateful scenario, ConnectX-4 is about the same as XL710.

          Stateful comparison of XL710 (trex08) vs. ConnextX-4 (trex07)


Stateless comparison of XL710 (trex08) vs. ConnextX-4 (trex07)  (cycle per packet in number of usecase)




SR-IOV support

TRex supports paravirtualized interfaces such as VMXNET3/virtio/E1000 however when connected to a vSwitch, the vSwitch limits the performance. VPP or OVS-DPDK can improve the performance but require more software resources to handle the rate.

SR-IOV can accelerate the performance and reduce CPU resource usage as well as latency by utilizing NIC hardware switch capability (the switching  is done by hardware).

TRex version 2.15 now includes SR-IOV support for XL710 and X710.

The following diagram compares between vSwitch and SR-IOV.




One use case which shows the performance gain that can be acheived by using SR-IOV is when a user wants to create a pool of TRex VMs that tests a pool of virtual DUTs (e.g. ASAv,CSR etc.)

When using newly supported SR-IOV, compute, storage and networking resources can be controlled dynamically (e.g by using OpenStack)





The above diagram is an example of one server with two NICS. TRex VMs can be allocated on one NIC while the DUTs can be allocated on another.


Stateless GUI  with Packet builder

Using the new stateless GUI advanced packet builder you can build any type of packet. It utilizes

TRex scapy server for building the packets and can be easily extended.







more information can be found under


Stateful scale improvement

Starting from version v2.15, TRex uses a new Stateful scheduler that works better in cases where the number of active flows is high. When tested with a typical EMIX traffic profile, a 70% performance improvement was observed.


In order to increase the maximum number of active flows that TRex can allocate the user should edit the dp_flows field in the TRex config file. The following picture shows and example of the config file.


Config File


The following CLI can be used along with the --active-flows switch in order to scale the number of flows:



This following chart shows the MPPS per core when using TRex with a traffic profile of small UDP flows (containing 10 packets of 64B each). PQ label relates to the old scheduler version whereas TW2 label relates to te latest scheduler version (TW0 an TW1 are results of different execution permutations). It can be seen that the new version behaves better in case of a high number of active-flows and in the common case (~30K-300K flows) there is a significant difference in performance.





The following chart shows extrapolation with average packet size of 600B. It can be seen that with the new scheduler version, TRex can scale up to 350Gbps with one UCS.




More information can be found under:


Please register to our TRex session that will be presented in the devnet zone  Cisco Live

Berlin 2017 in order to learn more on our latest enhancements and advanced capabilities.


Content Catalog - Cisco Live EMEA 2017



more resource can be found and





APIC EM and TRex collaboration is a representive example on how a softeware based traffic generator can adapt

to the needs of the network in almost zero time.


Using APIC EM controller for configuring your network might impose a challenge on how to test a complex topology.

This has led the APIC EM team to request TRex dev team to provide a way to test a more complex topology.


Let's take a look at the following topology:




We wish to test this toplogy with a range of clients communicating with a range of servers.

Testing this kind of a network topology requires a way to divide the range of the clients to the clusters by providing:


1. Different VLANs for both directions

2. Different MAC addresses for each cluster

3. A way to direct different amount of traffic to each cluster


In TRex we were able to adapt to the need in less than two weeks !

The solution is based on a well known standards such as YAML format for configuration:




TRex Stateless GUI was released

Posted by hhaim Jul 10, 2016

This is the message from Sari from Exalt-tech


"Hi All,

Please note that we have released a first early Beta version of the TRex Stateless GUI. This Beta version is still under development and continuous improvement  which includes source code and functionality enhancements / optimization.


You can find the latest binaries in the following GitHub repository:


In addition, you can find instructions to build the project from source or to install the binaries as well as the current supported features:


Your contribution to this project is more than welcome.





Meet us@Cisco Live Vegas

Posted by hhaim Jul 10, 2016

Meet us in Cisco Live US for Demo/Question/Roadmap






TRex Stateless support

Posted by hhaim May 22, 2016


TRex started as an open source, low cost, stateful traffic generator fuelled by DPDK. TRex  generates L7 traffic based on pre-processing and smart replay of real traffic templates. TRex amplifies both client and server side traffic and can scale up to 200Gb/sec with one UCS using Intel XL710.


Lately Stateless support was added, including the following feature set:

  • High scale –~10M-22MPPS/core
  • Support 1/10/25/40 Gb/sec interfaces
  • Profile can support multiple streams, scalable to 10K parallel streams
  • Supported for each stream
    • Packet template
    • Field engine program (src_ip =
    • Send Mode : Continues/Burst/Multi burst support
  • Interactive support – GUI/TUI
  • Statistic per port
  • Statistic per stream (by Hardware)
  • Latency, Jitter per stream
  • Fast Python automation support
    • Python 2.7/3.0 Client API
    • Python HLTAPI Client API
  • Multi-user support


The following diagram depicts an example of a Stateless traffic profile



If you want to get a glimpse of TRex, and see for yourself how dramatically it can improve your product, try the new Sandbox available at Cisco DevNet.


For further information, check out   TRex Documentation



TRex Dev team