Trex cannot effectively use 400G network cards

[da1234cao]

Environment Introduction

I have a Mellanox connectx-7 NIC which is 400Gbps.

root@~# mlxlink -d 0000:c1:00.0

Operational Info
----------------
State                              : Active 
Physical state                     : LinkUp 
Speed                              : 400G 
Width                              : 4x 
FEC                                : Standard_RS-FEC - (544,514) 
Loopback Mode                      : No Loopback 
Auto Negotiation                   : ON 

Supported Info
--------------
Enabled Link Speed (Ext.)          : 0x0001bff2 (400G_4X,400G_8X,200G_2X,200G_4X,100G_1X,100G_2X,100G_4X,50G_1X,50G_2X,40G,25G,10G,1G) 
Supported Cable Speed (Ext.)       : 0x00012800 (400G_4X,200G_2X,100G_1X) 

Troubleshooting Info
--------------------
Status Opcode                      : 0 
Group Opcode                       : N/A 
Recommendation                     : No issue was observed 

Tool Information
----------------
Firmware Version                   : 28.43.1014 
amBER Version                      : 3.3 
MFT Version                        : mft 4.29.0-131 

Problem Description

However, the output information of trex indicates that its speed is 200G.

port : 1 
------------
link         :  link : Link Up - speed 200000 Mbps - full-duplex

I looked at the source code and I think the problem might be here.

When the network card is larger than 200G, the maximum is 200G.

// src/trex_port_attr.h
virtual uint32_t get_link_speed() { return m_link.link_speed < RTE_ETH_SPEED_NUM_200G? m_link.link_speed: RTE_ETH_SPEED_NUM_200G; } // L1 Mbps

In actual use, with a 400G network card, Trex can only send about 217G.

Currently, is there any way I can bypass the 200G limit and let Trex fully utilize the 400G capacity of the network card?

[noble778]

ok，en....，that's all right。

[rohithkumar7582]

I have tried running trex two CX7 Nvidia Mellanox NIC cards attached to a DELL server. Each NIC card (single port) can support 400Gbps Tx+Rx and when checked with mlnx tools, we see 400Gbps link in the vendor information. But the ethtool (v4.8) displays a max of 200000 Mbps as the max link speed and not 400000 Mbps.

But surprisingly, I was able to simulate around 400Gbps Tx-Rx both incoming and outgoing. Whereas, when I ran it in client-only mode to max of 400Gbps (Tx only) per port, the max we could simulate was 392Gbps outgoing in STF mode. There seems to be a loss of 8Gbps or approx 2% drop on the NIC card.

On the ASTF mode, the max throughput per port simulated was 200 Gbps which was limited by the ethtool itself. On using ethtool v6.x, I was able to see a 400Gbps link attributes. @hhaim , could you let us know like in what all modes were the Connect CX7 NIC cards were qualified and perf numbers if any?

Interface link mode output from ethtool:

Settings for NIC interface:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                .
                                .
                                200000baseKR4/Full
                                200000baseSR4/Full
                                200000baseLR4_ER4_FR4/Full
                                200000baseDR4/Full
                                200000baseCR4/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None BaseR RS
        Advertised link modes:  1000baseT/Full
                                .
                                .
                                200000baseKR4/Full
                                200000baseSR4/Full
                                200000baseLR4_ER4_FR4/Full
                                200000baseDR4/Full
                                200000baseCR4/Full ----------------------> capped at 200Gbps
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: RS
        Speed: **400000Mb/s**
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: yes

Trex console logs after jumping into interactive ASTF mode:

./trex-console

Using 'python3' as Python interpeter


Connecting to RPC server on localhost:4501                   [SUCCESS]


Connecting to publisher server on localhost:4500             [SUCCESS]

[SUCCESS]


Acquiring ports [0, 1]:                                      [SUCCESS]


Server Info:

Server version:   v3.06 @ ASTF
Server mode:      Advanced Stateful
Server CPU:       1 x Intel(R) Xeon(R) Gold 6438N
Ports count:      **2 x 200Gbps** @ MT2910 Family [ConnectX-7] ----------> shows 2 x 200 instead of 2 x 400

[PanPaskuda]

Hi @da1234cao I have the same issue, but it seems that adding --force workaround that problem and you can pass 400G as bpsL1 either via api:
trex_client.start(ports=[0], force=True, mult="400000mbps", total=False, synchronized=False, duration=1)
or via cmd:
trex>start --force -f /path/to/teststream.py -m 400000mbpsl1

[da1234cao]

Thank you.

After testing, only the get_link_speed function returned 200G.

However, the actual network speed can reach 390G+.

The previous test showed a network speed of 217G, which was my mistake. My CPU doesn't support PCIe 5.0. After switching to a machine that supports PCIe 5.0, the network speed has reached 390G+.