Friday, 19 May 2017

100G Ethernet NICs - Broadcom joins Mellanox, Cavium's QLogic, and Chelsio

New 100Gb Ethernet NICs announced last month from Broadcom bring the number of mainstream 100Gb Ethernet NIC vendors to four. There are a number of FPGA vendors also offering solutions for 100Gb Ethernet but here I've just decided to meander through the usual NIC vendors. Remember, if you want to trade at sub 100 nanoseconds, you'll have to avoid the PCIe bus transfer times and stick to FPGA tick to trade solutions.

Mellanox has been shipping 100GbE for around two years. Cavium's QLogic 100GbE NICs are a bit over a year old. Chelsio started with their 100GbE solution earlier this year.

Broadcom is due to soon ship both an Open Compute Project (OCP) Mezzanine Card 2.0 multi-host form factor (M1100PM) and the usual PCIe NIC form factor (P1100P).
OCP Mezzanine Card 2.0 form factor from the OCP specification
FreeBSD added support for Broadcom's 100GbE a couple of weeks ago, so the ducks are lining up for rollout. The BCM57454 chipset supports Nvidia's GPUDirect via RDMA for your HPC and ML enjoyment. Virtual switch and embedded security may take some load off your hosts. These are single port QSFP28 solutions.

Mellanox ConnectX-6 EN
(click to enlarge)
All the 100GbE vendors so far have settled on QSFP28. We'll have to wait a little longer for both Broadcom's multi-port 100GbE solutions, and a nod to PCIe 4.0. It is worth noting that PCIe 3 x16 does not support enough bandwidth for concurrent dual 100Gbps transfers. 

Most vendors' 100GbE solutions support versions of RoCE, RDMA, GPUDirect, et al, but there are a quite a lot of differences in the details. Such details may be important to you at 100GbE as your offloaded loads can help your CPUs out quite a bit. CPUs need help. 100Gbps is a lot of data for a little CPU to worry about, especially as packet loads climb to 200 million per second. Think about that. If you only had one CPU you'd only have 5ns per packet of processing time to keep up with that flow. We rely on offloaded help, direct memory access methods, and steering to multiple CPUs to stop from drowning.

Here is a list, for your convenience, of the current main network vendors' 100GbE solutions:

Mellanox

QLogic FastLinQ QL45611HLCU
(click to enlarge)

Cavium - QLogic

Chelsio

Broadcom


Solarflare makes excellent 10GbE and 40GbE solutions in both PCIe and OCP form factors. Solarflare and Mellanox are currently the "goto" NICs for low-latency trading. Hopefully, Solarflare will not be too far away from 100GbE delivery and be ready for exchanges reaching beyond their current 40Gbps maximum offerings.

Happy trading,

--Matt.

2 comments:

  1. are the connectX latencies for ethernet or infiniband?

    ReplyDelete
    Replies
    1. They are the numbers Mellanox publish for Ethernet. On Connect-X4 EN 100G there has been reported 60-90ns difference between IB and RoCE FWIW

      Delete