Reference:
https://docs.ebpf.io/
When building high-frequency trading systems, real-time audio pipelines, or distributed low-latency services, the Linux kernel's standard TCP/IP networking stack can become a major bottleneck. Every packet copy, interrupt moderation delay, and kernel-space traversal adds precious microseconds of jitter.
In this post, we explore how to optimize your network interface cards (NICs) by scaling ring buffers, disabling interrupt coalescing, and bypassing the kernel entirely using modern zero-copy primitives like AF_XDP.
1. Maximizing NIC Ring Buffers (ethtool)
Network interface cards use circular ring buffers (queues of buffer descriptors) to handle incoming (RX) and outgoing (TX) packets. Under heavy traffic bursts, default ring sizes (often 256 or 512 descriptors) fill up instantly, causing dropped packets and costly retransmissions.
Scaling Ring Sizes
Check your NIC's maximum supported ring limits and current settings:
ethtool -g eth0
sudo ethtool -G eth0 rx 4096 tx 4096Eliminating Interrupt Coalescing
By default, NIC drivers use interrupt coalescing—delaying hardware interrupts for a few microseconds (rx-usecs) to batch packets and save CPU cycles. For low-latency systems, this delay is unacceptable. To force immediate interrupt delivery upon packet arrival:
sudo ethtool -C eth0 rx-usecs 0 rx-frames 12. Kernel Socket Bypass Technologies
Standard socket calls (recv(), send()) force packets to copy through the Linux kernel's networking stack, incurring context switches and memory allocation overhead. Kernel bypass technologies eliminate this by mapping NIC ring buffers directly into user space.
Native Zero-Copy with AF_XDP
AF_XDP is an address family built directly into the Linux kernel specifically optimized for high-performance packet processing.
How it works: An unprivileged application allocates an aligned memory region (
UMEM), registers it with the kernel, and maps kernel ring descriptors (fill_ring,rx_ring) into user space usingmmap.The Result: The NIC driver performs DMA transfers directly into user-space buffers, allowing your application to poll and process packets instantly with zero syscall overhead or memory copying.
Alternative Enterprise Bypass (DPDK & Onload)
For sub-microsecond HFT environments, standard OS primitives are often replaced entirely:
DPDK (Data Plane Development Kit): Replaces the kernel network driver with a poll-mode driver (PMD) running in a tight user-space polling loop.
Solarflare OpenOnload: Transparently bypasses the OS kernel stack via user-space virtual interfaces (
EF_VI) to accelerate standard TCP/UDP socket calls.
Summary Blueprint for Low-Latency Networking
Enlarge Ring Buffers: Maximize RX/TX ring sizes with
ethtool -Gto prevent buffer overflows under burst loads.Disable Coalescing: Set
rx-usecs 0to fire hardware interrupts immediately.Deploy Kernel Bypass: Use
AF_XDPor DPDK to map NIC queues directly into user space, eliminating kernel stack traversal and memory copying overhead.
#include <iostream>
#include <vector>
#include <cstring>
#include <unistd.h>
#include <poll.h>
#include <sys/socket.h>
#include <linux/if_xdp.h>
// Dummy structures representing your mapped ring buffers and context
// (Assumes socket, rings, and UMEM have already been initialized and mmap'd as shown previously)
struct XdpPollerContext {
int xsk_fd = -1;
// Ring pointers mapped from kernel via mmap()
// In production, these point to the actual mmap regions for fill, rx, tx, and comp rings.
void* umem_area = nullptr;
// Tracking free frame indices for the UMEM buffer
std::vector<__u64> free_frames;
};
// Helper to reserve and push free buffers into the Fill Ring
void replenish_fill_ring(int xsk_fd, struct xdp_ring* fill_ring, std::vector<__u64>& free_frames) {
__u32 index;
// Reserve slots in the fill ring
unsigned int rcvd = xsk_ring_prod__reserve(fill_ring, free_frames.size(), &index);
if (rcvd == 0) return;
for (unsigned int i = 0; i < rcvd; ++i) {
*xsk_ring_prod__fill_addr(fill_ring, index++) = free_frames.back();
free_frames.pop_back();
}
// Submit the filled entries to the kernel
xsk_ring_prod__submit(fill_ring, rcvd);
}
// Main packet processing loop
void poll_rx_packets(XdpPollerContext& ctx, struct xdp_ring* rx_ring, struct xdp_ring* fill_ring) {
struct pollfd pfd = {};
pfd.fd = ctx.xsk_fd;
pfd.events = POLLIN;
std::cout << "Starting AF_XDP packet polling loop...\n";
bool running = true;
while (running) {
// 1. Ensure the Fill ring has available buffers for incoming packets
if (!ctx.free_frames.empty()) {
// (Assuming helper or direct ring producer macros are used)
// replenish_fill_ring(ctx.xsk_fd, fill_ring, ctx.free_frames);
}
// 2. Poll the socket for incoming traffic (blocks until packets arrive or timeout)
int ret = poll(&pfd, 1, 1000); // 1-second timeout
if (ret < 0) {
if (errno == EINTR) continue;
std::cerr << "Poll error: " << std::strerror(errno) << "\n";
break;
} else if (ret == 0) {
// Timeout - no packets received in this interval
continue;
}
// 3. Check the RX ring for completed packet descriptors
// unsigned int rcvd = xsk_ring_cons__peek(rx_ring, 32, &idx);
// if (rcvd > 0) {
// for (unsigned int i = 0; i < rcvd; ++i) {
// const struct xdp_desc* desc = xsk_ring_cons__rx_desc(rx_ring, idx++);
//
// // Access packet data directly in user-space UMEM area!
// void* packet_data = (char*)ctx.umem_area + desc->addr;
// uint32_t packet_len = desc->len;
//
// // Process packet here...
// std::cout << "Received packet of size: " << packet_len << " bytes\n";
//
// // After processing, add desc->addr back to free_frames to recycle the UMEM chunk
// ctx.free_frames.push_back(desc->addr);
// }
//
// // Release consumed descriptors back to the RX ring
// xsk_ring_cons__release(rx_ring, rcvd);
// }
}
}
High-Performance Networking: Deep Dive into XDP and AF_XDP
When building ultra-low latency systems, high-throughput firewalls, or custom load balancers, the Linux kernel's standard TCP/IP networking stack can become a major bottleneck. Every packet copy, interrupt moderation delay, and kernel-space traversal adds precious microseconds of overhead.
Here is a focused, end-to-end breakdown of XDP and AF_XDP, covering how they work, when to use them, and how to implement zero-copy packet processing in C++.
1. What is XDP?
XDP stands for eXpress Data Path. It is a programmable, high-performance packet processing framework built directly into the Linux kernel network stack, operating via eBPF (Extended Berkeley Packet Filter) programs attached directly to the network interface card (NIC) driver.
Earliest Possible Hook: XDP runs eBPF code the exact moment a packet is received by the NIC driver—long before the kernel allocates an
sk_buffmemory structure or traverses the heavy TCP/IP stack.Line-Rate Execution: Capable of processing millions of packets per second (PPS) for tasks like DDoS mitigation, packet filtering, and early routing.
2. What is AF_XDP and When Should You Choose It?
AF_XDP (Address Family XDP) is a specialized socket family designed for high-speed packet I/O that bridges the gap between raw kernel-bypass frameworks and standard Linux sockets.
When to Choose AF_XDP:
Extreme Packet Rates: Your application needs to ingest or filter millions of packets per second where standard sockets (
AF_INET/AF_PACKET) saturate the CPU.Coexistence with the Linux Kernel: Unlike legacy kernel-bypass frameworks (like DPDK) which completely take over the NIC driver,
AF_XDPcoexists with the Linux kernel. Regular management traffic, SSH, and standard tools (tcpdump,iptables) continue to run seamlessly on the same interface.Zero-Copy Performance: You want bare-metal DMA (Direct Memory Access) speeds—where the NIC writes incoming packets directly into your user-space memory buffers (
UMEM) with zero copying.
3. Tight Busy-Polling Loop (xsk_ring_cons)
To process packets with absolute minimum latency without the context-switch overhead of poll(), use an unblocked while(true) busy-polling loop on your consumer ring (xsk_ring_cons), pinned to an isolated CPU core:
#include <linux/if_xdp.h>
void tight_busy_poll_loop(struct xsk_ring_cons* rx_ring,
struct xsk_ring_prod* fill_ring,
int xsk_fd,
void* umem_area) {
constexpr uint32_t BATCH_SIZE = 64;
uint32_t rx_idx = 0;
while (true) {
// 1. Non-blocking peek: Check for new packet descriptors in the RX ring
uint32_t rcvd = xsk_ring_cons__peek(rx_ring, BATCH_SIZE, &rx_idx);
if (rcvd > 0) {
for (uint32_t i = 0; i < rcvd; ++i) {
const struct xdp_desc* desc = xsk_ring_cons__rx_desc(rx_ring, rx_idx++);
// Zero-copy access to packet payload in registered UMEM
void* packet_data = static_cast<char*>(umem_area) + desc->addr;
uint32_t packet_len = desc->len;
// Process packet instantly...
(void)packet_data;
(void)packet_len;
}
// 2. Release consumed descriptors back to the RX ring
xsk_ring_cons__release(rx_ring, rcvd);
} else {
// 3. Handle kernel wakeup requirement if driver is sleeping
if (xsk_ring_prod__needs_wakeup(fill_ring)) {
char dummy;
recvfrom(xsk_fd, &dummy, 0, MSG_DONTWAIT, nullptr, nullptr);
}
}
}
}Simplifying High-Performance AF_XDP Networking with libbpf Helpers
Writing raw AF_XDP code in C++ for ultra-low latency networking is powerful, but it traditionally comes with a steep learning curve and a mountain of boilerplate. Manually allocating page-aligned memory for UMEM, configuring multiple socket options via setsockopt, calculating ring offsets with mmap, and binding queues can quickly clutter your codebase.
Fortunately, libbpf provides clean, high-level helper functions—namely xsk_umem__create() and xsk_socket__create()—that drastically streamline AF_XDP setup.
The Power of libbpf XSK Helpers
Instead of manually wrestling with low-level kernel descriptors and memory mapping, libbpf wraps the entire initialization ritual into clean library APIs:
xsk_umem__create(): Handles raw UMEM registration (XDP_UMEM_REG), memory allocation mapping offsets, and automatically sets up your Fill and Completion rings.xsk_socket__create(): Binds the AF_XDP socket to your target network interface and hardware queue while initializing your RX and TX consumer/producer rings.
Clean C++ Implementation
Here is how straightforward it is to initialize an AF_XDP session using libbpf helpers:
#include <iostream>
#include <vector>
#include <stdexcept>
#include <cstring>
#include <unistd.h>
#include <sys/mman.h>
#include <bpf/xsk.h>
struct XdpSession {
struct xsk_umem* umem = nullptr;
struct xsk_socket* xsk = nullptr;
void* umem_buffer = nullptr;
struct xsk_ring_prod fill_ring;
struct xsk_ring_cons comp_ring;
struct xsk_ring_cons rx_ring;
struct xsk_ring_prod tx_ring;
~XdpSession() {
if (xsk) xsk_socket__delete(xsk);
if (umem) xsk_umem__delete(umem);
if (umem_buffer) free(umem_buffer);
}
};
XdpSession init_af_xdp(const char* ifname, int queue_id) {
XdpSession session;
// 1. Allocate page-aligned UMEM memory area (4096 frames of 4096 bytes)
const size_t frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE;
const size_t num_frames = 4096;
const size_t umem_size = frame_size * num_frames;
if (posix_memalign(&session.umem_buffer, getpagesize(), umem_size) != 0) {
throw std::runtime_error("Failed to allocate aligned memory for UMEM");
}
// 2. Create and register UMEM using libbpf
struct xsk_umem_config umem_cfg = {
.fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
.comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
.frame_size = (unsigned int)frame_size,
.frame_headroom = 0,
.flags = 0
};
int ret = xsk_umem__create(&session.umem,
session.umem_buffer,
umem_size,
&session.fill_ring,
&session.comp_ring,
&umem_cfg);
if (ret) {
throw std::runtime_error("Failed to create xsk umem: " + std::string(std::strerror(-ret)));
}
// 3. Create the AF_XDP socket
struct xsk_socket_config xsk_cfg = {
.rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS,
.tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS,
.libbpf_flags = XSK_SLIBBPF_FLAGS__INHIBIT_PROG_LOAD,
.xdp_flags = 0,
.bind_flags = XDP_USE_NEED_WAKEUP
};
ret = xsk_socket__create(&session.xsk,
ifname,
queue_id,
session.umem,
&session.rx_ring,
&session.tx_ring,
&xsk_cfg);
if (ret) {
throw std::runtime_error("Failed to create xsk socket: " + std::string(std::strerror(-ret)));
}
std::cout << "Successfully created libbpf AF_XDP socket on " << ifname << " queue " << queue_id << "\n";
return session;
}Summary
By leveraging libbpf's built-in XSK helpers, you eliminate dozens of lines of complex socket option and memory mapping code. This lets you jump straight into what matters most: writing high-performance, zero-copy packet processing loops for your low-latency systems.
