DPDKSMTF-StackPAUSE instruction

Improving Effective CPU Utilization for Kernel Bypass Networking under SMT

Research resolving how DPDK's spin loop wastes other logical cores' resources under SMT, by selectively inserting the PAUSE instruction — improving a compute-intensive task's throughput by up to 18.5% without degrading the network application's own throughput (Kono Lab, Keio University — undergraduate thesis).

Background

DPDK achieves fast packet processing by having a Poll Mode Driver (PMD) poll the NIC queue directly, but that polling is implemented as a spin loop, occupying the CPU even while idle waiting for I/O. Under SMT, this steals resources from other logical cores sharing the same physical core, lowering effective CPU utilization. Running a spin loop alongside a compute-intensive task under SMT was confirmed to drop that task's throughput by 16% versus running it alone.

Proposed approach

Proposed inserting the PAUSE instruction (equivalent to cpu_relax) only when a poll finds no packet received. PAUSE is a hint instruction that tells the processor "this is a spin-wait," encouraging it to free up resources for other logical cores. Inserting it only on empty polls — not every iteration — avoids the packet-detection delay (and resulting throughput loss) that would come from pausing unconditionally, while still cutting wasted CPU cycles.

Experimental setup

On a server with SMT enabled (6-core Xeon E-2336), placed a DPDK network application and sysbench on logical cores of the same physical core, then measured both throughputs across varying send rates. DPDK applications tested: a self-built UDP echo server, f-stack nginx, and f-stack redis. Turbo Boost, P-State, and C-State were disabled to remove measurement noise from CPU state changes.

Results

The proposed method showed almost no throughput loss for the DPDK application itself (0.54% at worst), while the co-running sysbench's throughput improved: up to 18.5% for the echo server, 1.37% for f-stack nginx, and 1.14% for f-stack redis. Across all three, the improvement was clearly larger at lower network load — i.e., the more often PAUSE actually executed.

Discussion

The gap between the echo server and f-stack came down to how much each application itself consumes. The echo server is lightweight — just copying and resending packets — with a stable ~30μs latency. f-stack nginx/redis do much more complex processing, with latency starting above 450μs and climbing to nearly 300ms under heavy load. That heavier per-iteration work means more cycles per spin-loop pass, so PAUSE executes less often (fewer chances to hand resources to the other core), which is likely why sysbench saw a smaller benefit.

Future work

This experiment only ran two tasks — a DPDK application and sysbench. sysbench's CPU test is integer-arithmetic-heavy with almost no memory access or cache pressure, so it doesn't reproduce the resource contention a real data center workload would create. PAUSE's effectiveness is also known to vary by CPU microarchitecture, so verifying the approach on hardware beyond the Xeon Silver 4110 used here is still needed.