Showing posts with label cpp_low_latency. Show all posts
Showing posts with label cpp_low_latency. Show all posts

Sep 12, 2026

C++ template code COMDAT folding

Reference:
Reducing C++ template bloat by factoring out the type-dependent portions of the function


COMDAT folding (also known as Identical Code Folding or ICF/Safe ICF) is a link-time optimization where the linker detects functions or read-only data sections that compile into byte-for-byte identical machine code, merges them into a single instance, and points all call sites to that unified copy.

When instantiates templates over types that share the same underlying memory layout and operations, the compiler often emits redundant assembly:

template <typename T>
void push_item(T* item) {
    // Operations on item...
}

// In translation unit A:
push_item<Dog>(dog_ptr);

// In translation unit B:
push_item<Cat>(cat_ptr);
Because Dog* and Cat* are both raw machine pointers (typically 8 bytes), the generated machine instructions for push_item and push_item are identical. 

Normally, C++ templates emit code into COMDAT sections (Common Data sections) with "link-once" semantics (select any). This allows the linker to discard duplicates of push_item compiled across multiple .cpp files. However, normal deduplication only matches identical symbol names. It cannot eliminate the identical assembly between push_item and push_item because their mangled symbol names differ.

How COMDAT Folding Works

During the link step, the linker inspects all candidate sections:

Content Comparison: The linker compares the byte streams, relocation targets, and alignment requirements of functions marked as foldable (typically COMDAT sections generated by inline functions, template instantiations, and virtual tables).

Section Merging: If two distinct functions (e.g., std::vector<int*>::size() and std::vector<char*>::size()) produce identical instructions and reference identical offsets, the linker discards one function body.

Symbol Redirection: The symbol table entry for the discarded function is updated to point directly to the entry point of the retained function.

Key Benefits

Reduced Binary Footprint: Prevents template-heavy code (like std::vector<T*> for hundreds of pointer types) from inflating executable size.

Instruction Cache (I-Cache) Efficiency: Multiple logically distinct types share hot cache lines instead of thrashing the instruction cache with duplicate code.


The Function Pointer Quirk

Under strict C++ rules, every distinct function must have a unique address:

assert(&push_item<Dog> != &push_item<Cat>);

When COMDAT folding collapses these functions, &push_item<Dog> == &push_item<Cat> evaluates to true. This can break code that relies on function pointer uniqueness for type-tagging or callback registries.

To handle this, linkers provide different safety levels:
LLVM (lld)  --icf=safe Only merges functions whose addresses are never taken, preserving C++ address-uniqueness guarantees.
LLVM (lld)  --icf=all Aggressively merges all identical functions, even if addresses are taken.


Instead of generating redundant machine code for hundreds of pointer instantiations and hoping the linker cleans it up with COMDAT folding / ICF, standard library implementations and runtime systems use type erasure with non-templated (or void-pointer-based) base classes.

By shifting the heavy procedural logic into a shared base class, the exposed template becomes a thin, inline wrapper that carries zero runtime binary overhead.

The Architecture: Base-Derived Split

The pattern divides a container or utility into two layers:

The Erased Base Class: Implements memory allocation, capacity resizing, buffer shifts, and element index arithmetic using void* or raw byte buffers. This code is compiled once into the runtime library or translation unit.

The Typed Template Wrapper: Derives from or wraps the base class. It only exposes strongly typed interfaces, using zero-cost casts (reinterpret_cast or static_cast) to translate between T* and void*.


// --- Shared Implementation (compiled once into binary/lib) ---
class VectorPtrBase {
protected:
    void** data_ = nullptr;
    size_t size_ = 0;
    size_t capacity_ = 0;

    void grow_and_insert(size_t index, void* element) {
        // All heavy buffer reallocation, index shifting,
        // and boundary checks happen HERE once.
        if (size_ == capacity_) {
            size_t new_cap = capacity_ == 0 ? 8 : capacity_ * 2;
            void** new_data = new void*[new_cap];
            for (size_t i = 0; i < size_; ++i) new_data[i] = data_[i];
            delete[] data_;
            data_ = new_data;
            capacity_ = new_cap;
        }
        data_[index] = element;
        ++size_;
    }

    void* get_element(size_t index) const {
        return data_[index];
    }
};

// --- Thin Typed Wrapper (specialization for any pointer type) ---
template <typename T>
class Vector<T*> : private VectorPtrBase {
public:
    void push_back(T* val) {
        // Zero-cost static cast; inlines down to a direct call to the base
        grow_and_insert(size_, static_cast<void*>(val));
    }

    T* operator[](size_t index) const {
        return static_cast<T*>(get_element(index));
    }

    size_t size() const { return size_; }
};

Why This Beats Relying Purely on COMDAT Folding

Faster Compilation Times: The compiler does not have to parse, instantiate, type-check, and generate intermediate representation (IR) / assembly for grow_and_insert across Vector<Apple*>, Vector<Banana*>, and Vector<Car*>.

Lower Linker Overhead: Linker-time ICF requires analyzing section hashes, inspecting instruction bytes, and walking relocation tables to prove equivalence. Pointer erasure eliminates the duplicate sections before the object files reach the linker.

Guaranteed Code Sharing: COMDAT folding is sensitive to subtle differences (such as debug information emission, compiler optimization levels, or platform-specific pointer calling conventions). The base-class approach enforces code sharing by construction.

Preserved Pointer Address Guarantees: Because the wrapper's member functions inline entirely into the caller or delegate to the shared base, it avoids issues where --icf=safe refuses to fold functions whose addresses were taken.

Sep 11, 2026

[left-right datastructure] The Cost of Concurrency Coordination

I used to use this dual-copy/pointer swap trick for Linkedin's ATS server.

Reference:
The Cost of Concurrency Coordination with Jon Gjengset
[low latency] What is low latency (definition)

In concurrent systems programming, conventional wisdom often reduces synchronization performance to simple rules of thumb:

  • "Mutexes are slow because threads get descheduled."
  • "If your workload is read-heavy, swap out your Mutex for a Reader-Writer Lock (RwLock)."
  • "Lock-free data structures are inherently fast."
Experience engineer should know these statements aren't true.

It really based on the workload patterns. (DoD)

The RwLock Trap

When developers see read contention on a mutex, the standard instinct is reaching for a Reader-Writer Lock (std::sync::RwLock in Rust, std::shared_mutex in C++). In theory, multiple readers are non-exclusive and should execute cleanly in parallel.

In practice, the benchmark reveals something surprising:
The RwLock read throughput starts roughly equal to the Mutex.
As the number of concurrent reader threads grows, RwLock performance degrades faster and ends up significantly worse than the basic Mutex.


The Hardware Reality: Cache Lines and the MESI Protocol


Why RwLock::read() is Actually a Write

Here lies the catch: Taking a read lock is not a read operation.

To track how many active readers hold the lock, an RwLock maintains an internal reader counter. When a reader calls read(), the CPU must execute an atomic increment (fetch_add) on that shared counter.

Core 0 (Reader 1)   --> fetch_add(reader_count) --> Needs Exclusive Ownership (Line in 'M')
Core 1 (Reader 2)   --> fetch_add(reader_count) --> Forces Core 0 invalidation; pulls line to Core 1
Core 0 (Releasing)  --> fetch_sub(reader_count) --> Pulls line back from Core 1

Every single acquire and release triggers cache-line bouncing across the processor interconnect. Each transition costs ~30 ns. Across acquire and release, a thread spends ~60 ns just synchronizing the lock state—over half the latency of a trip out to main RAM.




Why does Mutex hold up better under extreme thread contention than RwLock?

With a Mutex, only the current owner touches the lock line. Other threads block or queue up in sequence.
With an RwLock, dozens of reader threads aggressively hammer the exact same counter simultaneously, creating intense cache-line ping-pong across every core.


The Left-Right Data Structure: Coordination Without Contention

To make readers truly scalable, readers must never write to a shared cache line.

Jon presents Left-Right, a concurrency primitive, designed for workloads with frequent reads and infrequent writes (e.g., in-memory key-value lookups, routing tables, configuration maps).

High-Level Architecture

Instead of locking access to a single instance, Left-Right maintains two identical copies of the underlying data structure (the Left copy and the Right copy), mediated by an atomic pointer.



The Read Path (Wait-Free & Shared-State Free)

Every reader thread is registered with a private, thread-local counter aligned to its own cache line.
When reading:
  • The reader announces its entry by updating its own thread-local counter.
  • It loads the global atomic pointer to find the current active copy (Left or Right).
  • It executes the read directly on that copy without taking any locks.
  • It signals completion on its private counter.
  • Because each reader modifies only its own cache line, there is zero cross-core invalidation between readers. Their cache lines stay in the Exclusive/Modified state within their local L1/L2 caches.

The Write Path (Two-Phase Reconciliation)

  • Step 1: Writer mutates Right Copy (inactive)
  • Step 2: Writer swaps Atomic Pointer to Right
  • Step 3: Writer waits for readers in Left Copy to exit (epochs/counters)
  • Step 4: Writer replays mutations onto Left Copy
Mutate Inactive Copy: The writer applies the update to the copy that readers aren't currently directed to (e.g., the Right copy).
Atomic Pointer Swap: The writer atomically swings the pointer to Right. All new incoming read operations will now read from Right.
Wait for Old Readers: Readers that entered before the pointer swap are still safely executing inside the Left copy. The writer scans the per-thread counters in a loop until it confirms that all readers active during the switch have finished.
Replay & Synchronize: Once the Left copy is completely drained of readers, the writer replays the exact same update from an operational log onto the Left copy. Both copies are now identical again.

The "Four-Core Drop": A Real-World Lesson in False Sharing

While benchmarking Left-Right, Jon observed expected linear scalability up to three cores—then suddenly, at four cores, throughput plummeted by nearly an order of magnitude:

Throughput
    ^
    |         /
    |       /
    |     /   <-- Expected linear scaling
    |   /
    |  *
    |       |     \__ * <-- Plummeted 10x at 4 cores!
    +----------------------------------------> Cores

The bug wasn't an algorithmic flaw or a NUMA boundary traversal. It was False Sharing:
  • The internal implementation stored per-thread reader counters together in an array.
  • Multiple 64-bit counter values fit inside a single 64-byte cache line (8 bytes × 8 = 64 bytes).
  • Even though Core 0 and Core 1 were updating completely independent counter variables, those variables shared the exact same physical cache line. The CPU was forced to ping-pong the line between cores on every single reader check-in.

The Fix

Enforce cache-line alignment on the counter type:

#[repr(align(64))]
struct AlignedReaderCounter {
    counter: AtomicUsize,
}

By ensuring each thread's counter lived on its own dedicated 64-byte boundary, the false sharing vanished, and performance restored to ~3 billion reads/second across 10 cores—scaling linearly.

Engineering Trade-offs: When Should You Use Left-Right?

Left-Right is not a magic drop-in replacement for every concurrency scenario. It trades memory and write performance for extreme read throughput:

ConstraintLeft-Right Trade-Off
Memory FootprintDoubled (2×), because two full copies of the data structure must live in memory.
Write OverheadHigh. Writers must apply changes twice (once per copy), keep an operation log, and wait for reader epochs to drain.
Write ConcurrencySingle writer only. Multiple concurrent writers require an external lock.
Consistency ModelEventually consistent / Non-linearizable. Readers might see slightly stale data before a pointer swap, and writers cannot immediately read-your-own-writes from the reader handle.
DeterminismOperations must be completely deterministic so that replaying them on the second copy yields identical internal state.


Summary Takeaways

  • "Lock-Free" does not mean "Contention-Free": Eliminating OS-level locks doesn't matter if your threads are repeatedly modifying the same atomic variable on a single shared cache line.
  • Short critical sections expose synchronization overhead: If your protected work takes 5 ns (like a hash map lookup) and your lock acquire/release costs 60 ns in cache line bounces, synchronization dominates your runtime.
  • Align for the hardware: Always guard against false sharing in concurrent per-thread arrays using 64-byte alignment (alignas(64) in C++, #[repr(align(64))] in Rust).
  • Tailor algorithms to your access patterns: When your system is 99% reads and you can afford the memory overhead and deterministic write logs, patterns like Left-Right turn cache lines from a bottleneck into an advantage.

Aug 4, 2026

[CppNow][summary] Lock-free Programming is Dead - Long Live Lock-free Programming! - Fedor G Pikus - C++Now 2026

Deep Dive Summary: Lock-free Programming is Dead - Long Live Lock-free Programming!

Speaker: Fedor Pikus
Conference: C++Now 2026
Topic: Microarchitecture, Concurrency, Lock-Free Data Structures, and Low-Level C++ Optimization

Reference: 


Executive Overview

For decades, modern C++ concurrent programming followed a simple, widely accepted rule: Lock-free algorithms should be used for high-contention, performance-critical paths, while locks should be reserved for low-contention or non-critical code.

In this landmark C++Now 2026 presentation, Fedor Pikus demonstrates that advancements in modern CPU microarchitecture (x86-64 Intel/AMD, ARM Grace/Graviton, Apple Silicon) have completely inverted this paradigm:

  1. At High Contention: Properly engineered spin locks outperform lock-free (compare_exchange loops) and wait-free (fetch_add) atomic algorithms.
  2. At Low Contention: Taking a spin lock—even once every 100 iterations—severely degrades surrounding CPU out-of-order execution performance ("pipeline poisoning"), whereas atomics excel without stalling execution pipelines.
  3. The Optimal Modern Paradigm: High-performance concurrent data structures must combine both techniques—utilizing custom spin locks for high-contention index allocation/state transitions and atomic handoffs for low-contention payload access.

Progress Guarantees & Theoretical Definitions

Before diving into hardware microarchitecture, Pikus clarifies the classical computer science definitions of thread progress:

Guarantee CS Definition Typical C++ Primitive Hardware Reality
Wait-Free Every thread completes its operation in a bounded number of algorithmic steps. std::atomic::fetch_add Not constant time. Hardware cache-line invalidation forces sequential memory execution.
Lock-Free At least one thread makes progress overall; losers retry in a loop. std::atomic::compare_exchange_weak / strong High contention causes severe CAS-retry loops and cache coherence thrashing.
Lock-Based One thread holds exclusive access; all other contending threads wait/block. std::mutex, Custom Spin Locks OS mutexes incur context switch overhead, but well-tuned spin locks eliminate CAS retry overhead.

Key Distinction: Computer science definitions measure algorithmic steps, not CPU clock cycles. A "wait-free" instruction executes a single instruction step, but at the hardware level, cache coherency mechanisms force memory accesses to serialize, causing hardware-level waiting.


Microarchitectural Deep Dive: Why Atomics Fail Under High Contention

1. Cache Coherency and Read-For-Ownership (RFO)

Modifying any atomic variable requires exclusive access to its underlying 64-byte cache line:

  • To write to a cache line, a CPU core must issue a Read-For-Ownership (RFO) request across the interconnect.
  • The requesting core must wait for all other cores holding that cache line in a Shared (S) state to invalidate their local L1/L2 caches and send back an Acknowledgment (ACK) signal.
    (MESI)
  • Under heavy multi-threaded contention, cores spend the majority of their clock cycles waiting for speed-of-light electrical signal propagation across the chip to complete RFO invalidation handshakes.

2. True Sharing vs. False Sharing

Experimentation proves that false sharing (multiple independent atomics residing on the same 64-byte cache line) and true sharing (all threads hammering the exact same atomic variable) suffer from the exact same latency penalty at high thread counts. The hardware bottleneck is the cache line granularity, not the specific integer being modified.


Engineering a High-Performance Spin Lock

To outperform atomics at high contention, a spin lock must be engineered specifically to respect hardware cache coherency protocols.

Critical Design Features:

  1. Pre-Read Probe (Test-and-Test-and-Set):
    • Before attempting an expensive atomic operation (atomic_exchange), the thread performs a relaxed, read-only load of the lock flag.
    • Reading allows the core to acquire the cache line in a Shared (S) state without revoking exclusive access from the thread currently holding the lock.
  2. Pre-Read Retries:
    • Performing ~8 relaxed reads on x86 before attempting an atomic swap ensures that in-flight RFO signals have time to settle, preventing premature cache-line stealing.
  3. Aggressive Back-off:
    • Unlocking a spin lock requires writing 0 to memory, which itself requires an RFO request. If waiting threads continuously hammer the lock with atomic writes, they steal the cache line from the lock holder, severely delaying the unlock operation.
    • Back-off logic (or yielding) keeps waiting threads from stealing the cache line during lock release.

The Low-Contention Asymmetry: "Pipeline Poisoning"

When benchmarking code that mixes parallel payload computation (e.g., local mathematical tasks) with synchronization (e.g., updating a shared counter):

  • High Contention Domain: Spin locks deliver up to 2.5x higher overall application throughput compared to atomics.
  • Low Contention Domain: When synchronization occurs rarely (e.g., 1 out of 100 iterations), using a spin lock causes program throughput to plummet compared to atomics.
High Contention (1:1 Parallel to Shared Work):
[Spin Lock]  ========================> 2.5x Throughput vs Atomics
[Atomic CAS] ========> 1.0x

Low Contention (100:1 Parallel to Shared Work):
[Atomic CAS] ========================> 1.0x (Fast Execution)
[Spin Lock]  =====> 0.25x (Severe Pipeline Poisoning)

Hardware Profiling & Root Cause Analysis (Intel VTune / Linux Perf)

By inspecting CPU hardware performance counters (RESOURCE_STALLS.STORE_BUFFER), Pikus identified the exact microarchitectural bottleneck:

  1. Store Buffer Stalls: Spin locks force massive store buffer stalls in x86 execution pipelines. On x86 architectures, memory writes exit the store buffer in strict program retirement order.
  2. Implicit vs. Explicit Dependencies:
    • Atomics (fetch_add, CAS): Fuse control and data into a single variable. The CPU out-of-order execution engine can inspect the instruction stream and recognize explicit data dependencies, allowing non-dependent payload instructions to flow around the atomic operation.
    • Locks: Separate control (the lock flag) from data (the guarded payload). Because the CPU cannot reason through implicit memory dependencies across bidirectional acquire/release memory barriers, it cannot predict safety across the critical section. As a result, the out-of-order execution pipeline flushes and stalls until the lock and unlock operations fully retire.

Practical Application: The Hybrid MPMC Ring-Buffer Queue

To prove these findings, Fedor Pikus constructed a high-throughput Multi-Producer Multi-Consumer (MPMC) ring-buffer queue designed around hybrid synchronization:

+-----------------------------------------------------------------------+
|                         MPMC QUEUE DESIGN                             |
+-----------------------------------------------------------------------+
| High-Contention Domain  -->  Spin Lock (Guards Head / Tail Indices)  |
| Separate Cache Lines    -->  Prevents Producer/Consumer Thrashing     |
+-----------------------------------------------------------------------+
| Low-Contention Domain   -->  Atomic Slot Keys (std::atomic<Key>)    |
| Exclusive Slot Access   -->  Zero Lock Overhead for Payload Handoff   |
+-----------------------------------------------------------------------+

Performance Benchmark Results:

  • Throughput: Dramatically outperforms traditional pure lock-free MPMC queues across Intel Granite Rapids, AMD Zen 5, Nvidia Grace, and Apple Silicon.
  • Average Latency: Significantly lower mean and 95th/99th percentile latency compared to pure lock-free implementations.
  • Tail Latency Exception (99.99%+): Pure lock-free queues only win at extreme tail latencies where thread preemption risks affect lock-based structures.

Duration & Power Considerations: When Spin Locks Fail

While spin locks excel at short critical sections, they carry severe operational trade-offs if critical section processing times grow long:

  1. Busy-Waiting Energy Waste: A thread waiting on a spin lock burns 100% CPU core utilization, drawing maximum power, generating heat, and causing thermal throttling.
  2. Thread Preemption Disaster: If the thread holding a spin lock is preempted by the OS scheduler (context-switched out), all contending threads will actively spin for their entire OS time slice doing zero productive work while waiting for the lock holder to be rescheduled.
  3. The Industrial Solution (Adaptive / Two-Phase Locks): Production systems (e.g., database kernels, runtime engines) utilize adaptive locks:
    • Phase 1: Spin for a short, bounded duration (~50–100 iterations using CPU pause hints like _mm_pause() or YIELD).
    • Phase 2: If the lock remains unacquired, yield execution to the OS kernel via a futex sleep system call.

Architectural Anomalies & Micro-architectural Surprises

  1. Cache-Line Separation of Lock and Payload: Placing the spin lock flag and the guarded data variable on different cache lines improves high-contention throughput. It prevents waiting threads (doing relaxed pre-reads on the lock flag) from invalidating the lock holder's cache line while it modifies the payload variable.
  2. AMD Zen 4 / Zen 5 Near-Memory Atomics: AMD Zen 4 introduced hardware execution ALUs directly inside the L3 memory controller. If a core attempts a fetch_add without owning the cache line, it offloads the operation directly to the L3 controller, bypassing L1/L2 cache-line invalidation cycles.
  3. Apple Silicon CAS Back-off: Apple M-series chips use a power-efficient, high-latency directory interconnect. Implementing explicit back-off inside a compare_exchange loop on Apple Silicon improves throughput by 10x, elevating CAS performance close to spin lock levels.

Summary Principles for Modern C++ Concurrency

  1. High Contention: Abandon pure lock-free CAS loops. Use properly engineered spin locks featuring relaxed pre-read probes, iteration limits, and back-off logic.
  2. Low Contention: Avoid locks completely. Use atomic primitives (std::atomic) to prevent CPU store buffer flushes and out-of-order pipeline stalls.
  3. Hybrid Architecture: Structure concurrent data structures to use spin locks for high-contention indexing (e.g., ring-buffer head/tail allocation) and atomic state flags for low-contention data transfer.
  4. Bounded Work: Keep spin lock critical sections strictly bounded to a few dozen nanoseconds; fall back to adaptive OS-backed locks (futex) if processing times can exceed thread time-slices.

#include <atomic>
#include <chrono>
#include <new>
#include <thread>

#if defined(__x86_64__) || defined(_M_X64)
#include <emmintrin.h> // For _mm_pause()
#endif

// Align to prevent false sharing with adjacent cache lines
class alignas(std::hardware_destructive_interference_size) AdaptiveSpinLock {
public:
    AdaptiveSpinLock() noexcept = default;

    // Non-copyable, non-movable
    AdaptiveSpinLock(const AdaptiveSpinLock&) = delete;
    AdaptiveSpinLock& operator=(const AdaptiveSpinLock&) = delete;

    void lock() noexcept {
        // Phase 1: Fast path (Uncontended)
        if (!state_.exchange(true, std::memory_order_acquire)) {
            return;
        }

        // Phase 2: Active Spin Loop with Pre-Read (Test-and-Test-and-Set)
        int spin_count = 0;
        constexpr int MAX_SPINS = 64; // Max active CPU spinning iterations

        while (true) {
            // Pre-read probe: Read in a relaxed loop to stay in 'Shared' (S) cache state.
            // Avoids sending RFO (Read-For-Ownership) signals while waiting.
            while (state_.load(std::memory_order_relaxed)) {
                pause_cpu(spin_count);

                // Phase 3: Adaptive Fallback to OS-backed Sleeping
                // If spinning takes too long, stop burning CPU/power and park the thread.
                if (++spin_count > MAX_SPINS) {
                    // C++20/23 std::atomic::wait uses OS futex/synch primitives natively
                    state_.wait(true, std::memory_order_relaxed);
                }
            }

            // Attempt to acquire lock via atomic exchange once pre-read detects key is free
            if (!state_.exchange(true, std::memory_order_acquire)) {
                return; // Lock acquired successfully
            }
        }
    }

    bool try_lock() noexcept {
        // First check relaxed state to avoid cache invalidation on failure
        if (state_.load(std::memory_order_relaxed)) {
            return false;
        }
        return !state_.exchange(true, std::memory_order_acquire);
    }

    void unlock() noexcept {
        // Unconditionally release lock
        state_.store(false, std::memory_order_release);

        // Notify any waiting thread that was parked in state_.wait()
        state_.notify_one();
    }

private:
    // Flushes execution pipeline & saves power during short spinning
    static void pause_cpu(int spin_count) noexcept {
        if (spin_count < 16) {
#if defined(__x86_64__) || defined(_M_X64)
            _mm_pause(); // Informs x86 pipeline of spin loop; reduces power & store buffer pressure
#elif defined(__aarch64__)
            asm volatile("yield" ::: "memory"); // ARM yield hint
#endif
        } else {
            // Hint to OS scheduler to run other threads on the same core/time-slice
            std::this_thread::yield();
        }
    }

    // False = Unlocked, True = Locked
    std::atomic<bool> state_{false};
};

Aug 3, 2026

[benchmark] how `void DoNotOptimize(Tp const& value)` works under the hood

Reference:
How does Google's `DoNotOptimize()` function enforce statement ordering
[gcc] Constraint Modifier Characters
[google menchmark] void DoNotOptimize()

code by example:
https://godbolt.org/z/EzTr4afEM


How Google's DoNotOptimize Enforces Statement Ordering

This document summarizes the StackOverflow thread How does Google's DoNotOptimize function enforce statement ordering?, focusing in detail on the accepted answer by Peter Cordes (answer 69288151).


🧩 Summary of Ordering Mechanics

Mechanism Targeted Ordering Implementation Compiler's Interpretation
Opaque "memory" Clobber Ordering between DoNotOptimize() and external calls (e.g., time()) asm volatile("" : ... : : "memory") Treated as an opaque, non-inline function call that can read/write any globally-reachable memory, stopping compile-time reordering.
Data Dependency ("+r,m") Ordering of the benchmarked computation itself (input -> output) Input/Output constraints on the variable Forces the compiler to materialize values, forget compile-time constants (defeating constant propagation), and execute the code.

1. The "memory" Clobber acts as an Opaque Function Call

The inline assembly in DoNotOptimize() contains a "memory" clobber.

  • Opaque External Call Simulation: To the compiler's optimizer, a "memory" clobber is treated just like an external, non-inline function call (whose source code is hidden). The compiler must assume this block can read or write any globally-reachable object.
  • Preventing Reordering with time(): Since time() itself is not defined inline in C++ headers, the compiler cannot see its definition and treats it as an opaque external function. Because both time() and DoNotOptimize() (due to its "memory" clobber) are treated as opaque operations that could modify global state, the compiler is strictly forbidden from reordering them. It is the exact same reason a compiler will not reorder two sequential calls to unknown functions like puts("first"); puts("second");.

2. The "+r,m" Constraint & Data Dependency

While the "memory" clobber handles ordering against other opaque calls, the actual benchmark computation (e.g., run_bench()) is ordered using strict C++ data dependencies.

  • Forcing Materialization: The "+r,m" constraint declares a read/write operand. This forces the compiler to materialize the target variable's value in a register or memory, meaning the variable must actually be computed up to that point.
  • Defeating Constant Propagation: Because of the + (read/write) modifier, the compiler must assume that the assembly block could arbitrarily change the variable's value. Consequently, the compiler has to forget all compile-time knowledge of that variable (such as it being a constant, positive, etc.). This prevents the optimizer from constant-folding or optimizing away the benchmarked code entirely.
  • Enforcing the Pipeline:
    1. DoNotOptimize(input) forces the input to be computed and clears compiler assumptions.
    2. The compiler runs the benchmark calculation, because the output depends on this newly materialized input.
    3. DoNotOptimize(output) forces the output of the benchmark to be materialized in memory or a register, ensuring the compiler doesn't optimize away the entire benchmark calculation as an "unused result".

3. Key Misconceptions Corrected by the Answer

  • Reference Arguments and the Stack: The original poster suspected that reference arguments force variables onto the stack (since you cannot have a pointer to a register). However, because DoNotOptimize() is forced to inline (always_inline), the by-reference argument is optimized away entirely. The assembly operates directly on the underlying C++ variable.
  • Register vs. Memory Choice: The compiler can always choose to load a variable's value into a register before the asm statement and store it back afterward, even with the "m" constraint option.
  • Clang Workaround: The use of "r,m" (instead of "rm") specifically bypasses a Clang missed-optimization bug where Clang aggressively defaults to memory spilling for "rm" constraints even when a register is readily available.

Jul 2, 2026

[C++] is inline a function always a good idea?

Reference:
A deep dive into SmallVector::push_back
https://llvm-compile-time-tracker.com/


Shrink-wrapping optimization

Shrink-wrapping is a compiler optimization technique designed to minimize the overhead of a function's prologue and epilogue by moving them so they execute only when absolutely necessary.

To understand why this is useful, it helps to look at how a standard function handles memory and registers at the assembly level.

The Problem: Preemptive Over-Allocation

When a function executes, it often needs to save certain CPU registers to the stack (callee-saved registers) and allocate space for local variables. This setup is called the prologue. Before the function returns, it restores those registers and cleans up the stack, which is called the epilogue.

By default, standard compilers place the prologue at the very beginning of a function and the epilogue at the very end.

If a function has a hot fast-path (e.g., an early return or a quick validation check) that doesn't actually use those registers or local variables, it still pays the CPU cycle tax of executing the prologue and epilogue.


How Shrink-Wrapping Fixes This

Shrink-wrapping analyzes the control flow graph (CFG) of a function to find the precise basic blocks where the saved registers or stack space are actually required. It then "shrinks" the scope of the prologue and epilogue, wrapping them tightly only around those specific paths.


Before Shrink-Wrapping (Standard Layout)

Function_Start:
  PUSH registers      <-- Prologue runs every single time
  ALLOCATE stack
  
  IF (condition) THEN
    GOTO Fast_Path    
  ELSE
    GOTO Slow_Path
    
Fast_Path:
  Result = 0          <-- Doesn't even need those registers!
  GOTO Function_End

Slow_Path:
  Use registers to do heavy math...
  GOTO Function_End

Function_End:
  DEALLOCATE stack    <-- Epilogue runs every single time
  POP registers
  RETURN

After Shrink-Wrapping

Function_Start:
  IF (condition) THEN
    GOTO Fast_Path    
  ELSE
    GOTO Slow_Path
    
Fast_Path:
  Result = 0          <-- Pure profit: No stack overhead, no register spills
  RETURN              <-- Direct return

Slow_Path:
  PUSH registers      <-- Prologue is "wrapped" only around the slow path
  ALLOCATE stack
  Use registers to do heavy math...
  DEALLOCATE stack
  POP registers       <-- Epilogue wrapped here too
  RETURN


Limitations: The "Rejoin" Problem

Shrink-wrapping is incredibly effective for functions with early exits. However, it can fail or get disabled if the control flow graph gets messy.

If a fast path and a slow path diverge but then rejoin later in the function before a single, shared return point, the compiler may not be able to safely decouple the stack frames.


clang and gcc cmd:
$ clang -mllvm -debug-only=shrink-wrap
$ gcc -fshrink-wrap-separate (on at -O2) 


How to solve if shrink-wrap not working?

Hand coding the tail duplication with fast and slow code path.
e.g.
LLVM_ATTRIBUTE_NOINLINE void growAndPushBack(ValueParamT Elt) {
  // in case Elt aliases storage that grow() invalidates
  // This is very much edge-case considered.
  T Tmp = Elt;
  // +1 is sufficient, while internally doing
  // exponential growth algorithm.
  this->grow(this->size() + 1);
  std::memcpy(reinterpret_cast<void *>(this->end()), &Tmp, sizeof(T));
  // size is not just +1 but applied with 
  // exponential growth algorithm.
  this->set_size(this->size() + 1);
}

void push_back(ValueParamT Elt) {
  if (LLVM_UNLIKELY(this->size() >= this->capacity()))
    return growAndPushBack(Elt);
  std::memcpy(reinterpret_cast<void *>(this->end()), &Elt, sizeof(T));
  this->set_size(this->size() + 1);
}

mov    eax, [rdi + 8]
cmp    eax, [rdi + 12]
jae    growAndPushBack         # TAILCALL
mov    rcx, [rdi]
mov    [rcx + rax*4], esi
inc    dword ptr [rdi + 8]
ret
Since there are no callee-saved registers, shrink-wrapping is unnecessary.
Moving the slow path into a separate COMDAT/section groups section as an out-of-line function further degrades its performance.

Use:
// noinline growAndPushBack is load-bearing for both Clang and GCC.
void DecodeMOVDDUPMask(unsigned n, llvm::SmallVectorImpl<int> &v) {
  for (unsigned l = 0; l < n; l += 2)
    for (unsigned i = 0; i < 2; ++i)
      v.push_back(i);
}
The noinline attribute is required here. Without it, Clang and GCC may inline the helper function, which defeats the optimization by reintroducing the prologue.

Creating a temporary copy (T Tmp = Elt) safely handles cases where Elt references the vector's own storage. While this copy is easily elided for small, trivially-copyable types, passing a larger element by reference to the out-of-line growAndPushBack() forces it to be memory-materialized. Because its address must remain stable across a non-inlined call, this defeats construct-in-place optimization for large types. However, this overhead is negligible compared to the cost of grow(), which must already copy or move all existing size() elements.

Jul 1, 2026

[benchmark] using rdtsc counter

RDTSC (Read Time-Stamp Counter) instruction returns the number of clock cycles since the processor was last reset.

On modern processors, RDTSC does not count actual, variable CPU clock cycles affected by power-saving states (C-states) or Turbo Boost. Instead, it uses a feature called Invariant TSC.

Invariant TSC: The counter increments at a constant, fixed frequency (usually the base/nominal frequency of the processor), regardless of the current operational frequency or power state.

Implication: Because it measures reference time rather than actual executed core clock cycles, if CPU turbos up to 5GHz but its base frequency is 2.5GHz, RDTSC will still increment at the 2.5GHz rate.


Because modern CPUs execute instructions out-of-order, RDTSC can float ahead or behind the block of code we are trying to benchmark.

To get an accurate cycle count for a specific code snippet, we must fence the instruction.

RDTSCP: A serialized variant that waits until all previous instructions have executed before reading the counter (though it doesn't prevent subsequent instructions from moving above it).

LFENCE; RDTSC: The industry-standard way to benchmark. Placing an LFENCE (Load Fence) right before RDTSC forces the CPU to serialize execution, ensuring measure exactly what happens between fences.


The Fence Strategy

lfence (Load Fence): This instruction acts as a serializing barrier for instruction execution. It forces the CPU to wait until all previous instructions in the pipeline have completed execution before it allows any subsequent instructions to begin.

The Goal: Putting lfence before rdtsc prevents the CPU from executing rdtsc early (out-of-order). It guarantees that everything you wanted to measure before this benchmark has truly finished before the timer starts.

The Register Mapping

The x86 rdtsc instruction reads the 64-bit Time-Stamp Counter and splits the value across two 32-bit registers:
EDX gets the high-order 32 bits.
EAX gets the low-order 32 bits.
The inline assembly output constraints capture this:
"=a"(lo): Tells the compiler to bind the value in EAX (a) to the C variable lo.
"=d"(hi): Tells the compiler to bind the value in EDX (d) to the C variable hi.
uint64_t rdtsc_start() {
    uint32_t lo; uint32_t hi;
    __asm__ __volatile__(
        "lfence\n\t"
        "rdtsc"
        : "=a"(lo), "=d"(hi)
        :
        : "memory");
    return ((uint64_t)hi << 32) | lo;
}

The Fence Strategy

rdtscp (Read Time-Stamp Counter and Processor ID): Unlike rdtsc, rdtscp is natively partially serialized. The hardware guarantees that rdtscp will wait for all prior instructions to complete before executing. This ensures that the code you are benchmarking has fully finished before the stop-timer runs.

lfence afterward: While rdtscp prevents earlier instructions from slipping past it downward, it does not prevent later instructions from leaking upward (executing early before the timestamp is taken). Adding lfence right after rdtscp pins the timestamp in place, ensuring that nothing belonging after your benchmark executes before rdtscp completes.

The Clobber List Change
"rcx": In addition to writing the 64-bit timestamp to EDX:EAX, the rdtscp instruction also writes the hardware processor ID (core ID) into the ECX register. Because the inline assembly overwrites RCX as a side effect, you must declare "rcx" in the clobber list so the compiler knows its previous contents are ruined.
uint64_t rdtsc_end() {
    uint32_t lo; uint32_t hi;
    __asm__ __volatile__(
        "rdtscp\n\t"
        "lfence"
        : "=a"(lo), "=d"(hi)
        :
        : "rcx", "memory");
    return ((uint64_t)hi << 32) | lo;
}


Why __volatile__?

Without __volatile__, the compiler's optimization passes might conclude that reading a hardware counter is a pure function or that its order doesn't matter relative to adjacent C statements. 
__volatile__ forces the compiler to leave the assembly block exactly where you put it and prevents it from being optimized away.


The "memory" Clobber

The "memory" token tells the compiler that this assembly block read or wrote to arbitrary locations in RAM. This creates a compiler-level memory fence, forcing the compiler to flush registers back to memory before the block and reload them afterward. This stops the compiler from scheduling code movements across the boundary.



[ Pre-benchmarking Code ]
----------------------------------- <- lfence forces completion of above
RDTSC (Start Timer)
===================================
[ Critical Code Block to Measure ] <- Cannot leak upwards (lfence blocks it)
=================================== <- Cannot leak downwards (rdtscp blocks it)
RDTSCP (Stop Timer)
----------------------------------- <- lfence forces completion of rdtscp
[ Post-benchmarking Code ]

Jun 30, 2026

[Design] Cache friendly design, CPU/Memory level

Reference:
What do you mean by "Cache Friendly"? - Björn Fahller
What Every Programmer Should Know About Memory

Simplistic model of cache behaviour

Includes

  • The cache is small
  • and consists of fixed size lines
  • and is very very fast when hit
  • and missing is very slow

Excludes

  • Multiple levels of caches
  • Associativity
  • Threading

All models are wrong, but some are useful


$ valgrind --tool=callgrind --cache-sim=yes --dump-instr=yes --branch-sim=yes
$ perf stat -e cycles,instructions,L1-dcache-loads,L1-dcache-Load-misses --call-graph=dwarf
$ perf record -e cycles,instructions,L1-dcache-loads,L1-dcache-Load-misses --call-graph=dwarf
$ pert report
$ hotspot perf.data

macos:
$ samply


Rule of thumb

follow/chasing pointer -> cache miss.

So,
Get rid of pointers.
Use std::vector (contiguous memory) vs. prev/next pointers.
std::vector be aware of memmove/memcpy.
memmove cause cache miss; if possible, memcpy.
Thus try to move less data.

binary tree/search still chasing pointer, 
can we get Log(n) lookup/insert without chasing pointers?
Heap. Use contiguous memory(e.g. std::vector) to implement.

But the heap is not searchable. Need extra contiguous memory to store the elements in order;
and the element holds the index to the contigous memory. Thus when the smallest heap
is processed, use the index to the contiguous memory and process that chunk.
Trade off: extra memory usage for speed and less cache miss.

Can we make it better?
B-Heap(binary heap implemented to keep subtrees in a single page); smaller heap fit into cache-line.
(not really better, depends on use case. i.e. if within a loop, the cache is hot, good.
otherwise, not worth the extra instructions due to cache is flushed anyways.)










Linear search is good iff the data set is small. (cpu prefetch within the cache size)
If data set is large, big O still dominates.




Rules of thumb

  • Folling a pointer is a cache miss, unless have information to the contrary.
  • Smaller working data set is better.
  • Use as much of a cache entry as we can.
  • Sequential memory accesses can be very fast due to prefetching.
  • Fewer eviceted cache lines means more data in hot cache for the rest of the program.
  • Mispredicted branches can evict cache entries (spectre/meltdown)
  • Linear access in contigous memory rules for small data sets.
  • Measure measure measure. (intuition can be wrong.)

Reference:

Important things to know:

  • The hardware prefetchers only work for streams within 4KiB pages.
    (There is a page-crossing L1 prefetcher in Ivy bridge, but it does not seem to make a lot of difference in my tests.)
  • The L2 hardware prefetchers can only track a limited number of pages (16 to 32, depending on the processor model).
  • Hardware prefetchers tend to be much more aggressive at fetching read streams than at fetching write streams.
  • Transpose operations have to deal with non-contiguous accesses over some large area -- it helps if that area is small enough to be mapped by the TLBs.
    For Ivy Bridge, this is 4KiB * 512 entries = 2 MiB when using the default page size.
  • Strided accesses in transpositions can very easily cause pathological cache conflicts.
  • If the stride is a multiple of 4KiB, the L1 Data Cache will only hold 8 entries before beginning to overflow.
  • Memory bandwidth performance in Intel processors is strongly degraded if a store maps to the same congruence class in the L1 cache as a preceding load.
    This will happen whenever a store maps to the same location in its 4KiB page as a recently preceding load.
From these principles, it is pretty easy to come up with tuning guidelines that work well for transpositions.
  1. Pad arrays to avoid power-of-two strides (or offsets) whenever possible.
    Unrolling loops by 8 is usually a very good strategy.
  2. This minimizes cache conflicts, so you (typically) get to read all the elements in a cache line before it gets evicted.
  3. The number is small enough that the compiler does not typically get terribly confused about register scheduling.
  4. When working in higher dimensions, make sure that the sub-blocks resulting from the unrolling don't cover more space than the TLB can cover.
  5. This may mean unrolling by less than 8 for some or all of the loops.
  6. Analysis from first principles is challenging for 2D transpositions, and is probably not practical for higher than 3D.
  7. Favor contiguous reads over contiguous stores, but remember that there is minimal benefit to contiguous reads longer than 4 KiB.
  8. If using streaming stores, remember to construct the code to store 64 contiguous bytes all at once, then store to the next block.
    This will maximize the number of full write-combining buffers that get used and will minimize the read/modify/write cycles at the DRAM (which are required if a partially full write-combining buffer gets flushed).
Unrolling and blocking are required to get good performance -- especially with arrays that have large power-of-two dimensions.



Hardware CPU Prefetching, Cache Friendliness, and Big-O Complexity

1. How CPU Prefetching Works on std::vector

Hardware CPU prefetchers actually work better on large std::vectors than small ones, and function seamlessly regardless of how large the vector grows (up to the physical limits of RAM).

In fact, CPU hardware prefetchers rely on streaming through large blocks of memory to function effectively.

Mechanics of Hardware Prefetching

std::vector stores its elements in a single, contiguous block of dynamically allocated memory. When iterating through a vector linearly, memory accesses occur at sequential byte addresses ($A, A+64, A+128, \dots$).

  1. Pattern Detection: Modern CPU cores feature dedicated Stream Prefetchers and Spatial Prefetchers (L1/L2 data prefetchers). When the CPU detects sequential cache misses (typically 2 to 4 consecutive cache line accesses in the same direction), it identifies a linear stream.
  2. Ahead-of-Time Fetching: Once a stream is detected, the prefetcher speculatively issues memory reads for upcoming cache lines ($N+2, N+3, \dots$) into L2 or L1 cache before execution requests them.
  3. Pipelining Latency: As long as scanning continues sequentially, memory fetch latency is completely hidden behind instruction execution.

Impact of Dataset Size

  • Small Vectors (< 1 KB / ~16 Cache Lines): The vector fits entirely within the L1 or L2 CPU cache after the first access anyway. Furthermore, by the time the hardware prefetcher detects the sequential pattern (after 2–3 misses), the iteration may already be near the end of the vector.
  • Large / "Unlimited" Vectors (Gigabytes): The vector exceeds CPU cache capacity, meaning every element must be fetched from main RAM. Because the access pattern is perfectly predictable and long-running, the hardware prefetcher stays active at full throughput, streaming cache lines continuously from RAM with minimal cache-miss stalls.

Caveats: When Prefetching Breaks Down

While the size of std::vector does not limit the hardware prefetcher, the access pattern and data layout do:

  • Non-Sequential Access: Accessing elements randomly (vec[rand()]) or via pointer chasing (e.g., storing raw pointers inside std::vector<T*>) prevents the prefetcher from predicting the next address.
  • Page Boundaries (TLB Misses): Every 4 KB (or 2 MB) OS memory page boundary forces the prefetcher to pause until the CPU translates the next virtual page address via the Translation Lookaside Buffer (TLB).
  • Page Faults / Virtual Memory: If a massive vector spills out of RAM into swap disk space, page faults will stall the CPU regardless of prefetching.
  • Cache Pollution & Bandwidth Saturation: Iterating over a multi-gigabyte vector with minimal compute per element turns the execution into a memory-bandwidth-bound bottleneck.

2. Linear Search vs. Binary Search: Hardware Cache vs. Big-O Complexity

The assertion that "Linear search is good iff the data set is small (cpu prefetch within the cache size). If data set is large, big O still dominates" highlights a fundamental crossover point in systems programming: hardware cache efficiency vs. asymptotic algorithmic complexity ($Big-O$).

Why Linear Search Wins for Small Data Sets ($N$ is Small)

Actual execution time is modeled by:

$$T(N) = C \cdot f(N)$$

where $C$ is the constant factor determined by memory layout and hardware overhead, and $f(N)$ is the algorithmic growth function.

For small data sets (e.g., $N < 64$ elements):

$$C{\text{linear}} \cdot N < C{\text{binary}} \cdot \log_2 N$$

Structural Reasons for a Smaller $C_{\text{linear}}$:

  1. Hardware Prefetching & Spatial Locality: Linear search scans contiguous memory (std::vector/array). As soon as element 0 is accessed, the CPU cache line (64 bytes) automatically pulls in adjacent elements. The CPU stream prefetcher speculatively fetches subsequent cache lines directly into L1/L2 cache before execution reaches them.
  2. Zero Pointer Chasing: Operating on contiguous memory incurs L1 cache latency ($\sim 1\text{ ns}$) rather than random memory node jumps ($\sim 50\text{--}100\text{ ns}$ DRAM latency).
  3. Branch Predictability: A simple sequential loop is trivial for the CPU's branch predictor to handle, minimizing pipeline stalls and flushes.

At small $N$, the data set fits entirely within L1/L2 cache, making memory access essentially instantaneous. The structural overhead of binary search (midpoint calculation, branches, non-sequential jumps) outweighs the cost of scanning a few adjacent cache lines sequentially.


Why Big-O Dominates as Data Grows ($N$ is Large)

As $N$ grows large (e.g., $N = 1,000,000$), hardware prefetching cannot overcome mathematical growth rate differences.

Metric Linear Search $O(N)$ Binary Search $O(\log_2 N)$
Operations for $N = 1,000,000$ $\sim 500,000$ comparisons (avg) $\sim 20$ comparisons
Memory Access Pattern Sequential / Prefetched Non-sequential / Cache line jumps
Primary Bottleneck Memory Bandwidth & Total Operations Cache Latency

Why Hardware Prefetching Fails to Save Linear Search at Scale:

  1. Arithmetic Overwhelms Prefetching: Even if hardware prefetching streams data from main RAM at maximum bus bandwidth ($\sim 50\text{--}100\text{ GB/s}$), scanning $10^6$ elements still requires visiting half a million items on average.
  2. Memory Bandwidth Saturation: Once the data set exceeds cache capacity (L3/RAM), linear search becomes entirely memory-bandwidth bound. Prefetching keeps the CPU pipeline fed, but the bus can only transport data so fast.
  3. Logarithmic Superiority: In Binary Search, even if every single step of the 20 comparisons incurs a full DRAM cache miss ($\sim 70\text{ ns}$ each):

$$20 \text{ misses} \times 70\text{ ns} = 1.4 \;\mu\text{s}$$

Scanning megabytes of data sequentially via Linear Search takes hundreds of microseconds, making Binary Search orders of magnitude faster despite poor prefetch alignment.


Summary: The Crossover Point

Time (ns)
   ^
   |       /  Linear Search O(N) [Small C, bad growth]
   |      /
   |     /    <--- Crossover Point (N32 - 128 elements)
   |    /___________ Binary Search O(log N) [Large C, excellent growth]
   |   /
   +-------------------------------------> N (Data Size)
  • Small $N$ ($N \lesssim 64$): Constant factor $C$ dominates. The cache-friendliness, vectorization potential, and prefetching of linear search make $O(N)$ faster than $O(\log N)$ or $O(1)$.
  • Large $N$ ($N \gg 1000$): Algorithmic growth $f(N)$ dominates. The exponential reduction in operations from $O(\log N)$ or $O(1)$ easily beats reading gigabytes of prefetched memory linearly.

Jun 16, 2026

[low latency] What is low latency (definition)

Reference:
https://github.com/crill-dev/crill

Low latency <-> High throughput

Server side is in the middle.

HFT, Audio, Game is more lean to the left.

2 Categories

  • Efficient programming
  • Programming for deterministic execution time
Most crucial thing: measuring!







Efficient programming

Microbenchmarks are tricky

  • Warm the cache
  • Randomise the heap
  • Measure release build with same compiler flags
  • But optimisations change what code you are measuring!
    • A lot of stuff can "constexpr away" in microbenchmark but not in production code


Writing efficient code requires...

  • Knowledge of the programming language
  • Knowledge of the libraries used
  • Knowledge of the compiler
  • Optimiser
    • ABI (Itanium, Microsoft, ...)
  • Knowledge of the hardware architecture
    • CPU architecture (Instruction set, pipeline, SIMD...)
    • Cache hierarchy (Registers, L1/2/3 cache)
    • Prefetcher, translation look-aside buffer
    • Branch predictor, branch target buffer


Avoid unnecessary work

  • Avoid unnecessary copies
  • Avoid unnecessary function calls / indirections
  • inline functions
  • prefer std::variant / CRTP / "deducing this" (since C++23)
    over virtual functions
  • Make as many decisions as possible at compile time
    • constexpr all the things
    • Template meta-programming
    • Generate lookup tables at compile time
  • Use efficient mathematical operations
    • Fast approximations
    • Use powers of two for sizes (compiler can replace division/mod with bit shifts)
    • Lookup tables
    • Many other techniques


Low-level bit manipulation in C++

  • Object lifetime rules
  • Aliasing rules
  • Alignment rules
  • Object representations
  • Value representations
  • std::bit_cast (since C++20)
  • Implicit-lifetime types (since C++20)
  • std::start_lifetime_as (since C++23)


The optimiser & undefined behaviour

  • memory-related UB
  • type system violation
  • out-of-bounds
  • lifetime violation (dangling pointers, use-after-free)
  • uninitialised variables
  • data races
  • signed integer overflow
  • infinite loops with no side effects


The optimiser & undefined behaviour


Sub-properties of reproducible & unsequenced

  • stateless: function that does not define mutable static or thread-local objects (nor do functions that are called by it)
  • effect-less: function that does not have observable side effects
  • idempotent: repeated evaluation gives the same result (hence may read global state)
  • independent: does not depend on other state than the arguments or constants (hence may write to globals)
  • reproducible: effect-less and idempotent
  • un-sequenced: stateless, effect-less, idempotent, and independent

Assume explained:

  int f(int i) {
  	[[assume(++i == 43)]]
    return i;
  }
  // function f can be optimized to '42'
  


C++ currently does not have a way to tell the compiler that the pointer are not alias:



CPU pipeline hazards

  • Branch hazard
  • Data hazard
  • Hardware hazard
    • Limited amount of adders/shifters
    • Limited amount of load/store units
    • (latest Intel CPUs: 3 loads + 2 stores per cycle)
    • → sometimes you can replace loads by shifts → increase bandwidth of every load using SIMD



[likely]] and [unlikely]]
https://vsdmars.blogspot.com/2016/01/likely-or-unlikely-easy-misleading.html

  • Does not affect branch predictor!
  • Can affect code layout
  • Have various of pitfalls
    • Aaron Ballman: "Don't use the [likely]] or [unlikely]] attributes"
    • Amir Kirsh & Tomer Vromen:"C++20 Likely and Unlikely:
    • A Journey Through Branch Prediction and Compiler Optimizations" (2022)

The 3 Types of Data Hazards

  • RAW (Read After Write): The most common type. Instruction B tries to read data before Instruction A finishes writing it.
  • WAR (Write After Read): Instruction B tries to write to a location before Instruction A has a chance to read it.
  • WAW (Write After Write): Instruction B tries to write to a destination before Instruction A writes to it, potentially leaving the wrong final value.

Rarely have to worry about this breaking your program because the system handles it automatically:

  • The Hardware (CPU): Modern CPUs use a trick called Data Forwarding (passing the result directly from the math unit to the next instruction before it's even written to memory) or they just stall the processor for a cycle to let the data catch up.
  • The Compiler: Optimization flags (like -O2 or -O3 in GCC/Clang) allow the C++ compiler to smartly rearrange your instructions so that independent calculations are placed in between dependent ones, keeping the pipeline moving smoothly without stalls.


SIMD

  • "Single instruction, multiple data"
  • CPU-specific: MMX, SSE 1/2/3/4, AVX, AVX2, AVX-512, AMX, NEON, SVE...
  • How to use?
  • Auto-vectorisation
  • Explicit vectorisation using SIMD libraries
    • Google Highway, xsimd, vectorclass, eve, std::simd proposal for C++26 (P1928)
  • Jeff Garland: "SIMD Libraries in C++" (CppNow 2023)
  • Writing intrinsics
  • Writing assembly
  • SWAR ("SIMD Within A Register")

Other SIMD considerations

  • If you know the exact target CPU:
    • use arch compiler flags
  • If you don't:
    • dynamic dispatch
    • function multiversioning (GCC/Clang only)

function multiversioning:

#include <iostream>

// 1. Version optimized for modern CPUs with AVX2
__attribute__((target("avx2"))) 
void process_data() {
    std::cout << "Running high-performance AVX2 vectorized version!\n";
    // Fast vector math goes here
}

// 2. Version optimized for older SSE4.2 capable CPUs
__attribute__((target("sse4.2"))) 
void process_data() {
    std::cout << "Running mid-tier SSE4.2 version!\n";
}

// 3. The mandatory default fallback version
__attribute__((target("default"))) 
void process_data() {
    std::cout << "Running generic baseline version.\n";
}

int main() {
    // You call it like a regular function. 
    // The resolution happens completely behind the scenes.
    process_data(); 
    return 0;
}  

Autovectorisation

  • Highly dependent on compiler
  • Only works with vectors/arrays of int, char, double etc. - no structs
  • Workaround: struct of arrays instead of array of structs
  • Traverse data linearly
  • for loop, not while loop
  • Number of iterations predetermined (ideally, known at compile time)
  • No data-dependent break, goto, etc.
  • No conditionals
  • No data dependencies between array elements
  • No aliasing




Cache miss

any optimization is pointless if there's a cache miss.
Minimise data cache misses
  • Data locality
  • Align data on cachelines
  • Concurrency aspect (true/false sharing)
  • Contiguous data traversal (also good for prefetcher)
  • "Almost always vector"
  • Cache-friendly associative containers
  • std::flat_set/std::flat_map (C++23), Abseil containers
  • Cache-friendly algorithms
    • Cache-friendly binary search
Minimise instruction cache misses
  • Consider generated code layout & alignment
  • Avoid branches
  • Avoid virtual functions
    • std::variant
    • Compile-time polymorphism
    • CRTP, mixins, "deducing this" (since C++23)

Keep the cache warm
  • Data cache
    • Periodically poke data on a timer
  • Instruction cache
    • Periodically run the hot path with dummy input/output


Other fun hardware problems
  • CPU throttling
    • Due to overheating
      • spread thermal dissipation
    • Due to low activity
      • keep CPU busy with pause instructions




Programming for deterministic execution time

What not to do in the hot path:

  • Dynamic memory allocations/deallocations
  • blocking the thread
  • I/O
  • exceptions
  • context switches / mode switches (user/kernel space)
  • syscalls
  • calling into unknown code
  • loops without definite bounds
  • algorithms > O(1), or with no statically known upper bound on N

Dynamic memory allocations/deallocations

Avoiding allocations
  • Do not use data types that allocate
  • Do not use algorithms that allocate
  • Do not use data structures that allocate
Don't use(in real time):
  • std::stable_sort 
  • std::stable_partition 
  • std::inplace_merge

Do use(in real time):
std::array
std::pair
std::tuple
std::optional 
std::variant

Don't use(in real time):
std::any
std::function
std::vector, std::deque, std::list etc.


Customer allocators(also for coroutine):
tcmalloc, rpmalloc...
are not good for ultra low latency and real-time:
  • minimising average cost, not worst case
  • not constant time
  • multithreaded (locks)
  • eventually go to OS to request dynamic memory

Custom Allocators
Preallocate everything
  • Monotonic allocators
  • std::pmr::monotonic_buffer_resource
  • Pool allocators
  • std::pmr::unsynchronised_pool_resource
  • Frame allocators
  • Arena allocators
  • Double-ended allocators for big/small buffers
  • Lock-free allocator (require helper thread)


Lambda does not malloc.

Coroutine might.
rely on the optimiser?
→ Eyal Zedaka: "Using Coroutines to Implement C++
Exceptions for Freestanding Environments" (CppCon 2021)
  • create and suspend coroutine frame upfront
  • write your own promise type, defining its own custom operator new and operator delete
  • Don't use coroutines in a low-latency/real-time scenario


Blocking the thread

Don't use mutex. (and spanner violates all these rules in every aspects lol)


Concurrency
  • atomic == indivisible, race-free
  • lock-free == at least one thread is guaranteed to make progress
  • wait-free == all threads are guaranteed to make progress

Wait-free concurrency
  • Can't block hot path (no mutexes, no spinning)
  • Can't do any syscalls
  • Can't do anything with unbounded/non-deterministic runtime
  • Only available synchronisation mechanism:
  • always:
    • static_assert(std::atomic<T>::is_always_lock_free);
How:
  • Passing data to/from "hot path" thread
    • wait-free queue
  • Sharing data with "hot path" thread
    • Hot path reads
      • spinlock try_lock (involve syscall.) Solution: spinlock with try_lock.
      • "spin-on-write"
        • Easy to use
        • Reader always wait-free
        • Tradeoffs:
          • Single reader
          •  read is 2x faster then std::atomic::exchange; because .load read from
            local CPU cache(since the value is MESI shared)
          •  reader blocks writer(s) who need to spin
          • one writer (or multiple writers who block each other)
          • writer needs to heap-allocate + copy
        std::unique_ptr‹biquad_coefficients> storage;
        std::atomic<biquad_coefficients*> coeffs;
        void process(audio_buffer& buffer) { auto* current_coeffs = coeffs.exchange(nullptr); process_biquad(buffer, *current_coffs) ; coeffs.store(current_coeffs) ; } void update_coeffs (biquad_coefficients new_coeffs) { auto new_coeffs = std::make_unique‹biquad_coefficients>(new_coeffs); for (auto* expected = storage.get(); !coeffs.compare_exchange_weak(expected, new_coeffs.get()); expected = storage.get() /* spin */;) storage = std::move(new_coffs); } // Or just use crill lib crill::spin_on_write_object<biquad_coefficients> coeffs; void process (audio_buffer& buffer) { auto read_ptr = coeffs.lock_read(); process_biquad (buffer, *read_ptr); } void update_coeffs(biquad_coefficients new_coeffs) { coeffs.update(new_coeffs); }
      • RCU
        • read is now a single atomic load (= no overhead on modern platforms)
        • Multiple concurrent readers & writers
        • Readers don't block writers
        • But we need to solve deferred reclamation problem!
          • hard...
          • https://www.youtube.com/watch?v=7fKxIZOyBCE
            crill::defer_reclaim_object<biquad_coefficients> coeffs;
            void process(audio_buffer& buffer) {
              uto read_ptr = coeffs.Lock_read() ;
              process_biquad (buffer, *read_ptr);
            }
            
            void update_coeffs (biquad_coefficients new_coeffs) {
              coeffs.update(new_coeffs);
            }
            
            void timer_callback(){
              coeffs.reclaim();
            }
          • Reading always wait-free
          • Multiple readers
          • algorithm simplifies greatly if single reader
          • read is single atomic load (= no overhead on modern platforms)
          • Readers do not block writers
          • Writer needs to do heap allocation
          • User needs to manage reclamation
        • Variation: reclaim_on_write_object
          • Reading always wait-free
          • Multiple readers
          • algorithm simplifies greatly if single reader
          • read is single atomic load (= no overhead on modern platforms)
          • Writer waits for active reader(s) to finish
          • Writing is fast and does not require heap allocation
          • No need to manage reclamation
    • Hot path writes
      • Double-buffering (just atomic SWAP to update the data.)
        • Always be aware of ABA problem. Thus always having state in the single
          atomic variable.
          std::array<frequency_spectrum, 2> slots;
          std::atomic<int> idx = {0};
          
          void process (audio_buffer& audio_in) {
          	auto spectrum = calculate_spectrum(audio_in);
          	int write_id = id.load();
          	slots [write_idx] = spectrum;
          }
          
          void update_spectrum() {
          	int read_id = idx.fetch_xor(1);
          	draw_spectrum(slots[read_idx]);
          }
          
          // Solve the ABA problem
          std::array<frequency_spectrum, 2> slots;
          std::atomic<int> idx = {0};
          enum {
          	BIT_IDX = (1 << 0),
          	BIT_NEWDATA = (1 << 1), 
              BIT_BUSY = (1 << 2),
          };
          void process (audio_buffer& audio_in) {
          	auto spectrum = calculate_spectrum(audio_in);
          	int write_idx = idx.fetch_or(BIT_BUSY) & BIT_IDX;
          	slots [write_idx] = spectrum;
          	idx.store ((write_idx & BIT_IDX) | BIT_NEWDATA);
          }
          // ...
          
      • SeqLock (HFT technique)
        • Writing always wait-free
        • Good solution if writing happens more rarely, data not too large
        • Tradeoffs:
          • Single writer, multiple readers
          • Readers lock-free but not wait-free (might have to retry unbounded nr of times)
          • Data must be trivially copyable
          • Overhead of copying data atomically on both reader & writer
      • std::atomic<std::size_t> seq = 0;
        // single thread doing the write.
        void store(T t) noexcept {
        	std::size_t old_seq = seq.fetch_add(1);
            /*
            // Faster
            auto old_seq = seq.load(std::memory_order_relaxed);
            seq.store(old_seq + 1, std::memory_order_relaxed);
        
            std::atomic_thread_fence(std::memory_order_release);
            */
        	// write data...
        	seq.store(old_seq + 2);
        }
        
        bool try_load(T& t) const noexcept {
        	std::size_t seq1 = seq.load(std::memory_order_acquired);
        	if (seq1 % 2 != 0) return false;
        	
            // read data...
            /*
            
            Use Byte-wise atomic memcpy to read the data;
            or chunk it; DO NOT USE memcpy
            
            for (size_t i = 0; i < count; ++i) {
        		reinterpret_cast<char*>(dest)[i] =
        			atomic_ref<char>(reinterpret_cast<char*>(source)[i]).load(memory_order_relaxed);
            }
        	
            atomic_thread_fence(order);
            */
            
            std::atomic_thread_fence(std::memory_order_acquired);
        	std: :size_t seq2 = seq.load(std::memory_order_relaxed);
        	return seq1 == seq2;
        }
Ring-Buffer, as always



I/O

  • With other threads: Push message into wait-free specific queue
  • With other processes: Shared memory
  • With hardware: Direct Memory Access (DMA)

Reading data from disk on the hot path
  • Pre-load into RAM and lock address range (to prevent swap-out) / MMAP
    • mlock (POSIX)
      mlock(), mlock2(), and mlockall() lock part or all of the calling process's virtual address space into RAM, preventing that memory from being paged to the swap area. munlock() and munlockall() perform the converse operation, unlocking part or all of the calling process's virtual address space, so that pages in the specified virtual address range can be swapped out again if required by the kernel memory manager. Memory locking and unlocking are performed in units of whole pages.
    • VirtualLock (Windows)
  • Disk streaming
    • Pre-load into RAM first ~ 100 ms of every possible sound
    • Once it starts playing, start filling in the rest from disk (on a background thread)

No exceptions.

Avoiding context switches/mode switches
  • Mainstream operating systems: Thread priority
  • Real-time operating systems: deterministic thread scheduler
  • If you control the hardware:
    • Kernel bypass
  • If your hot path is in a single thread,
  • and you don't care about efficiency of other threads:
    • Turn off hyperthreading
    • Pin hot path thread to one CPU core