[C++/Rust] use of thread_local in code.

Reference:

1. All about thread-local storage by MaskRay
2. A Deep dive into (implicit) Thread Local Storage; in detail about use cases for thread_local.
3. ELF Handling For Thread-Local Storage by Ulrich Drepper
4. clang attribute 'tls-model'
5. How fast is thread local variable access on Linux
6. Mastering x86 Memory Segmentation
7. x86 and amd64 instruction reference

This note is focused on C++ coding practice with thread_local; knowledge are collected from daily engineering and references above.

C++ Language definitions:

1. Zero-initialization
   https://en.cppreference.com/w/cpp/language/zero_initialization
   https://vsdmars.blogspot.com/2014/04/c11-zero-initialisation-for-classes.html
2. Constant initialization
   https://en.cppreference.com/w/cpp/language/constant_initialization
   Init. Rule memorize: C.Z , Constant first if possible, then Zero init.
3. constinit specifier
   https://en.cppreference.com/w/cpp/language/constinit
4. Potentially-evaluated expressions
   https://en.cppreference.com/w/cpp/language/expressions#Potentially-evaluated_expressions
5. [C++20] consteval / constexpr
   https://vsdmars.blogspot.com/2022/06/cc20-consteval-constexpr.html
   
   

   

   
   
6. The thread_local keyword is only allowed for objects declared at namespace scope, objects declared at block scope, and static data members.
   It indicates that the object has thread storage duration.
   If thread_local is the only storage class specifier applied to a block scope variable, static is also implied.
   It can be combined with static or extern to specify internal or external linkage (except for static data members which always have external linkage) respectively.
   It can be combined with constinit to reduce overhead that would otherwise be incurred by a hidden guard variable. (i.e. static is thread safe guarded)
7. thread storage duration. The storage for the object is allocated when the thread begins and deallocated when the thread ends. Each thread has its own instance of the object(i.e. clone()). Only objects declared thread_local have this storage duration. thread_local can appear together with static or extern to adjust linkage.
8. thread_local is init. ordered in C.Z; i.e first init. with const-init; if can't, do zero-init.

When variable decorated with static or thread_local it will be constant initialized if possible than a runtime zero initialization. [[basic.start.static]]

i.e.

#include <iostream>
using namespace std;

bool runtimeFunc() {
  return std::is_constant_evaluated(); // always false
}

constexpr bool constexprFunc() {
  return std::is_constant_evaluated(); // may be false or true
}

consteval bool constevalFunc() {
  return std::is_constant_evaluated(); // always true
}

void foo() {
  static bool v1 = constexprFunc();       // T

  // implicit static
  thread_local bool v2 = constexprFunc(); // T
  thread_local bool v3 = constevalFunc(); // T

  int y = 42;
  static int v4 = y + runtimeFunc();         // 42
  static int v5 = y + constexprFunc();       // 42
  static int v6 = y + constevalFunc();       // 43

  // implicit static
  thread_local int v7 = y + runtimeFunc();   // 42
  thread_local int v8 = y + constexprFunc(); // 42
  thread_local int v9 = y + constevalFunc(); // 43
}

int main() { foo(); }

Usage

• thread_local should not be used in signal handler; while signal handler can be called in different threads, thus the fact that thread_local is not sync between threads can introduce buggy logic.
• thread_local is relatively slow in DSO use cases, use local caching instead.
      1 instruction in Windows, Linux
      3-4 in OSX
• in dlopen; DSO interacts with thread_local as follows:
• When a thread starts(i.e. clone()), init. thread_local objects with thread storage duration at namespace scope.
  When a thread exits, destruct objects with thread storage duration.
• What happens if the library is unloaded before all threads exit?
• In glibc, use RTLD_NODELETE, this will have DF_1_NODELETE set in ELF, thus does not unload the shared object during dlclose().
• Consequently, the object's static and global variables are not reinitialized if the object is reloaded with dlopen() at a later time.
• Also,  dlclose() in the middle of destructing thread_local objects is a no-op when RTLD_NODELETE is used.
• Use cases for thread_local in DSO can be slow due to __tls_get_addr@plt to get the address of the thread_local variable out of the DSO.
• Thread local variables should not be used in coroutines to prevent buggy logic.
  https://rules.sonarsource.com/cpp/RSPEC-6367
  If you have to use thread local inside a signal handler function, read:
  https://vsdmars.blogspot.com/2025/11/c-avoid-compiler-reordering-statements.html (std::atomic_signal_fence)
  
  
  
  
  

TCB: Thread control block

[C++] pointer-compression; from v8 oilpan-library point of view.

Reference:

• https://v8.dev/blog/oilpan-pointer-compression
• https://v8.dev/blog/oilpan-library
• oilpan code [google git]
• https://en.wikipedia.org/wiki/64-bit_computing#Limits_of_processors
• https://vsdmars.blogspot.com/2020/07/virtual-memory-refresh.html
• Pointer Compression in Oilpan design doc
• https://v8.dev/blog/pointer-compression

Pointer compression has been used in many opensource projects (e.g. cachelib, chrome);

the idea is to use less bits in 64-bit arch (usually 1 word/8 bytes for pointer, 2-words for pointer to member function) to present virtual memory address.

Thus the pointer size compression implementation design can be done as follows:

1. cage' (or slab) a range of heap memory block
2. The size of a heap cage is limited by the available bits for the offset. e.g., a 4GB heap cage requires 32-bit offsets.
   The compressed pointer contains only the offset index from the base address of the 'cage' heap virtual memory.
3. the 'cage' continuously heap virtual memory base address is per thread, thus, thread_local base pointer can be used here. However, thread local storage (TLS) is slow; thus Oilpan uses single caged heap memory per process.

Oilpan design requirements:

'Member' type instance(i.e. ref counted smart pointer) can take:

1. A valid heap pointer to an object;
2. The C++ nullptr (or similar);
3. A sentinel value which must be known at compile time. The sentinel value can e.g. be used to signal deleted values in hash tables that also support nullptr as entries.

nullptr has its own type domain; what 's value of compress(nullptr) ?

Is it nullptr means deleted object or just pointing to null?

Extra requirements:

1. Compress/decompress should be inlined at call site. (i.e. __attribute__((always_inline)) )
2. Fast and compact instruction sequence to minimize i-cache misses.
3. Branchless instruction sequence to avoid using up branch predictors.
4. Consider read/write separately. Read > Write counts; thus:
   Fast decompression is preferred.

Design idea:

The main idea for the scheme that is implemented as of today is to separate regular heap pointers from nullptr and sentinel by relying on alignment of the heap cage.

Thus, for cage heap memory, allocated it with alignment such that the least significant bit

of the upper half-word is always set.

cage heap memory allocated with alignment as base address:

0x00 00 00 01  |  00 00 00 00

nullptr:

0x00 00 00 00  |  00 00 00 00

sentinel:

0x00 00 00 00  |  00 00 00 02

Compression generates a compressed value by merely right-shifting by one and truncating away the upper half of the value. In this way, the alignment bit (which now becomes the most significant bit of the compressed value) signals a valid heap pointer.

e.g.

original heap memory:

00000000 00000000 00000000 00000001 00000000 11111111 00000000 00000000

compressed:

00000000 00000000 00000000 00000000 10000000 01111111 10000000 00000000

and truncating away the upper half:

10000000 01111111 10000000 00000000 (half word, the msb 1 indicates a valid heap memory address)

With this implementation, compressed nullptr become:

00000000 00000000 00000000 00000000

With this implementation, compressed sentinel become:

00000000 00000000 00000000 00000001

Note that this allows for figuring out whether a compressed value represents a heap pointer, nullptr, or the sentinel value, which is important to avoid useless decompressions in user code.

Decompression relies on a specifically crafted base pointer, in which the least significant 32 bits are set to 1.

Base:

0x00 00 00 01  |  FF FF FF FF

The decompression operation first sign extends the compressed value and then left-shifts to undo the compression operation for the sign bit.

And  the decompressed pointer is just the result of a bitwise and between this intermediate value and the base pointer. 

Heap pointer:

10000000 01111111 10000000 00000000 to

00000000 00000000 00000000 00000000 10000000 01111111 10000000 00000000 to

00000000 00000000 00000000 00000001 00000000 11111111 00000000 00000000 (first stage decompressed)

00000000 00000000 00000000 00000001 11111111 11111111 11111111 11111111 &

----------------------------------------------------------------------------------------------------------------

00000000 00000000 00000000 00000001 00000000 11111111 00000000 00000000 (decompressed)

nullptr:

00000000 00000000 00000000 00000000 000000000 00000000 00000000 00000000

00000000 00000000 00000000 00000001 111111111 11111111 11111111 11111111 &

----------------------------------------------------------------------------------------------------------------

00000000 00000000 00000000 00000000 000000000 00000000 00000000 00000000 (decompressed)

sentinel:

00000000 00000000 00000000 00000000 000000000 00000000 00000000 00000010

00000000 00000000 00000000 00000001 111111111 11111111 11111111 11111111 &

----------------------------------------------------------------------------------------------------------------

00000000 00000000 00000000 00000000 000000000 00000000 00000000 00000010 (decompressed)

Several gotcha in the article mentioned worth noted here:

1. Optimizing cage base load, the cage base pointer an't constexpr at runtime thus impedes compilter to reason for generating faster code. The Oilpan team tackle this with clang's attributes; i.e. using 
   __attribute__((require_constant_initialization));
   (https://chromium-review.googlesource.com/c/v8/v8/+/2739979/17/include/cppgc/member.h#38
   https://chromium.googlesource.com/chromium/src/+/f47da96363899cbe1b3b851119bb3409eac253e1/base/allocator/partition_allocator/pcscan.h#17
   https://clang.llvm.org/docs/AttributeReference.html#require-constant-initialization-constinit-c-20)
2. Avoiding decompression at all;
1. decompress nullptr to check if it's null
2. constructing or assigning a Member from another Member needs no decompression/compression
3. Comparison of pointers is preserved by compression, so we can avoid transformations for them as well.
4. Hashing can be sped up with compressed pointers. Decompression for hash calculation is redundant, because the fixed base does not increase the hash entropy. Instead, a simpler hashing function for 32-bit integers can be used.
   Blink has many hash tables that use Member as a key; the 32-bit hashing resulted in faster collections!
3. Helping clang where it fails to optimize; remove unnecessary decompression in memory barriar blocks.
4. While now the pointer has been compressed, be ware of padding since pointer is now size of int_32; using compressed pointer inside the structure should be padding considered.

TBD:

oilpan-library code dig. 

https://chromium.googlesource.com/v8/v8.git/+/HEAD/include/cppgc/

https://chromium.googlesource.com/v8/v8/+/main/samples/cppgc/hello-world.cc

[concurrent][mutex] Use queuing the spinner to accelerate lock acquisition

reference:
https://sourceware.org/ml/libc-alpha/2018-12/msg00808.html

Paper:
The performance of spinlock alternatives for shared-memory multiprocessors: https://www.cc.gatech.edu/classes/AY2009/cs4210_fall/papers/anderson-spinlock.pdf
Algorithms for scalable sync on shared-memory multiprocessors: http://www.cs.rochester.edu/~scott/papers/1991_TOCS_synch.pdf

Blog Reference:
http://vsdmars.blogspot.com/2018/09/c-something-about-spinlock.html

[Concurrency] [C++][Go] Wrap up - 2018

Modified

• Thie Core's cache line has the modified data.
• Data in memory won't be in other Core's cache line.

Exclusive

• Data aren't modified.
• Data in memory is the latest.
• If the Core has to evict the data inside the cache line,
  nothing need to be write back to memory.

Shared

• Data are shared between Cores' cache line.
• If this core has to modify the data, it needs to ask for data from other cores first.

Invalid

• The data in the cache line is null.

Operations:

Read

• read data from cache lines. Ask for other cores' for data.  

Read response

• Data for read request. Either from Memory or from Cores' cache.

Invalidate

• Invalidate the particular data inside Cores' cache lines.

Invalidate ack

• Response to invalidate request that the request is in the queue.      

Read invalidate

• Read + Invalidate.
• Will get read's data response and invalidate ack.

Writeback

• Write the data from cache line to memory.

Store buffer:

• Read invalidate request to other Cores' cache line for the to be modified data  is not necessary since the data response isn't needed because this Core is going to modify the data anyway.
• Thus, this core will write data to Store buffer first.
• Thus, Core will read data inside the Store Buffer with highest priority if the data exist in Store buffer and Cache line.
• However, there's an issue for Read invalidate request to other cores.
  Consider that data A in Core 1 cache, data B in Core 2 cache.
  And data A, data B has this happen before relationship.
  i.e
  In Core 1, update data A(Core 1), then update data B(Core 2).
  In Core 2, read data A(Core 1), verify data B(Core 2).

 
You see the problem. Between operation on A/B there's interleaves.
 
Core 1 updates data B, write into store buffer, send read invalidate to Core 2 on data B, at the same time, Core 2 reads data A, receives data A from Core 1, and verify data B as well, at the time, data B isn't invalidated yet.

Thus we need some wait mechanism.
i.e memory barrier.

In the above case, we need

• WMB(write memory barrier) on Core 1,
• RMB(read memory barrier) on Core 2. 

For Core 1, we do:
Update B, then WMB, then update data A.

For Core 2, we do:
Read data A, if A is old data, verify B won't happen, and if A is new data.
i.e
Core 1 updated data A, data B in Core 2 SHOULD use new one from Core 1, since it's invalidate queued.
But how to trigger Core 2 to verify invalidate queue? Before verify data B in Core 2, call RMB.
So the sequence for Core 2 will become:
Read Data A, RMB, verify data B.
     
 
WMB:

• Send read invalidate to Core 2 and till Core 2 send back invalidate ack will Core 1 flush data A from store buffer to it's cache line.

RMW:

• Verify invalidate queue, make data that is in the invalidate queue invalid the Core 2's cache line.
• While Core 2 tries to read the data, it will send a 'read' request to Core 1 for data B.

Take away:

• WMB should be issued AFTER a write to the shared data.
• RMW should be issued BEFORE a read to the shared data.

     
Reference:

1. Memory Barriers: a Hardware View for Software Hackers
2. https://en.cppreference.com/w/cpp/atomic/memory_order
3. https://en.cppreference.com/w/cpp/atomic/atomic_thread_fence

Code:

#include <atomic>
#include <iostream>
#include <thread>

using namespace std;
atomic<int> A{0};
atomic<int> B{0};

void t1()
{
    this_thread::sleep_for(1s);
    B.store(38, memory_order_relaxed);

    int a = 42;
    // WMB, prevents 'a' passing A.store.
    A.store(a, memory_order_release);
}

void t2()
{
    // RMB
    while (A.load(memory_order_acquire) != 42) {
        cout << "in while" << endl;
        // Prints 0 or 38 if B.store hasn't process yet.
        cout << B.load(memory_order_relaxed) << endl;
    }
    cout << "out while" << endl;
    // Always print 38.
    cout << B << endl;
}
int main()
{
    thread T1{t1};
    thread T2{t2};
    T1.join();
    T2.join();
}

Fence:

#include <atomic>
#include <iostream>
#include <thread>
using namespace std;
atomic<int> A{0};
atomic<int> B{0};

void t1()
{
    A.store(42, memory_order_relaxed);
    atomic_thread_fence(memory_order_release);
    // B.store can NEVER before A.store.
    B.store(38, memory_order_relaxed);
}

void t2()
{
    while (B.load(memory_order_relaxed) == 38) {
        cout << "in while loop\n";
        cout << A << endl; // Must be 42
        break;
    }
    cout << "out while loop\n";
}

int main()
{
    thread T1{t1};
    thread T2{t2};
    T1.join();
    T2.join();
}

Fence:

Release/Store

• Prevents all preceding memory operations from being reordered past subsequent writes.
• Prevents all following memory operations from being memory reordered before the write.

Acquire/Load

• Prevents all following memory operations from being reordered before this fence.
• Prevents all preceding memory operations from being reordered pass this fence.

Memory fences are NOT an acquire or release operation.

Operation:

Release operation:  store

• Cannot be reordered by compiler.
• Prevents preceding memory operations from being reordered past itself.
  i.e Any operations after a store can be reordered before the store operation.
• Any read or write operation that precedes it in program order.
  i.e those in memory_order_relaxed mode can't be reordered.

     
Acquire operation:  load

• https://en.cppreference.com/w/cpp/atomic/atomic_load
• Cannot be reordered by compiler.
• Any read or write operation that follows it in program order.
  i.e those in memory_order_relaxed mode can't be reordered.
• Those before the load can be memory ordered pass the load. Not as strong as fence.

BUT with different variable(object):
A release operation followed by a acquire operation CAN be reordered.
A acquire operation followed by a release operation CAN be reordered.
i.e
    A.store(1, std::memory_order_release);
    int b = B.load(std::memory_order_acquire);
=>
    int b = B.load(std::memory_order_acquire);
    A.store(1, std::memory_order_release);

However, keep in mind that (different variable/object) even reorder is OK for release/acquire operation, standard also depicts:
http://eel.is/c++draft/intro.multithread#intro.progress-18
An implementation should ensure that the last value (in modification order) assigned by an atomic or synchronization operation will become visible to all other threads in a finite period of time.

Reference:

https://stackoverflow.com/questions/14687703/what-c11-atomic-operations-memory-orders-guarantees-freshness

https://stackoverflow.com/questions/8819095/concurrency-atomic-and-volatile-in-c11-memory-model/8833218#8833218

i.e
If an store operation follows a for/while loop and a load operation,
the compiler shouldn't reorder load before store due to it can't
reasoning that the for/while loop is finite thus other thread can see thread store result in a finite time.
   
Reference:

1. [Preshing] Can Reordering of Release/Acquire Operations Introduce Deadlock?
2. [Bruce Dawson] In Praise of Idleness
3. [Golang bug list] cmd/compile: go1.8 regression: sync/atomic loop elided #19182

Hardware Memory Model:

Weak memory model

• ARM/Power PC

Strong memory model

• x86/64
• Sequential Consistence (software)

Reordering:

Sequentially Consistent types:

• Java: volatile variables
• C++11: atomic (default)

Explicit Compiler Barriers:
gcc:
    asm volatile("" ::: "memory");
Macro:
    #define COMPILER_BARRIER() asm volatile("" ::: "memory")

source:
https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/barrier.h#L22

figure source: (https://www.youtube.com/watch?v=ZQFzMfHIxng)

Don't consider Sequential Consistency is SLOW; correctness is the highest priority.

Implied Compiler Barriers:
C++11:

• Every non­-relaxed atomic operation acts as a compiler barrier.
  這裡是compiler reordering, 不是memory reordering.
• Every function containing a compiler barrier must act as a compiler barrier itself, even the function is inlined.
• The majority of function calls act as compiler barriers, whether they contain their own compiler barrier or not due to it's coming from different TU. 
• But does not include inline functions, functions declared with the pure attribute, and cases where link ­time code generation is used.          

Reference:

1. [Implications of pure and constant functions] https://lwn.net/Articles/285332/

What about Golang?

goroutine act as 'user space thread', 

i.e it's runtime memory location is allocated on the heap.

Inside a single goroutine, the compiler is allowed to re-arrange

expressions as long as the reordering does not change the behavior within that goroutine as defined by the language specification. 

Within a single goroutine, the happens-before order is the order expressed by the program.

i.e

a := 42
if a == 42 {
    // always true.
}

// b will always be 42
b := a
// within other goroutines, the observe of a == 42 and c == 38 sequence is not guaranteed.
c := 38  

Reads and writes of values larger than a single machine word behave as multiple machine-word-sized operations in an unspecified order.

Initialization:

• If a package p imports package q, the completion of q's init functions happens before the start of any of p's.
• The start of the function main.main happens after all init functions have finished.

Goroutine creation:

• The go statement that starts a new goroutine happens before the goroutine's execution begins.
• i.e go statement act as a barrier function call in C/C++ which won't be reordered.

Goroutine destruction:

• Can happen any time.
• Be ware that if a goroutine that updates a global value and do nothing (i.e using channel etc.), an aggressive compiler could delete the goroutine entirely for optimization.
  i.e just modify the global variable without creating a goroutine.
  https://github.com/golang/go/issues/19182

Channel communication:

• A send on a channel happens before the corresponding receive from that channel completes.
• The closing of a channel happens before a receive that returns a zero value because the channel is closed.
• A receive from an un-buffered channel happens before the send on that channel completes.
• The kth receive on a channel with capacity C happens before the k+Cth send from that channel completes.

i.e the famous:
Do not communicate by sharing memory; instead, share memory by communicating.

No more memory store/load operation from programmer's point of view.

The Channel acted as the sequential consistence/memory fence contract.

Reference:

https://golang.org/ref/mem

https://blog.golang.org/share-memory-by-communicating

https://github.com/8l/go-learn

https://dougrichardson.org/2016/01/23/go-memory-allocations.html

[Preventing stack guard-page hopping] https://lwn.net/Articles/725832/

[C++] Always consider function thread safeness drags performance in single thread code.

std::cin / std::cout  are thread safe according to CPP ISO (30.2.3 Thread safety),
thus means it's slow in single thread.

Use std::fstream instead.

std::shared_ptr is thread safe for internal ref counting.
That is to say, using std::shared_ptr is slow in single thread as well.
Use _move_ for std::shared_ptr or just using std::unique_ptr.


When running single threaded code, for performance, always think
potential function call is thread safe or not, thus could impact performance.

Ref:

• reddit: YET ANOTHER "JAVA FASTER THAN C++" CLAIM
• GCC's shared_ptr detects whether the executable is linked to libpthread and uses non-atomic updates when possible.

code:

#ifdef __GLIBCXX__
template<typename T>
  using single_threaded_shared_ptr = std::__shared_ptr<T, std::_S_single>;
#else
template<typename T>
  using single_threaded_shared_ptr = std::shared_ptr<T>;
#endif

auto p = std::__make_shared<T, std::_S_single>(args...);

Reference:
https://stackoverflow.com/questions/32317370/avoid-cost-of-stdmutex-when-not-multi-threading
https://stackoverflow.com/questions/3652056/how-efficient-is-locking-an-unlocked-mutex-what-is-the-cost-of-a-mutex
Why is the std::function () operator const?

[C++11][NOTE] About memory barrier

Reference:

Memory Barriers Are Like Source Control Operations
Why Memory Barrier？
CPU Cache Flushing Fallacy
Memory Barriers/Fences
The SC- Memory Model for Java