[errno] errno, base::context has the same idea(used in Dapper trace), stored in TLS.

errno IS NOT A REGULAR VARIABLE

• Stored in Thread Local Storage (TLS)
• A pointer to errno is returned via libc’s __errno_location() function
• On i686 the TLS block is referenced by %gs segment register; on x86-64 via %fs.
• All syscalls return between -4095 and -1 on failure.
• Even the syscall() function sets errno

More about why [-1, -4095] range.

This specific range (-4095 to -1) is a clever hack to solve the "Pointer Ambiguity" problem.

The kernel developers needed a way for a function to return either a valid memory address (a pointer) or an error code, using only a single return register (like rax).

1) The Conflict

Imagine a system call like mmap(), which asks the kernel for memory.

Success: It returns a pointer (e.g., memory address 0x7f43a...).

Failure: It needs to return an error code (e.g., -12 for "Out of Memory").

If the kernel just returned -12, the computer sees 0xFFFFFFFFFFFFFFF4 (in 64-bit unsigned hex). 

The Risk: What if there is actually a valid piece of memory at address 0xFFFFFFFFFFFFFFF4? The program wouldn't know if it got a valid pointer or an error.

2) The Solution: The "Forbidden" Page

To fix this, the kernel reserves the very last page of virtual memory (the top 4KB) as a "No-Go Zone."

The Range: -4095 corresponds to the start of that last 4KB page.

-1 = 0xFFFFFFFFFFFFFFFF (The very top)

-4095 = 0xFFFFFFFFFFFFF001 (4KB down)

The Logic: No valid pointer is allowed to point to this very last page of memory. It is reserved specifically to hold error codes.

Why 4095? Because the standard memory page size is 4096 bytes (4KB). By reserving exactly one page, they minimized wasted address space while ensuring there's enough room for all standard error codes (there are usually only ~130 distinct errno values).

[auxiliary vector] auxv

man:
getauxval

$ LD_SHOW_AUXV=1 /bin/ls

#include <sys/auxv.h>
#include <stdio.h>

int main() {
    // 1. Check for specific hardware features (simplified example)
    unsigned long hwcaps = getauxval(AT_HWCAP);
    
    // 2. Get the pointer to the 16 random bytes provided by kernel
    void *random_ptr = (void *)getauxval(AT_RANDOM);
    
    printf("Hardware Caps Bitmask: %lx\n", hwcaps);
    printf("Address of random seed: %p\n", random_ptr);
    
    return 0;
}

[futex2] io_uring_prep_futex_waitv

man:
io_uring_prep_futex_waitv

Why futex2?

1. Wait on Multiple Mutexes (futex_waitv)
   The single biggest driver for futex2 was the gaming industry (Valve/CodeWeavers).
   
   The Problem: Windows has a function called WaitForMultipleObjects. It allows a thread to sleep until any one of a list of locks/events becomes available.
   
   Game Engine Logic: "Sleep until the GPU is done OR a network packet arrives OR the user presses a key."
   
   
1. Old Linux futex: Could only wait on one address. To emulate Windows, Wine had to create complex polling loops or use expensive eventfd file descriptors, killing performance.
2. The futex2 Solution: The new syscall sys_futex_waitv accepts an array of futexes. The kernel puts the thread to sleep and wakes it up if any of the futexes in the array are triggered.
   
   
2. Variable Sized Locks (Not just 32-bit)
   The original sys_futex was strictly designed for 32-bit integers.
   If you wanted a tiny lock (boolean flag), you wasted 32 bits.
   If you wanted a 64-bit lock (e.g., storing a pointer or a full counter), you couldn't do it atomically via the kernel.
   
   futex2 introduces explicit sizing:
1. 8-bit: Great for boolean flags.
2. 16-bit: Useful for small counters.
3. 32-bit: Legacy standard.
4. 64-bit: Crucial for modern 64-bit pointer tagging and high-contention counters

This allows languages like Rust, Go, or C++ to implement synchronization primitives that map naturally to their native data types without padding or casting.

#include <linux/futex.h>
#include <unistd.h>
#include <liburing.h>

struct futex_waitv {
    uint64_t val;  // Expected value
    uint64_t uaddr; // Address of the futex
    uint32_t flags; // Flags (e.g., 32-bit vs 64-bit)
    uint32_t __reserved;
};

// Syscall prototype
// blocks until at least one futex wakes up
syscall(SYS_futex_waitv, struct futex_waitv *waiters, unsigned int nr_futexes, ...);

// io_uring lib api
void io_uring_prep_futex_waitv(struct io_uring_sqe *sqe,
	struct futex_waitv *futexv,
	uint32_t nr_futex,
	unsigned int flags);

[Unix IO] minute for IO models - W. Richard Stevens

Blocking IO

Nonblocking I/O Model

I/O Multiplexing Model

Disadvantage: using select requires two system calls (select and recvfrom) instead of one

Advantage: we can wait for more than one descriptor to be ready (see the select function later in this chapter)

Signal-Driven I/O Mode

Signal-driven I/O is rarely used in modern applications because its disadvantages generally outweigh its benefits. Newer APIs like epoll (Linux), kqueue (BSD/macOS), and IOCP (Windows) are far superior.

1. Complexity of Signal Handling: Dealing with signals is notoriously difficult and error-prone. Signal handlers have many restrictions (e.g., only a limited set of functions, known as async-signal-safe functions, can be safely called from within them).

2. Unreliable Signal Queuing: Signals are not queued. If two I/O events occur in rapid succession, the kernel might only deliver a single SIGIO signal. This means the signal handler must be written in a loop to read (or write) until the operation would block (returning an EWOULDBLOCK or EAGAIN error). Forgetting this leads to lost I/O events.

3. Complexity in Multithreading: Signal handling in a multithreaded program is extremely complex. It's often unclear which thread will receive the signal, leading to difficult synchronization problems.

4. Still a Synchronous Model: According to the POSIX standard, signal-driven I/O is still a synchronous I/O model. While the notification is asynchronous, the actual I/O call (e.g., recvfrom()) is initiated by the process and can still block in the signal handler or main loop. This is different from true asynchronous I/O (AIO), where the kernel performs the entire operation (including the data copy) and only notifies the process upon completion.

Asynchronous I/O Model
The main difference between this model and the signal-driven I/O model is that with signal-driven I/O, the kernel tells us when an I/O operation can be initiated, but with asynchronous I/O, the kernel tells us when an I/O operation is complete.

Advantage:

• True Asynchronicity: This is the only model (besides the modern io_uring) that is truly asynchronous. The application is completely unblocked and can perform other computations while the I/O is in progress.
• Parallel I/O and Computation: It allows an application to overlap its computations with its I/O operations, which can lead to significant performance gains, especially in data-intensive applications like database servers.
• Request Queuing: You can submit (queue) multiple I/O requests to the kernel at once, allowing the kernel to potentially optimize the scheduling of these operations (e.g., re-ordering disk reads).

Disadvantages:

The POSIX AIO model, despite its theoretical benefits, is rarely used and generally not recommended on Linux for several critical reasons:

• Poor Linux Implementation: This is the biggest problem. The standard glibc implementation of POSIX AIO is not a true kernel-level AIO. Instead, it's implemented in user-space by creating a pool of worker threads. When you call aio_read(), it just hands the request to one of these hidden threads, which then performs a normal blocking read(). This adds all the overhead of threading and synchronization, often making it slower and more resource-intensive than just managing your own thread pool.
• Limited to Disk Files: Even the "true" kernel AIO support (which requires using O_DIRECT) works only for disk files. It does not work for network sockets. For high-performance networking, epoll (I/O multiplexing) is the standard.
• API Complexity: The API is complex, requiring you to manage aiocb (AIO control block) structures for every request and handle notifications, which often fall back to signals (with all their associated problems) or require you to poll for completion.
• Superseded by io_uring: On modern Linux, AIO is considered obsolete. io_uring is the modern, high-performance interface for true asynchronous I/O. It works for both file I/O and network I/O, is vastly more efficient, and is designed to eliminate the flaws of POSIX AIO.

Comparison chart:

[kernel] Restartable sequences

Restartable sequences in glibc:
https://lwn.net/Articles/883104/

resq struct

[kernnel][C][C++] ternary conditional operator trick in action

Reference:
Return type of '?:' (ternary conditional operator): https://stackoverflow.com/questions/8535226/return-type-of-ternary-conditional-operator

__is_constexpr() macro is dark magic: https://lore.kernel.org/linux-hardening/20220131204357.1133674-1-keescook@chromium.org/?fbclid=IwAR0Rgg_tGDk0qiEyuDsuZwERITSdstxmU2-bOtadb7iOOCtw3tgOPKEQ5hE

Using ternary conditional operator to get the type we want for a template type is often used in C++.

Here the same idea applies in C macro:

#define __is_constexpr(x) \
	(sizeof(int) == sizeof(*(8 ? ((void *)((long)(x) * 0l)) : (int *)8)))

Details:

 - sizeof() is an integer constant expression, and does not evaluate the

   value of its operand; it only examines the type of its operand.

 - The results of comparing two integer constant expressions is also

   an integer constant expression.

 - The use of literal "8" is to avoid warnings about unaligned pointers;

   these could otherwise just be "1"s.

 - (long)(x) is used to avoid warnings about 64-bit types on 32-bit

   architectures.

 - The C standard defines an "integer constant expression" as different

   from a "null pointer constant" (an integer constant 0 pointer).

 - The conditional operator ("... ? ... : ...") returns the type of the

   operand that isn't a null pointer constant. This behavior is the

   central mechanism of the macro.

 - If (x) is an integer constant expression, then the "* 0l" resolves it

   into a null pointer constant, which forces the conditional operator

   to return the type of the last operand: "(int *)".

 - If (x) is not an integer constant expression, then the type of the

   conditional operator is from the first operand: "(void *)".

 - sizeof(int) == 4 and sizeof(void) == 1.

 - The ultimate comparison to "sizeof(int)" chooses between either:

     sizeof(*((int *) (8)) == sizeof(int)   (x was a constant expression)

     sizeof(*((void *)(8)) == sizeof(void)  (x was not a constant expression)

For a conditional expression (?:) to be an lvalue, the second and third operands must be lvalues of the same type.

This is because the type and value category of a conditional expression is determined at compile time and must be appropriate whether or not the condition is true.

If one of the operands must be converted to a different type to match the other than the conditional expression cannot be an lvalue as the result of this conversion would not be an lvalue (but a r-value).

Thus:

OK:

int x = 1;
int y = 2;
(x > y ? x : y) = 100; // l-value on the left side of =

Not OK:

int x = 1;
long y = 2;
(x > y ? x : y) = 100; // type conversion to long as r-value type

[kernel] Thread Control Block (TCB) and Process control block(PCB)

Reference:

https://en.wikipedia.org/wiki/Process_control_block

https://en.wikipedia.org/wiki/Thread_control_block

https://en.wikipedia.org/wiki/X86_memory_segmentation

https://en.wikipedia.org/wiki/Protection_ring

A Deep dive into (implicit) Thread Local Storage: https://chao-tic.github.io/blog/2018/12/25/tls

[split stack] reading notes and references: https://vsdmars.blogspot.com/2019/01/split-stack-reading-notes-and-references.html

TCB(Thread control block) or TLS(Thread Local Storage) setup is done somewhat differently for statically linked executables and dynamically linked executables.

dynamically linked executables

TLS is initialised differently for the main thread and the other threads that begin later in the execution.

When it comes to dynamically linked ELF programs, it’s useful to know that once they are loaded and mapped into memory, the kernel would then take its hands off and pass the execution baton to the dynamic linker (ld.so on Linux).

The main thread’s TLS is setup in the function init_tls, which calls _dl_allocate_tls_storage to allocate the TCB or struct pthread, eventually invokes the aforementioned macro TLS_INIT_TP to bind the pthread TCB to the main thread.

A module in glibc can refer to either an executable or a dynamically shared object, a module ID therefore is an index number for a loaded ELF object in a process.

Note that for a given running process the module ID for the main executable will always be 1, whereas the shared objects don’t know their module IDs until they are loaded and assigned by the linker.

Dynamically-loaded modules here don’t mean any dynamic shared objects, they only refer to the shared objects that are loaded by explicitly calling dlopen.

[virtual memory] recap

Reference:

Set-associative cache

Recap

Program address in linux/elf

Virtual Memory is used due to:

• RAM is not sufficient
• Holes in address space
• Programs writing over each other

Programs can access any byte in their 64-bit address space

Page Out

• While not enough memory, paging out(base on different algorithm) for free physical memory and flush data into next level storage system(usually disk)

    

Page Tables

• Page Table Entry(PTE) (usually 4kb page size (32bit) or 2mb (64bit))

Page faults (slow)

• Pages not in RAM
• CPU generates a page fault exception caught by OS
• OS flush out data from memory to give room for data on disk

Virtual Address <-> Page Table <-> Physical Address(PA)

Virtual Address layout: (2^32 or 2^64 physical memory wide)

20bit virtual page number (virtual page number, map to physical page number) + 12bit page offset (does not get translated into PA, the size 

depends on page size, e.g 4KB we need 2^12, 12 bits offset, i.e 2^12 = 4096 bytes = 4KB, remember, in memory, the basic unit is byte)

Physical Address layout: (2^N where N is the size of physical memory are installed on this box)

16bit physical page number + 12bit page offset (same as VA's page offset)

Page Table Size

Page table remains in the memory all the time, for lookup.

Page Table Lookup

Page Table Indirect

How to make virtual memory FAST?

List operations:

• Access the page table in RAM
• Translate the address
• Access the data in RAM

Cache for page table: Translation Lookaside Buffer(TLB)

    Virtual Address <-> TLB(small and fast) <-> PA

TLB should be small

• iTLB (for instructions)
• dTLB (for data)
• Second TLB
• 64 entries, 4 way (4kb pages)
• 32 entries, 4 way (2mb pages)

i.e If program runs in 4kb/2mb pages of data, there's no update needed for TLB.

Good: Page in RAM

• PTE (page table entry) in the TLB
• PTE not in the TLB

Bad: Page not in RAM

• PTE in the TLB(unlikely)
• PTE not in the TLB

TLB layout

tag + physical page number

• tag: physical memory index (virtual address(VA)'s virtual page number)
• physical page number: physical address

TLBs and CPU caches(L1/L2/L3)

• Physical Cache (slow, must do TLB lookup before accessing the cache)
• Virtual Cache (fast, TLB lookups only when having a cache miss)

VIPT (Virtually Indexed, Physically Tagged)

• Data in the cache in indexed by the virtual address
• But tagged by the physical address
• Only get a hit if the tag matches the physical address, but can start looking with the virtual address
  i.e TLB translation and Cache lookup at the same time(in parallel).

Reference:
David Black-Schaffer - virtual memory

[kernel] livepatch

Check livepatch enabled in the kernel:

$ zcat /proc/config.gz | grep LIVEPATCH
$ cat /boot/config-$(uname -r) | grep LIVEPATCH
$ ls -ld /sys/kernel/livepatch

How to know if the kernel is patched properly?

Look into directory:

$ /sys/kernel/livepatch

Check tainted flag from /proc

$ cat /proc/sys/kernel/tainted

Caveats

• Make sure kernel has module load function enabled
  $ cat /proc/sys/kernel/modules_disabled

[kernel] Address Space Layout Randomization

Reference:
Address space layout randomization (linux):
https://en.wikipedia.org/wiki/Address_space_layout_randomization#Linux
Modern Binary Exploitation:
http://security.cs.rpi.edu/courses/binexp-spring2015/lectures/15/09_lecture.pdf
How Effective is ASLR on Linux Systems?
https://securityetalii.es/2013/02/03/how-effective-is-aslr-on-linux-systems/


interface:
/proc/sys/kernel/randomize_va_space

• 0 – No randomization. Everything is static. 
• 1 – Conservative randomization. Shared libraries, stack, mmap(), VDSO and heap are randomized. 
• 2 – Full randomization. In addition to elements listed in the previous point, memory managed through brk() is also randomized.

Prior to 2.6.22 had a similar problem where VDSO (linux-vdso.so) was always located at a fixed location. (https://vsdmars.blogspot.com/2018/06/vdso-function-exported-to-user-space.html)

Unless compiled with PIE elf executable is not guarded by ASLR.

[randomness][kernel][note]

Reference:
On Linux's Random Number Generation: https://research.nccgroup.com/2019/12/19/on-linuxs-random-number-generation/
Myths about /dev/urandom (nice read):
https://www.2uo.de/myths-about-urandom/
Removing the Linux /dev/random blocking pool:
https://lwn.net/Articles/808575/
ChaCha20: https://en.wikipedia.org/wiki/Salsa20#ChaCha20_adoption

/dev/random     # may blocks, don't use it.
/dev/urandom    # does not blocks, just use this <---
getrandom()

[kernel] io_uring

Reference:
System Calls http://man7.org/linux/man-pages/man2/syscalls.2.html
[LWN] Ringing in a new asynchronous I/O API
[LWN] Redesigned workqueues for io_uring

[kernel][C++] Linux-Kernel Memory Model

Reference:
Linux-Kernel Memory Model http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0124r5.html

Kernel Document https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/memory-model/Documentation/

What is RCU, Fundamentally? https://lwn.net/Articles/262464/

[futex] futex skim through

Notes from reading Eli Bendersky's blog post:
Basics of Futexes

Background:

System calls are expensive.
Context switch happens between userspace and kernel space.
Thus, we have VDSO ( http://vsdmars.blogspot.com/search/label/linux_vdso ) to relax some system calls' burden, as for locks, we have futex.
The difference here is that futex doesn't involve any business logic but simply to acquire the lock to access memory concurrently safe.


It is likely that when a thread acquires a lock, the lock hasn't been locked yet.
In this case, no system call involved, a compare_and_swap atomic instruction would be enough. ( cmpxhg , which is cheaper than a system call )
Reference:
https://stackoverflow.com/a/27856649
The high-level locks that lock-free algorithms try to avoid can guard arbitrary code fragments whose execution may take arbitrary time and thus, these locks will have to put threads into wait state until the lock is available which is a costly operation, e.g. implies maintaining a queue of waiting threads.

This is an entirely different thing than the CPU LOCK prefix feature which guards a single instruction only and thus might hold other threads for the duration of that single instruction only. Since this is implemented by the CPU itself, it doesn’t require additional software efforts.

Therefore the challenge of developing lock-free algorithms is not the removal of synchronization entirely, it boils down to reduce the critical section of the code to a single atomic operation which will be provided by the CPU itself.



However; if there's lock needed, the atomic CAS would fail.

2 choices here:

1. busy 'for loop CAS' (spinlock), which consumes CPU core power. Although it's in userspace, still a very bad idea.
   Reference:
   https://vsdmars.blogspot.com/2018/09/c-something-about-spinlock.html
2. "sleep" (i.e pause) until the lock free.
   Reference:https://vsdmars.blogspot.com/2018/09/c-something-about-spinlock.html
   As for 'pause':
   Pause Intrinsic can help prevent a busy wait from completely overwhelming the system, by inserting pauses in the instruction stream that prevent the busy loop from overwhelming the processor.
   This is particularly important on hyperthreaded systems since it gives the other logical core time to run.
   If must busy wait then be sure to use pause. 

Reference:
http://man7.org/linux/man-pages/man2/futex.2.html
The futex() system call provides a method for waiting until a certain condition becomes true.  It is typically used as a blocking construct in the context of shared-memory synchronization.  When using futexes, the majority of the synchronization operations are performed in user space.  A user-space program employs the futex() system call only when it is likely that the program has to block for a longer time until the condition becomes true.  Other futex() operations can be used to wake any processes or threads waiting for a particular condition.

Focus on 2 futex system calls:

• FUTEX_WAIT (mutex.Lock)
  waits on an event.
  Caller is suspended by the kernel and will only be scheduled awake when there's a wake-up signal.
• FUTEX_WAKE (mutex.Unlock)
  signals an event.

Go by example code from Eli Bendersky


Child process:

1. Waits for 0xA to be written into a shared memory slot.
2. Writes 0xB into the same memory slot.

Parent process:

1. Writes 0xA into the shared memory slot.
2. Waits for 0xB to be written into the slot.

wait_on_futex_value:
loop that waits
pause if the val is the expected value, but not yet being waked yet.
If val is not the expected value, continue looping.
Then another process sent out wake event, stop pausing, check if the val is the expected value, i.e not a spurious wake up call, if is, returns.
FUTEX_WAIT (since Linux 2.6.0)
This operation tests that the value at the futex word pointed to by the address uaddr still contains the expected value val,and if so, then sleeps waiting for a FUTEX_WAKE operation on the futex word.  The load of the value of the futex word is an atomic memory access (i.e., using atomic machine instructions of the respective architecture).  This load, the comparison with the expected value, and starting to sleep are performed atomically and totally ordered with respect to other futex operations on the same futex word.  If the thread starts to sleep, it is considered a waiter on this futex word.  If the futex value does not match val, then the call fails immediately with the error EAGAIN.

The purpose of the comparison with the expected value is to prevent lost wake-ups.  If another thread changed the value of the futex word after the calling thread decided to block based on the prior value, and if the other thread executed a FUTEX_WAKE operation (or similar wake-up) after the value change and before this FUTEX_WAIT operation, then the calling thread will observe the value change and will not start to sleep.

If the timeout is not NULL, the structure it points to specifies a timeout for the wait.  (This interval will be rounded up to the system clock granularity, and is guaranteed not to expire early.)  The timeout is by default measured according to the CLOCK_MONOTONIC clock, but, since Linux 4.5, the CLOCK_REALTIME clock can be selected by specifying FUTEX_CLOCK_REALTIME in futex_op.  If timeout is NULL, the call blocks indefinitely.
           

wake_futex_blocking:
send wake up event.
FUTEX_WAKE (since Linux 2.6.0)
This operation wakes at most val of the waiters that are waiting (e.g., inside FUTEX_WAIT) on the futex word at the address uaddr.  Most commonly, val is specified as either 1 (wake up a single waiter) or INT_MAX (wake up all waiters). No guarantee is provided about which waiters are awoken (e.g., a waiter with a higher scheduling priority is not guaranteed to be awoken in preference to a waiter with a lower priority).

• Futexes are kernel queues for userspace code.
• A futex is a queue the kernel manages for userspace convenience.
• Futex allows userspace code asking the kernel to suspend until a certain condition is meet, and allows other userspace code signal that condition and wake up the waiting processes.
• Futexes are implemented in kernel/futex.c
• Kernel keeps a hash table keyed by the address to quickly find the proper queue data structure and adds the calling process to the wait queue.

figure: https://lwn.net/Articles/360699/

Timed blocking with FUTEX_WAIT


Ok, isn't this familiar?
Golang's channel + context timeout.
And yet we could use Golang channel to implement a mutex lock.


Reference:
https://eli.thegreenplace.net/2018/basics-of-futexes/
http://man7.org/linux/man-pages/man2/futex.2.html
A futex overview and update
[C++] something about spinlock
Futexes are tricky [PDF]

[stack guard-page] Preventing stack guard-page hopping reading notes

Reference:
Preventing stack guard-page hopping

The stack area in a running process is, on most architectures,
placed at a relatively high virtual address; it grows downward as the process's stack needs increase.

A virtual-memory region that automatically grows as a result of page faults brings some inherent risks; in particular, it must be prevented from growing into another memory region placed below it.

In a single-threaded process, the address space reserved for the stack can be large and difficult to overflow.

Multi-threaded processes contain multiple stacks; those stacks are smaller and are likely to be placed between other virtual-memory areas of interest.

An accidental overflow could corrupt the area located below a stack; a deliberate overflow, if it can be arranged, could be used to compromise the system.

The kernel has long placed a guard page — a page that is inaccessible to the owning process — below each stack area.

A process that wanders off the bottom of a stack into the guard page will be rewarded with a segmentation-fault signal, which is likely to bring about the process's untimely end.

The world has generally assumed that the guard page is sufficient to protect against stack overflows but, it seems, the world was mistaken.

The fundamental problem with the guard page is that it is too small.
There are a number of ways in which the stack can be expanded by more than one page at a time.

• make large alloca() calls
• with large variable-length arrays or other large on-stack data structures.

It turns out to be relatively easy for an attacker to cause a program to generate stack addresses that hop over the guard page, stomping on whatever memory is placed below the stack.

Partial solution: extend the guard page size: 4KB -> 1MB
Configurable at boot time:
stack_guard_gap

[network][vxlan] wrap-up

rfc7348: https://tools.ietf.org/html/rfc7348

• VXLAN Tunnel End point(VTEP)
• Head-end Replication
• VxLAN MAC Distribution

( figure orig: http://underlays.blogspot.com/ )

Reference:
https://cn-blogs-prod-alln.cisco.com/datacenter/digging-deeper-into-vxlan
https://cn-blogs-prod-alln.cisco.com/datacenter/vxlan-deep-dive-part-2-looking-at-the-options
https://cn-blogs-prod-alln.cisco.com/datacenter/digging-deeper-in-vxlan-pt-3-more-faqs

[linux][namespace] Mount (mnt)

Mount namespaces and shared subtrees
Reference: 
https://lwn.net/Articles/689856/
https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt
https://www.kernel.org/doc/html/v5.0/
http://man7.org/linux/man-pages/man7/mount_namespaces.7.html

• Each mount namespace has its own list of mount points.
  When the system is first booted, there is a single mount namespace,
  the so-called "initial namespace".
• New mount namespaces are created by using the CLONE_NEWNS flag
  with either the clone() system call (to create a new child process in the new namespace)
  or the unshare() system call (to move the caller into the new namespace).
• When a new mount namespace is created, it receives a copy of the mount point list replicated from the namespace of the caller of clone() or unshare().
• Changes to the mount point list are (by default) visible only to processes in the mount namespace where the process resides;
  the changes are not visible in other mount namespaces.

Shared subtrees:
We do not want to re-mount a DVD-ROM in every mount space.
Each mount point is marked with a "propagation type",
which determines whether mount points created and removed
under this mount point are propagated to other mount points.

4 shared types:

• MS_SHARED
  Make this mount point shared.
  Mount and unmount events immediately under this mount point will propagate to the other mount points that are members of this mount's peer group. Propagation here means that the same mount or unmount will automatically occur under all of the other mount points in the peer group.
  Conversely, mount and unmount events that take place under peer mount points will propagate to this mount point.
• MS_PRIVATE
  Make this mount point private.
  Mount and unmount events do not propagate into or out of this mount point.
• MS_SLAVE
  If this is a shared mount point that is a member of a peer group that contains other members, convert it to a slave mount.
  If this is a shared mount point that is a member of a peer group that contains no other members, convert it to a private mount. 
  Otherwise, the propagation type of the mount point is left unchanged.
  When a mount point is a slave, mount and unmount events propagate into this mount point from the (master) shared peer group of which it was formerly a member.
  Mount and unmount events under this mount point do not propagate to any peer.
  A mount point can be the slave of another peer group while at the same time sharing mount and unmount events with a peer group of which it is a member.
• MS_UNBINDABLE
  Make this mount unbindable.
  This is like a private mount, and in addition this mount can't be bind mounted.  When a recursive bind mount (mount() with the MS_BIND and MS_REC flags) is performed on a directory subtree, any unbindable mounts within the subtree are automatically pruned (i.e., not replicated) when replicating that subtree to produce the target subtree.

Peer groups:
A peer group is a set of mount points that propagate mount and unmount events to one another.


Examining propagation types and peer groups via
/proc/PID/mountinfo:
The /proc/PID/mountinfo file (documented in the proc(5) manual page) displays a range of information about the mount points for the mount namespace in which the process PID resides. All processes that reside in the same mount namespace will see the same view in this file.


List current process's mount information:
$ cat /proc/self/mountinfo | sed 's/ - .*//'

[linux][namespace] wrap-up

Linux Namespace is relatively new idea in the linux space which is the
fundamental of containers as well as Kubernetes.

Those design/api were not mentioned in the TLPI book, which becomes more essential nowadays due to the rise of distributed computing(sever-less, lambda, whatever fancy words you name it~)

This page is used as index page for further linux namespace ideas/design/programming.
Currently my coding language are C++(modern)/Golang/Python.


Traditional process resource limit:
http://man7.org/linux/man-pages/man1/prlimit.1.html
$ prlimit --nofile=256 --nproc=512 --locks=32 /bin/bash


ps shows PPID/SID:
$ ps -efj


Linux namespace directories for debugging:
/proc/*/ns/*
/proc/*/task/*/ns/*
/proc/self/ns  # caller's namespace information
/proc/sys/kernel/ns_last_pid # the last PID that was allocated in this PID namespace.
/run/netns/netns-name  # created network namespace
/run/netns/default  # default network namespace
/sys/fs/cgroup # cgroup information

Namespace is differentiated by ID(integer)


List all namespace ID with in all processes:
$ readlink /proc/*/task/*/ns/* | sort -u


List all namespace under root PID 1 namespace:
This can be used to find the default linux namespaces
$ readlink /proc/1/task/*/ns/* | sort -u


Use bind mount to persist a linux namespace:
Reference:
https://unix.stackexchange.com/a/198591

$ mount --bind /proc/pid/ns/type /anywhere/you/want
Thus later on you could use nsenter(1)/unshare(1)(2)/setns(2) to
enter that namespace.


Linux PID namespace has some extended behaviors which should be noticed:

• A process's namespace is settled when it's created. Period.
  It CANNOT be changed even with 'setns'.
  'setns' will only associated the child created by the caller PID with the new
  namespace but not the caller itself.
  (Since Linux 4.12, that new PID namespace is shown via the
  /proc/[pid]/ns/pid_for_children file.)
  Once the caller PID calls 'setns', all it's children will be put into the new PID namespace.
  The children's call to 'getppid(2)' will return 0 since they
  CANNOT observe the PID outside it's own PID namespace.
  Beware, processes may not enter any ancestor namespaces (parent, grandparent, etc.).
  Changing PID namespaces is a one-way operation.
  Use ioctl_ns to get the parent namespace information.
  code: https://github.com/verbalsaintmars/ns_show
  
  That is to say,
  PID namespace parent/child namespace relationship honors the design of
  2 layer relationship in Session/Process Group(Job), Process Group/Process, Parent Process/Child Process.
• Ancestor namespace PIDs can send kill signals to other PID namespace's PID 1 which honors the 'kill' system call privilege checks, plus, the other PID namepace's PID 1 has the corresponding signal handlers installed.
• Starting with Linux 3.4, the reboot(2) system call causes a signal to be sent to the namespace "init" process.
• If the "init" process of a PID namespace terminates, the kernel
  terminates all of the processes in the namespace via a SIGKILL signal.
• If the 'init' process, which usually is PID 1, terminates, and later on there's new PID want's to join this PID linux namespace which has the 'init' process terminated, the new PID called by fork will error out with errorno: ENOMEM, which is: 'fork cannot allocate memory'
  (the ENOMEM comes from the 'PIDNS_HASH_ADDING' has been unset once PID 1 dies which calls disable_pid_allocation() and if a new PID intends to be created by calling alloc_pid(), ENOMEM is set.)
• Thus, 'unshare' with or without -f behaves as:
  -f (use fork):
  --fork will thus telling 'unshare' to fork the 'cmd' into the new namespace as the first existing process. (i.e PID 1)
  
  without -f (use exec):
  the 'cmd' is not running in the new pid namespace but it's fork process is.
  Reference:
  https://stackoverflow.com/a/45973522 https://unix.stackexchange.com/a/393279
• PID namespaces can be nested, except for the 'default' PID namespace.
  Since Linux 3.7, the kernel limits the maximum nesting depth for PID namespaces to 32 (Nesting PID namespaces).
  A process can see (e.g., send signals with kill(2), set nice values with setpriority(2), etc.) only processes contained in its own PID namespace and in descendants of that namespace.
• A call to getpid(2) always returns the PID associated with the
  namespace in which the process was created.
• In current versions of Linux,
  CLONE_NEWPID can't be combined with CLONE_THREAD.
  Threads are required to be in the same PID namespace such that the threads in a process can send signals to each other.
  Similarly, it must be possible to see all of the threads of a
  processes in the proc(5) filesystem.
• A /proc filesystem shows (in the /proc/[pid] directories) only processes visible in the PID namespace of the process that performed the mount, even if the /proc filesystem is viewed from processes in other namespaces.
  That's the reason 'unshare' provides '--mount-proc' argument, which mounts /proc with in the new created PID namespace with new Mount namespace.
• When a process ID is passed over a UNIX domain socket to a process in a different PID namespace, it is translated into the corresponding PID value in the receiving process's PID namespace.

'unshare' with or without -f behaves explained:

The error is caused by the PID 1 process exits in the new namespace.

After bash start to run, bash will fork several new sub-processes to do somethings.
If you run unshare without -f, bash will have the same pid as the current "unshare" process.
The current "unshare" process call the unshare systemcall, create a new pid namespace, but the current "unshare" process is not in the new pid namespace.
It is the desired behavior of linux kernel: process A creates a new namespace, the process A itself won't be put into the new namespace, only the sub-processes of process A will be put into the new namespace. So when you run:
$ unshare -p /bin/bash

The unshare process will exec /bin/bash, and /bin/bash forks several sub-processes, the first sub-process of bash will become PID 1 of the new namespace, and the subprocess will exit after it completes its job.
So the PID 1 of the new namespace exits.

The PID 1 process has a special function:
It should become all the orphan processes' parent process.
If PID 1 process in the root namespace exits, kernel will panic.
If PID 1 process in a sub namespace exits, linux kernel will call the disable_pid_allocation function, which will clean the PIDNS_HASH_ADDING flag in that namespace.
When linux kernel create a new process, kernel will call alloc_pid function to allocate a PID in a namespace, and if the PIDNS_HASH_ADDING flag is not set, alloc_pid function will return a -ENOMEM error. That's why you got the "Cannot allocate memory" error.

You can resolve this issue by use the '-f' option:
$ unshare -fp /bin/bash

If you run unshare with '-f' option, unshare will fork a new process after it create the new pid namespace. And run /bin/bash in the new process. The new process will be the pid 1 of the new pid namespace.
Then bash will also fork several sub-processes to do some jobs.
As bash itself is the pid 1 of the new pid namespace, its sub-processes can exit without any problem.

Reference:
Namespaces in operation(lwn.net): https://lwn.net/Articles/531114/#series_index

Resource management: Linux kernel Namespaces and cgroups: http://www.haifux.org/lectures/299/netLec7.pdf
Control groups series by Neil Brown https://lwn.net/Articles/604609/

[Go][goroutine] linux namespace with goroutine

Linux Namespaces and Go Don't Mix
HN: Linux Namespaces and Go Don't Mix (weave.works)
reddit: https://www.reddit.com/r/golang/comments/6ew883/linux_namespaces_and_go_dont_mix/


Cause:
When ever things are blocking, go runtime will fork a M(physical thread) into spinning mode for new goroutine to run before every things are blocked.

However, go runtime will fork the thread with whatever Linux Namespace it's in with the same Linux Namespace scope, which causing problem.

[linux][kernel] linux namespace

Namespace kinds:

• Mount (mnt) $ 
• CLONE_NEWNS
• clone(child_fn, child_stack+1048576, CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | SIGCHLD, NULL)
• Process ID (pid)
• CLONE_NEWPID
• pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | SIGCHLD, NULL);
• Network (net)
• CLONE_NEWNET
• pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | CLONE_NEWNET | SIGCHLD, NULL);
• $ ip link add name veth0 type veth peer name veth1 netns <pid>
• Interprocess Communication (ipc)
• CLONE_NEWIPC
• Having their own interprocess communication resources, i.e  System V IPC and POSIX messages.
• UTS
• CLONE_NEWUTS
• Isolates two specific identifiers of the system: nodename and domainname.
• pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWUTS | SIGCHLD, NULL);
• User ID (user)
• CLONE_NEWUSER
• Allows a process to have root privileges within the namespace, without giving it that access to processes outside of the namespace.
• Control group (cgroup)
  
• CLONE_NEWCGROUP
• Time namespace
  The process can have a distinct view of CLOCK_MONOTONIC and/or CLOCK_BOOTTIME which can be changed using /proc/self/timens_offsets
  man : time_namespaces

Three syscalls can directly manipulate namespaces: 

• clone, flags to specify which new namespace the new process should be migrated to. 
• unshare, Allows the caller process (or thread) to disassociate parts of its execution context that are currently being shared with other processes (or threads) , and put the caller process into the NEW namespace. (Except for PID unshare)
• setns, The caller process joins a particular namespace through FD (Except for PID setns, which the caller's child will be put into new namespace).
• ioctl(2),  Various ioctl(2) operations can be used to discover information about namespaces.  These operations are described in ioctl_ns(2).

Cross-Namespace Communication:

• create namespace PID child process first.
• since child and parent share the same network namespace at the moment, child new PID namespace process can establish unix socket FD first and then call unshare() to create a new network namespace.

Creation of new namespaces using clone(2) and unshare(2) in most cases requires the CAP_SYS_ADMIN capability, since, in the new namespace, the creator will have the power to change global resources that are visible to other processes that are subsequently created in, or join the namespace. 

User namespaces are the exception: since Linux 3.8, no privilege is required to create a user namespace.

reference:

http://man7.org/linux/man-pages/man7/namespaces.7.html
https://www.toptal.com/linux/separation-anxiety-isolating-your-system-with-linux-namespaces
How to list all namespaces:
https://unix.stackexchange.com/a/113561

read:

Linux Namespaces and Go Don't Mix
The Curious Case of Pid Namespaces:

• From TLPI Ch.20.5 (p.402)
• The init process (process ID 1), which runs with user and group of root, is a special case. It can be sent only signals for which it has a handler installed. This prevents the system administrator from accidentally killing init, which is fundamental to the operation of the system.
• https://github.com/krallin/tini
• https://docs.docker.com/engine/reference/run/#specify-an-init-process