[errno] errno, base::context has the same idea(used in Dapper trace), stored in TLS.

errno IS NOT A REGULAR VARIABLE

• Stored in Thread Local Storage (TLS)
• A pointer to errno is returned via libc’s __errno_location() function
• On i686 the TLS block is referenced by %gs segment register; on x86-64 via %fs.
• All syscalls return between -4095 and -1 on failure.
• Even the syscall() function sets errno

More about why [-1, -4095] range.

This specific range (-4095 to -1) is a clever hack to solve the "Pointer Ambiguity" problem.

The kernel developers needed a way for a function to return either a valid memory address (a pointer) or an error code, using only a single return register (like rax).

1) The Conflict

Imagine a system call like mmap(), which asks the kernel for memory.

Success: It returns a pointer (e.g., memory address 0x7f43a...).

Failure: It needs to return an error code (e.g., -12 for "Out of Memory").

If the kernel just returned -12, the computer sees 0xFFFFFFFFFFFFFFF4 (in 64-bit unsigned hex). 

The Risk: What if there is actually a valid piece of memory at address 0xFFFFFFFFFFFFFFF4? The program wouldn't know if it got a valid pointer or an error.

2) The Solution: The "Forbidden" Page

To fix this, the kernel reserves the very last page of virtual memory (the top 4KB) as a "No-Go Zone."

The Range: -4095 corresponds to the start of that last 4KB page.

-1 = 0xFFFFFFFFFFFFFFFF (The very top)

-4095 = 0xFFFFFFFFFFFFF001 (4KB down)

The Logic: No valid pointer is allowed to point to this very last page of memory. It is reserved specifically to hold error codes.

Why 4095? Because the standard memory page size is 4096 bytes (4KB). By reserving exactly one page, they minimized wasted address space while ensuring there's enough room for all standard error codes (there are usually only ~130 distinct errno values).

[Unix IO] minute for IO models - W. Richard Stevens

Blocking IO

Nonblocking I/O Model

I/O Multiplexing Model

Disadvantage: using select requires two system calls (select and recvfrom) instead of one

Advantage: we can wait for more than one descriptor to be ready (see the select function later in this chapter)

Signal-Driven I/O Mode

Signal-driven I/O is rarely used in modern applications because its disadvantages generally outweigh its benefits. Newer APIs like epoll (Linux), kqueue (BSD/macOS), and IOCP (Windows) are far superior.

1. Complexity of Signal Handling: Dealing with signals is notoriously difficult and error-prone. Signal handlers have many restrictions (e.g., only a limited set of functions, known as async-signal-safe functions, can be safely called from within them).

2. Unreliable Signal Queuing: Signals are not queued. If two I/O events occur in rapid succession, the kernel might only deliver a single SIGIO signal. This means the signal handler must be written in a loop to read (or write) until the operation would block (returning an EWOULDBLOCK or EAGAIN error). Forgetting this leads to lost I/O events.

3. Complexity in Multithreading: Signal handling in a multithreaded program is extremely complex. It's often unclear which thread will receive the signal, leading to difficult synchronization problems.

4. Still a Synchronous Model: According to the POSIX standard, signal-driven I/O is still a synchronous I/O model. While the notification is asynchronous, the actual I/O call (e.g., recvfrom()) is initiated by the process and can still block in the signal handler or main loop. This is different from true asynchronous I/O (AIO), where the kernel performs the entire operation (including the data copy) and only notifies the process upon completion.

Asynchronous I/O Model
The main difference between this model and the signal-driven I/O model is that with signal-driven I/O, the kernel tells us when an I/O operation can be initiated, but with asynchronous I/O, the kernel tells us when an I/O operation is complete.

Advantage:

• True Asynchronicity: This is the only model (besides the modern io_uring) that is truly asynchronous. The application is completely unblocked and can perform other computations while the I/O is in progress.
• Parallel I/O and Computation: It allows an application to overlap its computations with its I/O operations, which can lead to significant performance gains, especially in data-intensive applications like database servers.
• Request Queuing: You can submit (queue) multiple I/O requests to the kernel at once, allowing the kernel to potentially optimize the scheduling of these operations (e.g., re-ordering disk reads).

Disadvantages:

The POSIX AIO model, despite its theoretical benefits, is rarely used and generally not recommended on Linux for several critical reasons:

• Poor Linux Implementation: This is the biggest problem. The standard glibc implementation of POSIX AIO is not a true kernel-level AIO. Instead, it's implemented in user-space by creating a pool of worker threads. When you call aio_read(), it just hands the request to one of these hidden threads, which then performs a normal blocking read(). This adds all the overhead of threading and synchronization, often making it slower and more resource-intensive than just managing your own thread pool.
• Limited to Disk Files: Even the "true" kernel AIO support (which requires using O_DIRECT) works only for disk files. It does not work for network sockets. For high-performance networking, epoll (I/O multiplexing) is the standard.
• API Complexity: The API is complex, requiring you to manage aiocb (AIO control block) structures for every request and handle notifications, which often fall back to signals (with all their associated problems) or require you to poll for completion.
• Superseded by io_uring: On modern Linux, AIO is considered obsolete. io_uring is the modern, high-performance interface for true asynchronous I/O. It works for both file I/O and network I/O, is vastly more efficient, and is designed to eliminate the flaws of POSIX AIO.

Comparison chart:

[elf][note] entry function

ELF executable entry function

_start function facts

• supplied by libc
• written in assembly
• store in a file called crt0.s
• Compilers typically ship with pre-compiled crt0.o object files for each supported architecture.
  https://wiki.osdev.org/Creating_a_C_Library

_start function implements

1. Early low-level initialization, such as
1. Configuring processor registers
2. Initializing external memory
3. Enabling caches
4. Configuring the MMU
2. Stack initialization, making sure that the stack is properly aligned per the ABI requirements
3. Frame pointer initialization
4. Initialization of the C/C++ runtime
   Relocate any relocatable sections (if not handled by the loader or linker)
1. Initializing global and static memory
2. Runtime initializes a subset of uninitialized memory (no = in the declaration) to 0.
   This includes global and static variables, but not stack variables. All uninitialized data that needs to be set to 0 is placed into the .bss section of the compiled program image by the linker. The location of the .bss section is identified during initialization, and the memory is typically set to 0 with memset.
3. C++ global objects must be constructed before calling main. The linker places these constructors into the .init, .init_array, or .ctors section of the image.
   Some compilers also allow C and C++ functions to be marked as a constructor using a compiler attribute (e.g., __attribute__((constuctor))). The constructors are stored in a list by the linker.
   The runtime initialization process iterates through the list and calls each constructor.
4. Prepare the argc and argv variables for invoking main (even if it’s just setting these to 0/NULL)
5. Perform any additional setup steps required by the C/C++ standard library implementation.
   These additional runtime initialization steps are run for many programs (but not all):
1. Heap initialization
2. Initialize stdio (i.e., stdin, stdout, stderr)
3. Initialize exception support (if using C++)
4. Register destructors and other cleanup functions that will run when exiting the program (using atexit and __cxa_atexit)
5. Assembly files commonly found during this portion of the startup process are crtbegin.s, crtend.s, crti.s, and crtn.s.
6. Prepare environment variables
5. Initialization of other scaffolding required by the system
     Program scaffolding setup before main might include:
1. Threading support and thread local storage
2. Buffer overrun detection
3. Stack logging
4. Run-time error checks
5. Locale settings
6. Math error handling
7. Default math library precision
6. Jumping to main
7. Exiting the program with the return code from main

So, how do we get to the _start?

• Baremetal: Reset Vector
• Bootloader Launches Application
• OS Calls an exec function
  Loaders will often perform the following actions:
• Check permissions
• Allocate space for the program’s stack
• Allocate space for the program’s heap
• Initialize registers (e.g., stack pointer)
• Push argc, argv, and envp onto the program stack
• Map virtual address spaces
• Dynamic linking
• Relocations
• Call pre-initialization functions

[C++][CPPCON 2021] s.t about Dynamically Loaded Libraries

Reference:
https://youtu.be/-dxCaM4GOqs

https://man7.org/linux/man-pages/man3/dlopen.3.html

https://man7.org/linux/man-pages/man3/dlmopen.3.html

https://sourceware.org/glibc/wiki/LinkerNamespaces

https://github.com/yugr/Implib.so

Definition:

Dynamic linking, opposed to static linking form a physical aspect of a program relocation is done at load time.

Dynamic loading ask for additional functionalities often involve library discovery relocation is done at run time maybe capable of 'unloading'.

in C++, object do not relocate but functions do.

Or to be precise, C/Assembly function kind do relocate, thus Golang could have dynamic growing stack size which relocate function at runtime.

After dlopen, unloading is often avoided due to this makes function having a 'lifetime'.

e.g. musl's dlcose is a noop. (complicating thread-local storage(TLS) implementation if library may unload)

Can objects from a library outlive the library?

Library lifetime realized in C++

- When loading a library, init. objects with static storage duration at namespace scope.

- When unloading a library, destruct objects with static storage duration.

- Loaded libraries are reference counted.

Interaction with thread_local

- When a thread starts, init. objects with thread storage duration at namespace scope.

- When a thread exits, destruct objects with thread storage duration.

- What happens if the library is unloaded before all threads exit?

glibc:

RTLD_NODELETE

Do not unload the shared object during dlclose().

Consequently, the object's static and global variables are not reinitialized if the object is reloaded with dlopen() at a later time.

DF_1_NODELETE (elf.h)

stackoverflow answer

- Set flag on the DSO until all thread_local objects defined in the DSO are destroyed

- After the flag being cleared, a subsequent dlclose() unloads the DSO

- i.e. dlclose() in the middle of destructing thread_local objects is a no-op

Unloading summary:

- Functions may have lifetime

- Implementations need to prevent objects with thread storage duration from outliving their destructors

[linux] missing zh-tw characters fix

$ cd /etc/fonts/conf.d

modify 65-nonlatin.conf and change <prefer> with font family contains zh-TW fonts.
e.g

  Set preferable fonts for non-Latin
	
		serif
		
			LiHei Pro
		
	
	
		sans-serif
		
			LiHei Pro
		
	
	
		monospace
		
			LiHei Pro
		
	

[unix][programming] EINTR and What It Is Good For (http://250bpm.com/blog:12)

Reference:
EINTR and What It Is Good For Martin Sústrik, zeromq


Before we dive, this concept is well mentioned in Richard Stevens's UNIX Network Programming - Ch.20.5, thus Martin Sústrik's blog post can be considered as a recap of EINTR error.

Rule of thumb: 

When handling EINTR error, check any conditions that may have been altered by signal handlers.
Then restart the blocking function.

Additionally, If you are implementing a blocking function yourself, take care to return EINTR when you encounter a signal.

Beware those 2 POSIX functions which don't honor EINTR

• pthread_cond_wait 
  From the emit event side should honor EINTR. or use sem_wait 
• pthread_mutex_lock
  Not usually a problem due to lock will be unlocked soon.

Consider this code:

volatile int stop = 0;

void handler (int)
{
    stop = 1;
}

void event_loop (int sock)
{
    signal (SIGINT, handler);

    while (1) {
        if (stop) {  // never hit if recv is blocked
            printf ("do cleanup\n");
            return;
        }
        char buf [1];
        recv (sock, buf, 1, 0);  // block call
        printf ("perform an action\n");
    }
}

Above is the reason POSIX has EINTR error.

Modify code to this:
noted that to make blocking functions like recv return EINTR you may have to use sigaction() with SA_RESTART set to zero instead of signal() on some operating systems.

volatile int stop = 0;

void handler (int)
{
    stop = 1;
}

void event_loop (int sock)
{
    signal (SIGINT, handler);

    while (1) {
        if (stop) {
            printf ("do cleanup\n");
            return;
        }
        char buf [1];
        int rc = recv (sock, buf, 1, 0);
        if (rc == -1 && errno == EINTR)  // if interrupted by signal, continue while loop
            continue;
        printf ("perform an action\n");
    }
}

But, this isn't a graceful shutdown.
We have to exhaust the incoming message before exit.
When you press Ctrl+C, program exits performing the clean-up beforehand.

The morale of this story is that common advice to just restart the blocking function when EINTR is returned doesn't quite work:

volatile int stop = 0;

void handler (int)
{
    stop = 1;
}

void event_loop (int sock)
{
    signal (SIGINT, handler);

    while (1) {
        if (stop) {
            printf ("do cleanup\n");
            return;
        }
        char buf [1];
        while (1) {
            // even signaled with stop == 1, and no more incoming data, we are stucked here..
            int rc = recv (sock, buf, 1, 0); 
            // if signaled, continue to recv, otherwise, message consumed, break inner loop
            if (rc == -1 && errno == EINTR) 
                continue;
            break;
        }
        printf ("perform an action\n");
    }
}

Even EINTR is not completely water-proof, check this code:

volatile int stop = 0;

void handler (int)
{
    stop = 1;
}

void event_loop (int sock)
{
    signal (SIGINT, handler);

    while (1) {
        if (stop) {
            printf ("do cleanup\n");
            return;
        }

        /*  What if signal handler is executed at this point? */
        /* pressing Ctrl+C for the second time sorts the problem out */

        char buf [1];
        // even stop == 1, and no more data coming, we are stucked here...
        int rc = recv (sock, buf, 1, 0);
        if (rc == -1 && errno == EINTR)
            continue;
        printf ("perform an action\n");
    }
}

Ultimate solution

use pselect, which mask the signals before calling pselect, and allow signal to pass during the pselect(which if signal occurs, pselect returns).

select

int select(int nfds,
                    fd_set *readfds,
                    fd_set *writefds,
                    fd_set *exceptfds, 
                    struct timeval *timeout);
                    

nfds should be n + 1 (exclusive bound), this optimizing the linear check of fds.

Be sure to check the definition under what conditions is a Descriptor ready for network FDs.

Notice that when an error occurs on a socket, both readable and writable is marked by select.

Although the timeval structure lets us specify a resolution in microseconds, the actual resolution supported by the kernel is often more coarse.
Many Unix kernels round the timeout value up to a multiple of 10ms. There is also a scheduling latency involved, meaning it takes some time after the timer expires before the kernel schedules this process to run.

void FD_ZERO(fd_set *fdset);
void FD_SET(int fd, fd_set *fdset);
void FD_CLR(int fd, fd_set *fdset);
int FD_ZERO(int fd, fd_set *fdset);

pselect

 int pselect(
            int nfds,
            fd_set *restrict readfds,
            fd_set *restrict writefds,
            fd_set *restrict errorfds,
            const struct timespec *restrict timeout,
            const sigset_t *restrict sigmask);

example:

// https://github.com/k84d/unpv13e/blob/master/bcast/dgclibcast4.c
#include "unp.h"

static void recvfrom_alarm(int);

void
dg_cli(FILE *fp, int sockfd, const SA *pservaddr, socklen_t servlen)
{
 int    n;
 const int  on = 1;
 char   sendline[MAXLINE], recvline[MAXLINE + 1];
 fd_set   rset;
 sigset_t  sigset_alrm, sigset_empty;
 socklen_t  len;
 struct sockaddr *preply_addr;
 
 preply_addr = Malloc(servlen);

 Setsockopt(sockfd, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on));

 FD_ZERO(&rset);

 Sigemptyset(&sigset_empty);
 Sigemptyset(&sigset_alrm);
 Sigaddset(&sigset_alrm, SIGALRM);

 Signal(SIGALRM, recvfrom_alarm);

 while (Fgets(sendline, MAXLINE, fp) != NULL) {
  Sendto(sockfd, sendline, strlen(sendline), 0, pservaddr, servlen);

  Sigprocmask(SIG_BLOCK, &sigset_alrm, NULL);
  alarm(5);
  for ( ; ; ) {
   FD_SET(sockfd, &rset);
   n = pselect(sockfd+1, &rset, NULL, NULL, NULL, &sigset_empty);
   if (n < 0) {
    if (errno == EINTR)
     break;
    else
     err_sys("pselect error");
   } else if (n != 1)
    err_sys("pselect error: returned %d", n);

   len = servlen;
   n = Recvfrom(sockfd, recvline, MAXLINE, 0, preply_addr, &len);
   recvline[n] = 0; /* null terminate */
   printf("from %s: %s",
     Sock_ntop_host(preply_addr, len), recvline);
  }
 }
 free(preply_addr);
}

static void
recvfrom_alarm(int signo)
{
 return;  /* just interrupt the recvfrom() */
}

poll

int poll(
        struct pollfd *fds,
        nfds_t nfds,
        const struct timespec *tmo_p,
        const sigset_t *sigmask);
        

[linux][performance] tools for monitoring

Reference:
https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55

cmds:

$ uptime
$ dmesg | tail
$ vmstat -m (virtual memory statistics, slab memory info)
$ mpstat -P ALL (Report processors related statistics.)
$ pidstat (Report statistics for Linux tasks)
$ iostat -xz (Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.)
$ free -m (memory info)
$ top

https://serverfault.com/a/913089
$ systemctl start  sysstat sysstat-collect.timer sysstat-summary.timer
$ systemctl enable sysstat sysstat-collect.timer sysstat-summary.timer
$ sar -n DEV 1 (Collect, report, or save system activity information: )
$ sar -n TCP,ETCP 1

[hugepage][kernel][notes]

Reference:
k8s feature-gates:
https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

HugePages:
https://wiki.debian.org/Hugepages

Huge pages part 1, Introduction: https://lwn.net/Articles/374424/
Huge pages part 2, Interfaces: https://lwn.net/Articles/375096/

mmap: http://man7.org/linux/man-pages/man2/mmap.2.html

golang mmap: https://godoc.org/golang.org/x/exp/mmap

Use mmap With Care:
https://www.sublimetext.com/blog/articles/use-mmap-with-care
Nice write up, taking advantage of memory pages to load large files into memory with mmap(). Some gotcha to be aware of if files are located on network drive(e.g nfs).

[linux][kernel] linux namespace

Namespace kinds:

• Mount (mnt) $ 
• CLONE_NEWNS
• clone(child_fn, child_stack+1048576, CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS | SIGCHLD, NULL)
• Process ID (pid)
• CLONE_NEWPID
• pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | SIGCHLD, NULL);
• Network (net)
• CLONE_NEWNET
• pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWPID | CLONE_NEWNET | SIGCHLD, NULL);
• $ ip link add name veth0 type veth peer name veth1 netns <pid>
• Interprocess Communication (ipc)
• CLONE_NEWIPC
• Having their own interprocess communication resources, i.e  System V IPC and POSIX messages.
• UTS
• CLONE_NEWUTS
• Isolates two specific identifiers of the system: nodename and domainname.
• pid_t child_pid = clone(child_fn, child_stack+1048576, CLONE_NEWUTS | SIGCHLD, NULL);
• User ID (user)
• CLONE_NEWUSER
• Allows a process to have root privileges within the namespace, without giving it that access to processes outside of the namespace.
• Control group (cgroup)
  
• CLONE_NEWCGROUP
• Time namespace
  The process can have a distinct view of CLOCK_MONOTONIC and/or CLOCK_BOOTTIME which can be changed using /proc/self/timens_offsets
  man : time_namespaces

Three syscalls can directly manipulate namespaces: 

• clone, flags to specify which new namespace the new process should be migrated to. 
• unshare, Allows the caller process (or thread) to disassociate parts of its execution context that are currently being shared with other processes (or threads) , and put the caller process into the NEW namespace. (Except for PID unshare)
• setns, The caller process joins a particular namespace through FD (Except for PID setns, which the caller's child will be put into new namespace).
• ioctl(2),  Various ioctl(2) operations can be used to discover information about namespaces.  These operations are described in ioctl_ns(2).

Cross-Namespace Communication:

• create namespace PID child process first.
• since child and parent share the same network namespace at the moment, child new PID namespace process can establish unix socket FD first and then call unshare() to create a new network namespace.

Creation of new namespaces using clone(2) and unshare(2) in most cases requires the CAP_SYS_ADMIN capability, since, in the new namespace, the creator will have the power to change global resources that are visible to other processes that are subsequently created in, or join the namespace. 

User namespaces are the exception: since Linux 3.8, no privilege is required to create a user namespace.

reference:

http://man7.org/linux/man-pages/man7/namespaces.7.html
https://www.toptal.com/linux/separation-anxiety-isolating-your-system-with-linux-namespaces
How to list all namespaces:
https://unix.stackexchange.com/a/113561

read:

Linux Namespaces and Go Don't Mix
The Curious Case of Pid Namespaces:

• From TLPI Ch.20.5 (p.402)
• The init process (process ID 1), which runs with user and group of root, is a special case. It can be sent only signals for which it has a handler installed. This prevents the system administrator from accidentally killing init, which is fundamental to the operation of the system.
• https://github.com/krallin/tini
• https://docs.docker.com/engine/reference/run/#specify-an-init-process

[VDSO] system call functions exported to user space through VDSO

clock_gettime
getcpu
getpid
getppid
gettimeofday
set_tid_address  <= interesting

reference:
vdso (7) - Linux Man Pages -- vdso: overview of the virtual ELF dynamic shared object
What are vdso and vsyscall?

[system] linux perf

http://www.brendangregg.com/linuxperf.html
https://www.reddit.com/r/linux/comments/4x4smu/linux_performance_tools_full_version_draft/

[linux] linux inside book

Linux Inside

[Linus] Memory barriers discuss

Linus: memory_barriers discuss


The "barrier()" macro is there entirely to defeat the compiler reordering
that is sometimes deadly.

For example, the compiler doesn't know _squat_ about locking, and that's
just as well, since compiler-generated locking tends to suck dead donkeys
through a straw (ie monitors etc that some languages have, and that are
tied to the data structures and tend to be overly careful since the
compiler doesn't actually understand what's going on).

But not knowing about locking means that if you have code like this:

        local_cpu_lock = 1;

        .. do something critical ..

        local_cpu_lock = 0;

and you use the "local_cpu_lock" to tell interrupts to not do certain
things, the compiler will be totally clueless.

[C++] Performance

Linux Performance

[C++][Linux][Memory] Heap Memory Profiler for Linux

• libmemusage.so
• gperftools
• Heaptrack - A Heap Memory Profiler for Linux

[Books] Linux Kernel

UNIX® Systems for Modern Architectures: Symmetric Multiprocessing and Caching for Kernel Programmers Read this book before delving into linux kernel

[linux] Tweak & Tips

[T410]
Bright key not working.
in 50-device.conf
Option "RegistryDwords" "EnableBrightnessControl=1"
----------