[AI] Tuning jargon

Supervised Fine-Tuning (SFT)

Parameter-Efficient Fine-Tuning (PEFT):
LoRA / QLoRA: The industry standard. Instead of changing the whole model, you add tiny "adapter" layers. This reduces the VRAM requirement by up to 90%.
DoRA (Weight-Decomposed Low-Rank Adaptation): A newer 2025/2026 favorite that decouples the magnitude and direction of weight updates, often yielding better results than LoRA.

SDFT (Self-Distillation Fine-Tuning): A breakthrough method (popularized by MIT in early 2026) where the model uses its own reasoning to generate better training data for itself, reducing the need for human-labeled sets.

SFT vs. RLHF (The "Teacher" vs. the "Critic")
It is helpful to think of SFT as the first step in a two-part education:

SFT (The Teacher): Tells the model, "Here is exactly how a good answer looks. Copy this."
RLHF/RFT (The Critic): Comes after SFT. It tells the model, "You gave me three answers; this one is better than that one." This is used for "alignment"—making the model safer, more polite, or better at complex reasoning (like DeepSeek-R1 or OpenAI’s o1).

Why Use SFT?

Domain Expertise: Teaching a model medical, legal, or proprietary company jargon.
Style/Voice: Ensuring the AI sounds like your specific brand (e.g., "Professional yet cheeky").
Format Constraints: Forcing the model to always output valid JSON or specific code structures.
Efficiency: A fine-tuned 7B model can often outperform a generic 70B model on a specific, narrow task.
Pro Tip: In 2026, the mantra is "Quality over Quantity." 1,000 extremely high-quality, human-verified examples will almost always result in a better model than 50,000 noisy, machine-generated ones.

[AI math] Proximal Policy Optimization (PPO)

Proximal Policy Optimization(PPO 2017)

It tries to solve destructive updates issue.

In Reinforcement Learning, if you take a step that is too large, the agent "falls off a cliff." It adopts a policy that causes it to fail immediately, which generates bad data, which leads to even worse updates. It never recovers.

The Deep Dive: The Mechanics of "Clipping"

The genius of PPO lies in its pessimistic view of the world. It doesn't trust its own recent success.

When the agent collects data, it calculates the Ratio between the new policy and the old policy for a specific action:

The "Pessimistic" Update

Standard algorithms simply push this ratio as high as possible for good actions. PPO says: "Stop."

• It looks at the Advantage (how good the action was) and applies a Clip:
  If the action was good (Positive Advantage): PPO increases the probability of doing it again, BUT it caps the Ratio at roughly 1.2 (increasing probability by 20%).
  Even if the math says "this action was amazing, boost it by 500%," PPO clips the update at 20%. It refuses to overcommit to a single piece of evidence.
• If the action was bad (Negative Advantage):
  PPO decreases the probability, BUT it caps the decrease at 0.8. It refuses to completely destroy the possibility of taking that action again based on one failure.

This results in a "Trust Region"—a safe zone around the current behavior where the agent is free to learn, but beyond which it is physically prevented from moving.

Usage

Reinforcement Learning from Human Feedback (RLHF).

The Setup

• The Agent (Policy): The LLM itself (e.g., Llama-2-70b). Its "Action" is choosing the next token (word) in a sentence.
• The Environment: The conversation window.
• The Reward Model: A separate, smaller AI that has been trained to mimic a human grader. It looks at a full sentence and outputs a score (e.g., 7/10 for helpfulness).

The PPO Training Loop

1. Rollout (Data Collection): The LLM generates a response to a prompt like "Explain gravity."
   LLM: "Gravity is a force..." It records the probability of every token it chose (e.g., it was 99% sure about "force").
   
2. Advantage Calculation: The Reward Model looks at the finished sentence and gives it a score (Reward). The PPO algorithm compares this score to what it expected to get.
1. Scenario: The model usually writes boring answers (Expected Reward: 5). This answer was witty and accurate (Actual Reward: 8).
2. Advantage: +3. This was a "better than expected" sequence of actions.
3. The PPO Update (The Critical Step): We now update the LLM's neural weights to make those specific tokens more likely next time.
1. Without PPO: The model might see that high reward and drastically boost the probability of those specific words, potentially overfitting and making the model speak in repetitive loops or gibberish just to chase that score.
2. With PPO: The algorithm checks the Ratio.
   "Did we already increase the probability of the word 'force' by 20% compared to the old model?"
   Yes? -> CLIP. Stop updating. Do not push the weights further.

PPO is not just an optimization method; it is a constraint method.
It allows AI to run training loops that would otherwise be unstable.

Policy

Policy is a function that maps a State (what the agent sees) to an Action (what the agent does).

In mathematical formulas, the policy is almost always represented by the Greek letter π.

a = π(s)

s: State(input)

π: Policy(logic)

a: Action(output)

The Two Types of Policies

• Deterministic Policy
  This policy has no randomness. For a specific situation, it will always output the exact same action.
  Example: A chess bot. If the board is in arrangement X, it always moves the Knight to E5
  a = π(s)
• Stochastic Policy (Used in PPO)
  This policy deals in probabilities. Instead of outputting a single action, it outputs a probability distribution over all possible actions. The agent then samples from this distribution.
  Example: A robot learning to walk. If it is tilting left, it might be 80% likely to step right and 20% likely to wave its arm. It rolls the dice to decide.
  π(a | s) : read as "the probability of taking action a given state s
  

Stochastic policies are essential for learning. If a policy is 100% deterministic, the agent never tries anything new; it just repeats the same mistakes. By adding randomness (probabilities), the agent explores different actions, which allows it to discover better strategies.

In modern AI (like PPO), the "Policy" is a Neural Network.

• Input: The network receives the State (e.g., the pixels of a video game screen, or the text of a user prompt).
• Hidden Layers: The network processes this information.
• Output: The network outputs numbers representing the probability of each action.
  Action 1 (Jump): 0.1
  Action 2 (Run): 0.8
  Action 3 (Duck): 0.1

When we say we are "training the policy," we are simply adjusting the weights of this neural network so that it assigns higher probabilities to "good" actions and lower probabilities to "bad" actions.

Value Function (V)

often called the Critic, is the partner to the Policy (the Actor).

While the Policy answers "What should I do?", the Value Function answers: "How good is it to be in this situation?"

1. 1. The Core Concept: Prediction
   Imagine you are playing a video game.
   The Policy (Actor): Looks at the screen and presses the "Jump" button.
   The Value Function (Critic): Looks at the screen and says, "We currently have a 70% chance of winning."
   The Critic doesn't play the game; it predicts the outcome.
2. The Math V(s)
   The Value function maps a State s to a single number (Scalar).
   
   
3. Why PPO Needs the Critic (The Advantage)

   This is the most important part. PPO updates the Policy based on the Advantage. The Advantage is calculated using the Value Function.
   The Logic: To know if an action was "good," we can't just look at the reward.
   Example: If you get a reward of +10, is that good?
   If you usually get +1, then +10 is amazing.
   If you usually get +100, then +10 is terrible.
   The Value Function provides that baseline "usual" expectation.
   

   

   
   
4. How the Critic Learns
   

   While the Actor learns to maximize reward, the Critic learns to be a better predictor.

   It uses a simple regression loss function (Mean Squared Error):
   

   
   

   At every step, the Critic looks at the reward actually received and updates itself: "I predicted this state was worth 5 points, but we actually got 8. I should update my weights to predict higher next time."
5. The PPO Team:
   Actor (Policy π): "I think we should move Left."
   Environment: (Agent moves Left, gets +5 reward, lands in new state).
   Critic (Value V): "I expected a reward of +2. You got +5. That was a great move!"
   PPO Update: "Since the Critic said that was 'great' (Positive Advantage), let's adjust the Actor's weights to make 'Move Left' more likely next time—but clip it so we don't go crazy."

[deep dive llm (gpt-2)]

Reference:
https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1
Let's reproduce GPT-2 (1.6B): one 8XH100 node, 24 hours, $672, in llm.c

Tool:
https://tiktokenizer.vercel.app/

Library:
https://github.com/google/sentencepiece

Flow

[AI Math] Embedding, RAG, and Vector index

https://gemini.google.com/share/587581ab91ce

[AI Math] Cosine similarity

Cosine similarity is a metric used to measure how similar two things are, mathematically represented as vectors.

1. The Core Concept: It's All About the Angle

Identical: If the arrows point in the exact same direction, the angle is 0. The cosine of 0 is 1. (Maximum similarity).
Unrelated: If the arrows are at a 90 angle (perpendicular), the cosine of 90 is 0. (No similarity).
Opposite: If they point in opposite directions 180, the cosine is -1.

2. The "Killer Feature": Magnitude Invariance

Why do we use this instead of just measuring distance (Euclidean distance)?

The problem with distance: Imagine you have two documents about "Apples."

Doc A: A short sentence: "I like apples."

Doc B: A massive 500-page book about apples.

If you count the word frequencies, Doc A might be vector [1, 0...] and Doc B might be [1000, 0...].

Euclidean Distance: These two are very far apart because Doc B has way more words.

Cosine Similarity: These two point in the exact same direction. They are both about apples.

Reference:
https://youtu.be/zcUGLp5vwaQ?si=IhjCEhAj6YoEsCtA

[ACCU 2005] Learning To Stop Writing C++ Code minute

Learning To Stop Writing C++ Code (and Why You Won’t Miss It) - Daisy Hollman - ACCU 2025
https://www.youtube.com/watch?v=mpGx-_uLPDM&t=870s

Best Practices for coding with LLMs

• Use smaller files
• Over-test everything.
• LLMs are pretty good at generating tests for existing code
• But they also pretty decent at helping you with Test-Driven Development
• Well-contained unit tests are much easier for LLMs to reason about
• Encapsulation is critical!

LLMs currently struggle with "long-term learning"
Whereas a human working on the same project for weeks or months can abstract away the details of a complicated workflow and "learn" which poorly encapsulated sharp edges are ignorable, LLMs currently struggle with this kind of thing. (early 2025)

In other words, code coupling is bad—don't connect dissimilar things from different units of encapsulation in unintuitive ways.

• Naming is more important than ever
• Intuitive abstraction design goes a long ways
• Agents often don't know to "check" for unintuitive behavior
  ...or they might "check" sometimes and not other times
• Writing abstractions that are easy to correctly "guess" how they work is important
• Write better (but still concise!) comments and documentation
  QUOTE
  The compiler does not read comments and neither do I — Bjarne Stroustrup
  Maybe it's time to revise this? LLMs do read comments

High cohesion is good

• This is the opposite of code coupling—similar things within a given unit of encapsulation should be grouped together.
• "Don't Repeat Yourself" (DRY) coding helps make efficient use of the LLM's context window
• Don't do unexpected things
• Especially if those things often don't have syntax (e.g., copy constructors in C++, auto-dereferencing in Rust, non-idiomatic __getattribute__ in Python, etc.)
• In C++, use regular types whenever possible. (read my note about regular type: https://vsdmars.blogspot.com/2018/06/c-regular-type.html , basically, design by contract, precondition)
• Don't mix owning and non-owning semantics in the same type or template
• Don't mix value and reference semantics in the same type or template

KEY TAKEAWAY

Writing code that LLMs will understand is not that different from writing code that humans will understand, except that we can start to understand and quantify why these best practices increase understandability.

Design by contract

• Both contracts and effects systems are ways of encapsulating information and reducing code coupling.
• Encapsulation is key to effectively working with LLMs because of the context window size constraints.
• But also, it's a lot easier to train LLMs on small, well-contained problems.
• Contracts promote Liskov Substitutability, allowing LLMs to infer behavior of a broader category of types.

An "Effects System" is a way for a programming language to track what a function does (its side effects), not just what it returns (its data type).

While C++ doesn't have a full academic effects system (like the research language Koka), it has a "pragmatic" one built into keywords you use every day.

Here is how const and noexcept act as an Effects System to help both the Compiler and LLMs.

The Concept: Labeling the "Black Box"

Without an effects system, a function signature void process(T& data) is a black box. It could do anything: write to disk, throw an error, modify global state, or format your hard drive.

An effects system puts warning labels on the box.

Why LLMs Love This (The "Context" Win)

The slide mentioned "Context Window Constraints." This is where effects systems shine for AI.

If you give an LLM this code:

C++

// Case A: No "Effects"

void transform(Data& d);

The LLM has to "hallucinate" or guess: Does this function throw? Do I need a try-catch block? Does it invalidate my iterators? It has to consider all possibilities, which wastes "reasoning tokens."

If you give it this:

C++

// Case B: Constrained Effects

void transform(Data& d) noexcept;

The search space collapses. The LLM immediately knows: No exception handling logic is needed here. It can focus its limited attention on the actual logic rather than defensive coding.

[Book] Programming massively parallel processors(PMPP) - Wen-mei Hwu, reading minute - ch1 - ch6

Reference:

CUDA C++ Programming Guide
NVIDIA Nsight Compute CUDA code optimization

[virtual memory] recap

Intel TBB
Intel TBB Task Scheduler
[oneTBB] accessor note

Error handling recap; TLPI
EINTR and What It Is Good For

kernal namespace recap
https://vsdmars.blogspot.com/2018/12/linuxnamespace-wrap-up.html
https://vsdmars.blogspot.com/2018/12/linuxnamespace-mount-mnt.html
https://vsdmars.blogspot.com/2018/06/linuxkernel-namespace.html

cache
[Pacific++ 2018][re-read] "Designing for Efficient Cache Usage" - Scott McMillan
[Go][design] high through-put low contention in memory cache

Locking
[futex] futex skim through
[Concurrency] [C++][Go] Wrap up - 2018
[C++] memory model in depth; memory spilling.

Algorithm/Implementation
Under my github/leetcode project

Software abstraction (CUDA runtime)

Similar idea e.g.  Golang  ref: [Go][note] Analysis of the Go runtime scheduler paper note, the difference coming from Go/C++ as multi-purpose language running on CPU (sequential), CUDA/OpenCL as C extension running on GPU. While the underneath hardware architecture difference, the abstraction diffs.

SPMD single-program multiple-data 

host CPU based code

kernel GPU Device code / function, more details later. Basically same code / IR run in parallel.

grid threads group

_h host variable in CPU code

_d device variable in CPU code

__host__  callable from host, executed on host, executed by host thread(e.g. linux thread).

__global__ callable from host or device, executed on device, executed by grid of device threads

__device__ callable from device, executed on device, executed by caller device thread.

If function declared with __host__ and __device__ macro, NVCC generates two version, one for host and one for device.

block 32-based size(hardware efficiency reason), all blocks are in same size. Size of how many GPU threads. Threads in a block can execute in any order with respect to each other.

SM Streaming multiprocessor; each SM has several processing units called CUDA cores. It is designed to execute all threads in a warp following the single-instruction multiple-data(SIMD) model.

HBM high-bandwidth memory

Warp a warp groups 32-threads together. Thus a block of threads will be group into warps, which each warp has 32-threads. Scheduling is based on Warp. Also think as single-instruction, multiple-threads.

FLOP floating-point operations

FLOP/B FLOP to byte ratio.

GPU global memory bandwidth: 1555GB/second; 1555 * 0.25(FLOP/B) = 389 GFLOPS

const readonly variables

 blockIdx; area code
 blockDim; row idx
 threadIdx; phoneline
Those three variable gives the kernel realize which data it is running on.

OUR_GLOBAL_FUNC<<<number of block, threads per block>>>(args...);

Thread Scheduling

Block scheduling

When a kernel is called, the CUDA runtime launches a grid of threads execute the kernel code. These threads are assigned to SMs on a block-by-block basis. All threads in a block are simultaneously assigned to the same SM. There are reserved blocks for system to executed, thus a SMs' blocks are not all scheduled to the user kernel.
Multiple blocks are likely to be simultaneously assigned to the same SM. The concept of Warp scheduling is that, those threads inside the same Warp runs the same instruction set(same kernel), thus the fetch of instruction is one time efforts. Also, the data those threads in the same Warp access are linear thus are prefetchable/cache friendly.

Moreover, threads in the same block can interact with each other in ways that threads across different blocks cannot, such as barrier synchronization,

synchronization / transparent scalability

__syncthreads()

  block until every thread in the same block reaches the code location. if a __syncthreads() statement is present, it must be executed by all threads in a block. i.e. 

void incorrect_barries_example(int n) {
	if (threadIdx.x % 2) {
		__syncthreads(); // sync point-1
	} else {
		__syncthreads(); // sync point-2
	}
} 

Wrong due to not all threads runs into the same barrier synchronization points.

Not only do all threads in a block have to be assigned to the same SM, but also they need to to be assigned to that SM simultaneously. i.e. a block can begin execution only when the runtime system has secured all the resources needed by all threads in the block to complete execution.

The ability to execute the same application code on different hardware with different amounts of execution resources is referred to as transparent scalability.

Control Divergence

The execution works well when either all threads in a warp execute the if-path or all execute the else-path. Otherwise, it has to go with the code twice. One run with the core running if path code and the other core with else path is doing noop. (In the same Warp). Another run with the core doing noop on the if path code and the other core wile else path is running. In old architecture, those 2 runs run in sequence. In new architecture, those 2 runs can run in parallel. This is called independent thread scheduling.

Thus, due to this fact, do not use threadIdx for if branching. But use data for divergence control, this also related to data locality in cache. One important fact, the performance impact of control divergence decreases as the size of the vectors being processed increases.

One cannot assume that all threads in a warp have the same execution timing.(even they are running the same fetched instruction). Thus, use __syncwarp() barrier synchronization instead.

Latency tolerance

simple, i.e. CPU, context switch on single code due to limited of resource(registers, cache etc.) and makes sure code runs preemptive-scheduling fashion.

Thus, SM only has enough execution units to execute a subset of all the threads assigned to it at any point in time.

In recent SM, each SM can execute instructions for a small number of warps at any given point in time.

GPU SMs achieves zero-overhead scheduling by holding all the execution states for the assigned warps in the hardware registers so there is no need to save and restore states when switching from one warp to another.

Thus, allows GPU oversubscription of threads to SMs.

Automatic/Local variables declared in the kernel are placed into registers.

Each SM in A100 GPU has 65,536 registers.

65536/2048(threads) = 32 registers per thread/kernel.

In cases, the compiler may perform register spilling to reduce the register requirement per thread and thus elevate the level of occupancy. However, this could increase latency due to need to fetch data from the memory instead directly from the register.

Nsight Compute Occupancy Calculator

cudaDeviceProp struct has bunch of variable represents the hardware SPEC.

e.g.

 multiProcessorCount Number of multiprocessors on device

 clockRate Clock frequency in kilohertz

 regsPerBlock 32-bit registers available per block

 warpSize  Warp size in threads

Variable declaration scope and lifetime

automatic scalar variables    [mem]register    [scope]thread    [lifetime]grid

automatic array variables    [mem]local    [scope]thread    [lifetime]grid

__device__ __shared__    [mem]shared    [scope]block    [lifetime]grid

__device__    [mem]global    [scope]grid    [lifetime]application

__device__ __constant_    [mem]constant    [scope]grid    [lifetime]application

API

cudaMalloc

cudaMemcpy

[LLM] basic

 

The algorithm underlying BPE breaks down words that aren’t in its predefined

vocabulary into smaller subword units or even individual characters, enabling it to

handle out-of-vocabulary words. So, thanks to the BPE algorithm, if the tokenizer

encounters an unfamiliar word during tokenization, it can represent it as a sequence

of subword tokens or characters.