[godbo.lt architecture]

 godbolt.org

[system design] Scalable and Reliable Logging at Pinterest (DataEngConf SF16)

Reference:
https://youtu.be/DphnpWVYeG8?t=2250

Although this talk was back in 2016, there are several points to take away:

1. the way project 'merced' operates reminds me of paper 'Sparrow'

https://cs.stanford.edu/~matei/papers/2013/sosp_sparrow.pdf

And their use of zk as message delivery is fun.

2. back then Kafka doesn't support exactly once delivery and transaction commit;

thus they've spent quite of efforts for that.

3. back then K8S an't popular/feature rich enough which addresses HA issues~

[monitoring] Book: Monitoring Distributed Systems

Book:
Monitoring Distributed Systems

Monitoring

Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes.

• White-box monitoring (k8s readiness probe)
  Monitoring based on metrics exposed by the internals of the system, including logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP handler that emits internal statistics.
• Black-box monitoring (k8s liveness probe)
  Testing externally visible behavior as a user would see it.
• Dashboard
  An application (usually web-based) that provides a summary view of a service’s core metrics. A dashboard may have filters, selectors, and so on, but is prebuilt to expose the metrics most important to its users. The dashboard might also display team information such as ticket queue length, a list of high-priority bugs, the current on-call engineer for a given area of responsi‐ bility, or recent pushes.
• Alert
  A notification intended to be read by a human and that is pushed to a system such as a bug or ticket queue, an email alias, or a pager. Respectively, these alerts are classified as tickets, email alerts, and pages.
• Root cause
  A defect in a software or human system that, if repaired, instills confidence that this event won’t happen again in the same way. A given incident might have multiple root causes: for example, perhaps it was caused by a combination of insufficient process automation, software that crashed on bogus input, and insuffi‐ cient testing of the script used to generate the configuration. Each of these factors might stand alone as a root cause, and each should be repaired.
• Node (or machine)
  Used interchangeably to indicate a single instance of a running kernel in either a physical server, virtual machine, or container. There might be multiple services worth monitoring on a single machine. The services may either be:
• Related to each other: for example, a caching server and a web server
• Unrelated services sharing hardware: for example, a code repository and a master for a configuration system like Puppet or Chef
• Push
  Any change to a service’s running software or its configuration.

Why Monitor?

• Analyzing long-term trends
• Comparing over time or experiment groups
• Alerting
• Building dashboards (“The Four Golden Signals”: latency, traffic, errors, and saturation.)
• Conducting ad hoc retrospective analysis (i.e., debugging)

Monitoring and alerting enables a system to tell us when it’s broken, or perhaps to tell us what’s about to break. When the system isn’t able to automatically fix itself, we want a human to investigate the alert, determine if there’s a real problem at hand, mitigate the problem, and determine the root cause of the problem.

In general, Google has trended toward simpler and faster monitor‐ ing systems, with better tools for post hoc analysis. We avoid “magic” systems that try to learn thresholds or automatically detect causality.

To keep noise low and signal high, the elements of your monitoring system that direct to a pager need to be very simple and robust.

Rules that generate alerts for humans should be simple to understand and represent a clear failure.

Your monitoring system should address two questions: 

• what’s broken,
• and why?

“What” versus “why” is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise.

In Google’s experience, basic collection and aggregation of metrics, paired with alerting and dashboards, has worked well as a relatively standalone system.

When creating rules for monitoring and alerting, asking the following questions can help you avoid false positives and pager burnout:

• Does this rule detect an otherwise undetected condition that is urgent, actionable, and actively or imminently user-visible?
• Will I ever be able to ignore this alert, knowing it’s benign? When and why will I be able to ignore this alert, and how can I avoid this scenario?
• Does this alert definitely indicate that users are being negatively affected? Are there detectable cases in which users aren’t being negatively impacted, such as drained traffic or test deployments, that should be filtered out?
• Can I take action in response to this alert? Is that action urgent, or could it wait until morning? Could the action be safely automated? Will that action be a long-term fix, or just a short-term workaround?
• Are other people getting paged for this issue, therefore rendering at least one of the pages unnecessary?

These questions reflect a fundamental philosophy on pages and pagers:

• Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued.
• Every page should be actionable.
• Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page.
• Pages should be about a novel problem or an event that hasn’t been seen before.

[scale] maxim

QCON 2017 Scaling Instagram Infrastructure 

[Actor model] Some real-world implementation details from AKKA

Found this talk on YT recently, my comments + notes below:

Introduction to the Actor Model for Concurrent Computation - John Murray

Actor is:

• persistent
• has internal state
• asyc
• Independent event-loop + memory
• Mailbox (receive message)
• React in FIFO order

Well,
actor model is concurrent with regard to a system of actions.
actor model is Not concurrent with regard to data.
(surely we can do implement concurrent thread inside one actor.)

Using channel as mutex to single variable

So, how to make a mutable variable access by multiple threads?

In Golang, we have channel.

• Create a channel with the length of 1.
• Put the mutable variable in, and mutiple goroutines have access to that channel.
• However, only one goroutine can successful retrieve the mutable varible inside the length of 1 channel, other goroutines since the channel has no value, it will block.
• This acts as a MUTEX for goroutines.
• Once the goroutine which having the mutable variable changed,
  it puts it back to the channel thus other goroutines can access.

Actor can:

• create more Actors
• receive messages and response
• make local decisions
• perform arbitrary, side-effecting action
• send messages
• respond to the sender 0 or more times
  (to clearfy here, by "respond" means
  send message to the sender Actor
  Not duplex channel respond)
• Process exactly one message at a time

Actor communication is:

• No channels or internediaries (such as in CSP, e.g. golang)
• "best effor" delivery
• at-most-once delivery
• Messages can take arbitrary long to be delivered (has no concept of time)
• No message ordering guarantees

Actor address:

• identify the actor
• may also represent a proxy / forwarder to an Actor
• contains location and transport information
• don't care where the Actor lives
  (can be inside the same process, different nodes, different containers, etc)
• one address may represent many Actors(pool)

Error handling:

• Supervision
• the running state of an Actor is monitored and managed by another Actor(the Supervisor)
• Supervision has:
• constantly monitors running state of actor
• can perform actions based on the state of the Actor( e.g unhandled error, restart Actor)
  (In Golang, we take advantage of context.Context)
• transparent life-cycle management
• addresses do not change during restarts
  (we implemented with Actor's hash(name + uuid + hostname) as 'address')
  BE WARE.
  This only has meaning iff Actor is not pure, which is STATEFUL.
• Persist state into local sqlite, loaded with (name + uuid)
• mailboxes are persisted outside the Actor instances (Auh, K.I.S.S)
  I doubt the use of Supervisor idea.
  Think about this, does the Sender care how the Receiver act when it receives the message? ;-P

 

Implement Address contains these as a group:

• mailbox
• Actor

Anti use-case:

• Working on a non-concurrent system
• performance critical applications
• non-concurrent communication is involved
• no mutable state

Draw backs:

• too many Actors
• testing
• debugging

Extra material:
Don't use Actors for concurrency


CRDT:
https://medium.com/@istanbul_techie/a-look-at-conflict-free-replicated-data-types-crdt-221a5f629e7e

[Note] design principle

While researching into LMAX found Martin Thompson's video,
although it's cliche
(in the way that senior engineers should have known),
still worth jotting down a note here.

Design Principle

1. Clear separation of concerns
2. Garbage free in steady state running
3. Lock-free, wait-free, and copy-free in data structures in the message path
4. Repsect the Single Writer Principle
5. Major data structures are not shared
6. Don't burden the main path with exceptional cases
7. Non-blocking in the message path

Succint into 3 things:

1. System Architecture
2. Data structures
3. Protocols of interactions

Data structures:

• Map
• IPC Ring Buffers
• IPC Broadcast Buffers
• ITC Queues
• Dynamic Arrays
• Log Buffers

[distributed system] opentracing

Paper:
Dapper, a Large-Scale Distributed Systems Tracing Infrastructure: https://ai.google/research/pubs/pub36356

Reference:
The OpenTracing Semantic Specification:
https://opentracing.io/specification/
Towards Turnkey Distributed Tracing
https://www.jaegertracing.io/

Dapper uses annotation to tag records with global identifier.

• span
  Tree node
  edge indicates relationship between current span and it's parent span.
  Contains timestamp: start time, end time.
  Contains span name: human readable.
  Contains span id Contains span parent id
  Root span: has no parent span.
  Share same trace ID iff associated with a specific trace.
  Beware time skew since span flows from client to server on different host.
• tree
  span forms the tree
• annotation

Implement:

• when use with thread, attaches a trace context to thread-local storage.(slow in dynamic library)
• when use with async calls, put the trace context into the async function call's argument.
• embedded trace context into each RPC/IPC call.
• Use customize annotation to trace data. Although, tracing is not logging, there's upper bound of calls to the trace API.
• Annotation is not an indication to the behavior of the tracing.
• Supports key/value(global) data structure to assist tracing calls.
• Use sampling to restrict number of calls to the tracing core.
• Provides out-of-bands trace information due to there are times when the callee is returing the result but the callee's callees haven't returning the data yet!

Tracer can be used as security check as well, which checks the code is actually hitting the code that is supposed to be hit.

• sampling statistic skew
• aggressive sampling
  Does not hinder high-throughput services.
• adaptive sampling
• trace generation overhead
• trace collection overhead
• effect on production workloads 
• addressing long tail latency 
• beware of coalescing effects (read the paper)

[distributed system][design] Real-Time Delivery Architecture at Twitter

• nothing is stable, embrace and deal with it.
• keep it simple.
• LRU for Redis.
• CQRS Always consider separately from read and write design.
• Event-based 'router'
• Fan-out :-) It's been used back in the day Jesus was born :-P ext4 files system/inode, fanout with another indirection.
  Prof. David Wheeler : "All problems in computer science can be solved by another level of indirection" (the "fundamental theorem of software engineering")
• consider about edge cases.
• Technology makes lives, better, keep on!

[LMAX][ACTOR MODEL][Note] A bit about LMAX

Reference:
[AKKA] How the Actor Model Meets the Needs of Modern, Distributed Systems
[AKKA] What problems does the actor model solve?



[Video] The Actor Model (everything you wanted to know...)

• Processing
• Storage
• Communication

One actor is no actor

Actor comes in system

Actor has address

Everything is an actor

When actor receives messages, it can

• Create more actors
• Send messages to other actors that it has the addresses to send to
• Designate what to do with the next message it receives

Future

They are actors can be send around(like a lambda expression/closure)

Address

Address : Actor is many-to-many arch.

Message

Messages are received NO guarantee in order.

It's like the IP(not TCP) model.
IP sequence without considering order(upper layer will do the check).

Sender persists message.

There's NO channels.

Same message can be send at MOST 1 time. No duplicate.

Message sent is not time-bounded. Which can take a long time to arrive to another actor.

Message being handled one at a time, which gives us synchronization.

non-deterministic vs. in-deterministic

Nondeterministic (turing) vs Indeterministic (actors)

E.g: A message that increments a number and then sends the same message again.
Another message, to stop and report the number is on its way. How many increments can happen before the stop message arrives.

arbiter

• Not something you can make out of just (and gates/or gates) and other boolean components.
• Equal number of outputs to inputs. 
• The output order is not determined by the order of input. 
• Unbounded by processing time but the probability of output goes down exponentially as time goes on. 
• Mutated state isn't for the current message.
  Mutated state is for the next message.

LMAX

Paper:

Disruptor: High performance alternative to bounded queues for exchanging data between concurrent threads
(neat paper touch the basic idea of MESI / false sharing / prefetch etc.)


The LMAX Architecture

• Event sourcing (log, RAFT)
• snapshot (RAFT, taking snap shot with commited log)
• unlike RAFT, Business Logic Processor keep multiple active node, each input event is processed by multiple active node.

Concurrent execution of code is about two things:

• mutual exclusion and
• visibility of change.

CAS approach is significantly more efficient than locks because it does not require a context switch to the kernel for arbitration.

However CAS operations are not free of cost.
(Details can be read from book: The art of multiprocessor programming)

The processor must lock its instruction pipeline to ensure atomicity and employ a memory barrier to make the changes visible to other threads.

The ideal algorithm would be one with only a single thread owning all writes to a single resource with other threads reading the results.

To read the results in a multi-processor environment requires memory barriers to make the changes
visible to threads running on other processors.


Solution:
Ensuring that any data should be owned by only one thread for write access, therefore eliminating write contention.


When designing a financial exchange in a language that uses garbage collection, too much memory allocation can be problematic. (golang, and the benefit of using C++)

The LMAX Architecture
https://martinfowler.com/articles/lmax.html

The best programmers prefer profilers and test cases to speculation.

reference:
https://en.wikipedia.org/wiki/Little%27s_law
Ring buffer basics
Circular Buffers in C/C++

[accu2018][design] Read/Write thinking would be harmful for system design

Read and write considered harmful - Hubert Matthews [ACCU 2018]

Could be the best presentation in this year's ACCU talk.

Also reference:
Mike Acton's Data-oriented Design cppcon 2014 presentation

Notes below:

Basics of Read/Write

• Business processes, rules and schemas
• Performance, scaling, concurrency
• Six questions about data
• Asynchrony and queues
• Structure changing
• State management

Reads

• Can be cached, at multiple levels (e.g CPU)
• Caching is transparent(mostly)
• Idempotent (can be retried without side-effects)
• Can be partitioned easily (routing, i.e load balancer, API gateway etc.)
• Access control rules only
• Synchronous and blocking
• Scalable bandwidth - fanout reduces contention

Writes

• Caching is horrible for writes
• writeback/through, eviction policy, coherence, etc.
• Scaling writes is horrible - fan-in creates contention
• Sharding works well only for primary key writes
• Access control rules plus update rules
• Can be delayed or asynchronous (write back cache, i.e store buffer)
• Idempotent is a design issue/choice

Dependencies

• Even simple code has dependencies caused by read and write
• Asymmetric - caller object doesn't know who calls it
• Makes testing difficult
• Substitution
• Mocks, etc
• Introduces notion of push and pull

REST APIs and rules

• REST = getters/setters on steroids
• Industrial-scale anti-pattern
• Separates code and data
• Opposite of encapsulation
• Very non-OO
• Duplicated logic/rules in every client
• e.g stock_level >= 0
• If rule is broken

Tradeoffs: early/late validation failure, schema migration, versioning, code/rule duplication

REST = CRUD

• statechart has one state and four transitions
• OK for metadata
• Real processes have multiple states
• Have different entities per state
• (Possiblly sub-entities)

Typical system scaling path (important!)

Think it in the WRONG WAY.

• Application is too slow
• Get more front-end boxes(scale R + W)
• Application is still too slow
• Get bigger DB box(scale R + W)
• Run out of read bandwidth
• Replicate or cache data(Scale R)
• Run out of write bandwidth
• Shard/Partition data on primary key(scale R + W)
• Create separate services per entity/component
• Cross-service joins done in client(scale entities)

Scaling problems

• Partitioning or sharding works to an extent
• If access is strongly biased around primary key
• 例如: access flight dates(平均平坦) vs. access every days meal(sequential even if hashed keys)
• Nasty to scale cross-partition operations
• Particularly for write (usually not idempotent)
• Partial failure on write, cross-box transactions, concurrency, latency, etc.
• Service boundaries aligned with operations boundaries and failure boundaries
• Bulk access to multiple records can cause N+1 access problems
  (get primary keys then N single-row accesses:
  beware of REST and ORMs) 可用batch access解決.

Avoid sharing mutable data

• Shared mutable data is the evil of all computing
• Read-only data can be shared safely without locks
• Const is our friend!
• Pure message-passing approach avoids this

Shared writes don't scale

• In memory programming
• 0 scalability!

Why shared writes don't scale?

MESI / Cores

• Caches have to communicate to ensure coherent view
• MESI protocol passes messages between caches
• Shared writes limited by MESI communication
• JUST DON'T DO SHARE WRITE!

6 questions about data access

Primary key: hashing, partitioning, op==

• Most common from of access
• database primary key
• std::map/unordered_map
• Can use hashing [O(1)], binary search [O(logN)] or linear search for small N [O(N)]
• Requires only operator ==
• Partition on PK into multiple parts that can operate in parallel or to avoid contention
  e.g Product catalogue, customer records, sticky web sessions, NoSQL, memcache, REST

Non-Primary key: search, secondary indexes, full-text search

• Finding items by value, not by key
• Need for secondary indexes(e.g database indexes)
• Search on parts of a record
• Metadata search (data/time, etc)
• Full-text search
• May require substantially more work to build compared to PK-based access
• Usually slower than PK access for lookup

Range scans: ordering, iteration, bulk vs. single values, op<

• Requires ordering, i.e operator<, (ordering costs)
• Requires iterators/cursors/traversal state
• Seek then scan - first find is slow, then fast
• Dense linear access and prefetch
• Watch out for read/write amplification
• Bulk, not single record, access - may require bulk aggregate operations rather than N times single record operations for speed (DB N+1 problem)

R/W ratio: caching, cost of lookups vs. cost of updates

• Not all data in a system has similar R/W ratios
• e.g metadata is often read-heavy
• High reads
• caches are effective
• cache writethrough/back
• cache eviction policy
• cache coherency
• index structures useful
• High writes
• caches don't help much (except for metadata)
• locking overhead
• index structure require updating

Working set: how big is the commonly accessed set of data(RAM)

• How much of the common data will fit in main memory, the L1/L2/L3 cache
• Will the index structures fit but not the main data
• Index data tends to be 'HOT', main data may be 'COLD'
• Depends on data access patterns - 80/20 rule

Consistency: exact results vs. fast approximations, eventual consistency, replication, batched updates

• Do all copies of the data need to be exactly up-to-date right now
• ACID, 2-phase commit, centralised, locking, slow
• How often are copies updated
• Batch updates (e.g overnight)
• Data vs. metadata (transactions vs. reference data)
• ACID vs. BASE(eventual consistency)
• BASE allows for decoupled asynchronous systems

Read or write, pull or push?

• How to read a diagram?
• Is X reading Y or writing to it?
• Or both at different times?
• Is X pushing or Y pulling?
• Where is the THREAD of control?
• Is this a full batch update or a partial incremental change?
• Is this an asynchronous push (fire and forget) or a synchronous blocking call?
• When? Who? What?
• Horizontal dataflow thinking!

Full vs. incremental change

• Full changes allow the state to be reset on a regular basis
• Prevents build-up of errors or divergence from base data
• Slow, long latency, partial failure problems
• One big transaction
• Incremental changes are fast but don't guarantee to keep state changes synchronised
• lost messages because of unavailability
• transactionality only on each update

Lambda architecture

Reader/writer vs. data flow

• Readers and writers view hides inherent data flow in systems
• Split R and W into micro-services
  i.e Command Query Responsibility Segregation(CQRS)
• separates rules, performance,  scaling, access control, etc.
• Often W and R are very different
• W usually reads from somewhere

Sync vs. async systems

• ACID is hard to scale, partition, get right, can promote failures, makes for a more fragile system as everything has to be up all the time(brittle)
• 10 sync systems with 99% uptime => 90% uptime
• 10 async systems => 99% uptime for end system
• Can maybe delay writes or cover them up
  (lambda arch, i.e layer by layer)
• Reads are sync but recent data may be sufficient, particularly for metadata(caching helps lots!)

Data flow and sync/async

• Data flows can be point-to-point or broadcast
• Can be synchronous or asynchronous
• message queues provide simple sync intermediary
• flat file batch transfer is popular for a reason
• Queues can also be event stores with reread!!(RAFT)
• Makes queue reads idempotent

Content management example

• Keep two APIs separate
• security, clarity of purpose
• separation of concerns
• horizontal not vertical thinking!!

Larger example

• Data flow makes us think about where data comes from and goes to
• Who is actually going to read the data I'm writing?
• Micro-services may have two APIs for sending and receiving
• 'Vertical' thinking may lead to trying to fit both into one API

When? Who? What?

Horizontal dataflow thinking!

Command Query Representation Separation (CQRS)

• Need to change the data structure from the form that best suits the write API to the structure that best suits the read API
• Can be synchronous or asynchronous transformation
• Think as READ and WRITE separately

CQRS examples

• Structure change can be on read or on write
• Twitter does it on read for high-value users, not write
• Log-structured merge systems do this internally and asynchronously (e.g HBase, RocksDB)
• Other examples:
• time-series db
• event sourcing
• struct-of-arrays vs. array-of-structs(e.g non-OO, data oriented)
• Columnar analytics db(e.g BigQuery)

State management

• Read and write focus doesn't help manage state across a complex system
• State management needs to address:
• transactions vs. eventual consistency
• failure management
• availability and MTTR (Mean Time To Repair)
• MTBF(Mean Time Between Failure)
• immutability
• Align boundaries with failure and aggregate boundaries
• REST API: /resourceA/1/resourceB/2
• Fragmentation and transactionality problems

Failure management

• Distributed systems can suffer from partial failures on writes
• Writes in distributed are inherently concurrent
• Recovery and re-sync state is 'fun'
• Idempotent writes allow for replay and deduping
• Make deduping easy: serial number, timestamp, etc.
• Repeatable queues are useful (e.g Kafka, flat files)
• Check-pointing of known good state
• Point-in-time recovery
• Full vs. incremental update problem again

Bell-LaPadula and Biba models

• Bell-LaPadula: confidentiality
• Biba: integrity

You CANNOT have BOTH.

Immutability (makes things simpler)

• Functional programming languages have immutable data
• Make sharing and reasoning about data easier
• Russian Doll caching, MVCC
• Change the key not the value
• Pets vs. cattle - infrastructure
• SSA - compilers and CPI reservation stations
• Lambda architecture - immutable master data

Availability

• Availability = MTBF / (MTBF + MTTR)
  (where MTBF = mean time between failures
  MTTR = mean time to repair)
• Maximise software practices, hardware failover, reliable well-known technology choices
• Minimise MTTR by making systems easier to understand, debug and restart
• minimise state management(redo logs, fsck, etc)
• use immutability where possible

Read and write are too low level

• They don't help us to design or analyse systems
• they are the assembler-level of data (CRUD)
• They don't relate to the larger picture
• It is too easy to deal with them in isolation
• REST APIs are an all-too common example
• Looking at data flow, push vs. pull, sync/async, business processes, operational profiles, state management, etc. are much more fruitful approaches

[design][microservice] Microservices at Netflix scale

Watch note for Microservices at Netflix scale

Microservices

What's the cost?

Make assumptions

Then we could prioritize which is important.

Services should be stateless

• Not rely on sticky session
• Chaos testing
• Verify stateless (kill the instance randomly)

Scale out vs. scale up

• NoSQL at scale (Cassandra, Scylla)
• CAP available
  (Local quorum vs. async copy to remote region)

Redundancy and Isolation for resiliency

• Avoid single point of failure
• Make more than one of _anything_
• Isolate the blast radius for any given failure

Destructive testing

    Check check check

Billing services

Arch:

Request Cache!

Consider maintenance efforts

Writing tools for automation, less human involve would be better!

Reliability

Cascading failures

Tools to check microservices failure

Failure has grades. Some low grade failure is OK with backup services.
Tools: Hystrix, or Envoy

Containers

[design][microservice] Known Before Scaling

Watch note for What I Wish I Had Known Before Scaling Uber to 1000 Services • Matt Ranney

Microservices

• immutable?
• append only?

Move and release independently
Own your uptime
Use the best tool for the job

What are the costs?
Everything is RPC
What if it breaks?

Everything is a trade off.

[design][microservice] The hardest part of microservices is your data

Watch note for The hardest part of microservices is your data


slides:
https://www.slideshare.net/ceposta/the-hardest-part-of-microservices-your-data

• Microservices is about optimizing for SPEED.
• Manage (aka. reduce) dependencies.
• Data is a major dependency.
• Focus on domain models, not data models
• Stick with these conveniences as long as you can.
• A microservice has its own database
• We need to understand something about the data inside our services and the data outside our services
• Thus, we are building a distributed system, thus, beware of 
• network latency, 
• network partition
• Plan for failures.
• Build concepts of time, delay, network, and failures into the design as a first-class citizen
• How do you “read” data and how do you “update” data.
• Separate reads and writes
  (CQRS, Command Query Responsibility Segregation
  refer to: http://vsdmars.blogspot.com/2018/05/design-even-driven-system-pattern.html
• Performance: N+1 Query Problem
• https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/
• Use Batching Queries to fix it!
• Load all your data before iterating through it.
• Detecting the Problem
• If seeing a bunch of similar queries, this often indicates an N+1 query issue(or a similar kind of query batching problem)
• For our reads and writes, we need some "consistency"
• We need reads and writes. But we expect failures. 
• CAP tells us to pick 2: Consistency, Availability, Partition Tolerance
• CAP is a bad way to think about this.
• Consistency model
  https://en.wikipedia.org/wiki/Consistency_model
• What consistency model do you need, depending on what role you’re playing?
• https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf
• Tool: debezium.io

[design] Serverless Architecture (FaaS)

Watching note for Serverless: the Future of Software Architecture • Peter Sbarski

cross reference:
Designing distributed system

Function as Service (execute custom code)
Backend as a Service (3rd-party services)

Adopt

• Pipelines as code
• APIs as a product
• Decoupling secret management from source code
• Hosting PII data in the EU
• Legacy in a box
• Lightweight architecture decision records
• Progressive web applications
• Prototyping with invision and sketch
• Serverless architecture

e.g

AWS Lambda for compute

S3 for storage

Principles of Serverless Architecture

• Use a code-execution compute service to run code on demand
• Write single-purpose statelesss functions
• Design push-based, event-driver pipelines
• Create thicker, more powerful front ends
• Embrace third party services

Use a code-execution compute service to run code on demand

Do not run on a server.
Focus on function, not application update/upgrade
(e.g patch Apache web server etc.)

Compute as backend:
API-Gateway is needed for REST API

Compute as glue:
Act as Pipeline

Write single-purpose statelesss functions

Pure function

Only 0 or 1 transformation

Design push-based, event-driver pipelines

One event triggers another event.
e.g

e.g.

Use websocket, push based.

Create thicker, more powerful front ends

traditional way:

Changed way:

Embrace third party services

Benefits

• time to market
• scale effortlessly (deal with spikes traffic)
• disruptive cost model
• no more server to manage
• versatile
• lower cost
• less code
• easy to scale and flexible

Cons

• Not for everyone
• Service level and customization
• Vendor lock-in
• Decentralization

Architecture

Go for microservices

Use API-Gateway

https://www.algolia.com/ as search service

[design] Event-driven system pattern

Watch note for The Many Meanings of Event-Driven Architecture • Martin Fowler

4 event driven patterns

Event notification

    啊就是類似 GUI  一個thread在while loop 聽event...

• Decouple receiver from sender

Event-carried state transfer

    啊就是將event存成state 傳遞...

• Decoupling
• Reduced load on supplier 

Event sourcing

    啊就是 log 咩 RAFT it is...

• Create event object and persist the object
• Process the persisted object
• No state, can restore the state machine from log    (like git(or other version application) system is using event sourcing)

can be used in:

• audit
• debugging
• historic state
• alternative state
• memory image        

Command Query Responsibility Segregation(CQRS)

    啊就是 rwlock啊...

• Seperate read/write into two application.
• Write: Command model
• Read: Query model

Reference:

https://martinfowler.com/

[design] 4 patterns for distributed sysmtem architecture

4 patterns for distributed sysmtem architecture

watching note for: Four Distributed Systems Architectural Patterns by Tim Berglund

Modern three-tier

Strengths

• Rich front-end framework
• Hip, scalable middle tier
• Basically infinitely scalable data tier

Weaknesses

• State in the middle tier

Sharded

Strengths

• Client isolation is easy (data and deployment)
• Known, simple technologies

Weaknesses

• Complexity
• No comprehensive view of data
• Oversized shards

DB:

read replicate

• Partition problem.
• Master is dead, reelect master.

Lambda

Streaming vs. Batch
Unbounded(immutible data) vs. Bounded

e.g
event system

Strengths

• Optimizes subsystems based on operational requirements
• Good at unbounded data

Weaknesses

• Complex to operate and maintain
• write same code twice
• For analysis for best!

Streaming

• Integration is a first-class concern
• Life is dynamic; databases are static
• Tables are streams and streams are tables
• Keep your services close, your computation closer

Integration

Bad:

Kafka:

Event stream:
transfer database/table to stream...
i.e into a log!

Storing data in messages
First-class message system

• events get consumed though message stream system
• Then, each request/response from the sub-system which hooked on to the stream system.
  i.e stream system act like the 'HUB'
• Keep stream computation near our code.

Winner is: Streaming :-D

[design] thinking process

A strong process is crucial to successfully solving system design questions.

4 steps:

• Scope the problem: Don't make assumptions; Ask questions; Understand the constraints and use cases.
• Sketch up an abstract design that illustrates the basic components of the system and the relationships between them.
• Think about the bottlenecks these components face when the system scales.
• Address these bottlenecks by using the fundamentals principles of scalable system design.

About design:

• Everything is a tradeoff
• there is no one optimal system design.
• staying up to date

At the interview

First of all, follow the System Design Process. You already know how to apply it, so we'll be brief. Don't skip steps, don't make assumptions, start broad and go deep when asked.

Second, keep in mind that system design questions serve as an idea exchange platform. Be prepared for discussions about tradeoffs, about pros and cons. Be prepared to give alternatives, to ask questions, to identify and solve bottlenecks, to go broad or deep depending on your interviewer's preferences.

Don't get defensive: whenever your interviewer challenges your architectural choices, acknowledge that rarely an idea is perfect, and outline the advantages and disadvantages of your choice. Be open to new constraints to pop up during the discussion and to adjust your architecture on the fly.

1. Use cases
2. Constraints (Math)
3. Abstract design

• Outline all the important components that your architecture will need.
• Sketch your main components and the connections between them. 
• If you do this, very quickly you will be able to get feedback if you are moving in the right direction.
• Of course, you must be able to justify the high-level design that you just drew.
• Make sure you sketch the important components and the connections between them.
• Justify your ideas in front of the interviewer and try to address every constraint and use case.

4. Understanding bottlenecks

• Needs to be scalable, in order for you to be able to improve it using some standard tools and techniques.
• It may be the case that the interviewer wants to direct the discussion in one particular direction.
• Then, maybe you won't need to address all the bottlenecks but rather talk in more depth about one particular area.
• In any case, you need to be able to identify the weak spots in a system and be able to resolve them.
• Remember, usually each solution is a trade-off of some kind. Changing something will worsen something else.
• However, the important thing is to be able to talk about these trade-offs, and to measure their impact on the system given the constraints and use cases defined.

5. Scaling your abstract design


Scale:

• Vertical scaling
• Horizontal scaling
• Caching / Sticky Session
• Load balancing
• Database replication
• Database partitioning

Divide and Conquer - The Scalability Technique

• This is the scalability technique. Everything is about partitioning out work. Deciding how to execute it. Applies to many things, from web tier, you have a lot of web servers that are more or less identically and independently and you grow them horizontally. That’s divide and conquer.
• This is the crux of database sharding. How do you partitions things out and communicate between the parts that you’ve subdivided. These are things you want to figure out early on because they influence how you grow.
• Simple and loose connections are really valuable.
• The dynamic nature of Python is a win here. No matter how bad your API is you can stub or modify or decorate your way out of a lot of problems.

Avoid Thundering herd problem

Same as design in RAFT, use random!

[distributed system build up][book reading notes] Designing distributed system

Sidekick pattern

Ambassadors

Adapter

Replicated Load-Balanced Services

• Stateless Services

• Session Tracked Services
• Session tracking is accomplished via a consistent hashing function

• Application-Layer Replicated Services
• Introducing a Caching Layer
• A cache exists between your stateless application and the end-user request. 

• Deploy using the sidecar pattern

Using https://varnish-cache.org/ for HTTP cacheing

Sharded Services

• Sharded Caching

Many sharding functions use consistent hashing functions. 

Consistent hashing functions are special hash functions that are guaranteed to only remap # keys / # shards, when being resized to # shards.

For example, if we use a consistent hashing function for our sharded cache, moving from 10 to 11 shards will only result in remapping < 10% (K / 11) keys.

This is dramatically better than losing the entire sharded service.

The performance of your cache is defined in terms of its hit rate. 

The hit rate is the percentage of the time that your cache contains the data for a user request.

Ultimately, the hit rate determines the overall capacity of your distributed system and affects the overall capacity and performance of your system.

Sharding Functions
Shard = ShardingFunction(Req)
or, for programming languages,
Shard = hash(Req) % 10

Commonly, the sharding function is defined using a hashing function and the modulo(%) operator.

Hashing functions are functions that transform an arbitrary object into an integer hash.
The hash function has two important characteristics for our sharding:

• Determinism
• The output should always be the same for a unique input.
• Uniformity
• The distribution of outputs across the output space should be equal.

Selecting a Key

A better sharding function would be shard(request.path). 

When we use request.path as the shard key, then we map both requests to the same shard, and thus the response to one request can be served out of the cache to service the other.

Hot Sharding Systems
Ideally the load on a sharded cache will be perfectly even, but in many cases this isn't true and “hot shards” appear because organic load patterns drive more traffic to one particular shard.

As an example of this, consider a sharded cache for a user's photos; when a particular photo goes viral and suddenly receives a disproportionate amount of traffic, the cache shard containing that photo will become “hot.”

When this happens, with a replicated, sharded cache, you can scale the cache shard to respond to the increased load.

Indeed, if you set up auto scaling for each cache shard, you can dynamically grow and shrink each replicated shard as the organic traffic to your service shifts around.

An illustration of this process is shown in Figure 6-3. Initially the sharded service receives equal traffic to all three shards.

Then the traffic shifts so that Shard A is
receiving four times as much traffic as Shard B and Shard C.

The hot sharding system moves Shard B to the same machine as Shard C, and replicates Shard A to a second machine.

Traffic is now, once again, equally shared between replicas.

Scatter/Gather

Uses replication for scalability in terms of time.

Like replicated and sharded systems, the scatter/gather pattern is a tree pattern with a root that distributes requests and leaves that process those requests.

However, in contrast to replicated and sharded systems, with scatter/gather requests are simultaneously farmed out to all of the replicas in the system.

Each replica does a small amount of processing and then returns a fraction of the result to the root.

The root server then combines the various partial results together to form a single complete response to the request and then sends this request back out to the client.

example:

scatter/gather systems lead us to some conclusions:

• Increased parallelism doesn't always speed things up because of overhead on each node.
• Increased parallelism doesn't always speed things up because of the straggler problem. (one process is slow)
• The performance of the 99th percentile is more important than in other systems because each user request actually becomes numerous requests to the service.

The same straggler problem applies to availability.

If you issue a request to 100 leaf nodes, and the probability that any leaf node failing is 1 in 100, you are again practically guaranteed to fail every single user request.

Below:

Built this way, each leaf request from the root is actually load balanced across all healthy replicas of the shard. 

This means that if there are any failures, they won't result in a user visible outage for your system.

Likewise, you can safely perform an upgrade under load, since each replicated shard can be upgraded one replica at a time.

Indeed, you can perform the upgrade across multiple shards simultaneously, depending on how quickly you want to perform the upgrade.

Functions and Event-Driven Processing

Function-as-a-service (FaaS)

When FaaS Makes Sense:

• Functions are stateless and thus any system you build on top of functions is inherently more modular and decoupled than a similar system built into a single binary. 
• Each function is entirely independent.
• The only communication is across the network, 
• And each function instance cannot have local memory, requiring all states to be stored in a storage service.
• Additionally, the request-based and serverless nature of functions means that certain problems are quite difficult to detect.
• FaaS is inherently an event-based application model. Functions are executed in response to discrete events that occur and trigger the execution of the functions.

The Decorator Pattern: Request or Response Transformation

Kubeless is deployed on top of the Kubernetes

container orchestration service. Assuming that you have provisioned a Kubernetes cluster, you can install Kubeless from its releases page. 

Once you have the kubeless binary installed, you can

install it into your cluster with the following cmd: 

kubeless install

To can see deployed functions cmd:

kubectl get functions.

Handling Events

Ownership Election

Work Queue Systems

In the containerized work queue, there are two interfaces:

• the source container interface(like event sourcing), which provides a stream(也就是log, unbounded, immutable logs) of work items that need processing,
• and the worker container interface, which knows how to actually process a work item.

URLs:

• GET http://localhost/api/v1/items
• GET http://localhost/api/v1/items/<item-name>