[design][microservice] Microservices at Netflix scale

Watch note for Microservices at Netflix scale

Microservices

What's the cost?

Make assumptions

Then we could prioritize which is important.

Services should be stateless

• Not rely on sticky session
• Chaos testing
• Verify stateless (kill the instance randomly)

Scale out vs. scale up

• NoSQL at scale (Cassandra, Scylla)
• CAP available
  (Local quorum vs. async copy to remote region)

Redundancy and Isolation for resiliency

• Avoid single point of failure
• Make more than one of _anything_
• Isolate the blast radius for any given failure

Destructive testing

    Check check check

Billing services

Arch:

Request Cache!

Consider maintenance efforts

Writing tools for automation, less human involve would be better!

Reliability

Cascading failures

Tools to check microservices failure

Failure has grades. Some low grade failure is OK with backup services.
Tools: Hystrix, or Envoy

Containers

[design][microservice] Known Before Scaling

Watch note for What I Wish I Had Known Before Scaling Uber to 1000 Services • Matt Ranney

Microservices

• immutable?
• append only?

Move and release independently
Own your uptime
Use the best tool for the job

What are the costs?
Everything is RPC
What if it breaks?

Everything is a trade off.

[design][microservice] The hardest part of microservices is your data

Watch note for The hardest part of microservices is your data


slides:
https://www.slideshare.net/ceposta/the-hardest-part-of-microservices-your-data

• Microservices is about optimizing for SPEED.
• Manage (aka. reduce) dependencies.
• Data is a major dependency.
• Focus on domain models, not data models
• Stick with these conveniences as long as you can.
• A microservice has its own database
• We need to understand something about the data inside our services and the data outside our services
• Thus, we are building a distributed system, thus, beware of 
• network latency, 
• network partition
• Plan for failures.
• Build concepts of time, delay, network, and failures into the design as a first-class citizen
• How do you “read” data and how do you “update” data.
• Separate reads and writes
  (CQRS, Command Query Responsibility Segregation
  refer to: http://vsdmars.blogspot.com/2018/05/design-even-driven-system-pattern.html
• Performance: N+1 Query Problem
• https://secure.phabricator.com/book/phabcontrib/article/n_plus_one/
• Use Batching Queries to fix it!
• Load all your data before iterating through it.
• Detecting the Problem
• If seeing a bunch of similar queries, this often indicates an N+1 query issue(or a similar kind of query batching problem)
• For our reads and writes, we need some "consistency"
• We need reads and writes. But we expect failures. 
• CAP tells us to pick 2: Consistency, Availability, Partition Tolerance
• CAP is a bad way to think about this.
• Consistency model
  https://en.wikipedia.org/wiki/Consistency_model
• What consistency model do you need, depending on what role you’re playing?
• https://www.microsoft.com/en-us/research/wp-content/uploads/2011/10/ConsistencyAndBaseballReport.pdf
• Tool: debezium.io