CHAOS ENGINEERING

with

SERVICE MESH

Julien Bisconti

SRE / Data Engineer

contact

g.dev/julien

slides: bisconti.cloud

Outline

Genesis
Kunernetes networking model
Service mesh: architecture and features
Demo of Istio
Chaos Engineering: concepts & origin
Demo of fault-injection
Q&A

Questions to the audience

Who uses containers ?
Who uses an orchestrator such as kubernetes ?
Who knows what is a service mesh ?
Who knows what is a SLI, SLO, SLA ?
Who knows what is Chaos Engineering ?
Who already did Chaos Engineering ?

Service Mesh

CONNECT

SECURE

CONTROL

OBSERVE

source: istio.io

VIDEO: Istio a la carte by Dan Ciruli

8 fallacies of distributed computing

The network is reliable.
Latency is zero.
Bandwidth is infinite.
The network is secure.

Topology doesn't change.
There is one administrator.
Transport cost is zero.
The network is homogeneous.

source (wikipedia)
RFC 1925 ( 12 Networking Truths )

using a library, framework or service for:

circuit breaking
timeouts
retries
service discovery
client-side loadbalancing
metrics
traffic shaping
rate limiting
...

for which languages?

drawbacks

combination language/framework/version/feature
maintain, upgrade, migrate, retire
code pollution and complexity (+ testing)
deployment / rolling update
language/framework/version lock-in
debugging

➡️ ️ move it to the infrastructure

Kubernetes networking model

1. all containers → all other containers without NAT

2. all nodes → all containers
all nodes ← all containers
without NAT

3. the IP that a container sees itself as
is the SAME
IP that others see it as

NAT (wikipedia)
VIDEO: Kubernetes Deconstructed (1h)

What is a service mesh

What problems does it solve

Communication between services

A network for services, not bytes

How does it solve inter service communication

The overall architecture of an Istio-based application.

source

What's in the code


details = {
    "name" : "http://details:9080",
    "endpoint" : "details",
    "children" : []
}
ratings = {
    "name" : "http://ratings:9080",
    "endpoint" : "ratings",
    "children" : []
}

source code

Traffic Management


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
  ...
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

Resiliency


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v2
      retries:
        attempts: 3
        perTryTimeout: 2s

Security

namespace-level and service-level policies
mutual TLS Authentication
role-based access control (RBAC)

Observability

Metrics (prometheus)
Logs (fluentd)
Tracing (jaeger)
Cluster traffic (kiali)

DEMO

QUESTIONS

about service mesh

CHAOS ENGINEERING

Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.
— principlesofchaos.org

Thoughtful, planned experiments designed to reveal the weakness in our systems.
— Kolton Andrus (cofounder and CEO of Gremlin Inc.)

Chaos Engineering isn't done to cause problems; it is done to reveal them.
— Nora Jones (Netflix)

Chaos Engineering is exploratory testing of non-functional requirements where ‘non-functional requirements’ are the requirements that if not met render a service non-functional.
— @littleidea