CHAOS ENGINEERING

with

SERVICE MESH

Julien Bisconti

SRE / Data Engineer

contact

g.dev/julien

slides: bisconti.cloud

Outline

Genesis
Service mesh: architecture and features
Demo of Envoy and Istio
Chaos Engineering: concepts & origin
Demo of fault-injection
Q&A

at the beginning there was an

APP

and the app was code

that needed to scale

👉 microservices

Deployment

Containers: lightweight VMs

12 factor app
easier deploy
reproducible build

but ...

Deployment concerns

Scaling up and down
Redundancy
Scheduling / Orchestration
Service Discovery

Resiliency
Rolling out and back
Health checks
Secret and config

➡️ kubernetes

but ...

8 fallacies of distributed computing

The network is reliable.
Latency is zero.
Bandwidth is infinite.
The network is secure.

Topology doesn't change.
There is one administrator.
Transport cost is zero.
The network is homogeneous.

source (wikipedia)
RFC 1925 ( 12 Networking Truths )

Kubernetes concerns

Logging
Tracing
Metrics
Dependency visualisation
Service identity and Auth

Circuit breaking
Traffic flow and policies
Failover
Fault injection
...

➡️ ️ use code?

drawbacks

combination language/framework/version/feature
maintain, upgrade, migrate, retire
code pollution and complexity (+ testing)
deployment / rolling update
language/framework/version lock-in
debugging

➡️ ️ move it to the infrastructure

Data plane

The network should be transparent to applications.
When network and application problems do occur it should be easy to determine the source of the problem.

The overall architecture of an Istio-based application.

How to manage a fleet of envoy proxy?

Service Mesh

CONNECT

SECURE

CONTROL

OBSERVE

source: istio.io

VIDEO: Istio a la carte by Dan Ciruli

What is a service mesh

What problems does it solve

Communication between services

A network for services, not bytes

How does it solve inter service communication

The overall architecture of an Istio-based application.

source

What's in the code


details = {
    "name" : "http://details:9080",
    "endpoint" : "details",
    "children" : []
}
ratings = {
    "name" : "http://ratings:9080",
    "endpoint" : "ratings",
    "children" : []
}

source code

Traffic Management


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
  ...
spec:
  hosts:
  - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

Resiliency


apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v2
      retries:
        attempts: 3
        perTryTimeout: 2s

Security

namespace-level and service-level policies
mutual TLS Authentication
role-based access control (RBAC)

Observability

Metrics (prometheus)
Logs (fluentd)
Tracing (jaeger)
Cluster traffic (kiali)

DEMO

QUESTIONS about service mesh

List of service meshes

Comparison: Consult vs Istio

Resources

CHAOS ENGINEERING

Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.
— principlesofchaos.org

Thoughtful, planned experiments designed to reveal the weakness in our systems.
— Kolton Andrus (cofounder and CEO of Gremlin Inc.)

Chaos Engineering isn't done to cause problems; it is done to reveal them.
— Nora Jones (Netflix)

Chaos Engineering is exploratory testing of non-functional requirements where ‘non-functional requirements’ are the requirements that if not met render a service non-functional.
— @littleidea