Logging & Monitoring


  • Introduction
  • Container technology
  • Kubernetes
  • Logging architecture
  • Monitoring

Julien Bisconti

SRE / Data Engineer

Google Cloud Platform icon



slides: bisconti.cloud

How Long

from monolith to microservices ?

8 fallacies of distributed computing

  1. The network is reliable.
  2. Latency is zero.
  3. Bandwidth is infinite.
  4. The network is secure.
  5. Topology doesn't change.
  6. There is one administrator.
  7. Transport cost is zero.
  8. The network is homogeneous.

Source: wikipedia

RFC 1925 - 12 Networking Truths

Logging & Monitoring:

monolithic app


distributed system

Loggingrecording events
Metricsdata combined from measuring events
Tracingrecording events with causal ordering

credit @coda


Log Levels for dev

  • Info
  • Debug
  • Warning
  • Error
  • Fatal
  • Zombie-Apocalypse
  • Meteor
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€“  when necessary
  •    πŸ€·β€β™‚οΈ  don't care
  •    🧐  to investigate
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€·β€β™‚οΈ  don't care
string perf tweet string perf


  • Application errors πŸ‘‰ where to look
  • Business metrics πŸ‘‰ money
  • Latency πŸ‘‰ user experience

Metrics, Metrics everywhere


build OR buy

Tweet about datacenter

whole thread


What is a container

Not a real thing. An application delivery mechanism with process isolation based on several Linux kernel features.

kernel features

Dev πŸ‘‰ inside container (build)

Ops πŸ‘‰ outside container (run)

container = common interface for deploying services


docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \

Logs for containers

πŸ‘‰ Treat logs as event streams


  • ❌ no routing
  • ❌ no storage
  • ❌ no file handling

write logs to: stdout/stderr

Log Levels for dev

  • Info
  • Debug
  • Warning
  • Error
  • Fatal
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€“  stdout
  •    πŸ€·β€β™‚οΈ  don't care
  •    🧐  stderr
  •    πŸ€·β€β™‚οΈ  don't care


kubernetes architecture

Logging architecture

Node level logging

node level logging
  • JSON (no multiline)
  • /var/log/
  • keep previous pod logs
  • pod eviction = ❌ no logs
  • logrotate script

cluster level logging

Logs lifecycle & storage

independent of nodes, pods, or containers

logging with node agent

logging with node agent
  • per node agent pod (DaemonSet)
  • centralized logging
  • fluentd
  • logs to stdout/stderr

logging with streaming side car

logging with streaming side car
  • logs to shared volumes
  • sidecar streams logs to its own stdout
  • separate log streams
  • double disk usage
  • better to directly write to stdout/stderr

logging with sidecar agent

logging with sidecar agent
  • per pod agent (resources!)
  • no kubectl logs

logging from application

logs from application

which logs

typical app



CPU & RAM should be enough

not really...

  • docker stats
  • kubectl top nodes
  • kubectl top pods

Why monitoring

  • detect/prevent outages (alerting)
  • entry price for chaos engineering
  • auto-scaling (HPA)
  • optimize (cost & perfs)

different levels of monitoring

  • Infrastructure level - U.S.E
  • Application level - R.E.D
USE method: for every resource, check:
  • utilization
  • saturation
  • errors
RED method: for every service, check request:
  • rate
  • error rate
  • duration (distributions)
are withing SLO/A
Site Reliability Engineering book by Google
4 Golden Signals

  • Latency
  • Traffic
  • Errors
  • Saturation


What to monitor

  1. request time/rate (if it's fast, it works)
  2. connections (health check, DB, pods)
  3. kubernetes pods (CrashLoopBackOff,...)
  4. kubernetes internals (control plane, kubelet, ...)
  5. infrastructure (disk space, CPU, RAM, network,...)

Health check

what does "healthy" mean?

typical app

Where to monitor

maybe not on the cluster that you are monitoring

don't take my word for it


These measurements describe basic properties of metrics that matter, what values we want those metrics to have, and how we’ll react if we can’t provide the expected service

Define SLI SLO

Going further

K8s for dev

Open data

example: makebook.io/open


doesn't track your users

GDPR friendly


and I'm sorry πŸ™
If you had to maintain my code
I hope you learned more by maintaining it
than me by writing it




Slides made with Reveal.js and hugo-reveal