Logging & Monitoring


  • Introduction
  • Container technology
  • Kubernetes
  • Logging architecture
  • Monitoring

Julien Bisconti

Software Engineer
specialized in Google Cloud

Google Cloud Platform icon



How Long

from monolith to microservices ?

8 fallacies of distributed computing

  1. The network is reliable.
  2. Latency is zero.
  3. Bandwidth is infinite.
  4. The network is secure.
  5. Topology doesn't change.
  6. There is one administrator.
  7. Transport cost is zero.
  8. The network is homogeneous.

Source: wikipedia

RFC 1925 - 12 Networking Truths

Logging & Monitoring:

monolithic app


distributed system

Loggingrecording events
Metricsdata combined from measuring events
Tracingrecording events with causal ordering

credit @coda


Log Levels for dev

  • Info
  • Debug
  • Warning
  • Error
  • Fatal
  • Zombie-Apocalypse
  • Meteor
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€“  when necessary
  •    πŸ€·β€β™‚οΈ  don't care
  •    🧐  to investigate
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€·β€β™‚οΈ  don't care
string perf tweet string perf


From user to metrics

  • Application errors πŸ‘‰ where to look πŸ‘€
  • Business metrics πŸ‘‰ money
  • Latency πŸ‘‰ user experience
funny cartoon

Metrics, Metrics everywhere


build OR buy

Tweet about datacenter

whole thread


What is a container

Not a real thing. An application delivery mechanism with process isolation based on several Linux kernel features.

kernel features

Dev πŸ‘‰ inside container (build)

Ops πŸ‘‰ outside container (run)

container = common interface


docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \

Logs for containers

πŸ‘‰ Treat logs as event streams


  • ❌ no routing
  • ❌ no storage
  • ❌ no file handling

write logs to: stdout/stderr

Log Levels for dev

  • Info
  • Debug
  • Warning
  • Error
  • Fatal
  •    πŸ€·β€β™‚οΈ  don't care
  •    πŸ€“  stdout
  •    πŸ€·β€β™‚οΈ  don't care
  •    🧐  stderr
  •    πŸ€·β€β™‚οΈ  don't care


kubernetes architecture

Logging architecture

Node level logging

node level logging
  • JSON (no multiline)
  • /var/log/
  • keep previous pod logs
  • pod eviction = ❌ no logs
  • logrotate script

cluster level logging

Logs lifecycle & storage

independent of nodes, pods, or containers

logging with node agent

logging with node agent
  • per node agent pod (DaemonSet)
  • centralized logging
  • fluentd
  • logs to stdout/stderr

logging with streaming side car

logging with streaming side car
  • logs to shared volumes
  • sidecar streams logs to its own stdout
  • separate log streams
  • double disk usage
  • better to directly write to stdout/stderr

logging with sidecar agent

logging with sidecar agent
  • per pod agent (resources!)
  • no kubectl logs

logging from application

logs from application

which logs

typical app


  • logs don't replace tests
  • "error" means someone should investigate
  • maybe no need to log everything (GDPR?)
  • business metrics to the queue
  • `kubectl logs --all-container` PR
  • /dev/termination-log


CPU & RAM should be enough

not really...

  • docker stats
  • kubectl top nodes
  • kubectl top pods

Why monitoring

  • detect/prevent outages (alerting)
  • entry price for chaos engineering
  • auto-scaling (HPA)
  • optimize (cost & perfs)

What to monitor

  1. request time/rate (if it's fast, it works)
  2. connections (health check, DB, pods)
  3. kubernetes pods (CrashLoopBackOff,...)
  4. kubernetes internals (control plane, kubelet, ...)
  5. infrastructure (disk space, CPU, RAM, network,...)
RED = Request rate, Error and Duration - by Tom Wilkie

Health check

what does "healthy" mean?

typical app

Where to monitor

maybe not on the cluster that you are monitoring

don't take my word for it


These measurements describe basic properties of metrics that matter, what values we want those metrics to have, and how we’ll react if we can’t provide the expected service

Define SLI SLO

Going further

Open data

example: makebook.io/open


doesn't track your users

GDPR friendly


Github: @veggiemonk
Twitter: @veggiemonk
LinkedIn: julienbisconti
blog: https://bisconti.cloud/
slides: https://www.slideshare.net/julienbisconti

Slides made with Revealjs