Kubernetes

Logging & Monitoring

Plan

• Introduction
• Container technology
• Kubernetes
• Logging architecture
• Monitoring

How Long

from monolith to microservices ?

8 fallacies of distributed computing

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

Source: wikipedia

RFC 1925 - 12 Networking Truths

Logging & Monitoring:

monolithic app

-vs-

distributed system

credit @coda

Logging

Log Levels for dev

Monitoring

• Application errors 👉 where to look
• Business metrics 👉 money
• Latency 👉 user experience

Metrics, Metrics everywhere

youtu.be/czes-oa0yik

build OR buy

whole thread

Containers

What is a container

Not a real thing. An application delivery mechanism with process isolation based on several Linux kernel features.

kernel features

Dev 👉 inside container (build)

Ops 👉 outside container (run)

container = common interface for deploying services

cAdvisor

docker run \
--volume=/:/rootfs:ro \
--volume=/var/run:/var/run:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--volume=/dev/disk/:/dev/disk:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
google/cadvisor:latest
    

Logs for containers

👉 Treat logs as event streams

12factor.net

• ❌ no routing
• ❌ no storage
• ❌ no file handling

write logs to: stdout/stderr

Log Levels for dev

Kubernetes

Logging architecture

Node level logging

• JSON (no multiline)
• /var/log/
• keep previous pod logs
• pod eviction = ❌ no logs
• logrotate script

cluster level logging

Logs lifecycle & storage

independent of nodes, pods, or containers

logging with node agent

• per node agent pod (DaemonSet)
• centralized logging
• fluentd
• logs to stdout/stderr

logging with streaming side car

• logs to shared volumes
• sidecar streams logs to its own stdout
• separate log streams
• double disk usage
• better to directly write to stdout/stderr

logging with sidecar agent

• per pod agent (resources!)
• no kubectl logs

logging from application

which logs

tips

• logs don't replace tests or monitoring
• ❌"error" means someone should investigate
• maybe no need to log everything (GDPR?)
• business metrics to the queue
• github.com/wercker/stern: logs multiple containers in parallel
• /dev/termination-log 👉 can be specified in YAML
• Kubernetes deconstructed (Vimeo)

Monitoring

CPU & RAM should be enough

not really...

• docker stats
• kubectl top nodes
• kubectl top pods

Why monitoring

• detect/prevent outages (alerting)
• entry price for chaos engineering
• auto-scaling (HPA)
• optimize (cost & perfs)

different levels of monitoring

• Infrastructure level - U.S.E
• Application level - R.E.D

What to monitor

1. request time/rate (if it's fast, it works)
2. connections (health check, DB, pods)
3. kubernetes pods (CrashLoopBackOff,...)
4. kubernetes internals (control plane, kubelet, ...)
5. infrastructure (disk space, CPU, RAM, network,...)

Health check

what does "healthy" mean?

Where to monitor

maybe not on the cluster that you are monitoring

don't take my word for it

These measurements describe basic properties of metrics that matter, what values we want those metrics to have, and how we’ll react if we can’t provide the expected service

Going further

K8s for dev

Open data

example: makebook.io/open

doesn't track your users

GDPR friendly