Building data/ML platforms
Lessons from the trenches
What is the purpose of
ML/data platforms ?
scaling
people & systems
( Skills & Tools )
Conway’s Law
"organizations [...] are constrained to produce designs which are copies of the communication structures of these organizations"
💡
tools
can help fix an
organization
but not its
culture
how to scale a system
- vertically
- horizontally
- deeply
village (100+) -> city (10k+) -> megacity (1m+)
same same but different
💡 we are not scaling a system up
we are creating a new system
that can sustain 100x the load
and we continously migrate to it
💡 Always be migrating
or at least plan for it
Migration
create the new
deprecate the old
problem definition
how to:
- train
- build
- deploy
- monitor
Machine Learning models
in a repoducible manner
at scale ?
Hidden Technical Debt in Machine Learning Systems
Source: D. Sculley, et al.: Hidden Technical Debt in Machine Learning Systemscost of context switching
Mental limitations
- # decisions / day
- # things to remember
- speed of memory / reflexes
Reliable
def: consistently good in quality or performance; able to be trusted.
“holistic approach toward deciding how to integrate development, deployment, production operations, and long-term care.”
CHAOS ENGINEERING*
* Requires good monitoring
What Chaos Engineering is not
Hope is not a strategy
Strategy >< tactics
Tactic is a specific action to achieve a goal
Strategy is a plan to achieve a goal
ML platform assembly kit
Source: article by Clemens Mewald
Yevgeniy Brikman - Lessons from 300k+ Lines of Infrastructure Code
Main blockers (Infra)
- networking
- identity and access management (IAM)
- resources organization
- biling
- policy management
source: GCP blog
IAM
- how to get access to the data
- how to remove access to the data
Good to understand OAuth & TLS certificates
build OR buy
💡
Laws might dictate that you have
to build your own
Tradeoffs
Build >< buy
time & skills >< Money
When things go wrong
At scale
everything hurts
[the cloud bill is not proportional to the number of customers but to the number of developers with access to the cloud account]
“Corey Quinn (@QuinnyPig)”
Cloud operation team
basically restricting what devs can do
💡
Paying for a platform costs more
but allows us to validate our use cases faster
💡
What is measured can be improved
Infrastructure as “Code”
- Terraform (HCL)
- CloudFormation (YAML)
- Jsonnet (Json)
- Pulumi (TS, Python, Go, …)
- CDKs (TS, Python, Go, …)
70% of outages are caused by a config change
prog. lang. (dynamic)
vs
config. lang. (static)
Error messages
source: Jenni Nadler (c)
converting types
the most awkwards and common things to do!
- gRPC/protobuf
- AVRO
- JSON
- DBs
- language types
- ORMs
💡 Everything is a tradeoff
“best” is the enemy of “good”
Main blockers (ML)
- data organization
- data quality
- data access/security
- data privacy
- data governance
- data lineage
- deployment
Deployment concerns
- Scaling up and down
- Redundancy
- Scheduling / Orchestration
- Service Discovery
- Resiliency
- Rolling out and back
- Health checks
- Secret and config
Kubernetes concerns
- Logging
- Tracing
- Metrics
- Dependency visualisation
- Service identity and Auth
- Circuit breaking
- Traffic flow and policies
- Failover
- Fault injection
- ...
using a library, framework or service for:
- circuit breaking
- timeouts
- retries
- service discovery
- client-side loadbalancing
- metrics
- traffic shaping
- rate limiting
- ...
for which languages?
Programming languages
- How many languages do we use ?
- How many languages do we know really well ?
source: "thread by Jaana Dogan" (c)
Go
It is not the language, it is the tooling/opinions
testing, benchmarks, deployments, dependency management, security, code formating, distribution …
Consistency is key
source
💡 Consistency is a myth but a worthy goal
💡
DevOps
basically people talking to each other through APIs
#thisisdevops
Yevgeniy Brikman - Lessons from 300k+ Lines of Infrastructure Code
Dev + Ops = DevOps
DevOps + Data = DataOps
DataOps + ML = MLOps
SRE ?
how different is ML
- Various hardware
- Resources heavy
- Various cycles
- Many languages
- Dependencies
- Explainability >< debugging
- Composability of models
- Huge amount of data
Testing
what if we had 2 clusters + LB
instead of a “clone” of production
Observability
is a key component of a platform
Monitoring
Answers the question: Are we making money now ?
Observability >< Monitoring
Observability deduces the state of the system based on its output
These measurements describe basic properties of metrics that matter, what values we want those metrics to have, and how we’ll react if we can’t provide the expected service
SLI / SLO / SLA
Start from the user experience
Nothing is perfect but we can always approximate
Security
Hackers are bots now
CVEs are monitored by hackers
They scan the internet for outdated software to exploit
Deploying
bake the model in the container or not ?
CICD
how to make changes to the system
And after a while
More models, more requests and more data
DORA
DevOps Research and Assessment report
https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-outSpeed is what matters
💡 Getting better at getting better
💡 If it hurts, do it more often
Julien Bisconti
SRE / Data Engineer
