The journey continues! Day 2 to 4 where packed with talks! I tried to create briefs from each one I followed.
From Day 2 onward the talks where early and late in the afternoon, leaving space for keynotes in the middle. With that many interesting talks, I have to admit I skipped all keynotes after the first one in Day 2, but I'm watching them on-demand (it was easier to interact with speakers timely after their talks, as there were no talk dedicated channels but topic dedicated channels, so following conversations long after the talk was complex at times).
As organization embrace the Cloud and it's tools, managing fragmentation becomes more and more complex. The idea of applying design patterns to create composable and reusable infrastructure components is very cool. It also allows to abstract in a reusable way different cloud provider patterns.
They integrate pipelines in the equation, which is awesome. Flexibility of pipelines is one of the key advantages of being in the Cloud, leveraging that is very powerful.
It's not a one-size-fit-all approach, but their solution is inspiring. Based on Infrastructure as a Code and CueLang for configurations.
The talk explores a powerful concept: configuration as data >> configuration as code.
Kelsey Hightower@kelseyhightowerDuring the @ArrestedDevOps podcast, hosted by @bridgetkromhout, I was able to dive into what I refer to as infrastructure as data. Its the idea that configuration should be treated as data and leverage pipelines for manipulation and policy enforcement. arresteddevops.com23:13 PM - 04 Dec 2019
Their solution does not take into account state management, that should be separately handled (they use Pulumi, previously they used Terraform).
Companion book: Kubernetes Patterns (Free download https://k8spatterns.io/)
Nothing particularly new to me, I've had my chance to review and study Kubernetes patterns some time ago.
Worth checking out if you're a developer using K8s for app delivery. The book is definitely to be downloaded for study and future reference.
Progressive delivery is a term to describe deployment techniques to avoid common all-or-nothing deployment pitfalls.
The goal is to shift traffic slowly from the old to the new version, being able to roll back immediately in case of unexpected behaviours of the new deployment.
Avoid downtime, limit the blast radius, ship code faster!
I was positively impressed by the visual explanations of techniques. The concepts were clear to me before the presentation, but have been explained so well it has been a pleasurable review.
Monitoring is the new testing
Today monitoring allow to know when users are experiencing issues and react automatically.
How to manage databases with progressive delivery?
- Use Cloud hosted solutions
- Make you app always backward compatible
- Only non-breaking database schema changes.
Who is for: Ops
by Ying Chun Guo
I was a bit mislead by the title of this talk. The focus is on Knative (a serverless platform on top of Kubernetes) and Tekton (a Kubernetes-native CI/CD system). Truly Kubernetes-native but very "locked in".
Seeing how the two systems play together has been interesting, but unless you are planning to adopt such technologies there is little knowledge that you can reuse in other environment.
This has been one of the most interesting talk I've followed.
Scanning for vulnerabilities does not scale. In 2019 there have been little less than 50 vulnerabilities per day.
How can the team keep up with that? By reducing noise and focusing on what matters to us.
Use Open Policy Agent (OPA) to define how to handle vulnerabilities discovered by a scanner. This allows to create policies to handle vulnerabilities based on internal knowledge and requirements.
Trivy has direct OPA integration.
trivy-enforcer is a custom K8s controller that allows enforcing policies on images to be deployed on K8s. Still experimental, looks awesome!
Do you happen to need caching data in your services? Generally, is quite a common need.
The overview of caching pattern was quite complete but the talk felt centred around promoting the speaker company product more than giving a solid understanding of different caching patterns.
Still worth to watch if you want an overview of caching patterns or if you're interested in the product (Hazelcast).
This talk is presented by a UI engineer! It's so interesting to see tracing applied to frontend because can really help understand where customers are having issues using your application.
Why only Backend and Ops should have the cool tools? OpenTelemetry.js
UI is much more than a "service". Users do lots of things in UI and with complex UI apps (microfrontends, complex journeys) tracing is powerful.
By carefully tagging traces is possible to correlate frontend and backend traces to look at the entire user journey. COOL!
Database data resiliency is always an interesting topic for me. Nobody cares if the app is up and the code works if the database is broken.
The talk had cool tips, and some very nice troubleshooting/discovery commands. Highly recommended!
It starts with a clear explanation on Kubernetes related risks for stateful applications and mitigations.
The main mentioned topics have been: replicas, affinity, distance, ephemeral storage and operator error.
Replica is one of the ways to avoid node failure.
How does Replicas cope with blast radius (how fare away you need to be to not be affected by a failure)?
The process of requesting different replica to be nearby or distanced. Kubernetes supports Pod affinity natively.
Protect replicas using distance (affinity/anti-affinity).
Take backup in object storage (it's easy and durable).
How does distance cope with blast radius? How to protect across regions or K8s clusters? It's complicated: it's not just a data problem, is a resource problem too (think load, DNS). (This part was out of scope for the talk, unfortunately!)
The persistent volume that doesn't
In Kubernetes ephemeral storage is a feature. Unless you want it to be persistent 😅
A pod cannot understand if storage is ephemeral or not.
Persistent Volumes are not enough if you don't use them! When you expect your Pod to use persistent storage, verify it writes data to be persisted to the proper mount point, otherwise ephemeral storage is used.
Fat fingers of fate
The saddest one. When an operator deletes applications. In Kubernetes deleting workloads is very easy. Once an application is deleted, dynamically provisioned persistent volumes (like the ones created by cloud provider integrations) are deleted by default.
But fear no more, you can avoid that changing the Reclaim Policy to
Retain. You can achieve the same by creating a new
Retain policy and use that.
When a company starts it's journey, there is a blank slate. Then, after a while, you get in the jungle, where everything is in your stack and there is tech debt everywhere.
It's time to find the Golden Path.
Golden Path is not "the only" path, but it makes easier for the organization to work together. It's the supported path, out of it, you're on your own.
Meanwhile, you may want to build Cloud Native apps. And what about CI/CD? What is a Cloud Native CI/CD?
"Avoid e2e test at all, they are very brittle. Adopt Contract Testing instead."
When building the Golden Path, build the map of where we are going and embed it in the pipelines.
This one has been one of the most interesting for me, as surfing the amount of different tech stacks in a company can be incredibly daunting and I can see the Golden Path value proposition.
That's the end for Day 2! Any talk you think is missing? Let me know in the comments!