EnqueueZero Techshack 2019-06
A Kubernetes Developer Workflow for MacOS
This article introduces a set of kube-extended tools for macOS users: kubectx, kubectl alias, krew, skaffold, dive, stern.
WHAT YOU NEED TO KNOW - GitOps
GitOps is a way to do Continuous Delivery. It works by using Git as a single source of truth for declarative infrastructure and applications.
Rules:
- Everything that can be described must be stored in git.
- Kubectl should not be used directly.
- Use a Kubernetes controller that follows an operator pattern.
Below is a typical GitOps flow.
- A pull request for a new feature is pushed to GitHub for review.
- The code is reviewed and approved by a colleague. After the code is revised, and re-approved it is merged to Git.
- The Git merge triggers the CI and build pipeline, runs a series of tests and then eventually builds a new image and deposits to the new image to a registry.
- "Deployment Automator" watches the image registry, notices the image, pulls the new image from the registry and updates its YAML in the config repo.
- "Deployment Synchronizer" (installed to the cluster), detects that the cluster is out of date. It pulls the changed manifests from the config repo and deploys the new feature to production.
Introducing AresDB: Uber’s GPU-Powered Open Source, Real-time Analytics Engine
AresDB is an open source, real-time analytics engine that leverages an unconventional power source, graphics processing units (GPUs), to enable our analytics to grow at scale. The article describes the design of AresDB.
At a high level, AresDB stores most of its data in host memory (RAM that is connected to CPUs), handling data ingestion using CPUs and data recovery via disks. At query time, AresDB transfers data from host memory to GPU memory for parallel processing on GPU.
AresDB’s architecture supports the following features:
- Column-based storage with compression
- Real-time upsert with primary key deduplication
- GPU powered query processing
From ActiveMQ To Amazon MQ : Why And How We Moved To AWS’s Managed Solution
The article describes the solution migrating from self-hosted ActiveMQ to Amazon MQ.
Kubernetes Failure Stories
A compiled list of links to public failure stories related to Kubernetes. This compilation of failure stories should make it easier for people dealing with Kubernetes operations (SRE, Ops, platform/infrastructure teams) to learn from others and reduce the unknown unknowns of running Kubernetes in production.
Surviving On-Call: Tips from a Hosted Graphite SRE
- Set up "Do Not Disturb". The only audible notifications you get are either from close family, phone alarms, Pagerduty notifications or emergency work calls.
- Look after yourself while on-call. Grab a snap or have something to eat.
- Have Phone / 4G connection, laptop, power bank, backpack prepared.
CNAB: Packagaging for Applications with Multiple Toolchains
Large organizations have a diversity of toolchains, meaning lots more versions of lots more tools.
CNAB allows for packaging the tools along with the app they manage.
It's not about to create yet another package standard; it's about compatibility between tools.
CNAB provides compatibility between repositories and registries, makes it easier to integrate any package format.
It has below concepts: metadata (bundle.json), on-disk representation (fs layout), execution (oci runtime entrypoint with arguments), and distribution (oci image and distribution specifications).
Key takeaways:
- CNAB is a specification. Think MSI or OSI rather than Helm charts.
- Early days, but with lots of hacking potentials.
Distributing with Distribution: Upcoming Changes to Helm Chart Repositories
The article shows how to store Helm charts in Distribution (also known as Docker Registry v2). Simply put, a Chart Repository is a basic HTTP server that houses an index.yaml file and some packaged charts. Docker’s Distribution project is not just for container images, but for any form of content, including Helm charts.
oras is an open source project developed during the Helm 3 discussions to push and pull any content from Distribution.
Usage:
- Launch Registry locally.
$ docker run -dp 5000:5000 --restart=always --name registry registry:2
- Upload any file.
$ echo "hello world!" > helloworld.txt
$ oras push localhost:5000/helloworld:latest helloworld.txt:text/plain
- Download any file.
$ oras pull localhost:5000/helloworld:latest -t text/plain
Coroutines in C
The article describes how coroutine works, from the ground up. By introducing the coroutine, we keep the code almost the same, but change the execution flow completely. The basic idea is by using Duff's device.
#define crBegin static int state=0; switch(state) { case 0:
#define crReturn(i,x) do { state=i; return x; case i:; } while (0)
#define crFinish }
int function(void) {
static int i;
crBegin;
for (i = 0; i < 10; i++)
crReturn(1, i);
crFinish;
}
Target labels are for life, not just for Christmas
How should you choose the labels to put on your Prometheus monitoring targets?
- Target Labels Should be Constant.
- Target Labels should be Minimal.
API Gateways Are Going Through an Identity Crisis
API is an explicitly and purposefully defined interface designed to be invoked over a network that enables software developers to get programmatic access to data and functionality within an organization in a controlled and comfortable way. The API-gateway pattern is about curating an API for more optimal usage by different classes of consumers.
Journey to Event Driven – Part 1: Why Event-First Thinking Changes Everything
- Why do events matter? There has been a revolution; organizations must become real time; to become real time, they must become event driven. The value of events is that a sequence of related events represent behavior.
- Two patterns:
- Event-first patterns, sending out FACTS.
- Event-command patterns, sending out COMMANDS.
- The cost of the event-first approach
- Need to support traceability, failure paths, scaling and explanation about why things have gone wrong.
- Benefits of the event-first approach
- Decoupling
- Encapsulation
- Inverted knowledge
- Evolutionary change
- Event sourcing
Overload control for scaling WeChat microservices
The paper describes the design of the battle hardened overload control system DAGOR that has been in production at WeChat for five years.
Challenges:
- The backend has over 3k services, including instant messaging, social networking, mobile payment, and third-party authorization.
- The platform sees between 1010 to 1011 external requests per day, fanning more services requests such that the WeChat backend as a whole needs to handle hundreds of millions of requests per second.
- The microservices system runs over 20k machines, and the number is increasing.
- There are a thousand system changes per day on average.
- At certain times of year (e.g. around the Chinese Lunar New Year) peak workload can rise to 10x the daily average.
Architecture - 3-tier services:
- Entry leap services: front-end services receiving external requests
- Shared leap services: middle-tier orchestration services
- Basic services: services that don’t fan out to any other services, and thus act as sinks for requests
WeChat’s overload control system is called DAGOR. It aims to provide overload control to all services and thus is designed to be service agnostic.
- For overload detection, DAGOR uses the average waiting time of requests in the pending queue (i.e., queuing time).
- Once overload is detected, a curated feature priority table is used for dropping some requests. It adds a second layer of admission control based on user-id. Considering a total number of 128 levels, when overload control is set to business priority level n, all requests from levels n+1 will be dropped.
Lesson Learned:
- Overload control in a large-scale microservice architecture must be decentralized and autonomous in each service
- Overload control should take into account a variety of feedback mechanisms (e.g. DAGOR’s collaborative admission control) rather than relying solely on open-loop heuristics.
- Overload control design should be informed by profiling the processing behaviour of your actual workloads.
StranglerApplication
^ Over many years the huge strangler vines seed in the upper branches of a host tree, grow into fantastic and beautiful shapes, meanwhile strangling and killing the host.
The metaphor is, instead of making a new system that replaces the existing one, an alternative route is to gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled.
The fundamental strategy is EventInterception, which can be used to gradually move functionality to the strangler and to enable AssetCapture.
There's another important idea here - when designing a new application you should design it in such a way as to make it easier for it to be strangled in the future.
Git History
Quickly browse the history of any GitHub file:
- Replace github.com with github.githistory.xyz in any file url
- There's no step two
Dotfile madness
"My own home directory contains 25 ordinary files and 144 hidden files." This particular problem has been noticed and solved a long time ago with the creation of XDG Base Directory Specification.
Using the standard is very simple. Read the relevant environment variable and use the default paths defined by the standard if it is missing. You should then append a program-specific directory name to it and create the entire directory tree to store your data.
Original published: http://enqueuezero.com/techshack.weekly/2019-06.html
Top comments (0)