Appwrite is an open-source, self-hosted Backend-as-a-Service that makes app development easier with a suite of SDKs and APIs to accelerate app development. #30DaysOfAppwrite is a month-long event focused on giving developers a walkthrough of all of Appwrite's features, starting from the basics to more advanced features like cloud functions! Alongside we will also be building a full-featured Medium clone to demonstrate how these concepts can be applied when building a real-world app. We also have some exciting prizes for developers who follow along with us!
Welcome to Day 3 👋
Today, we take a deep dive into Appwrite's technology stack and learn what goes on under the hood. We will cover Appwrite's docker-compose file and go through all the containers that Appwrite uses. Appwrite was designed to work in a Cloud Native environment, and in keeping this spirit alive, Appwrite is conveniently packaged as a set of docker containers (18, to be precise!).
This is an overview of Appwrite's architecture, and we will talk about most of these components in the upcoming section. Each container in Appwrite handles a single microservice on its own. Since they've been containerized, each service can scale independently of any of the other microservices.
Currently, all of the Appwrite microservices communicate over the TCP protocol over a private network. You should be aware not to expose any of the services to the public-facing network besides the public ports 80 and 443, which, by default, are used to expose the Appwrite HTTP API.
This is the main Appwrite container, and this is where all the fancy things happen! This container is built off a Dockerfile hosted here. The main Appwrite container implements the Appwrite API protocols, and handles authentication, authorization, and rate-limiting. This microservice is completely stateless and can be easily replicated for scalability.
Traefik is a modern reverse proxy and load balancer written in Go that makes deploying microservices easy. Traefik integrates with your existing infrastructure components and configures itself automatically and dynamically. We use Traefik as the main entry point for the different Appwrite APIs. Traefik is also responsible for serving Appwrite's auto-generated SSL certificates. This microservice is completely stateless.
Appwrite uses Redis to serve three main functions.
- Caching: Appwrite uses Redis in-memory cache to fetch database queries faster.
- Pub/Sub: Appwrite uses Redis with Resque as a pub/sub mechanism to transmit messages between the Appwrite API and the different workers.
- Scheduled tasks: Appwrite uses Redis to store and trigger future tasks using the schedule container.
There are a lot of asynchronous tasks that need to happen in Appwrite - a good example is recording usage stats for the Appwrite API.
We use an internal pub/sub system - Resque - to accumulate all these tasks. The respective workers consume these tasks and perform the executions independently. We have eight message queues and eight workers paired with them.
The Audits worker consumes messages from the
v1-auditsqueue. Appwrite has a defined set of system events that can be found here. When these events occur, the Audits worker logs them into
mariadb. The Audits worker makes use of the utopia-php/audit library.
The Certificates worker consumes messages from the
v1-certificatesqueue. The certificate worker uses
certbotfrom Let's Encrypt to create and periodically renew SSL certificates. We will be covering SSL certificates and more about the certificates worker in our upcoming article. So stay tuned to learn more.
The Deletes worker consumes messages from the
v1-deletesqueue. As the name suggests, it performs deletions in the Appwrite Database. Delete requests for Documents, Users, Projects, Functions etc. are handled by the Deletes Worker. In the present state, the deletes worker is triggered on certain API requests, as well as by the Maintenance worker.
The Functions worker consumes messages from the
v1-functionsqueue and handles all of the tasks related to Appwrite's Asynchronous Cloud Functions. Synchronous functions skip the Functions worker and are directly sent to the Executor.
The Executor is responsible for all communication between Appwrite and the orchestration service in use. It handles executing functions, deleting functions, building the functions, and more.
Cloud functions in Appwrite can be triggered in three ways:
The Executor does all the heavy lifting required to get Cloud Functions up and running. From pulling Docker images for the respective environments on startup to managing and running containers and responding to errors, the Executor takes care of it all!
The Tasks worker consumes messages from the
Appwrite's Tasks API allows you to schedule any repeating tasks your app might need to run in the background. Each task is created by defining a CRON schedule and a target HTTP endpoint.
Each task can define any HTTP endpoint with any HTTP method, headers or basic HTTP authentication. Inside your Appwrite console, you can view all your tasks, their current statuses, previous and next runtime, and a response log to view the result of previous executions.
The Usage worker consumes messages from the
v1-usagequeue and makes use of
statsdto send messages to
telegrafover a UDP connection. The usage stats are then logged in
influxDB, including function execution stats, the total number of requests, storage stats, etc.
The Database worker handles the creation and deletion of database attributes and indexes.
The Webhooks worker consumes messages from the
v1-webhooksqueue and triggers the webhooks that were registered in the Appwrite console. The worker checks for the event that occurs and fires the corresponding webhook by making a CURL request.
Webhooks allow you to build or set up integrations that subscribe to certain events on Appwrite. When one of those events is triggered, we send an HTTP POST payload to the webhook's configured URL. Webhooks can be used to purge cache from CDNs, calculate data or send a Slack notification. You're only limited by your imagination.
Additionally we have two more workers that delegate tasks to other workers.
The maintenance worker corresponds to the
appwrite-maintenanceservice in the docker-compose file. The Maintenance worker lies here and performs some housekeeping tasks so your Appwrite server does not blow up over time! In its current state, the maintenance worker delegates deletion tasks to
appwrite-worker-deleteswhich then performs the actual deletion. We use the Maintenance worker to schedule three kinds of deletions:
Cleaning up Abuse logs
Cleaning up Audit Logs
Cleaning up Execution Logs
The Schedules worker corresponds to the
appwrite-scheduleservice in the docker-compose file. The Schedules worker uses a Resque Scheduler under the hood and handles the scheduling of CRON jobs across Appwrite. This includes CRON jobs from the Tasks API, Webhooks API and the functions API.
Appwrite uses MariaDB as the default database for project collections, documents and all other metadata. Appwrite is agnostic to the database you use under the hood and support for more databases like Postgres, CockroachDB, MySQL and MongoDB is currently under active development! 😊
ClamAV is a TCP Anti-virus server responsible for scanning all user uploads to the Appwrite storage. The ClamAV microservice is optional and can be disabled using Appwrite environment variables. Starting with Appwrite verson 0.8, this functionality is disabled by default to save memory on smaller setups. If you are having issues with excessive memory utilisation, you can learn to disable it here
Appwrite uses InfluxDB for storing your projects' API usage metrics and stats. This is the engine used for generating your API usage graphs and handling time-series data.
Telegraf is a plugin-driven server agent for collecting and sending metrics and events from multiple sources to multiple destinations. Telegraf kind of protects InfluxDB by aggregating the data before sending it to the database. Telegraf operates on the UDP protocol, which makes data transfer blazing fast!
Feel free to reach out to us on Discord if you would like to learn more about Appwrite, Aliens or Unicorns 🦄. Stay tuned for tomorrow's article! Until then 👋