STATERA Library · Series 08 · Volume 08 · Chapter 10 · EN
Source: https://github.com/Statera-Library/08_Vol_08_Robot_Database_and_Storage/blob/main/Chapter_10_Database_Replication_and_HA/Chapter_10_Database_Replication_and_HA_EN.md
Volume 08 Robot Database and Storage
10. Database Replication and HA
10.01 DB HA Patterns: Active.Passive / Active.Active
High availability in database systems is the architectural discipline of keeping data services accessible when servers, storage devices, networks, operating systems, or database processes fail. In robotics, this requirement can be especially important because fleet management, mission scheduling, robot configuration, telemetry, maps, and operational state may depend continuously on database services. A high-availability design therefore combines redundancy, replication, failure detection, traffic routing, and controlled recovery rather than relying on a single powerful database server.
Database high availability is commonly described through active-passive and active-active patterns. Both maintain more than one database instance, but they differ in how those instances process workloads. Active-passive architectures normally designate one database node as the operational primary while another remains synchronized and prepared to take over. Active-active architectures allow multiple nodes or sites to process workloads simultaneously, requiring additional mechanisms for coordinating writes, replication, consistency, and conflict resolution.
In an active-passive architecture, applications send normal database traffic to the active node. The passive node receives replicated data but generally does not accept application writes while operating as a standby. Replication may be synchronous, asynchronous, or a combination depending on durability and latency requirements. When the active node fails, the standby is promoted and client connections are redirected to it, restoring database service without requiring the failed machine to return immediately.
The major advantage of active-passive design is architectural clarity. Because one node normally owns the writable database state, application developers face fewer concurrent-write conflicts and the system can preserve familiar transactional behavior. Operations teams can also reason more easily about primary ownership, replication state, and failover. The main disadvantage is that standby resources may provide limited productive capacity during normal operation even though they consume infrastructure, maintenance, monitoring, and licensing resources.
Failover is the central operational process in active-passive HA. A monitoring component detects that the active database is unavailable or unhealthy, verifies whether promotion is safe, selects an eligible standby, promotes it, and redirects database clients. This sequence must distinguish genuine failures from temporary network interruptions. An incorrect promotion can create two nodes that both believe they are primary, producing the dangerous split-brain condition addressed later in this chapter structure.
Synchronous replication can reduce potential data loss by requiring a replica to acknowledge selected changes before a transaction is considered committed. This strengthens durability across node failure but adds communication latency to transaction processing. Asynchronous replication allows the primary to commit without waiting for the replica, usually improving response time but creating replication lag. If the primary fails before outstanding changes reach the standby, recently committed transactions may be unavailable after promotion.
Active-active architecture changes the model by allowing two or more database nodes, clusters, or geographic sites to serve application workloads concurrently. Instead of reserving one environment mainly for failure recovery, infrastructure can contribute to normal service capacity. Requests may be distributed by geography, tenant, robot fleet, shard, or application function. If one active location fails, traffic can potentially move toward another location that was already processing live workloads.
The difficulty of active-active operation lies primarily in data coordination. When multiple nodes can modify the same logical information, concurrent updates may occur before replication delivers each change everywhere. The database architecture must therefore define ownership, consensus, conflict detection, conflict resolution, or application-level reconciliation. Some designs avoid conflicts by assigning specific records or partitions to particular writers, while distributed databases may use quorum or consensus mechanisms to coordinate state across nodes.
Consistency requirements should determine replication architecture rather than treating active-active as automatically superior. Strongly consistent transactions may require coordination among nodes before accepting a write, which can increase latency as network distance grows. Systems prioritizing availability and low-latency local operations may permit temporary divergence and converge later. Robot systems often contain both categories: mission ownership and safety-relevant configuration may demand strict consistency, while some telemetry or historical observations can tolerate delayed synchronization.
Geographic distribution makes this tradeoff more visible. A robot fleet operating across several factories, warehouses, or cities may benefit from databases positioned close to each operational site. Local applications can continue using nearby services when a remote region becomes unreachable. However, inter-region latency and network partitions prevent designers from assuming that every location can synchronously coordinate every transaction at negligible cost. Data domains must therefore be classified according to consistency, latency, and recovery requirements.
High availability should also be distinguished from backup and disaster recovery. Replication creates another current or nearly current copy of database state, but replication can also reproduce accidental deletions, corrupted records, or malicious modifications. Backup retains recoverable historical states, while HA focuses primarily on service continuity. A robust robot data platform consequently combines replication and failover with backup, point-in-time recovery, integrity verification, and disaster-recovery procedures rather than replacing one mechanism with another.
Recovery objectives provide measurable requirements for this architecture. Recovery Time Objective expresses how long a database service may remain unavailable, while Recovery Point Objective expresses how much recent data loss can be tolerated. These targets influence whether manual failover is sufficient, whether automatic promotion is required, how many replicas should exist, and whether synchronous replication is justified. They also connect HA design directly with the backup and recovery concepts developed in the preceding chapter.
Robot databases should not assign identical HA requirements to every dataset. A fleet scheduler, robot authorization database, mission state store, or operational configuration service may require rapid recovery because interruption can stop multiple robots. Historical logs and large sensor archives may accept longer restoration periods. Separating these service classes prevents organizations from applying expensive synchronous redundancy universally while still protecting database functions that directly affect ongoing robot operations.
Client connectivity is another essential layer. Promoting a standby accomplishes little if applications continue connecting to the failed address. HA systems therefore commonly use virtual endpoints, proxies, service discovery, cluster-aware drivers, load balancers, or orchestrators to direct applications toward healthy database instances. Connection pools must also recognize broken sessions and reconnect safely. Robot applications should be designed for transient database failures instead of assuming that every connection remains permanently valid.
Failover must preserve application semantics as well as connectivity. A transaction may have reached the database immediately before a connection was lost, leaving the client uncertain whether the operation committed. Blindly repeating such requests can create duplicate mission records, commands, or events. Idempotent operations, transaction identifiers, unique constraints, retry policies, and explicit state verification help applications recover safely. Database HA is therefore partly an application architecture problem rather than purely an infrastructure configuration problem.
Monitoring determines whether an HA design works reliably in production. Operators should observe node health, replication delay, transaction throughput, storage capacity, connection counts, error rates, replication state, and failover events. Alerting should identify degradation before redundancy disappears completely. A primary database that remains operational while its only replica has been disconnected for hours is technically available, but the system has already lost the redundancy required to survive the next failure.
HA testing must intentionally exercise failure conditions. Teams should test database-process termination, server loss, storage faults, network partitions, replica lag, exhausted disk capacity, and loss of communication between cluster members. Tests should measure detection time, promotion time, application reconnection, transaction behavior, and data consistency after recovery. Periodic exercises reveal assumptions that normal monitoring cannot expose and prepare operators to respond predictably when real infrastructure failures occur.
For robot fleets, database availability should be considered together with edge autonomy. Robots should not become immediately unsafe merely because a central database connection disappears. Essential local control, collision avoidance, and short-term mission execution can remain at the robot or edge layer while centralized databases maintain fleet-level state and coordination. This separation reduces the consequence of temporary database outages while HA mechanisms restore centralized services and reconcile accumulated operational information afterward.
The choice between active-passive and active-active therefore depends on workload characteristics rather than terminology. Active-passive is often appropriate when transactional simplicity, deterministic ownership, and straightforward failover are priorities. Active-active becomes attractive when multiple locations must serve traffic concurrently, geographic latency matters, or service capacity should remain distributed during failures. Its benefits must be balanced against greater complexity in synchronization, consistency management, routing, conflict handling, and operational diagnosis.
A mature architecture may combine both patterns. A regional robot platform could operate active-active services across major sites while each regional database cluster internally uses an active-passive or consensus-based redundancy mechanism. Data can also be partitioned so that each region owns particular robot fleets while replicating selected information globally. Such layered designs demonstrate that database HA is not a single configuration but a hierarchy of failure domains, replication relationships, and application responsibilities.
Ultimately, database high availability is achieved when infrastructure failure becomes an anticipated operating condition rather than an exceptional event. Redundant nodes, appropriate replication modes, reliable health detection, controlled failover, safe client retries, continuous monitoring, and repeated failure testing must work together. These principles provide the conceptual foundation for the subsequent PostgreSQL, Patroni, MySQL or MariaDB, Redis, Elasticsearch, geo-distributed replication, monitoring, and split-brain topics defined in this chapter.
10.02 PostgreSQL Streaming Replication Deep Dive [w/Code]
PostgreSQL streaming replication is a physical replication mechanism that continuously transfers Write-Ahead Log (WAL) records from a primary database server to one or more standby servers. Instead of reproducing individual SQL statements, the standby applies low-level changes recorded in WAL, allowing it to maintain a binary-compatible copy of the primary database cluster. This mechanism forms a fundamental building block for PostgreSQL high availability, read scaling, and disaster recovery architectures.
Write-Ahead Logging is the foundation of PostgreSQL replication. Before modified database pages are permanently written to data files, PostgreSQL records the corresponding changes in WAL. This ordering supports crash recovery because committed changes can be reconstructed from the log after a failure. Streaming replication extends the same principle across servers by transmitting WAL records to standby systems, where PostgreSQL replays them to reproduce the primary server\'s database state.
The primary server generates WAL continuously as transactions insert, update, delete, or modify database objects. A WAL sender process on the primary communicates with a WAL receiver process running on each connected standby. The receiver obtains WAL records through PostgreSQL\'s replication protocol and writes them locally. A startup process on the standby then replays those records against its database files, progressively advancing the standby toward the current state of the primary.
A standby normally begins from a consistent physical copy of the primary database cluster. PostgreSQL tools such as pg_basebackup can create this initial base backup while the primary remains operational. Once the required database files and replication configuration are available, the standby starts recovery and connects to the primary. Streaming replication then supplies WAL generated after the base backup, allowing the standby to catch up without repeatedly copying the entire database.
Physical replication requires compatibility between primary and standby environments because WAL describes changes at PostgreSQL\'s storage level rather than as portable logical operations. The standby represents the database cluster as a whole rather than selectively subscribing to arbitrary tables. This differs from logical replication, which operates on logical data changes and can replicate selected tables. Streaming replication is therefore particularly suitable when the objective is maintaining a complete standby for availability and recovery.
PostgreSQL supports asynchronous streaming replication as the common low-latency operating model. In this configuration, the primary can report a transaction as committed without waiting for the standby to confirm that the corresponding WAL has been received or persisted. Application response time is therefore less dependent on replica communication. However, if the primary fails while some WAL remains in transit, the promoted standby may not contain the most recently committed transactions.
Replication lag describes the distance between the primary\'s current WAL position and the progress achieved by a standby. Lag may arise from limited network bandwidth, storage latency, high transaction volume, insufficient standby resources, long recovery operations, or temporary connectivity loss. A replica that is technically connected but significantly behind the primary may provide weaker recovery characteristics than expected. Monitoring replication progress is consequently essential when streaming replication supports production HA.
Synchronous replication changes the commit path by requiring confirmation from designated standby systems according to the configured synchronization policy. A transaction on the primary may wait until the required replica has received, written, flushed, or applied the relevant WAL, depending on configuration. This can substantially reduce the risk of committed transaction loss during primary failure, but database latency becomes more sensitive to network conditions and standby performance.
The choice between synchronous and asynchronous replication should reflect Recovery Point Objective (RPO), transaction criticality, network characteristics, and acceptable application latency. A mission-state database for a robot fleet may justify stronger durability guarantees than a database containing reconstructable operational statistics. Conversely, forcing geographically distant replicas into every synchronous commit path can increase transaction latency and make remote infrastructure conditions directly visible to applications.
PostgreSQL provides replication slots to help coordinate WAL retention for replicas and other replication consumers. A physical replication slot records the WAL position still required by a standby, preventing the primary from removing necessary WAL prematurely. This improves resilience when a replica disconnects temporarily. However, an inactive slot can cause WAL files to accumulate on the primary, potentially exhausting storage, so slot status and retained WAL volume must be continuously monitored.
WAL retention can also be supported through configuration and WAL archiving. Streaming replication delivers records directly while the connection is available, whereas archived WAL provides another path for retrieving historical log segments. A standby can therefore combine restored archive files with streamed WAL. This combination is valuable for resilient recovery architectures because temporary streaming interruptions do not necessarily require rebuilding the replica when the missing WAL remains available from an archive.
Hot standby capability allows read-only queries to execute while a standby continues replaying WAL. This makes replicas useful for reporting, monitoring, analytics, or other read workloads rather than reserving all standby resources exclusively for failure recovery. Read scaling must nevertheless account for replication delay because queries executed on a standby may observe an older database state than queries executed on the primary, especially when asynchronous replication is used.
Read workloads can also interact with WAL replay. A long-running query on the standby may require row versions that incoming WAL needs to remove or modify as part of recovery. PostgreSQL must then resolve the conflict, potentially delaying replay or canceling the conflicting standby query depending on configuration and workload behavior. HA replicas used simultaneously for analytical workloads should therefore be designed so that reporting activity does not undermine replication freshness or recovery readiness.
When a primary database becomes unavailable, a sufficiently current standby can be promoted to become a new writable primary. Promotion ends recovery mode and allows normal read-write transactions. Streaming replication itself provides the replicated database state required for this transition, but complete high availability also requires failure detection, promotion policy, client redirection, and protection against multiple primaries. These responsibilities are commonly handled by additional orchestration components.
After promotion, the old primary cannot simply resume normal writable operation without considering the new cluster history. If both systems accept writes independently, their timelines diverge and split-brain can occur. PostgreSQL uses timeline histories to represent changes in recovery lineage following promotion. Administrators or HA automation must ensure that the former primary is isolated, repaired, rewound when appropriate, or rebuilt before it rejoins the replication topology.
pg_rewind can be useful when a former primary has diverged from the newly promoted server but remains sufficiently compatible for synchronization. Rather than performing another complete base backup, pg_rewind identifies changed blocks and adjusts the old server to follow the new timeline. This can reduce recovery time and network transfer for large database clusters, although correct prerequisites, WAL availability, and operational procedures remain essential before the server is safely returned as a standby.
Cascading replication allows a standby to provide WAL to additional downstream standbys. Instead of every replica connecting directly to the primary, intermediate replicas distribute replication traffic. This architecture can reduce network and WAL-sender load on the primary and can be useful across sites or network boundaries. The trade-off is that downstream replicas inherit additional replication paths and may experience greater lag or become disconnected when an upstream standby fails.
For robot and Physical AI systems, streaming replication can protect operational databases containing fleet state, mission records, robot configuration, user authorization, maintenance information, and structured telemetry metadata. A local standby can provide rapid recovery from server failure, while additional replicas may support read-intensive fleet dashboards or remote recovery. Large raw sensor datasets, video, point clouds, and training archives are normally better handled by storage architectures designed for those data types.
Replication health should be monitored through PostgreSQL statistics and operational metrics rather than inferred only from whether a standby process is running. Important observations include replica connection state, WAL sender and receiver activity, WAL positions, replay progress, replication lag, replication-slot status, retained WAL volume, transaction latency, storage consumption, and standby recovery state. Alert thresholds should reflect the actual RPO and RTO required by each database service.
Failure testing is necessary because a replication topology that appears healthy during normal operation may behave differently during network partitions, storage exhaustion, abrupt server loss, or heavy transaction load. Operators should validate replica catch-up, WAL retention, promotion, client reconnection, old-primary isolation, and reconstruction of failed replicas. Tests should also verify whether the measured data-loss window and recovery duration remain within the intended RPO and RTO.
PostgreSQL streaming replication should therefore be understood as a continuous WAL transport and replay system rather than a complete HA solution by itself. It supplies synchronized standby database state, while production availability depends on surrounding mechanisms for monitoring, quorum or leader decisions, automated failover, service discovery, fencing, and recovery. This distinction provides the foundation for the next topic, where Patroni coordinates PostgreSQL replication nodes into an automated high-availability cluster.
10.03 PostgreSQL Patroni HA Cluster: Auto Failover [w/Code]
Patroni is an open-source high-availability orchestration framework designed to manage PostgreSQL clusters and automate leader election, replication control, and failover. PostgreSQL streaming replication provides synchronized database copies, but replication alone does not decide which server should become primary after a failure. Patroni adds this coordination layer, transforming multiple PostgreSQL instances into an operational HA cluster capable of responding automatically to node and service failures.
A typical Patroni architecture consists of several PostgreSQL nodes running Patroni agents together with a distributed configuration store or distributed consensus system. Patroni continuously observes PostgreSQL health and cluster membership while coordinating leadership through shared state. Instead of allowing each database server to independently decide whether it should become primary, the cluster uses coordinated leader information to maintain a single authoritative writable PostgreSQL instance.
Distributed consensus is essential because failover decisions must remain coordinated even when servers or network paths fail. Patroni commonly integrates with distributed systems such as etcd, Consul, or ZooKeeper, while deployments may also use supported alternatives depending on architecture. The distributed coordination layer stores cluster metadata and leader information. Patroni nodes compete for or renew leadership according to defined rules, allowing the cluster to distinguish the current leader from replicas.
The Patroni leader normally hosts the writable PostgreSQL primary. It periodically renews a leader lock or equivalent leadership state in the distributed configuration store. As long as the node remains healthy and can maintain this leadership information, other Patroni members continue operating as replicas. If the leader becomes unavailable or can no longer renew its leadership state within the configured timing conditions, the remaining members can begin an election and failover process.
PostgreSQL replicas under Patroni normally use streaming replication to receive WAL from the current primary. Patroni therefore does not replace PostgreSQL\'s native replication mechanism; instead, it manages and coordinates it. PostgreSQL remains responsible for WAL generation, transmission, and replay, while Patroni supervises node roles, replication configuration, promotion, demotion, and topology changes. This separation is important for understanding where database durability ends and HA orchestration begins.
Automatic failover begins when the current leader is considered unavailable according to Patroni\'s health and coordination logic. Eligible replicas are evaluated as candidates for promotion. Replication state matters because promoting a severely delayed replica could increase data loss. After a suitable replica obtains leadership, Patroni promotes its PostgreSQL instance from recovery mode into a writable primary and updates cluster state so that other members can recognize and follow the new leader.
Failover timing requires careful configuration. Patroni parameters governing the control loop, leader lock lifetime, retry behavior, and failover decisions determine how rapidly the cluster reacts to failures. Aggressive timing may reduce service interruption but can make the system more sensitive to temporary network instability or overloaded infrastructure. Conservative timing can reduce unnecessary role changes but increases recovery time. These settings should therefore be aligned with the database service\'s Recovery Time Objective (RTO).
Recovery Point Objective (RPO) depends strongly on PostgreSQL replication mode rather than Patroni alone. With asynchronous replication, transactions acknowledged by the failed primary may not yet exist on the promoted replica. Patroni can automate promotion, but it cannot reconstruct WAL that never reached another server. Synchronous replication can reduce this exposure by requiring selected replicas to confirm WAL progress before commits complete, at the cost of additional transaction latency and availability considerations.
Candidate selection is consequently more than simply choosing any running replica. A production HA design should consider replication lag, timeline state, node health, topology, and whether a member is intended to participate in promotion. Patroni provides mechanisms for controlling member behavior so that particular nodes can serve specialized roles. For example, a remote disaster-recovery replica or delayed replica may be valuable for recovery while being inappropriate as an automatic failover candidate.
Split-brain prevention is one of the most important responsibilities of a PostgreSQL HA architecture. If two database nodes simultaneously accept writes as primary, their histories can diverge and automated reconciliation becomes difficult. Patroni uses coordinated leadership to reduce this risk, but reliable operation also depends on sound network design and operational controls. In critical environments, fencing mechanisms may be added to ensure that an isolated former primary cannot continue accepting application writes.
When a failed former primary returns, it cannot automatically assume its previous role because another server may already have accepted new transactions as the current primary. Its database timeline may have diverged from the cluster. Patroni can coordinate reinitialization or use PostgreSQL recovery mechanisms such as pg_rewind where appropriate. The returning node must be synchronized with the current leader before it can safely rejoin the cluster as a replica.
Patroni exposes a REST API that supports health checks, cluster inspection, and administrative operations. Load balancers, proxies, monitoring platforms, or service-discovery mechanisms can use node state to determine which PostgreSQL instance currently serves the primary role and which instances are replicas. This interface helps separate database role management from client routing, allowing infrastructure components to direct write traffic toward the leader and, when appropriate, read traffic toward replicas.
Client routing remains a separate architectural concern because Patroni does not automatically make every application connection follow a newly promoted primary. Production environments commonly place a routing or proxy layer in front of PostgreSQL or integrate service discovery with application connection logic. Components such as HAProxy can use Patroni health information to identify the writable leader. Connection pools must also discard failed sessions and reconnect applications after failover.
Patroni supports controlled administrative operations in addition to unexpected automatic failover. A planned switchover intentionally transfers leadership from a healthy primary to an eligible replica, which is useful during maintenance, operating-system updates, hardware replacement, or infrastructure migration. Unlike emergency failover, switchover can be coordinated while the existing primary remains available, allowing administrators to reduce disruption and verify replica readiness before changing database leadership.
Cluster topology should account for failure domains rather than merely counting database servers. Three nodes located on the same physical host, storage system, power domain, or network switch may not provide meaningful resilience against infrastructure failure. PostgreSQL and Patroni nodes should be distributed according to the failures the system is expected to survive. Coordination services must likewise remain available because loss of distributed consensus can affect the cluster\'s ability to safely perform leadership changes.
Monitoring a Patroni cluster requires observing both PostgreSQL replication health and orchestration state. Important information includes the current leader, member availability, replication lag, WAL positions, timeline changes, leader-lock status, failover history, PostgreSQL process health, storage capacity, connection load, and distributed coordination service health. Monitoring only PostgreSQL can miss orchestration failures, while monitoring only Patroni can overlook database-level degradation that makes a replica unsuitable for promotion.
Operational testing should deliberately reproduce realistic failure scenarios. Tests can terminate PostgreSQL, stop Patroni, isolate a node from the network, interrupt the coordination service, introduce replication lag, or simulate host loss. Engineers should measure detection time, leader election, promotion time, application reconnection, transaction behavior, and replica recovery. The objective is not simply to demonstrate that failover occurs, but to confirm that the entire database service returns within its intended RTO and RPO.
For robot fleet platforms, Patroni can protect centralized operational databases containing mission assignments, robot status, configuration, authorization, maintenance records, and fleet coordination information. If the active database server fails, automatic promotion can restore centralized data services without waiting for manual database intervention. Robots should nevertheless preserve essential local autonomy so that collision avoidance, safety control, and short-term execution do not depend entirely on continuous database availability.
A practical robot deployment may place PostgreSQL and Patroni nodes across independent edge or on-premise servers while keeping low-level robot control outside the HA database path. The database cluster maintains persistent fleet-level information, whereas robots and edge controllers continue safety-critical operations locally. After a database failover, applications reconnect to the new primary and synchronize operational state, limiting the effect of infrastructure failure on physical robot behavior.
Security must also cover the entire HA control plane. PostgreSQL authentication, replication credentials, Patroni API access, coordination-service permissions, network segmentation, encryption, and secret management should be treated as related controls. An attacker capable of manipulating leadership information or administrative APIs could affect database availability even without directly modifying application tables. HA infrastructure therefore requires security controls comparable to those protecting the database itself.
Patroni should ultimately be understood as the orchestration layer surrounding PostgreSQL streaming replication rather than as a replacement for PostgreSQL replication. PostgreSQL maintains replicated data through WAL, while Patroni determines roles, coordinates leadership, automates promotion, and manages cluster membership. Combined with distributed coordination, client routing, monitoring, fencing, backups, and tested recovery procedures, it provides a foundation for automated PostgreSQL high availability in robot and Physical AI platforms.
10.04 MySQL / MariaDB Group Replication [w/Code]
MySQL Group Replication is a distributed replication technology designed to maintain a group of MySQL servers that coordinate database changes and provide fault tolerance. Instead of depending only on a traditional primary-to-replica replication chain, members participate in a replication group and exchange transaction information through a group communication mechanism. This architecture can support automated membership management, failure detection, transaction coordination, and high-availability database services.
Group Replication operates by combining MySQL transaction processing with distributed agreement among participating members. Transactions are first executed locally, but before they are finally committed, information about their writes is distributed to the group for certification. Members evaluate whether concurrently executed transactions conflict with one another. Transactions that successfully pass certification can be applied consistently, while conflicting transactions may be rejected and rolled back.
The certification process is a fundamental difference from simple asynchronous replication. Group members examine transaction write sets representing the rows affected by transactions and use them to identify conflicting updates. This approach enables multiple servers to coordinate transaction ordering without simply replaying an unconstrained stream of independent writes. Applications must nevertheless be prepared for transactions to fail certification and should implement appropriate retry behavior.
MySQL Group Replication can operate in single-primary mode or multi-primary mode. Single-primary mode designates one member as the writable primary while the remaining members normally serve as read-only replicas. This model simplifies application behavior because writes have a clear destination. If the primary leaves the group or fails, another eligible member can be selected, allowing the database service to continue with a new primary.
Multi-primary mode allows multiple members to accept transactions concurrently. This can distribute write activity and remove the requirement that every application write target a single server. However, concurrent modifications to overlapping data can create certification conflicts. Multi-primary deployment therefore requires careful workload design, transaction retry handling, application awareness, and avoidance of write patterns that repeatedly modify the same records across different members.
Group membership is managed dynamically as servers join, leave, fail, or recover. Members exchange information through the group communication layer and maintain a consistent view of participating nodes. Failure detection allows remaining members to update their membership view when a server becomes unreachable. This group-oriented model provides the coordination required for automated HA behavior, but reliable network connectivity and appropriate timeout configuration remain important operational requirements.
Quorum is essential to maintaining safe group operation. A replication group must retain a majority of members to make progress safely when communication failures divide the cluster. For example, a three-member deployment can generally tolerate the loss of one member while retaining a majority. If a network partition isolates nodes without a majority, those nodes should not independently continue normal group decisions, helping reduce the possibility of divergent database histories.
Odd-sized groups are commonly useful because they provide clear majority decisions without adding unnecessary voting ambiguity. Three members are a practical minimum for tolerating one member failure while maintaining majority availability, while larger groups can tolerate additional failures depending on topology. Simply adding members does not automatically improve performance because each member participates in communication, certification, replication, and state maintenance, increasing coordination overhead.
MySQL Group Replication relies on replicated transaction information rather than shared storage. Each participating server maintains its own database copy and local storage subsystem. This eliminates a single shared database disk as the fundamental replication dependency, but consistency also depends on each member correctly receiving and applying replicated transactions. Storage latency, CPU pressure, network delay, and transaction volume can therefore influence replication throughput and member health.
A joining or recovering member must obtain the database state required to become consistent with the group. MySQL can use recovery mechanisms to transfer transactions that the member missed while unavailable, and appropriate provisioning may be required when the difference is too large or necessary history is unavailable. Administrators should monitor recovery progress because a server listed in the topology is not necessarily ready to carry production traffic immediately after starting.
Binary logging remains an important part of the replication architecture. Group Replication depends on transaction metadata and configuration that allow changes to be identified and replicated consistently. Server identifiers, global transaction identifiers, row-based logging, and other replication-related settings must be correctly configured. Inconsistent configuration among members can prevent nodes from joining the group or produce operational behavior different from the intended HA design.
Global Transaction Identifiers (GTIDs) provide unique identities for replicated transactions and help servers track which transactions have already been executed. This is valuable when members leave and later rejoin because replication recovery can determine which transaction history is missing. GTID-based tracking also simplifies many operational tasks compared with relying only on binary-log file names and positions, particularly when database roles change during failover.
MySQL InnoDB Cluster builds a broader HA solution around Group Replication. Group Replication supplies distributed database membership and transaction replication, while MySQL Router can direct application connections toward appropriate cluster members. MySQL Shell provides administration and cluster-management capabilities. Together, these components can create an integrated architecture in which replication, topology management, routing, and administrative automation cooperate rather than being independently assembled.
MySQL Router is especially important when the writable member changes. Applications should not require manually updated database addresses whenever a failover occurs. A routing layer can maintain knowledge of the cluster topology and direct read-write connections toward the appropriate primary while optionally routing read-only traffic toward replicas. Existing application connections may still fail during role changes, so connection pools and applications need reconnect and retry logic.
MariaDB provides related high-availability approaches, but its replication architecture should not be assumed to be identical to MySQL Group Replication. MariaDB deployments commonly use technologies such as Galera Cluster for virtually synchronous multi-primary clustering, alongside traditional replication options. Although both ecosystems address database redundancy and availability, their protocols, configuration, transaction behavior, failure handling, and operational tooling differ and should be evaluated independently.
Galera-based MariaDB clustering replicates transactional changes across cluster nodes while using certification-based mechanisms to coordinate concurrent writes. Multiple nodes can accept application traffic, but this does not mean that conflicting writes become free of coordination cost. Workloads with frequent concurrent updates to the same records can experience certification conflicts or rollbacks, making application transaction design and retry behavior important for stable multi-primary operation.
Network latency has a stronger architectural effect in coordinated replication than in loosely coupled asynchronous replicas. Transactions that require communication or certification across members are affected by communication delay and cluster health. Nodes should therefore be placed according to realistic latency and failure-domain requirements. Spreading replicas across independent infrastructure improves resilience, but excessive geographic distance can directly affect transaction performance and recovery behavior.
Database consistency should be considered together with availability requirements. A robot fleet database may contain mission ownership, configuration, authorization, maintenance state, and scheduling information for which conflicting concurrent updates are undesirable. Single-primary operation may therefore provide a simpler operational model for many transactional workloads. Multi-primary operation can be appropriate when the data model and workload are deliberately designed to minimize conflicting writes.
Robot systems should also separate database HA from real-time robot safety. A MySQL or MariaDB cluster can maintain fleet-level operational state, business records, user information, maintenance history, and mission metadata, but low-level motion control and collision avoidance should not require continuous access to the central database. Edge controllers and robots can preserve essential local operation while the database cluster detects failures, changes membership, and restores centralized service.
Monitoring should cover both database performance and distributed cluster state. Important observations include member role, group membership, transaction throughput, certification conflicts, replication queues, recovery progress, connection errors, storage utilization, CPU load, network latency, and failed members. A cluster can appear operational while one member is falling behind or repeatedly leaving and rejoining, so HA monitoring must evaluate redundancy quality rather than only whether a database endpoint responds.
Failure testing should validate member loss, primary failure, network partition, routing changes, node recovery, and transaction behavior during topology transitions. Engineers should measure failure detection, election or role-change time, application reconnection, transaction rollback, and data consistency after recovery. Testing should also verify whether the remaining group retains quorum and whether isolated nodes behave safely instead of continuing as independent writable database services.
For production robotics platforms, the appropriate architecture depends on workload, consistency requirements, latency, and operational complexity. MySQL Group Replication with single-primary operation provides coordinated redundancy with a relatively clear write model, while multi-primary designs offer greater write distribution at the cost of conflict management. MariaDB clustering offers related alternatives with different implementation characteristics. Both require monitoring, routing, backup, recovery testing, and application-aware retry mechanisms.
Group replication should therefore be understood as more than copying data between database servers. It combines replication with membership management, transaction certification, distributed coordination, and failure handling to maintain a coherent database service across multiple nodes. When integrated with routing, monitoring, appropriate quorum design, backups, and tested recovery procedures, MySQL or MariaDB clustering can provide a resilient database foundation for robot fleet and Physical AI services.
10.05 Redis Sentinel Cluster Auto Failover [w/Code]
Redis Sentinel is a high-availability mechanism for Redis deployments that monitors primary and replica instances, detects failures, coordinates automatic failover, and informs clients about the current primary. It is designed for replicated Redis environments where service continuity is required without introducing Redis Cluster sharding. Sentinel therefore focuses on availability and role management, while Redis replication continues to provide copies of the primary dataset across replica nodes.
A typical Sentinel deployment contains one Redis primary, one or more Redis replicas, and multiple independent Sentinel processes. The primary accepts normal write operations and propagates changes to replicas through Redis replication. Sentinel processes continuously observe these Redis instances and communicate with one another. By separating monitoring and failover coordination from the data nodes, Sentinel can make collective decisions about whether a primary is actually unavailable.
Redis replication is normally asynchronous, meaning the primary can acknowledge writes before replicas confirm that they have received them. This design provides low write latency and high throughput, but it also means recently acknowledged data can be lost if the primary fails before its latest changes reach a replica. Sentinel can automate recovery from primary failure, but automatic failover does not by itself guarantee zero data loss because durability remains dependent on replication progress.
Sentinel continuously performs health checks against monitored Redis servers and exchanges information with other Sentinel instances. A temporary connection failure observed by one Sentinel should not immediately trigger a cluster-wide role change. Instead, Sentinel distinguishes an individual observer\'s assessment from a collectively confirmed failure. This prevents many short-lived network or process disturbances from unnecessarily promoting a replica and changing the database topology.
A subjectively down condition, commonly called SDOWN, occurs when one Sentinel concludes that a monitored instance has failed to respond within the configured timeout. For a primary, this local judgment is not sufficient to initiate automatic failover. Sentinel communicates with peer Sentinels and determines whether enough observers also consider the primary unavailable. When the required quorum agrees, the primary can be classified as objectively down, or ODOWN.
Quorum is therefore central to Sentinel\'s failure-detection model. It specifies how many Sentinel processes must agree that a primary is unavailable before objective failure is established. Quorum should not be confused with every condition required to authorize a failover, because Sentinel also uses leader-election logic among available Sentinels. A sound deployment needs enough independent Sentinel processes to make reliable decisions when individual monitoring processes or network paths fail.
Deploying three Sentinel processes is a common architecture because it allows majority-based coordination while tolerating the loss of one Sentinel under appropriate conditions. These processes should ideally be distributed across independent hosts or failure domains rather than placed together on one machine. Running several Sentinels on a single host increases process redundancy but provides little protection if that host, power source, operating system, or network path becomes unavailable.
When a primary is confirmed unavailable, the Sentinel processes coordinate to determine which Sentinel will lead the failover operation. The elected Sentinel evaluates available replicas and selects a suitable candidate for promotion. Candidate selection considers replication-related information and replica suitability rather than randomly choosing any reachable node. The selected replica is then instructed to become the new writable primary.
After promotion, the failover process must reorganize the remaining replication topology. Other replicas are directed to replicate from the new primary rather than the failed server. If the former primary later returns, Sentinel can reconfigure it to follow the current primary as a replica instead of allowing it to automatically resume its old writable role. This behavior helps restore a consistent primary-replica topology after a server failure.
Replica freshness is important because the most useful failover candidate is generally one that has received as much of the primary\'s recent replication stream as possible. A replica that has been disconnected for a long period may be operational but contain stale data. Sentinel\'s failover process therefore needs replication state and configuration information when evaluating candidates. Operational monitoring should likewise track replica connectivity and replication delay before an emergency occurs.
Sentinel supports replica priority settings that influence whether a replica should be considered for promotion and how candidates are ranked. This allows administrators to reflect infrastructure design in failover policy. For example, a well-provisioned local replica may be preferred over a remote recovery node. A replica can also be configured so that it remains available for replication purposes without becoming an automatic promotion candidate when its role or location makes promotion undesirable.
Client discovery is another important Sentinel function. Applications should not permanently assume that one fixed Redis address will always represent the primary because automatic failover can move the writable role to another server. Sentinel-aware clients can query Sentinel to discover the address of the current primary. After failover, clients reconnect using the updated topology, allowing the Redis service to continue without manually changing every application\'s configuration.
Existing client connections do not automatically survive a server failure simply because Sentinel promotes another replica. Applications and connection pools must recognize broken connections, rediscover the current primary, reconnect, and safely retry operations where appropriate. Retry behavior should consider whether an operation may already have been executed before the connection failed. High availability therefore requires cooperation between Sentinel, Redis replication, client libraries, and application logic.
Sentinel itself does not provide data sharding. This distinction is important because Redis Sentinel and Redis Cluster solve different architectural problems. Sentinel manages high availability for primary-replica deployments, while Redis Cluster partitions the keyspace across multiple masters and can provide replication and failover within a sharded topology. Choosing between them depends on whether the primary requirement is availability for one logical dataset or horizontal distribution of data and throughput.
Persistence configuration also remains independent of Sentinel failover. Redis can use Redis Database files (RDB), Append Only File (AOF), or combinations of persistence mechanisms depending on durability and recovery requirements. Sentinel does not replace these mechanisms and should not be treated as a backup system. Replication may reproduce unwanted modifications or deletions across replicas, so durable persistence and independent backups remain necessary when historical recovery is required.
Network partitions create one of the most difficult HA scenarios. A primary may continue running while becoming unreachable from enough Sentinel processes to trigger failover elsewhere. If clients can still reach the isolated old primary, writes could occur while another replica is promoted. Deployment design should therefore consider network topology, client routing, Redis configuration, and operational safeguards that limit the possibility and impact of divergent writes during partitions.
Failover parameters must balance recovery speed against stability. A very short failure-detection timeout may allow rapid reaction to genuine outages but can make the system sensitive to temporary latency, CPU stalls, or network congestion. Longer thresholds reduce false failure detection but increase service interruption before promotion begins. These values should be selected according to measured infrastructure behavior and the Recovery Time Objective rather than copied unchanged between environments.
Recovery Point Objective is strongly affected by asynchronous replication. Sentinel can reduce the time required to restore a writable Redis service, but it cannot guarantee that every acknowledged write exists on the promoted replica. Applications storing critical robot state should understand this limitation. If a value cannot tolerate loss or reconstruction, Redis may need to be combined with a more durable system of record rather than being treated as the sole authoritative storage layer.
For robotics, Sentinel is particularly suitable for Redis services used as fast operational state stores, caches, event coordination layers, session stores, or short-lived fleet information. Robot status summaries, task queues, temporary mission context, coordination flags, and rapidly accessed configuration data may benefit from Redis performance and automatic failover. The exact suitability depends on whether the application can tolerate Redis replication and persistence characteristics.
Safety-critical robot behavior should remain separated from Sentinel-controlled infrastructure. Motion control, emergency stopping, collision avoidance, and immediate actuator decisions should not depend on successful communication with a centralized Redis primary. Robots and edge controllers should preserve essential local autonomy while centralized services recover. After failover, fleet applications can reconnect to the new Redis primary and resume higher-level coordination without compromising local safety functions.
Monitoring should include both Redis data nodes and Sentinel processes. Important observations include primary and replica roles, replica connectivity, replication offsets, replication delay, Sentinel reachability, SDOWN and ODOWN events, quorum state, leader elections, failover duration, memory utilization, persistence health, client connections, and network latency. Monitoring only the current primary can hide degraded redundancy until the moment a real failure occurs.
Failure testing should intentionally stop the primary, disconnect replicas, terminate Sentinel processes, create network partitions, and restore failed nodes. Engineers should measure detection time, quorum formation, leader election, replica promotion, client rediscovery, application reconnection, and topology reconstruction. Data consistency after failover should also be examined so that the actual recovery behavior and potential loss window are understood before the system is deployed in production.
Redis Sentinel should ultimately be understood as an HA coordination layer built around Redis primary-replica replication. Replication maintains redundant data copies, while Sentinel provides monitoring, collective failure detection, leader election, replica promotion, topology reconfiguration, and primary discovery. Combined with persistence, backups, resilient clients, observability, failure-domain separation, and tested recovery procedures, it can provide a practical high-availability Redis service for robot fleet and Physical AI platforms.
10.06 Elasticsearch Cluster Shard Replication Design [w/Code]
Elasticsearch cluster architecture is designed around distributed storage, parallel search, and fault tolerance by dividing index data into shards and distributing those shards across multiple nodes. An index is not stored as one monolithic structure. Instead, Elasticsearch partitions it into primary shards and can maintain replica shards for redundancy. This distributed model allows search and indexing workloads to scale beyond the capacity of a single server while reducing the impact of individual node failures.
A shard is the fundamental unit of data distribution in Elasticsearch and internally represents an independent Lucene index. Documents assigned to an Elasticsearch index are distributed among its primary shards according to Elasticsearch routing logic. Because each shard can reside on a different node, indexing and search operations can execute across multiple machines in parallel. Shard design therefore directly influences scalability, resource utilization, recovery behavior, and operational complexity.
Primary shards define the original partitions of an index. When an index is created, its configured number of primary shards determines how the document space is partitioned. Documents are routed to specific primary shards, which become responsible for accepting the corresponding indexing operations. The number and expected size of primary shards should therefore reflect anticipated data volume, growth, node capacity, and workload characteristics rather than being selected arbitrarily.
Replica shards are copies of primary shards maintained on other eligible nodes. Their first purpose is availability: if the node containing a primary shard fails, an available replica can be promoted so that the shard remains accessible. Replicas can also contribute to search throughput because read operations may be served by either primary or replica copies. Replication therefore supports both fault tolerance and horizontal distribution of search workloads.
Elasticsearch avoids placing a replica shard on the same node as its corresponding primary shard because doing so would provide little protection against node failure. In a healthy multi-node cluster, primary and replica copies are distributed across different nodes according to shard allocation rules. If one node becomes unavailable, the cluster can continue using surviving shard copies while reallocating missing replicas to restore the intended redundancy level.
When a document is indexed, the coordinating node determines the target shard from the document\'s routing information and forwards the operation to the node containing the relevant primary shard. The primary processes the operation and coordinates replication to its replica copies. This primary-replica relationship establishes a consistent write path for each shard while allowing the overall index to remain distributed across many nodes.
Search operations follow a different distributed pattern. A coordinating node identifies the shards that must participate in the query and sends requests to suitable shard copies across the cluster. Each participating shard performs local search processing and returns partial results. The coordinating node then merges these results into the final response. Elasticsearch can therefore parallelize searches across shards, although excessive shard counts can also create coordination and resource overhead.
Cluster nodes can perform different responsibilities depending on their configured roles. Master-eligible nodes participate in cluster coordination and management of cluster metadata, while data nodes store shards and execute data operations. Other specialized roles can support ingest processing, machine learning, transforms, or coordinating workloads. Separating roles can become useful as deployments grow because cluster management, indexing, searching, and specialized processing place different demands on infrastructure.
Cluster state contains essential metadata describing indices, mappings, shard allocation, node membership, and other configuration required for coordinated operation. The elected master node manages changes to this state and distributes updated information to the cluster. Master responsibilities should therefore be distinguished from ordinary document processing. A stable cluster-management layer is necessary for safely responding to node membership changes, index creation, shard allocation, and recovery events.
Node failure triggers shard availability and recovery mechanisms. When a node disappears, Elasticsearch detects the membership change and evaluates the shards that were hosted on the failed node. If a lost primary has an available replica, a surviving replica can be promoted to primary. Elasticsearch can then create additional replica copies on suitable nodes to return the index to its configured redundancy level once sufficient resources are available.
Cluster health is commonly summarized as green, yellow, or red. Green indicates that primary and replica shards are allocated as intended. Yellow means all primary shards are available but at least one replica shard is unassigned, so data remains accessible while redundancy is reduced. Red indicates that one or more primary shards are unavailable, which means some index data cannot currently be accessed. These states provide a concise view of shard availability rather than complete system health.
Shard allocation determines where shard copies are placed and can incorporate node availability, disk conditions, allocation awareness, and configured constraints. Production deployments can use awareness attributes to distribute copies across failure domains such as hosts, racks, or availability zones. The objective is not merely to spread shards evenly, but to avoid placing redundant copies where a single infrastructure failure could simultaneously remove both the primary and its replicas.
Replication does not replace backup. Replica shards protect against node and storage failures by maintaining additional active copies, but accidental deletion, incorrect updates, or application errors can propagate through the cluster. Elasticsearch snapshots provide a separate recovery mechanism for preserving index data outside the active shard topology. High-availability design should therefore combine shard replication with snapshot policies and tested restore procedures.
Shard sizing is one of the most important design decisions in Elasticsearch. Too few very large shards can limit parallelism and make recovery operations lengthy, while too many small shards consume memory, file handles, CPU time, and cluster-management resources. The appropriate balance depends on document size, query patterns, indexing rate, retention period, hardware capacity, and recovery objectives. Shard strategy should consequently be measured and adjusted using realistic workloads.
Oversharding can become especially problematic when many indices each contain numerous small shards. Every shard introduces Lucene structures and operational overhead even when it stores relatively little data. A cluster with excessive shard counts may experience increased heap pressure, slower cluster-state operations, longer recovery, and inefficient searches. Index lifecycle and data organization should therefore consider shard count as an infrastructure resource rather than treating shards as cost-free logical partitions.
Time-oriented data is often organized into multiple indices according to operational periods or lifecycle policies. Logs, events, telemetry, and robot observations may continuously accumulate, making lifecycle management important for controlling storage growth. Elasticsearch data streams and index lifecycle mechanisms can support rollover, retention, movement between storage tiers, and deletion according to policy, allowing shard organization to evolve as data ages.
Recovery traffic must be considered when sizing the cluster. After a node failure, Elasticsearch may need to copy large shard datasets across the network to recreate missing replicas. This recovery consumes network bandwidth, storage I/O, CPU resources, and time while normal search and indexing workloads continue. A cluster designed only for steady-state utilization may therefore perform poorly during failure recovery. Operational headroom is necessary to restore redundancy without overwhelming surviving nodes.
Replica count represents a trade-off among availability, search capacity, storage consumption, and indexing cost. Additional replicas create more surviving copies and can increase read capacity, but every indexed change must also be replicated and each copy consumes storage. Replica configuration should therefore correspond to failure tolerance and workload requirements. Increasing replicas without considering node count and failure domains may provide less resilience than expected.
For robot and Physical AI platforms, Elasticsearch can store and search structured operational events, robot logs, mission histories, diagnostic messages, alerts, semantic metadata, and searchable telemetry. Sharding allows these datasets to expand across multiple data nodes, while replicas maintain availability when individual servers fail. Elasticsearch is particularly valuable when operators need rapid filtering, full-text search, aggregation, and investigation across large volumes of machine-generated operational information.
Raw camera streams, dense point clouds, and large training datasets should not automatically be placed directly into Elasticsearch simply because they originate from robots. Elasticsearch is generally more appropriate for searchable metadata, events, derived features, references, and indexed operational records. Large binary objects can remain in object or file storage while Elasticsearch maintains identifiers, timestamps, robot metadata, mission context, semantic labels, and locations needed for efficient discovery.
A robot fleet deployment can distribute Elasticsearch nodes across independent infrastructure while applications access the cluster through stable service endpoints. If one data node fails, surviving shard copies can continue serving accessible data while the cluster reconstructs redundancy. Fleet dashboards, diagnostic systems, mission analysis services, and observability platforms can therefore remain available despite individual server failures, provided shard allocation and capacity have been designed for the expected failure scenarios.
Monitoring should extend beyond the simple green, yellow, and red health indicator. Important observations include shard allocation, unassigned shards, node availability, indexing throughput, search latency, JVM heap usage, garbage collection, disk utilization, disk watermarks, recovery activity, thread pools, query load, cluster-state stability, and snapshot status. A green cluster can still suffer severe performance problems, so availability and performance must be monitored together.
Failure testing should remove data nodes, interrupt network paths, exhaust controlled storage capacity, and restore nodes while observing shard promotion and reallocation. Engineers should measure detection time, search availability, indexing behavior, recovery throughput, replica reconstruction, and the time required to return to full redundancy. Snapshot restoration should also be tested independently because successful shard replication does not prove that disaster recovery procedures will work.
Elasticsearch shard replication should ultimately be understood as a distributed data-placement and availability architecture rather than merely a mechanism for maintaining duplicate files. Primary shards partition the index, replicas provide redundant searchable copies, cluster coordination manages placement and promotion, and recovery reconstructs lost redundancy. Combined with deliberate shard sizing, failure-domain awareness, monitoring, snapshots, lifecycle management, and capacity headroom, this design can provide a scalable and resilient search foundation for robot fleet and Physical AI platforms.
10.07 Geo-Distributed Replication: Multi-Region Design
Geo-distributed replication extends database high availability beyond a single data center or availability zone by maintaining database copies across geographically separated regions. Its objective is to preserve service continuity when an entire site becomes unavailable because of power, network, infrastructure, or regional cloud failures. Unlike local replication, multi-region design must account for wide-area latency, limited bandwidth, network partitions, regulatory boundaries, and substantially longer recovery paths.
A multi-region architecture normally separates local high availability from geographic disaster recovery. Within each region, multiple database nodes can provide rapid failover for ordinary server or rack failures. Replication between regions then protects against larger site-level failures. This layered design prevents every small local incident from becoming a cross-region event while maintaining a remote database copy that can assume service when an entire regional deployment is lost.
Network latency is the dominant constraint in geographically distributed replication. Communication between nearby servers may require only a small delay, while communication between distant regions can add tens or hundreds of milliseconds. If every transaction must wait for confirmation from another region, application write latency increases accordingly. Database architecture must therefore decide which operations require cross-region durability and which can complete using only local acknowledgements.
Asynchronous cross-region replication is widely used because the primary region can commit transactions without waiting for a remote replica. This keeps application latency close to local database performance and allows geographically distant replicas to remain practical. The trade-off is a replication lag window. If the primary region fails suddenly, transactions that were committed locally but had not yet reached the remote region may be unavailable after regional failover.
Synchronous or strongly coordinated replication can reduce the potential data-loss window by requiring remote acknowledgement before completing selected writes. However, geographic network delay becomes part of transaction latency, and loss of connectivity between regions can directly affect write availability. Strong cross-region consistency is therefore not automatically preferable. It must be justified by the value of the protected data and the application\'s tolerance for latency and temporary unavailability.
Recovery Point Objective defines how much recent data loss can be tolerated after a regional disaster, while Recovery Time Objective defines how long the service can remain unavailable before recovery. These objectives should drive replication topology rather than being documented after implementation. A low RPO may require tighter replication guarantees, while a low RTO requires prepared infrastructure, automated promotion, traffic redirection, validated configuration, and continuously monitored replica readiness.
An active-passive multi-region design normally directs writes to one primary region while another region maintains a replicated standby environment. This model provides a clear authority for updates and simplifies conflict management. During regional failure, the standby is promoted and application traffic is redirected. The design is operationally straightforward, but failover procedures must verify replica freshness and prevent the failed region from unexpectedly returning as an independent writable database.
Active-active architecture allows applications in multiple regions to access writable database services closer to their location. This can reduce regional access latency and improve geographic service availability, but concurrent writes introduce significantly more complex consistency requirements. If two regions modify the same logical data while communication is delayed or interrupted, the system requires conflict detection, deterministic resolution, ownership rules, or a data model designed to avoid conflicting updates.
Data ownership can simplify multi-region operation. Instead of allowing every region to update every record, applications can assign ownership according to robot fleet, customer, facility, geographic area, or another stable partition. Most writes then remain within the owning region while selected information is replicated globally. This approach can reduce cross-region conflicts and coordination traffic while still supporting geographically distributed services and consolidated analytics.
Network partitions require explicit design because both regions may remain operational while losing communication with one another. Allowing every isolated region to continue unrestricted writes can produce divergent histories that are difficult to reconcile. Systems may instead preserve a single writable authority, use quorum-based decisions, restrict selected operations, or temporarily operate with reduced functionality. Partition behavior must be defined before deployment rather than improvised during an outage.
Failover begins with reliable failure detection, but detecting a regional outage is more difficult than detecting a failed database process. Monitoring must distinguish database failure from local routing problems, inter-region network interruption, control-plane failure, and application connectivity issues. Incorrectly declaring a healthy region unavailable can trigger an unnecessary regional failover and create two competing writable environments, making conservative coordination and fencing important.
Fencing prevents a failed or isolated primary region from continuing to accept authoritative writes after another region has been promoted. Techniques depend on infrastructure and may involve database controls, network isolation, service discovery, load balancers, orchestration systems, or access-policy changes. The essential objective is to maintain one clearly recognized write authority whenever the database architecture requires single-primary semantics and thereby reduce split-brain risk.
Traffic management is part of database disaster recovery even though it may exist outside the database itself. DNS, global load balancers, service discovery, application gateways, or orchestration platforms can redirect clients toward the surviving region. Traffic redirection must be coordinated with database promotion because sending writes to a standby before it is ready can cause failures, while continuing to route traffic to an unavailable region extends the outage.
Read locality can improve performance without requiring every region to become writable. Regional replicas can serve dashboards, search operations, reporting, or other workloads that tolerate replication delay while authoritative writes remain in one primary region. Applications must understand that remote replica reads may be stale. Features requiring immediate read-after-write behavior should therefore access an appropriate authoritative path or use database mechanisms that provide the required consistency.
Replication bandwidth must be included in capacity planning. Robot fleets can generate continuous mission events, telemetry, logs, configuration changes, and operational records, producing sustained replication traffic between sites. Large bursts after temporary disconnection may create additional catch-up load. Network links must therefore support both normal replication throughput and recovery conditions without allowing replication queues to grow indefinitely during peak operation.
Initial synchronization and regional rebuilds can transfer much more data than ordinary incremental replication. Creating a new replica may require a full database copy, snapshot restoration, or bulk synchronization before continuous replication can resume. For large databases, transferring this state across a wide-area network may take considerable time and bandwidth. Seed copies, snapshots, object storage, or staged provisioning can therefore become important parts of multi-region recovery planning.
Data placement must also consider regulatory and organizational boundaries. Some information may need to remain within a country, customer environment, factory, or controlled security domain even when the application operates globally. Multi-region replication should therefore classify data according to residency, confidentiality, durability, and sharing requirements. Replicating every table or record everywhere can increase regulatory exposure, network cost, and operational complexity without providing useful resilience.
Backups remain necessary even when several geographic replicas exist. Replication protects service availability by maintaining operational copies, but corruption, accidental deletion, incorrect application updates, or malicious changes can propagate between regions. Independent backups and point-in-time recovery provide a historical recovery path outside the live replication topology. Backup copies should also be protected from the same credentials and failure domains as the production database where practical.
Monitoring must provide both regional and global visibility. Engineers should observe database health, replication lag, transaction throughput, replication queues, network latency, bandwidth, replica freshness, regional role, failover readiness, storage capacity, backup status, and routing state. A remote replica that is technically running but several hours behind does not provide the recovery capability assumed by the architecture, making measurable readiness more important than simple node availability.
Multi-region failure testing should include complete regional isolation rather than only stopping a database process. Tests should examine loss of inter-region connectivity, primary-region shutdown, standby promotion, traffic redirection, application reconnection, fencing, replica recovery, and reintegration of the former primary. Engineers should measure actual RTO and RPO behavior and confirm that recovery procedures remain valid as database size, network conditions, and infrastructure configurations change.
For robot fleet and Physical AI platforms, geographic replication can protect mission records, robot configuration, maintenance history, fleet metadata, user authorization, scheduling information, and operational state shared across sites. However, robots should not require continuous access to a distant region for immediate motion decisions. Local edge systems should retain the autonomy necessary for safe operation while regional database services recover or temporarily lose inter-region connectivity.
A practical architecture may therefore combine robot-local autonomy, site-level edge databases, regional high-availability clusters, and asynchronous replication toward a secondary region or central platform. Data can move upward according to importance and timing requirements while essential control remains local. This hierarchy reduces dependence on wide-area connectivity and allows different consistency, latency, retention, and recovery policies to be applied to different classes of robot data.
Geo-distributed replication should ultimately be designed as a coordinated system of database replication, failure domains, network behavior, application consistency, traffic management, fencing, monitoring, and disaster recovery. Multi-region copies alone do not create resilience. When replication mode, data ownership, RPO/RTO, failover authority, backup strategy, and edge autonomy are deliberately aligned, the architecture can sustain regional failures while preserving dependable robot fleet and Physical AI services.
10.08 HA Configuration Monitoring: Prometheus / Alertmanager [w/Code]
High-availability monitoring must verify more than whether a database process is running. A healthy HA system requires correct node roles, reachable replicas, acceptable replication lag, valid quorum, sufficient storage, stable network paths, and functioning failover mechanisms. Prometheus provides time-series metric collection for these conditions, while Alertmanager transforms monitoring rules into controlled notifications that operators can use to respond before redundancy is lost.
Prometheus follows a pull-oriented monitoring model in which the server periodically scrapes HTTP metric endpoints exposed by monitored systems or exporters. Database servers commonly require dedicated exporters that translate internal statistics into Prometheus-compatible metrics. Infrastructure exporters can simultaneously expose CPU, memory, disk, filesystem, and network information, allowing database behavior to be correlated with the health of the underlying host.
A typical HA monitoring architecture includes database nodes, database-specific exporters, node-level exporters, Prometheus, alerting rules, Alertmanager, and visualization systems. Prometheus periodically collects metrics from every relevant component and stores them as labeled time series. Operators can then query current and historical behavior using PromQL, while dashboards provide a consolidated view of replication, availability, capacity, and failover readiness.
Labels are fundamental to organizing HA metrics. Metrics can identify cluster, database, instance, region, site, role, environment, or service so that the same monitoring rules can operate across many deployments. A metric indicating replication lag becomes substantially more useful when it can be grouped by primary, replica, cluster, and region. Label design should remain controlled because excessive high-cardinality labels can increase Prometheus resource consumption.
Database availability metrics should distinguish primary, replica, and unavailable states rather than reporting only process uptime. A running database process may be unable to accept connections, may have the wrong role, or may be isolated from its peers. HA monitoring should therefore verify service reachability, database role, replication state, cluster membership, leader status, and whether the expected number of healthy members is currently available.
Replication lag is one of the most important HA indicators because a replica that is running but significantly behind the primary may provide poor recovery capability. Monitoring should observe lag in time, bytes, log positions, transaction identifiers, or database-specific replication queues as appropriate. Alert thresholds should reflect the actual Recovery Point Objective and workload characteristics instead of treating every temporary increase in replication delay as a critical failure.
Failover readiness should be monitored before a failure occurs. A cluster can appear healthy while lacking a usable promotion candidate because replicas are disconnected, stale, misconfigured, or located in the same failure domain as the primary. Monitoring rules can evaluate replica count, synchronization state, quorum availability, leader-election components, replication health, and infrastructure placement so that degraded redundancy is detected while the primary is still operating.
Prometheus recording rules can precompute frequently used or computationally expensive expressions. Instead of repeatedly calculating complex HA conditions across many raw metrics, a recording rule can produce a simplified time series representing replica health, service availability, or cluster redundancy. This improves query consistency and can make dashboards and alert expressions easier to maintain, particularly across large robot-fleet infrastructure.
Alerting rules evaluate Prometheus expressions and create alerts when defined conditions are satisfied. Effective HA alerts should represent actionable operational states rather than every unusual metric movement. Examples include primary unavailable, no promotable replica, quorum lost, replication lag above an RPO-related threshold, storage approaching exhaustion, repeated failover, backup failure, or an unexpected change in cluster membership.
The for\ duration in a Prometheus alerting rule can require a condition to remain true for a specified period before an alert becomes active. This is useful for avoiding notifications caused by very short network delays, brief load spikes, or transient replication lag. However, excessive delay can hide genuine failures. The duration should therefore correspond to the urgency of the condition and the system\'s expected transient behavior.
Alertmanager receives alerts generated by Prometheus and handles grouping, routing, inhibition, silencing, and notification delivery. Grouping combines related alerts so that one infrastructure failure does not produce hundreds of independent notifications. Routing can direct database alerts to the appropriate operations team, while severity, environment, service, region, or cluster labels can determine how and where notifications are delivered.
Inhibition reduces redundant alert noise by suppressing secondary alerts when a higher-level failure already explains them. For example, if an entire database node is unavailable, alerts for every individual service or replica process on that node may add little operational value. Carefully designed inhibition relationships help operators focus on the most relevant failure while preserving lower-level metrics for diagnosis.
Silences provide temporary suppression of selected alerts during known maintenance or controlled testing. They should be targeted using labels and limited in duration so that unrelated failures remain visible. Permanent or excessively broad silences can conceal real incidents. Maintenance workflows should therefore define when silences begin, when they expire, and how monitoring coverage is restored after planned database switchovers or infrastructure changes.
Alert severity should reflect operational impact. A warning may indicate reduced redundancy, increasing replication lag, or declining disk capacity while the database service remains available. A critical alert should represent conditions requiring immediate action, such as loss of the writable primary without successful failover, loss of quorum, unavailable primary shards, or complete loss of a required database service. Severity should be tied to response procedures rather than visual preference.
Infrastructure metrics provide essential context for database HA. CPU saturation can delay replication processing, memory pressure can destabilize database processes, disk latency can slow transaction persistence, and filesystem exhaustion can stop writes entirely. Network packet loss or latency can disrupt consensus and replication even when every server is running. Database metrics and host metrics should therefore be analyzed together rather than as separate monitoring domains.
Capacity alerts should identify trends before hard limits are reached. Disk utilization is especially important because databases, transaction logs, snapshots, and replication backlogs can consume storage rapidly during abnormal conditions. Monitoring can combine current utilization with growth rate and operational thresholds. Capacity planning should also preserve enough free resources for failover and recovery, when surviving nodes may temporarily process additional workload.
Monitoring the monitoring system is itself necessary. Prometheus scrape failures, stale targets, missing exporters, Alertmanager failures, or notification delivery problems can create a false impression that the infrastructure is healthy. Operators should track scrape success, target availability, rule evaluation, Prometheus storage health, Alertmanager cluster state where applicable, and notification pipeline status so that loss of observability becomes an explicit incident.
HA dashboards should present operational relationships rather than isolated graphs. A useful cluster view can combine current primary, replica states, replication lag, transaction rate, connection count, disk utilization, host load, quorum state, recent failovers, and backup status. Regional or fleet-level dashboards can summarize many clusters and allow engineers to drill down from degraded service indicators to individual database nodes and infrastructure metrics.
Alert annotations should contain enough context to begin investigation without forcing an operator to reconstruct the problem from a metric name. Useful information can include affected cluster, instance, current role, observed value, threshold, duration, and a concise description of the expected response. Links to dashboards or operational runbooks can further reduce response time when they are maintained alongside the monitored system.
HA configuration and monitoring should be managed as version-controlled infrastructure where practical. Prometheus scrape configuration, recording rules, alerting rules, Alertmanager routing, dashboard definitions, and database HA parameters can be reviewed and tested through controlled changes. Configuration consistency is particularly important across multiple robot sites because manually maintained monitoring rules can drift and create different operational behavior for otherwise identical deployments.
Testing should deliberately trigger conditions that monitoring is expected to detect. Engineers can stop a primary database, disconnect a replica, increase replication delay, remove quorum members, fill a controlled test filesystem, or interrupt an exporter. The objective is not only to confirm that failover works but also to verify that the correct alerts appear, notifications reach the intended destination, dashboards reflect the event, and recovery clears the alerts correctly.
For robot fleet and Physical AI infrastructure, monitoring can cover PostgreSQL, MySQL or MariaDB, Redis, Elasticsearch, host systems, and supporting HA components through a common Prometheus-based observability layer. Each database exposes different internal signals, but labels and dashboards can normalize operational concepts such as primary availability, replica health, lag, quorum, capacity, and recovery readiness across heterogeneous storage technologies.
Robot-side safety should remain independent of centralized monitoring availability. Prometheus and Alertmanager can notify operators that fleet databases or regional services are degraded, but they are not real-time safety controllers. Robots and edge systems should preserve local motion safety, collision avoidance, and essential autonomous behavior while central database infrastructure is failing or recovering. Monitoring supports operational response rather than immediate physical control.
A mature HA monitoring design therefore connects technical measurements to recovery objectives and operational actions. Prometheus collects and evaluates database and infrastructure signals, while Alertmanager converts meaningful conditions into grouped, routed, and controlled notifications. Combined with dashboards, runbooks, version-controlled configuration, capacity monitoring, and repeated failure testing, this observability layer makes database redundancy measurable and operationally usable for robot fleet and Physical AI platforms.
10.09 HA Database Split-Brain Response
Split brain is one of the most dangerous failure conditions in a high-availability database because two or more nodes may simultaneously believe they are authorized to operate as the writable primary. Instead of producing a simple outage, the system can continue accepting transactions along separate paths. This can create divergent database histories, conflicting updates, duplicated operations, and uncertainty about which data should be treated as authoritative.
Split brain commonly begins with a communication failure rather than a complete server failure. A primary database may continue running while losing contact with replicas, cluster managers, or other regions. Another part of the HA system can interpret this isolation as primary failure and promote a replica. If the original primary remains reachable by some applications, both sides may accept writes independently even though they can no longer coordinate.
Network partitions are therefore a major source of split-brain risk. A partition divides cluster members into groups that remain internally functional but cannot communicate reliably with one another. From the perspective of one partition, the nodes on the other side may appear failed even though they are still running. HA architecture must distinguish loss of communication from confirmed loss of authority and define which partition is permitted to continue processing writes.
Quorum mechanisms reduce this risk by requiring sufficient agreement before important cluster decisions can be made. A partition that retains the required majority can continue leader election or promotion, while a minority partition should normally lose the ability to make authoritative changes. Quorum does not eliminate every operational problem, but it provides a formal rule for determining which part of a distributed system retains decision-making authority during communication failures.
Odd-sized voting groups are commonly used because they make majority decisions easier to maintain. In a three-member coordination group, two members can form a majority if one member becomes unavailable. However, simply adding voting processes to the same physical server or network segment does not create meaningful fault tolerance. Quorum members should be distributed across appropriate failure domains so that one infrastructure failure does not remove the majority simultaneously.
Fencing is a critical mechanism for preventing an old primary from continuing to accept writes after another node has been promoted. The objective is to isolate or disable the node whose authority has been revoked before the replacement becomes fully authoritative. Fencing can involve stopping database services, blocking network access, revoking storage access, changing routing, disabling virtual addresses, modifying access policies, or using infrastructure management interfaces to power off an unsafe node.
STONITH, commonly interpreted as "Shoot The Other Node In The Head," is a strong form of fencing used in some HA systems. Despite the dramatic name, its purpose is precise: ensure that a node cannot continue modifying shared or authoritative state when its cluster role is uncertain. Power control, hypervisor operations, cloud instance controls, or storage fencing can provide stronger guarantees than merely asking an unresponsive database process to shut down.
Leader leases and distributed locks can also help preserve a single writable authority. A node may be permitted to act as primary only while it owns a valid lease or lock maintained through a consensus or coordination service. If it loses access to that authority and cannot renew the lease, it should stop accepting writes. Such mechanisms are useful only when expiration, clock assumptions, network behavior, and failure semantics are carefully designed and tested.
Client routing must follow the same authority model as the database cluster. Even if the cluster correctly promotes a new primary, stale DNS records, load balancers, proxies, connection pools, or application configuration can continue sending requests to the old node. HA design should therefore coordinate database role changes with service discovery and routing updates so that clients converge on the currently authorized writable endpoint.
Detection should combine multiple signals rather than interpreting one failed health check as proof that a primary has disappeared. Database reachability, replication state, cluster membership, coordination-service status, network paths, and peer observations can provide a more reliable picture. Aggressive failure detection may reduce outage time but can also increase false promotions, so timeout and election settings must balance Recovery Time Objective with split-brain protection.
When split brain is suspected, the first operational priority is to stop additional divergence. Engineers should identify which nodes are currently writable, which clients can reach them, and which side retains legitimate cluster authority. Unsafe writers should be isolated before normal traffic is restored. Continuing to accept transactions on both sides while investigating may increase the amount of conflicting data and make subsequent reconciliation significantly more difficult.
After write authority has been stabilized, transaction histories must be compared. Administrators may examine transaction logs, WAL positions, GTIDs, replication offsets, timestamps, application identifiers, or database-specific metadata to determine where histories diverged. The exact evidence depends on the database technology. The objective is to establish a reliable boundary between common replicated history and transactions that occurred independently after the partition.
Choosing the authoritative history should follow predefined ownership and business rules rather than simply selecting the node containing the largest amount of data. One branch may contain later transactions while another contains operations that are more important or externally committed. Mission assignments, payments, inventory changes, robot commands, maintenance records, and authorization updates can have different reconciliation requirements, making application semantics essential to recovery.
Automatic merging is safe only when conflicts can be resolved deterministically. Independent inserts with globally unique identifiers may sometimes be combined, while concurrent updates to the same record may require business-level decisions. Blindly copying one database over another can discard valid transactions, but blindly merging every record can create impossible states. Recovery procedures should classify data according to whether it can be replayed, merged, regenerated, or manually reviewed.
Idempotency helps reduce damage when operations must be retried or replayed after a split-brain incident. If a mission request, event, or command has a unique operation identifier, the application can recognize that the same logical action was already processed and avoid duplicating it. This is especially valuable when clients experienced timeouts during the partition and cannot determine whether their original transaction completed before reconnecting to another primary.
Write ownership can prevent many conflicts before they occur. Data can be partitioned by site, robot, customer, region, or another stable key so that only one authority normally updates a given logical record. Even in active-active systems, ownership rules reduce the probability that isolated regions modify the same data concurrently. Global information that requires strict consistency can use stronger coordination while locally owned data continues with fewer dependencies.
Monitoring should detect conditions that indicate increased split-brain risk before conflicting writes occur. Important signals include loss of quorum, multiple writable nodes, unexpected leader changes, failed fencing, replication interruption, rapidly increasing lag, coordination-service failures, inconsistent cluster membership, and clients connected to obsolete primaries. Alerts should emphasize authority violations because a functioning but unauthorized writer can be more dangerous than an unavailable node.
Audit logs are essential during incident reconstruction. Database logs, cluster-manager events, proxy records, infrastructure events, application request identifiers, and accurate timestamps can reveal when the partition began, when promotion occurred, and which clients wrote to each branch. Clock synchronization across database, application, and infrastructure systems improves the ability to reconstruct the sequence of events and determine the scope of divergent transactions.
Backups and point-in-time recovery provide an additional safety mechanism when reconciliation cannot confidently restore a consistent state. Replication alone is insufficient because conflicting or incorrect changes may already exist across live nodes. Historical backups, archived transaction logs, and recovery checkpoints allow administrators to reconstruct a known state and selectively replay validated operations when direct reconciliation would create unacceptable uncertainty.
Failure testing should intentionally reproduce network partitions and isolation scenarios rather than testing only clean process shutdowns. Engineers should disconnect the primary from coordination services, isolate replicas, interrupt client routes, and verify quorum behavior, fencing, promotion, routing convergence, and recovery. Tests should confirm that an isolated node cannot remain an authoritative writer after losing the conditions required to hold the primary role.
Operational runbooks should define split-brain response before an incident occurs. They should identify who can declare a primary authoritative, how unsafe nodes are fenced, how traffic is stopped or redirected, which evidence must be preserved, and how divergent data is classified. Recovery steps should be rehearsed because improvised decisions made during an outage can unintentionally destroy evidence or overwrite transactions needed for later reconciliation.
For robot fleet and Physical AI systems, conflicting database authority can affect mission ownership, scheduling, robot configuration, maintenance state, user permissions, and fleet coordination. Immediate robot safety should therefore remain local rather than depending on the availability of a globally authoritative database. During a database partition, robots can preserve safe local behavior while higher-level operations restrict or suspend actions that require globally consistent state.
A resilient architecture treats split-brain prevention as a combination of quorum, fencing, authority management, routing, application semantics, monitoring, and tested recovery procedures. No single mechanism is sufficient in every failure scenario. The essential principle is that losing communication must never silently create multiple unrestricted authorities. When authority is explicit and unsafe writers can be isolated quickly, HA systems can fail predictably instead of allowing hidden data divergence.
10.10 Robot Fleet Database HA Operation Case
A robot fleet database is a shared operational foundation that stores information required to coordinate many robots, missions, users, facilities, and supporting services. High availability becomes important because a database outage can interrupt mission assignment, fleet dashboards, configuration access, maintenance workflows, and coordination between robots. The HA design must therefore protect centralized operations without making immediate robot safety dependent on continuous database connectivity.
A practical fleet architecture separates real-time robot control from fleet-level database services. Motion control, emergency stopping, obstacle avoidance, localization, and essential navigation remain on the robot or local edge system. The central database manages slower-changing operational state such as robot registration, mission queues, schedules, maps, configuration versions, user permissions, maintenance records, and historical events. This separation limits the physical consequences of database failure.
The primary operational database can be deployed as an HA cluster with one writable primary and one or more replicas. Applications access the database through a stable service endpoint rather than connecting permanently to a specific server. Replication continuously transfers committed changes to standby nodes, while an HA manager observes node health and promotes a suitable replica when the primary becomes unavailable. Client routing is then updated toward the new writable node.
Fleet applications should not assume that every database request will succeed immediately. A primary failure can interrupt existing connections while promotion and routing changes occur. Connection pools, API services, mission managers, and fleet orchestration components should detect broken sessions and reconnect using bounded retry policies. Transactions must be designed so that retries do not accidentally create duplicate missions, commands, reservations, or maintenance records.
Idempotency is especially important in robot fleet operations because communication failures can leave clients uncertain about whether a request was completed. A mission creation request can include a globally unique operation identifier so that repeating the same logical request does not create another mission. Similar identifiers can protect charging requests, task assignments, configuration updates, and other operations that may be retried after database or network recovery.
Robot state should be classified according to durability and timing requirements. Persistent mission definitions, configuration, authorization, maintenance history, and business records generally require durable storage. High-frequency telemetry, transient positions, heartbeats, and temporary sensor-derived states may use specialized time-series, cache, streaming, or observability systems. Treating every robot message as an equally critical relational transaction can unnecessarily increase database load and complicate HA recovery.
A fleet database should also distinguish authoritative state from derived state. Mission ownership, robot assignment, configuration versions, and access permissions may represent authoritative records that require carefully controlled updates. Dashboard summaries, utilization statistics, search indexes, and analytical views can often be reconstructed from primary records or event streams. This classification helps determine which data requires strict replication guarantees and which services can temporarily operate with stale information.
During normal operation, fleet services write authoritative transactions to the primary database while replicas remain synchronized and may support selected read workloads. Monitoring continuously evaluates primary availability, replica health, replication lag, storage capacity, connection count, transaction rate, and HA coordination state. The objective is to detect degraded redundancy before an actual primary failure occurs rather than discovering that no usable replica exists during an outage.
Consider a fleet control system operating many AMRs across a warehouse or industrial site. The database primary becomes unavailable because of a server or storage failure. Robots already executing missions should continue their locally validated motion behavior, while the fleet orchestration layer temporarily pauses database-dependent actions that cannot be safely confirmed. The HA system detects primary failure, evaluates available replicas, and begins controlled failover.
Before promotion, the HA mechanism should verify that the candidate replica is sufficiently current and still has legitimate authority to become primary. Quorum or coordination services can prevent isolated nodes from independently promoting themselves. If the previous primary might still be running, fencing should remove its ability to accept writes. This step is critical because restoring service quickly is less valuable than creating two writable primaries with divergent mission and robot state.
After a replica is promoted, the database service endpoint or proxy redirects new application connections toward the new primary. Fleet APIs and orchestration services reconnect and resume transactions according to their retry logic. Operations whose results were uncertain during the failure are checked using transaction or operation identifiers before being repeated. This prevents a temporary database outage from turning into duplicated robot missions or contradictory fleet assignments.
Replication lag determines how much recently committed information may be missing after an asynchronous failover. The operational impact depends on the lost data type. Losing a dashboard update may be tolerable, while losing a newly issued mission assignment or authorization change may require additional verification. Recovery Point Objective should therefore be connected to application semantics, and critical records may require stronger durability policies than high-volume operational observations.
When the failed server returns, it should not automatically resume its previous primary role. Its database history may be behind or may have diverged from the promoted node. The returning server should be isolated, compared with the current authoritative history, resynchronized, and reintroduced as a replica. Automated rejoin mechanisms can simplify this process, but they must preserve the rule that only one node owns writable authority in a single-primary architecture.
Fleet operation may also span multiple buildings, factories, or geographic regions. Each site can maintain local edge services and selected local data so that temporary WAN failure does not stop safe robot operation. Regional or central databases can receive synchronized mission history, fleet metadata, maintenance information, and analytical records. This hierarchical approach combines local autonomy with centralized visibility while reducing dependence on continuous wide-area connectivity.
A site disconnected from the central platform should enter a defined degraded operating mode rather than behaving unpredictably. Existing missions may continue when their required maps, permissions, and resources are locally available, while new globally coordinated operations can be restricted. Locally generated events can be buffered and synchronized after connectivity returns. The precise policy depends on whether each data category is locally authoritative, centrally authoritative, or safely mergeable.
Conflicting ownership must be prevented when multiple sites can operate during disconnection. Robot identifiers, mission identifiers, resource reservations, and task ownership should use rules that remain unambiguous across failure domains. Globally unique identifiers and explicit site ownership reduce collision risk. Data requiring one global authority may need to become read-only or unavailable during partition, while site-owned records can continue to evolve locally and synchronize later.
Caching can improve resilience but must not silently replace authoritative database state. Fleet services may cache robot metadata, maps, configuration, or permission information to survive short database interruptions. Every cached item should have defined validity and refresh behavior. Safety-sensitive or authorization-related decisions should not rely indefinitely on stale information merely because a cached copy remains technically available after the central database has failed.
Message queues and event streams can decouple robot communication from immediate database availability. Robot events can enter a durable messaging layer and be written to the operational database when database service is available. This reduces direct connection pressure and allows short outages to be absorbed without losing every incoming event. However, consumers must handle duplicate delivery, ordering, replay, and backpressure so that delayed processing does not corrupt fleet state.
Monitoring should connect database health to fleet-level consequences. Engineers need visibility into primary role, replicas, replication lag, failover state, storage, backups, API errors, connection retries, queued events, mission-processing delays, and the number of robots operating in degraded mode. A database cluster can technically be healthy while fleet services remain unable to reconnect, so infrastructure metrics and application-level service indicators should be examined together.
Alerting should distinguish reduced redundancy from actual service loss. A failed replica may generate a warning because the primary remains available but the fleet has lost part of its recovery protection. Loss of the writable primary without successful promotion is more urgent, while multiple writable nodes should be treated as an authority violation. Alerts should lead to specific operational procedures rather than merely report that a metric crossed a threshold.
Backups remain necessary even when the fleet database has several replicas. Replication protects availability but can rapidly reproduce accidental deletion, corrupted application updates, or logically incorrect records across all nodes. Scheduled backups, transaction-log archiving, and point-in-time recovery provide a historical recovery path. Restore procedures should be tested using representative fleet data so that backup existence is not mistaken for proven recoverability.
HA testing should reproduce realistic fleet failures rather than only stopping database processes in isolation. Tests can terminate the primary, disconnect replicas, interrupt network paths, remove coordination members, restart the former primary, and temporarily isolate fleet applications. Engineers should observe robot behavior, mission continuity, failover time, reconnection, retries, duplicate prevention, monitoring, alerts, and the restoration of full redundancy after recovery.
A useful test also verifies what happens during database unavailability rather than concentrating only on recovery. Robots should maintain local safety, applications should reject or queue operations according to defined policies, operators should receive understandable status information, and uncertain transactions should remain traceable. This confirms that the fleet fails in a controlled mode instead of allowing infrastructure failure to create unsafe or logically inconsistent behavior.
The resulting HA architecture is therefore more than a replicated database cluster. It combines local robot autonomy, authoritative data classification, database replication, automated failover, fencing, stable client routing, idempotent applications, buffering, monitoring, backups, and tested recovery procedures. When these layers are designed together, database failures can degrade fleet coordination temporarily without becoming uncontrolled robot failures or permanent corruption of operational state.
Author: Richard Minsoo Go
LinkedIn: Richard Minsoo Go










Top comments (0)