How it works · one resource, one control loop

Service-aware failover
for active-active services.

elasticd turns service-health evidence into bounded eligibility, exclusive authority, exact Linux VIP state, and explicit route intent. Each resource reconciles independently—and when authority can no longer be proven, the safe answer is withdrawal, not optimism.

the control path before the configuration reference
01 / WATCH

Watch health

External probes report expiring evidence. All required checks must remain fresh before a node may contend.

02 / MOVE

Move the VIP

Only current authority may add the exact configured addresses; every mutation is observed before it counts.

03 / REPORT

Report the owner

The publisher turns ownership into explicit route intent and separately reports local publication and peer state.

client
traffic
VIP
edge-1
service healthy
current owner
edge-2
service healthy
eligible
edge-3
service healthy
eligible
203.0.113.20/32 · all three services are active · one explicit owner
Resource model · steady state

vip/web gets its own control loop.

09:41:00 · ACTIVE

One long-lived controller serializes health, ownership, address, publisher, timer, and shutdown events for this resource. Its authority generation and desired state belong to vip/web alone.

controllerone serialized state owner per resourceauthoritygeneration-scoped and deadline-bound
INPUThealth/webfresh · ttl 5s
RESOURCEvip/webreconcile()
DESIREDedge-1owner · active

Why is reconciliation scoped per resource?

Web, mail, and DNS can live in the same daemon while preferring different nodes and failing independently. Each resource controller owns only its mutable lifecycle state, operation generation, timers, and status snapshot. Shared capabilities such as the BGP runtime keep one lifecycle, but they do not merge resource authority.

Every asynchronous result carries the authority and operation generation that started it. If ownership changes while a netlink or publisher operation is still in flight, the late result is rejected instead of completing a newer transition. One resource failure does not reset unrelated controllers unless a shared runtime is unsafe for all of them.

Read resource reconciliation and state-machine invariants →

Health evidence · local eligibility

Health is evidence. It is never authority.

09:41:05 · CHECK EXPIRED

External producers report bounded, versioned observations for the workload. When one required check expires or becomes unacceptable, edge-1 becomes ineligible immediately—without the health listener touching Raft, Consul, netlink, or BGP.

evidencelocal, expiring, and in-memory onlyeligibleall required checks fresh and accepted
edge-1 / webexpired
edge-2 / webeligible
edge-3 / webeligible

How does elasticd know a service is healthy?

Service-specific checks run outside the elasticd authority daemon. Monit, Consul, or root-managed scripts submit fresh health reports through its local health socket. This keeps application credentials and protocol logic outside the privileged HA daemon.

Every required check begins as unknown after restart. Acceptable states carry a bounded TTL derived from the daemon’s monotonic receive time; producer timestamps cannot extend it. Missing, expired, critical, withdrawn, malformed, or unauthorized evidence never satisfies eligibility.

Health can remove a node from contention, but it cannot create ownership or request a VIP. Read local health notification and eligibility →

Ownership · provable authority

A lock or grant is not permission forever.

09:41:06 · GENERATION CHANGED

The selected driver identifies one authority generation. The controller may act only while that generation still matches and its conservative local deadline remains unexpired. Failed proof never extends the clock.

previousedge-1 proof expired or invalidatedcurrentedge-3 confirmed a new generation

How does elasticd choose the owner?

Resource configuration defines ownership priorities and required health checks. Fresh health reports determine which nodes are eligible, and the configured preference guides owner selection. The selected node still needs a committed Raft ownership grant and an unexpired local lease before it can act. Health or priority alone never grants ownership.

Handover is break-before-make: the current owner withdraws routes, waits for the required propagation delay, removes managed VIPs, and confirms cleanup before releasing ownership. If it crashes or cleanup cannot be proven, the successor waits a conservative safety delay before taking over. Leases and takeover safeguards prevent the previous and next owners from overlapping.

The design prefers an explicit outage to two correct owners. Understand ownership leases and safe takeover →

Actuation · managed VIP

Authority becomes exact Linux address state.

09:41:07 · VIP USABLE

edge-3 adds only the configured host prefixes on the configured existing interface. An acknowledgement is not enough: fresh netlink observation must prove every required address usable before route publication can begin.

desiredconfigured /32 and /128 host prefixes presentobservedexact addresses usable on current link
edge-1withdrawn
VIP /32
edge-3usable · observed
desired vip0 + 203.0.113.20/32observed exact address usable

What does “move the VIP” actually change?

The address capability owns only exact configured pairs: interface name plus canonical host prefix. It may add, observe, or remove those pairs. It does not create or bring up the interface, adopt an address it discovers, infer resources from host networking, or touch unrelated addresses, routes, rules, neighbors, namespaces, sysctls, or firewall state.

Netlink events are wake-up hints, not durable truth. Full link-and-address observation is authoritative at startup, after event loss or decode failure, after link replacement, and periodically. IPv4 must be observed on the expected link; IPv6 must also clear tentative, duplicate-address, and deprecation checks required by the address contract.

If an active VIP disappears, elasticd withdraws publication before repairing the address and republishing. Read managed VIP and netlink reconciliation →

Publisher · explicit route intent

The authorized owner is reported upstream.

09:41:08 · LOCAL RIB PRESENT

Only after every required VIP is observed usable may edge-3 insert the resource’s exact configured path. The publisher reports local route state and peer availability separately, without pretending either proves Internet-wide convergence.

intent203.0.113.20/32 · current generationpublisherembedded BGP leaf · explicit allow-list
intent203.0.113.20/32 · edge-3
local-ribexact path present
peer/websession established
local RIB ≠ the whole Internet

What does “published” prove—and what does it not?

The publisher accepts only validated route intent from resource configuration. It does not scan interfaces, redistribute kernel routes, derive aggregates, import routes into the host, or let an arbitrary path cross the publisher boundary. For v0.1, one embedded GoBGP server and peer set are shared across resources, while desired routes remain resource-scoped.

Local publication proves the exact elasticd-originated path exists in the embedded BGP local RIB for the current authority generation. It does not prove that a peer selected it or that every downstream router converged. Peer failure is operational degradation, not a new ownership election.

Clean drain proves local route absence and maintains it through the configured propagation delay before removing the VIP and releasing authority. Unclean takeover waits the declared stale-route fence. Read publisher intent, BGP observation, and fencing →

Replay the incident

Different failure.
Same control loop.

Health expiry, owner loss, and quorum loss change different inputs. The output stays explainable—and sometimes the safe output is no VIP at all.

VIP
edge-1
healthy · owner
edge-2
healthy · eligible
edge-3
healthy · eligible
desired/owner: edge-1route published · evidence fresh · quorum healthy
Operator notes · where the details matter

Three promises worth checking before production.

READ / 01

Evidence freshness

Know exactly when a sample becomes too old to justify ownership. Health model →

READ / 02

Driver guarantees

The authority backend defines the fencing and lease guarantees you actually get. Driver contract →

READ / 03

Withdrawal behavior

When ownership cannot be proven, disappearing can be safer than guessing. Failure model →

Continue down the stack

The page explains the loop. The design explains the promises.

Read the ownership model, evidence semantics, driver contracts, and the assumptions still being challenged in public.