Building an Autoscaler for Docker Swarm: Inside swarmctl

Series note: this is the deep dive behind the autoscaling section of Running Production Swarm, which covers the whole architecture it plugs into. You do not need to read that first, but it explains where swarmctl sits: on a manager node, reading cAdvisor and Traefik metrics, scaling services that opt in through labels.

A deep dive into swarmctl: a lightweight daemon that watches Swarm services, decides when to scale, executes the change, and coordinates node-level scaling with OpenStack.

Why build this at all?

Kubernetes gives you the Horizontal Pod Autoscaler, the Cluster Autoscaler, and a metrics pipeline out of the box. Docker Swarm gives you docker service scale <name> <n> and nothing else. No reactive scaling, no load-aware placement, no graceful node drain before a VM is terminated.

If you are running Swarm in production (and plenty of people still are, because it is simple, stable, and fits on three small VMs without a control-plane tax), you have to build that layer yourself. swarmctl is the layer I built.

It does not try to be Kubernetes. It is explicitly not:

• a replacement for docker stack deploy
• a full observability stack (no Prometheus, no Grafana)
• a node provisioner (OpenStack's autoscaling group handles that)

It is a daemon that:

1. collects per-service metrics every cycle (CPU, memory, CPU throttling, request rate, latency, errors, connections, pending/rejected task counts),
2. resolves a scaling policy per service from labels, templates, and defaults,
3. feeds metrics + policy into a pure decision engine,
4. executes the resulting decision through the Docker API with verification and partial-success handling,
5. signals OpenStack to add or remove whole VMs when the cluster itself runs out of room,
6. drains nodes gracefully before OpenStack terminates them.

In this post I walk through the whole thing, with the real code. There are five parts worth understanding in depth:

• The collector: turning raw cAdvisor and Traefik numbers into clean per-service metrics.
• The policy layer: resolver (config → policy) and the engine (metrics + policy → decision).
• The executor: making a decision real, safely.
• The watch loop: the stateful orchestrator that ties it all together.
• The OpenStack integration: scale-out signals, scale-in approval, and graceful drain.

Before the code, here is the control loop in miniature. This is a simplified model, not swarmctl itself: watch how startup lag, cooldowns, and a minimum replica count shape what an autoscaler does with a traffic spike. The rest of the post is about the guards that make the real one behave this way.

Architecture at a glance

There are three paths through swarmctl, and the diagram below follows each one: the watch-loop cycle that does most of the work, OpenStack asking permission to scale in, and the drain that runs just before a VM is terminated. Press Play to watch a path, or Step to go one stage at a time. Each stage shows the code that does the work, and you can click any box to jump to the step where it is used.

swarmctl watch --interval 30

Two concurrency boundaries matter: the watch loop runs on the main thread on a fixed interval, and the hook server runs on a daemon thread so OpenStack can call in at any time. They share a small amount of state (last scale-out timestamp per pool) through a lock-protected module-level dict.

Part 1: The Collector

The engine is only as good as its inputs. The collector's job is to produce one clean ServiceMetrics object per enabled service, every cycle, without ever blocking the loop for longer than one interval.

Only manage what opts in

Services opt in with a Swarm label. Nothing is scaled unless it explicitly asks to be:

deploy:
  labels:
    - "swarmctl.enable=true"
    - "swarmctl.policy=blazor-standard"   # named template
    - "swarmctl.max_replicas=8"           # override any single field
    - "swarmctl.scale_up_cpu_avg=60"

Concurrent collection with a hard deadline

Collecting stats for a service means hitting cAdvisor on whichever node each task runs on. Done serially, total time is O(n_services × latency). The collector fans out across a thread pool and enforces a deadline so one slow service can never stall the whole loop:

def collect_all(self) -> List[Tuple[ServiceMetrics, Dict[str, str]]]:
    s = get_settings()
    self._docker.refresh_infra_maps()

    enabled = self._docker.list_enabled_services()
    if not enabled:
        return []

    # Scrape Traefik ONCE per cycle. Calling get_metrics() per service would reset
    # the rate-state on every call and give zero deltas for all but the first service.
    traffic_cache = self._docker.traefik.get_metrics()
    deadline = time.monotonic() + s.collection_timeout

    with ThreadPoolExecutor(max_workers=min(len(enabled), 16),
                            thread_name_prefix="collect") as pool:
        future_to_svc = {
            pool.submit(self._docker.get_service_metrics, svc.id, traffic_cache): (svc, labels)
            for svc, labels in enabled
        }
        remaining = max(0.1, deadline - time.monotonic())
        done, not_done = _wait(future_to_svc, timeout=remaining)

        if not_done:
            skipped = [future_to_svc[f][0].name for f in not_done]
            logger.warning(f"Collection deadline: {len(not_done)} skipped this cycle: {skipped}")
            for fut in not_done:
                fut.cancel()
        ...

Two design decisions worth calling out:

• Traefik is scraped once per cycle, not once per service. The scraper keeps per-node rate state (previous counter values). If you scraped per service, every call but the first would see a zero time delta and report a zero rate. Scrape once, pass the cache down.
• Concurrency is capped at 16. Beyond that, the Docker daemon socket becomes the bottleneck, not the thread count.

Computing CPU the right way

Docker/cAdvisor expose raw cumulative CPU counters, not percentages. You need two samples and the standard delta formula:

cpu_delta    = cpu_usage.total      - precpu_usage.total
system_delta = system_cpu_usage     - pre_system_cpu_usage
cpu_percent  = (cpu_delta / system_delta) * online_cpus * 100

swarmctl pulls two samples straight from cAdvisor (?count=2) rather than Docker's own stats endpoint, because cAdvisor also exposes CPU throttling, which is the single most useful signal for a CPU-limited container:

# cpu_throttle_pct: % of CFS scheduling periods that were throttled. A container
#                    showing low raw CPU% but high throttle% is hitting its CPU ceiling.

A container pinned to 0.5 CPU can sit at "50% CPU" forever and look healthy while being heavily throttled and serving slow responses. Raw CPU percentage hides that; throttle percentage exposes it. This is why the engine has a dedicated throttle trigger.

Two memory denominators

Memory gets reported two ways because there are two meaningful questions:

• node_mem_pct: memory as a percentage of the host node's total RAM. Always present, always a consistent denominator across services. This is what the policy engine uses.
• limit_mem_pct: memory as a percentage of the container's own limit. Only present when a real limit is set; None otherwise.

The "no limit set" case is a trap. cAdvisor reports an unlimited container's limit as a kernel sentinel (9223372036854771712, i.e. max int64). Dividing by that gives a meaningless near-zero percentage, so the code treats anything over 128 GB as "no limit":

_MAX_SANE_LIMIT = 128 * 1024 * 1024 * 1024  # 128 GB
mem_limit_bytes = entry.get("spec", {}).get("memory", {}).get("limit", 0)
has_real_limit  = 0 < mem_limit_bytes < _MAX_SANE_LIMIT
limit_mem_pct   = (mem_bytes / mem_limit_bytes * 100.0) if has_real_limit else None

Memory also uses working_set (not raw usage) to match what docker stats shows: working_set excludes reclaimable page cache, so you measure what the process actually needs.

A stale-data guard

cAdvisor's default housekeeping interval is 10 seconds. If your watch interval is 5 seconds, every other cycle you'd re-read the exact same sample, compute a zero delta, and report 0% CPU, which would spuriously suppress scale-up triggers on alternating cycles. The collector caches the last result keyed by timestamp and returns the cache when the sample hasn't advanced:

latest_ts = stats_list[-1].get("timestamp", "")
cached_ts, cached_result = self._cadvisor_cache.get(container_id, ("", None))
if latest_ts and latest_ts == cached_ts and cached_result is not None:
    return cached_result   # don't report a false 0%

Replica state and "stuck pending"

The collector counts tasks by state and, critically, distinguishes why a task is pending. A task pending because of insufficient cluster capacity is a scale-out signal. A task pending because of a bad image or a missing secret is a broken deployment that must not trigger scaling:

_CAPACITY_PATTERNS = ("no suitable node", "insufficient resources",
                      "scheduling constraints not satisfied")
_CONFIG_ERROR_PATTERNS = ("no such image", "pull access denied", "secret not found", ...)

Only capacity-pending tasks feed longest_pending_seconds, which the engine turns into a host-scale signal.

Service name matching with Traefik

Traefik labels services as myapp_blazor@docker; Docker calls them myapp_blazor. The scraper strips the @docker / @swarm provider suffix with a regex, and the actual Traefik service name is derived from the service's own Traefik label rather than guessed:

# traefik.http.services.<name>.loadbalancer.server.port  →  <name>

Because Traefik can run on several nodes (one per manager in our cluster), the scraper discovers all Traefik endpoints and keeps per-node rate state, summing deltas across nodes each cycle. Counter resets (a Traefik container restart) are guarded with max(0.0, current - previous) so a reset never produces a negative rate.

The output of all this is one ServiceMetrics per service:

class ServiceMetrics(BaseModel):
    service: str
    replicas_running: int
    replicas_desired: int
    replicas_pending: int
    replicas_failed: int
    replicas_rejected: int = 0

    avg_cpu: float; max_cpu: float
    avg_throttle: float; max_throttle: float      # the throttle signal
    avg_mem: float; max_mem: float                 # % of node RAM
    avg_limit_mem: Optional[float]; max_limit_mem: Optional[float]  # % of container limit

    request_rate: float; avg_response_ms: float; error_rate: float; open_connections: int
    longest_pending_seconds: int
    tasks: List[TaskStat]

Part 2: The Policy Layer

The policy layer has two halves that must stay strictly separated:

• The resolver does I/O. It reads config, merges labels and templates, reloads when the config file changes. It produces a ResolvedPolicy.
• The engine does logic. It takes (metrics, policy, timers) and returns a decision. No I/O, no clock reads, no randomness. Same inputs → same outputs, every time.

That separation is what makes the engine fully unit-testable without mocking anything, and it's why the test file for it is ~28 KB on its own.

The resolver: three layers, highest wins

Every policy field is resolved from three layers, in priority order:

1. per-service label   (swarmctl.scale_up_cpu_avg=60)   highest
2. named template      (swarmctl.policy=blazor-standard) middle
3. global default      (from config.json)                lowest, always present

def resolve(self, service, labels) -> ResolvedPolicy | None:
    self._reload_if_changed()                 # hot-reload config.json on mtime change
    resolved = self.config["defaults"].copy()  # layer 3
    resolved["service"] = service.name
    resolved["enabled"] = True

    template = labels.get("swarmctl.policy")   # layer 2
    if template and template in self.config["templates"]:
        resolved.update(self.config["templates"][template])

    for label_key, (field, typ) in label_map.items():   # layer 1
        if label_key in labels:
            try:
                val = labels[label_key]
                resolved[field] = (val.lower() == "true") if typ is bool else typ(val)
            except (ValueError, AttributeError):
                pass   # invalid label value → silently fall through to template/default

    policy = ResolvedPolicy(**resolved)
    return policy if self._validate(policy) else None

Note three things:

• Config hot-reload. _reload_if_changed() checks the file mtime each cycle, so operators can tune thresholds in config.json live without restarting the daemon.
• Invalid labels fail silently. A typo'd scale_up_cpu_avg=sixty doesn't crash the loop or disable the service: it just falls through to the next layer.
• Validation can veto a service. If the resolved policy is incoherent, the service is skipped for the cycle rather than scaled with nonsense thresholds:

def _validate(self, policy) -> bool:
    if policy.min_replicas < 1: return False
    if policy.min_replicas > policy.max_replicas: return False
    if policy.scale_down_cpu >= policy.scale_up_cpu_avg: return False   # no sane band
    if policy.scale_down_mem >= policy.scale_up_mem_avg: return False
    if policy.cooldown_after_up <= 0 or policy.cooldown_after_down <= 0: return False
    if policy.scale_down_response_ms > policy.scale_up_response_ms: return False
    return True

The engine: the decision pipeline

The engine signature tells you everything about what state lives outside it:

def evaluate(
    self,
    metrics: ServiceMetrics,
    policy: ResolvedPolicy,
    time_since_last_action: int,        # from watch loop
    last_scale_action: ScalingAction,   # from watch loop
    elapsed_scale_up_seconds: int,      # from watch loop (duration timer)
    elapsed_scale_down_seconds: int,    # from watch loop (duration timer)
    suppressed_triggers: frozenset = frozenset(),  # per-trigger cooldown
) -> Tuple[ScalingDecision, bool]:      # (decision, host_scale_signal)

The engine never reads the clock. All timing is passed in. That means a test can simulate "it's been 119 seconds since the last scale-up" by passing 119: no sleeping, no mocking time.time().

The pipeline runs in a strict order, short-circuiting as early as possible.

Step 1: Guard checks

Never scale a service that is stopped or crash-looping:

if not policy.enabled:
    return self._none(svc, "policy disabled", BLOCKED_DISABLED), False
if metrics.replicas_running == 0 and metrics.replicas_desired == 0:
    return self._none(svc, "service stopped", BLOCKED_SERVICE_STOPPED), False
if metrics.replicas_running == 0 and metrics.replicas_desired > 0:
    return self._none(svc, "crash loop: 0 running, desired > 0", BLOCKED_CRASH_LOOP), False

Scaling a crash-looping service up just creates more crashing containers. The engine refuses.

Step 2: Emergency check (bypasses cooldown)

Hard ceilings fire regardless of cooldown. A container at 96% max CPU can't wait two minutes:

if metrics.max_cpu > self._EMERGENCY_CPU_HARD_CEILING:   # e.g. 95%
    emergency_trigger_names.add(TRIGGER_MAX_CPU)
if metrics.max_mem > self._EMERGENCY_MEM_HARD_CEILING:   # e.g. 92%
    emergency_trigger_names.add(TRIGGER_MAX_MEM)

The rejected-tasks case is special and important: it is the cluster-full signal:

if metrics.replicas_rejected > 0:
    return ScalingDecision(..., action=NONE, blocked_by=BLOCKED_CLUSTER_FULL), True
    #                                 ^^^^                                    ^^^^
    #       Don't try to add containers: there's no room.    But DO raise host_scale_signal.

When tasks are being rejected, adding more replicas is pointless: the cluster has no capacity. So the engine returns NONE for the service but sets host_scale_signal = True, telling the watch loop to ask OpenStack for a new VM instead.

If an emergency fires and there is room, it scales by the emergency step (3) immediately, waiving cooldown.

Step 3: Cooldown check

After an ordinary scale, respect the configured quiet period so the system can stabilize before reacting again:

if last_scale_action == SCALE_UP and time_since_last_action < policy.cooldown_after_up:
    return self._none(svc, f"cooldown after scale-up ...", BLOCKED_COOLDOWN_UP), False
if last_scale_action == SCALE_DOWN and time_since_last_action < policy.cooldown_after_down:
    return self._none(svc, f"cooldown after scale-down ...", BLOCKED_COOLDOWN_DOWN), False

Steps 4 & 5: Weighted multi-trigger scoring

This is the heart of scale-up. Each trigger is evaluated independently; any single one firing is enough to scale up. But the engine also computes a weighted score that drives how aggressively to scale (by 1, 2, or 3 replicas).

Each trigger contributes overage × weight, where overage is how far past the threshold the metric is, normalized to the headroom between threshold and ceiling:

def _check(trigger, condition, weight, overage):
    if not condition:
        return
    if trigger in suppressed_triggers:   # per-trigger cooldown
        suppressed.append(trigger)
    else:
        fired.append(trigger)
        weighted_overage_sum += overage * weight

# T1 avg_cpu   weight 0.30
_check(TRIGGER_AVG_CPU, metrics.avg_cpu > policy.scale_up_cpu_avg, 0.30,
       (metrics.avg_cpu - policy.scale_up_cpu_avg) / (100.0 - policy.scale_up_cpu_avg))
# T2 max_cpu   weight 0.30
# T3 avg_mem   weight 0.30   (same as CPU: sustained high memory should reach MODERATE)
# T4 max_mem   weight 0.25   (slightly lower: one hot container shouldn't alone drive emergency)
# T5 cpu_throttle weight 0.30 (fires even when raw CPU% looks low)
# T7 connections weight 0.10  (overage capped at 1.0, a 10x spike shouldn't alone hit EMERGENCY)
# T8 request_rate weight 0.15 (only when a rate threshold is explicitly configured)

The pending-stuck trigger is a special case: it fires, adds its weight, and raises the host-scale signal, because stuck tasks mean the cluster is out of room:

if metrics.longest_pending_seconds > policy.pending_stuck_seconds:
    fired.append(TRIGGER_PENDING_STUCK)
    weighted_overage_sum += 0.30
    host_scale_signal = True

The score has a built-in floor, so any fired trigger yields at least a NORMAL scale:

score = min(1.0, self._SCORE_BASE + weighted_overage_sum)   # SCORE_BASE = 0.30

urgency, step = Urgency.NORMAL, 1
if score >= self._SCORE_EMERGENCY_THRESHOLD:   # 0.75 → EMERGENCY, step 3
    urgency, step = Urgency.EMERGENCY, 3
elif score >= self._SCORE_MODERATE_THRESHOLD:  # 0.45 → MODERATE, step 2
    urgency, step = Urgency.MODERATE, 2

The defaults (tunable in config): a floor of 0.30 so any single fired trigger yields at least a NORMAL +1; 0.45 crosses into MODERATE +2 (one trigger well above threshold, or two triggers firing together); 0.75 into EMERGENCY +3 (several triggers blown out at once). The combined trigger weights sum to roughly 1.80, so the score saturates well before every trigger has to fire.

In short, barely over one threshold gives +1, and multiple thresholds blown out give +3. The weighting means a service that is simultaneously CPU-bound, memory-bound, and throttled escalates fast, while a service that merely has slightly elevated connections nudges up by one.

Play with the numbers yourself. Slide each metric past its threshold (the small tick on each slider) and watch the score and the step size change. The "throttled, CPU looks fine" preset is the case that makes the throttle trigger worth having.

Step 5.5: Scale-up must be sustained

A single spiky cycle shouldn't scale you up. Scale-up conditions must hold for a configured duration, except pending-stuck, which bypasses the timer because waiting would only make a capacity shortage worse:

duration_waived = TRIGGER_PENDING_STUCK in fired
if not duration_waived and elapsed_scale_up_seconds < policy.scale_up_duration:
    return self._none(svc, f"...met for {elapsed_scale_up_seconds}s, need {policy.scale_up_duration}s",
                      BLOCKED_SCALE_UP_NOT_SUSTAINED), False

Step 6: Scale-up bounds

If already at max_replicas, don't scale, but raise the host signal, because the service wants more and the only way to give it more is a bigger cluster:

if current_total >= policy.max_replicas:
    return ScalingDecision(..., action=NONE, blocked_by=BLOCKED_AT_MAX_REPLICAS), True   # host signal
if current_total + step > policy.max_replicas:
    step = policy.max_replicas - current_total
if current_total + step == policy.max_replicas:
    host_scale_signal = True   # hitting the ceiling this cycle: warn OpenStack early

Scale-down: conservative by design

Scale-up is aggressive (any trigger fires). Scale-down is the opposite: every condition must be true at once, stay true for a full duration, and then survive a simulation.

Step 7: All conditions, simultaneously

response_ms_ok = metrics.traffic is None or metrics.avg_response_ms < policy.scale_down_response_ms
down_eligible = (
    metrics.avg_cpu < policy.scale_down_cpu
    and metrics.max_cpu < policy.scale_up_cpu_max      # one hot replica blocks scale-down
    and metrics.avg_mem < policy.scale_down_mem
    and metrics.open_connections < policy.scale_down_connections
    and response_ms_ok
    and metrics.replicas_running > policy.min_replicas
    and metrics.replicas_pending == 0
    and metrics.replicas_failed == 0
)

Two subtleties:

• Response time is only enforced when Traefik data exists. A non-HTTP worker reports 0ms, which would always satisfy < scale_down_response_ms and bias every worker toward scale-down. Guarding on metrics.traffic is None prevents that false signal.
• max_cpu is checked here, not just avg_cpu. One hot replica among several idle ones can hide in the average. Checking the max means a single busy container blocks removal.

Step 8: Scale-down must also be sustained

if elapsed_scale_down_seconds < policy.scale_down_duration:
    return self._none(svc, "...conditions met for Xs, need Ys", BLOCKED_SCALE_DOWN_NOT_SUSTAINED), False

Step 9: Proportional step

Scale-up steps by urgency; scale-down normally steps by one. But if a service is deeply idle, removing one replica per cooldown period is painfully slow. So the step is proportional to how far below the threshold the load is, hard-capped so you can never halve capacity in one move:

step = 1
if n > policy.min_replicas + 1 and policy.scale_down_cpu > 0 and policy.scale_down_mem > 0:
    cpu_r  = metrics.avg_cpu / policy.scale_down_cpu
    mem_r  = metrics.avg_mem / policy.scale_down_mem
    load_r = max(cpu_r, mem_r, 0.0)
    target = max(policy.min_replicas, math.ceil(load_r * n))
    step   = max(1, min(n - target, n - policy.min_replicas))
    step   = min(step, self._SCALE_DOWN_MAX_STEP)   # never remove too many at once

Step 10: Bounds check

Clamp the step so the service can never drop below min_replicas, and skip the scale-down entirely if there is nothing left to remove.

Step 11: The removal simulation

Before removing a replica, the engine simulates what happens to the survivors. This is the guard that stops a classic oscillation: scale down → survivors overload → immediate scale up → cooldown → scale down → ...

The crucial insight is that different metrics redistribute differently, and our first implementation treated them all the same, which was wrong.

CPU redistributes proportionally. Remove a replica and the same request volume lands on fewer containers, so projecting cpu × n / (n-step) is valid:

projected_cpu = (metrics.avg_cpu * n) / projected_n
if projected_cpu > policy.scale_up_cpu_avg * self._SIMULATION_SAFETY_MARGIN:   # 85%
    return self._none(svc, "simulation failed: projected avg cpu ...", BLOCKED_SIM_FAILED_CPU), False

Memory does NOT redistribute. A container's heap, page cache, and mmapped files are process-private: they don't transfer to siblings when you remove one. The old formula avg_mem × n / (n-step) treated memory like a liquid and made scale-down nearly impossible for any memory-heavy service. The fix is to check that current memory already has headroom below the scale-up ceiling:

# memory is process-private, not redistributed
if metrics.avg_mem > policy.scale_up_mem_avg * self._SIMULATION_SAFETY_MARGIN:
    return self._none(svc, "simulation failed: current avg mem ...", BLOCKED_SIM_FAILED_MEM), False

Response time is non-linear. Latency is governed by queue depth and upstream bottlenecks (DB, cache, network), not simply by 1/n servers. Projecting it proportionally is meaningless: if a database is the bottleneck, removing a replica doesn't change it. So again the engine checks for genuine current headroom rather than projecting:

if metrics.traffic is not None and \
   metrics.avg_response_ms > policy.scale_up_response_ms * self._SIMULATION_SAFETY_MARGIN:
    return self._none(svc, "simulation failed: current response ...", BLOCKED_SIM_FAILED_RESPONSE), False

If every projection passes, the decision is finally a SCALE_DOWN.

Try it. The default preset is a service that looks idle enough to shrink but would overload the survivors. Then try the memory-heavy preset, where the old formula would have wrongly refused.

The host_scale_signal

Threaded through the whole engine is a second return value: host_scale_signal. It is set True whenever the service wants capacity the cluster can't give:

• tasks are being rejected (replicas_rejected > 0),
• tasks are stuck pending past the threshold,
• triggers are firing but the service is already at max_replicas.

This signal is deliberately separate from the ScalingDecision. The decision is about containers; the signal is about VMs. The watch loop reads the signal and forwards it to the OpenStack ASG.

Part 3: The Executor

The executor has exactly one job: make a decision real, verify it, and report honestly. No decisions of its own, no retries.

Calculate the target, then skip if it's a no-op

if decision.action == ScalingAction.NONE:
    return None   # the watch loop handles NONE logging; executor never touches Docker for NONE

target = self._calculate_target(decision, metrics, policy)
if target == metrics.replicas_running + metrics.replicas_pending:
    return None   # nothing to do

Don't fight an active deployment

A subtle production hazard: if an operator is mid docker stack deploy (a rolling update), scaling the service at the same time corrupts the rollout. The executor checks for an in-progress update and backs off, unless the decision is a forced ghost-recovery:

update_status = service.attrs.get("UpdateStatus", {})
is_updating   = update_status.get("State") == "updating"
if is_updating and not decision.force:
    return self._result(..., success=False, error="active_deployment")

Scale and classify failures precisely

try:
    service = self.client.services.get(metrics.service)
    service.scale(target)
except docker.errors.NotFound:
    return self._result(..., error=EXEC_ERROR_NOT_FOUND)    # service removed between eval & exec
except Exception as exc:
    return self._result(..., error=str(exc))

Verify by polling: partial success is a real outcome

Calling scale() returns immediately; it does not mean the replicas are running. The executor polls until the running count matches the target, or a rejection appears, or it times out:

while time.time() < deadline:
    time.sleep(_POLL_INTERVAL)
    running, rejected = self._count_task_states(service_name)

    if rejected > 0:
        return self._result(..., replicas_after=running, success=False, error=EXEC_ERROR_REJECTED)
    if running == target:
        return self._result(..., replicas_after=running, success=True)

# timeout: record what actually happened, not what we asked for
running, _ = self._count_task_states(service_name)
return self._result(..., replicas_after=running, success=False, error=EXEC_ERROR_TIMEOUT)

The ExecutionResult records replicas_before, replicas_after, requested_change, and actual_change separately, so a scale from 3→5 that only reached 4 is recorded faithfully as a partial success rather than a lie.

No retries, on purpose

The executor never retries. If a scale fails, the watch loop will re-evaluate next cycle, and if the service still needs scaling the engine will fire again. Retrying inside the executor would risk double-scaling during a transient blip. Let the loop's natural cadence be the retry.

Part 4: The Watch Loop

The engine is pure, the executor is stateless, so all mutable state lives in the watch loop. It owns the cooldown timestamps, the duration timers, the per-trigger cooldowns, and the record of what each service's target replica count should be.

last_scale_time:   Dict[str, float]            = {}
last_scale_action: Dict[str, ScalingAction]    = {}
last_trigger_time: Dict[str, Dict[str, float]] = {}   # svc -> {trigger -> ts}
scale_up_start:    Dict[str, Optional[float]]  = {}    # duration timer
scale_down_start:  Dict[str, Optional[float]]  = {}    # duration timer
target_replicas:   Dict[str, int]              = audit.get_last_targets()   # hydrated from DB

That last line matters: the loop hydrates its target state from the audit log on startup, so a restart doesn't forget what each service's intended replica count was. Cooldown and duration timers do reset on restart (the first cycle after a restart is treated as a warm-up), but the intended targets survive.

The cycle, in three phases

# Phase 1: evaluate every service (no blocking I/O)
for sm, labels in all_service_data:
    policy = policy_svc.resolve(sm.info, labels)
    if policy is None:
        continue
    ... compute time_since, elapsed_up, elapsed_down, suppressed triggers ...
    decision, host_signal = policy_svc.evaluate(sm, policy, time_since, last_action,
                                                elapsed_up, elapsed_down, suppressed)
    evals.append(_Eval(sm, labels, policy, decision, host_signal))

# Phase 2: execute all non-NONE decisions in PARALLEL
active = [e for e in evals if e.decision.action != NONE]
futures = {_exec_pool.submit(_exec_one, ev): ev for ev in active}
# total blocking time is max(individual poll_timeout), not the sum

# Phase 3: process results, update state, write audit, update the live table

Separating evaluation (pure, instant) from execution (slow, I/O-bound) lets the loop fire all scale operations concurrently. Ten services that each need scaling cost one poll-timeout, not ten.

Managing the duration timers

The engine is told how long conditions have held, but it's the loop that measures it. Each cycle it asks the policy's own condition helpers whether the up and down conditions are currently met, and starts or resets a timer accordingly:

if policy_svc.up_conditions_met(sm, policy):
    if scale_up_start.get(svc) is None:
        scale_up_start[svc] = now
    elapsed_up = int(now - scale_up_start[svc])
else:
    scale_up_start[svc] = None
    elapsed_up = 0

Those up_conditions_met / down_conditions_met helpers live on the PolicyService as the single source of truth, so the loop can't drift out of sync with the engine's own thresholds.

Per-trigger cooldown

The ordinary cooldown blocks all scaling for a period. But imagine a service whose CPU is chronically a hair over threshold: it would scale +1, cool down, scale +1 again, forever, from the same unresolved cause. Per-trigger cooldown suppresses an individual trigger for a while after it caused a scale, while leaving the other triggers free to fire:

suppressed = frozenset(
    t for t, ts in last_trigger_time.get(svc, {}).items()
    if now - ts < policy.trigger_cooldown_seconds
)

Genuine emergencies (urgency EMERGENCY) are never suppressed: a real CPU fire always gets through. pending_stuck is never suppressed either, since it drives capacity signaling.

Reconciliation and manual changes

The loop doesn't assume it's the only actor. An operator might run docker service scale by hand. On each cycle the loop compares Swarm's actual desired count to its own remembered target and reacts intelligently:

• Swarm has more than expected → an intentional manual scale-up. Accept it silently and adopt the new number as the target. Don't fight the operator.
• Swarm has fewer than expected → could be a crash, or a manual scale-down. Run the policy first. If load actually justifies the old higher count (or the service is below min_replicas), reconcile back up. Otherwise accept the operator's lower number.

if current_swarm_desired > known_target:
    target_replicas[svc] = current_swarm_desired          # accept manual scale-up
else:
    probe_decision, _ = policy_svc.evaluate(... )
    if probe_decision.action == SCALE_UP or below_min:
        reconcile_decision = ScalingDecision(action=SCALE_UP, by=diff, urgency=EMERGENCY, ...)
    else:
        target_replicas[svc] = current_swarm_desired       # accept manual scale-down

Ghost-task recovery

Sometimes a service ends up with desired > running, a task stuck pending, and no real load to justify it: the classic "ghost update" where a new task spec can never schedule (bad constraint, image that vanished). Scaling up won't help; the right move is to clear the unreachable pending task. The loop waits a configurable number of cycles (so slow image pulls and genuine rolling updates resolve themselves first), then force-scales down to clear the ghost:

if sm.replicas_desired > sm.replicas_running and sm.replicas_pending > 0 and load_is_idle:
    stuck_desired_cycles[svc] += 1
    if stuck_desired_cycles[svc] >= _s.ghost_recovery_min_cycles:
        recovery = ScalingDecision(action=SCALE_DOWN,
                                   by=sm.replicas_desired - sm.replicas_running,
                                   force=True, ...)   # force bypasses the active-deployment guard
        executor.execute(recovery, sm, ev.policy)

Writing the audit trail

Every evaluation is written to SQLite, including NONE decisions. A NONE with a nonzero score ("almost fired") is exactly the data you want when tuning thresholds later. The table is append-only and indexed by service, timestamp, and action.

Ephemeral node cleanup

As a housekeeping task, the loop watches nodes labeled lifecycle=ephemeral (VMs added by the ASG). If one stays down for enough consecutive cycles (~24h worth), it removes it from the swarm so the member list doesn't fill with dead OpenStack instances.

Part 5: The OpenStack Integration

This is what turns a service autoscaler into a cluster autoscaler. swarmctl scales replicas; OpenStack's Heat autoscaling group (ASG) scales VMs. They talk through three channels.

Scale-out: swarmctl asks OpenStack for a VM

When the engine raises host_scale_signal because the cluster is out of room, the watch loop counts how many consecutive cycles the demand has persisted, per pool, and only then fires an HTTP webhook at Heat:

for ev in evals:
    pool = ev.labels.get("swarmctl.pool") or _s.DEFAULT_POOL
    key  = (pool, ev.sm.service)
    is_pending_stuck = ev.sm.longest_pending_seconds > ev.policy.pending_stuck_seconds
    if is_pending_stuck and ev.decision.action == ScalingAction.SCALE_UP:
        svc_stuck_cycles[key] = svc_stuck_cycles.get(key, 0) + 1
    else:
        svc_stuck_cycles.pop(key, None)

# aggregate the worst stuck-count per pool, then signal each pool over the threshold
for pool, max_stuck in pool_max_stuck.items():
    if max_stuck >= _s.asg_pending_cycles:
        asg.trigger_scale_out(pool)

The pool concept matters. A cluster can have several ASG pools (say default and gpu). A service is routed to a pool by its Swarm placement constraint (node.labels.pool == gpu), the same value Swarm itself uses to schedule it, so routing can't drift from reality. A stuck GPU service must grow the GPU pool, not some unrelated one. Scale-out demand is therefore tracked and aggregated per pool.

AsgService.trigger_scale_out enforces a per-pool cooldown (one new VM takes time to boot and join before its effect shows up) and records the scale-out timestamp, which, as you will see, the scale-in path reads to avoid flapping:

def trigger_scale_out(self, pool=None) -> bool:
    cfg = s.get_pool(pool)
    if cfg is None or not cfg.scale_out_url:
        return False
    if (now - self._last_trigger.get(pool, 0.0)) < cfg.cooldown:
        return False   # a VM is already on its way
    response = httpx.post(cfg.scale_out_url, timeout=10.0)
    if response.status_code in (200, 201, 202, 204):
        self._last_trigger[pool] = now
        with _ts_lock:
            _last_scale_out_ts[pool] = now   # cross-thread handoff to scale-in logic
        return True

When the new VM boots, cloud-init runs docker swarm join, swarmctl sees the new node next cycle, the stuck tasks schedule onto it, and host_scale_signal goes quiet.

Scale-in: OpenStack asks swarmctl for permission

The reverse direction is more dangerous. OpenStack's Aodh cpu_low alarm wants to remove a VM. But Aodh only sees aggregate cluster CPU: it has no idea whether removing that node would strand services. So instead of pointing the alarm straight at Heat, it points at swarmctl, which approves or denies:

@router.post("/scale-in-request")
async def scale_in_request(pool: str = "default", ...):
    result = await asyncio.to_thread(service.evaluate_scale_in, pool)
    if result["approved"]:
        await asyncio.to_thread(AsgService().trigger_scale_in, pool)
    return ScaleInResponse(**result)   # always 200 so Aodh never retries

evaluate_scale_in runs a sequence of safety checks, all scoped to the pool that fired:

# 1. Post-scale-out cooldown. If this pool just gained a VM, don't immediately remove one,
#    otherwise Aodh sees the fresh node idle, approves removal, and could delete the ORIGINAL
#    production node, briefly dropping to zero replicas (1 → 0 → 1).
last_out = _asg_mod.last_scale_out_ts(pool)
if (now - last_out) < cooldown:
    return {"approved": False, "reason": "post-scale-out cooldown"}

# 2. Scale-in's own cooldown.
# 3. Minimum node count: strict <=, so a single-node pool can never be scaled to zero.
if len(active_workers) <= min_nodes:
    return {"approved": False, "reason": "at minimum node count"}

# 4. Any pending task in THIS pool blocks removal: the cluster is already short on capacity.
for svc in services_in_pool:
    if svc has a pending task:
        return {"approved": False, "reason": f"{svc.name} has pending tasks"}

return {"approved": True, "reason": "safe"}

The post-scale-out cooldown is the subtle one, and it needs cross-thread state: the watch loop (main thread) records the last scale-out timestamp per pool, and the hook server (daemon thread) reads it. That handoff goes through a lock-protected module dict:

_ts_lock = threading.Lock()
_last_scale_out_ts: dict[str, float] = {}

def last_scale_out_ts(pool: str) -> float:
    with _ts_lock:
        return _last_scale_out_ts.get(pool, 0.0)

Pre-terminate: drain a node gracefully

Once a scale-in is approved and Heat picks a victim VM, the VM's own systemd shutdown hook calls POST /hooks/pre-terminate before Docker stops. This is the graceful-drain path, and it is the most involved piece of orchestration in the system:

def handle(self, node_name, instance_id) -> dict:
    node = self._find_node(node_name)
    if node is None:
        return {"status": "approve", "reason": "node not found in swarm"}

    running_tasks = self._tasks_on_node(node.id)
    if running_tasks:
        # 1. Pause the node so Swarm schedules nothing new onto a dying host.
        if not self._set_availability(node, AVAIL_PAUSE):
            return {"status": "reject", "reason": "failed to pause node"}

        # 2. Drain: scale affected services DOWN by their task count on this node,
        #    so Swarm reschedules those replicas elsewhere.
        adjustments = self._compute_adjustments(node.id, running_tasks)
        self._apply_scale_adjustments(adjustments, scale_down=True)

        # 3. Wait (bounded) for tasks to actually leave the node.
        drained = self._wait_for_drain(node.id)

        # 4. Scale back UP to the original counts: the new replicas land on OTHER nodes.
        self._apply_scale_adjustments(adjustments, scale_down=False)

        # 5. Write REBASELINE events so the watch loop treats the restored counts as the
        #    authoritative target, instead of trying to "reconcile" the drain-time dip.
        self._write_rebaseline_events(adjustments)

        self._set_availability(self._client.nodes.get(node.id), AVAIL_ACTIVE)

    # 6. Finally, simulate removing the node for real.
    return self._simulate_removal(node.id)

The drain is a temporary scale-down-then-up dance: you shrink each service by exactly the number of its tasks on the doomed node, let Swarm move them, then grow back so the replacements schedule onto healthy nodes. If the drain times out, swarmctl restores the counts and rejects the termination: better to keep a node than to lose replicas.

Here is the whole sequence, including the final simulation that can still say no:

The rebaseline step solves a nasty interaction with the watch loop: during the drain the loop would see service counts dip and might try to "fix" them. Writing a REBASELINE audit event tells the loop "this new count is intentional, adopt it as the target," so the two subsystems don't fight.

Finally, _simulate_removal is the last safety gate: it projects every surviving node's memory if this node's load is redistributed, and rejects the termination if any node would exceed the rebound threshold:

threshold = self._settings.hook_rebound_threshold   # e.g. 65%
for nid, used_mb in node_used_mem.items():
    pct = (used_mb * 1024 * 1024) / node_total_mem[nid]
    if pct > threshold:
        return {"status": "reject", "reason": HOOK_REASON_REBOUND}
return {"status": "approve", "reason": "safe"}

The one rule OpenStack must obey

The webhook can only ever return approve or reject. The critical operational contract is on the OpenStack side:

OpenStack must be configured to reject a termination by default if the webhook is unreachable. A node must never be removed without positive approval.

Fail-safe, not fail-open. If swarmctl is down, nodes stay up: a slightly-too-big cluster is always cheaper than an outage.

Design principles that run through the whole thing

A few ideas show up again and again, and they're worth stating plainly because they're what make the system trustworthy:

The engine is pure. No I/O, no clock, no randomness. Every bit of timing is injected by the watch loop. This is why it can be exhaustively unit-tested, and why you can reason about its behavior by reading one function.

Scale up fast, scale down slow. Up: any single trigger fires, step 1 to 3 by urgency. Down: every condition true, sustained for a full duration, and a per-metric simulation must pass. The cost of under-provisioning (dropped requests) is far higher than the cost of a spare replica.

Different metrics behave differently. CPU redistributes across replicas; memory is process-private; latency is non-linear and often upstream-bound. The simulation treats each correctly instead of applying one proportional formula to all three, a bug our first version actually had.

Separate "scale containers" from "scale VMs". The ScalingDecision and the host_scale_signal are two independent outputs. One drives the Docker API; the other drives OpenStack. Keeping them separate keeps each layer simple.

Fail safe. Crash-looping services aren't scaled. Broken deployments (bad image, missing secret) don't trigger capacity signals. Node termination defaults to reject. Partial successes are recorded as partial, not as success.

Every decision is auditable. Every cycle, every service, every NONE-with-a-score lands in SQLite. When you later ask "why did it scale at 14:03?" or "why didn't it scale?", the answer is a query away.

Running it

Run it as a Swarm service on a manager node, with the Docker socket mounted and the hook port exposed:

services:
  swarmctl:
    image: yourrepo/swarmctl:latest
    command: ["swarmctl", "watch", "--interval", "30"]
    deploy:
      replicas: 1
      placement:
        constraints: ["node.role == manager"]
      restart_policy:
        condition: any
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - swarmctl_data:/data
    ports:
      - "8889:8889"   # OpenStack lifecycle hook

Then watch it work: swarmctl prints a live Rich table every cycle showing each service's replicas, CPU avg/max, throttle, memory, request rate, latency, error rate, the decision taken, and a ⚠ marker when the host-scale signal is raised.

Opt a service in with a single label (swarmctl.enable=true), optionally pick a template, and let the loop do the rest.

Closing thought

An autoscaler is mostly a study in restraint. The hard part isn't deciding to scale up when CPU is high: that's one if. The hard part is not scaling down into an oscillation, not fighting an operator's manual change, not killing a node that would strand a service, not reacting to a single spiky cycle, and not trusting a proportional formula that happens to be wrong for memory.

Every guard, cooldown, duration timer, simulation, and reconciliation branch in swarmctl exists because our naive first version did the wrong thing in production. That's the real content of this post: not that you can call service.scale(), but all the things you have to check before you dare to.