[{"data":1,"prerenderedAt":11934},["ShallowReactive",2],{"nav-stories":3,"blog-running-production-swarm":24,"ref-\u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm":2171,"ref-\u002Fblog\u002Flgtm-stack\u002Fpart-1":10886},[4,15],{"id":5,"color":6,"extension":7,"image":8,"label":9,"link":6,"meta":10,"order":11,"stem":12,"text":13,"__hash__":14},"stories\u002Fstories\u002F01-traffic.yml",null,"yml","\u002Fimages\u002Fstories\u002Fapp-traffic.png","TRAFFIC",{},1,"stories\u002F01-traffic","74.1 million requests in a single day, peaking at about 8,000 requests per second, with 97.3% of them successful.","ZAWpOSlORUptBXwG92wUBweOVfkhWbeLh2pEzmDrejU",{"id":16,"color":17,"extension":7,"image":6,"label":18,"link":6,"meta":19,"order":20,"stem":21,"text":22,"__hash__":23},"stories\u002Fstories\u002F02-swarmctl.yml","#1d1d1f","SWARMCTL",{},2,"stories\u002F02-swarmctl","swarmctl is the autoscaler I built for Docker Swarm. It scales services on CPU, memory and traffic, and asks OpenStack for new VMs when the cluster is full. Should I open-source it? Tell me what you think. \u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm","e1bgTtDeqxwbYkPP3QR4l79ZvA47Oljb4dfnbh43Ftw",{"id":25,"title":26,"body":27,"category":2153,"date":2154,"description":2155,"extension":2156,"meta":2157,"navigation":529,"path":2158,"readTime":2159,"seo":2160,"stem":2161,"tags":2162,"thumbnail":2169,"__hash__":2170},"blog\u002Fblog\u002Frunning-production-swarm\u002Findex.md","Running Production Swarm: Serving Containerized Apps Without Kubernetes",{"type":28,"value":29,"toc":2125},"minimark",[30,41,46,49,52,55,59,62,140,143,146,168,175,177,181,192,195,198,228,231,234,236,240,245,256,259,269,272,275,302,306,309,364,379,386,388,392,399,570,573,625,629,632,639,644,647,653,658,689,691,695,698,716,851,854,856,860,863,873,880,883,886,906,909,912,915,919,933,943,945,949,956,1009,1050,1057,1060,1067,1179,1183,1186,1220,1231,1234,1251,1257,1259,1263,1270,1284,1357,1368,1371,1374,1377,1409,1419,1422,1425,1428,1430,1434,1480,1482,1486,1548,1551,1554,1569,1573,1576,1582,1588,1593,1599,1605,1610,1617,1619,1623,1626,1658,1669,1727,1735,1737,1741,1744,1775,1777,1781,1784,1810,1813,1815,1819,2121],[31,32,33],"blockquote",{},[34,35,36,40],"p",{},[37,38,39],"strong",{},"Heads up:"," the diagrams in this post are interactive. Break things, drag sliders, click nodes. They are simplified models of the real cluster, not live data.",[42,43,45],"h2",{"id":44},"why-this-post-exists","Why this post exists",[34,47,48],{},"Containers have become the default way to ship software. A container carries its dependencies with it, so the app that worked on a laptop behaves the same in production, and the \"it works on my machine\" argument mostly goes away. If an app can be containerized, it can run the same way everywhere.",[34,50,51],{},"The journey to containers is only beginning for us. Most internally built systems are on virtual machines or bare servers, and the move to containers is gradual. This post documents how I designed our containerized deployment platform: the architecture, the reasons behind it, the incidents that shaped it, and the parts I know are imperfect.",[53,54],"hr",{},[42,56,58],{"id":57},"the-ladder-from-one-host-to-an-orchestrator","The ladder: from one host to an orchestrator",[34,60,61],{},"There are several ways to run containers in production, and they sit on a ladder of capability and complexity.",[63,64,65,84],"table",{},[66,67,68],"thead",{},[69,70,71,75,78,81],"tr",{},[72,73,74],"th",{},"Option",[72,76,77],{},"What it is",[72,79,80],{},"Good at",[72,82,83],{},"Costs you",[85,86,87,108,124],"tbody",{},[69,88,89,95,102,105],{},[90,91,92],"td",{},[37,93,94],{},"Docker on individual hosts",[90,96,97,101],{},[98,99,100],"code",{},"docker run"," or Compose on each server",[90,103,104],{},"Zero new concepts if you know Linux",[90,106,107],{},"You decide which host runs what. Every host needs credentials in CI\u002FCD. No self-healing across machines.",[69,109,110,115,118,121],{},[90,111,112],{},[37,113,114],{},"Docker Swarm",[90,116,117],{},"Orchestration built into the Docker Engine",[90,119,120],{},"Clustering, rolling updates, rescheduling, overlay networking, with a tiny learning curve",[90,122,123],{},"No autoscaling, a small ecosystem, an uncertain long-term industry position",[69,125,126,131,134,137],{},[90,127,128],{},[37,129,130],{},"Kubernetes",[90,132,133],{},"The industry-standard orchestrator",[90,135,136],{},"Scheduling, autoscaling, huge ecosystem",[90,138,139],{},"Steep learning curve and real engineering capacity to run it well",[34,141,142],{},"I started on the first rung. For a year or two, a handful of systems ran on standalone Docker hosts: web applications, internal tools, and the monitoring stack. That worked well. Several services shared one server, deployment was quick, and nobody had to learn anything exotic.",[34,144,145],{},"But standalone hosts behave like a single VM you can stack many containers on, and the questions pile up:",[147,148,149,158,161],"ul",{},[150,151,152,153,157],"li",{},"Which machine serves which app, and ",[154,155,156],"em",{},"where"," should the next container go?",[150,159,160],{},"What happens to the apps on a host when that host dies at 2 a.m.?",[150,162,163,164,167],{},"CI\u002FCD becomes a problem, because it has to connect to every host to deploy, which means ",[37,165,166],{},"each pipeline needs credentials for every host it deploys to",".",[34,169,170,171,174],{},"Those are orchestration questions. Kubernetes answers them, and it is where I want the team's skills to go. But it demands operational knowledge that our team does not have in depth yet. I have managed a few Kubernetes deployments before, and I wanted this to be a gradual transition so that the entire team could learn the fundamentals of containers and orchestration without being overwhelmed. Swarm answers the same questions with a few commands, using concepts a Linux admin already holds: nodes, services, networks, and volumes. ",[37,172,173],{},"Swarm is the entry point I can adopt without a steep learning curve, and Kubernetes remains the destination."," Because everything is containerized, the move later is mostly a change of manifest format rather than a rewrite.",[53,176],{},[42,178,180],{"id":179},"the-architecture","The architecture",[34,182,183,184,187,188,191],{},"Here is the whole cluster at a glance. A request enters through a floating IP, passes a load balancer, is routed by a reverse proxy, and arrives at a container on a worker node. Press ",[37,185,186],{},"Send a request"," to watch one request travel down the stack, or press ",[37,189,190],{},"Step"," to move it one hop at a time. The explanation under the diagram only changes when the packet moves to the next step, and you can click any box to jump straight to it. The dashed ring around the packet means it is still encrypted, and switching to plain HTTP shows where the redirect happens.",[193,194],"swarm-architecture",{},[34,196,197],{},"A few design facts drive everything else:",[147,199,200,206,212,218],{},[150,201,202,205],{},[37,203,204],{},"TLS is terminated by Traefik, not HAProxy."," HAProxy passes port 443 through as raw TCP and prepends a PROXY protocol header so Traefik still sees the real client address.",[150,207,208,211],{},[37,209,210],{},"Traefik runs only on managers",", one replica per manager, and publishes host-mode ports so HAProxy talks straight to the Traefik on the chosen node.",[150,213,214,217],{},[37,215,216],{},"Routing is label-driven."," Nothing about an app's hostname is hard-coded in Traefik. A service becomes reachable by carrying labels.",[150,219,220,223,224,227],{},[37,221,222],{},"Swarm does not scale anything by itself."," A separate in-house service that I built, ",[98,225,226],{},"swarmctl",", does that.",[34,229,230],{},"Now the same path, but with things going wrong. Break something, send a request, and step through it to see exactly where it stops. Try breaking HAProxy on the manager that holds the VIP and sending a request straight away, then send another once the election has finished. After that, stop all four tasks and compare where the request dies.",[232,233],"swarm-request-flow",{},[53,235],{},[42,237,239],{"id":238},"the-edge-keepalived-and-haproxy","The edge: keepalived and HAProxy",[241,242,244],"h3",{"id":243},"one-address-that-never-moves-keepalived","One address that never moves (keepalived)",[34,246,247,248,251,252,255],{},"Clients and firewalls need one stable address. ",[37,249,250],{},"keepalived"," provides it using VRRP: every manager advertises, and the one with the highest priority holds a ",[154,253,254],{},"virtual IP"," (VIP). If that manager disappears, the others notice the missed advertisements and one of them takes over the address.",[34,257,258],{},"I also tie the priority to what actually matters, which is whether HAProxy is answering:",[260,261,267],"pre",{"className":262,"code":264,"language":265,"meta":266},[263],"language-text","vrrp_script chk_haproxy {\n    script   \"curl -sf http:\u002F\u002Flocalhost:80\"\n    interval 2\n    weight   -20        # failing check costs 20 priority points\n}\n\nvrrp_instance VI_1 {\n    state             MASTER      # BACKUP on the others\n    interface         eth0\n    virtual_router_id 51\n    priority          110         # 100 and 90 on the others\n    advert_int        1\n    virtual_ipaddress { 10.0.0.100\u002F23 }\n    track_script      { chk_haproxy }\n}\n","text","",[98,268,264],{"__ignoreMap":266},[34,270,271],{},"If HAProxy stops responding on the master, it drops from 110 to 90, the node with 100 takes the VIP, and when HAProxy recovers the original master preempts back. Failover takes about three missed adverts plus up to two seconds for the check, so roughly five seconds. The animation above shortens it, but you can see requests time out during the hand-over.",[34,273,274],{},"Two lessons from running this:",[276,277,278,284],"ol",{},[150,279,280,283],{},[37,281,282],{},"Priorities and weights interact."," Think through what happens when two nodes fail, not just one. The tie-break between equal priorities is the higher IP address.",[150,285,286,289,290,293,294,297,298,301],{},[37,287,288],{},"VRRP is multicast by default."," A manager on a different network (this cluster has one in OpenStack) may never hear its peers and will promote itself, producing two nodes holding the same VIP. The fix is unicast VRRP (",[98,291,292],{},"unicast_src_ip"," \u002F ",[98,295,296],{},"unicast_peer","), or keeping that node out of the VRRP group and using it only as an HAProxy backend. Verify with ",[98,299,300],{},"ip -br a"," on every manager.",[241,303,305],{"id":304},"spreading-load-haproxy","Spreading load (HAProxy)",[34,307,308],{},"Every manager runs HAProxy with an identical file. It does two jobs:",[63,310,311,324],{},[66,312,313],{},[69,314,315,318,321],{},[72,316,317],{},"Listener",[72,319,320],{},"Mode",[72,322,323],{},"Behavior",[85,325,326,347],{},[69,327,328,333,336],{},[90,329,330],{},[98,331,332],{},":80",[90,334,335],{},"http",[90,337,338,339,342,343,346],{},"Adds ",[98,340,341],{},"X-Forwarded-*"," headers, rate-limits per source IP with a stick table (HTTP 429 above 200 requests per 10 s), balances ",[98,344,345],{},"leastconn"," across the three Traefiks",[69,348,349,354,357],{},[90,350,351],{},[98,352,353],{},":443",[90,355,356],{},"tcp",[90,358,359,360,363],{},"TLS passthrough to the same Traefiks with ",[98,361,362],{},"send-proxy"," so the client IP survives",[34,365,366,367,370,371,374,375,378],{},"Health checks run every 5 s with ",[98,368,369],{},"rise 2 fall 3",", so a dead Traefik is pulled out in about 15 s. During that window ",[98,372,373],{},"option redispatch"," and ",[98,376,377],{},"retries 3"," retry on another backend, so most users never notice.",[34,380,381,382,385],{},"HAProxy and keepalived are both ",[37,383,384],{},"native services",", not containers. That is on purpose: the thing that gives containers a front door should not depend on the orchestrator being healthy.",[53,387],{},[42,389,391],{"id":390},"traefik-routing-without-editing-config","Traefik: routing without editing config",[34,393,394,395,398],{},"Traefik is the only containerized part of the edge. It watches the Swarm API, discovers services that opt in, and builds routes from their ",[37,396,397],{},"labels",". Adding an app to production means adding labels to its stack file. Nobody edits a proxy config or reloads anything.",[260,400,404],{"className":401,"code":402,"language":403,"meta":266,"style":266},"language-yaml shiki shiki-themes vitesse-light","services:\n  web:\n    image: registry.example.com\u002Fteam\u002Fweb:1.4.2\n    networks: [proxy-net]\n    deploy:\n      replicas: 3\n      labels:\n        - traefik.enable=true\n        - traefik.http.routers.web.rule=Host(`app.example.com`)\n        - traefik.http.routers.web.entrypoints=websecure\n        - traefik.http.routers.web.tls=true\n        - traefik.http.services.web.loadbalancer.server.port=8080\n\nnetworks:\n  proxy-net:\n    external: true\n    name: prod-proxy-net\n","yaml",[98,405,406,418,425,438,455,463,475,483,492,500,508,516,524,531,539,547,559],{"__ignoreMap":266},[407,408,410,414],"span",{"class":409,"line":11},"line",[407,411,413],{"class":412},"su6XF","services",[407,415,417],{"class":416},"sYZai",":\n",[407,419,420,423],{"class":409,"line":20},[407,421,422],{"class":412},"  web",[407,424,417],{"class":416},[407,426,428,431,434],{"class":409,"line":427},3,[407,429,430],{"class":412},"    image",[407,432,433],{"class":416},":",[407,435,437],{"class":436},"spphp"," registry.example.com\u002Fteam\u002Fweb:1.4.2\n",[407,439,441,444,446,449,452],{"class":409,"line":440},4,[407,442,443],{"class":412},"    networks",[407,445,433],{"class":416},[407,447,448],{"class":416}," [",[407,450,451],{"class":436},"proxy-net",[407,453,454],{"class":416},"]\n",[407,456,458,461],{"class":409,"line":457},5,[407,459,460],{"class":412},"    deploy",[407,462,417],{"class":416},[407,464,466,469,471],{"class":409,"line":465},6,[407,467,468],{"class":412},"      replicas",[407,470,433],{"class":416},[407,472,474],{"class":473},"s-TwI"," 3\n",[407,476,478,481],{"class":409,"line":477},7,[407,479,480],{"class":412},"      labels",[407,482,417],{"class":416},[407,484,486,489],{"class":409,"line":485},8,[407,487,488],{"class":416},"        -",[407,490,491],{"class":436}," traefik.enable=true\n",[407,493,495,497],{"class":409,"line":494},9,[407,496,488],{"class":416},[407,498,499],{"class":436}," traefik.http.routers.web.rule=Host(`app.example.com`)\n",[407,501,503,505],{"class":409,"line":502},10,[407,504,488],{"class":416},[407,506,507],{"class":436}," traefik.http.routers.web.entrypoints=websecure\n",[407,509,511,513],{"class":409,"line":510},11,[407,512,488],{"class":416},[407,514,515],{"class":436}," traefik.http.routers.web.tls=true\n",[407,517,519,521],{"class":409,"line":518},12,[407,520,488],{"class":416},[407,522,523],{"class":436}," traefik.http.services.web.loadbalancer.server.port=8080\n",[407,525,527],{"class":409,"line":526},13,[407,528,530],{"emptyLinePlaceholder":529},true,"\n",[407,532,534,537],{"class":409,"line":533},14,[407,535,536],{"class":412},"networks",[407,538,417],{"class":416},[407,540,542,545],{"class":409,"line":541},15,[407,543,544],{"class":412},"  proxy-net",[407,546,417],{"class":416},[407,548,550,553,555],{"class":409,"line":549},16,[407,551,552],{"class":412},"    external",[407,554,433],{"class":416},[407,556,558],{"class":557},"sbBg2"," true\n",[407,560,562,565,567],{"class":409,"line":561},17,[407,563,564],{"class":412},"    name",[407,566,433],{"class":416},[407,568,569],{"class":436}," prod-proxy-net\n",[34,571,572],{},"Details that matter in production:",[147,574,575,589,601,607,613,619],{},[150,576,577,580,581,584,585,588],{},[37,578,579],{},"Three replicas, one per manager",", constrained to ",[98,582,583],{},"node.role == manager",", with ports published in ",[37,586,587],{},"host mode",". That bypasses Swarm's built-in routing mesh, which is simple but hides the client IP and adds a hop.",[150,590,591,596,597,600],{},[37,592,593,167],{},[98,594,595],{},"exposedByDefault: false"," A service must set ",[98,598,599],{},"traefik.enable=true",". No accidental exposure.",[150,602,603,606],{},[37,604,605],{},"The provider refreshes every 15 s"," and on Docker events, so new services appear quickly and dead tasks drop out.",[150,608,609,612],{},[37,610,611],{},"Shared middlewares"," live in one dynamic file: HSTS and security headers, rate limiting, compression, an IP allow-list for admin tools, and basic auth for the dashboard.",[150,614,615,618],{},[37,616,617],{},"A catch-all router"," with priority 1 sends any unknown hostname to a maintenance page (explained just below).",[150,620,621,624],{},[37,622,623],{},"Logs and metrics"," go out through OpenTelemetry and Prometheus, so I can monitor incoming traffic. They are also the signal the autoscaler reads. Monitoring is very important to me, and the observability section below explains what we monitor on this cluster.",[241,626,628],{"id":627},"why-a-catch-all-maintenance-page","Why a catch-all maintenance page",[34,630,631],{},"Every hostname served by the cluster points at the same address. When a service fails to start or is unreachable, Traefik falls back to its default response, a plain \"404 page not found\" (or a \"bad gateway\"). That error is not user-friendly: imagine opening an address and seeing a black page with small text saying \"404 page not found\".",[34,633,634],{},[635,636],"img",{"alt":637,"src":638},"Traefik's default \"404 page not found\" response, a black page with a single line of text","\u002Fimages\u002Fblog\u002Frunning-production-swarm\u002Ftraefik-404.png",[34,640,641],{},[154,642,643],{},"Traefik's default response when no router matches the request.",[34,645,646],{},"So I created a catch-all router that sends all unknown hostnames to a maintenance page. It is a small React app that reads the requested route, checks it against a YAML config, and decides what to show: either a nicely designed \"not found\" page or an \"under maintenance\" page. Every DNS record pointed at this cluster expects a service, so a request that matches nothing means the service has either failed to start or is unreachable, and a maintenance page is a better experience than a bare error.",[34,648,649],{},[635,650],{"alt":651,"src":652},"The maintenance page: a \"We are under maintenance\" message with an illustration and a Try Again button","\u002Fimages\u002Fblog\u002Frunning-production-swarm\u002Fmaintenance-page.png",[34,654,655],{},[154,656,657],{},"The maintenance page that unknown or unreachable hostnames now land on.",[31,659,660],{},[34,661,662,663,666,667,670,671,674,675,678,679,681,682,685,686,167],{},"One gotcha I hit: restart policies with ",[98,664,665],{},"max_attempts: 3"," and no ",[98,668,669],{},"window"," mean a service that crashes three times ",[154,672,673],{},"gives up forever",". After a network event, two of three Traefik replicas stopped restarting and one replica carried all traffic until someone ran ",[98,676,677],{},"docker service update --force",". For anything on the edge, set a ",[98,680,669],{}," or ",[98,683,684],{},"max_attempts: 0",", and ",[37,687,688],{},"alert when running replicas are below desired",[53,690],{},[42,692,694],{"id":693},"networks-one-front-door-private-back-rooms","Networks: one front door, private back rooms",[34,696,697],{},"Swarm overlay networks (VXLAN over UDP 4789, with gossip on 7946 and raft on 2377) let containers on different hosts talk as if they shared a switch. I use two kinds:",[147,699,700,706],{},[150,701,702,705],{},[37,703,704],{},"One shared proxy network."," Every web-facing container joins it, along with Traefik. About half of all services are on it.",[150,707,708,711,712,715],{},[37,709,710],{},"One private overlay per stack"," for databases, queues, and workers. Those containers are ",[154,713,714],{},"not"," attached to the proxy network, so Traefik can never reach them. Say you want to deploy a stack with a frontend, a backend, a worker (or consumer), Redis, and Postgres. Only the frontend and backend are reachable from Traefik, because those are the ones we expose to clients. The worker and the databases are internal only, and each stack defines its own internal network. An example stack file looks like this:",[260,717,719],{"className":401,"code":718,"language":403,"meta":266,"style":266},"services:\n  api:        # reachable from Traefik and from its own backend\n    networks: [proxy-net, backend]\n  worker:     # internal only\n    networks: [backend]\n  postgres:   # internal only\n    networks: [backend]\n\nnetworks:\n  proxy-net: { external: true, name: prod-proxy-net }\n  backend:   {}\n",[98,720,721,727,738,756,766,779,789,801,805,811,841],{"__ignoreMap":266},[407,722,723,725],{"class":409,"line":11},[407,724,413],{"class":412},[407,726,417],{"class":416},[407,728,729,732,734],{"class":409,"line":20},[407,730,731],{"class":412},"  api",[407,733,433],{"class":416},[407,735,737],{"class":736},"s8zF2","        # reachable from Traefik and from its own backend\n",[407,739,740,742,744,746,748,751,754],{"class":409,"line":427},[407,741,443],{"class":412},[407,743,433],{"class":416},[407,745,448],{"class":416},[407,747,451],{"class":436},[407,749,750],{"class":416},",",[407,752,753],{"class":436}," backend",[407,755,454],{"class":416},[407,757,758,761,763],{"class":409,"line":440},[407,759,760],{"class":412},"  worker",[407,762,433],{"class":416},[407,764,765],{"class":736},"     # internal only\n",[407,767,768,770,772,774,777],{"class":409,"line":457},[407,769,443],{"class":412},[407,771,433],{"class":416},[407,773,448],{"class":416},[407,775,776],{"class":436},"backend",[407,778,454],{"class":416},[407,780,781,784,786],{"class":409,"line":465},[407,782,783],{"class":412},"  postgres",[407,785,433],{"class":416},[407,787,788],{"class":736},"   # internal only\n",[407,790,791,793,795,797,799],{"class":409,"line":477},[407,792,443],{"class":412},[407,794,433],{"class":416},[407,796,448],{"class":416},[407,798,776],{"class":436},[407,800,454],{"class":416},[407,802,803],{"class":409,"line":485},[407,804,530],{"emptyLinePlaceholder":529},[407,806,807,809],{"class":409,"line":494},[407,808,536],{"class":412},[407,810,417],{"class":416},[407,812,813,815,817,820,823,825,828,830,833,835,838],{"class":409,"line":502},[407,814,544],{"class":412},[407,816,433],{"class":416},[407,818,819],{"class":416}," {",[407,821,822],{"class":412}," external",[407,824,433],{"class":416},[407,826,827],{"class":557}," true",[407,829,750],{"class":416},[407,831,832],{"class":412}," name",[407,834,433],{"class":416},[407,836,837],{"class":436}," prod-proxy-net",[407,839,840],{"class":416}," }\n",[407,842,843,846,848],{"class":409,"line":510},[407,844,845],{"class":412},"  backend",[407,847,433],{"class":416},[407,849,850],{"class":416},"   {}\n",[34,852,853],{},"That pattern gives you network isolation between stacks for free, without writing a single firewall rule.",[53,855],{},[42,857,859],{"id":858},"managers-workers-and-quorum","Managers, workers, and quorum",[34,861,862],{},"A Swarm cluster has two kinds of nodes.",[34,864,865,868,869,872],{},[37,866,867],{},"Managers"," hold the cluster's state in a Raft log and make the decisions: where a task lives, when to reschedule it, when a node has joined or left. ",[37,870,871],{},"Workers"," run the actual workloads. I keep application workloads on workers. Managers run only the pieces that make the cluster resilient or operable: Traefik, the autoscaler, and a few internal tools.",[34,874,875,876,879],{},"Raft needs a ",[37,877,878],{},"majority"," of managers to agree on anything. With three managers, you can lose one. Lose two and the cluster has no leader.",[881,882],"swarm-quorum",{},[34,884,885],{},"Things this taught me:",[147,887,888,894,900],{},[150,889,890,893],{},[37,891,892],{},"Existing containers keep running"," when quorum is lost, but that does not always mean users can still reach them (more on that below). What you lose is the ability to deploy, scale, drain, or reschedule, which is exactly when you need it most.",[150,895,896,899],{},[37,897,898],{},"Never restart two managers at once."," Confirm the other two are reachable first, and go one by one.",[150,901,902,905],{},[37,903,904],{},"Managers do little work",", so adding more is about resilience, not capacity.",[34,907,908],{},"The first point is a bit tricky. Our research suggested that you can still serve traffic when quorum is lost. That is not the case for this cluster, because Traefik reads the cluster state from Swarm and builds its routes from it. When the managers lose quorum and enter a read-only state, Traefik cannot read the cluster state and cannot route traffic to the right service. So even though the containers are still running, they are not reachable from outside.",[34,910,911],{},"The workaround is to define routes statically in Traefik instead of reading them from Swarm. That way even one manager can serve traffic, because the routes live in Traefik's own configuration. But it is a trade-off: static routes have to be updated by hand whenever a new service is added, and new services are added every single day.",[34,913,914],{},"I am considering a hybrid approach: define static routes for the critical services and read the rest from Swarm, so that critical services stay reachable even when quorum is lost. Another option is to let the autoscaler edit the routes configuration every time a new service is added.",[241,916,918],{"id":917},"two-kinds-of-workers","Two kinds of workers",[276,920,921,927],{},[150,922,923,926],{},[37,924,925],{},"Hand-joined workers."," A small number of nodes an admin joined manually with the CLI. Examples: a node that was originally meant for one app and now hosts team services, a dedicated data node, and a test VM.",[150,928,929,932],{},[37,930,931],{},"Autoscaling-group workers."," Because we run our own private cloud, most workers come from autoscaling groups in OpenStack. The group creates and removes VMs based on load, and they join the cluster without anyone touching them. This is the big win of putting an orchestrator on top of an elastic cloud: capacity is just a number.",[34,934,935,936,942],{},"Because those VMs are disposable, ",[37,937,938,939],{},"I label them ",[98,940,941],{},"lifecycle=ephemeral"," and treat everything on them as stateless. That brings me to labels.",[53,944],{},[42,946,948],{"id":947},"labels-and-placement","Labels and placement",[34,950,951,952,955],{},"Labels are the most useful feature in our cluster. A label is a key-value tag on a node, and a ",[37,953,954],{},"placement constraint"," on a service says \"only run on nodes where this is true\". Together they answer \"where does this container go?\" without anyone choosing a host.",[63,957,958,968],{},[66,959,960],{},[69,961,962,965],{},[72,963,964],{},"Label",[72,966,967],{},"Meaning",[85,969,970,979,989,999],{},[69,971,972,976],{},[90,973,974],{},[98,975,941],{},[90,977,978],{},"Disposable cloud worker. Workloads must be stateless.",[69,980,981,986],{},[90,982,983],{},[98,984,985],{},"pool=web",[90,987,988],{},"Sample pool for public-facing web apps",[69,990,991,996],{},[90,992,993],{},[98,994,995],{},"pool=batch",[90,997,998],{},"Sample pool for background workers and builds",[69,1000,1001,1006],{},[90,1002,1003],{},[98,1004,1005],{},"role=test",[90,1007,1008],{},"Preview and test deployments",[260,1010,1012],{"className":401,"code":1011,"language":403,"meta":266,"style":266},"deploy:\n  placement:\n    constraints:\n      - node.role == worker\n      - node.labels.pool == web\n",[98,1013,1014,1021,1028,1035,1043],{"__ignoreMap":266},[407,1015,1016,1019],{"class":409,"line":11},[407,1017,1018],{"class":412},"deploy",[407,1020,417],{"class":416},[407,1022,1023,1026],{"class":409,"line":20},[407,1024,1025],{"class":412},"  placement",[407,1027,417],{"class":416},[407,1029,1030,1033],{"class":409,"line":427},[407,1031,1032],{"class":412},"    constraints",[407,1034,417],{"class":416},[407,1036,1037,1040],{"class":409,"line":440},[407,1038,1039],{"class":416},"      -",[407,1041,1042],{"class":436}," node.role == worker\n",[407,1044,1045,1047],{"class":409,"line":457},[407,1046,1039],{"class":416},[407,1048,1049],{"class":436}," node.labels.pool == web\n",[34,1051,1052,1053,1056],{},"A task is scheduled only where ",[37,1054,1055],{},"all"," of a service's constraints are true. Experiment below. Select service A, then click nodes to relabel or drain them. Then hit \"Replay the incident\".",[1058,1059],"swarm-placement",{},[34,1061,1062,1063,1066],{},"That replay is a real event. During a storage stall in October, the workers in one pool showed I\u002FO pressure near 99% and tasks never got past ",[98,1064,1065],{},"Assigned",". I did not touch a single stack file. I labeled four healthy nodes into the pool, drained the stalled ones, and the replicas of the affected services came up within minutes:",[260,1068,1072],{"className":1069,"code":1070,"language":1071,"meta":266,"style":266},"language-bash shiki shiki-themes vitesse-light","docker node update --label-add pool=web \u003Chealthy-node>\ndocker node update --availability drain \u003Csick-node>\n# ...and undo afterwards\ndocker node update --label-rm pool \u003Chealthy-node>\ndocker node update --availability active \u003Cnode>\n","bash",[98,1073,1074,1107,1130,1135,1157],{"__ignoreMap":266},[407,1075,1076,1080,1083,1086,1090,1093,1097,1100,1104],{"class":409,"line":11},[407,1077,1079],{"class":1078},"sySUi","docker",[407,1081,1082],{"class":436}," node",[407,1084,1085],{"class":436}," update",[407,1087,1089],{"class":1088},"sEi1f"," --label-add",[407,1091,1092],{"class":436}," pool=web",[407,1094,1096],{"class":1095},"si04Y"," \u003C",[407,1098,1099],{"class":436},"healthy-nod",[407,1101,1103],{"class":1102},"suHK_","e",[407,1105,1106],{"class":1095},">\n",[407,1108,1109,1111,1113,1115,1118,1121,1123,1126,1128],{"class":409,"line":20},[407,1110,1079],{"class":1078},[407,1112,1082],{"class":436},[407,1114,1085],{"class":436},[407,1116,1117],{"class":1088}," --availability",[407,1119,1120],{"class":436}," drain",[407,1122,1096],{"class":1095},[407,1124,1125],{"class":436},"sick-nod",[407,1127,1103],{"class":1102},[407,1129,1106],{"class":1095},[407,1131,1132],{"class":409,"line":427},[407,1133,1134],{"class":736},"# ...and undo afterwards\n",[407,1136,1137,1139,1141,1143,1146,1149,1151,1153,1155],{"class":409,"line":440},[407,1138,1079],{"class":1078},[407,1140,1082],{"class":436},[407,1142,1085],{"class":436},[407,1144,1145],{"class":1088}," --label-rm",[407,1147,1148],{"class":436}," pool",[407,1150,1096],{"class":1095},[407,1152,1099],{"class":436},[407,1154,1103],{"class":1102},[407,1156,1106],{"class":1095},[407,1158,1159,1161,1163,1165,1167,1170,1172,1175,1177],{"class":409,"line":457},[407,1160,1079],{"class":1078},[407,1162,1082],{"class":436},[407,1164,1085],{"class":436},[407,1166,1117],{"class":1088},[407,1168,1169],{"class":436}," active",[407,1171,1096],{"class":1095},[407,1173,1174],{"class":436},"nod",[407,1176,1103],{"class":1102},[407,1178,1106],{"class":1095},[241,1180,1182],{"id":1181},"the-stateful-problem-honestly","The stateful problem, honestly",[34,1184,1185],{},"Containers are ephemeral, but databases are not. Kubernetes has StatefulSets and a storage ecosystem for this. We do not have an equivalent, so our approach is deliberately blunt:",[147,1187,1188,1199,1206,1213],{},[150,1189,1190,1191,1194,1195,1198],{},"One dedicated ",[37,1192,1193],{},"data node"," carries ",[98,1196,1197],{},"node-type=data",", and stateful services are constrained to it.",[150,1200,1201,1202,1205],{},"The data disk is a ",[37,1203,1204],{},"Ceph volume mounted on that VM"," under a fixed path, so the data does not live on the VM itself. It is also backed up frequently.",[150,1207,1208,1209,1212],{},"Services reach the database by a ",[37,1210,1211],{},"DNS name",", never by an IP address.",[150,1214,1215,1216,1219],{},"Some databases live ",[37,1217,1218],{},"outside Swarm"," entirely, on conventional servers.",[34,1221,1222,1223,1226,1227,1230],{},"This works, but it is a ",[37,1224,1225],{},"single point of failure",". In one outage, that node was down for days. Everything pinned to it stayed ",[98,1228,1229],{},"Pending",", and everything that depended on it (SSO, docs, dashboards) returned 5xx errors.",[34,1232,1233],{},"The Ceph disk and the DNS name are what turn that outage into a recovery procedure instead of a rebuild. If the data node fails, we can recover like this:",[276,1235,1236,1239,1242,1248],{},[150,1237,1238],{},"Make sure the failed VM can no longer write to the disk (power it off, or blocklist it in Ceph).",[150,1240,1241],{},"Attach the same Ceph volume to a healthy VM and mount it at the same path.",[150,1243,1244,1245,1247],{},"Give that VM the ",[98,1246,1197],{}," label, so the pinned services are scheduled onto it.",[150,1249,1250],{},"Repoint the DNS name at the new VM.",[34,1252,1253,1254,1256],{},"Does it work? Yes, with two caveats. First, a Ceph block device is meant to be written by one machine at a time, so step 1 is not optional. If two VMs mount the same disk, the filesystem gets corrupted. Second, it is a manual failover, so recovery time is detection plus those four steps plus DNS caching (keep the TTL low). It turns days into minutes, but it is not instant. If I could fix one thing next, it would be to automate those steps and add an alert on ",[98,1255,1229],{}," tasks.",[53,1258],{},[42,1260,1262],{"id":1261},"autoscaling-swarm-has-none-so-i-built-a-controller","Autoscaling: Swarm has none, so I built a controller",[34,1264,1265,1266,1269],{},"Swarm will keep N replicas alive. It will never decide that N should be 8 because a busy season is starting. To scale, you run ",[98,1267,1268],{},"docker service scale"," by hand.",[34,1271,1272,1273,1277,1278,1283],{},"I built ",[37,1274,1275],{},[98,1276,226],{}," (",[1279,1280,1282],"a",{"href":1281},"\u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm","the full write-up is here","), a single in-house service that runs on a manager. Services opt in with labels:",[260,1285,1287],{"className":401,"code":1286,"language":403,"meta":266,"style":266},"deploy:\n  replicas: 3\n  labels:\n    - swarmctl.enable=true\n    - swarmctl.min_replicas=3\n    - swarmctl.max_replicas=10\n    - swarmctl.scale_up_cpu_avg=70\n    - swarmctl.cooldown_after_up=300\n    - swarmctl.simulation_enabled=true   # check survivors before scaling down\n",[98,1288,1289,1295,1304,1311,1319,1326,1333,1340,1347],{"__ignoreMap":266},[407,1290,1291,1293],{"class":409,"line":11},[407,1292,1018],{"class":412},[407,1294,417],{"class":416},[407,1296,1297,1300,1302],{"class":409,"line":20},[407,1298,1299],{"class":412},"  replicas",[407,1301,433],{"class":416},[407,1303,474],{"class":473},[407,1305,1306,1309],{"class":409,"line":427},[407,1307,1308],{"class":412},"  labels",[407,1310,417],{"class":416},[407,1312,1313,1316],{"class":409,"line":440},[407,1314,1315],{"class":416},"    -",[407,1317,1318],{"class":436}," swarmctl.enable=true\n",[407,1320,1321,1323],{"class":409,"line":457},[407,1322,1315],{"class":416},[407,1324,1325],{"class":436}," swarmctl.min_replicas=3\n",[407,1327,1328,1330],{"class":409,"line":465},[407,1329,1315],{"class":416},[407,1331,1332],{"class":436}," swarmctl.max_replicas=10\n",[407,1334,1335,1337],{"class":409,"line":477},[407,1336,1315],{"class":416},[407,1338,1339],{"class":436}," swarmctl.scale_up_cpu_avg=70\n",[407,1341,1342,1344],{"class":409,"line":485},[407,1343,1315],{"class":416},[407,1345,1346],{"class":436}," swarmctl.cooldown_after_up=300\n",[407,1348,1349,1351,1354],{"class":409,"line":494},[407,1350,1315],{"class":416},[407,1352,1353],{"class":436}," swarmctl.simulation_enabled=true",[407,1355,1356],{"class":736},"   # check survivors before scaling down\n",[34,1358,1359,1360,1363,1364,1367],{},"It collects container metrics from ",[37,1361,1362],{},"cAdvisor"," (a ",[154,1365,1366],{},"global"," service: one instance on every node) and request metrics from Traefik, then adjusts replicas between the min and max. It also has a dashboard where admins can see replicas and scaling events.",[34,1369,1370],{},"The control loop is simple, but the details are where autoscalers go wrong. Throw a spike at this simulation and watch what the lag does:",[1372,1373],"swarm-autoscale",{},[34,1375,1376],{},"Four things worth noticing:",[276,1378,1379,1385,1391,1397],{},[150,1380,1381,1384],{},[37,1382,1383],{},"New tasks are not instant."," The pulsing, paler ones are still starting. CPU stays high until they are ready, and a naive controller would panic and add more.",[150,1386,1387,1390],{},[37,1388,1389],{},"Cooldowns exist to prevent flapping."," Without them, the controller scales up, sees a temporary dip, scales down, and repeats.",[150,1392,1393,1396],{},[37,1394,1395],{},"Scaling in is slower than scaling out"," on purpose. Being short of capacity is worse than having extra.",[150,1398,1399,1405,1406,1408],{},[37,1400,1401,1404],{},[98,1402,1403],{},"min_replicas"," is enforced."," If you manually scale a managed service below its minimum, ",[98,1407,226],{}," puts it back. Change the label first.",[34,1410,1411,1412,1277,1415,1418],{},"Scale-down is guarded by a ",[37,1413,1414],{},"removal simulation",[98,1416,1417],{},"simulation_enabled",", on by default). Before taking a replica away, swarmctl projects what the survivors would look like and refuses if they would end up too hot. That is what stops the classic loop of scaling down, overloading what is left, and immediately scaling back up.",[34,1420,1421],{},"If you want the full story (the collector, the decision engine and its simulation guards, the executor, and how it asks OpenStack for new VMs and drains nodes before removing them), I wrote it up separately:",[1423,1424],"reference",{"path":1281},[34,1426,1427],{},"Is this a replacement for the Kubernetes HPA? No. It is predictive only in a basic way, but it is good enough for this load profile, which follows a predictable seasonal pattern. If demand grows beyond what a hand-built controller should handle, that is a good reason to move.",[53,1429],{},[42,1431,1433],{"id":1432},"state-secrets-and-the-registry","State, secrets, and the registry",[147,1435,1436,1447,1453,1474],{},[150,1437,1438,1441,1442,681,1444,167],{},[37,1439,1440],{},"Images"," come from a private registry. If a node cannot pull from it, tasks sit in ",[98,1443,1065],{},[98,1445,1446],{},"Rejected: No such image",[150,1448,1449,1452],{},[37,1450,1451],{},"TLS material"," is kept as Docker secrets; ACME data lives in a volume on whichever manager holds it.",[150,1454,1455,1458,1459,1462,1463,1462,1466,1469,1470,1473],{},[37,1456,1457],{},"Configs"," (Traefik's static and dynamic files) are versioned Swarm configs. A warning: configs are immutable, so every change creates a new name, and without discipline you end up with ",[98,1460,1461],{},"traefik_dynamic",", ",[98,1464,1465],{},"_v2",[98,1467,1468],{},"_v4",", and a few typos. Clean up unused versions, and only trust what ",[98,1471,1472],{},"docker service inspect"," says is mounted.",[150,1475,1476,1479],{},[37,1477,1478],{},"Secrets are not environment variables."," Our inventory tooling deliberately never reads service environment variables.",[53,1481],{},[42,1483,1485],{"id":1484},"observability-you-cannot-run-what-you-cannot-see","Observability: you cannot run what you cannot see",[63,1487,1488,1498],{},[66,1489,1490],{},[69,1491,1492,1495],{},[72,1493,1494],{},"Piece",[72,1496,1497],{},"Purpose",[85,1499,1500,1508,1516,1524,1532,1540],{},[69,1501,1502,1505],{},[90,1503,1504],{},"Traefik metrics (Prometheus)",[90,1506,1507],{},"Request rate, latency, errors by router and service",[69,1509,1510,1513],{},[90,1511,1512],{},"Traefik access logs via OpenTelemetry",[90,1514,1515],{},"Central log pipeline",[69,1517,1518,1521],{},[90,1519,1520],{},"cAdvisor (global service)",[90,1522,1523],{},"Per-container CPU and memory, also fuel for the autoscaler",[69,1525,1526,1529],{},[90,1527,1528],{},"HAProxy stats + Prometheus exporter",[90,1530,1531],{},"Edge health. Keep the stats UI off public interfaces and use hashed credentials.",[69,1533,1534,1537],{},[90,1535,1536],{},"Grafana, Loki, Alloy",[90,1538,1539],{},"Dashboards and logs",[69,1541,1542,1545],{},[90,1543,1544],{},"Gatus",[90,1546,1547],{},"Uptime checks from outside, so you learn about failures before users tell you",[34,1549,1550],{},"I wrote up how this monitoring stack was built as a four-part series, from Prometheus and Grafana to Alloy, Loki and Tempo. Start with Part 1:",[1423,1552],{"path":1553},"\u002Fblog\u002Flgtm-stack\u002Fpart-1",[34,1555,1556,1557,1462,1561,685,1565,167],{},"The rest of the series: ",[1279,1558,1560],{"href":1559},"\u002Fblog\u002Flgtm-stack\u002Fpart-2","Part 2, Grafana Alloy and the push versus pull problem",[1279,1562,1564],{"href":1563},"\u002Fblog\u002Flgtm-stack\u002Fpart-3","Part 3, Loki, Tempo and the full observability picture",[1279,1566,1568],{"href":1567},"\u002Fblog\u002Flgtm-stack\u002Fpart-4","Part 4, the Enrollment API",[241,1570,1572],{"id":1571},"what-the-dashboards-look-like","What the dashboards look like",[34,1574,1575],{},"Everything above ends up in Grafana. Here are two of the dashboards.",[34,1577,1578,1581],{},[37,1579,1580],{},"HAProxy: edge health."," Servers up and down, backend status, queue depth, 4xx and 5xx rates, and connection and session rates. If the edge is unhealthy, this is where it shows first.",[34,1583,1584],{},[635,1585],{"alt":1586,"src":1587},"Grafana dashboard \"Apps Traffic - HAProxy\" showing threads, connections, session rate, servers up and down, backend status, queue depth and error rates","\u002Fimages\u002Fblog\u002Frunning-production-swarm\u002Fhaproxy-dashboard.png",[34,1589,1590],{},[154,1591,1592],{},"The HAProxy dashboard: three servers up, none down, an empty queue, and no 4xx or 5xx errors.",[34,1594,1595,1598],{},[37,1596,1597],{},"Traefik: traffic over time."," Total requests, the success percentage, request volume, and a breakdown of HTTP status codes over the last 12 hours. It is useful for spotting a sudden rise in 4xx or 5xx responses.",[34,1600,1601],{},[635,1602],{"alt":1603,"src":1604},"Grafana dashboard \"App Traffic\" showing 3.2 million requests, a 99.2 percent success rate, request volume and HTTP status codes over 12 hours","\u002Fimages\u002Fblog\u002Frunning-production-swarm\u002Ftraefik-traffic.png",[34,1606,1607],{},[154,1608,1609],{},"The Traefik traffic dashboard over the last 12 hours.",[34,1611,1612,1613,1616],{},"The metric I wish I had alerted on from day one is simple: ",[37,1614,1615],{},"running replicas below desired replicas",". Almost every incident I have had would have paged me within a minute.",[53,1618],{},[42,1620,1622],{"id":1621},"what-a-deployment-looks-like-now","What a deployment looks like now",[34,1624,1625],{},"Compare this with the standalone-host days:",[260,1627,1629],{"className":1069,"code":1628,"language":1071,"meta":266,"style":266},"# the CI\u002FCD agent runs on one manager and deploys straight from there\ndocker stack deploy --with-registry-auth -c stack.yml myapp\n",[98,1630,1631,1636],{"__ignoreMap":266},[407,1632,1633],{"class":409,"line":11},[407,1634,1635],{"class":736},"# the CI\u002FCD agent runs on one manager and deploys straight from there\n",[407,1637,1638,1640,1643,1646,1649,1652,1655],{"class":409,"line":20},[407,1639,1079],{"class":1078},[407,1641,1642],{"class":436}," stack",[407,1644,1645],{"class":436}," deploy",[407,1647,1648],{"class":1088}," --with-registry-auth",[407,1650,1651],{"class":1088}," -c",[407,1653,1654],{"class":436}," stack.yml",[407,1656,1657],{"class":436}," myapp\n",[147,1659,1660,1663,1666],{},[150,1661,1662],{},"No credential in the pipeline at all. The CI\u002FCD agent runs on one of the managers, so it talks to the local Docker socket and deploys from there. Nothing needs to be stored in the pipeline or sent over the network.",[150,1664,1665],{},"Swarm picks the nodes based on constraints.",[150,1667,1668],{},"Rolling updates are built in:",[260,1670,1672],{"className":401,"code":1671,"language":403,"meta":266,"style":266},"deploy:\n  update_config:\n    parallelism: 1\n    delay: 30s\n    order: stop-first\n    failure_action: rollback\n",[98,1673,1674,1680,1687,1697,1707,1717],{"__ignoreMap":266},[407,1675,1676,1678],{"class":409,"line":11},[407,1677,1018],{"class":412},[407,1679,417],{"class":416},[407,1681,1682,1685],{"class":409,"line":20},[407,1683,1684],{"class":412},"  update_config",[407,1686,417],{"class":416},[407,1688,1689,1692,1694],{"class":409,"line":427},[407,1690,1691],{"class":412},"    parallelism",[407,1693,433],{"class":416},[407,1695,1696],{"class":473}," 1\n",[407,1698,1699,1702,1704],{"class":409,"line":440},[407,1700,1701],{"class":412},"    delay",[407,1703,433],{"class":416},[407,1705,1706],{"class":436}," 30s\n",[407,1708,1709,1712,1714],{"class":409,"line":457},[407,1710,1711],{"class":412},"    order",[407,1713,433],{"class":416},[407,1715,1716],{"class":436}," stop-first\n",[407,1718,1719,1722,1724],{"class":409,"line":465},[407,1720,1721],{"class":412},"    failure_action",[407,1723,433],{"class":416},[407,1725,1726],{"class":436}," rollback\n",[147,1728,1729],{},[150,1730,1731,1732,1734],{},"Test deployments land on the ",[98,1733,1005],{}," node by constraint, so previews never compete with production.",[53,1736],{},[42,1738,1740],{"id":1739},"what-i-would-improve","What I would improve",[34,1742,1743],{},"I would rather publish this list than pretend it is empty:",[147,1745,1746,1752,1758,1766],{},[150,1747,1748,1751],{},[37,1749,1750],{},"Unicast VRRP"," for the manager in a different network, to remove the duplicate-VIP risk.",[150,1753,1754,1757],{},[37,1755,1756],{},"Fix restart policies"," on edge services and alert on replica deficits.",[150,1759,1760,1763,1764,1256],{},[37,1761,1762],{},"Faster data node recovery."," The data disk is on Ceph, so a failed node means remounting that disk elsewhere and repointing the DNS name. The remaining problem is how fast that can be done, so I want to script it and alert on ",[98,1765,1229],{},[150,1767,1768,1771,1772,1774],{},[37,1769,1770],{},"Move the edge and internal tools off the managers."," Traefik and tools like ",[98,1773,226],{}," could move to a separate pool of nodes, so that the managers are used only for cluster state and not for running workloads.",[53,1776],{},[42,1778,1780],{"id":1779},"why-swarm-was-the-right-call","Why Swarm was the right call",[34,1782,1783],{},"Looking back, these are the reasons I chose it:",[276,1785,1786,1792,1798,1804],{},[150,1787,1788,1791],{},[37,1789,1790],{},"The learning curve fit the team."," An admin who understands Linux, Docker, and networks can be productive in days.",[150,1793,1794,1797],{},[37,1795,1796],{},"It solved the real problems",": placement, self-healing, rolling updates, one deployment endpoint, and overlay networking.",[150,1799,1800,1803],{},[37,1801,1802],{},"The gaps were small enough to patch."," Autoscaling and state are the two big ones, and a small controller plus a labeled data node covered both.",[150,1805,1806,1809],{},[37,1807,1808],{},"It keeps the Kubernetes door open."," The workloads are containers behind labels and health checks. When capacity and demand justify it, there is a deliberate path forward, and the team will arrive with the container fundamentals already solid.",[34,1811,1812],{},"If you are where I was, running containers on a few hosts with deployment credentials scattered everywhere, Swarm is a very reasonable next rung. Do not wait for the perfect platform. Get the fundamentals right (an edge you can fail over, routing from labels, placement from node labels, and alerts on replica deficits), and you will learn what you actually need from the next platform.",[53,1814],{},[42,1816,1818],{"id":1817},"appendix-a-runbook-that-earned-its-place","Appendix: a runbook that earned its place",[260,1820,1822],{"className":1069,"code":1821,"language":1071,"meta":266,"style":266},"# Who holds the VIP?\nfor h in \u003Cmgr-a> \u003Cmgr-b> \u003Cmgr-c>; do ssh $h 'hostname; ip -br a | grep \u003Cvip>'; done\n\n# Services below desired replicas\ndocker service ls --format '{{.Name}} {{.Replicas}}' \\\n  | awk '{split($2,a,\"\u002F\"); if(a[1]!=a[2]) print}'\n\n# Why is it not running?\ndocker service ps \u003Csvc> --no-trunc\n\n# Gossip trouble (run on each manager)\njournalctl -u docker --since \"5 min ago\" | grep -c -E 'unknown to memberlist|rejoinClusterBootstrap'\n\n# Reset a service that gave up\ndocker service update --force \u003Csvc>\n\n# Is a worker disk-stalled? (I\u002FO pressure near 99, many D-state processes)\nssh \u003Cnode> 'cat \u002Fproc\u002Fpressure\u002Fio; ps -eo stat | grep -c ^D'\n\n# Restart a manager's daemon: confirm the other two are Reachable first, one at a time\ndocker node ls --format '{{.Hostname}} {{.ManagerStatus}}'\nsudo systemctl restart docker\n",[98,1823,1824,1829,1890,1894,1899,1922,1938,1942,1947,1969,1973,1978,2019,2023,2028,2047,2051,2056,2077,2082,2088,2106],{"__ignoreMap":266},[407,1825,1826],{"class":409,"line":11},[407,1827,1828],{"class":736},"# Who holds the VIP?\n",[407,1830,1831,1834,1838,1841,1843,1846,1849,1851,1854,1856,1858,1861,1863,1866,1869,1872,1875,1879,1882,1885,1887],{"class":409,"line":20},[407,1832,1833],{"class":557},"for",[407,1835,1837],{"class":1836},"svycV"," h",[407,1839,1840],{"class":557}," in",[407,1842,1096],{"class":1095},[407,1844,1845],{"class":436},"mgr-a",[407,1847,1848],{"class":1095},">",[407,1850,1096],{"class":1095},[407,1852,1853],{"class":436},"mgr-b",[407,1855,1848],{"class":1095},[407,1857,1096],{"class":1095},[407,1859,1860],{"class":436},"mgr-c",[407,1862,1848],{"class":1095},[407,1864,1865],{"class":416},";",[407,1867,1868],{"class":557}," do",[407,1870,1871],{"class":1078}," ssh",[407,1873,1874],{"class":1836}," $h",[407,1876,1878],{"class":1877},"sSP4y"," '",[407,1880,1881],{"class":436},"hostname; ip -br a | grep \u003Cvip>",[407,1883,1884],{"class":1877},"'",[407,1886,1865],{"class":416},[407,1888,1889],{"class":557}," done\n",[407,1891,1892],{"class":409,"line":427},[407,1893,530],{"emptyLinePlaceholder":529},[407,1895,1896],{"class":409,"line":440},[407,1897,1898],{"class":736},"# Services below desired replicas\n",[407,1900,1901,1903,1906,1909,1912,1914,1917,1919],{"class":409,"line":457},[407,1902,1079],{"class":1078},[407,1904,1905],{"class":436}," service",[407,1907,1908],{"class":436}," ls",[407,1910,1911],{"class":1088}," --format",[407,1913,1878],{"class":1877},[407,1915,1916],{"class":436},"{{.Name}} {{.Replicas}}",[407,1918,1884],{"class":1877},[407,1920,1921],{"class":1088}," \\\n",[407,1923,1924,1927,1930,1932,1935],{"class":409,"line":465},[407,1925,1926],{"class":1095},"  |",[407,1928,1929],{"class":1078}," awk",[407,1931,1878],{"class":1877},[407,1933,1934],{"class":436},"{split($2,a,\"\u002F\"); if(a[1]!=a[2]) print}",[407,1936,1937],{"class":1877},"'\n",[407,1939,1940],{"class":409,"line":477},[407,1941,530],{"emptyLinePlaceholder":529},[407,1943,1944],{"class":409,"line":485},[407,1945,1946],{"class":736},"# Why is it not running?\n",[407,1948,1949,1951,1953,1956,1958,1961,1964,1966],{"class":409,"line":494},[407,1950,1079],{"class":1078},[407,1952,1905],{"class":436},[407,1954,1955],{"class":436}," ps",[407,1957,1096],{"class":1095},[407,1959,1960],{"class":436},"sv",[407,1962,1963],{"class":1102},"c",[407,1965,1848],{"class":1095},[407,1967,1968],{"class":1088}," --no-trunc\n",[407,1970,1971],{"class":409,"line":502},[407,1972,530],{"emptyLinePlaceholder":529},[407,1974,1975],{"class":409,"line":510},[407,1976,1977],{"class":736},"# Gossip trouble (run on each manager)\n",[407,1979,1980,1983,1986,1989,1992,1995,1998,2001,2004,2007,2009,2012,2014,2017],{"class":409,"line":518},[407,1981,1982],{"class":1078},"journalctl",[407,1984,1985],{"class":1088}," -u",[407,1987,1988],{"class":436}," docker",[407,1990,1991],{"class":1088}," --since",[407,1993,1994],{"class":1877}," \"",[407,1996,1997],{"class":436},"5 min ago",[407,1999,2000],{"class":1877},"\"",[407,2002,2003],{"class":1095}," |",[407,2005,2006],{"class":1078}," grep",[407,2008,1651],{"class":1088},[407,2010,2011],{"class":1088}," -E",[407,2013,1878],{"class":1877},[407,2015,2016],{"class":436},"unknown to memberlist|rejoinClusterBootstrap",[407,2018,1937],{"class":1877},[407,2020,2021],{"class":409,"line":526},[407,2022,530],{"emptyLinePlaceholder":529},[407,2024,2025],{"class":409,"line":533},[407,2026,2027],{"class":736},"# Reset a service that gave up\n",[407,2029,2030,2032,2034,2036,2039,2041,2043,2045],{"class":409,"line":541},[407,2031,1079],{"class":1078},[407,2033,1905],{"class":436},[407,2035,1085],{"class":436},[407,2037,2038],{"class":1088}," --force",[407,2040,1096],{"class":1095},[407,2042,1960],{"class":436},[407,2044,1963],{"class":1102},[407,2046,1106],{"class":1095},[407,2048,2049],{"class":409,"line":549},[407,2050,530],{"emptyLinePlaceholder":529},[407,2052,2053],{"class":409,"line":561},[407,2054,2055],{"class":736},"# Is a worker disk-stalled? (I\u002FO pressure near 99, many D-state processes)\n",[407,2057,2059,2062,2064,2066,2068,2070,2072,2075],{"class":409,"line":2058},18,[407,2060,2061],{"class":1078},"ssh",[407,2063,1096],{"class":1095},[407,2065,1174],{"class":436},[407,2067,1103],{"class":1102},[407,2069,1848],{"class":1095},[407,2071,1878],{"class":1877},[407,2073,2074],{"class":436},"cat \u002Fproc\u002Fpressure\u002Fio; ps -eo stat | grep -c ^D",[407,2076,1937],{"class":1877},[407,2078,2080],{"class":409,"line":2079},19,[407,2081,530],{"emptyLinePlaceholder":529},[407,2083,2085],{"class":409,"line":2084},20,[407,2086,2087],{"class":736},"# Restart a manager's daemon: confirm the other two are Reachable first, one at a time\n",[407,2089,2091,2093,2095,2097,2099,2101,2104],{"class":409,"line":2090},21,[407,2092,1079],{"class":1078},[407,2094,1082],{"class":436},[407,2096,1908],{"class":436},[407,2098,1911],{"class":1088},[407,2100,1878],{"class":1877},[407,2102,2103],{"class":436},"{{.Hostname}} {{.ManagerStatus}}",[407,2105,1937],{"class":1877},[407,2107,2109,2112,2115,2118],{"class":409,"line":2108},22,[407,2110,2111],{"class":1078},"sudo",[407,2113,2114],{"class":436}," systemctl",[407,2116,2117],{"class":436}," restart",[407,2119,2120],{"class":436}," docker\n",[2122,2123,2124],"style",{},"html pre.shiki code .su6XF, html code.shiki .su6XF{--shiki-default:#998418}html pre.shiki code .sYZai, html code.shiki .sYZai{--shiki-default:#999999}html pre.shiki code .spphp, html code.shiki .spphp{--shiki-default:#B56959}html pre.shiki code .s-TwI, html code.shiki .s-TwI{--shiki-default:#2F798A}html pre.shiki code .sbBg2, html code.shiki .sbBg2{--shiki-default:#1E754F}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s8zF2, html code.shiki .s8zF2{--shiki-default:#A0ADA0}html pre.shiki code .sySUi, html code.shiki .sySUi{--shiki-default:#59873A}html pre.shiki code .sEi1f, html code.shiki .sEi1f{--shiki-default:#A65E2B}html pre.shiki code .si04Y, html code.shiki .si04Y{--shiki-default:#AB5959}html pre.shiki code .suHK_, html code.shiki .suHK_{--shiki-default:#393A34}html pre.shiki code .svycV, html code.shiki .svycV{--shiki-default:#B07D48}html pre.shiki code .sSP4y, html code.shiki .sSP4y{--shiki-default:#B5695977}",{"title":266,"searchDepth":20,"depth":20,"links":2126},[2127,2128,2129,2130,2134,2137,2138,2141,2144,2145,2146,2149,2150,2151,2152],{"id":44,"depth":20,"text":45},{"id":57,"depth":20,"text":58},{"id":179,"depth":20,"text":180},{"id":238,"depth":20,"text":239,"children":2131},[2132,2133],{"id":243,"depth":427,"text":244},{"id":304,"depth":427,"text":305},{"id":390,"depth":20,"text":391,"children":2135},[2136],{"id":627,"depth":427,"text":628},{"id":693,"depth":20,"text":694},{"id":858,"depth":20,"text":859,"children":2139},[2140],{"id":917,"depth":427,"text":918},{"id":947,"depth":20,"text":948,"children":2142},[2143],{"id":1181,"depth":427,"text":1182},{"id":1261,"depth":20,"text":1262},{"id":1432,"depth":20,"text":1433},{"id":1484,"depth":20,"text":1485,"children":2147},[2148],{"id":1571,"depth":427,"text":1572},{"id":1621,"depth":20,"text":1622},{"id":1739,"depth":20,"text":1740},{"id":1779,"depth":20,"text":1780},{"id":1817,"depth":20,"text":1818},"Blog","2025-11-10","Our company is early in its containerization journey, so I started with Docker Swarm instead of Kubernetes. This is the whole architecture: edge, routing, placement, autoscaling, failures, and what I would do differently.","md",{},"\u002Fblog\u002Frunning-production-swarm","18 min",{"title":26,"description":2155},"blog\u002Frunning-production-swarm\u002Findex",[114,2163,2164,2165,2166,2167,2168],"OpenStack","DevOps","Traefik","HAProxy","Containers","Infrastructure","\u002Fimages\u002Fthumbnails\u002Frunning-production-swarm\u002Fthumbnail.png","jMzQi7unz9USxaSINQe5QRHkl-09zxpXZanzAO9BtZA",{"id":2172,"title":2173,"body":2174,"category":2153,"date":10875,"description":10876,"extension":2156,"meta":10877,"navigation":529,"path":1281,"readTime":10878,"seo":10879,"stem":10880,"tags":10881,"thumbnail":10884,"__hash__":10885},"blog\u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm\u002Findex.md","Building an Autoscaler for Docker Swarm: Inside swarmctl",{"type":28,"value":2175,"toc":10831},[2176,2191,2196,2200,2207,2213,2219,2233,2240,2263,2266,2298,2300,2303,2305,2307,2311,2321,2324,2335,2337,2341,2348,2352,2355,2423,2427,2434,2987,2990,3004,3008,3011,3017,3030,3045,3052,3056,3059,3081,3088,3240,3258,3262,3265,3392,3396,3406,3492,3499,3503,3521,3530,3541,3547,3784,3786,3790,3793,3814,3817,3821,3824,3830,4267,4270,4306,4509,4513,4516,4660,4670,4673,4678,4681,4857,4860,4864,4867,4954,4961,5029,5043,5049,5053,5056,5176,5180,5191,5198,5432,5438,5500,5503,5656,5671,5674,5677,5680,5687,5694,5794,5798,5804,5932,5936,5943,5947,6138,6141,6169,6173,6228,6232,6235,6469,6473,6479,6483,6490,6497,6507,6600,6614,6685,6695,6780,6786,6789,6792,6796,6807,6823,6833,6835,6839,6842,6846,6950,6954,6960,7092,7096,7245,7249,7256,7503,7524,7528,7531,7533,7537,7543,7731,7738,7742,8043,8046,8050,8057,8176,8189,8193,8199,8275,8282,8286,8292,8315,8474,8478,8489,8663,8667,8674,8678,8688,8690,8694,8707,8711,8721,8986,9004,9010,9270,9280,9284,9294,9453,9458,9745,9748,9850,9854,9864,10318,10329,10332,10335,10345,10354,10525,10529,10532,10540,10543,10545,10549,10552,10558,10564,10570,10582,10588,10598,10600,10604,10607,10775,10781,10787,10789,10793,10814,10828],[31,2177,2178],{},[34,2179,2180,2183,2184,2187,2188,2190],{},[37,2181,2182],{},"Series note:"," this is the deep dive behind the autoscaling section of ",[1279,2185,2186],{"href":2158},"Running Production Swarm",", which covers the whole architecture it plugs into. You do not need to read that first, but it explains where ",[98,2189,226],{}," sits: on a manager node, reading cAdvisor and Traefik metrics, scaling services that opt in through labels.",[34,2192,2193],{},[154,2194,2195],{},"A deep dive into swarmctl: a lightweight daemon that watches Swarm services, decides when to scale, executes the change, and coordinates node-level scaling with OpenStack.",[42,2197,2199],{"id":2198},"why-build-this-at-all","Why build this at all?",[34,2201,2202,2203,2206],{},"Kubernetes gives you the Horizontal Pod Autoscaler, the Cluster Autoscaler, and a metrics\npipeline out of the box. Docker Swarm gives you ",[98,2204,2205],{},"docker service scale \u003Cname> \u003Cn>"," and nothing else.\nNo reactive scaling, no load-aware placement, no graceful node drain before a VM is terminated.",[34,2208,2209,2210,2212],{},"If you are running Swarm in production (and plenty of people still are, because it is simple,\nstable, and fits on three small VMs without a control-plane tax), you have to build that layer\nyourself. ",[98,2211,226],{}," is the layer I built.",[34,2214,2215,2216,2218],{},"It does ",[37,2217,714],{}," try to be Kubernetes. It is explicitly not:",[147,2220,2221,2227,2230],{},[150,2222,2223,2224],{},"a replacement for ",[98,2225,2226],{},"docker stack deploy",[150,2228,2229],{},"a full observability stack (no Prometheus, no Grafana)",[150,2231,2232],{},"a node provisioner (OpenStack's autoscaling group handles that)",[34,2234,2235,2236,2239],{},"It ",[154,2237,2238],{},"is"," a daemon that:",[276,2241,2242,2245,2248,2254,2257,2260],{},[150,2243,2244],{},"collects per-service metrics every cycle (CPU, memory, CPU throttling, request rate, latency, errors, connections, pending\u002Frejected task counts),",[150,2246,2247],{},"resolves a scaling policy per service from labels, templates, and defaults,",[150,2249,2250,2251,750],{},"feeds metrics + policy into a ",[37,2252,2253],{},"pure decision engine",[150,2255,2256],{},"executes the resulting decision through the Docker API with verification and partial-success handling,",[150,2258,2259],{},"signals OpenStack to add or remove whole VMs when the cluster itself runs out of room,",[150,2261,2262],{},"drains nodes gracefully before OpenStack terminates them.",[34,2264,2265],{},"In this post I walk through the whole thing, with the real code. There are five parts worth\nunderstanding in depth:",[147,2267,2268,2274,2280,2286,2292],{},[150,2269,2270,2273],{},[37,2271,2272],{},"The collector",": turning raw cAdvisor and Traefik numbers into clean per-service metrics.",[150,2275,2276,2279],{},[37,2277,2278],{},"The policy layer",": resolver (config → policy) and the engine (metrics + policy → decision).",[150,2281,2282,2285],{},[37,2283,2284],{},"The executor",": making a decision real, safely.",[150,2287,2288,2291],{},[37,2289,2290],{},"The watch loop",": the stateful orchestrator that ties it all together.",[150,2293,2294,2297],{},[37,2295,2296],{},"The OpenStack integration",": scale-out signals, scale-in approval, and graceful drain.",[53,2299],{},[34,2301,2302],{},"Before the code, here is the control loop in miniature. This is a simplified model, not swarmctl itself: watch how startup lag, cooldowns, and a minimum replica count shape what an autoscaler does with a traffic spike. The rest of the post is about the guards that make the real one behave this way.",[1372,2304],{},[53,2306],{},[42,2308,2310],{"id":2309},"architecture-at-a-glance","Architecture at a glance",[34,2312,2313,2314,2317,2318,2320],{},"There are three paths through swarmctl, and the diagram below follows each one: the watch-loop cycle that does most of the work, OpenStack asking permission to scale in, and the drain that runs just before a VM is terminated. Press ",[37,2315,2316],{},"Play"," to watch a path, or ",[37,2319,190],{}," to go one stage at a time. Each stage shows the code that does the work, and you can click any box to jump to the step where it is used.",[2322,2323],"swarmctl-architecture",{},[34,2325,2326,2327,2330,2331,2334],{},"Two concurrency boundaries matter: the ",[37,2328,2329],{},"watch loop"," runs on the main thread on a fixed\ninterval, and the ",[37,2332,2333],{},"hook server"," runs on a daemon thread so OpenStack can call in at any time.\nThey share a small amount of state (last scale-out timestamp per pool) through a lock-protected\nmodule-level dict.",[53,2336],{},[42,2338,2340],{"id":2339},"part-1-the-collector","Part 1: The Collector",[34,2342,2343,2344,2347],{},"The engine is only as good as its inputs. The collector's job is to produce one clean\n",[98,2345,2346],{},"ServiceMetrics"," object per enabled service, every cycle, without ever blocking the loop\nfor longer than one interval.",[241,2349,2351],{"id":2350},"only-manage-what-opts-in","Only manage what opts in",[34,2353,2354],{},"Services opt in with a Swarm label. Nothing is scaled unless it explicitly asks to be:",[260,2356,2358],{"className":401,"code":2357,"language":403,"meta":266,"style":266},"deploy:\n  labels:\n    - \"swarmctl.enable=true\"\n    - \"swarmctl.policy=blazor-standard\"   # named template\n    - \"swarmctl.max_replicas=8\"           # override any single field\n    - \"swarmctl.scale_up_cpu_avg=60\"\n",[98,2359,2360,2366,2372,2384,2398,2412],{"__ignoreMap":266},[407,2361,2362,2364],{"class":409,"line":11},[407,2363,1018],{"class":412},[407,2365,417],{"class":416},[407,2367,2368,2370],{"class":409,"line":20},[407,2369,1308],{"class":412},[407,2371,417],{"class":416},[407,2373,2374,2376,2378,2381],{"class":409,"line":427},[407,2375,1315],{"class":416},[407,2377,1994],{"class":1877},[407,2379,2380],{"class":436},"swarmctl.enable=true",[407,2382,2383],{"class":1877},"\"\n",[407,2385,2386,2388,2390,2393,2395],{"class":409,"line":440},[407,2387,1315],{"class":416},[407,2389,1994],{"class":1877},[407,2391,2392],{"class":436},"swarmctl.policy=blazor-standard",[407,2394,2000],{"class":1877},[407,2396,2397],{"class":736},"   # named template\n",[407,2399,2400,2402,2404,2407,2409],{"class":409,"line":457},[407,2401,1315],{"class":416},[407,2403,1994],{"class":1877},[407,2405,2406],{"class":436},"swarmctl.max_replicas=8",[407,2408,2000],{"class":1877},[407,2410,2411],{"class":736},"           # override any single field\n",[407,2413,2414,2416,2418,2421],{"class":409,"line":465},[407,2415,1315],{"class":416},[407,2417,1994],{"class":1877},[407,2419,2420],{"class":436},"swarmctl.scale_up_cpu_avg=60",[407,2422,2383],{"class":1877},[241,2424,2426],{"id":2425},"concurrent-collection-with-a-hard-deadline","Concurrent collection with a hard deadline",[34,2428,2429,2430,2433],{},"Collecting stats for a service means hitting cAdvisor on whichever node each task runs on.\nDone serially, total time is ",[98,2431,2432],{},"O(n_services × latency)",". The collector fans out across a\nthread pool and enforces a deadline so one slow service can never stall the whole loop:",[260,2435,2439],{"className":2436,"code":2437,"language":2438,"meta":266,"style":266},"language-python shiki shiki-themes vitesse-light","def collect_all(self) -> List[Tuple[ServiceMetrics, Dict[str, str]]]:\n    s = get_settings()\n    self._docker.refresh_infra_maps()\n\n    enabled = self._docker.list_enabled_services()\n    if not enabled:\n        return []\n\n    # Scrape Traefik ONCE per cycle. Calling get_metrics() per service would reset\n    # the rate-state on every call and give zero deltas for all but the first service.\n    traffic_cache = self._docker.traefik.get_metrics()\n    deadline = time.monotonic() + s.collection_timeout\n\n    with ThreadPoolExecutor(max_workers=min(len(enabled), 16),\n                            thread_name_prefix=\"collect\") as pool:\n        future_to_svc = {\n            pool.submit(self._docker.get_service_metrics, svc.id, traffic_cache): (svc, labels)\n            for svc, labels in enabled\n        }\n        remaining = max(0.1, deadline - time.monotonic())\n        done, not_done = _wait(future_to_svc, timeout=remaining)\n\n        if not_done:\n            skipped = [future_to_svc[f][0].name for f in not_done]\n            logger.warning(f\"Collection deadline: {len(not_done)} skipped this cycle: {skipped}\")\n            for fut in not_done:\n                fut.cancel()\n        ...\n","python",[98,2440,2441,2492,2506,2523,2527,2548,2561,2569,2573,2578,2583,2608,2637,2641,2678,2701,2711,2765,2783,2788,2820,2852,2856,2867,2907,2954,2968,2981],{"__ignoreMap":266},[407,2442,2443,2446,2449,2452,2455,2458,2461,2464,2467,2470,2472,2474,2476,2479,2481,2484,2486,2489],{"class":409,"line":11},[407,2444,2445],{"class":1095},"def",[407,2447,2448],{"class":1078}," collect_all",[407,2450,2451],{"class":416},"(",[407,2453,2454],{"class":1102},"self",[407,2456,2457],{"class":416},")",[407,2459,2460],{"class":416}," ->",[407,2462,2463],{"class":1102}," List",[407,2465,2466],{"class":416},"[",[407,2468,2469],{"class":1102},"Tuple",[407,2471,2466],{"class":416},[407,2473,2346],{"class":1102},[407,2475,750],{"class":416},[407,2477,2478],{"class":1102}," Dict",[407,2480,2466],{"class":416},[407,2482,2483],{"class":412},"str",[407,2485,750],{"class":416},[407,2487,2488],{"class":412}," str",[407,2490,2491],{"class":416},"]]]:\n",[407,2493,2494,2497,2500,2503],{"class":409,"line":20},[407,2495,2496],{"class":1102},"    s ",[407,2498,2499],{"class":416},"=",[407,2501,2502],{"class":1102}," get_settings",[407,2504,2505],{"class":416},"()\n",[407,2507,2508,2511,2513,2516,2518,2521],{"class":409,"line":427},[407,2509,2510],{"class":1088},"    self",[407,2512,167],{"class":416},[407,2514,2515],{"class":1102},"_docker",[407,2517,167],{"class":416},[407,2519,2520],{"class":1102},"refresh_infra_maps",[407,2522,2505],{"class":416},[407,2524,2525],{"class":409,"line":440},[407,2526,530],{"emptyLinePlaceholder":529},[407,2528,2529,2532,2534,2537,2539,2541,2543,2546],{"class":409,"line":457},[407,2530,2531],{"class":1102},"    enabled ",[407,2533,2499],{"class":416},[407,2535,2536],{"class":1088}," self",[407,2538,167],{"class":416},[407,2540,2515],{"class":1102},[407,2542,167],{"class":416},[407,2544,2545],{"class":1102},"list_enabled_services",[407,2547,2505],{"class":416},[407,2549,2550,2553,2556,2559],{"class":409,"line":465},[407,2551,2552],{"class":557},"    if",[407,2554,2555],{"class":1095}," not",[407,2557,2558],{"class":1102}," enabled",[407,2560,417],{"class":416},[407,2562,2563,2566],{"class":409,"line":477},[407,2564,2565],{"class":557},"        return",[407,2567,2568],{"class":416}," []\n",[407,2570,2571],{"class":409,"line":485},[407,2572,530],{"emptyLinePlaceholder":529},[407,2574,2575],{"class":409,"line":494},[407,2576,2577],{"class":736},"    # Scrape Traefik ONCE per cycle. Calling get_metrics() per service would reset\n",[407,2579,2580],{"class":409,"line":502},[407,2581,2582],{"class":736},"    # the rate-state on every call and give zero deltas for all but the first service.\n",[407,2584,2585,2588,2590,2592,2594,2596,2598,2601,2603,2606],{"class":409,"line":510},[407,2586,2587],{"class":1102},"    traffic_cache ",[407,2589,2499],{"class":416},[407,2591,2536],{"class":1088},[407,2593,167],{"class":416},[407,2595,2515],{"class":1102},[407,2597,167],{"class":416},[407,2599,2600],{"class":1102},"traefik",[407,2602,167],{"class":416},[407,2604,2605],{"class":1102},"get_metrics",[407,2607,2505],{"class":416},[407,2609,2610,2613,2615,2618,2620,2623,2626,2629,2632,2634],{"class":409,"line":518},[407,2611,2612],{"class":1102},"    deadline ",[407,2614,2499],{"class":416},[407,2616,2617],{"class":1102}," time",[407,2619,167],{"class":416},[407,2621,2622],{"class":1102},"monotonic",[407,2624,2625],{"class":416},"()",[407,2627,2628],{"class":1095}," +",[407,2630,2631],{"class":1102}," s",[407,2633,167],{"class":416},[407,2635,2636],{"class":1102},"collection_timeout\n",[407,2638,2639],{"class":409,"line":526},[407,2640,530],{"emptyLinePlaceholder":529},[407,2642,2643,2646,2649,2651,2654,2656,2659,2661,2664,2666,2669,2672,2675],{"class":409,"line":533},[407,2644,2645],{"class":557},"    with",[407,2647,2648],{"class":1102}," ThreadPoolExecutor",[407,2650,2451],{"class":416},[407,2652,2653],{"class":1836},"max_workers",[407,2655,2499],{"class":416},[407,2657,2658],{"class":412},"min",[407,2660,2451],{"class":416},[407,2662,2663],{"class":412},"len",[407,2665,2451],{"class":416},[407,2667,2668],{"class":1102},"enabled",[407,2670,2671],{"class":416},"),",[407,2673,2674],{"class":473}," 16",[407,2676,2677],{"class":416},"),\n",[407,2679,2680,2683,2685,2687,2690,2692,2694,2697,2699],{"class":409,"line":541},[407,2681,2682],{"class":1836},"                            thread_name_prefix",[407,2684,2499],{"class":416},[407,2686,2000],{"class":1877},[407,2688,2689],{"class":436},"collect",[407,2691,2000],{"class":1877},[407,2693,2457],{"class":416},[407,2695,2696],{"class":557}," as",[407,2698,1148],{"class":1102},[407,2700,417],{"class":416},[407,2702,2703,2706,2708],{"class":409,"line":549},[407,2704,2705],{"class":1102},"        future_to_svc ",[407,2707,2499],{"class":416},[407,2709,2710],{"class":416}," {\n",[407,2712,2713,2716,2718,2721,2723,2725,2727,2729,2731,2734,2736,2739,2741,2744,2746,2749,2752,2754,2757,2759,2762],{"class":409,"line":561},[407,2714,2715],{"class":1102},"            pool",[407,2717,167],{"class":416},[407,2719,2720],{"class":1102},"submit",[407,2722,2451],{"class":416},[407,2724,2454],{"class":1088},[407,2726,167],{"class":416},[407,2728,2515],{"class":1102},[407,2730,167],{"class":416},[407,2732,2733],{"class":1102},"get_service_metrics",[407,2735,750],{"class":416},[407,2737,2738],{"class":1102}," svc",[407,2740,167],{"class":416},[407,2742,2743],{"class":1102},"id",[407,2745,750],{"class":416},[407,2747,2748],{"class":1102}," traffic_cache",[407,2750,2751],{"class":416},"):",[407,2753,1277],{"class":416},[407,2755,2756],{"class":1102},"svc",[407,2758,750],{"class":416},[407,2760,2761],{"class":1102}," labels",[407,2763,2764],{"class":416},")\n",[407,2766,2767,2770,2772,2774,2777,2780],{"class":409,"line":2058},[407,2768,2769],{"class":557},"            for",[407,2771,2738],{"class":1102},[407,2773,750],{"class":416},[407,2775,2776],{"class":1102}," labels ",[407,2778,2779],{"class":557},"in",[407,2781,2782],{"class":1102}," enabled\n",[407,2784,2785],{"class":409,"line":2079},[407,2786,2787],{"class":416},"        }\n",[407,2789,2790,2793,2795,2798,2800,2803,2805,2808,2811,2813,2815,2817],{"class":409,"line":2084},[407,2791,2792],{"class":1102},"        remaining ",[407,2794,2499],{"class":416},[407,2796,2797],{"class":412}," max",[407,2799,2451],{"class":416},[407,2801,2802],{"class":473},"0.1",[407,2804,750],{"class":416},[407,2806,2807],{"class":1102}," deadline ",[407,2809,2810],{"class":1095},"-",[407,2812,2617],{"class":1102},[407,2814,167],{"class":416},[407,2816,2622],{"class":1102},[407,2818,2819],{"class":416},"())\n",[407,2821,2822,2825,2827,2830,2832,2835,2837,2840,2842,2845,2847,2850],{"class":409,"line":2090},[407,2823,2824],{"class":1102},"        done",[407,2826,750],{"class":416},[407,2828,2829],{"class":1102}," not_done ",[407,2831,2499],{"class":416},[407,2833,2834],{"class":1102}," _wait",[407,2836,2451],{"class":416},[407,2838,2839],{"class":1102},"future_to_svc",[407,2841,750],{"class":416},[407,2843,2844],{"class":1836}," timeout",[407,2846,2499],{"class":416},[407,2848,2849],{"class":1102},"remaining",[407,2851,2764],{"class":416},[407,2853,2854],{"class":409,"line":2108},[407,2855,530],{"emptyLinePlaceholder":529},[407,2857,2859,2862,2865],{"class":409,"line":2858},23,[407,2860,2861],{"class":557},"        if",[407,2863,2864],{"class":1102}," not_done",[407,2866,417],{"class":416},[407,2868,2870,2873,2875,2877,2879,2881,2884,2887,2890,2893,2896,2898,2901,2903,2905],{"class":409,"line":2869},24,[407,2871,2872],{"class":1102},"            skipped ",[407,2874,2499],{"class":416},[407,2876,448],{"class":416},[407,2878,2839],{"class":1102},[407,2880,2466],{"class":416},[407,2882,2883],{"class":1102},"f",[407,2885,2886],{"class":416},"][",[407,2888,2889],{"class":473},"0",[407,2891,2892],{"class":416},"].",[407,2894,2895],{"class":1102},"name ",[407,2897,1833],{"class":557},[407,2899,2900],{"class":1102}," f ",[407,2902,2779],{"class":557},[407,2904,2864],{"class":1102},[407,2906,454],{"class":416},[407,2908,2910,2913,2915,2918,2920,2922,2925,2928,2930,2932,2935,2937,2940,2943,2945,2948,2950,2952],{"class":409,"line":2909},25,[407,2911,2912],{"class":1102},"            logger",[407,2914,167],{"class":416},[407,2916,2917],{"class":1102},"warning",[407,2919,2451],{"class":416},[407,2921,2883],{"class":1095},[407,2923,2924],{"class":436},"\"Collection deadline: ",[407,2926,2927],{"class":1088},"{",[407,2929,2663],{"class":412},[407,2931,2451],{"class":416},[407,2933,2934],{"class":1102},"not_done",[407,2936,2457],{"class":416},[407,2938,2939],{"class":1088},"}",[407,2941,2942],{"class":436}," skipped this cycle: ",[407,2944,2927],{"class":1088},[407,2946,2947],{"class":1102},"skipped",[407,2949,2939],{"class":1088},[407,2951,2000],{"class":436},[407,2953,2764],{"class":416},[407,2955,2957,2959,2962,2964,2966],{"class":409,"line":2956},26,[407,2958,2769],{"class":557},[407,2960,2961],{"class":1102}," fut ",[407,2963,2779],{"class":557},[407,2965,2864],{"class":1102},[407,2967,417],{"class":416},[407,2969,2971,2974,2976,2979],{"class":409,"line":2970},27,[407,2972,2973],{"class":1102},"                fut",[407,2975,167],{"class":416},[407,2977,2978],{"class":1102},"cancel",[407,2980,2505],{"class":416},[407,2982,2984],{"class":409,"line":2983},28,[407,2985,2986],{"class":1088},"        ...\n",[34,2988,2989],{},"Two design decisions worth calling out:",[147,2991,2992,2998],{},[150,2993,2994,2997],{},[37,2995,2996],{},"Traefik is scraped once per cycle, not once per service."," The scraper keeps per-node\nrate state (previous counter values). If you scraped per service, every call but the first\nwould see a zero time delta and report a zero rate. Scrape once, pass the cache down.",[150,2999,3000,3003],{},[37,3001,3002],{},"Concurrency is capped at 16."," Beyond that, the Docker daemon socket becomes the\nbottleneck, not the thread count.",[241,3005,3007],{"id":3006},"computing-cpu-the-right-way","Computing CPU the right way",[34,3009,3010],{},"Docker\u002FcAdvisor expose raw cumulative CPU counters, not percentages. You need two samples and\nthe standard delta formula:",[260,3012,3015],{"className":3013,"code":3014,"language":265},[263],"cpu_delta    = cpu_usage.total      - precpu_usage.total\nsystem_delta = system_cpu_usage     - pre_system_cpu_usage\ncpu_percent  = (cpu_delta \u002F system_delta) * online_cpus * 100\n",[98,3016,3014],{"__ignoreMap":266},[34,3018,3019,3021,3022,3025,3026,3029],{},[98,3020,226],{}," pulls two samples straight from cAdvisor (",[98,3023,3024],{},"?count=2",") rather than Docker's own\nstats endpoint, because cAdvisor also exposes ",[37,3027,3028],{},"CPU throttling",", which is the single most\nuseful signal for a CPU-limited container:",[260,3031,3033],{"className":2436,"code":3032,"language":2438,"meta":266,"style":266},"# cpu_throttle_pct: % of CFS scheduling periods that were throttled. A container\n#                    showing low raw CPU% but high throttle% is hitting its CPU ceiling.\n",[98,3034,3035,3040],{"__ignoreMap":266},[407,3036,3037],{"class":409,"line":11},[407,3038,3039],{"class":736},"# cpu_throttle_pct: % of CFS scheduling periods that were throttled. A container\n",[407,3041,3042],{"class":409,"line":20},[407,3043,3044],{"class":736},"#                    showing low raw CPU% but high throttle% is hitting its CPU ceiling.\n",[34,3046,3047,3048,3051],{},"A container pinned to ",[98,3049,3050],{},"0.5"," CPU can sit at \"50% CPU\" forever and look healthy while being\nheavily throttled and serving slow responses. Raw CPU percentage hides that; throttle percentage\nexposes it. This is why the engine has a dedicated throttle trigger.",[241,3053,3055],{"id":3054},"two-memory-denominators","Two memory denominators",[34,3057,3058],{},"Memory gets reported two ways because there are two meaningful questions:",[147,3060,3061,3069],{},[150,3062,3063,3068],{},[37,3064,3065],{},[98,3066,3067],{},"node_mem_pct",": memory as a percentage of the host node's total RAM. Always present,\nalways a consistent denominator across services. This is what the policy engine uses.",[150,3070,3071,3076,3077,3080],{},[37,3072,3073],{},[98,3074,3075],{},"limit_mem_pct",": memory as a percentage of the container's own limit. Only present when\na real limit is set; ",[98,3078,3079],{},"None"," otherwise.",[34,3082,3083,3084,3087],{},"The \"no limit set\" case is a trap. cAdvisor reports an unlimited container's limit as a kernel\nsentinel (",[98,3085,3086],{},"9223372036854771712",", i.e. max int64). Dividing by that gives a meaningless\nnear-zero percentage, so the code treats anything over 128 GB as \"no limit\":",[260,3089,3091],{"className":2436,"code":3090,"language":2438,"meta":266,"style":266},"_MAX_SANE_LIMIT = 128 * 1024 * 1024 * 1024  # 128 GB\nmem_limit_bytes = entry.get(\"spec\", {}).get(\"memory\", {}).get(\"limit\", 0)\nhas_real_limit  = 0 \u003C mem_limit_bytes \u003C _MAX_SANE_LIMIT\nlimit_mem_pct   = (mem_bytes \u002F mem_limit_bytes * 100.0) if has_real_limit else None\n",[98,3092,3093,3121,3183,3203],{"__ignoreMap":266},[407,3094,3095,3098,3101,3104,3107,3110,3112,3114,3116,3118],{"class":409,"line":11},[407,3096,3097],{"class":1088},"_MAX_SANE_LIMIT",[407,3099,3100],{"class":416}," =",[407,3102,3103],{"class":473}," 128",[407,3105,3106],{"class":1095}," *",[407,3108,3109],{"class":473}," 1024",[407,3111,3106],{"class":1095},[407,3113,3109],{"class":473},[407,3115,3106],{"class":1095},[407,3117,3109],{"class":473},[407,3119,3120],{"class":736},"  # 128 GB\n",[407,3122,3123,3126,3128,3131,3133,3136,3138,3140,3143,3145,3147,3150,3152,3154,3156,3159,3161,3163,3165,3167,3169,3171,3174,3176,3178,3181],{"class":409,"line":20},[407,3124,3125],{"class":1102},"mem_limit_bytes ",[407,3127,2499],{"class":416},[407,3129,3130],{"class":1102}," entry",[407,3132,167],{"class":416},[407,3134,3135],{"class":1102},"get",[407,3137,2451],{"class":416},[407,3139,2000],{"class":1877},[407,3141,3142],{"class":436},"spec",[407,3144,2000],{"class":1877},[407,3146,750],{"class":416},[407,3148,3149],{"class":416}," {}).",[407,3151,3135],{"class":1102},[407,3153,2451],{"class":416},[407,3155,2000],{"class":1877},[407,3157,3158],{"class":436},"memory",[407,3160,2000],{"class":1877},[407,3162,750],{"class":416},[407,3164,3149],{"class":416},[407,3166,3135],{"class":1102},[407,3168,2451],{"class":416},[407,3170,2000],{"class":1877},[407,3172,3173],{"class":436},"limit",[407,3175,2000],{"class":1877},[407,3177,750],{"class":416},[407,3179,3180],{"class":473}," 0",[407,3182,2764],{"class":416},[407,3184,3185,3188,3190,3192,3194,3197,3200],{"class":409,"line":427},[407,3186,3187],{"class":1102},"has_real_limit  ",[407,3189,2499],{"class":416},[407,3191,3180],{"class":473},[407,3193,1096],{"class":1095},[407,3195,3196],{"class":1102}," mem_limit_bytes ",[407,3198,3199],{"class":1095},"\u003C",[407,3201,3202],{"class":1088}," _MAX_SANE_LIMIT\n",[407,3204,3205,3208,3210,3212,3215,3218,3220,3223,3226,3228,3231,3234,3237],{"class":409,"line":440},[407,3206,3207],{"class":1102},"limit_mem_pct   ",[407,3209,2499],{"class":416},[407,3211,1277],{"class":416},[407,3213,3214],{"class":1102},"mem_bytes ",[407,3216,3217],{"class":1095},"\u002F",[407,3219,3196],{"class":1102},[407,3221,3222],{"class":1095},"*",[407,3224,3225],{"class":473}," 100.0",[407,3227,2457],{"class":416},[407,3229,3230],{"class":557}," if",[407,3232,3233],{"class":1102}," has_real_limit ",[407,3235,3236],{"class":557},"else",[407,3238,3239],{"class":557}," None\n",[34,3241,3242,3243,3246,3247,3250,3251,3254,3255,3257],{},"Memory also uses ",[98,3244,3245],{},"working_set"," (not raw ",[98,3248,3249],{},"usage",") to match what ",[98,3252,3253],{},"docker stats"," shows:\n",[98,3256,3245],{}," excludes reclaimable page cache, so you measure what the process actually needs.",[241,3259,3261],{"id":3260},"a-stale-data-guard","A stale-data guard",[34,3263,3264],{},"cAdvisor's default housekeeping interval is 10 seconds. If your watch interval is 5 seconds,\nevery other cycle you'd re-read the exact same sample, compute a zero delta, and report 0% CPU,\nwhich would spuriously suppress scale-up triggers on alternating cycles. The collector caches\nthe last result keyed by timestamp and returns the cache when the sample hasn't advanced:",[260,3266,3268],{"className":2436,"code":3267,"language":2438,"meta":266,"style":266},"latest_ts = stats_list[-1].get(\"timestamp\", \"\")\ncached_ts, cached_result = self._cadvisor_cache.get(container_id, (\"\", None))\nif latest_ts and latest_ts == cached_ts and cached_result is not None:\n    return cached_result   # don't report a false 0%\n",[98,3269,3270,3307,3350,3381],{"__ignoreMap":266},[407,3271,3272,3275,3277,3280,3282,3284,3287,3289,3291,3293,3295,3298,3300,3302,3305],{"class":409,"line":11},[407,3273,3274],{"class":1102},"latest_ts ",[407,3276,2499],{"class":416},[407,3278,3279],{"class":1102}," stats_list",[407,3281,2466],{"class":416},[407,3283,2810],{"class":1095},[407,3285,3286],{"class":473},"1",[407,3288,2892],{"class":416},[407,3290,3135],{"class":1102},[407,3292,2451],{"class":416},[407,3294,2000],{"class":1877},[407,3296,3297],{"class":436},"timestamp",[407,3299,2000],{"class":1877},[407,3301,750],{"class":416},[407,3303,3304],{"class":1877}," \"\"",[407,3306,2764],{"class":416},[407,3308,3309,3312,3314,3317,3319,3321,3323,3326,3328,3330,3332,3335,3337,3339,3342,3344,3347],{"class":409,"line":20},[407,3310,3311],{"class":1102},"cached_ts",[407,3313,750],{"class":416},[407,3315,3316],{"class":1102}," cached_result ",[407,3318,2499],{"class":416},[407,3320,2536],{"class":1088},[407,3322,167],{"class":416},[407,3324,3325],{"class":1102},"_cadvisor_cache",[407,3327,167],{"class":416},[407,3329,3135],{"class":1102},[407,3331,2451],{"class":416},[407,3333,3334],{"class":1102},"container_id",[407,3336,750],{"class":416},[407,3338,1277],{"class":416},[407,3340,3341],{"class":1877},"\"\"",[407,3343,750],{"class":416},[407,3345,3346],{"class":557}," None",[407,3348,3349],{"class":416},"))\n",[407,3351,3352,3355,3358,3361,3363,3366,3369,3371,3373,3375,3377,3379],{"class":409,"line":427},[407,3353,3354],{"class":557},"if",[407,3356,3357],{"class":1102}," latest_ts ",[407,3359,3360],{"class":1095},"and",[407,3362,3357],{"class":1102},[407,3364,3365],{"class":1095},"==",[407,3367,3368],{"class":1102}," cached_ts ",[407,3370,3360],{"class":1095},[407,3372,3316],{"class":1102},[407,3374,2238],{"class":1095},[407,3376,2555],{"class":1095},[407,3378,3346],{"class":557},[407,3380,417],{"class":416},[407,3382,3383,3386,3389],{"class":409,"line":440},[407,3384,3385],{"class":557},"    return",[407,3387,3388],{"class":1102}," cached_result   ",[407,3390,3391],{"class":736},"# don't report a false 0%\n",[241,3393,3395],{"id":3394},"replica-state-and-stuck-pending","Replica state and \"stuck pending\"",[34,3397,3398,3399,3402,3403,3405],{},"The collector counts tasks by state and, critically, distinguishes ",[154,3400,3401],{},"why"," a task is pending.\nA task pending because of insufficient cluster capacity is a scale-out signal. A task pending\nbecause of a bad image or a missing secret is a broken deployment that must ",[37,3404,714],{}," trigger\nscaling:",[260,3407,3409],{"className":2436,"code":3408,"language":2438,"meta":266,"style":266},"_CAPACITY_PATTERNS = (\"no suitable node\", \"insufficient resources\",\n                      \"scheduling constraints not satisfied\")\n_CONFIG_ERROR_PATTERNS = (\"no such image\", \"pull access denied\", \"secret not found\", ...)\n",[98,3410,3411,3439,3451],{"__ignoreMap":266},[407,3412,3413,3416,3418,3420,3422,3425,3427,3429,3431,3434,3436],{"class":409,"line":11},[407,3414,3415],{"class":1088},"_CAPACITY_PATTERNS",[407,3417,3100],{"class":416},[407,3419,1277],{"class":416},[407,3421,2000],{"class":1877},[407,3423,3424],{"class":436},"no suitable node",[407,3426,2000],{"class":1877},[407,3428,750],{"class":416},[407,3430,1994],{"class":1877},[407,3432,3433],{"class":436},"insufficient resources",[407,3435,2000],{"class":1877},[407,3437,3438],{"class":416},",\n",[407,3440,3441,3444,3447,3449],{"class":409,"line":20},[407,3442,3443],{"class":1877},"                      \"",[407,3445,3446],{"class":436},"scheduling constraints not satisfied",[407,3448,2000],{"class":1877},[407,3450,2764],{"class":416},[407,3452,3453,3456,3458,3460,3462,3465,3467,3469,3471,3474,3476,3478,3480,3483,3485,3487,3490],{"class":409,"line":427},[407,3454,3455],{"class":1088},"_CONFIG_ERROR_PATTERNS",[407,3457,3100],{"class":416},[407,3459,1277],{"class":416},[407,3461,2000],{"class":1877},[407,3463,3464],{"class":436},"no such image",[407,3466,2000],{"class":1877},[407,3468,750],{"class":416},[407,3470,1994],{"class":1877},[407,3472,3473],{"class":436},"pull access denied",[407,3475,2000],{"class":1877},[407,3477,750],{"class":416},[407,3479,1994],{"class":1877},[407,3481,3482],{"class":436},"secret not found",[407,3484,2000],{"class":1877},[407,3486,750],{"class":416},[407,3488,3489],{"class":1088}," ...",[407,3491,2764],{"class":416},[34,3493,3494,3495,3498],{},"Only capacity-pending tasks feed ",[98,3496,3497],{},"longest_pending_seconds",", which the engine turns into a\nhost-scale signal.",[241,3500,3502],{"id":3501},"service-name-matching-with-traefik","Service name matching with Traefik",[34,3504,3505,3506,3509,3510,3513,3514,293,3517,3520],{},"Traefik labels services as ",[98,3507,3508],{},"myapp_blazor@docker","; Docker calls them ",[98,3511,3512],{},"myapp_blazor",". The scraper\nstrips the ",[98,3515,3516],{},"@docker",[98,3518,3519],{},"@swarm"," provider suffix with a regex, and the actual Traefik service name\nis derived from the service's own Traefik label rather than guessed:",[260,3522,3524],{"className":2436,"code":3523,"language":2438,"meta":266,"style":266},"# traefik.http.services.\u003Cname>.loadbalancer.server.port  →  \u003Cname>\n",[98,3525,3526],{"__ignoreMap":266},[407,3527,3528],{"class":409,"line":11},[407,3529,3523],{"class":736},[34,3531,3532,3533,3536,3537,3540],{},"Because Traefik can run on several nodes (one per manager in our cluster), the scraper discovers all Traefik\nendpoints and keeps ",[37,3534,3535],{},"per-node"," rate state, summing deltas across nodes each cycle. Counter\nresets (a Traefik container restart) are guarded with ",[98,3538,3539],{},"max(0.0, current - previous)"," so a reset\nnever produces a negative rate.",[34,3542,3543,3544,3546],{},"The output of all this is one ",[98,3545,2346],{}," per service:",[260,3548,3550],{"className":2436,"code":3549,"language":2438,"meta":266,"style":266},"class ServiceMetrics(BaseModel):\n    service: str\n    replicas_running: int\n    replicas_desired: int\n    replicas_pending: int\n    replicas_failed: int\n    replicas_rejected: int = 0\n\n    avg_cpu: float; max_cpu: float\n    avg_throttle: float; max_throttle: float      # the throttle signal\n    avg_mem: float; max_mem: float                 # % of node RAM\n    avg_limit_mem: Optional[float]; max_limit_mem: Optional[float]  # % of container limit\n\n    request_rate: float; avg_response_ms: float; error_rate: float; open_connections: int\n    longest_pending_seconds: int\n    tasks: List[TaskStat]\n",[98,3551,3552,3569,3579,3589,3598,3607,3616,3631,3635,3653,3672,3691,3725,3729,3759,3768],{"__ignoreMap":266},[407,3553,3554,3557,3561,3563,3566],{"class":409,"line":11},[407,3555,3556],{"class":1095},"class",[407,3558,3560],{"class":3559},"sUxyF"," ServiceMetrics",[407,3562,2451],{"class":416},[407,3564,3565],{"class":1078},"BaseModel",[407,3567,3568],{"class":416},"):\n",[407,3570,3571,3574,3576],{"class":409,"line":20},[407,3572,3573],{"class":1102},"    service",[407,3575,433],{"class":416},[407,3577,3578],{"class":412}," str\n",[407,3580,3581,3584,3586],{"class":409,"line":427},[407,3582,3583],{"class":1102},"    replicas_running",[407,3585,433],{"class":416},[407,3587,3588],{"class":412}," int\n",[407,3590,3591,3594,3596],{"class":409,"line":440},[407,3592,3593],{"class":1102},"    replicas_desired",[407,3595,433],{"class":416},[407,3597,3588],{"class":412},[407,3599,3600,3603,3605],{"class":409,"line":457},[407,3601,3602],{"class":1102},"    replicas_pending",[407,3604,433],{"class":416},[407,3606,3588],{"class":412},[407,3608,3609,3612,3614],{"class":409,"line":465},[407,3610,3611],{"class":1102},"    replicas_failed",[407,3613,433],{"class":416},[407,3615,3588],{"class":412},[407,3617,3618,3621,3623,3626,3628],{"class":409,"line":477},[407,3619,3620],{"class":1102},"    replicas_rejected",[407,3622,433],{"class":416},[407,3624,3625],{"class":412}," int",[407,3627,3100],{"class":416},[407,3629,3630],{"class":473}," 0\n",[407,3632,3633],{"class":409,"line":485},[407,3634,530],{"emptyLinePlaceholder":529},[407,3636,3637,3640,3642,3645,3648,3650],{"class":409,"line":494},[407,3638,3639],{"class":1102},"    avg_cpu",[407,3641,433],{"class":416},[407,3643,3644],{"class":412}," float",[407,3646,3647],{"class":1102},"; max_cpu",[407,3649,433],{"class":416},[407,3651,3652],{"class":412}," float\n",[407,3654,3655,3658,3660,3662,3665,3667,3669],{"class":409,"line":502},[407,3656,3657],{"class":1102},"    avg_throttle",[407,3659,433],{"class":416},[407,3661,3644],{"class":412},[407,3663,3664],{"class":1102},"; max_throttle",[407,3666,433],{"class":416},[407,3668,3644],{"class":412},[407,3670,3671],{"class":736},"      # the throttle signal\n",[407,3673,3674,3677,3679,3681,3684,3686,3688],{"class":409,"line":510},[407,3675,3676],{"class":1102},"    avg_mem",[407,3678,433],{"class":416},[407,3680,3644],{"class":412},[407,3682,3683],{"class":1102},"; max_mem",[407,3685,433],{"class":416},[407,3687,3644],{"class":412},[407,3689,3690],{"class":736},"                 # % of node RAM\n",[407,3692,3693,3696,3698,3701,3703,3706,3709,3712,3714,3716,3718,3720,3722],{"class":409,"line":518},[407,3694,3695],{"class":1102},"    avg_limit_mem",[407,3697,433],{"class":416},[407,3699,3700],{"class":1102}," Optional",[407,3702,2466],{"class":416},[407,3704,3705],{"class":412},"float",[407,3707,3708],{"class":416},"]",[407,3710,3711],{"class":1102},"; max_limit_mem",[407,3713,433],{"class":416},[407,3715,3700],{"class":1102},[407,3717,2466],{"class":416},[407,3719,3705],{"class":412},[407,3721,3708],{"class":416},[407,3723,3724],{"class":736},"  # % of container limit\n",[407,3726,3727],{"class":409,"line":526},[407,3728,530],{"emptyLinePlaceholder":529},[407,3730,3731,3734,3736,3738,3741,3743,3745,3748,3750,3752,3755,3757],{"class":409,"line":533},[407,3732,3733],{"class":1102},"    request_rate",[407,3735,433],{"class":416},[407,3737,3644],{"class":412},[407,3739,3740],{"class":1102},"; avg_response_ms",[407,3742,433],{"class":416},[407,3744,3644],{"class":412},[407,3746,3747],{"class":1102},"; error_rate",[407,3749,433],{"class":416},[407,3751,3644],{"class":412},[407,3753,3754],{"class":1102},"; open_connections",[407,3756,433],{"class":416},[407,3758,3588],{"class":412},[407,3760,3761,3764,3766],{"class":409,"line":541},[407,3762,3763],{"class":1102},"    longest_pending_seconds",[407,3765,433],{"class":416},[407,3767,3588],{"class":412},[407,3769,3770,3773,3775,3777,3779,3782],{"class":409,"line":549},[407,3771,3772],{"class":1102},"    tasks",[407,3774,433],{"class":416},[407,3776,2463],{"class":1102},[407,3778,2466],{"class":416},[407,3780,3781],{"class":1102},"TaskStat",[407,3783,454],{"class":416},[53,3785],{},[42,3787,3789],{"id":3788},"part-2-the-policy-layer","Part 2: The Policy Layer",[34,3791,3792],{},"The policy layer has two halves that must stay strictly separated:",[147,3794,3795,3804],{},[150,3796,3797,3800,3801,167],{},[37,3798,3799],{},"The resolver"," does I\u002FO. It reads config, merges labels and templates, reloads when the\nconfig file changes. It produces a ",[98,3802,3803],{},"ResolvedPolicy",[150,3805,3806,3809,3810,3813],{},[37,3807,3808],{},"The engine"," does logic. It takes ",[98,3811,3812],{},"(metrics, policy, timers)"," and returns a decision. No\nI\u002FO, no clock reads, no randomness. Same inputs → same outputs, every time.",[34,3815,3816],{},"That separation is what makes the engine fully unit-testable without mocking anything, and it's\nwhy the test file for it is ~28 KB on its own.",[241,3818,3820],{"id":3819},"the-resolver-three-layers-highest-wins","The resolver: three layers, highest wins",[34,3822,3823],{},"Every policy field is resolved from three layers, in priority order:",[260,3825,3828],{"className":3826,"code":3827,"language":265},[263],"1. per-service label   (swarmctl.scale_up_cpu_avg=60)   highest\n2. named template      (swarmctl.policy=blazor-standard) middle\n3. global default      (from config.json)                lowest, always present\n",[98,3829,3827],{"__ignoreMap":266},[260,3831,3833],{"className":2436,"code":3832,"language":2438,"meta":266,"style":266},"def resolve(self, service, labels) -> ResolvedPolicy | None:\n    self._reload_if_changed()                 # hot-reload config.json on mtime change\n    resolved = self.config[\"defaults\"].copy()  # layer 3\n    resolved[\"service\"] = service.name\n    resolved[\"enabled\"] = True\n\n    template = labels.get(\"swarmctl.policy\")   # layer 2\n    if template and template in self.config[\"templates\"]:\n        resolved.update(self.config[\"templates\"][template])\n\n    for label_key, (field, typ) in label_map.items():   # layer 1\n        if label_key in labels:\n            try:\n                val = labels[label_key]\n                resolved[field] = (val.lower() == \"true\") if typ is bool else typ(val)\n            except (ValueError, AttributeError):\n                pass   # invalid label value → silently fall through to template\u002Fdefault\n\n    policy = ResolvedPolicy(**resolved)\n    return policy if self._validate(policy) else None\n",[98,3834,3835,3868,3882,3915,3940,3959,3963,3990,4021,4055,4059,4097,4110,4117,4133,4191,4208,4216,4220,4240],{"__ignoreMap":266},[407,3836,3837,3839,3842,3844,3846,3848,3850,3852,3854,3856,3858,3861,3864,3866],{"class":409,"line":11},[407,3838,2445],{"class":1095},[407,3840,3841],{"class":1078}," resolve",[407,3843,2451],{"class":416},[407,3845,2454],{"class":1102},[407,3847,750],{"class":416},[407,3849,1905],{"class":1102},[407,3851,750],{"class":416},[407,3853,2761],{"class":1102},[407,3855,2457],{"class":416},[407,3857,2460],{"class":416},[407,3859,3860],{"class":1102}," ResolvedPolicy ",[407,3862,3863],{"class":1095},"|",[407,3865,3346],{"class":557},[407,3867,417],{"class":416},[407,3869,3870,3872,3874,3877,3879],{"class":409,"line":20},[407,3871,2510],{"class":1088},[407,3873,167],{"class":416},[407,3875,3876],{"class":1102},"_reload_if_changed",[407,3878,2625],{"class":416},[407,3880,3881],{"class":736},"                 # hot-reload config.json on mtime change\n",[407,3883,3884,3887,3889,3891,3893,3896,3898,3900,3903,3905,3907,3910,3912],{"class":409,"line":427},[407,3885,3886],{"class":1102},"    resolved ",[407,3888,2499],{"class":416},[407,3890,2536],{"class":1088},[407,3892,167],{"class":416},[407,3894,3895],{"class":1102},"config",[407,3897,2466],{"class":416},[407,3899,2000],{"class":1877},[407,3901,3902],{"class":436},"defaults",[407,3904,2000],{"class":1877},[407,3906,2892],{"class":416},[407,3908,3909],{"class":1102},"copy",[407,3911,2625],{"class":416},[407,3913,3914],{"class":736},"  # layer 3\n",[407,3916,3917,3920,3922,3924,3927,3929,3931,3933,3935,3937],{"class":409,"line":440},[407,3918,3919],{"class":1102},"    resolved",[407,3921,2466],{"class":416},[407,3923,2000],{"class":1877},[407,3925,3926],{"class":436},"service",[407,3928,2000],{"class":1877},[407,3930,3708],{"class":416},[407,3932,3100],{"class":416},[407,3934,1905],{"class":1102},[407,3936,167],{"class":416},[407,3938,3939],{"class":1102},"name\n",[407,3941,3942,3944,3946,3948,3950,3952,3954,3956],{"class":409,"line":457},[407,3943,3919],{"class":1102},[407,3945,2466],{"class":416},[407,3947,2000],{"class":1877},[407,3949,2668],{"class":436},[407,3951,2000],{"class":1877},[407,3953,3708],{"class":416},[407,3955,3100],{"class":416},[407,3957,3958],{"class":557}," True\n",[407,3960,3961],{"class":409,"line":465},[407,3962,530],{"emptyLinePlaceholder":529},[407,3964,3965,3968,3970,3972,3974,3976,3978,3980,3983,3985,3987],{"class":409,"line":477},[407,3966,3967],{"class":1102},"    template ",[407,3969,2499],{"class":416},[407,3971,2761],{"class":1102},[407,3973,167],{"class":416},[407,3975,3135],{"class":1102},[407,3977,2451],{"class":416},[407,3979,2000],{"class":1877},[407,3981,3982],{"class":436},"swarmctl.policy",[407,3984,2000],{"class":1877},[407,3986,2457],{"class":416},[407,3988,3989],{"class":736},"   # layer 2\n",[407,3991,3992,3994,3997,3999,4001,4003,4005,4007,4009,4011,4013,4016,4018],{"class":409,"line":485},[407,3993,2552],{"class":557},[407,3995,3996],{"class":1102}," template ",[407,3998,3360],{"class":1095},[407,4000,3996],{"class":1102},[407,4002,2779],{"class":1095},[407,4004,2536],{"class":1088},[407,4006,167],{"class":416},[407,4008,3895],{"class":1102},[407,4010,2466],{"class":416},[407,4012,2000],{"class":1877},[407,4014,4015],{"class":436},"templates",[407,4017,2000],{"class":1877},[407,4019,4020],{"class":416},"]:\n",[407,4022,4023,4026,4028,4031,4033,4035,4037,4039,4041,4043,4045,4047,4049,4052],{"class":409,"line":494},[407,4024,4025],{"class":1102},"        resolved",[407,4027,167],{"class":416},[407,4029,4030],{"class":1102},"update",[407,4032,2451],{"class":416},[407,4034,2454],{"class":1088},[407,4036,167],{"class":416},[407,4038,3895],{"class":1102},[407,4040,2466],{"class":416},[407,4042,2000],{"class":1877},[407,4044,4015],{"class":436},[407,4046,2000],{"class":1877},[407,4048,2886],{"class":416},[407,4050,4051],{"class":1102},"template",[407,4053,4054],{"class":416},"])\n",[407,4056,4057],{"class":409,"line":502},[407,4058,530],{"emptyLinePlaceholder":529},[407,4060,4061,4064,4067,4069,4071,4074,4076,4079,4081,4083,4086,4088,4091,4094],{"class":409,"line":510},[407,4062,4063],{"class":557},"    for",[407,4065,4066],{"class":1102}," label_key",[407,4068,750],{"class":416},[407,4070,1277],{"class":416},[407,4072,4073],{"class":1102},"field",[407,4075,750],{"class":416},[407,4077,4078],{"class":1102}," typ",[407,4080,2457],{"class":416},[407,4082,1840],{"class":557},[407,4084,4085],{"class":1102}," label_map",[407,4087,167],{"class":416},[407,4089,4090],{"class":1102},"items",[407,4092,4093],{"class":416},"():",[407,4095,4096],{"class":736},"   # layer 1\n",[407,4098,4099,4101,4104,4106,4108],{"class":409,"line":518},[407,4100,2861],{"class":557},[407,4102,4103],{"class":1102}," label_key ",[407,4105,2779],{"class":1095},[407,4107,2761],{"class":1102},[407,4109,417],{"class":416},[407,4111,4112,4115],{"class":409,"line":526},[407,4113,4114],{"class":557},"            try",[407,4116,417],{"class":416},[407,4118,4119,4122,4124,4126,4128,4131],{"class":409,"line":533},[407,4120,4121],{"class":1102},"                val ",[407,4123,2499],{"class":416},[407,4125,2761],{"class":1102},[407,4127,2466],{"class":416},[407,4129,4130],{"class":1102},"label_key",[407,4132,454],{"class":416},[407,4134,4135,4138,4140,4142,4144,4146,4148,4151,4153,4156,4158,4161,4163,4166,4168,4170,4172,4175,4177,4180,4183,4185,4187,4189],{"class":409,"line":541},[407,4136,4137],{"class":1102},"                resolved",[407,4139,2466],{"class":416},[407,4141,4073],{"class":1102},[407,4143,3708],{"class":416},[407,4145,3100],{"class":416},[407,4147,1277],{"class":416},[407,4149,4150],{"class":1102},"val",[407,4152,167],{"class":416},[407,4154,4155],{"class":1102},"lower",[407,4157,2625],{"class":416},[407,4159,4160],{"class":1095}," ==",[407,4162,1994],{"class":1877},[407,4164,4165],{"class":436},"true",[407,4167,2000],{"class":1877},[407,4169,2457],{"class":416},[407,4171,3230],{"class":557},[407,4173,4174],{"class":1102}," typ ",[407,4176,2238],{"class":1095},[407,4178,4179],{"class":412}," bool",[407,4181,4182],{"class":557}," else",[407,4184,4078],{"class":1102},[407,4186,2451],{"class":416},[407,4188,4150],{"class":1102},[407,4190,2764],{"class":416},[407,4192,4193,4196,4198,4201,4203,4206],{"class":409,"line":549},[407,4194,4195],{"class":557},"            except",[407,4197,1277],{"class":416},[407,4199,4200],{"class":412},"ValueError",[407,4202,750],{"class":416},[407,4204,4205],{"class":412}," AttributeError",[407,4207,3568],{"class":416},[407,4209,4210,4213],{"class":409,"line":561},[407,4211,4212],{"class":557},"                pass",[407,4214,4215],{"class":736},"   # invalid label value → silently fall through to template\u002Fdefault\n",[407,4217,4218],{"class":409,"line":2058},[407,4219,530],{"emptyLinePlaceholder":529},[407,4221,4222,4225,4227,4230,4232,4235,4238],{"class":409,"line":2079},[407,4223,4224],{"class":1102},"    policy ",[407,4226,2499],{"class":416},[407,4228,4229],{"class":1102}," ResolvedPolicy",[407,4231,2451],{"class":416},[407,4233,4234],{"class":1095},"**",[407,4236,4237],{"class":1102},"resolved",[407,4239,2764],{"class":416},[407,4241,4242,4244,4247,4249,4251,4253,4256,4258,4261,4263,4265],{"class":409,"line":2084},[407,4243,3385],{"class":557},[407,4245,4246],{"class":1102}," policy ",[407,4248,3354],{"class":557},[407,4250,2536],{"class":1088},[407,4252,167],{"class":416},[407,4254,4255],{"class":1102},"_validate",[407,4257,2451],{"class":416},[407,4259,4260],{"class":1102},"policy",[407,4262,2457],{"class":416},[407,4264,4182],{"class":557},[407,4266,3239],{"class":557},[34,4268,4269],{},"Note three things:",[147,4271,4272,4286,4296],{},[150,4273,4274,4277,4278,4281,4282,4285],{},[37,4275,4276],{},"Config hot-reload."," ",[98,4279,4280],{},"_reload_if_changed()"," checks the file mtime each cycle, so operators\ncan tune thresholds in ",[98,4283,4284],{},"config.json"," live without restarting the daemon.",[150,4287,4288,4291,4292,4295],{},[37,4289,4290],{},"Invalid labels fail silently."," A typo'd ",[98,4293,4294],{},"scale_up_cpu_avg=sixty"," doesn't crash the loop or\ndisable the service: it just falls through to the next layer.",[150,4297,4298,4301,4302,4305],{},[37,4299,4300],{},"Validation can veto a service."," If the resolved policy is incoherent, the service is\n",[154,4303,4304],{},"skipped for the cycle"," rather than scaled with nonsense thresholds:",[260,4307,4309],{"className":2436,"code":4308,"language":2438,"meta":266,"style":266},"def _validate(self, policy) -> bool:\n    if policy.min_replicas \u003C 1: return False\n    if policy.min_replicas > policy.max_replicas: return False\n    if policy.scale_down_cpu >= policy.scale_up_cpu_avg: return False   # no sane band\n    if policy.scale_down_mem >= policy.scale_up_mem_avg: return False\n    if policy.cooldown_after_up \u003C= 0 or policy.cooldown_after_down \u003C= 0: return False\n    if policy.scale_down_response_ms > policy.scale_up_response_ms: return False\n    return True\n",[98,4310,4311,4335,4359,4384,4415,4441,4477,4503],{"__ignoreMap":266},[407,4312,4313,4315,4318,4320,4322,4324,4327,4329,4331,4333],{"class":409,"line":11},[407,4314,2445],{"class":1095},[407,4316,4317],{"class":1078}," _validate",[407,4319,2451],{"class":416},[407,4321,2454],{"class":1102},[407,4323,750],{"class":416},[407,4325,4326],{"class":1102}," policy",[407,4328,2457],{"class":416},[407,4330,2460],{"class":416},[407,4332,4179],{"class":412},[407,4334,417],{"class":416},[407,4336,4337,4339,4341,4343,4346,4348,4351,4353,4356],{"class":409,"line":20},[407,4338,2552],{"class":557},[407,4340,4326],{"class":1102},[407,4342,167],{"class":416},[407,4344,4345],{"class":1102},"min_replicas ",[407,4347,3199],{"class":1095},[407,4349,4350],{"class":473}," 1",[407,4352,433],{"class":416},[407,4354,4355],{"class":557}," return",[407,4357,4358],{"class":557}," False\n",[407,4360,4361,4363,4365,4367,4369,4371,4373,4375,4378,4380,4382],{"class":409,"line":427},[407,4362,2552],{"class":557},[407,4364,4326],{"class":1102},[407,4366,167],{"class":416},[407,4368,4345],{"class":1102},[407,4370,1848],{"class":1095},[407,4372,4326],{"class":1102},[407,4374,167],{"class":416},[407,4376,4377],{"class":1102},"max_replicas",[407,4379,433],{"class":416},[407,4381,4355],{"class":557},[407,4383,4358],{"class":557},[407,4385,4386,4388,4390,4392,4395,4398,4400,4402,4405,4407,4409,4412],{"class":409,"line":440},[407,4387,2552],{"class":557},[407,4389,4326],{"class":1102},[407,4391,167],{"class":416},[407,4393,4394],{"class":1102},"scale_down_cpu ",[407,4396,4397],{"class":1095},">=",[407,4399,4326],{"class":1102},[407,4401,167],{"class":416},[407,4403,4404],{"class":1102},"scale_up_cpu_avg",[407,4406,433],{"class":416},[407,4408,4355],{"class":557},[407,4410,4411],{"class":557}," False",[407,4413,4414],{"class":736},"   # no sane band\n",[407,4416,4417,4419,4421,4423,4426,4428,4430,4432,4435,4437,4439],{"class":409,"line":457},[407,4418,2552],{"class":557},[407,4420,4326],{"class":1102},[407,4422,167],{"class":416},[407,4424,4425],{"class":1102},"scale_down_mem ",[407,4427,4397],{"class":1095},[407,4429,4326],{"class":1102},[407,4431,167],{"class":416},[407,4433,4434],{"class":1102},"scale_up_mem_avg",[407,4436,433],{"class":416},[407,4438,4355],{"class":557},[407,4440,4358],{"class":557},[407,4442,4443,4445,4447,4449,4452,4455,4457,4460,4462,4464,4467,4469,4471,4473,4475],{"class":409,"line":465},[407,4444,2552],{"class":557},[407,4446,4326],{"class":1102},[407,4448,167],{"class":416},[407,4450,4451],{"class":1102},"cooldown_after_up ",[407,4453,4454],{"class":1095},"\u003C=",[407,4456,3180],{"class":473},[407,4458,4459],{"class":1095}," or",[407,4461,4326],{"class":1102},[407,4463,167],{"class":416},[407,4465,4466],{"class":1102},"cooldown_after_down ",[407,4468,4454],{"class":1095},[407,4470,3180],{"class":473},[407,4472,433],{"class":416},[407,4474,4355],{"class":557},[407,4476,4358],{"class":557},[407,4478,4479,4481,4483,4485,4488,4490,4492,4494,4497,4499,4501],{"class":409,"line":477},[407,4480,2552],{"class":557},[407,4482,4326],{"class":1102},[407,4484,167],{"class":416},[407,4486,4487],{"class":1102},"scale_down_response_ms ",[407,4489,1848],{"class":1095},[407,4491,4326],{"class":1102},[407,4493,167],{"class":416},[407,4495,4496],{"class":1102},"scale_up_response_ms",[407,4498,433],{"class":416},[407,4500,4355],{"class":557},[407,4502,4358],{"class":557},[407,4504,4505,4507],{"class":409,"line":485},[407,4506,3385],{"class":557},[407,4508,3958],{"class":557},[241,4510,4512],{"id":4511},"the-engine-the-decision-pipeline","The engine: the decision pipeline",[34,4514,4515],{},"The engine signature tells you everything about what state lives outside it:",[260,4517,4519],{"className":2436,"code":4518,"language":2438,"meta":266,"style":266},"def evaluate(\n    self,\n    metrics: ServiceMetrics,\n    policy: ResolvedPolicy,\n    time_since_last_action: int,        # from watch loop\n    last_scale_action: ScalingAction,   # from watch loop\n    elapsed_scale_up_seconds: int,      # from watch loop (duration timer)\n    elapsed_scale_down_seconds: int,    # from watch loop (duration timer)\n    suppressed_triggers: frozenset = frozenset(),  # per-trigger cooldown\n) -> Tuple[ScalingDecision, bool]:      # (decision, host_scale_signal)\n",[98,4520,4521,4531,4537,4548,4559,4573,4588,4602,4616,4636],{"__ignoreMap":266},[407,4522,4523,4525,4528],{"class":409,"line":11},[407,4524,2445],{"class":1095},[407,4526,4527],{"class":1078}," evaluate",[407,4529,4530],{"class":416},"(\n",[407,4532,4533,4535],{"class":409,"line":20},[407,4534,2510],{"class":1102},[407,4536,3438],{"class":416},[407,4538,4539,4542,4544,4546],{"class":409,"line":427},[407,4540,4541],{"class":1102},"    metrics",[407,4543,433],{"class":416},[407,4545,3560],{"class":1102},[407,4547,3438],{"class":416},[407,4549,4550,4553,4555,4557],{"class":409,"line":440},[407,4551,4552],{"class":1102},"    policy",[407,4554,433],{"class":416},[407,4556,4229],{"class":1102},[407,4558,3438],{"class":416},[407,4560,4561,4564,4566,4568,4570],{"class":409,"line":457},[407,4562,4563],{"class":1102},"    time_since_last_action",[407,4565,433],{"class":416},[407,4567,3625],{"class":412},[407,4569,750],{"class":416},[407,4571,4572],{"class":736},"        # from watch loop\n",[407,4574,4575,4578,4580,4583,4585],{"class":409,"line":465},[407,4576,4577],{"class":1102},"    last_scale_action",[407,4579,433],{"class":416},[407,4581,4582],{"class":1102}," ScalingAction",[407,4584,750],{"class":416},[407,4586,4587],{"class":736},"   # from watch loop\n",[407,4589,4590,4593,4595,4597,4599],{"class":409,"line":477},[407,4591,4592],{"class":1102},"    elapsed_scale_up_seconds",[407,4594,433],{"class":416},[407,4596,3625],{"class":412},[407,4598,750],{"class":416},[407,4600,4601],{"class":736},"      # from watch loop (duration timer)\n",[407,4603,4604,4607,4609,4611,4613],{"class":409,"line":485},[407,4605,4606],{"class":1102},"    elapsed_scale_down_seconds",[407,4608,433],{"class":416},[407,4610,3625],{"class":412},[407,4612,750],{"class":416},[407,4614,4615],{"class":736},"    # from watch loop (duration timer)\n",[407,4617,4618,4621,4623,4626,4628,4630,4633],{"class":409,"line":494},[407,4619,4620],{"class":1102},"    suppressed_triggers",[407,4622,433],{"class":416},[407,4624,4625],{"class":412}," frozenset",[407,4627,3100],{"class":416},[407,4629,4625],{"class":412},[407,4631,4632],{"class":416},"(),",[407,4634,4635],{"class":736},"  # per-trigger cooldown\n",[407,4637,4638,4640,4642,4645,4647,4650,4652,4654,4657],{"class":409,"line":502},[407,4639,2457],{"class":416},[407,4641,2460],{"class":416},[407,4643,4644],{"class":1102}," Tuple",[407,4646,2466],{"class":416},[407,4648,4649],{"class":1102},"ScalingDecision",[407,4651,750],{"class":416},[407,4653,4179],{"class":412},[407,4655,4656],{"class":416},"]:",[407,4658,4659],{"class":736},"      # (decision, host_scale_signal)\n",[34,4661,4662,4663,4666,4667,167],{},"The engine never reads the clock. All timing is passed in. That means a test can simulate\n\"it's been 119 seconds since the last scale-up\" by passing ",[98,4664,4665],{},"119",": no sleeping, no mocking\n",[98,4668,4669],{},"time.time()",[34,4671,4672],{},"The pipeline runs in a strict order, short-circuiting as early as possible.",[4674,4675,4677],"h4",{"id":4676},"step-1-guard-checks","Step 1: Guard checks",[34,4679,4680],{},"Never scale a service that is stopped or crash-looping:",[260,4682,4684],{"className":2436,"code":4683,"language":2438,"meta":266,"style":266},"if not policy.enabled:\n    return self._none(svc, \"policy disabled\", BLOCKED_DISABLED), False\nif metrics.replicas_running == 0 and metrics.replicas_desired == 0:\n    return self._none(svc, \"service stopped\", BLOCKED_SERVICE_STOPPED), False\nif metrics.replicas_running == 0 and metrics.replicas_desired > 0:\n    return self._none(svc, \"crash loop: 0 running, desired > 0\", BLOCKED_CRASH_LOOP), False\n",[98,4685,4686,4700,4733,4765,4797,4825],{"__ignoreMap":266},[407,4687,4688,4690,4692,4694,4696,4698],{"class":409,"line":11},[407,4689,3354],{"class":557},[407,4691,2555],{"class":1095},[407,4693,4326],{"class":1102},[407,4695,167],{"class":416},[407,4697,2668],{"class":1102},[407,4699,417],{"class":416},[407,4701,4702,4704,4706,4708,4711,4713,4715,4717,4719,4722,4724,4726,4729,4731],{"class":409,"line":20},[407,4703,3385],{"class":557},[407,4705,2536],{"class":1088},[407,4707,167],{"class":416},[407,4709,4710],{"class":1102},"_none",[407,4712,2451],{"class":416},[407,4714,2756],{"class":1102},[407,4716,750],{"class":416},[407,4718,1994],{"class":1877},[407,4720,4721],{"class":436},"policy disabled",[407,4723,2000],{"class":1877},[407,4725,750],{"class":416},[407,4727,4728],{"class":1088}," BLOCKED_DISABLED",[407,4730,2671],{"class":416},[407,4732,4358],{"class":557},[407,4734,4735,4737,4740,4742,4745,4747,4749,4752,4754,4756,4759,4761,4763],{"class":409,"line":427},[407,4736,3354],{"class":557},[407,4738,4739],{"class":1102}," metrics",[407,4741,167],{"class":416},[407,4743,4744],{"class":1102},"replicas_running ",[407,4746,3365],{"class":1095},[407,4748,3180],{"class":473},[407,4750,4751],{"class":1095}," and",[407,4753,4739],{"class":1102},[407,4755,167],{"class":416},[407,4757,4758],{"class":1102},"replicas_desired ",[407,4760,3365],{"class":1095},[407,4762,3180],{"class":473},[407,4764,417],{"class":416},[407,4766,4767,4769,4771,4773,4775,4777,4779,4781,4783,4786,4788,4790,4793,4795],{"class":409,"line":440},[407,4768,3385],{"class":557},[407,4770,2536],{"class":1088},[407,4772,167],{"class":416},[407,4774,4710],{"class":1102},[407,4776,2451],{"class":416},[407,4778,2756],{"class":1102},[407,4780,750],{"class":416},[407,4782,1994],{"class":1877},[407,4784,4785],{"class":436},"service stopped",[407,4787,2000],{"class":1877},[407,4789,750],{"class":416},[407,4791,4792],{"class":1088}," BLOCKED_SERVICE_STOPPED",[407,4794,2671],{"class":416},[407,4796,4358],{"class":557},[407,4798,4799,4801,4803,4805,4807,4809,4811,4813,4815,4817,4819,4821,4823],{"class":409,"line":457},[407,4800,3354],{"class":557},[407,4802,4739],{"class":1102},[407,4804,167],{"class":416},[407,4806,4744],{"class":1102},[407,4808,3365],{"class":1095},[407,4810,3180],{"class":473},[407,4812,4751],{"class":1095},[407,4814,4739],{"class":1102},[407,4816,167],{"class":416},[407,4818,4758],{"class":1102},[407,4820,1848],{"class":1095},[407,4822,3180],{"class":473},[407,4824,417],{"class":416},[407,4826,4827,4829,4831,4833,4835,4837,4839,4841,4843,4846,4848,4850,4853,4855],{"class":409,"line":465},[407,4828,3385],{"class":557},[407,4830,2536],{"class":1088},[407,4832,167],{"class":416},[407,4834,4710],{"class":1102},[407,4836,2451],{"class":416},[407,4838,2756],{"class":1102},[407,4840,750],{"class":416},[407,4842,1994],{"class":1877},[407,4844,4845],{"class":436},"crash loop: 0 running, desired > 0",[407,4847,2000],{"class":1877},[407,4849,750],{"class":416},[407,4851,4852],{"class":1088}," BLOCKED_CRASH_LOOP",[407,4854,2671],{"class":416},[407,4856,4358],{"class":557},[34,4858,4859],{},"Scaling a crash-looping service up just creates more crashing containers. The engine refuses.",[4674,4861,4863],{"id":4862},"step-2-emergency-check-bypasses-cooldown","Step 2: Emergency check (bypasses cooldown)",[34,4865,4866],{},"Hard ceilings fire regardless of cooldown. A container at 96% max CPU can't wait two minutes:",[260,4868,4870],{"className":2436,"code":4869,"language":2438,"meta":266,"style":266},"if metrics.max_cpu > self._EMERGENCY_CPU_HARD_CEILING:   # e.g. 95%\n    emergency_trigger_names.add(TRIGGER_MAX_CPU)\nif metrics.max_mem > self._EMERGENCY_MEM_HARD_CEILING:   # e.g. 92%\n    emergency_trigger_names.add(TRIGGER_MAX_MEM)\n",[98,4871,4872,4897,4914,4939],{"__ignoreMap":266},[407,4873,4874,4876,4878,4880,4883,4885,4887,4889,4892,4894],{"class":409,"line":11},[407,4875,3354],{"class":557},[407,4877,4739],{"class":1102},[407,4879,167],{"class":416},[407,4881,4882],{"class":1102},"max_cpu ",[407,4884,1848],{"class":1095},[407,4886,2536],{"class":1088},[407,4888,167],{"class":416},[407,4890,4891],{"class":1088},"_EMERGENCY_CPU_HARD_CEILING",[407,4893,433],{"class":416},[407,4895,4896],{"class":736},"   # e.g. 95%\n",[407,4898,4899,4902,4904,4907,4909,4912],{"class":409,"line":20},[407,4900,4901],{"class":1102},"    emergency_trigger_names",[407,4903,167],{"class":416},[407,4905,4906],{"class":1102},"add",[407,4908,2451],{"class":416},[407,4910,4911],{"class":1088},"TRIGGER_MAX_CPU",[407,4913,2764],{"class":416},[407,4915,4916,4918,4920,4922,4925,4927,4929,4931,4934,4936],{"class":409,"line":427},[407,4917,3354],{"class":557},[407,4919,4739],{"class":1102},[407,4921,167],{"class":416},[407,4923,4924],{"class":1102},"max_mem ",[407,4926,1848],{"class":1095},[407,4928,2536],{"class":1088},[407,4930,167],{"class":416},[407,4932,4933],{"class":1088},"_EMERGENCY_MEM_HARD_CEILING",[407,4935,433],{"class":416},[407,4937,4938],{"class":736},"   # e.g. 92%\n",[407,4940,4941,4943,4945,4947,4949,4952],{"class":409,"line":440},[407,4942,4901],{"class":1102},[407,4944,167],{"class":416},[407,4946,4906],{"class":1102},[407,4948,2451],{"class":416},[407,4950,4951],{"class":1088},"TRIGGER_MAX_MEM",[407,4953,2764],{"class":416},[34,4955,4956,4957,4960],{},"The rejected-tasks case is special and important: it is the ",[37,4958,4959],{},"cluster-full"," signal:",[260,4962,4964],{"className":2436,"code":4963,"language":2438,"meta":266,"style":266},"if metrics.replicas_rejected > 0:\n    return ScalingDecision(..., action=NONE, blocked_by=BLOCKED_CLUSTER_FULL), True\n    #                                 ^^^^                                    ^^^^\n    #       Don't try to add containers: there's no room.    But DO raise host_scale_signal.\n",[98,4965,4966,4983,5019,5024],{"__ignoreMap":266},[407,4967,4968,4970,4972,4974,4977,4979,4981],{"class":409,"line":11},[407,4969,3354],{"class":557},[407,4971,4739],{"class":1102},[407,4973,167],{"class":416},[407,4975,4976],{"class":1102},"replicas_rejected ",[407,4978,1848],{"class":1095},[407,4980,3180],{"class":473},[407,4982,417],{"class":416},[407,4984,4985,4987,4990,4992,4995,4997,5000,5002,5005,5007,5010,5012,5015,5017],{"class":409,"line":20},[407,4986,3385],{"class":557},[407,4988,4989],{"class":1102}," ScalingDecision",[407,4991,2451],{"class":416},[407,4993,4994],{"class":1088},"...",[407,4996,750],{"class":416},[407,4998,4999],{"class":1836}," action",[407,5001,2499],{"class":416},[407,5003,5004],{"class":1088},"NONE",[407,5006,750],{"class":416},[407,5008,5009],{"class":1836}," blocked_by",[407,5011,2499],{"class":416},[407,5013,5014],{"class":1088},"BLOCKED_CLUSTER_FULL",[407,5016,2671],{"class":416},[407,5018,3958],{"class":557},[407,5020,5021],{"class":409,"line":427},[407,5022,5023],{"class":736},"    #                                 ^^^^                                    ^^^^\n",[407,5025,5026],{"class":409,"line":440},[407,5027,5028],{"class":736},"    #       Don't try to add containers: there's no room.    But DO raise host_scale_signal.\n",[34,5030,5031,5032,5035,5036,5038,5039,5042],{},"When tasks are being ",[154,5033,5034],{},"rejected",", adding more replicas is pointless: the cluster has no capacity.\nSo the engine returns ",[98,5037,5004],{}," for the service but sets ",[98,5040,5041],{},"host_scale_signal = True",", telling the\nwatch loop to ask OpenStack for a new VM instead.",[34,5044,5045,5046,5048],{},"If an emergency fires and there ",[154,5047,2238],{}," room, it scales by the emergency step (3) immediately,\nwaiving cooldown.",[4674,5050,5052],{"id":5051},"step-3-cooldown-check","Step 3: Cooldown check",[34,5054,5055],{},"After an ordinary scale, respect the configured quiet period so the system can stabilize before\nreacting again:",[260,5057,5059],{"className":2436,"code":5058,"language":2438,"meta":266,"style":266},"if last_scale_action == SCALE_UP and time_since_last_action \u003C policy.cooldown_after_up:\n    return self._none(svc, f\"cooldown after scale-up ...\", BLOCKED_COOLDOWN_UP), False\nif last_scale_action == SCALE_DOWN and time_since_last_action \u003C policy.cooldown_after_down:\n    return self._none(svc, f\"cooldown after scale-down ...\", BLOCKED_COOLDOWN_DOWN), False\n",[98,5060,5061,5089,5120,5146],{"__ignoreMap":266},[407,5062,5063,5065,5068,5070,5073,5075,5078,5080,5082,5084,5087],{"class":409,"line":11},[407,5064,3354],{"class":557},[407,5066,5067],{"class":1102}," last_scale_action ",[407,5069,3365],{"class":1095},[407,5071,5072],{"class":1088}," SCALE_UP",[407,5074,4751],{"class":1095},[407,5076,5077],{"class":1102}," time_since_last_action ",[407,5079,3199],{"class":1095},[407,5081,4326],{"class":1102},[407,5083,167],{"class":416},[407,5085,5086],{"class":1102},"cooldown_after_up",[407,5088,417],{"class":416},[407,5090,5091,5093,5095,5097,5099,5101,5103,5105,5108,5111,5113,5116,5118],{"class":409,"line":20},[407,5092,3385],{"class":557},[407,5094,2536],{"class":1088},[407,5096,167],{"class":416},[407,5098,4710],{"class":1102},[407,5100,2451],{"class":416},[407,5102,2756],{"class":1102},[407,5104,750],{"class":416},[407,5106,5107],{"class":1095}," f",[407,5109,5110],{"class":436},"\"cooldown after scale-up ...\"",[407,5112,750],{"class":416},[407,5114,5115],{"class":1088}," BLOCKED_COOLDOWN_UP",[407,5117,2671],{"class":416},[407,5119,4358],{"class":557},[407,5121,5122,5124,5126,5128,5131,5133,5135,5137,5139,5141,5144],{"class":409,"line":427},[407,5123,3354],{"class":557},[407,5125,5067],{"class":1102},[407,5127,3365],{"class":1095},[407,5129,5130],{"class":1088}," SCALE_DOWN",[407,5132,4751],{"class":1095},[407,5134,5077],{"class":1102},[407,5136,3199],{"class":1095},[407,5138,4326],{"class":1102},[407,5140,167],{"class":416},[407,5142,5143],{"class":1102},"cooldown_after_down",[407,5145,417],{"class":416},[407,5147,5148,5150,5152,5154,5156,5158,5160,5162,5164,5167,5169,5172,5174],{"class":409,"line":440},[407,5149,3385],{"class":557},[407,5151,2536],{"class":1088},[407,5153,167],{"class":416},[407,5155,4710],{"class":1102},[407,5157,2451],{"class":416},[407,5159,2756],{"class":1102},[407,5161,750],{"class":416},[407,5163,5107],{"class":1095},[407,5165,5166],{"class":436},"\"cooldown after scale-down ...\"",[407,5168,750],{"class":416},[407,5170,5171],{"class":1088}," BLOCKED_COOLDOWN_DOWN",[407,5173,2671],{"class":416},[407,5175,4358],{"class":557},[4674,5177,5179],{"id":5178},"steps-4-5-weighted-multi-trigger-scoring","Steps 4 & 5: Weighted multi-trigger scoring",[34,5181,5182,5183,5186,5187,5190],{},"This is the heart of scale-up. Each trigger is evaluated independently; ",[37,5184,5185],{},"any single one firing\nis enough to scale up",". But the engine also computes a weighted ",[154,5188,5189],{},"score"," that drives how\naggressively to scale (by 1, 2, or 3 replicas).",[34,5192,5193,5194,5197],{},"Each trigger contributes ",[98,5195,5196],{},"overage × weight",", where overage is how far past the threshold the\nmetric is, normalized to the headroom between threshold and ceiling:",[260,5199,5201],{"className":2436,"code":5200,"language":2438,"meta":266,"style":266},"def _check(trigger, condition, weight, overage):\n    if not condition:\n        return\n    if trigger in suppressed_triggers:   # per-trigger cooldown\n        suppressed.append(trigger)\n    else:\n        fired.append(trigger)\n        weighted_overage_sum += overage * weight\n\n# T1 avg_cpu   weight 0.30\n_check(TRIGGER_AVG_CPU, metrics.avg_cpu > policy.scale_up_cpu_avg, 0.30,\n       (metrics.avg_cpu - policy.scale_up_cpu_avg) \u002F (100.0 - policy.scale_up_cpu_avg))\n# T2 max_cpu   weight 0.30\n# T3 avg_mem   weight 0.30   (same as CPU: sustained high memory should reach MODERATE)\n# T4 max_mem   weight 0.25   (slightly lower: one hot container shouldn't alone drive emergency)\n# T5 cpu_throttle weight 0.30 (fires even when raw CPU% looks low)\n# T7 connections weight 0.10  (overage capped at 1.0, a 10x spike shouldn't alone hit EMERGENCY)\n# T8 request_rate weight 0.15 (only when a rate threshold is explicitly configured)\n",[98,5202,5203,5232,5242,5247,5264,5280,5287,5302,5318,5322,5327,5361,5402,5407,5412,5417,5422,5427],{"__ignoreMap":266},[407,5204,5205,5207,5210,5212,5215,5217,5220,5222,5225,5227,5230],{"class":409,"line":11},[407,5206,2445],{"class":1095},[407,5208,5209],{"class":1078}," _check",[407,5211,2451],{"class":416},[407,5213,5214],{"class":1102},"trigger",[407,5216,750],{"class":416},[407,5218,5219],{"class":1102}," condition",[407,5221,750],{"class":416},[407,5223,5224],{"class":1102}," weight",[407,5226,750],{"class":416},[407,5228,5229],{"class":1102}," overage",[407,5231,3568],{"class":416},[407,5233,5234,5236,5238,5240],{"class":409,"line":20},[407,5235,2552],{"class":557},[407,5237,2555],{"class":1095},[407,5239,5219],{"class":1102},[407,5241,417],{"class":416},[407,5243,5244],{"class":409,"line":427},[407,5245,5246],{"class":557},"        return\n",[407,5248,5249,5251,5254,5256,5259,5261],{"class":409,"line":440},[407,5250,2552],{"class":557},[407,5252,5253],{"class":1102}," trigger ",[407,5255,2779],{"class":1095},[407,5257,5258],{"class":1102}," suppressed_triggers",[407,5260,433],{"class":416},[407,5262,5263],{"class":736},"   # per-trigger cooldown\n",[407,5265,5266,5269,5271,5274,5276,5278],{"class":409,"line":457},[407,5267,5268],{"class":1102},"        suppressed",[407,5270,167],{"class":416},[407,5272,5273],{"class":1102},"append",[407,5275,2451],{"class":416},[407,5277,5214],{"class":1102},[407,5279,2764],{"class":416},[407,5281,5282,5285],{"class":409,"line":465},[407,5283,5284],{"class":557},"    else",[407,5286,417],{"class":416},[407,5288,5289,5292,5294,5296,5298,5300],{"class":409,"line":477},[407,5290,5291],{"class":1102},"        fired",[407,5293,167],{"class":416},[407,5295,5273],{"class":1102},[407,5297,2451],{"class":416},[407,5299,5214],{"class":1102},[407,5301,2764],{"class":416},[407,5303,5304,5307,5310,5313,5315],{"class":409,"line":485},[407,5305,5306],{"class":1102},"        weighted_overage_sum ",[407,5308,5309],{"class":416},"+=",[407,5311,5312],{"class":1102}," overage ",[407,5314,3222],{"class":1095},[407,5316,5317],{"class":1102}," weight\n",[407,5319,5320],{"class":409,"line":494},[407,5321,530],{"emptyLinePlaceholder":529},[407,5323,5324],{"class":409,"line":502},[407,5325,5326],{"class":736},"# T1 avg_cpu   weight 0.30\n",[407,5328,5329,5332,5334,5337,5339,5341,5343,5346,5348,5350,5352,5354,5356,5359],{"class":409,"line":510},[407,5330,5331],{"class":1102},"_check",[407,5333,2451],{"class":416},[407,5335,5336],{"class":1088},"TRIGGER_AVG_CPU",[407,5338,750],{"class":416},[407,5340,4739],{"class":1102},[407,5342,167],{"class":416},[407,5344,5345],{"class":1102},"avg_cpu ",[407,5347,1848],{"class":1095},[407,5349,4326],{"class":1102},[407,5351,167],{"class":416},[407,5353,4404],{"class":1102},[407,5355,750],{"class":416},[407,5357,5358],{"class":473}," 0.30",[407,5360,3438],{"class":416},[407,5362,5363,5366,5369,5371,5373,5375,5377,5379,5381,5383,5386,5388,5391,5394,5396,5398,5400],{"class":409,"line":518},[407,5364,5365],{"class":416},"       (",[407,5367,5368],{"class":1102},"metrics",[407,5370,167],{"class":416},[407,5372,5345],{"class":1102},[407,5374,2810],{"class":1095},[407,5376,4326],{"class":1102},[407,5378,167],{"class":416},[407,5380,4404],{"class":1102},[407,5382,2457],{"class":416},[407,5384,5385],{"class":1095}," \u002F",[407,5387,1277],{"class":416},[407,5389,5390],{"class":473},"100.0",[407,5392,5393],{"class":1095}," -",[407,5395,4326],{"class":1102},[407,5397,167],{"class":416},[407,5399,4404],{"class":1102},[407,5401,3349],{"class":416},[407,5403,5404],{"class":409,"line":526},[407,5405,5406],{"class":736},"# T2 max_cpu   weight 0.30\n",[407,5408,5409],{"class":409,"line":533},[407,5410,5411],{"class":736},"# T3 avg_mem   weight 0.30   (same as CPU: sustained high memory should reach MODERATE)\n",[407,5413,5414],{"class":409,"line":541},[407,5415,5416],{"class":736},"# T4 max_mem   weight 0.25   (slightly lower: one hot container shouldn't alone drive emergency)\n",[407,5418,5419],{"class":409,"line":549},[407,5420,5421],{"class":736},"# T5 cpu_throttle weight 0.30 (fires even when raw CPU% looks low)\n",[407,5423,5424],{"class":409,"line":561},[407,5425,5426],{"class":736},"# T7 connections weight 0.10  (overage capped at 1.0, a 10x spike shouldn't alone hit EMERGENCY)\n",[407,5428,5429],{"class":409,"line":2058},[407,5430,5431],{"class":736},"# T8 request_rate weight 0.15 (only when a rate threshold is explicitly configured)\n",[34,5433,5434,5435,5437],{},"The pending-stuck trigger is a special case: it fires, adds its weight, ",[37,5436,3360],{}," raises the\nhost-scale signal, because stuck tasks mean the cluster is out of room:",[260,5439,5441],{"className":2436,"code":5440,"language":2438,"meta":266,"style":266},"if metrics.longest_pending_seconds > policy.pending_stuck_seconds:\n    fired.append(TRIGGER_PENDING_STUCK)\n    weighted_overage_sum += 0.30\n    host_scale_signal = True\n",[98,5442,5443,5465,5481,5491],{"__ignoreMap":266},[407,5444,5445,5447,5449,5451,5454,5456,5458,5460,5463],{"class":409,"line":11},[407,5446,3354],{"class":557},[407,5448,4739],{"class":1102},[407,5450,167],{"class":416},[407,5452,5453],{"class":1102},"longest_pending_seconds ",[407,5455,1848],{"class":1095},[407,5457,4326],{"class":1102},[407,5459,167],{"class":416},[407,5461,5462],{"class":1102},"pending_stuck_seconds",[407,5464,417],{"class":416},[407,5466,5467,5470,5472,5474,5476,5479],{"class":409,"line":20},[407,5468,5469],{"class":1102},"    fired",[407,5471,167],{"class":416},[407,5473,5273],{"class":1102},[407,5475,2451],{"class":416},[407,5477,5478],{"class":1088},"TRIGGER_PENDING_STUCK",[407,5480,2764],{"class":416},[407,5482,5483,5486,5488],{"class":409,"line":427},[407,5484,5485],{"class":1102},"    weighted_overage_sum ",[407,5487,5309],{"class":416},[407,5489,5490],{"class":473}," 0.30\n",[407,5492,5493,5496,5498],{"class":409,"line":440},[407,5494,5495],{"class":1102},"    host_scale_signal ",[407,5497,2499],{"class":416},[407,5499,3958],{"class":557},[34,5501,5502],{},"The score has a built-in floor, so any fired trigger yields at least a NORMAL scale:",[260,5504,5506],{"className":2436,"code":5505,"language":2438,"meta":266,"style":266},"score = min(1.0, self._SCORE_BASE + weighted_overage_sum)   # SCORE_BASE = 0.30\n\nurgency, step = Urgency.NORMAL, 1\nif score >= self._SCORE_EMERGENCY_THRESHOLD:   # 0.75 → EMERGENCY, step 3\n    urgency, step = Urgency.EMERGENCY, 3\nelif score >= self._SCORE_MODERATE_THRESHOLD:  # 0.45 → MODERATE, step 2\n    urgency, step = Urgency.MODERATE, 2\n",[98,5507,5508,5542,5546,5570,5591,5613,5634],{"__ignoreMap":266},[407,5509,5510,5513,5515,5518,5520,5523,5525,5527,5529,5532,5534,5537,5539],{"class":409,"line":11},[407,5511,5512],{"class":1102},"score ",[407,5514,2499],{"class":416},[407,5516,5517],{"class":412}," min",[407,5519,2451],{"class":416},[407,5521,5522],{"class":473},"1.0",[407,5524,750],{"class":416},[407,5526,2536],{"class":1088},[407,5528,167],{"class":416},[407,5530,5531],{"class":1088},"_SCORE_BASE",[407,5533,2628],{"class":1095},[407,5535,5536],{"class":1102}," weighted_overage_sum",[407,5538,2457],{"class":416},[407,5540,5541],{"class":736},"   # SCORE_BASE = 0.30\n",[407,5543,5544],{"class":409,"line":20},[407,5545,530],{"emptyLinePlaceholder":529},[407,5547,5548,5551,5553,5556,5558,5561,5563,5566,5568],{"class":409,"line":427},[407,5549,5550],{"class":1102},"urgency",[407,5552,750],{"class":416},[407,5554,5555],{"class":1102}," step ",[407,5557,2499],{"class":416},[407,5559,5560],{"class":1102}," Urgency",[407,5562,167],{"class":416},[407,5564,5565],{"class":1088},"NORMAL",[407,5567,750],{"class":416},[407,5569,1696],{"class":473},[407,5571,5572,5574,5577,5579,5581,5583,5586,5588],{"class":409,"line":440},[407,5573,3354],{"class":557},[407,5575,5576],{"class":1102}," score ",[407,5578,4397],{"class":1095},[407,5580,2536],{"class":1088},[407,5582,167],{"class":416},[407,5584,5585],{"class":1088},"_SCORE_EMERGENCY_THRESHOLD",[407,5587,433],{"class":416},[407,5589,5590],{"class":736},"   # 0.75 → EMERGENCY, step 3\n",[407,5592,5593,5596,5598,5600,5602,5604,5606,5609,5611],{"class":409,"line":457},[407,5594,5595],{"class":1102},"    urgency",[407,5597,750],{"class":416},[407,5599,5555],{"class":1102},[407,5601,2499],{"class":416},[407,5603,5560],{"class":1102},[407,5605,167],{"class":416},[407,5607,5608],{"class":1088},"EMERGENCY",[407,5610,750],{"class":416},[407,5612,474],{"class":473},[407,5614,5615,5618,5620,5622,5624,5626,5629,5631],{"class":409,"line":465},[407,5616,5617],{"class":557},"elif",[407,5619,5576],{"class":1102},[407,5621,4397],{"class":1095},[407,5623,2536],{"class":1088},[407,5625,167],{"class":416},[407,5627,5628],{"class":1088},"_SCORE_MODERATE_THRESHOLD",[407,5630,433],{"class":416},[407,5632,5633],{"class":736},"  # 0.45 → MODERATE, step 2\n",[407,5635,5636,5638,5640,5642,5644,5646,5648,5651,5653],{"class":409,"line":477},[407,5637,5595],{"class":1102},[407,5639,750],{"class":416},[407,5641,5555],{"class":1102},[407,5643,2499],{"class":416},[407,5645,5560],{"class":1102},[407,5647,167],{"class":416},[407,5649,5650],{"class":1088},"MODERATE",[407,5652,750],{"class":416},[407,5654,5655],{"class":473}," 2\n",[34,5657,5658,5659,5662,5663,5666,5667,5670],{},"The defaults (tunable in config): a floor of ",[98,5660,5661],{},"0.30"," so any single fired trigger yields at least a\nNORMAL +1; ",[98,5664,5665],{},"0.45"," crosses into MODERATE +2 (one trigger well above threshold, or two triggers\nfiring together); ",[98,5668,5669],{},"0.75"," into EMERGENCY +3 (several triggers blown out at once). The combined\ntrigger weights sum to roughly 1.80, so the score saturates well before every trigger has to fire.",[34,5672,5673],{},"In short, barely over one threshold gives +1, and multiple thresholds blown out give +3. The weighting means a\nservice that is simultaneously CPU-bound, memory-bound, and throttled escalates fast, while a\nservice that merely has slightly elevated connections nudges up by one.",[34,5675,5676],{},"Play with the numbers yourself. Slide each metric past its threshold (the small tick on each slider) and watch the score and the step size change. The \"throttled, CPU looks fine\" preset is the case that makes the throttle trigger worth having.",[5678,5679],"swarm-scoring",{},[4674,5681,5683,5684],{"id":5682},"step-55-scale-up-must-be-sustained","Step 5.5: Scale-up must be ",[154,5685,5686],{},"sustained",[34,5688,5689,5690,5693],{},"A single spiky cycle shouldn't scale you up. Scale-up conditions must hold for a configured\nduration, ",[37,5691,5692],{},"except"," pending-stuck, which bypasses the timer because waiting would only make a\ncapacity shortage worse:",[260,5695,5697],{"className":2436,"code":5696,"language":2438,"meta":266,"style":266},"duration_waived = TRIGGER_PENDING_STUCK in fired\nif not duration_waived and elapsed_scale_up_seconds \u003C policy.scale_up_duration:\n    return self._none(svc, f\"...met for {elapsed_scale_up_seconds}s, need {policy.scale_up_duration}s\",\n                      BLOCKED_SCALE_UP_NOT_SUSTAINED), False\n",[98,5698,5699,5714,5739,5785],{"__ignoreMap":266},[407,5700,5701,5704,5706,5709,5711],{"class":409,"line":11},[407,5702,5703],{"class":1102},"duration_waived ",[407,5705,2499],{"class":416},[407,5707,5708],{"class":1088}," TRIGGER_PENDING_STUCK",[407,5710,1840],{"class":1095},[407,5712,5713],{"class":1102}," fired\n",[407,5715,5716,5718,5720,5723,5725,5728,5730,5732,5734,5737],{"class":409,"line":20},[407,5717,3354],{"class":557},[407,5719,2555],{"class":1095},[407,5721,5722],{"class":1102}," duration_waived ",[407,5724,3360],{"class":1095},[407,5726,5727],{"class":1102}," elapsed_scale_up_seconds ",[407,5729,3199],{"class":1095},[407,5731,4326],{"class":1102},[407,5733,167],{"class":416},[407,5735,5736],{"class":1102},"scale_up_duration",[407,5738,417],{"class":416},[407,5740,5741,5743,5745,5747,5749,5751,5753,5755,5757,5760,5762,5765,5767,5770,5772,5774,5776,5778,5780,5783],{"class":409,"line":427},[407,5742,3385],{"class":557},[407,5744,2536],{"class":1088},[407,5746,167],{"class":416},[407,5748,4710],{"class":1102},[407,5750,2451],{"class":416},[407,5752,2756],{"class":1102},[407,5754,750],{"class":416},[407,5756,5107],{"class":1095},[407,5758,5759],{"class":436},"\"...met for ",[407,5761,2927],{"class":1088},[407,5763,5764],{"class":1102},"elapsed_scale_up_seconds",[407,5766,2939],{"class":1088},[407,5768,5769],{"class":436},"s, need ",[407,5771,2927],{"class":1088},[407,5773,4260],{"class":1102},[407,5775,167],{"class":416},[407,5777,5736],{"class":1102},[407,5779,2939],{"class":1088},[407,5781,5782],{"class":436},"s\"",[407,5784,3438],{"class":416},[407,5786,5787,5790,5792],{"class":409,"line":440},[407,5788,5789],{"class":1088},"                      BLOCKED_SCALE_UP_NOT_SUSTAINED",[407,5791,2671],{"class":416},[407,5793,4358],{"class":557},[4674,5795,5797],{"id":5796},"step-6-scale-up-bounds","Step 6: Scale-up bounds",[34,5799,5800,5801,5803],{},"If already at ",[98,5802,4377],{},", don't scale, but raise the host signal, because the service wants\nmore and the only way to give it more is a bigger cluster:",[260,5805,5807],{"className":2436,"code":5806,"language":2438,"meta":266,"style":266},"if current_total >= policy.max_replicas:\n    return ScalingDecision(..., action=NONE, blocked_by=BLOCKED_AT_MAX_REPLICAS), True   # host signal\nif current_total + step > policy.max_replicas:\n    step = policy.max_replicas - current_total\nif current_total + step == policy.max_replicas:\n    host_scale_signal = True   # hitting the ceiling this cycle: warn OpenStack early\n",[98,5808,5809,5826,5861,5882,5901,5921],{"__ignoreMap":266},[407,5810,5811,5813,5816,5818,5820,5822,5824],{"class":409,"line":11},[407,5812,3354],{"class":557},[407,5814,5815],{"class":1102}," current_total ",[407,5817,4397],{"class":1095},[407,5819,4326],{"class":1102},[407,5821,167],{"class":416},[407,5823,4377],{"class":1102},[407,5825,417],{"class":416},[407,5827,5828,5830,5832,5834,5836,5838,5840,5842,5844,5846,5848,5850,5853,5855,5858],{"class":409,"line":20},[407,5829,3385],{"class":557},[407,5831,4989],{"class":1102},[407,5833,2451],{"class":416},[407,5835,4994],{"class":1088},[407,5837,750],{"class":416},[407,5839,4999],{"class":1836},[407,5841,2499],{"class":416},[407,5843,5004],{"class":1088},[407,5845,750],{"class":416},[407,5847,5009],{"class":1836},[407,5849,2499],{"class":416},[407,5851,5852],{"class":1088},"BLOCKED_AT_MAX_REPLICAS",[407,5854,2671],{"class":416},[407,5856,5857],{"class":557}," True",[407,5859,5860],{"class":736},"   # host signal\n",[407,5862,5863,5865,5867,5870,5872,5874,5876,5878,5880],{"class":409,"line":427},[407,5864,3354],{"class":557},[407,5866,5815],{"class":1102},[407,5868,5869],{"class":1095},"+",[407,5871,5555],{"class":1102},[407,5873,1848],{"class":1095},[407,5875,4326],{"class":1102},[407,5877,167],{"class":416},[407,5879,4377],{"class":1102},[407,5881,417],{"class":416},[407,5883,5884,5887,5889,5891,5893,5896,5898],{"class":409,"line":440},[407,5885,5886],{"class":1102},"    step ",[407,5888,2499],{"class":416},[407,5890,4326],{"class":1102},[407,5892,167],{"class":416},[407,5894,5895],{"class":1102},"max_replicas ",[407,5897,2810],{"class":1095},[407,5899,5900],{"class":1102}," current_total\n",[407,5902,5903,5905,5907,5909,5911,5913,5915,5917,5919],{"class":409,"line":457},[407,5904,3354],{"class":557},[407,5906,5815],{"class":1102},[407,5908,5869],{"class":1095},[407,5910,5555],{"class":1102},[407,5912,3365],{"class":1095},[407,5914,4326],{"class":1102},[407,5916,167],{"class":416},[407,5918,4377],{"class":1102},[407,5920,417],{"class":416},[407,5922,5923,5925,5927,5929],{"class":409,"line":465},[407,5924,5495],{"class":1102},[407,5926,2499],{"class":416},[407,5928,5857],{"class":557},[407,5930,5931],{"class":736},"   # hitting the ceiling this cycle: warn OpenStack early\n",[241,5933,5935],{"id":5934},"scale-down-conservative-by-design","Scale-down: conservative by design",[34,5937,5938,5939,5942],{},"Scale-up is aggressive (any trigger fires). Scale-down is the opposite: ",[37,5940,5941],{},"every"," condition must\nbe true at once, stay true for a full duration, and then survive a simulation.",[4674,5944,5946],{"id":5945},"step-7-all-conditions-simultaneously","Step 7: All conditions, simultaneously",[260,5948,5950],{"className":2436,"code":5949,"language":2438,"meta":266,"style":266},"response_ms_ok = metrics.traffic is None or metrics.avg_response_ms \u003C policy.scale_down_response_ms\ndown_eligible = (\n    metrics.avg_cpu \u003C policy.scale_down_cpu\n    and metrics.max_cpu \u003C policy.scale_up_cpu_max      # one hot replica blocks scale-down\n    and metrics.avg_mem \u003C policy.scale_down_mem\n    and metrics.open_connections \u003C policy.scale_down_connections\n    and response_ms_ok\n    and metrics.replicas_running > policy.min_replicas\n    and metrics.replicas_pending == 0\n    and metrics.replicas_failed == 0\n)\n",[98,5951,5952,5988,5998,6015,6038,6058,6078,6085,6104,6119,6134],{"__ignoreMap":266},[407,5953,5954,5957,5959,5961,5963,5966,5968,5970,5972,5974,5976,5979,5981,5983,5985],{"class":409,"line":11},[407,5955,5956],{"class":1102},"response_ms_ok ",[407,5958,2499],{"class":416},[407,5960,4739],{"class":1102},[407,5962,167],{"class":416},[407,5964,5965],{"class":1102},"traffic ",[407,5967,2238],{"class":1095},[407,5969,3346],{"class":557},[407,5971,4459],{"class":1095},[407,5973,4739],{"class":1102},[407,5975,167],{"class":416},[407,5977,5978],{"class":1102},"avg_response_ms ",[407,5980,3199],{"class":1095},[407,5982,4326],{"class":1102},[407,5984,167],{"class":416},[407,5986,5987],{"class":1102},"scale_down_response_ms\n",[407,5989,5990,5993,5995],{"class":409,"line":20},[407,5991,5992],{"class":1102},"down_eligible ",[407,5994,2499],{"class":416},[407,5996,5997],{"class":416}," (\n",[407,5999,6000,6002,6004,6006,6008,6010,6012],{"class":409,"line":427},[407,6001,4541],{"class":1102},[407,6003,167],{"class":416},[407,6005,5345],{"class":1102},[407,6007,3199],{"class":1095},[407,6009,4326],{"class":1102},[407,6011,167],{"class":416},[407,6013,6014],{"class":1102},"scale_down_cpu\n",[407,6016,6017,6020,6022,6024,6026,6028,6030,6032,6035],{"class":409,"line":440},[407,6018,6019],{"class":1095},"    and",[407,6021,4739],{"class":1102},[407,6023,167],{"class":416},[407,6025,4882],{"class":1102},[407,6027,3199],{"class":1095},[407,6029,4326],{"class":1102},[407,6031,167],{"class":416},[407,6033,6034],{"class":1102},"scale_up_cpu_max      ",[407,6036,6037],{"class":736},"# one hot replica blocks scale-down\n",[407,6039,6040,6042,6044,6046,6049,6051,6053,6055],{"class":409,"line":457},[407,6041,6019],{"class":1095},[407,6043,4739],{"class":1102},[407,6045,167],{"class":416},[407,6047,6048],{"class":1102},"avg_mem ",[407,6050,3199],{"class":1095},[407,6052,4326],{"class":1102},[407,6054,167],{"class":416},[407,6056,6057],{"class":1102},"scale_down_mem\n",[407,6059,6060,6062,6064,6066,6069,6071,6073,6075],{"class":409,"line":465},[407,6061,6019],{"class":1095},[407,6063,4739],{"class":1102},[407,6065,167],{"class":416},[407,6067,6068],{"class":1102},"open_connections ",[407,6070,3199],{"class":1095},[407,6072,4326],{"class":1102},[407,6074,167],{"class":416},[407,6076,6077],{"class":1102},"scale_down_connections\n",[407,6079,6080,6082],{"class":409,"line":477},[407,6081,6019],{"class":1095},[407,6083,6084],{"class":1102}," response_ms_ok\n",[407,6086,6087,6089,6091,6093,6095,6097,6099,6101],{"class":409,"line":485},[407,6088,6019],{"class":1095},[407,6090,4739],{"class":1102},[407,6092,167],{"class":416},[407,6094,4744],{"class":1102},[407,6096,1848],{"class":1095},[407,6098,4326],{"class":1102},[407,6100,167],{"class":416},[407,6102,6103],{"class":1102},"min_replicas\n",[407,6105,6106,6108,6110,6112,6115,6117],{"class":409,"line":494},[407,6107,6019],{"class":1095},[407,6109,4739],{"class":1102},[407,6111,167],{"class":416},[407,6113,6114],{"class":1102},"replicas_pending ",[407,6116,3365],{"class":1095},[407,6118,3630],{"class":473},[407,6120,6121,6123,6125,6127,6130,6132],{"class":409,"line":502},[407,6122,6019],{"class":1095},[407,6124,4739],{"class":1102},[407,6126,167],{"class":416},[407,6128,6129],{"class":1102},"replicas_failed ",[407,6131,3365],{"class":1095},[407,6133,3630],{"class":473},[407,6135,6136],{"class":409,"line":510},[407,6137,2764],{"class":416},[34,6139,6140],{},"Two subtleties:",[147,6142,6143,6157],{},[150,6144,6145,6148,6149,6152,6153,6156],{},[37,6146,6147],{},"Response time is only enforced when Traefik data exists."," A non-HTTP worker reports 0ms,\nwhich would always satisfy ",[98,6150,6151],{},"\u003C scale_down_response_ms"," and bias every worker toward scale-down.\nGuarding on ",[98,6154,6155],{},"metrics.traffic is None"," prevents that false signal.",[150,6158,6159,6168],{},[37,6160,6161,6164,6165,167],{},[98,6162,6163],{},"max_cpu"," is checked here, not just ",[98,6166,6167],{},"avg_cpu"," One hot replica among several idle ones can\nhide in the average. Checking the max means a single busy container blocks removal.",[4674,6170,6172],{"id":6171},"step-8-scale-down-must-also-be-sustained","Step 8: Scale-down must also be sustained",[260,6174,6176],{"className":2436,"code":6175,"language":2438,"meta":266,"style":266},"if elapsed_scale_down_seconds \u003C policy.scale_down_duration:\n    return self._none(svc, \"...conditions met for Xs, need Ys\", BLOCKED_SCALE_DOWN_NOT_SUSTAINED), False\n",[98,6177,6178,6196],{"__ignoreMap":266},[407,6179,6180,6182,6185,6187,6189,6191,6194],{"class":409,"line":11},[407,6181,3354],{"class":557},[407,6183,6184],{"class":1102}," elapsed_scale_down_seconds ",[407,6186,3199],{"class":1095},[407,6188,4326],{"class":1102},[407,6190,167],{"class":416},[407,6192,6193],{"class":1102},"scale_down_duration",[407,6195,417],{"class":416},[407,6197,6198,6200,6202,6204,6206,6208,6210,6212,6214,6217,6219,6221,6224,6226],{"class":409,"line":20},[407,6199,3385],{"class":557},[407,6201,2536],{"class":1088},[407,6203,167],{"class":416},[407,6205,4710],{"class":1102},[407,6207,2451],{"class":416},[407,6209,2756],{"class":1102},[407,6211,750],{"class":416},[407,6213,1994],{"class":1877},[407,6215,6216],{"class":436},"...conditions met for Xs, need Ys",[407,6218,2000],{"class":1877},[407,6220,750],{"class":416},[407,6222,6223],{"class":1088}," BLOCKED_SCALE_DOWN_NOT_SUSTAINED",[407,6225,2671],{"class":416},[407,6227,4358],{"class":557},[4674,6229,6231],{"id":6230},"step-9-proportional-step","Step 9: Proportional step",[34,6233,6234],{},"Scale-up steps by urgency; scale-down normally steps by one. But if a service is deeply idle,\nremoving one replica per cooldown period is painfully slow. So the step is proportional to how\nfar below the threshold the load is, hard-capped so you can never halve capacity in one move:",[260,6236,6238],{"className":2436,"code":6237,"language":2438,"meta":266,"style":266},"step = 1\nif n > policy.min_replicas + 1 and policy.scale_down_cpu > 0 and policy.scale_down_mem > 0:\n    cpu_r  = metrics.avg_cpu \u002F policy.scale_down_cpu\n    mem_r  = metrics.avg_mem \u002F policy.scale_down_mem\n    load_r = max(cpu_r, mem_r, 0.0)\n    target = max(policy.min_replicas, math.ceil(load_r * n))\n    step   = max(1, min(n - target, n - policy.min_replicas))\n    step   = min(step, self._SCALE_DOWN_MAX_STEP)   # never remove too many at once\n",[98,6239,6240,6249,6294,6315,6336,6362,6401,6442],{"__ignoreMap":266},[407,6241,6242,6245,6247],{"class":409,"line":11},[407,6243,6244],{"class":1102},"step ",[407,6246,2499],{"class":416},[407,6248,1696],{"class":473},[407,6250,6251,6253,6256,6258,6260,6262,6264,6266,6268,6270,6272,6274,6276,6278,6280,6282,6284,6286,6288,6290,6292],{"class":409,"line":20},[407,6252,3354],{"class":557},[407,6254,6255],{"class":1102}," n ",[407,6257,1848],{"class":1095},[407,6259,4326],{"class":1102},[407,6261,167],{"class":416},[407,6263,4345],{"class":1102},[407,6265,5869],{"class":1095},[407,6267,4350],{"class":473},[407,6269,4751],{"class":1095},[407,6271,4326],{"class":1102},[407,6273,167],{"class":416},[407,6275,4394],{"class":1102},[407,6277,1848],{"class":1095},[407,6279,3180],{"class":473},[407,6281,4751],{"class":1095},[407,6283,4326],{"class":1102},[407,6285,167],{"class":416},[407,6287,4425],{"class":1102},[407,6289,1848],{"class":1095},[407,6291,3180],{"class":473},[407,6293,417],{"class":416},[407,6295,6296,6299,6301,6303,6305,6307,6309,6311,6313],{"class":409,"line":427},[407,6297,6298],{"class":1102},"    cpu_r  ",[407,6300,2499],{"class":416},[407,6302,4739],{"class":1102},[407,6304,167],{"class":416},[407,6306,5345],{"class":1102},[407,6308,3217],{"class":1095},[407,6310,4326],{"class":1102},[407,6312,167],{"class":416},[407,6314,6014],{"class":1102},[407,6316,6317,6320,6322,6324,6326,6328,6330,6332,6334],{"class":409,"line":440},[407,6318,6319],{"class":1102},"    mem_r  ",[407,6321,2499],{"class":416},[407,6323,4739],{"class":1102},[407,6325,167],{"class":416},[407,6327,6048],{"class":1102},[407,6329,3217],{"class":1095},[407,6331,4326],{"class":1102},[407,6333,167],{"class":416},[407,6335,6057],{"class":1102},[407,6337,6338,6341,6343,6345,6347,6350,6352,6355,6357,6360],{"class":409,"line":457},[407,6339,6340],{"class":1102},"    load_r ",[407,6342,2499],{"class":416},[407,6344,2797],{"class":412},[407,6346,2451],{"class":416},[407,6348,6349],{"class":1102},"cpu_r",[407,6351,750],{"class":416},[407,6353,6354],{"class":1102}," mem_r",[407,6356,750],{"class":416},[407,6358,6359],{"class":473}," 0.0",[407,6361,2764],{"class":416},[407,6363,6364,6367,6369,6371,6373,6375,6377,6379,6381,6384,6386,6389,6391,6394,6396,6399],{"class":409,"line":465},[407,6365,6366],{"class":1102},"    target ",[407,6368,2499],{"class":416},[407,6370,2797],{"class":412},[407,6372,2451],{"class":416},[407,6374,4260],{"class":1102},[407,6376,167],{"class":416},[407,6378,1403],{"class":1102},[407,6380,750],{"class":416},[407,6382,6383],{"class":1102}," math",[407,6385,167],{"class":416},[407,6387,6388],{"class":1102},"ceil",[407,6390,2451],{"class":416},[407,6392,6393],{"class":1102},"load_r ",[407,6395,3222],{"class":1095},[407,6397,6398],{"class":1102}," n",[407,6400,3349],{"class":416},[407,6402,6403,6406,6408,6410,6412,6414,6416,6418,6420,6423,6425,6428,6430,6432,6434,6436,6438,6440],{"class":409,"line":477},[407,6404,6405],{"class":1102},"    step   ",[407,6407,2499],{"class":416},[407,6409,2797],{"class":412},[407,6411,2451],{"class":416},[407,6413,3286],{"class":473},[407,6415,750],{"class":416},[407,6417,5517],{"class":412},[407,6419,2451],{"class":416},[407,6421,6422],{"class":1102},"n ",[407,6424,2810],{"class":1095},[407,6426,6427],{"class":1102}," target",[407,6429,750],{"class":416},[407,6431,6255],{"class":1102},[407,6433,2810],{"class":1095},[407,6435,4326],{"class":1102},[407,6437,167],{"class":416},[407,6439,1403],{"class":1102},[407,6441,3349],{"class":416},[407,6443,6444,6446,6448,6450,6452,6455,6457,6459,6461,6464,6466],{"class":409,"line":485},[407,6445,6405],{"class":1102},[407,6447,2499],{"class":416},[407,6449,5517],{"class":412},[407,6451,2451],{"class":416},[407,6453,6454],{"class":1102},"step",[407,6456,750],{"class":416},[407,6458,2536],{"class":1088},[407,6460,167],{"class":416},[407,6462,6463],{"class":1088},"_SCALE_DOWN_MAX_STEP",[407,6465,2457],{"class":416},[407,6467,6468],{"class":736},"   # never remove too many at once\n",[4674,6470,6472],{"id":6471},"step-10-bounds-check","Step 10: Bounds check",[34,6474,6475,6476,6478],{},"Clamp the step so the service can never drop below ",[98,6477,1403],{},", and skip the scale-down entirely if there is nothing left to remove.",[4674,6480,6482],{"id":6481},"step-11-the-removal-simulation","Step 11: The removal simulation",[34,6484,6485,6486,6489],{},"Before removing a replica, the engine ",[154,6487,6488],{},"simulates"," what happens to the survivors. This is the\nguard that stops a classic oscillation: scale down → survivors overload → immediate scale up →\ncooldown → scale down → ...",[34,6491,6492,6493,6496],{},"The crucial insight is that ",[37,6494,6495],{},"different metrics redistribute differently",", and our first\nimplementation treated them all the same, which was wrong.",[34,6498,6499,6502,6503,6506],{},[37,6500,6501],{},"CPU redistributes proportionally."," Remove a replica and the same request volume lands on\nfewer containers, so projecting ",[98,6504,6505],{},"cpu × n \u002F (n-step)"," is valid:",[260,6508,6510],{"className":2436,"code":6509,"language":2438,"meta":266,"style":266},"projected_cpu = (metrics.avg_cpu * n) \u002F projected_n\nif projected_cpu > policy.scale_up_cpu_avg * self._SIMULATION_SAFETY_MARGIN:   # 85%\n    return self._none(svc, \"simulation failed: projected avg cpu ...\", BLOCKED_SIM_FAILED_CPU), False\n",[98,6511,6512,6538,6568],{"__ignoreMap":266},[407,6513,6514,6517,6519,6521,6523,6525,6527,6529,6531,6533,6535],{"class":409,"line":11},[407,6515,6516],{"class":1102},"projected_cpu ",[407,6518,2499],{"class":416},[407,6520,1277],{"class":416},[407,6522,5368],{"class":1102},[407,6524,167],{"class":416},[407,6526,5345],{"class":1102},[407,6528,3222],{"class":1095},[407,6530,6398],{"class":1102},[407,6532,2457],{"class":416},[407,6534,5385],{"class":1095},[407,6536,6537],{"class":1102}," projected_n\n",[407,6539,6540,6542,6545,6547,6549,6551,6554,6556,6558,6560,6563,6565],{"class":409,"line":20},[407,6541,3354],{"class":557},[407,6543,6544],{"class":1102}," projected_cpu ",[407,6546,1848],{"class":1095},[407,6548,4326],{"class":1102},[407,6550,167],{"class":416},[407,6552,6553],{"class":1102},"scale_up_cpu_avg ",[407,6555,3222],{"class":1095},[407,6557,2536],{"class":1088},[407,6559,167],{"class":416},[407,6561,6562],{"class":1088},"_SIMULATION_SAFETY_MARGIN",[407,6564,433],{"class":416},[407,6566,6567],{"class":736},"   # 85%\n",[407,6569,6570,6572,6574,6576,6578,6580,6582,6584,6586,6589,6591,6593,6596,6598],{"class":409,"line":427},[407,6571,3385],{"class":557},[407,6573,2536],{"class":1088},[407,6575,167],{"class":416},[407,6577,4710],{"class":1102},[407,6579,2451],{"class":416},[407,6581,2756],{"class":1102},[407,6583,750],{"class":416},[407,6585,1994],{"class":1877},[407,6587,6588],{"class":436},"simulation failed: projected avg cpu ...",[407,6590,2000],{"class":1877},[407,6592,750],{"class":416},[407,6594,6595],{"class":1088}," BLOCKED_SIM_FAILED_CPU",[407,6597,2671],{"class":416},[407,6599,4358],{"class":557},[34,6601,6602,6605,6606,6609,6610,6613],{},[37,6603,6604],{},"Memory does NOT redistribute."," A container's heap, page cache, and mmapped files are\nprocess-private: they don't transfer to siblings when you remove one. The old formula\n",[98,6607,6608],{},"avg_mem × n \u002F (n-step)"," treated memory like a liquid and made scale-down nearly impossible for\nany memory-heavy service. The fix is to check that ",[154,6611,6612],{},"current"," memory already has headroom below\nthe scale-up ceiling:",[260,6615,6617],{"className":2436,"code":6616,"language":2438,"meta":266,"style":266},"# memory is process-private, not redistributed\nif metrics.avg_mem > policy.scale_up_mem_avg * self._SIMULATION_SAFETY_MARGIN:\n    return self._none(svc, \"simulation failed: current avg mem ...\", BLOCKED_SIM_FAILED_MEM), False\n",[98,6618,6619,6624,6653],{"__ignoreMap":266},[407,6620,6621],{"class":409,"line":11},[407,6622,6623],{"class":736},"# memory is process-private, not redistributed\n",[407,6625,6626,6628,6630,6632,6634,6636,6638,6640,6643,6645,6647,6649,6651],{"class":409,"line":20},[407,6627,3354],{"class":557},[407,6629,4739],{"class":1102},[407,6631,167],{"class":416},[407,6633,6048],{"class":1102},[407,6635,1848],{"class":1095},[407,6637,4326],{"class":1102},[407,6639,167],{"class":416},[407,6641,6642],{"class":1102},"scale_up_mem_avg ",[407,6644,3222],{"class":1095},[407,6646,2536],{"class":1088},[407,6648,167],{"class":416},[407,6650,6562],{"class":1088},[407,6652,417],{"class":416},[407,6654,6655,6657,6659,6661,6663,6665,6667,6669,6671,6674,6676,6678,6681,6683],{"class":409,"line":427},[407,6656,3385],{"class":557},[407,6658,2536],{"class":1088},[407,6660,167],{"class":416},[407,6662,4710],{"class":1102},[407,6664,2451],{"class":416},[407,6666,2756],{"class":1102},[407,6668,750],{"class":416},[407,6670,1994],{"class":1877},[407,6672,6673],{"class":436},"simulation failed: current avg mem ...",[407,6675,2000],{"class":1877},[407,6677,750],{"class":416},[407,6679,6680],{"class":1088}," BLOCKED_SIM_FAILED_MEM",[407,6682,2671],{"class":416},[407,6684,4358],{"class":557},[34,6686,6687,6690,6691,6694],{},[37,6688,6689],{},"Response time is non-linear."," Latency is governed by queue depth and upstream bottlenecks\n(DB, cache, network), not simply by ",[98,6692,6693],{},"1\u002Fn"," servers. Projecting it proportionally is meaningless:\nif a database is the bottleneck, removing a replica doesn't change it. So again the engine checks\nfor genuine current headroom rather than projecting:",[260,6696,6698],{"className":2436,"code":6697,"language":2438,"meta":266,"style":266},"if metrics.traffic is not None and \\\n   metrics.avg_response_ms > policy.scale_up_response_ms * self._SIMULATION_SAFETY_MARGIN:\n    return self._none(svc, \"simulation failed: current response ...\", BLOCKED_SIM_FAILED_RESPONSE), False\n",[98,6699,6700,6720,6748],{"__ignoreMap":266},[407,6701,6702,6704,6706,6708,6710,6712,6714,6716,6718],{"class":409,"line":11},[407,6703,3354],{"class":557},[407,6705,4739],{"class":1102},[407,6707,167],{"class":416},[407,6709,5965],{"class":1102},[407,6711,2238],{"class":1095},[407,6713,2555],{"class":1095},[407,6715,3346],{"class":557},[407,6717,4751],{"class":1095},[407,6719,1921],{"class":416},[407,6721,6722,6725,6727,6729,6731,6733,6735,6738,6740,6742,6744,6746],{"class":409,"line":20},[407,6723,6724],{"class":1102},"   metrics",[407,6726,167],{"class":416},[407,6728,5978],{"class":1102},[407,6730,1848],{"class":1095},[407,6732,4326],{"class":1102},[407,6734,167],{"class":416},[407,6736,6737],{"class":1102},"scale_up_response_ms ",[407,6739,3222],{"class":1095},[407,6741,2536],{"class":1088},[407,6743,167],{"class":416},[407,6745,6562],{"class":1088},[407,6747,417],{"class":416},[407,6749,6750,6752,6754,6756,6758,6760,6762,6764,6766,6769,6771,6773,6776,6778],{"class":409,"line":427},[407,6751,3385],{"class":557},[407,6753,2536],{"class":1088},[407,6755,167],{"class":416},[407,6757,4710],{"class":1102},[407,6759,2451],{"class":416},[407,6761,2756],{"class":1102},[407,6763,750],{"class":416},[407,6765,1994],{"class":1877},[407,6767,6768],{"class":436},"simulation failed: current response ...",[407,6770,2000],{"class":1877},[407,6772,750],{"class":416},[407,6774,6775],{"class":1088}," BLOCKED_SIM_FAILED_RESPONSE",[407,6777,2671],{"class":416},[407,6779,4358],{"class":557},[34,6781,6782,6783,167],{},"If every projection passes, the decision is finally a ",[98,6784,6785],{},"SCALE_DOWN",[34,6787,6788],{},"Try it. The default preset is a service that looks idle enough to shrink but would overload the survivors. Then try the memory-heavy preset, where the old formula would have wrongly refused.",[6790,6791],"swarm-scale-down-sim",{},[241,6793,6795],{"id":6794},"the-host_scale_signal","The host_scale_signal",[34,6797,6798,6799,6802,6803,6806],{},"Threaded through the whole engine is a second return value: ",[98,6800,6801],{},"host_scale_signal",". It is set\n",[98,6804,6805],{},"True"," whenever the service wants capacity the cluster can't give:",[147,6808,6809,6815,6818],{},[150,6810,6811,6812,2671],{},"tasks are being rejected (",[98,6813,6814],{},"replicas_rejected > 0",[150,6816,6817],{},"tasks are stuck pending past the threshold,",[150,6819,6820,6821,167],{},"triggers are firing but the service is already at ",[98,6822,4377],{},[34,6824,6825,6826,6829,6830,6832],{},"This signal is deliberately ",[37,6827,6828],{},"separate"," from the ",[98,6831,4649],{},". The decision is about\ncontainers; the signal is about VMs. The watch loop reads the signal and forwards it to the\nOpenStack ASG.",[53,6834],{},[42,6836,6838],{"id":6837},"part-3-the-executor","Part 3: The Executor",[34,6840,6841],{},"The executor has exactly one job: make a decision real, verify it, and report honestly. No\ndecisions of its own, no retries.",[241,6843,6845],{"id":6844},"calculate-the-target-then-skip-if-its-a-no-op","Calculate the target, then skip if it's a no-op",[260,6847,6849],{"className":2436,"code":6848,"language":2438,"meta":266,"style":266},"if decision.action == ScalingAction.NONE:\n    return None   # the watch loop handles NONE logging; executor never touches Docker for NONE\n\ntarget = self._calculate_target(decision, metrics, policy)\nif target == metrics.replicas_running + metrics.replicas_pending:\n    return None   # nothing to do\n",[98,6850,6851,6873,6882,6886,6915,6941],{"__ignoreMap":266},[407,6852,6853,6855,6858,6860,6863,6865,6867,6869,6871],{"class":409,"line":11},[407,6854,3354],{"class":557},[407,6856,6857],{"class":1102}," decision",[407,6859,167],{"class":416},[407,6861,6862],{"class":1102},"action ",[407,6864,3365],{"class":1095},[407,6866,4582],{"class":1102},[407,6868,167],{"class":416},[407,6870,5004],{"class":1088},[407,6872,417],{"class":416},[407,6874,6875,6877,6879],{"class":409,"line":20},[407,6876,3385],{"class":557},[407,6878,3346],{"class":557},[407,6880,6881],{"class":736},"   # the watch loop handles NONE logging; executor never touches Docker for NONE\n",[407,6883,6884],{"class":409,"line":427},[407,6885,530],{"emptyLinePlaceholder":529},[407,6887,6888,6891,6893,6895,6897,6900,6902,6905,6907,6909,6911,6913],{"class":409,"line":440},[407,6889,6890],{"class":1102},"target ",[407,6892,2499],{"class":416},[407,6894,2536],{"class":1088},[407,6896,167],{"class":416},[407,6898,6899],{"class":1102},"_calculate_target",[407,6901,2451],{"class":416},[407,6903,6904],{"class":1102},"decision",[407,6906,750],{"class":416},[407,6908,4739],{"class":1102},[407,6910,750],{"class":416},[407,6912,4326],{"class":1102},[407,6914,2764],{"class":416},[407,6916,6917,6919,6922,6924,6926,6928,6930,6932,6934,6936,6939],{"class":409,"line":457},[407,6918,3354],{"class":557},[407,6920,6921],{"class":1102}," target ",[407,6923,3365],{"class":1095},[407,6925,4739],{"class":1102},[407,6927,167],{"class":416},[407,6929,4744],{"class":1102},[407,6931,5869],{"class":1095},[407,6933,4739],{"class":1102},[407,6935,167],{"class":416},[407,6937,6938],{"class":1102},"replicas_pending",[407,6940,417],{"class":416},[407,6942,6943,6945,6947],{"class":409,"line":465},[407,6944,3385],{"class":557},[407,6946,3346],{"class":557},[407,6948,6949],{"class":736},"   # nothing to do\n",[241,6951,6953],{"id":6952},"dont-fight-an-active-deployment","Don't fight an active deployment",[34,6955,6956,6957,6959],{},"A subtle production hazard: if an operator is mid ",[98,6958,2226],{}," (a rolling update),\nscaling the service at the same time corrupts the rollout. The executor checks for an in-progress\nupdate and backs off, unless the decision is a forced ghost-recovery:",[260,6961,6963],{"className":2436,"code":6962,"language":2438,"meta":266,"style":266},"update_status = service.attrs.get(\"UpdateStatus\", {})\nis_updating   = update_status.get(\"State\") == \"updating\"\nif is_updating and not decision.force:\n    return self._result(..., success=False, error=\"active_deployment\")\n",[98,6964,6965,6997,7031,7051],{"__ignoreMap":266},[407,6966,6967,6970,6972,6974,6976,6979,6981,6983,6985,6987,6990,6992,6994],{"class":409,"line":11},[407,6968,6969],{"class":1102},"update_status ",[407,6971,2499],{"class":416},[407,6973,1905],{"class":1102},[407,6975,167],{"class":416},[407,6977,6978],{"class":1102},"attrs",[407,6980,167],{"class":416},[407,6982,3135],{"class":1102},[407,6984,2451],{"class":416},[407,6986,2000],{"class":1877},[407,6988,6989],{"class":436},"UpdateStatus",[407,6991,2000],{"class":1877},[407,6993,750],{"class":416},[407,6995,6996],{"class":416}," {})\n",[407,6998,6999,7002,7004,7007,7009,7011,7013,7015,7018,7020,7022,7024,7026,7029],{"class":409,"line":20},[407,7000,7001],{"class":1102},"is_updating   ",[407,7003,2499],{"class":416},[407,7005,7006],{"class":1102}," update_status",[407,7008,167],{"class":416},[407,7010,3135],{"class":1102},[407,7012,2451],{"class":416},[407,7014,2000],{"class":1877},[407,7016,7017],{"class":436},"State",[407,7019,2000],{"class":1877},[407,7021,2457],{"class":416},[407,7023,4160],{"class":1095},[407,7025,1994],{"class":1877},[407,7027,7028],{"class":436},"updating",[407,7030,2383],{"class":1877},[407,7032,7033,7035,7038,7040,7042,7044,7046,7049],{"class":409,"line":427},[407,7034,3354],{"class":557},[407,7036,7037],{"class":1102}," is_updating ",[407,7039,3360],{"class":1095},[407,7041,2555],{"class":1095},[407,7043,6857],{"class":1102},[407,7045,167],{"class":416},[407,7047,7048],{"class":1102},"force",[407,7050,417],{"class":416},[407,7052,7053,7055,7057,7059,7062,7064,7066,7068,7071,7073,7076,7078,7081,7083,7085,7088,7090],{"class":409,"line":440},[407,7054,3385],{"class":557},[407,7056,2536],{"class":1088},[407,7058,167],{"class":416},[407,7060,7061],{"class":1102},"_result",[407,7063,2451],{"class":416},[407,7065,4994],{"class":1088},[407,7067,750],{"class":416},[407,7069,7070],{"class":1836}," success",[407,7072,2499],{"class":416},[407,7074,7075],{"class":557},"False",[407,7077,750],{"class":416},[407,7079,7080],{"class":1836}," error",[407,7082,2499],{"class":416},[407,7084,2000],{"class":1877},[407,7086,7087],{"class":436},"active_deployment",[407,7089,2000],{"class":1877},[407,7091,2764],{"class":416},[241,7093,7095],{"id":7094},"scale-and-classify-failures-precisely","Scale and classify failures precisely",[260,7097,7099],{"className":2436,"code":7098,"language":2438,"meta":266,"style":266},"try:\n    service = self.client.services.get(metrics.service)\n    service.scale(target)\nexcept docker.errors.NotFound:\n    return self._result(..., error=EXEC_ERROR_NOT_FOUND)    # service removed between eval & exec\nexcept Exception as exc:\n    return self._result(..., error=str(exc))\n",[98,7100,7101,7108,7140,7156,7174,7202,7216],{"__ignoreMap":266},[407,7102,7103,7106],{"class":409,"line":11},[407,7104,7105],{"class":557},"try",[407,7107,417],{"class":416},[407,7109,7110,7113,7115,7117,7119,7122,7124,7126,7128,7130,7132,7134,7136,7138],{"class":409,"line":20},[407,7111,7112],{"class":1102},"    service ",[407,7114,2499],{"class":416},[407,7116,2536],{"class":1088},[407,7118,167],{"class":416},[407,7120,7121],{"class":1102},"client",[407,7123,167],{"class":416},[407,7125,413],{"class":1102},[407,7127,167],{"class":416},[407,7129,3135],{"class":1102},[407,7131,2451],{"class":416},[407,7133,5368],{"class":1102},[407,7135,167],{"class":416},[407,7137,3926],{"class":1102},[407,7139,2764],{"class":416},[407,7141,7142,7144,7146,7149,7151,7154],{"class":409,"line":427},[407,7143,3573],{"class":1102},[407,7145,167],{"class":416},[407,7147,7148],{"class":1102},"scale",[407,7150,2451],{"class":416},[407,7152,7153],{"class":1102},"target",[407,7155,2764],{"class":416},[407,7157,7158,7160,7162,7164,7167,7169,7172],{"class":409,"line":440},[407,7159,5692],{"class":557},[407,7161,1988],{"class":1102},[407,7163,167],{"class":416},[407,7165,7166],{"class":1102},"errors",[407,7168,167],{"class":416},[407,7170,7171],{"class":1102},"NotFound",[407,7173,417],{"class":416},[407,7175,7176,7178,7180,7182,7184,7186,7188,7190,7192,7194,7197,7199],{"class":409,"line":457},[407,7177,3385],{"class":557},[407,7179,2536],{"class":1088},[407,7181,167],{"class":416},[407,7183,7061],{"class":1102},[407,7185,2451],{"class":416},[407,7187,4994],{"class":1088},[407,7189,750],{"class":416},[407,7191,7080],{"class":1836},[407,7193,2499],{"class":416},[407,7195,7196],{"class":1088},"EXEC_ERROR_NOT_FOUND",[407,7198,2457],{"class":416},[407,7200,7201],{"class":736},"    # service removed between eval & exec\n",[407,7203,7204,7206,7209,7211,7214],{"class":409,"line":465},[407,7205,5692],{"class":557},[407,7207,7208],{"class":412}," Exception",[407,7210,2696],{"class":557},[407,7212,7213],{"class":1102}," exc",[407,7215,417],{"class":416},[407,7217,7218,7220,7222,7224,7226,7228,7230,7232,7234,7236,7238,7240,7243],{"class":409,"line":477},[407,7219,3385],{"class":557},[407,7221,2536],{"class":1088},[407,7223,167],{"class":416},[407,7225,7061],{"class":1102},[407,7227,2451],{"class":416},[407,7229,4994],{"class":1088},[407,7231,750],{"class":416},[407,7233,7080],{"class":1836},[407,7235,2499],{"class":416},[407,7237,2483],{"class":412},[407,7239,2451],{"class":416},[407,7241,7242],{"class":1102},"exc",[407,7244,3349],{"class":416},[241,7246,7248],{"id":7247},"verify-by-polling-partial-success-is-a-real-outcome","Verify by polling: partial success is a real outcome",[34,7250,7251,7252,7255],{},"Calling ",[98,7253,7254],{},"scale()"," returns immediately; it does not mean the replicas are running. The executor\npolls until the running count matches the target, or a rejection appears, or it times out:",[260,7257,7259],{"className":2436,"code":7258,"language":2438,"meta":266,"style":266},"while time.time() \u003C deadline:\n    time.sleep(_POLL_INTERVAL)\n    running, rejected = self._count_task_states(service_name)\n\n    if rejected > 0:\n        return self._result(..., replicas_after=running, success=False, error=EXEC_ERROR_REJECTED)\n    if running == target:\n        return self._result(..., replicas_after=running, success=True)\n\n# timeout: record what actually happened, not what we asked for\nrunning, _ = self._count_task_states(service_name)\nreturn self._result(..., replicas_after=running, success=False, error=EXEC_ERROR_TIMEOUT)\n",[98,7260,7261,7282,7299,7325,7329,7341,7384,7397,7429,7433,7438,7461],{"__ignoreMap":266},[407,7262,7263,7266,7268,7270,7273,7275,7277,7280],{"class":409,"line":11},[407,7264,7265],{"class":557},"while",[407,7267,2617],{"class":1102},[407,7269,167],{"class":416},[407,7271,7272],{"class":1102},"time",[407,7274,2625],{"class":416},[407,7276,1096],{"class":1095},[407,7278,7279],{"class":1102}," deadline",[407,7281,417],{"class":416},[407,7283,7284,7287,7289,7292,7294,7297],{"class":409,"line":20},[407,7285,7286],{"class":1102},"    time",[407,7288,167],{"class":416},[407,7290,7291],{"class":1102},"sleep",[407,7293,2451],{"class":416},[407,7295,7296],{"class":1088},"_POLL_INTERVAL",[407,7298,2764],{"class":416},[407,7300,7301,7304,7306,7309,7311,7313,7315,7318,7320,7323],{"class":409,"line":427},[407,7302,7303],{"class":1102},"    running",[407,7305,750],{"class":416},[407,7307,7308],{"class":1102}," rejected ",[407,7310,2499],{"class":416},[407,7312,2536],{"class":1088},[407,7314,167],{"class":416},[407,7316,7317],{"class":1102},"_count_task_states",[407,7319,2451],{"class":416},[407,7321,7322],{"class":1102},"service_name",[407,7324,2764],{"class":416},[407,7326,7327],{"class":409,"line":440},[407,7328,530],{"emptyLinePlaceholder":529},[407,7330,7331,7333,7335,7337,7339],{"class":409,"line":457},[407,7332,2552],{"class":557},[407,7334,7308],{"class":1102},[407,7336,1848],{"class":1095},[407,7338,3180],{"class":473},[407,7340,417],{"class":416},[407,7342,7343,7345,7347,7349,7351,7353,7355,7357,7360,7362,7365,7367,7369,7371,7373,7375,7377,7379,7382],{"class":409,"line":465},[407,7344,2565],{"class":557},[407,7346,2536],{"class":1088},[407,7348,167],{"class":416},[407,7350,7061],{"class":1102},[407,7352,2451],{"class":416},[407,7354,4994],{"class":1088},[407,7356,750],{"class":416},[407,7358,7359],{"class":1836}," replicas_after",[407,7361,2499],{"class":416},[407,7363,7364],{"class":1102},"running",[407,7366,750],{"class":416},[407,7368,7070],{"class":1836},[407,7370,2499],{"class":416},[407,7372,7075],{"class":557},[407,7374,750],{"class":416},[407,7376,7080],{"class":1836},[407,7378,2499],{"class":416},[407,7380,7381],{"class":1088},"EXEC_ERROR_REJECTED",[407,7383,2764],{"class":416},[407,7385,7386,7388,7391,7393,7395],{"class":409,"line":477},[407,7387,2552],{"class":557},[407,7389,7390],{"class":1102}," running ",[407,7392,3365],{"class":1095},[407,7394,6427],{"class":1102},[407,7396,417],{"class":416},[407,7398,7399,7401,7403,7405,7407,7409,7411,7413,7415,7417,7419,7421,7423,7425,7427],{"class":409,"line":485},[407,7400,2565],{"class":557},[407,7402,2536],{"class":1088},[407,7404,167],{"class":416},[407,7406,7061],{"class":1102},[407,7408,2451],{"class":416},[407,7410,4994],{"class":1088},[407,7412,750],{"class":416},[407,7414,7359],{"class":1836},[407,7416,2499],{"class":416},[407,7418,7364],{"class":1102},[407,7420,750],{"class":416},[407,7422,7070],{"class":1836},[407,7424,2499],{"class":416},[407,7426,6805],{"class":557},[407,7428,2764],{"class":416},[407,7430,7431],{"class":409,"line":494},[407,7432,530],{"emptyLinePlaceholder":529},[407,7434,7435],{"class":409,"line":502},[407,7436,7437],{"class":736},"# timeout: record what actually happened, not what we asked for\n",[407,7439,7440,7442,7444,7447,7449,7451,7453,7455,7457,7459],{"class":409,"line":510},[407,7441,7364],{"class":1102},[407,7443,750],{"class":416},[407,7445,7446],{"class":1102}," _ ",[407,7448,2499],{"class":416},[407,7450,2536],{"class":1088},[407,7452,167],{"class":416},[407,7454,7317],{"class":1102},[407,7456,2451],{"class":416},[407,7458,7322],{"class":1102},[407,7460,2764],{"class":416},[407,7462,7463,7466,7468,7470,7472,7474,7476,7478,7480,7482,7484,7486,7488,7490,7492,7494,7496,7498,7501],{"class":409,"line":518},[407,7464,7465],{"class":557},"return",[407,7467,2536],{"class":1088},[407,7469,167],{"class":416},[407,7471,7061],{"class":1102},[407,7473,2451],{"class":416},[407,7475,4994],{"class":1088},[407,7477,750],{"class":416},[407,7479,7359],{"class":1836},[407,7481,2499],{"class":416},[407,7483,7364],{"class":1102},[407,7485,750],{"class":416},[407,7487,7070],{"class":1836},[407,7489,2499],{"class":416},[407,7491,7075],{"class":557},[407,7493,750],{"class":416},[407,7495,7080],{"class":1836},[407,7497,2499],{"class":416},[407,7499,7500],{"class":1088},"EXEC_ERROR_TIMEOUT",[407,7502,2764],{"class":416},[34,7504,7505,7506,7509,7510,1462,7513,1462,7516,7519,7520,7523],{},"The ",[98,7507,7508],{},"ExecutionResult"," records ",[98,7511,7512],{},"replicas_before",[98,7514,7515],{},"replicas_after",[98,7517,7518],{},"requested_change",", and\n",[98,7521,7522],{},"actual_change"," separately, so a scale from 3→5 that only reached 4 is recorded faithfully as a\npartial success rather than a lie.",[241,7525,7527],{"id":7526},"no-retries-on-purpose","No retries, on purpose",[34,7529,7530],{},"The executor never retries. If a scale fails, the watch loop will re-evaluate next cycle, and if\nthe service still needs scaling the engine will fire again. Retrying inside the executor would\nrisk double-scaling during a transient blip. Let the loop's natural cadence be the retry.",[53,7532],{},[42,7534,7536],{"id":7535},"part-4-the-watch-loop","Part 4: The Watch Loop",[34,7538,7539,7540,7542],{},"The engine is pure, the executor is stateless, so ",[37,7541,1055],{}," mutable state lives in the watch loop.\nIt owns the cooldown timestamps, the duration timers, the per-trigger cooldowns, and the\nrecord of what each service's target replica count should be.",[260,7544,7546],{"className":2436,"code":7545,"language":2438,"meta":266,"style":266},"last_scale_time:   Dict[str, float]            = {}\nlast_scale_action: Dict[str, ScalingAction]    = {}\nlast_trigger_time: Dict[str, Dict[str, float]] = {}   # svc -> {trigger -> ts}\nscale_up_start:    Dict[str, Optional[float]]  = {}    # duration timer\nscale_down_start:  Dict[str, Optional[float]]  = {}    # duration timer\ntarget_replicas:   Dict[str, int]              = audit.get_last_targets()   # hydrated from DB\n",[98,7547,7548,7574,7598,7634,7666,7696],{"__ignoreMap":266},[407,7549,7550,7553,7555,7558,7560,7562,7564,7566,7568,7571],{"class":409,"line":11},[407,7551,7552],{"class":1102},"last_scale_time",[407,7554,433],{"class":416},[407,7556,7557],{"class":1102},"   Dict",[407,7559,2466],{"class":416},[407,7561,2483],{"class":412},[407,7563,750],{"class":416},[407,7565,3644],{"class":412},[407,7567,3708],{"class":416},[407,7569,7570],{"class":416},"            =",[407,7572,7573],{"class":416}," {}\n",[407,7575,7576,7579,7581,7583,7585,7587,7589,7591,7593,7596],{"class":409,"line":20},[407,7577,7578],{"class":1102},"last_scale_action",[407,7580,433],{"class":416},[407,7582,2478],{"class":1102},[407,7584,2466],{"class":416},[407,7586,2483],{"class":412},[407,7588,750],{"class":416},[407,7590,4582],{"class":1102},[407,7592,3708],{"class":416},[407,7594,7595],{"class":416},"    =",[407,7597,7573],{"class":416},[407,7599,7600,7603,7605,7607,7609,7611,7613,7615,7617,7619,7621,7623,7626,7628,7631],{"class":409,"line":427},[407,7601,7602],{"class":1102},"last_trigger_time",[407,7604,433],{"class":416},[407,7606,2478],{"class":1102},[407,7608,2466],{"class":416},[407,7610,2483],{"class":412},[407,7612,750],{"class":416},[407,7614,2478],{"class":1102},[407,7616,2466],{"class":416},[407,7618,2483],{"class":412},[407,7620,750],{"class":416},[407,7622,3644],{"class":412},[407,7624,7625],{"class":416},"]]",[407,7627,3100],{"class":416},[407,7629,7630],{"class":416}," {}",[407,7632,7633],{"class":736},"   # svc -> {trigger -> ts}\n",[407,7635,7636,7639,7641,7644,7646,7648,7650,7652,7654,7656,7658,7661,7663],{"class":409,"line":440},[407,7637,7638],{"class":1102},"scale_up_start",[407,7640,433],{"class":416},[407,7642,7643],{"class":1102},"    Dict",[407,7645,2466],{"class":416},[407,7647,2483],{"class":412},[407,7649,750],{"class":416},[407,7651,3700],{"class":1102},[407,7653,2466],{"class":416},[407,7655,3705],{"class":412},[407,7657,7625],{"class":416},[407,7659,7660],{"class":416},"  =",[407,7662,7630],{"class":416},[407,7664,7665],{"class":736},"    # duration timer\n",[407,7667,7668,7671,7673,7676,7678,7680,7682,7684,7686,7688,7690,7692,7694],{"class":409,"line":457},[407,7669,7670],{"class":1102},"scale_down_start",[407,7672,433],{"class":416},[407,7674,7675],{"class":1102},"  Dict",[407,7677,2466],{"class":416},[407,7679,2483],{"class":412},[407,7681,750],{"class":416},[407,7683,3700],{"class":1102},[407,7685,2466],{"class":416},[407,7687,3705],{"class":412},[407,7689,7625],{"class":416},[407,7691,7660],{"class":416},[407,7693,7630],{"class":416},[407,7695,7665],{"class":736},[407,7697,7698,7701,7703,7705,7707,7709,7711,7713,7715,7718,7721,7723,7726,7728],{"class":409,"line":465},[407,7699,7700],{"class":1102},"target_replicas",[407,7702,433],{"class":416},[407,7704,7557],{"class":1102},[407,7706,2466],{"class":416},[407,7708,2483],{"class":412},[407,7710,750],{"class":416},[407,7712,3625],{"class":412},[407,7714,3708],{"class":416},[407,7716,7717],{"class":416},"              =",[407,7719,7720],{"class":1102}," audit",[407,7722,167],{"class":416},[407,7724,7725],{"class":1102},"get_last_targets",[407,7727,2625],{"class":416},[407,7729,7730],{"class":736},"   # hydrated from DB\n",[34,7732,7733,7734,7737],{},"That last line matters: the loop ",[37,7735,7736],{},"hydrates its target state from the audit log on startup",", so\na restart doesn't forget what each service's intended replica count was. Cooldown and duration\ntimers do reset on restart (the first cycle after a restart is treated as a warm-up), but the\nintended targets survive.",[241,7739,7741],{"id":7740},"the-cycle-in-three-phases","The cycle, in three phases",[260,7743,7745],{"className":2436,"code":7744,"language":2438,"meta":266,"style":266},"# Phase 1: evaluate every service (no blocking I\u002FO)\nfor sm, labels in all_service_data:\n    policy = policy_svc.resolve(sm.info, labels)\n    if policy is None:\n        continue\n    ... compute time_since, elapsed_up, elapsed_down, suppressed triggers ...\n    decision, host_signal = policy_svc.evaluate(sm, policy, time_since, last_action,\n                                                elapsed_up, elapsed_down, suppressed)\n    evals.append(_Eval(sm, labels, policy, decision, host_signal))\n\n# Phase 2: execute all non-NONE decisions in PARALLEL\nactive = [e for e in evals if e.decision.action != NONE]\nfutures = {_exec_pool.submit(_exec_one, ev): ev for ev in active}\n# total blocking time is max(individual poll_timeout), not the sum\n\n# Phase 3: process results, update state, write audit, update the live table\n",[98,7746,7747,7752,7770,7800,7812,7817,7843,7882,7898,7935,7939,7944,7987,8029,8034,8038],{"__ignoreMap":266},[407,7748,7749],{"class":409,"line":11},[407,7750,7751],{"class":736},"# Phase 1: evaluate every service (no blocking I\u002FO)\n",[407,7753,7754,7756,7759,7761,7763,7765,7768],{"class":409,"line":20},[407,7755,1833],{"class":557},[407,7757,7758],{"class":1102}," sm",[407,7760,750],{"class":416},[407,7762,2776],{"class":1102},[407,7764,2779],{"class":557},[407,7766,7767],{"class":1102}," all_service_data",[407,7769,417],{"class":416},[407,7771,7772,7774,7776,7779,7781,7784,7786,7789,7791,7794,7796,7798],{"class":409,"line":427},[407,7773,4224],{"class":1102},[407,7775,2499],{"class":416},[407,7777,7778],{"class":1102}," policy_svc",[407,7780,167],{"class":416},[407,7782,7783],{"class":1102},"resolve",[407,7785,2451],{"class":416},[407,7787,7788],{"class":1102},"sm",[407,7790,167],{"class":416},[407,7792,7793],{"class":1102},"info",[407,7795,750],{"class":416},[407,7797,2761],{"class":1102},[407,7799,2764],{"class":416},[407,7801,7802,7804,7806,7808,7810],{"class":409,"line":440},[407,7803,2552],{"class":557},[407,7805,4246],{"class":1102},[407,7807,2238],{"class":1095},[407,7809,3346],{"class":557},[407,7811,417],{"class":416},[407,7813,7814],{"class":409,"line":457},[407,7815,7816],{"class":557},"        continue\n",[407,7818,7819,7822,7825,7827,7830,7832,7835,7837,7840],{"class":409,"line":465},[407,7820,7821],{"class":1088},"    ...",[407,7823,7824],{"class":1102}," compute time_since",[407,7826,750],{"class":416},[407,7828,7829],{"class":1102}," elapsed_up",[407,7831,750],{"class":416},[407,7833,7834],{"class":1102}," elapsed_down",[407,7836,750],{"class":416},[407,7838,7839],{"class":1102}," suppressed triggers ",[407,7841,7842],{"class":1088},"...\n",[407,7844,7845,7848,7850,7853,7855,7857,7859,7862,7864,7866,7868,7870,7872,7875,7877,7880],{"class":409,"line":477},[407,7846,7847],{"class":1102},"    decision",[407,7849,750],{"class":416},[407,7851,7852],{"class":1102}," host_signal ",[407,7854,2499],{"class":416},[407,7856,7778],{"class":1102},[407,7858,167],{"class":416},[407,7860,7861],{"class":1102},"evaluate",[407,7863,2451],{"class":416},[407,7865,7788],{"class":1102},[407,7867,750],{"class":416},[407,7869,4326],{"class":1102},[407,7871,750],{"class":416},[407,7873,7874],{"class":1102}," time_since",[407,7876,750],{"class":416},[407,7878,7879],{"class":1102}," last_action",[407,7881,3438],{"class":416},[407,7883,7884,7887,7889,7891,7893,7896],{"class":409,"line":485},[407,7885,7886],{"class":1102},"                                                elapsed_up",[407,7888,750],{"class":416},[407,7890,7834],{"class":1102},[407,7892,750],{"class":416},[407,7894,7895],{"class":1102}," suppressed",[407,7897,2764],{"class":416},[407,7899,7900,7903,7905,7907,7909,7912,7914,7916,7918,7920,7922,7924,7926,7928,7930,7933],{"class":409,"line":494},[407,7901,7902],{"class":1102},"    evals",[407,7904,167],{"class":416},[407,7906,5273],{"class":1102},[407,7908,2451],{"class":416},[407,7910,7911],{"class":1102},"_Eval",[407,7913,2451],{"class":416},[407,7915,7788],{"class":1102},[407,7917,750],{"class":416},[407,7919,2761],{"class":1102},[407,7921,750],{"class":416},[407,7923,4326],{"class":1102},[407,7925,750],{"class":416},[407,7927,6857],{"class":1102},[407,7929,750],{"class":416},[407,7931,7932],{"class":1102}," host_signal",[407,7934,3349],{"class":416},[407,7936,7937],{"class":409,"line":502},[407,7938,530],{"emptyLinePlaceholder":529},[407,7940,7941],{"class":409,"line":510},[407,7942,7943],{"class":736},"# Phase 2: execute all non-NONE decisions in PARALLEL\n",[407,7945,7946,7949,7951,7953,7956,7958,7961,7963,7966,7968,7971,7973,7975,7977,7979,7982,7985],{"class":409,"line":518},[407,7947,7948],{"class":1102},"active ",[407,7950,2499],{"class":416},[407,7952,448],{"class":416},[407,7954,7955],{"class":1102},"e ",[407,7957,1833],{"class":557},[407,7959,7960],{"class":1102}," e ",[407,7962,2779],{"class":557},[407,7964,7965],{"class":1102}," evals ",[407,7967,3354],{"class":557},[407,7969,7970],{"class":1102}," e",[407,7972,167],{"class":416},[407,7974,6904],{"class":1102},[407,7976,167],{"class":416},[407,7978,6862],{"class":1102},[407,7980,7981],{"class":1095},"!=",[407,7983,7984],{"class":1088}," NONE",[407,7986,454],{"class":416},[407,7988,7989,7992,7994,7996,7999,8001,8003,8005,8008,8010,8013,8015,8018,8020,8022,8024,8026],{"class":409,"line":526},[407,7990,7991],{"class":1102},"futures ",[407,7993,2499],{"class":416},[407,7995,819],{"class":416},[407,7997,7998],{"class":1102},"_exec_pool",[407,8000,167],{"class":416},[407,8002,2720],{"class":1102},[407,8004,2451],{"class":416},[407,8006,8007],{"class":1102},"_exec_one",[407,8009,750],{"class":416},[407,8011,8012],{"class":1102}," ev",[407,8014,2751],{"class":416},[407,8016,8017],{"class":1102}," ev ",[407,8019,1833],{"class":557},[407,8021,8017],{"class":1102},[407,8023,2779],{"class":557},[407,8025,1169],{"class":1102},[407,8027,8028],{"class":416},"}\n",[407,8030,8031],{"class":409,"line":533},[407,8032,8033],{"class":736},"# total blocking time is max(individual poll_timeout), not the sum\n",[407,8035,8036],{"class":409,"line":541},[407,8037,530],{"emptyLinePlaceholder":529},[407,8039,8040],{"class":409,"line":549},[407,8041,8042],{"class":736},"# Phase 3: process results, update state, write audit, update the live table\n",[34,8044,8045],{},"Separating evaluation (pure, instant) from execution (slow, I\u002FO-bound) lets the loop fire all\nscale operations concurrently. Ten services that each need scaling cost one poll-timeout, not ten.",[241,8047,8049],{"id":8048},"managing-the-duration-timers","Managing the duration timers",[34,8051,8052,8053,8056],{},"The engine is told ",[154,8054,8055],{},"how long"," conditions have held, but it's the loop that measures it. Each\ncycle it asks the policy's own condition helpers whether the up and down conditions are currently met,\nand starts or resets a timer accordingly:",[260,8058,8060],{"className":2436,"code":8059,"language":2438,"meta":266,"style":266},"if policy_svc.up_conditions_met(sm, policy):\n    if scale_up_start.get(svc) is None:\n        scale_up_start[svc] = now\n    elapsed_up = int(now - scale_up_start[svc])\nelse:\n    scale_up_start[svc] = None\n    elapsed_up = 0\n",[98,8061,8062,8083,8107,8123,8147,8153,8168],{"__ignoreMap":266},[407,8063,8064,8066,8068,8070,8073,8075,8077,8079,8081],{"class":409,"line":11},[407,8065,3354],{"class":557},[407,8067,7778],{"class":1102},[407,8069,167],{"class":416},[407,8071,8072],{"class":1102},"up_conditions_met",[407,8074,2451],{"class":416},[407,8076,7788],{"class":1102},[407,8078,750],{"class":416},[407,8080,4326],{"class":1102},[407,8082,3568],{"class":416},[407,8084,8085,8087,8090,8092,8094,8096,8098,8100,8103,8105],{"class":409,"line":20},[407,8086,2552],{"class":557},[407,8088,8089],{"class":1102}," scale_up_start",[407,8091,167],{"class":416},[407,8093,3135],{"class":1102},[407,8095,2451],{"class":416},[407,8097,2756],{"class":1102},[407,8099,2457],{"class":416},[407,8101,8102],{"class":1095}," is",[407,8104,3346],{"class":557},[407,8106,417],{"class":416},[407,8108,8109,8112,8114,8116,8118,8120],{"class":409,"line":427},[407,8110,8111],{"class":1102},"        scale_up_start",[407,8113,2466],{"class":416},[407,8115,2756],{"class":1102},[407,8117,3708],{"class":416},[407,8119,3100],{"class":416},[407,8121,8122],{"class":1102}," now\n",[407,8124,8125,8128,8130,8132,8134,8137,8139,8141,8143,8145],{"class":409,"line":440},[407,8126,8127],{"class":1102},"    elapsed_up ",[407,8129,2499],{"class":416},[407,8131,3625],{"class":412},[407,8133,2451],{"class":416},[407,8135,8136],{"class":1102},"now ",[407,8138,2810],{"class":1095},[407,8140,8089],{"class":1102},[407,8142,2466],{"class":416},[407,8144,2756],{"class":1102},[407,8146,4054],{"class":416},[407,8148,8149,8151],{"class":409,"line":457},[407,8150,3236],{"class":557},[407,8152,417],{"class":416},[407,8154,8155,8158,8160,8162,8164,8166],{"class":409,"line":465},[407,8156,8157],{"class":1102},"    scale_up_start",[407,8159,2466],{"class":416},[407,8161,2756],{"class":1102},[407,8163,3708],{"class":416},[407,8165,3100],{"class":416},[407,8167,3239],{"class":557},[407,8169,8170,8172,8174],{"class":409,"line":477},[407,8171,8127],{"class":1102},[407,8173,2499],{"class":416},[407,8175,3630],{"class":473},[34,8177,8178,8179,293,8181,8184,8185,8188],{},"Those ",[98,8180,8072],{},[98,8182,8183],{},"down_conditions_met"," helpers live on the ",[98,8186,8187],{},"PolicyService"," as the\nsingle source of truth, so the loop can't drift out of sync with the engine's own thresholds.",[241,8190,8192],{"id":8191},"per-trigger-cooldown","Per-trigger cooldown",[34,8194,8195,8196,8198],{},"The ordinary cooldown blocks ",[154,8197,1055],{}," scaling for a period. But imagine a service whose CPU is\nchronically a hair over threshold: it would scale +1, cool down, scale +1 again, forever, from\nthe same unresolved cause. Per-trigger cooldown suppresses an individual trigger for a while\nafter it caused a scale, while leaving the other triggers free to fire:",[260,8200,8202],{"className":2436,"code":8201,"language":2438,"meta":266,"style":266},"suppressed = frozenset(\n    t for t, ts in last_trigger_time.get(svc, {}).items()\n    if now - ts \u003C policy.trigger_cooldown_seconds\n)\n",[98,8203,8204,8215,8251,8271],{"__ignoreMap":266},[407,8205,8206,8209,8211,8213],{"class":409,"line":11},[407,8207,8208],{"class":1102},"suppressed ",[407,8210,2499],{"class":416},[407,8212,4625],{"class":412},[407,8214,4530],{"class":416},[407,8216,8217,8220,8222,8225,8227,8230,8232,8235,8237,8239,8241,8243,8245,8247,8249],{"class":409,"line":20},[407,8218,8219],{"class":1102},"    t ",[407,8221,1833],{"class":557},[407,8223,8224],{"class":1102}," t",[407,8226,750],{"class":416},[407,8228,8229],{"class":1102}," ts ",[407,8231,2779],{"class":557},[407,8233,8234],{"class":1102}," last_trigger_time",[407,8236,167],{"class":416},[407,8238,3135],{"class":1102},[407,8240,2451],{"class":416},[407,8242,2756],{"class":1102},[407,8244,750],{"class":416},[407,8246,3149],{"class":416},[407,8248,4090],{"class":1102},[407,8250,2505],{"class":416},[407,8252,8253,8255,8258,8260,8262,8264,8266,8268],{"class":409,"line":427},[407,8254,2552],{"class":557},[407,8256,8257],{"class":1102}," now ",[407,8259,2810],{"class":1095},[407,8261,8229],{"class":1102},[407,8263,3199],{"class":1095},[407,8265,4326],{"class":1102},[407,8267,167],{"class":416},[407,8269,8270],{"class":1102},"trigger_cooldown_seconds\n",[407,8272,8273],{"class":409,"line":440},[407,8274,2764],{"class":416},[34,8276,8277,8278,8281],{},"Genuine emergencies (urgency EMERGENCY) are never suppressed: a real CPU fire always gets\nthrough. ",[98,8279,8280],{},"pending_stuck"," is never suppressed either, since it drives capacity signaling.",[241,8283,8285],{"id":8284},"reconciliation-and-manual-changes","Reconciliation and manual changes",[34,8287,8288,8289,8291],{},"The loop doesn't assume it's the only actor. An operator might run ",[98,8290,1268],{},"\nby hand. On each cycle the loop compares Swarm's actual desired count to its own remembered\ntarget and reacts intelligently:",[147,8293,8294,8304],{},[150,8295,8296,8303],{},[37,8297,8298,8299,8302],{},"Swarm has ",[154,8300,8301],{},"more"," than expected"," → an intentional manual scale-up. Accept it silently and\nadopt the new number as the target. Don't fight the operator.",[150,8305,8306,8311,8312,8314],{},[37,8307,8298,8308,8302],{},[154,8309,8310],{},"fewer"," → could be a crash, or a manual scale-down. Run the policy\nfirst. If load actually justifies the old higher count (or the service is below\n",[98,8313,1403],{},"), reconcile back up. Otherwise accept the operator's lower number.",[260,8316,8318],{"className":2436,"code":8317,"language":2438,"meta":266,"style":266},"if current_swarm_desired > known_target:\n    target_replicas[svc] = current_swarm_desired          # accept manual scale-up\nelse:\n    probe_decision, _ = policy_svc.evaluate(... )\n    if probe_decision.action == SCALE_UP or below_min:\n        reconcile_decision = ScalingDecision(action=SCALE_UP, by=diff, urgency=EMERGENCY, ...)\n    else:\n        target_replicas[svc] = current_swarm_desired       # accept manual scale-down\n",[98,8319,8320,8334,8353,8359,8383,8405,8449,8455],{"__ignoreMap":266},[407,8321,8322,8324,8327,8329,8332],{"class":409,"line":11},[407,8323,3354],{"class":557},[407,8325,8326],{"class":1102}," current_swarm_desired ",[407,8328,1848],{"class":1095},[407,8330,8331],{"class":1102}," known_target",[407,8333,417],{"class":416},[407,8335,8336,8339,8341,8343,8345,8347,8350],{"class":409,"line":20},[407,8337,8338],{"class":1102},"    target_replicas",[407,8340,2466],{"class":416},[407,8342,2756],{"class":1102},[407,8344,3708],{"class":416},[407,8346,3100],{"class":416},[407,8348,8349],{"class":1102}," current_swarm_desired          ",[407,8351,8352],{"class":736},"# accept manual scale-up\n",[407,8354,8355,8357],{"class":409,"line":427},[407,8356,3236],{"class":557},[407,8358,417],{"class":416},[407,8360,8361,8364,8366,8368,8370,8372,8374,8376,8378,8380],{"class":409,"line":440},[407,8362,8363],{"class":1102},"    probe_decision",[407,8365,750],{"class":416},[407,8367,7446],{"class":1102},[407,8369,2499],{"class":416},[407,8371,7778],{"class":1102},[407,8373,167],{"class":416},[407,8375,7861],{"class":1102},[407,8377,2451],{"class":416},[407,8379,4994],{"class":1088},[407,8381,8382],{"class":416}," )\n",[407,8384,8385,8387,8390,8392,8394,8396,8398,8400,8403],{"class":409,"line":457},[407,8386,2552],{"class":557},[407,8388,8389],{"class":1102}," probe_decision",[407,8391,167],{"class":416},[407,8393,6862],{"class":1102},[407,8395,3365],{"class":1095},[407,8397,5072],{"class":1088},[407,8399,4459],{"class":1095},[407,8401,8402],{"class":1102}," below_min",[407,8404,417],{"class":416},[407,8406,8407,8410,8412,8414,8416,8419,8421,8424,8426,8429,8431,8434,8436,8439,8441,8443,8445,8447],{"class":409,"line":465},[407,8408,8409],{"class":1102},"        reconcile_decision ",[407,8411,2499],{"class":416},[407,8413,4989],{"class":1102},[407,8415,2451],{"class":416},[407,8417,8418],{"class":1836},"action",[407,8420,2499],{"class":416},[407,8422,8423],{"class":1088},"SCALE_UP",[407,8425,750],{"class":416},[407,8427,8428],{"class":1836}," by",[407,8430,2499],{"class":416},[407,8432,8433],{"class":1102},"diff",[407,8435,750],{"class":416},[407,8437,8438],{"class":1836}," urgency",[407,8440,2499],{"class":416},[407,8442,5608],{"class":1088},[407,8444,750],{"class":416},[407,8446,3489],{"class":1088},[407,8448,2764],{"class":416},[407,8450,8451,8453],{"class":409,"line":477},[407,8452,5284],{"class":557},[407,8454,417],{"class":416},[407,8456,8457,8460,8462,8464,8466,8468,8471],{"class":409,"line":485},[407,8458,8459],{"class":1102},"        target_replicas",[407,8461,2466],{"class":416},[407,8463,2756],{"class":1102},[407,8465,3708],{"class":416},[407,8467,3100],{"class":416},[407,8469,8470],{"class":1102}," current_swarm_desired       ",[407,8472,8473],{"class":736},"# accept manual scale-down\n",[241,8475,8477],{"id":8476},"ghost-task-recovery","Ghost-task recovery",[34,8479,8480,8481,8484,8485,8488],{},"Sometimes a service ends up with ",[98,8482,8483],{},"desired > running",", a task stuck pending, and ",[37,8486,8487],{},"no"," real load\nto justify it: the classic \"ghost update\" where a new task spec can never schedule (bad\nconstraint, image that vanished). Scaling up won't help; the right move is to clear the\nunreachable pending task. The loop waits a configurable number of cycles (so slow image pulls\nand genuine rolling updates resolve themselves first), then force-scales down to clear the ghost:",[260,8490,8492],{"className":2436,"code":8491,"language":2438,"meta":266,"style":266},"if sm.replicas_desired > sm.replicas_running and sm.replicas_pending > 0 and load_is_idle:\n    stuck_desired_cycles[svc] += 1\n    if stuck_desired_cycles[svc] >= _s.ghost_recovery_min_cycles:\n        recovery = ScalingDecision(action=SCALE_DOWN,\n                                   by=sm.replicas_desired - sm.replicas_running,\n                                   force=True, ...)   # force bypasses the active-deployment guard\n        executor.execute(recovery, sm, ev.policy)\n",[98,8493,8494,8531,8547,8573,8592,8616,8634],{"__ignoreMap":266},[407,8495,8496,8498,8500,8502,8504,8506,8508,8510,8512,8514,8516,8518,8520,8522,8524,8526,8529],{"class":409,"line":11},[407,8497,3354],{"class":557},[407,8499,7758],{"class":1102},[407,8501,167],{"class":416},[407,8503,4758],{"class":1102},[407,8505,1848],{"class":1095},[407,8507,7758],{"class":1102},[407,8509,167],{"class":416},[407,8511,4744],{"class":1102},[407,8513,3360],{"class":1095},[407,8515,7758],{"class":1102},[407,8517,167],{"class":416},[407,8519,6114],{"class":1102},[407,8521,1848],{"class":1095},[407,8523,3180],{"class":473},[407,8525,4751],{"class":1095},[407,8527,8528],{"class":1102}," load_is_idle",[407,8530,417],{"class":416},[407,8532,8533,8536,8538,8540,8542,8545],{"class":409,"line":20},[407,8534,8535],{"class":1102},"    stuck_desired_cycles",[407,8537,2466],{"class":416},[407,8539,2756],{"class":1102},[407,8541,3708],{"class":416},[407,8543,8544],{"class":416}," +=",[407,8546,1696],{"class":473},[407,8548,8549,8551,8554,8556,8558,8560,8563,8566,8568,8571],{"class":409,"line":427},[407,8550,2552],{"class":557},[407,8552,8553],{"class":1102}," stuck_desired_cycles",[407,8555,2466],{"class":416},[407,8557,2756],{"class":1102},[407,8559,3708],{"class":416},[407,8561,8562],{"class":1095}," >=",[407,8564,8565],{"class":1102}," _s",[407,8567,167],{"class":416},[407,8569,8570],{"class":1102},"ghost_recovery_min_cycles",[407,8572,417],{"class":416},[407,8574,8575,8578,8580,8582,8584,8586,8588,8590],{"class":409,"line":440},[407,8576,8577],{"class":1102},"        recovery ",[407,8579,2499],{"class":416},[407,8581,4989],{"class":1102},[407,8583,2451],{"class":416},[407,8585,8418],{"class":1836},[407,8587,2499],{"class":416},[407,8589,6785],{"class":1088},[407,8591,3438],{"class":416},[407,8593,8594,8597,8599,8601,8603,8605,8607,8609,8611,8614],{"class":409,"line":457},[407,8595,8596],{"class":1836},"                                   by",[407,8598,2499],{"class":416},[407,8600,7788],{"class":1102},[407,8602,167],{"class":416},[407,8604,4758],{"class":1102},[407,8606,2810],{"class":1095},[407,8608,7758],{"class":1102},[407,8610,167],{"class":416},[407,8612,8613],{"class":1102},"replicas_running",[407,8615,3438],{"class":416},[407,8617,8618,8621,8623,8625,8627,8629,8631],{"class":409,"line":465},[407,8619,8620],{"class":1836},"                                   force",[407,8622,2499],{"class":416},[407,8624,6805],{"class":557},[407,8626,750],{"class":416},[407,8628,3489],{"class":1088},[407,8630,2457],{"class":416},[407,8632,8633],{"class":736},"   # force bypasses the active-deployment guard\n",[407,8635,8636,8639,8641,8644,8646,8649,8651,8653,8655,8657,8659,8661],{"class":409,"line":477},[407,8637,8638],{"class":1102},"        executor",[407,8640,167],{"class":416},[407,8642,8643],{"class":1102},"execute",[407,8645,2451],{"class":416},[407,8647,8648],{"class":1102},"recovery",[407,8650,750],{"class":416},[407,8652,7758],{"class":1102},[407,8654,750],{"class":416},[407,8656,8012],{"class":1102},[407,8658,167],{"class":416},[407,8660,4260],{"class":1102},[407,8662,2764],{"class":416},[241,8664,8666],{"id":8665},"writing-the-audit-trail","Writing the audit trail",[34,8668,8669,8670,8673],{},"Every evaluation is written to SQLite, ",[37,8671,8672],{},"including NONE decisions",". A NONE with a nonzero\nscore (\"almost fired\") is exactly the data you want when tuning thresholds later. The table is\nappend-only and indexed by service, timestamp, and action.",[241,8675,8677],{"id":8676},"ephemeral-node-cleanup","Ephemeral node cleanup",[34,8679,8680,8681,8683,8684,8687],{},"As a housekeeping task, the loop watches nodes labeled ",[98,8682,941],{}," (VMs added by the\nASG). If one stays ",[98,8685,8686],{},"down"," for enough consecutive cycles (~24h worth), it removes it from the\nswarm so the member list doesn't fill with dead OpenStack instances.",[53,8689],{},[42,8691,8693],{"id":8692},"part-5-the-openstack-integration","Part 5: The OpenStack Integration",[34,8695,8696,8697,8699,8700,8703,8704,8706],{},"This is what turns a ",[154,8698,3926],{}," autoscaler into a ",[154,8701,8702],{},"cluster"," autoscaler. ",[98,8705,226],{}," scales replicas;\nOpenStack's Heat autoscaling group (ASG) scales VMs. They talk through three channels.",[241,8708,8710],{"id":8709},"scale-out-swarmctl-asks-openstack-for-a-vm","Scale-out: swarmctl asks OpenStack for a VM",[34,8712,8713,8714,8716,8717,8720],{},"When the engine raises ",[98,8715,6801],{}," because the cluster is out of room, the watch loop\ncounts how many consecutive cycles the demand has persisted, ",[37,8718,8719],{},"per pool",", and only then fires an\nHTTP webhook at Heat:",[260,8722,8724],{"className":2436,"code":8723,"language":2438,"meta":266,"style":266},"for ev in evals:\n    pool = ev.labels.get(\"swarmctl.pool\") or _s.DEFAULT_POOL\n    key  = (pool, ev.sm.service)\n    is_pending_stuck = ev.sm.longest_pending_seconds > ev.policy.pending_stuck_seconds\n    if is_pending_stuck and ev.decision.action == ScalingAction.SCALE_UP:\n        svc_stuck_cycles[key] = svc_stuck_cycles.get(key, 0) + 1\n    else:\n        svc_stuck_cycles.pop(key, None)\n\n# aggregate the worst stuck-count per pool, then signal each pool over the threshold\nfor pool, max_stuck in pool_max_stuck.items():\n    if max_stuck >= _s.asg_pending_cycles:\n        asg.trigger_scale_out(pool)\n",[98,8725,8726,8739,8776,8802,8832,8861,8896,8902,8921,8925,8930,8953,8970],{"__ignoreMap":266},[407,8727,8728,8730,8732,8734,8737],{"class":409,"line":11},[407,8729,1833],{"class":557},[407,8731,8017],{"class":1102},[407,8733,2779],{"class":557},[407,8735,8736],{"class":1102}," evals",[407,8738,417],{"class":416},[407,8740,8741,8744,8746,8748,8750,8752,8754,8756,8758,8760,8763,8765,8767,8769,8771,8773],{"class":409,"line":20},[407,8742,8743],{"class":1102},"    pool ",[407,8745,2499],{"class":416},[407,8747,8012],{"class":1102},[407,8749,167],{"class":416},[407,8751,397],{"class":1102},[407,8753,167],{"class":416},[407,8755,3135],{"class":1102},[407,8757,2451],{"class":416},[407,8759,2000],{"class":1877},[407,8761,8762],{"class":436},"swarmctl.pool",[407,8764,2000],{"class":1877},[407,8766,2457],{"class":416},[407,8768,4459],{"class":1095},[407,8770,8565],{"class":1102},[407,8772,167],{"class":416},[407,8774,8775],{"class":1088},"DEFAULT_POOL\n",[407,8777,8778,8781,8783,8785,8788,8790,8792,8794,8796,8798,8800],{"class":409,"line":427},[407,8779,8780],{"class":1102},"    key  ",[407,8782,2499],{"class":416},[407,8784,1277],{"class":416},[407,8786,8787],{"class":1102},"pool",[407,8789,750],{"class":416},[407,8791,8012],{"class":1102},[407,8793,167],{"class":416},[407,8795,7788],{"class":1102},[407,8797,167],{"class":416},[407,8799,3926],{"class":1102},[407,8801,2764],{"class":416},[407,8803,8804,8807,8809,8811,8813,8815,8817,8819,8821,8823,8825,8827,8829],{"class":409,"line":440},[407,8805,8806],{"class":1102},"    is_pending_stuck ",[407,8808,2499],{"class":416},[407,8810,8012],{"class":1102},[407,8812,167],{"class":416},[407,8814,7788],{"class":1102},[407,8816,167],{"class":416},[407,8818,5453],{"class":1102},[407,8820,1848],{"class":1095},[407,8822,8012],{"class":1102},[407,8824,167],{"class":416},[407,8826,4260],{"class":1102},[407,8828,167],{"class":416},[407,8830,8831],{"class":1102},"pending_stuck_seconds\n",[407,8833,8834,8836,8839,8841,8843,8845,8847,8849,8851,8853,8855,8857,8859],{"class":409,"line":457},[407,8835,2552],{"class":557},[407,8837,8838],{"class":1102}," is_pending_stuck ",[407,8840,3360],{"class":1095},[407,8842,8012],{"class":1102},[407,8844,167],{"class":416},[407,8846,6904],{"class":1102},[407,8848,167],{"class":416},[407,8850,6862],{"class":1102},[407,8852,3365],{"class":1095},[407,8854,4582],{"class":1102},[407,8856,167],{"class":416},[407,8858,8423],{"class":1088},[407,8860,417],{"class":416},[407,8862,8863,8866,8868,8871,8873,8875,8878,8880,8882,8884,8886,8888,8890,8892,8894],{"class":409,"line":465},[407,8864,8865],{"class":1102},"        svc_stuck_cycles",[407,8867,2466],{"class":416},[407,8869,8870],{"class":1102},"key",[407,8872,3708],{"class":416},[407,8874,3100],{"class":416},[407,8876,8877],{"class":1102}," svc_stuck_cycles",[407,8879,167],{"class":416},[407,8881,3135],{"class":1102},[407,8883,2451],{"class":416},[407,8885,8870],{"class":1102},[407,8887,750],{"class":416},[407,8889,3180],{"class":473},[407,8891,2457],{"class":416},[407,8893,2628],{"class":1095},[407,8895,1696],{"class":473},[407,8897,8898,8900],{"class":409,"line":477},[407,8899,5284],{"class":557},[407,8901,417],{"class":416},[407,8903,8904,8906,8908,8911,8913,8915,8917,8919],{"class":409,"line":485},[407,8905,8865],{"class":1102},[407,8907,167],{"class":416},[407,8909,8910],{"class":1102},"pop",[407,8912,2451],{"class":416},[407,8914,8870],{"class":1102},[407,8916,750],{"class":416},[407,8918,3346],{"class":557},[407,8920,2764],{"class":416},[407,8922,8923],{"class":409,"line":494},[407,8924,530],{"emptyLinePlaceholder":529},[407,8926,8927],{"class":409,"line":502},[407,8928,8929],{"class":736},"# aggregate the worst stuck-count per pool, then signal each pool over the threshold\n",[407,8931,8932,8934,8936,8938,8941,8943,8946,8948,8950],{"class":409,"line":510},[407,8933,1833],{"class":557},[407,8935,1148],{"class":1102},[407,8937,750],{"class":416},[407,8939,8940],{"class":1102}," max_stuck ",[407,8942,2779],{"class":557},[407,8944,8945],{"class":1102}," pool_max_stuck",[407,8947,167],{"class":416},[407,8949,4090],{"class":1102},[407,8951,8952],{"class":416},"():\n",[407,8954,8955,8957,8959,8961,8963,8965,8968],{"class":409,"line":518},[407,8956,2552],{"class":557},[407,8958,8940],{"class":1102},[407,8960,4397],{"class":1095},[407,8962,8565],{"class":1102},[407,8964,167],{"class":416},[407,8966,8967],{"class":1102},"asg_pending_cycles",[407,8969,417],{"class":416},[407,8971,8972,8975,8977,8980,8982,8984],{"class":409,"line":526},[407,8973,8974],{"class":1102},"        asg",[407,8976,167],{"class":416},[407,8978,8979],{"class":1102},"trigger_scale_out",[407,8981,2451],{"class":416},[407,8983,8787],{"class":1102},[407,8985,2764],{"class":416},[34,8987,7505,8988,8990,8991,374,8994,8997,8998,9001,9002,167],{},[37,8989,8787],{}," concept matters. A cluster can have several ASG pools (say ",[98,8992,8993],{},"default",[98,8995,8996],{},"gpu","). A\nservice is routed to a pool by its Swarm placement constraint\n(",[98,8999,9000],{},"node.labels.pool == gpu","), the same value Swarm itself uses to schedule it, so routing can't\ndrift from reality. A stuck GPU service must grow the GPU pool, not some unrelated one. Scale-out\ndemand is therefore tracked and aggregated ",[37,9003,8719],{},[34,9005,9006,9009],{},[98,9007,9008],{},"AsgService.trigger_scale_out"," enforces a per-pool cooldown (one new VM takes time to boot and\njoin before its effect shows up) and records the scale-out timestamp, which, as you will see, the\nscale-in path reads to avoid flapping:",[260,9011,9013],{"className":2436,"code":9012,"language":2438,"meta":266,"style":266},"def trigger_scale_out(self, pool=None) -> bool:\n    cfg = s.get_pool(pool)\n    if cfg is None or not cfg.scale_out_url:\n        return False\n    if (now - self._last_trigger.get(pool, 0.0)) \u003C cfg.cooldown:\n        return False   # a VM is already on its way\n    response = httpx.post(cfg.scale_out_url, timeout=10.0)\n    if response.status_code in (200, 201, 202, 204):\n        self._last_trigger[pool] = now\n        with _ts_lock:\n            _last_scale_out_ts[pool] = now   # cross-thread handoff to scale-in logic\n        return True\n",[98,9014,9015,9042,9062,9087,9093,9136,9145,9180,9216,9235,9245,9264],{"__ignoreMap":266},[407,9016,9017,9019,9022,9024,9026,9028,9030,9032,9034,9036,9038,9040],{"class":409,"line":11},[407,9018,2445],{"class":1095},[407,9020,9021],{"class":1078}," trigger_scale_out",[407,9023,2451],{"class":416},[407,9025,2454],{"class":1102},[407,9027,750],{"class":416},[407,9029,1148],{"class":1102},[407,9031,2499],{"class":1095},[407,9033,3079],{"class":557},[407,9035,2457],{"class":416},[407,9037,2460],{"class":416},[407,9039,4179],{"class":412},[407,9041,417],{"class":416},[407,9043,9044,9047,9049,9051,9053,9056,9058,9060],{"class":409,"line":20},[407,9045,9046],{"class":1102},"    cfg ",[407,9048,2499],{"class":416},[407,9050,2631],{"class":1102},[407,9052,167],{"class":416},[407,9054,9055],{"class":1102},"get_pool",[407,9057,2451],{"class":416},[407,9059,8787],{"class":1102},[407,9061,2764],{"class":416},[407,9063,9064,9066,9069,9071,9073,9075,9077,9080,9082,9085],{"class":409,"line":427},[407,9065,2552],{"class":557},[407,9067,9068],{"class":1102}," cfg ",[407,9070,2238],{"class":1095},[407,9072,3346],{"class":557},[407,9074,4459],{"class":1095},[407,9076,2555],{"class":1095},[407,9078,9079],{"class":1102}," cfg",[407,9081,167],{"class":416},[407,9083,9084],{"class":1102},"scale_out_url",[407,9086,417],{"class":416},[407,9088,9089,9091],{"class":409,"line":440},[407,9090,2565],{"class":557},[407,9092,4358],{"class":557},[407,9094,9095,9097,9099,9101,9103,9105,9107,9110,9112,9114,9116,9118,9120,9122,9125,9127,9129,9131,9134],{"class":409,"line":457},[407,9096,2552],{"class":557},[407,9098,1277],{"class":416},[407,9100,8136],{"class":1102},[407,9102,2810],{"class":1095},[407,9104,2536],{"class":1088},[407,9106,167],{"class":416},[407,9108,9109],{"class":1102},"_last_trigger",[407,9111,167],{"class":416},[407,9113,3135],{"class":1102},[407,9115,2451],{"class":416},[407,9117,8787],{"class":1102},[407,9119,750],{"class":416},[407,9121,6359],{"class":473},[407,9123,9124],{"class":416},"))",[407,9126,1096],{"class":1095},[407,9128,9079],{"class":1102},[407,9130,167],{"class":416},[407,9132,9133],{"class":1102},"cooldown",[407,9135,417],{"class":416},[407,9137,9138,9140,9142],{"class":409,"line":465},[407,9139,2565],{"class":557},[407,9141,4411],{"class":557},[407,9143,9144],{"class":736},"   # a VM is already on its way\n",[407,9146,9147,9150,9152,9155,9157,9160,9162,9165,9167,9169,9171,9173,9175,9178],{"class":409,"line":477},[407,9148,9149],{"class":1102},"    response ",[407,9151,2499],{"class":416},[407,9153,9154],{"class":1102}," httpx",[407,9156,167],{"class":416},[407,9158,9159],{"class":1102},"post",[407,9161,2451],{"class":416},[407,9163,9164],{"class":1102},"cfg",[407,9166,167],{"class":416},[407,9168,9084],{"class":1102},[407,9170,750],{"class":416},[407,9172,2844],{"class":1836},[407,9174,2499],{"class":416},[407,9176,9177],{"class":473},"10.0",[407,9179,2764],{"class":416},[407,9181,9182,9184,9187,9189,9192,9194,9196,9199,9201,9204,9206,9209,9211,9214],{"class":409,"line":485},[407,9183,2552],{"class":557},[407,9185,9186],{"class":1102}," response",[407,9188,167],{"class":416},[407,9190,9191],{"class":1102},"status_code ",[407,9193,2779],{"class":1095},[407,9195,1277],{"class":416},[407,9197,9198],{"class":473},"200",[407,9200,750],{"class":416},[407,9202,9203],{"class":473}," 201",[407,9205,750],{"class":416},[407,9207,9208],{"class":473}," 202",[407,9210,750],{"class":416},[407,9212,9213],{"class":473}," 204",[407,9215,3568],{"class":416},[407,9217,9218,9221,9223,9225,9227,9229,9231,9233],{"class":409,"line":494},[407,9219,9220],{"class":1088},"        self",[407,9222,167],{"class":416},[407,9224,9109],{"class":1102},[407,9226,2466],{"class":416},[407,9228,8787],{"class":1102},[407,9230,3708],{"class":416},[407,9232,3100],{"class":416},[407,9234,8122],{"class":1102},[407,9236,9237,9240,9243],{"class":409,"line":502},[407,9238,9239],{"class":557},"        with",[407,9241,9242],{"class":1102}," _ts_lock",[407,9244,417],{"class":416},[407,9246,9247,9250,9252,9254,9256,9258,9261],{"class":409,"line":510},[407,9248,9249],{"class":1102},"            _last_scale_out_ts",[407,9251,2466],{"class":416},[407,9253,8787],{"class":1102},[407,9255,3708],{"class":416},[407,9257,3100],{"class":416},[407,9259,9260],{"class":1102}," now   ",[407,9262,9263],{"class":736},"# cross-thread handoff to scale-in logic\n",[407,9265,9266,9268],{"class":409,"line":518},[407,9267,2565],{"class":557},[407,9269,3958],{"class":557},[34,9271,9272,9273,9276,9277,9279],{},"When the new VM boots, cloud-init runs ",[98,9274,9275],{},"docker swarm join",", swarmctl sees the new node next\ncycle, the stuck tasks schedule onto it, and ",[98,9278,6801],{}," goes quiet.",[241,9281,9283],{"id":9282},"scale-in-openstack-asks-swarmctl-for-permission","Scale-in: OpenStack asks swarmctl for permission",[34,9285,9286,9287,9290,9291,9293],{},"The reverse direction is more dangerous. OpenStack's Aodh ",[98,9288,9289],{},"cpu_low"," alarm wants to remove a VM.\nBut Aodh only sees aggregate cluster CPU: it has no idea whether removing that node would strand\nservices. So instead of pointing the alarm straight at Heat, it points at ",[37,9292,226],{},", which\napproves or denies:",[260,9295,9297],{"className":2436,"code":9296,"language":2438,"meta":266,"style":266},"@router.post(\"\u002Fscale-in-request\")\nasync def scale_in_request(pool: str = \"default\", ...):\n    result = await asyncio.to_thread(service.evaluate_scale_in, pool)\n    if result[\"approved\"]:\n        await asyncio.to_thread(AsgService().trigger_scale_in, pool)\n    return ScaleInResponse(**result)   # always 200 so Aodh never retries\n",[98,9298,9299,9322,9355,9388,9406,9434],{"__ignoreMap":266},[407,9300,9301,9304,9307,9309,9311,9313,9315,9318,9320],{"class":409,"line":11},[407,9302,9303],{"class":416},"@",[407,9305,9306],{"class":1078},"router",[407,9308,167],{"class":416},[407,9310,9159],{"class":1078},[407,9312,2451],{"class":416},[407,9314,2000],{"class":1877},[407,9316,9317],{"class":436},"\u002Fscale-in-request",[407,9319,2000],{"class":1877},[407,9321,2764],{"class":416},[407,9323,9324,9327,9330,9333,9335,9337,9339,9341,9343,9345,9347,9349,9351,9353],{"class":409,"line":20},[407,9325,9326],{"class":1095},"async",[407,9328,9329],{"class":1095}," def",[407,9331,9332],{"class":1078}," scale_in_request",[407,9334,2451],{"class":416},[407,9336,8787],{"class":1102},[407,9338,433],{"class":416},[407,9340,2488],{"class":412},[407,9342,3100],{"class":416},[407,9344,1994],{"class":1877},[407,9346,8993],{"class":436},[407,9348,2000],{"class":1877},[407,9350,750],{"class":416},[407,9352,3489],{"class":1102},[407,9354,3568],{"class":416},[407,9356,9357,9360,9362,9365,9368,9370,9373,9375,9377,9379,9382,9384,9386],{"class":409,"line":427},[407,9358,9359],{"class":1102},"    result ",[407,9361,2499],{"class":416},[407,9363,9364],{"class":557}," await",[407,9366,9367],{"class":1102}," asyncio",[407,9369,167],{"class":416},[407,9371,9372],{"class":1102},"to_thread",[407,9374,2451],{"class":416},[407,9376,3926],{"class":1102},[407,9378,167],{"class":416},[407,9380,9381],{"class":1102},"evaluate_scale_in",[407,9383,750],{"class":416},[407,9385,1148],{"class":1102},[407,9387,2764],{"class":416},[407,9389,9390,9392,9395,9397,9399,9402,9404],{"class":409,"line":440},[407,9391,2552],{"class":557},[407,9393,9394],{"class":1102}," result",[407,9396,2466],{"class":416},[407,9398,2000],{"class":1877},[407,9400,9401],{"class":436},"approved",[407,9403,2000],{"class":1877},[407,9405,4020],{"class":416},[407,9407,9408,9411,9413,9415,9417,9419,9422,9425,9428,9430,9432],{"class":409,"line":457},[407,9409,9410],{"class":557},"        await",[407,9412,9367],{"class":1102},[407,9414,167],{"class":416},[407,9416,9372],{"class":1102},[407,9418,2451],{"class":416},[407,9420,9421],{"class":1102},"AsgService",[407,9423,9424],{"class":416},"().",[407,9426,9427],{"class":1102},"trigger_scale_in",[407,9429,750],{"class":416},[407,9431,1148],{"class":1102},[407,9433,2764],{"class":416},[407,9435,9436,9438,9441,9443,9445,9448,9450],{"class":409,"line":465},[407,9437,3385],{"class":557},[407,9439,9440],{"class":1102}," ScaleInResponse",[407,9442,2451],{"class":416},[407,9444,4234],{"class":1095},[407,9446,9447],{"class":1102},"result",[407,9449,2457],{"class":416},[407,9451,9452],{"class":736},"   # always 200 so Aodh never retries\n",[34,9454,9455,9457],{},[98,9456,9381],{}," runs a sequence of safety checks, all scoped to the pool that fired:",[260,9459,9461],{"className":2436,"code":9460,"language":2438,"meta":266,"style":266},"# 1. Post-scale-out cooldown. If this pool just gained a VM, don't immediately remove one,\n#    otherwise Aodh sees the fresh node idle, approves removal, and could delete the ORIGINAL\n#    production node, briefly dropping to zero replicas (1 → 0 → 1).\nlast_out = _asg_mod.last_scale_out_ts(pool)\nif (now - last_out) \u003C cooldown:\n    return {\"approved\": False, \"reason\": \"post-scale-out cooldown\"}\n\n# 2. Scale-in's own cooldown.\n# 3. Minimum node count: strict \u003C=, so a single-node pool can never be scaled to zero.\nif len(active_workers) \u003C= min_nodes:\n    return {\"approved\": False, \"reason\": \"at minimum node count\"}\n\n# 4. Any pending task in THIS pool blocks removal: the cluster is already short on capacity.\nfor svc in services_in_pool:\n    if svc has a pending task:\n        return {\"approved\": False, \"reason\": f\"{svc.name} has pending tasks\"}\n\nreturn {\"approved\": True, \"reason\": \"safe\"}\n",[98,9462,9463,9468,9473,9478,9499,9521,9557,9561,9566,9571,9593,9628,9632,9637,9651,9660,9706,9710],{"__ignoreMap":266},[407,9464,9465],{"class":409,"line":11},[407,9466,9467],{"class":736},"# 1. Post-scale-out cooldown. If this pool just gained a VM, don't immediately remove one,\n",[407,9469,9470],{"class":409,"line":20},[407,9471,9472],{"class":736},"#    otherwise Aodh sees the fresh node idle, approves removal, and could delete the ORIGINAL\n",[407,9474,9475],{"class":409,"line":427},[407,9476,9477],{"class":736},"#    production node, briefly dropping to zero replicas (1 → 0 → 1).\n",[407,9479,9480,9483,9485,9488,9490,9493,9495,9497],{"class":409,"line":440},[407,9481,9482],{"class":1102},"last_out ",[407,9484,2499],{"class":416},[407,9486,9487],{"class":1102}," _asg_mod",[407,9489,167],{"class":416},[407,9491,9492],{"class":1102},"last_scale_out_ts",[407,9494,2451],{"class":416},[407,9496,8787],{"class":1102},[407,9498,2764],{"class":416},[407,9500,9501,9503,9505,9507,9509,9512,9514,9516,9519],{"class":409,"line":457},[407,9502,3354],{"class":557},[407,9504,1277],{"class":416},[407,9506,8136],{"class":1102},[407,9508,2810],{"class":1095},[407,9510,9511],{"class":1102}," last_out",[407,9513,2457],{"class":416},[407,9515,1096],{"class":1095},[407,9517,9518],{"class":1102}," cooldown",[407,9520,417],{"class":416},[407,9522,9523,9525,9527,9529,9531,9533,9535,9537,9539,9541,9544,9546,9548,9550,9553,9555],{"class":409,"line":465},[407,9524,3385],{"class":557},[407,9526,819],{"class":416},[407,9528,2000],{"class":1877},[407,9530,9401],{"class":436},[407,9532,2000],{"class":1877},[407,9534,433],{"class":416},[407,9536,4411],{"class":557},[407,9538,750],{"class":416},[407,9540,1994],{"class":1877},[407,9542,9543],{"class":436},"reason",[407,9545,2000],{"class":1877},[407,9547,433],{"class":416},[407,9549,1994],{"class":1877},[407,9551,9552],{"class":436},"post-scale-out cooldown",[407,9554,2000],{"class":1877},[407,9556,8028],{"class":416},[407,9558,9559],{"class":409,"line":477},[407,9560,530],{"emptyLinePlaceholder":529},[407,9562,9563],{"class":409,"line":485},[407,9564,9565],{"class":736},"# 2. Scale-in's own cooldown.\n",[407,9567,9568],{"class":409,"line":494},[407,9569,9570],{"class":736},"# 3. Minimum node count: strict \u003C=, so a single-node pool can never be scaled to zero.\n",[407,9572,9573,9575,9578,9580,9583,9585,9588,9591],{"class":409,"line":502},[407,9574,3354],{"class":557},[407,9576,9577],{"class":412}," len",[407,9579,2451],{"class":416},[407,9581,9582],{"class":1102},"active_workers",[407,9584,2457],{"class":416},[407,9586,9587],{"class":1095}," \u003C=",[407,9589,9590],{"class":1102}," min_nodes",[407,9592,417],{"class":416},[407,9594,9595,9597,9599,9601,9603,9605,9607,9609,9611,9613,9615,9617,9619,9621,9624,9626],{"class":409,"line":510},[407,9596,3385],{"class":557},[407,9598,819],{"class":416},[407,9600,2000],{"class":1877},[407,9602,9401],{"class":436},[407,9604,2000],{"class":1877},[407,9606,433],{"class":416},[407,9608,4411],{"class":557},[407,9610,750],{"class":416},[407,9612,1994],{"class":1877},[407,9614,9543],{"class":436},[407,9616,2000],{"class":1877},[407,9618,433],{"class":416},[407,9620,1994],{"class":1877},[407,9622,9623],{"class":436},"at minimum node count",[407,9625,2000],{"class":1877},[407,9627,8028],{"class":416},[407,9629,9630],{"class":409,"line":518},[407,9631,530],{"emptyLinePlaceholder":529},[407,9633,9634],{"class":409,"line":526},[407,9635,9636],{"class":736},"# 4. Any pending task in THIS pool blocks removal: the cluster is already short on capacity.\n",[407,9638,9639,9641,9644,9646,9649],{"class":409,"line":533},[407,9640,1833],{"class":557},[407,9642,9643],{"class":1102}," svc ",[407,9645,2779],{"class":557},[407,9647,9648],{"class":1102}," services_in_pool",[407,9650,417],{"class":416},[407,9652,9653,9655,9658],{"class":409,"line":541},[407,9654,2552],{"class":557},[407,9656,9657],{"class":1102}," svc has a pending task",[407,9659,417],{"class":416},[407,9661,9662,9664,9666,9668,9670,9672,9674,9676,9678,9680,9682,9684,9686,9688,9690,9692,9694,9696,9699,9701,9704],{"class":409,"line":549},[407,9663,2565],{"class":557},[407,9665,819],{"class":416},[407,9667,2000],{"class":1877},[407,9669,9401],{"class":436},[407,9671,2000],{"class":1877},[407,9673,433],{"class":416},[407,9675,4411],{"class":557},[407,9677,750],{"class":416},[407,9679,1994],{"class":1877},[407,9681,9543],{"class":436},[407,9683,2000],{"class":1877},[407,9685,433],{"class":416},[407,9687,5107],{"class":1095},[407,9689,2000],{"class":436},[407,9691,2927],{"class":1088},[407,9693,2756],{"class":1102},[407,9695,167],{"class":416},[407,9697,9698],{"class":1102},"name",[407,9700,2939],{"class":1088},[407,9702,9703],{"class":436}," has pending tasks\"",[407,9705,8028],{"class":416},[407,9707,9708],{"class":409,"line":561},[407,9709,530],{"emptyLinePlaceholder":529},[407,9711,9712,9714,9716,9718,9720,9722,9724,9726,9728,9730,9732,9734,9736,9738,9741,9743],{"class":409,"line":2058},[407,9713,7465],{"class":557},[407,9715,819],{"class":416},[407,9717,2000],{"class":1877},[407,9719,9401],{"class":436},[407,9721,2000],{"class":1877},[407,9723,433],{"class":416},[407,9725,5857],{"class":557},[407,9727,750],{"class":416},[407,9729,1994],{"class":1877},[407,9731,9543],{"class":436},[407,9733,2000],{"class":1877},[407,9735,433],{"class":416},[407,9737,1994],{"class":1877},[407,9739,9740],{"class":436},"safe",[407,9742,2000],{"class":1877},[407,9744,8028],{"class":416},[34,9746,9747],{},"The post-scale-out cooldown is the subtle one, and it needs cross-thread state: the watch loop\n(main thread) records the last scale-out timestamp per pool, and the hook server (daemon thread)\nreads it. That handoff goes through a lock-protected module dict:",[260,9749,9751],{"className":2436,"code":9750,"language":2438,"meta":266,"style":266},"_ts_lock = threading.Lock()\n_last_scale_out_ts: dict[str, float] = {}\n\ndef last_scale_out_ts(pool: str) -> float:\n    with _ts_lock:\n        return _last_scale_out_ts.get(pool, 0.0)\n",[98,9752,9753,9770,9794,9798,9821,9829],{"__ignoreMap":266},[407,9754,9755,9758,9760,9763,9765,9768],{"class":409,"line":11},[407,9756,9757],{"class":1102},"_ts_lock ",[407,9759,2499],{"class":416},[407,9761,9762],{"class":1102}," threading",[407,9764,167],{"class":416},[407,9766,9767],{"class":1102},"Lock",[407,9769,2505],{"class":416},[407,9771,9772,9775,9777,9780,9782,9784,9786,9788,9790,9792],{"class":409,"line":20},[407,9773,9774],{"class":1102},"_last_scale_out_ts",[407,9776,433],{"class":416},[407,9778,9779],{"class":1102}," dict",[407,9781,2466],{"class":416},[407,9783,2483],{"class":412},[407,9785,750],{"class":416},[407,9787,3644],{"class":412},[407,9789,3708],{"class":416},[407,9791,3100],{"class":416},[407,9793,7573],{"class":416},[407,9795,9796],{"class":409,"line":427},[407,9797,530],{"emptyLinePlaceholder":529},[407,9799,9800,9802,9805,9807,9809,9811,9813,9815,9817,9819],{"class":409,"line":440},[407,9801,2445],{"class":1095},[407,9803,9804],{"class":1078}," last_scale_out_ts",[407,9806,2451],{"class":416},[407,9808,8787],{"class":1102},[407,9810,433],{"class":416},[407,9812,2488],{"class":412},[407,9814,2457],{"class":416},[407,9816,2460],{"class":416},[407,9818,3644],{"class":412},[407,9820,417],{"class":416},[407,9822,9823,9825,9827],{"class":409,"line":457},[407,9824,2645],{"class":557},[407,9826,9242],{"class":1102},[407,9828,417],{"class":416},[407,9830,9831,9833,9836,9838,9840,9842,9844,9846,9848],{"class":409,"line":465},[407,9832,2565],{"class":557},[407,9834,9835],{"class":1102}," _last_scale_out_ts",[407,9837,167],{"class":416},[407,9839,3135],{"class":1102},[407,9841,2451],{"class":416},[407,9843,8787],{"class":1102},[407,9845,750],{"class":416},[407,9847,6359],{"class":473},[407,9849,2764],{"class":416},[241,9851,9853],{"id":9852},"pre-terminate-drain-a-node-gracefully","Pre-terminate: drain a node gracefully",[34,9855,9856,9857,4277,9860,9863],{},"Once a scale-in is approved and Heat picks a victim VM, the VM's own systemd shutdown hook calls\n",[98,9858,9859],{},"POST \u002Fhooks\u002Fpre-terminate",[37,9861,9862],{},"before"," Docker stops. This is the graceful-drain path, and it is\nthe most involved piece of orchestration in the system:",[260,9865,9867],{"className":2436,"code":9866,"language":2438,"meta":266,"style":266},"def handle(self, node_name, instance_id) -> dict:\n    node = self._find_node(node_name)\n    if node is None:\n        return {\"status\": \"approve\", \"reason\": \"node not found in swarm\"}\n\n    running_tasks = self._tasks_on_node(node.id)\n    if running_tasks:\n        # 1. Pause the node so Swarm schedules nothing new onto a dying host.\n        if not self._set_availability(node, AVAIL_PAUSE):\n            return {\"status\": \"reject\", \"reason\": \"failed to pause node\"}\n\n        # 2. Drain: scale affected services DOWN by their task count on this node,\n        #    so Swarm reschedules those replicas elsewhere.\n        adjustments = self._compute_adjustments(node.id, running_tasks)\n        self._apply_scale_adjustments(adjustments, scale_down=True)\n\n        # 3. Wait (bounded) for tasks to actually leave the node.\n        drained = self._wait_for_drain(node.id)\n\n        # 4. Scale back UP to the original counts: the new replicas land on OTHER nodes.\n        self._apply_scale_adjustments(adjustments, scale_down=False)\n\n        # 5. Write REBASELINE events so the watch loop treats the restored counts as the\n        #    authoritative target, instead of trying to \"reconcile\" the drain-time dip.\n        self._write_rebaseline_events(adjustments)\n\n        self._set_availability(self._client.nodes.get(node.id), AVAIL_ACTIVE)\n\n    # 6. Finally, simulate removing the node for real.\n    return self._simulate_removal(node.id)\n",[98,9868,9869,9898,9919,9932,9973,9977,10002,10011,10016,10040,10081,10085,10090,10095,10123,10148,10152,10157,10181,10185,10190,10212,10216,10221,10226,10241,10245,10286,10290,10296],{"__ignoreMap":266},[407,9870,9871,9873,9876,9878,9880,9882,9885,9887,9890,9892,9894,9896],{"class":409,"line":11},[407,9872,2445],{"class":1095},[407,9874,9875],{"class":1078}," handle",[407,9877,2451],{"class":416},[407,9879,2454],{"class":1102},[407,9881,750],{"class":416},[407,9883,9884],{"class":1102}," node_name",[407,9886,750],{"class":416},[407,9888,9889],{"class":1102}," instance_id",[407,9891,2457],{"class":416},[407,9893,2460],{"class":416},[407,9895,9779],{"class":412},[407,9897,417],{"class":416},[407,9899,9900,9903,9905,9907,9909,9912,9914,9917],{"class":409,"line":20},[407,9901,9902],{"class":1102},"    node ",[407,9904,2499],{"class":416},[407,9906,2536],{"class":1088},[407,9908,167],{"class":416},[407,9910,9911],{"class":1102},"_find_node",[407,9913,2451],{"class":416},[407,9915,9916],{"class":1102},"node_name",[407,9918,2764],{"class":416},[407,9920,9921,9923,9926,9928,9930],{"class":409,"line":427},[407,9922,2552],{"class":557},[407,9924,9925],{"class":1102}," node ",[407,9927,2238],{"class":1095},[407,9929,3346],{"class":557},[407,9931,417],{"class":416},[407,9933,9934,9936,9938,9940,9943,9945,9947,9949,9952,9954,9956,9958,9960,9962,9964,9966,9969,9971],{"class":409,"line":440},[407,9935,2565],{"class":557},[407,9937,819],{"class":416},[407,9939,2000],{"class":1877},[407,9941,9942],{"class":436},"status",[407,9944,2000],{"class":1877},[407,9946,433],{"class":416},[407,9948,1994],{"class":1877},[407,9950,9951],{"class":436},"approve",[407,9953,2000],{"class":1877},[407,9955,750],{"class":416},[407,9957,1994],{"class":1877},[407,9959,9543],{"class":436},[407,9961,2000],{"class":1877},[407,9963,433],{"class":416},[407,9965,1994],{"class":1877},[407,9967,9968],{"class":436},"node not found in swarm",[407,9970,2000],{"class":1877},[407,9972,8028],{"class":416},[407,9974,9975],{"class":409,"line":457},[407,9976,530],{"emptyLinePlaceholder":529},[407,9978,9979,9982,9984,9986,9988,9991,9993,9996,9998,10000],{"class":409,"line":465},[407,9980,9981],{"class":1102},"    running_tasks ",[407,9983,2499],{"class":416},[407,9985,2536],{"class":1088},[407,9987,167],{"class":416},[407,9989,9990],{"class":1102},"_tasks_on_node",[407,9992,2451],{"class":416},[407,9994,9995],{"class":1102},"node",[407,9997,167],{"class":416},[407,9999,2743],{"class":1102},[407,10001,2764],{"class":416},[407,10003,10004,10006,10009],{"class":409,"line":477},[407,10005,2552],{"class":557},[407,10007,10008],{"class":1102}," running_tasks",[407,10010,417],{"class":416},[407,10012,10013],{"class":409,"line":485},[407,10014,10015],{"class":736},"        # 1. Pause the node so Swarm schedules nothing new onto a dying host.\n",[407,10017,10018,10020,10022,10024,10026,10029,10031,10033,10035,10038],{"class":409,"line":494},[407,10019,2861],{"class":557},[407,10021,2555],{"class":1095},[407,10023,2536],{"class":1088},[407,10025,167],{"class":416},[407,10027,10028],{"class":1102},"_set_availability",[407,10030,2451],{"class":416},[407,10032,9995],{"class":1102},[407,10034,750],{"class":416},[407,10036,10037],{"class":1088}," AVAIL_PAUSE",[407,10039,3568],{"class":416},[407,10041,10042,10045,10047,10049,10051,10053,10055,10057,10060,10062,10064,10066,10068,10070,10072,10074,10077,10079],{"class":409,"line":502},[407,10043,10044],{"class":557},"            return",[407,10046,819],{"class":416},[407,10048,2000],{"class":1877},[407,10050,9942],{"class":436},[407,10052,2000],{"class":1877},[407,10054,433],{"class":416},[407,10056,1994],{"class":1877},[407,10058,10059],{"class":436},"reject",[407,10061,2000],{"class":1877},[407,10063,750],{"class":416},[407,10065,1994],{"class":1877},[407,10067,9543],{"class":436},[407,10069,2000],{"class":1877},[407,10071,433],{"class":416},[407,10073,1994],{"class":1877},[407,10075,10076],{"class":436},"failed to pause node",[407,10078,2000],{"class":1877},[407,10080,8028],{"class":416},[407,10082,10083],{"class":409,"line":510},[407,10084,530],{"emptyLinePlaceholder":529},[407,10086,10087],{"class":409,"line":518},[407,10088,10089],{"class":736},"        # 2. Drain: scale affected services DOWN by their task count on this node,\n",[407,10091,10092],{"class":409,"line":526},[407,10093,10094],{"class":736},"        #    so Swarm reschedules those replicas elsewhere.\n",[407,10096,10097,10100,10102,10104,10106,10109,10111,10113,10115,10117,10119,10121],{"class":409,"line":533},[407,10098,10099],{"class":1102},"        adjustments ",[407,10101,2499],{"class":416},[407,10103,2536],{"class":1088},[407,10105,167],{"class":416},[407,10107,10108],{"class":1102},"_compute_adjustments",[407,10110,2451],{"class":416},[407,10112,9995],{"class":1102},[407,10114,167],{"class":416},[407,10116,2743],{"class":1102},[407,10118,750],{"class":416},[407,10120,10008],{"class":1102},[407,10122,2764],{"class":416},[407,10124,10125,10127,10129,10132,10134,10137,10139,10142,10144,10146],{"class":409,"line":541},[407,10126,9220],{"class":1088},[407,10128,167],{"class":416},[407,10130,10131],{"class":1102},"_apply_scale_adjustments",[407,10133,2451],{"class":416},[407,10135,10136],{"class":1102},"adjustments",[407,10138,750],{"class":416},[407,10140,10141],{"class":1836}," scale_down",[407,10143,2499],{"class":416},[407,10145,6805],{"class":557},[407,10147,2764],{"class":416},[407,10149,10150],{"class":409,"line":549},[407,10151,530],{"emptyLinePlaceholder":529},[407,10153,10154],{"class":409,"line":561},[407,10155,10156],{"class":736},"        # 3. Wait (bounded) for tasks to actually leave the node.\n",[407,10158,10159,10162,10164,10166,10168,10171,10173,10175,10177,10179],{"class":409,"line":2058},[407,10160,10161],{"class":1102},"        drained ",[407,10163,2499],{"class":416},[407,10165,2536],{"class":1088},[407,10167,167],{"class":416},[407,10169,10170],{"class":1102},"_wait_for_drain",[407,10172,2451],{"class":416},[407,10174,9995],{"class":1102},[407,10176,167],{"class":416},[407,10178,2743],{"class":1102},[407,10180,2764],{"class":416},[407,10182,10183],{"class":409,"line":2079},[407,10184,530],{"emptyLinePlaceholder":529},[407,10186,10187],{"class":409,"line":2084},[407,10188,10189],{"class":736},"        # 4. Scale back UP to the original counts: the new replicas land on OTHER nodes.\n",[407,10191,10192,10194,10196,10198,10200,10202,10204,10206,10208,10210],{"class":409,"line":2090},[407,10193,9220],{"class":1088},[407,10195,167],{"class":416},[407,10197,10131],{"class":1102},[407,10199,2451],{"class":416},[407,10201,10136],{"class":1102},[407,10203,750],{"class":416},[407,10205,10141],{"class":1836},[407,10207,2499],{"class":416},[407,10209,7075],{"class":557},[407,10211,2764],{"class":416},[407,10213,10214],{"class":409,"line":2108},[407,10215,530],{"emptyLinePlaceholder":529},[407,10217,10218],{"class":409,"line":2858},[407,10219,10220],{"class":736},"        # 5. Write REBASELINE events so the watch loop treats the restored counts as the\n",[407,10222,10223],{"class":409,"line":2869},[407,10224,10225],{"class":736},"        #    authoritative target, instead of trying to \"reconcile\" the drain-time dip.\n",[407,10227,10228,10230,10232,10235,10237,10239],{"class":409,"line":2909},[407,10229,9220],{"class":1088},[407,10231,167],{"class":416},[407,10233,10234],{"class":1102},"_write_rebaseline_events",[407,10236,2451],{"class":416},[407,10238,10136],{"class":1102},[407,10240,2764],{"class":416},[407,10242,10243],{"class":409,"line":2956},[407,10244,530],{"emptyLinePlaceholder":529},[407,10246,10247,10249,10251,10253,10255,10257,10259,10262,10264,10267,10269,10271,10273,10275,10277,10279,10281,10284],{"class":409,"line":2970},[407,10248,9220],{"class":1088},[407,10250,167],{"class":416},[407,10252,10028],{"class":1102},[407,10254,2451],{"class":416},[407,10256,2454],{"class":1088},[407,10258,167],{"class":416},[407,10260,10261],{"class":1102},"_client",[407,10263,167],{"class":416},[407,10265,10266],{"class":1102},"nodes",[407,10268,167],{"class":416},[407,10270,3135],{"class":1102},[407,10272,2451],{"class":416},[407,10274,9995],{"class":1102},[407,10276,167],{"class":416},[407,10278,2743],{"class":1102},[407,10280,2671],{"class":416},[407,10282,10283],{"class":1088}," AVAIL_ACTIVE",[407,10285,2764],{"class":416},[407,10287,10288],{"class":409,"line":2983},[407,10289,530],{"emptyLinePlaceholder":529},[407,10291,10293],{"class":409,"line":10292},29,[407,10294,10295],{"class":736},"    # 6. Finally, simulate removing the node for real.\n",[407,10297,10299,10301,10303,10305,10308,10310,10312,10314,10316],{"class":409,"line":10298},30,[407,10300,3385],{"class":557},[407,10302,2536],{"class":1088},[407,10304,167],{"class":416},[407,10306,10307],{"class":1102},"_simulate_removal",[407,10309,2451],{"class":416},[407,10311,9995],{"class":1102},[407,10313,167],{"class":416},[407,10315,2743],{"class":1102},[407,10317,2764],{"class":416},[34,10319,10320,10321,10324,10325,10328],{},"The drain is a ",[154,10322,10323],{},"temporary"," scale-down-then-up dance: you shrink each service by exactly the\nnumber of its tasks on the doomed node, let Swarm move them, then grow back so the replacements\nschedule onto healthy nodes. If the drain times out, swarmctl restores the counts and ",[37,10326,10327],{},"rejects","\nthe termination: better to keep a node than to lose replicas.",[34,10330,10331],{},"Here is the whole sequence, including the final simulation that can still say no:",[10333,10334],"swarm-drain",{},[34,10336,7505,10337,10340,10341,10344],{},[37,10338,10339],{},"rebaseline"," step solves a nasty interaction with the watch loop: during the drain the loop\nwould see service counts dip and might try to \"fix\" them. Writing a ",[98,10342,10343],{},"REBASELINE"," audit event\ntells the loop \"this new count is intentional, adopt it as the target,\" so the two subsystems\ndon't fight.",[34,10346,10347,10348,10350,10351,10353],{},"Finally, ",[98,10349,10307],{}," is the last safety gate: it projects every surviving node's memory\nif this node's load is redistributed, and ",[37,10352,10327],{}," the termination if any node would exceed\nthe rebound threshold:",[260,10355,10357],{"className":2436,"code":10356,"language":2438,"meta":266,"style":266},"threshold = self._settings.hook_rebound_threshold   # e.g. 65%\nfor nid, used_mb in node_used_mem.items():\n    pct = (used_mb * 1024 * 1024) \u002F node_total_mem[nid]\n    if pct > threshold:\n        return {\"status\": \"reject\", \"reason\": HOOK_REASON_REBOUND}\nreturn {\"status\": \"approve\", \"reason\": \"safe\"}\n",[98,10358,10359,10381,10404,10438,10452,10487],{"__ignoreMap":266},[407,10360,10361,10364,10366,10368,10370,10373,10375,10378],{"class":409,"line":11},[407,10362,10363],{"class":1102},"threshold ",[407,10365,2499],{"class":416},[407,10367,2536],{"class":1088},[407,10369,167],{"class":416},[407,10371,10372],{"class":1102},"_settings",[407,10374,167],{"class":416},[407,10376,10377],{"class":1102},"hook_rebound_threshold   ",[407,10379,10380],{"class":736},"# e.g. 65%\n",[407,10382,10383,10385,10388,10390,10393,10395,10398,10400,10402],{"class":409,"line":20},[407,10384,1833],{"class":557},[407,10386,10387],{"class":1102}," nid",[407,10389,750],{"class":416},[407,10391,10392],{"class":1102}," used_mb ",[407,10394,2779],{"class":557},[407,10396,10397],{"class":1102}," node_used_mem",[407,10399,167],{"class":416},[407,10401,4090],{"class":1102},[407,10403,8952],{"class":416},[407,10405,10406,10409,10411,10413,10416,10418,10420,10422,10424,10426,10428,10431,10433,10436],{"class":409,"line":427},[407,10407,10408],{"class":1102},"    pct ",[407,10410,2499],{"class":416},[407,10412,1277],{"class":416},[407,10414,10415],{"class":1102},"used_mb ",[407,10417,3222],{"class":1095},[407,10419,3109],{"class":473},[407,10421,3106],{"class":1095},[407,10423,3109],{"class":473},[407,10425,2457],{"class":416},[407,10427,5385],{"class":1095},[407,10429,10430],{"class":1102}," node_total_mem",[407,10432,2466],{"class":416},[407,10434,10435],{"class":1102},"nid",[407,10437,454],{"class":416},[407,10439,10440,10442,10445,10447,10450],{"class":409,"line":440},[407,10441,2552],{"class":557},[407,10443,10444],{"class":1102}," pct ",[407,10446,1848],{"class":1095},[407,10448,10449],{"class":1102}," threshold",[407,10451,417],{"class":416},[407,10453,10454,10456,10458,10460,10462,10464,10466,10468,10470,10472,10474,10476,10478,10480,10482,10485],{"class":409,"line":457},[407,10455,2565],{"class":557},[407,10457,819],{"class":416},[407,10459,2000],{"class":1877},[407,10461,9942],{"class":436},[407,10463,2000],{"class":1877},[407,10465,433],{"class":416},[407,10467,1994],{"class":1877},[407,10469,10059],{"class":436},[407,10471,2000],{"class":1877},[407,10473,750],{"class":416},[407,10475,1994],{"class":1877},[407,10477,9543],{"class":436},[407,10479,2000],{"class":1877},[407,10481,433],{"class":416},[407,10483,10484],{"class":1088}," HOOK_REASON_REBOUND",[407,10486,8028],{"class":416},[407,10488,10489,10491,10493,10495,10497,10499,10501,10503,10505,10507,10509,10511,10513,10515,10517,10519,10521,10523],{"class":409,"line":465},[407,10490,7465],{"class":557},[407,10492,819],{"class":416},[407,10494,2000],{"class":1877},[407,10496,9942],{"class":436},[407,10498,2000],{"class":1877},[407,10500,433],{"class":416},[407,10502,1994],{"class":1877},[407,10504,9951],{"class":436},[407,10506,2000],{"class":1877},[407,10508,750],{"class":416},[407,10510,1994],{"class":1877},[407,10512,9543],{"class":436},[407,10514,2000],{"class":1877},[407,10516,433],{"class":416},[407,10518,1994],{"class":1877},[407,10520,9740],{"class":436},[407,10522,2000],{"class":1877},[407,10524,8028],{"class":416},[241,10526,10528],{"id":10527},"the-one-rule-openstack-must-obey","The one rule OpenStack must obey",[34,10530,10531],{},"The webhook can only ever return approve or reject. The critical operational contract is on the\nOpenStack side:",[31,10533,10534],{},[34,10535,10536,10539],{},[37,10537,10538],{},"OpenStack must be configured to reject a termination by default if the webhook is\nunreachable."," A node must never be removed without positive approval.",[34,10541,10542],{},"Fail-safe, not fail-open. If swarmctl is down, nodes stay up: a slightly-too-big cluster is\nalways cheaper than an outage.",[53,10544],{},[42,10546,10548],{"id":10547},"design-principles-that-run-through-the-whole-thing","Design principles that run through the whole thing",[34,10550,10551],{},"A few ideas show up again and again, and they're worth stating plainly because they're what make\nthe system trustworthy:",[34,10553,10554,10557],{},[37,10555,10556],{},"The engine is pure."," No I\u002FO, no clock, no randomness. Every bit of timing is injected by the\nwatch loop. This is why it can be exhaustively unit-tested, and why you can reason about its\nbehavior by reading one function.",[34,10559,10560,10563],{},[37,10561,10562],{},"Scale up fast, scale down slow."," Up: any single trigger fires, step 1 to 3 by urgency. Down:\nevery condition true, sustained for a full duration, and a per-metric simulation must pass. The\ncost of under-provisioning (dropped requests) is far higher than the cost of a spare replica.",[34,10565,10566,10569],{},[37,10567,10568],{},"Different metrics behave differently."," CPU redistributes across replicas; memory is\nprocess-private; latency is non-linear and often upstream-bound. The simulation treats each\ncorrectly instead of applying one proportional formula to all three, a bug our first version\nactually had.",[34,10571,10572,10575,10576,10578,10579,10581],{},[37,10573,10574],{},"Separate \"scale containers\" from \"scale VMs\"."," The ",[98,10577,4649],{}," and the\n",[98,10580,6801],{}," are two independent outputs. One drives the Docker API; the other drives\nOpenStack. Keeping them separate keeps each layer simple.",[34,10583,10584,10587],{},[37,10585,10586],{},"Fail safe."," Crash-looping services aren't scaled. Broken deployments (bad image, missing\nsecret) don't trigger capacity signals. Node termination defaults to reject. Partial successes\nare recorded as partial, not as success.",[34,10589,10590,10593,10594,10597],{},[37,10591,10592],{},"Every decision is auditable."," Every cycle, every service, every NONE-with-a-score lands in\nSQLite. When you later ask \"why did it scale at 14:03?\" or \"why ",[154,10595,10596],{},"didn't"," it scale?\", the answer\nis a query away.",[53,10599],{},[42,10601,10603],{"id":10602},"running-it","Running it",[34,10605,10606],{},"Run it as a Swarm service on a manager node, with the Docker socket mounted and the hook port exposed:",[260,10608,10610],{"className":401,"code":10609,"language":403,"meta":266,"style":266},"services:\n  swarmctl:\n    image: yourrepo\u002Fswarmctl:latest\n    command: [\"swarmctl\", \"watch\", \"--interval\", \"30\"]\n    deploy:\n      replicas: 1\n      placement:\n        constraints: [\"node.role == manager\"]\n      restart_policy:\n        condition: any\n    volumes:\n      - \u002Fvar\u002Frun\u002Fdocker.sock:\u002Fvar\u002Frun\u002Fdocker.sock:ro\n      - swarmctl_data:\u002Fdata\n    ports:\n      - \"8889:8889\"   # OpenStack lifecycle hook\n",[98,10611,10612,10618,10625,10634,10678,10684,10692,10699,10716,10723,10733,10740,10747,10754,10761],{"__ignoreMap":266},[407,10613,10614,10616],{"class":409,"line":11},[407,10615,413],{"class":412},[407,10617,417],{"class":416},[407,10619,10620,10623],{"class":409,"line":20},[407,10621,10622],{"class":412},"  swarmctl",[407,10624,417],{"class":416},[407,10626,10627,10629,10631],{"class":409,"line":427},[407,10628,430],{"class":412},[407,10630,433],{"class":416},[407,10632,10633],{"class":436}," yourrepo\u002Fswarmctl:latest\n",[407,10635,10636,10639,10641,10643,10645,10647,10649,10651,10653,10656,10658,10660,10662,10665,10667,10669,10671,10674,10676],{"class":409,"line":440},[407,10637,10638],{"class":412},"    command",[407,10640,433],{"class":416},[407,10642,448],{"class":416},[407,10644,2000],{"class":1877},[407,10646,226],{"class":436},[407,10648,2000],{"class":1877},[407,10650,750],{"class":416},[407,10652,1994],{"class":1877},[407,10654,10655],{"class":436},"watch",[407,10657,2000],{"class":1877},[407,10659,750],{"class":416},[407,10661,1994],{"class":1877},[407,10663,10664],{"class":436},"--interval",[407,10666,2000],{"class":1877},[407,10668,750],{"class":416},[407,10670,1994],{"class":1877},[407,10672,10673],{"class":436},"30",[407,10675,2000],{"class":1877},[407,10677,454],{"class":416},[407,10679,10680,10682],{"class":409,"line":457},[407,10681,460],{"class":412},[407,10683,417],{"class":416},[407,10685,10686,10688,10690],{"class":409,"line":465},[407,10687,468],{"class":412},[407,10689,433],{"class":416},[407,10691,1696],{"class":473},[407,10693,10694,10697],{"class":409,"line":477},[407,10695,10696],{"class":412},"      placement",[407,10698,417],{"class":416},[407,10700,10701,10704,10706,10708,10710,10712,10714],{"class":409,"line":485},[407,10702,10703],{"class":412},"        constraints",[407,10705,433],{"class":416},[407,10707,448],{"class":416},[407,10709,2000],{"class":1877},[407,10711,583],{"class":436},[407,10713,2000],{"class":1877},[407,10715,454],{"class":416},[407,10717,10718,10721],{"class":409,"line":494},[407,10719,10720],{"class":412},"      restart_policy",[407,10722,417],{"class":416},[407,10724,10725,10728,10730],{"class":409,"line":502},[407,10726,10727],{"class":412},"        condition",[407,10729,433],{"class":416},[407,10731,10732],{"class":436}," any\n",[407,10734,10735,10738],{"class":409,"line":510},[407,10736,10737],{"class":412},"    volumes",[407,10739,417],{"class":416},[407,10741,10742,10744],{"class":409,"line":518},[407,10743,1039],{"class":416},[407,10745,10746],{"class":436}," \u002Fvar\u002Frun\u002Fdocker.sock:\u002Fvar\u002Frun\u002Fdocker.sock:ro\n",[407,10748,10749,10751],{"class":409,"line":526},[407,10750,1039],{"class":416},[407,10752,10753],{"class":436}," swarmctl_data:\u002Fdata\n",[407,10755,10756,10759],{"class":409,"line":533},[407,10757,10758],{"class":412},"    ports",[407,10760,417],{"class":416},[407,10762,10763,10765,10767,10770,10772],{"class":409,"line":541},[407,10764,1039],{"class":416},[407,10766,1994],{"class":1877},[407,10768,10769],{"class":436},"8889:8889",[407,10771,2000],{"class":1877},[407,10773,10774],{"class":736},"   # OpenStack lifecycle hook\n",[34,10776,10777,10778,10780],{},"Then watch it work: ",[98,10779,226],{}," prints a live Rich table every cycle showing each service's\nreplicas, CPU avg\u002Fmax, throttle, memory, request rate, latency, error rate, the decision taken,\nand a ⚠ marker when the host-scale signal is raised.",[34,10782,10783,10784,10786],{},"Opt a service in with a single label (",[98,10785,2380],{},"), optionally pick a template, and\nlet the loop do the rest.",[53,10788],{},[42,10790,10792],{"id":10791},"closing-thought","Closing thought",[34,10794,10795,10796,10798,10799,10801,10802,10804,10805,10807,10808,10810,10811,10813],{},"An autoscaler is mostly a study in restraint. The hard part isn't deciding to scale up when CPU\nis high: that's one ",[98,10797,3354],{},". The hard part is ",[154,10800,714],{}," scaling down into an oscillation, ",[154,10803,714],{}," fighting\nan operator's manual change, ",[154,10806,714],{}," killing a node that would strand a service, ",[154,10809,714],{}," reacting to a\nsingle spiky cycle, and ",[154,10812,714],{}," trusting a proportional formula that happens to be wrong for memory.",[34,10815,10816,10817,10819,10820,10823,10824,10827],{},"Every guard, cooldown, duration timer, simulation, and reconciliation branch in ",[98,10818,226],{}," exists\nbecause our naive first version did the wrong thing in production. That's the real content of this post:\nnot that you ",[154,10821,10822],{},"can"," call ",[98,10825,10826],{},"service.scale()",", but all the things you have to check before you dare to.",[2122,10829,10830],{},"html pre.shiki code .su6XF, html code.shiki .su6XF{--shiki-default:#998418}html pre.shiki code .sYZai, html code.shiki .sYZai{--shiki-default:#999999}html pre.shiki code .sSP4y, html code.shiki .sSP4y{--shiki-default:#B5695977}html pre.shiki code .spphp, html code.shiki .spphp{--shiki-default:#B56959}html pre.shiki code .s8zF2, html code.shiki .s8zF2{--shiki-default:#A0ADA0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .si04Y, html code.shiki .si04Y{--shiki-default:#AB5959}html pre.shiki code .sySUi, html code.shiki .sySUi{--shiki-default:#59873A}html pre.shiki code .suHK_, html code.shiki .suHK_{--shiki-default:#393A34}html pre.shiki code .sEi1f, html code.shiki .sEi1f{--shiki-default:#A65E2B}html pre.shiki code .sbBg2, html code.shiki .sbBg2{--shiki-default:#1E754F}html pre.shiki code .svycV, html code.shiki .svycV{--shiki-default:#B07D48}html pre.shiki code .s-TwI, html code.shiki .s-TwI{--shiki-default:#2F798A}html pre.shiki code .sUxyF, html code.shiki .sUxyF{--shiki-default:#2E8F82}",{"title":266,"searchDepth":20,"depth":20,"links":10832},[10833,10834,10835,10844,10850,10857,10866,10872,10873,10874],{"id":2198,"depth":20,"text":2199},{"id":2309,"depth":20,"text":2310},{"id":2339,"depth":20,"text":2340,"children":10836},[10837,10838,10839,10840,10841,10842,10843],{"id":2350,"depth":427,"text":2351},{"id":2425,"depth":427,"text":2426},{"id":3006,"depth":427,"text":3007},{"id":3054,"depth":427,"text":3055},{"id":3260,"depth":427,"text":3261},{"id":3394,"depth":427,"text":3395},{"id":3501,"depth":427,"text":3502},{"id":3788,"depth":20,"text":3789,"children":10845},[10846,10847,10848,10849],{"id":3819,"depth":427,"text":3820},{"id":4511,"depth":427,"text":4512},{"id":5934,"depth":427,"text":5935},{"id":6794,"depth":427,"text":6795},{"id":6837,"depth":20,"text":6838,"children":10851},[10852,10853,10854,10855,10856],{"id":6844,"depth":427,"text":6845},{"id":6952,"depth":427,"text":6953},{"id":7094,"depth":427,"text":7095},{"id":7247,"depth":427,"text":7248},{"id":7526,"depth":427,"text":7527},{"id":7535,"depth":20,"text":7536,"children":10858},[10859,10860,10861,10862,10863,10864,10865],{"id":7740,"depth":427,"text":7741},{"id":8048,"depth":427,"text":8049},{"id":8191,"depth":427,"text":8192},{"id":8284,"depth":427,"text":8285},{"id":8476,"depth":427,"text":8477},{"id":8665,"depth":427,"text":8666},{"id":8676,"depth":427,"text":8677},{"id":8692,"depth":20,"text":8693,"children":10867},[10868,10869,10870,10871],{"id":8709,"depth":427,"text":8710},{"id":9282,"depth":427,"text":9283},{"id":9852,"depth":427,"text":9853},{"id":10527,"depth":427,"text":10528},{"id":10547,"depth":20,"text":10548},{"id":10602,"depth":20,"text":10603},{"id":10791,"depth":20,"text":10792},"2025-12-08","Swarm gives you `docker service scale` and nothing else. This is how I built swarmctl, a daemon that collects metrics, decides when to scale, executes safely, and coordinates VM scaling with OpenStack.",{},"25 min",{"title":2173,"description":10876},"blog\u002Fbuilding-an-autoscaler-for-docker-swarm\u002Findex",[114,10882,10883,2163,2164],"Autoscaling","Python","\u002Fimages\u002Fthumbnails\u002Fbuilding-an-autoscaler-for-docker-swarm\u002Fthumbnail.png","z-YSfLtRDHGY3g0RNQ1OHCalUJvSBOuT5sEtuFG57sU",{"id":10887,"title":10888,"body":10889,"category":2153,"date":11924,"description":11925,"extension":2156,"meta":11926,"navigation":529,"path":1553,"readTime":11927,"seo":11928,"stem":11929,"tags":11930,"thumbnail":11932,"__hash__":11933},"blog\u002Fblog\u002Flgtm-stack\u002Fpart-1.md","Building a Production Monitoring Stack from Scratch, Part 1: Prometheus, Grafana, Node Exporter & AlertManager",{"type":28,"value":10890,"toc":11912},[10891,10902,10904,10908,10911,10914,10917,10920,10922,10926,10929,10983,10994,10997,11003,11010,11012,11016,11023,11061,11071,11074,11081,11228,11295,11306,11308,11315,11321,11335,11341,11358,11364,11366,11370,11373,11376,11381,11390,11395,11404,11414,11439,11445,11454,11463,11473,11475,11479,11482,11485,11698,11712,11715,11786,11796,11798,11805,11815,11818,11867,11874,11883,11886,11888,11892,11895,11898,11901,11904,11907,11909],[31,10892,10893],{},[34,10894,10895,10898,10899],{},[37,10896,10897],{},"Series:"," From NagiosXI to a Modern Observability Stack\n",[37,10900,10901],{},"Part 1 of 4",[53,10903],{},[42,10905,10907],{"id":10906},"the-problem-with-nagiosxi","The Problem with NagiosXI",[34,10909,10910],{},"We had been running NagiosXI for a while. It worked, in the way that something can work while also quietly frustrating everyone who touches it. It checked hosts, fired alerts, and we had even wired up scripts to push notifications to Mattermost. But the gaps were real and getting harder to ignore.",[34,10912,10913],{},"It was a paid solution running on our own infrastructure, a licensing cost that got harder to justify every time someone asked for something it couldn't do. OpenTelemetry support was essentially nonexistent. Application log aggregation wasn't on the table at all. Every extension we had made through plugins had taken us about as far as plugins could go.",[34,10915,10916],{},"The conversation about replacing it had been happening for a while. Eventually it stopped being a conversation and became a project. The goal: a full open-source replacement covering host metrics, alerting, log aggregation, and eventually distributed tracing. One cohesive system instead of a patchwork.",[34,10918,10919],{},"I took on the work. Phase 1 was about standing up the foundation and proving it could actually replace what NagiosXI was doing before we went further.",[53,10921],{},[42,10923,10925],{"id":10924},"the-starting-point","The Starting Point",[34,10927,10928],{},"The first week was spent getting four things working together:",[63,10930,10931,10941],{},[66,10932,10933],{},[69,10934,10935,10938],{},[72,10936,10937],{},"Component",[72,10939,10940],{},"Role",[85,10942,10943,10953,10963,10973],{},[69,10944,10945,10950],{},[90,10946,10947],{},[37,10948,10949],{},"Prometheus",[90,10951,10952],{},"Time-series metrics database and scraping engine",[69,10954,10955,10960],{},[90,10956,10957],{},[37,10958,10959],{},"Grafana",[90,10961,10962],{},"Visualization and dashboarding",[69,10964,10965,10970],{},[90,10966,10967],{},[37,10968,10969],{},"Node Exporter",[90,10971,10972],{},"Host-level metrics (CPU, memory, disk, network)",[69,10974,10975,10980],{},[90,10976,10977],{},[37,10978,10979],{},"AlertManager",[90,10981,10982],{},"Alert routing, grouping, and silencing",[34,10984,10985,10986,10989,10990,10993],{},"The deployment runs across two nodes: ",[37,10987,10988],{},"mon-node-a"," for data collection (Prometheus, AlertManager, and agent-side components) and ",[37,10991,10992],{},"mon-node-b"," for presentation (Grafana). Keeping the presentation layer separate from the data layer was a deliberate decision: if we need to update or rebuild Grafana, it doesn't touch Prometheus, and vice versa. Everything runs in Docker.",[34,10995,10996],{},"How these pieces talk to each other matters, because one architectural choice here (pull vs push) ended up being the central problem in Part 2.",[260,10998,11001],{"className":10999,"code":11000,"language":265},[263],"[ Linux Hosts ]\n      |\n  node_exporter  (runs on each host, exposes \u002Fmetrics on port 9100)\n      |\n      ↓  (pull: Prometheus reaches out every 15s)\n[ Prometheus ]  ←── scrape_configs + alerting_rules\n      |\n      ├──→ [ AlertManager ]\n      |           |\n      |           └──→ Email \u002F Mattermost\n      |\n[ Grafana ]  ←── queries Prometheus via PromQL\n",[98,11002,11000],{"__ignoreMap":266},[34,11004,11005,11006,11009],{},"Prometheus is ",[37,11007,11008],{},"pull-based",". It reaches out to each target on a schedule and pulls metrics. The targets don't know Prometheus exists; they just expose an HTTP endpoint and wait. This distinction ends up mattering a lot.",[53,11011],{},[42,11013,11015],{"id":11014},"getting-host-metrics-in","Getting Host Metrics In",[34,11017,11018,11019,11022],{},"Node Exporter is a lightweight binary that runs on each host and exposes hardware and OS-level metrics at a ",[98,11020,11021],{},"\u002Fmetrics"," HTTP endpoint. Deploy one per machine, point Prometheus at it, done.",[260,11024,11026],{"className":1069,"code":11025,"language":1071,"meta":266,"style":266},"# Verify it's running\ncurl http:\u002F\u002F\u003Chost-ip>:9100\u002Fmetrics | head -50\n",[98,11027,11028,11033],{"__ignoreMap":266},[407,11029,11030],{"class":409,"line":11},[407,11031,11032],{"class":736},"# Verify it's running\n",[407,11034,11035,11038,11041,11043,11046,11048,11050,11053,11055,11058],{"class":409,"line":20},[407,11036,11037],{"class":1078},"curl",[407,11039,11040],{"class":436}," http:\u002F\u002F",[407,11042,3199],{"class":1095},[407,11044,11045],{"class":436},"host-i",[407,11047,34],{"class":1102},[407,11049,1848],{"class":1095},[407,11051,11052],{"class":436},":9100\u002Fmetrics",[407,11054,2003],{"class":1095},[407,11056,11057],{"class":1078}," head",[407,11059,11060],{"class":1088}," -50\n",[34,11062,11063,11064,374,11067,11070],{},"If you see ",[98,11065,11066],{},"# HELP",[98,11068,11069],{},"# TYPE"," blocks followed by metric lines, you're good.",[34,11072,11073],{},"Getting the metrics in wasn't the hard part. The harder part was getting them in cleanly, with enough context attached that alerts and dashboards would actually be useful. A raw IP address as the target label tells you very little when something breaks at 2am.",[34,11075,11076,11077,11080],{},"The solution was file-based service discovery with rich labels. Instead of listing targets directly in ",[98,11078,11079],{},"prometheus.yml",", Prometheus watches a directory of JSON files:",[260,11082,11086],{"className":11083,"code":11084,"language":11085,"meta":266,"style":266},"language-json shiki shiki-themes vitesse-light","[\n  {\n    \"targets\": [\"192.168.0.101:9100\"],\n    \"labels\": {\n      \"hostname\": \"web-server-01\",\n      \"environment\": \"production\",\n      \"location\": \"Primary Rack\",\n      \"maintainers\": \"admin@domain.com\"\n    }\n  }\n]\n","json",[98,11087,11088,11093,11098,11123,11135,11156,11176,11196,11214,11219,11224],{"__ignoreMap":266},[407,11089,11090],{"class":409,"line":11},[407,11091,11092],{"class":416},"[\n",[407,11094,11095],{"class":409,"line":20},[407,11096,11097],{"class":416},"  {\n",[407,11099,11100,11104,11107,11109,11111,11113,11115,11118,11120],{"class":409,"line":427},[407,11101,11103],{"class":11102},"s61at","    \"",[407,11105,11106],{"class":412},"targets",[407,11108,2000],{"class":11102},[407,11110,433],{"class":416},[407,11112,448],{"class":416},[407,11114,2000],{"class":1877},[407,11116,11117],{"class":436},"192.168.0.101:9100",[407,11119,2000],{"class":1877},[407,11121,11122],{"class":416},"],\n",[407,11124,11125,11127,11129,11131,11133],{"class":409,"line":440},[407,11126,11103],{"class":11102},[407,11128,397],{"class":412},[407,11130,2000],{"class":11102},[407,11132,433],{"class":416},[407,11134,2710],{"class":416},[407,11136,11137,11140,11143,11145,11147,11149,11152,11154],{"class":409,"line":457},[407,11138,11139],{"class":11102},"      \"",[407,11141,11142],{"class":412},"hostname",[407,11144,2000],{"class":11102},[407,11146,433],{"class":416},[407,11148,1994],{"class":1877},[407,11150,11151],{"class":436},"web-server-01",[407,11153,2000],{"class":1877},[407,11155,3438],{"class":416},[407,11157,11158,11160,11163,11165,11167,11169,11172,11174],{"class":409,"line":465},[407,11159,11139],{"class":11102},[407,11161,11162],{"class":412},"environment",[407,11164,2000],{"class":11102},[407,11166,433],{"class":416},[407,11168,1994],{"class":1877},[407,11170,11171],{"class":436},"production",[407,11173,2000],{"class":1877},[407,11175,3438],{"class":416},[407,11177,11178,11180,11183,11185,11187,11189,11192,11194],{"class":409,"line":477},[407,11179,11139],{"class":11102},[407,11181,11182],{"class":412},"location",[407,11184,2000],{"class":11102},[407,11186,433],{"class":416},[407,11188,1994],{"class":1877},[407,11190,11191],{"class":436},"Primary Rack",[407,11193,2000],{"class":1877},[407,11195,3438],{"class":416},[407,11197,11198,11200,11203,11205,11207,11209,11212],{"class":409,"line":485},[407,11199,11139],{"class":11102},[407,11201,11202],{"class":412},"maintainers",[407,11204,2000],{"class":11102},[407,11206,433],{"class":416},[407,11208,1994],{"class":1877},[407,11210,11211],{"class":436},"admin@domain.com",[407,11213,2383],{"class":1877},[407,11215,11216],{"class":409,"line":494},[407,11217,11218],{"class":416},"    }\n",[407,11220,11221],{"class":409,"line":502},[407,11222,11223],{"class":416},"  }\n",[407,11225,11226],{"class":409,"line":510},[407,11227,454],{"class":416},[260,11229,11231],{"className":401,"code":11230,"language":403,"meta":266,"style":266},"# prometheus.yml\nscrape_configs:\n  - job_name: \"node_exporter\"\n    file_sd_configs:\n      - files:\n          - \u002Fetc\u002Fprometheus\u002Ftargets\u002F*.json\n        refresh_interval: 30s\n",[98,11232,11233,11238,11245,11262,11269,11278,11286],{"__ignoreMap":266},[407,11234,11235],{"class":409,"line":11},[407,11236,11237],{"class":736},"# prometheus.yml\n",[407,11239,11240,11243],{"class":409,"line":20},[407,11241,11242],{"class":412},"scrape_configs",[407,11244,417],{"class":416},[407,11246,11247,11250,11253,11255,11257,11260],{"class":409,"line":427},[407,11248,11249],{"class":416},"  -",[407,11251,11252],{"class":412}," job_name",[407,11254,433],{"class":416},[407,11256,1994],{"class":1877},[407,11258,11259],{"class":436},"node_exporter",[407,11261,2383],{"class":1877},[407,11263,11264,11267],{"class":409,"line":440},[407,11265,11266],{"class":412},"    file_sd_configs",[407,11268,417],{"class":416},[407,11270,11271,11273,11276],{"class":409,"line":457},[407,11272,1039],{"class":416},[407,11274,11275],{"class":412}," files",[407,11277,417],{"class":416},[407,11279,11280,11283],{"class":409,"line":465},[407,11281,11282],{"class":416},"          -",[407,11284,11285],{"class":436}," \u002Fetc\u002Fprometheus\u002Ftargets\u002F*.json\n",[407,11287,11288,11291,11293],{"class":409,"line":477},[407,11289,11290],{"class":412},"        refresh_interval",[407,11292,433],{"class":416},[407,11294,1706],{"class":436},[34,11296,11297,11298,11301,11302,11305],{},"Drop a file in, get a monitored host within 30 seconds. No reload required. The labels on each target flow through to every metric scraped from that host, which means they're available in alert annotations, in Grafana, everywhere. When ",[98,11299,11300],{},"HostDown"," fires, the alert can say ",[154,11303,11304],{},"which"," host, in which environment, and who to contact. That's the payoff.",[53,11307],{},[42,11309,7505,11311,11314],{"id":11310},"the-up-metric",[98,11312,11313],{},"up"," Metric",[34,11316,11317,11318,11320],{},"One of Prometheus's built-in synthetic metrics is ",[98,11319,11313],{},". For every scrape target:",[147,11322,11323,11329],{},[150,11324,11325,11328],{},[98,11326,11327],{},"up = 1",": scrape succeeded",[150,11330,11331,11334],{},[98,11332,11333],{},"up = 0",": scrape failed",[34,11336,11337,11338,11340],{},"This is the most fundamental health signal in the stack. Everything else (CPU, memory, disk) is meaningless if you can't even reach the host. And because ",[98,11339,11313],{}," carries all the labels from your target file, you can immediately see which host is down, in which environment.",[260,11342,11346],{"className":11343,"code":11344,"language":11345,"meta":266,"style":266},"language-promql shiki shiki-themes vitesse-light","# All down hosts right now\nup{job=\"node_exporter\"} == 0\n","promql",[98,11347,11348,11353],{"__ignoreMap":266},[407,11349,11350],{"class":409,"line":11},[407,11351,11352],{},"# All down hosts right now\n",[407,11354,11355],{"class":409,"line":20},[407,11356,11357],{},"up{job=\"node_exporter\"} == 0\n",[34,11359,11360,11361,11363],{},"I keep coming back to ",[98,11362,11313],{}," throughout this series because it's also where things can silently break if you change the architecture carelessly. More on that in Part 2.",[53,11365],{},[42,11367,11369],{"id":11368},"dashboards","Dashboards",[34,11371,11372],{},"Grafana connects to Prometheus as a data source and queries it via PromQL. The community dashboards are easy to import and useful for getting started, but building your own is worth doing because it forces you to understand exactly what you're looking at.",[34,11374,11375],{},"The core panels and the queries behind them:",[34,11377,11378],{},[37,11379,11380],{},"CPU Usage (%)",[260,11382,11384],{"className":11343,"code":11383,"language":11345,"meta":266,"style":266},"100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)\n",[98,11385,11386],{"__ignoreMap":266},[407,11387,11388],{"class":409,"line":11},[407,11389,11383],{},[34,11391,11392],{},[37,11393,11394],{},"Memory Usage (%)",[260,11396,11398],{"className":11343,"code":11397,"language":11345,"meta":266,"style":266},"(1 - (node_memory_MemAvailable_bytes \u002F node_memory_MemTotal_bytes)) * 100\n",[98,11399,11400],{"__ignoreMap":266},[407,11401,11402],{"class":409,"line":11},[407,11403,11397],{},[34,11405,11406,11409,11410,11413],{},[37,11407,11408],{},"Disk Usage (%)",": the ",[98,11411,11412],{},"fstype"," filter excludes Docker overlays and tmpfs mounts that inflate results",[260,11415,11417],{"className":11343,"code":11416,"language":11345,"meta":266,"style":266},"(1 - (\n  node_filesystem_avail_bytes{fstype!~\"tmpfs|overlay\"} \u002F\n  node_filesystem_size_bytes{fstype!~\"tmpfs|overlay\"}\n)) * 100\n",[98,11418,11419,11424,11429,11434],{"__ignoreMap":266},[407,11420,11421],{"class":409,"line":11},[407,11422,11423],{},"(1 - (\n",[407,11425,11426],{"class":409,"line":20},[407,11427,11428],{},"  node_filesystem_avail_bytes{fstype!~\"tmpfs|overlay\"} \u002F\n",[407,11430,11431],{"class":409,"line":427},[407,11432,11433],{},"  node_filesystem_size_bytes{fstype!~\"tmpfs|overlay\"}\n",[407,11435,11436],{"class":409,"line":440},[407,11437,11438],{},")) * 100\n",[34,11440,11441,11444],{},[37,11442,11443],{},"Fleet status",": a stat panel showing every host's current state",[260,11446,11448],{"className":11343,"code":11447,"language":11345,"meta":266,"style":266},"up{job=\"node_exporter\"}\n",[98,11449,11450],{"__ignoreMap":266},[407,11451,11452],{"class":409,"line":11},[407,11453,11447],{},[34,11455,11456,11457,11459,11460,11462],{},"Value mappings: ",[98,11458,3286],{}," → 🟢 UP, ",[98,11461,2889],{}," → 🔴 DOWN.",[34,11464,11465,11466,1277,11469,11472],{},"Adding a dashboard variable for ",[98,11467,11468],{},"instance",[98,11470,11471],{},"label_values(up{job=\"node_exporter\"}, instance)",") gives you a dropdown to filter to a specific host or view the whole fleet. That one change makes the dashboard genuinely useful for day-to-day operations.",[53,11474],{},[42,11476,11478],{"id":11477},"alerting","Alerting",[34,11480,11481],{},"Prometheus evaluates alerting rules and forwards firing alerts to AlertManager. AlertManager handles the business logic: who gets notified, when, how often, and what to suppress.",[34,11483,11484],{},"The rules themselves live in separate YAML files:",[260,11486,11488],{"className":401,"code":11487,"language":403,"meta":266,"style":266},"groups:\n  - name: node_exporter_alerts\n    rules:\n\n      - alert: HostDown\n        expr: up{job=\"node_exporter\"} == 0\n        for: 2m\n        labels:\n          severity: critical\n        annotations:\n          summary: \"Host {{ $labels.instance }} is down\"\n          description: >\n            {{ $labels.hostname }} has been unreachable for more than 2 minutes.\n            Maintainers: {{ $labels.maintainers }}\n\n      - alert: HighCPUUsage\n        expr: >\n          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100) > 85\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"High CPU on {{ $labels.instance }}\"\n          description: >\n            CPU on {{ $labels.hostname }} has been above 85% for 5 minutes.\n            Current: {{ $value | printf \"%.1f\" }}%\n",[98,11489,11490,11497,11508,11515,11519,11531,11541,11551,11558,11568,11575,11589,11599,11604,11609,11613,11624,11632,11637,11646,11652,11661,11667,11680,11688,11693],{"__ignoreMap":266},[407,11491,11492,11495],{"class":409,"line":11},[407,11493,11494],{"class":412},"groups",[407,11496,417],{"class":416},[407,11498,11499,11501,11503,11505],{"class":409,"line":20},[407,11500,11249],{"class":416},[407,11502,832],{"class":412},[407,11504,433],{"class":416},[407,11506,11507],{"class":436}," node_exporter_alerts\n",[407,11509,11510,11513],{"class":409,"line":427},[407,11511,11512],{"class":412},"    rules",[407,11514,417],{"class":416},[407,11516,11517],{"class":409,"line":440},[407,11518,530],{"emptyLinePlaceholder":529},[407,11520,11521,11523,11526,11528],{"class":409,"line":457},[407,11522,1039],{"class":416},[407,11524,11525],{"class":412}," alert",[407,11527,433],{"class":416},[407,11529,11530],{"class":436}," HostDown\n",[407,11532,11533,11536,11538],{"class":409,"line":465},[407,11534,11535],{"class":412},"        expr",[407,11537,433],{"class":416},[407,11539,11540],{"class":436}," up{job=\"node_exporter\"} == 0\n",[407,11542,11543,11546,11548],{"class":409,"line":477},[407,11544,11545],{"class":412},"        for",[407,11547,433],{"class":416},[407,11549,11550],{"class":436}," 2m\n",[407,11552,11553,11556],{"class":409,"line":485},[407,11554,11555],{"class":412},"        labels",[407,11557,417],{"class":416},[407,11559,11560,11563,11565],{"class":409,"line":494},[407,11561,11562],{"class":412},"          severity",[407,11564,433],{"class":416},[407,11566,11567],{"class":436}," critical\n",[407,11569,11570,11573],{"class":409,"line":502},[407,11571,11572],{"class":412},"        annotations",[407,11574,417],{"class":416},[407,11576,11577,11580,11582,11584,11587],{"class":409,"line":510},[407,11578,11579],{"class":412},"          summary",[407,11581,433],{"class":416},[407,11583,1994],{"class":1877},[407,11585,11586],{"class":436},"Host {{ $labels.instance }} is down",[407,11588,2383],{"class":1877},[407,11590,11591,11594,11596],{"class":409,"line":518},[407,11592,11593],{"class":412},"          description",[407,11595,433],{"class":416},[407,11597,11598],{"class":557}," >\n",[407,11600,11601],{"class":409,"line":526},[407,11602,11603],{"class":436},"            {{ $labels.hostname }} has been unreachable for more than 2 minutes.\n",[407,11605,11606],{"class":409,"line":533},[407,11607,11608],{"class":436},"            Maintainers: {{ $labels.maintainers }}\n",[407,11610,11611],{"class":409,"line":541},[407,11612,530],{"emptyLinePlaceholder":529},[407,11614,11615,11617,11619,11621],{"class":409,"line":549},[407,11616,1039],{"class":416},[407,11618,11525],{"class":412},[407,11620,433],{"class":416},[407,11622,11623],{"class":436}," HighCPUUsage\n",[407,11625,11626,11628,11630],{"class":409,"line":561},[407,11627,11535],{"class":412},[407,11629,433],{"class":416},[407,11631,11598],{"class":557},[407,11633,11634],{"class":409,"line":2058},[407,11635,11636],{"class":436},"          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100) > 85\n",[407,11638,11639,11641,11643],{"class":409,"line":2079},[407,11640,11545],{"class":412},[407,11642,433],{"class":416},[407,11644,11645],{"class":436}," 5m\n",[407,11647,11648,11650],{"class":409,"line":2084},[407,11649,11555],{"class":412},[407,11651,417],{"class":416},[407,11653,11654,11656,11658],{"class":409,"line":2090},[407,11655,11562],{"class":412},[407,11657,433],{"class":416},[407,11659,11660],{"class":436}," warning\n",[407,11662,11663,11665],{"class":409,"line":2108},[407,11664,11572],{"class":412},[407,11666,417],{"class":416},[407,11668,11669,11671,11673,11675,11678],{"class":409,"line":2858},[407,11670,11579],{"class":412},[407,11672,433],{"class":416},[407,11674,1994],{"class":1877},[407,11676,11677],{"class":436},"High CPU on {{ $labels.instance }}",[407,11679,2383],{"class":1877},[407,11681,11682,11684,11686],{"class":409,"line":2869},[407,11683,11593],{"class":412},[407,11685,433],{"class":416},[407,11687,11598],{"class":557},[407,11689,11690],{"class":409,"line":2909},[407,11691,11692],{"class":436},"            CPU on {{ $labels.hostname }} has been above 85% for 5 minutes.\n",[407,11694,11695],{"class":409,"line":2956},[407,11696,11697],{"class":436},"            Current: {{ $value | printf \"%.1f\" }}%\n",[34,11699,7505,11700,11703,11704,11706,11707,1462,11709,11711],{},[98,11701,11702],{},"for: 2m"," on ",[98,11705,11300],{}," absorbs brief network glitches. Without it, a momentary scrape failure sends an alert. The rich labels on the target (",[98,11708,11142],{},[98,11710,11202],{},") show up directly in the alert annotations.",[34,11713,11714],{},"One AlertManager config worth explaining is the inhibit rule:",[260,11716,11718],{"className":401,"code":11717,"language":403,"meta":266,"style":266},"inhibit_rules:\n  - source_match:\n      alertname: \"HostDown\"\n    target_match_re:\n      alertname: \"HighCPUUsage|HighMemoryUsage|DiskSpaceLow\"\n    equal: [\"instance\"]\n",[98,11719,11720,11727,11736,11749,11756,11769],{"__ignoreMap":266},[407,11721,11722,11725],{"class":409,"line":11},[407,11723,11724],{"class":412},"inhibit_rules",[407,11726,417],{"class":416},[407,11728,11729,11731,11734],{"class":409,"line":20},[407,11730,11249],{"class":416},[407,11732,11733],{"class":412}," source_match",[407,11735,417],{"class":416},[407,11737,11738,11741,11743,11745,11747],{"class":409,"line":427},[407,11739,11740],{"class":412},"      alertname",[407,11742,433],{"class":416},[407,11744,1994],{"class":1877},[407,11746,11300],{"class":436},[407,11748,2383],{"class":1877},[407,11750,11751,11754],{"class":409,"line":440},[407,11752,11753],{"class":412},"    target_match_re",[407,11755,417],{"class":416},[407,11757,11758,11760,11762,11764,11767],{"class":409,"line":457},[407,11759,11740],{"class":412},[407,11761,433],{"class":416},[407,11763,1994],{"class":1877},[407,11765,11766],{"class":436},"HighCPUUsage|HighMemoryUsage|DiskSpaceLow",[407,11768,2383],{"class":1877},[407,11770,11771,11774,11776,11778,11780,11782,11784],{"class":409,"line":465},[407,11772,11773],{"class":412},"    equal",[407,11775,433],{"class":416},[407,11777,448],{"class":416},[407,11779,2000],{"class":1877},[407,11781,11468],{"class":436},[407,11783,2000],{"class":1877},[407,11785,454],{"class":416},[34,11787,11788,11789,11791,11792,11795],{},"When ",[98,11790,11300],{}," fires for a host, AlertManager suppresses all other alerts for that same host. There's no useful signal in a ",[98,11793,11794],{},"HighMemoryUsage"," alert for a machine that isn't reachable. Without this, a single dead host can generate a cascade of noise.",[53,11797],{},[42,11799,7505,11801,11804],{"id":11800},"the-last_seen-pattern",[98,11802,11803],{},"last_seen"," Pattern",[34,11806,11807,11808,11811,11812,11814],{},"When a host disappears completely, Prometheus eventually stops having active series data for it. ",[98,11809,11810],{},"up{instance=\"...\"}"," doesn't return ",[98,11813,2889],{},": it returns nothing, because there's no scrape happening. You lose the ability to answer \"when did this thing last check in?\"",[34,11816,11817],{},"A recording rule fixes this by continuously writing a timestamp whenever a host is up:",[260,11819,11821],{"className":401,"code":11820,"language":403,"meta":266,"style":266},"groups:\n  - name: recording_rules\n    rules:\n      - record: node_last_seen_timestamp\n        expr: time() * up{job=\"node_exporter\"}\n",[98,11822,11823,11829,11840,11846,11858],{"__ignoreMap":266},[407,11824,11825,11827],{"class":409,"line":11},[407,11826,11494],{"class":412},[407,11828,417],{"class":416},[407,11830,11831,11833,11835,11837],{"class":409,"line":20},[407,11832,11249],{"class":416},[407,11834,832],{"class":412},[407,11836,433],{"class":416},[407,11838,11839],{"class":436}," recording_rules\n",[407,11841,11842,11844],{"class":409,"line":427},[407,11843,11512],{"class":412},[407,11845,417],{"class":416},[407,11847,11848,11850,11853,11855],{"class":409,"line":440},[407,11849,1039],{"class":416},[407,11851,11852],{"class":412}," record",[407,11854,433],{"class":416},[407,11856,11857],{"class":436}," node_last_seen_timestamp\n",[407,11859,11860,11862,11864],{"class":409,"line":457},[407,11861,11535],{"class":412},[407,11863,433],{"class":416},[407,11865,11866],{"class":436}," time() * up{job=\"node_exporter\"}\n",[34,11868,11869,11870,11873],{},"This writes the current Unix timestamp on every evaluation cycle, but only when ",[98,11871,11872],{},"up == 1",". When a host goes dark, the last written value persists in storage. In Grafana:",[260,11875,11877],{"className":11343,"code":11876,"language":11345,"meta":266,"style":266},"time() - node_last_seen_timestamp\n",[98,11878,11879],{"__ignoreMap":266},[407,11880,11881],{"class":409,"line":11},[407,11882,11876],{},[34,11884,11885],{},"Format as duration and you get: \"last seen 3h 22m ago.\" It's a small thing but it's become one of the most-used panels.",[53,11887],{},[42,11889,11891],{"id":11890},"where-this-left-off","Where This Left Off",[34,11893,11894],{},"By the end of the first week, the stack was functionally replacing NagiosXI for host monitoring. Prometheus scraping every host every 15 seconds, dashboards showing the fleet, AlertManager routing alerts with inhibit rules and deduplication, recording rules keeping last-seen timestamps for hosts that went dark.",[34,11896,11897],{},"But there was a question I hadn't resolved yet.",[34,11899,11900],{},"Node Exporter is a single-purpose binary: host metrics and nothing else. The moment we wanted logs or traces from these same hosts, we'd need additional agents running alongside it. And adding a host to monitoring still meant four manual steps: SSH in, install Node Exporter, write the target file, reload Prometheus.",[34,11902,11903],{},"My colleague had been working in parallel, exploring the multiple-exporter approach: a separate binary for each signal type. I'd been looking at Grafana Alloy, which promised a single agent that could handle all of it. We hadn't converged yet, and there were real questions about whether Alloy was ready enough to build on.",[34,11905,11906],{},"That's what Part 2 is about.",[1423,11908],{"path":1559},[2122,11910,11911],{},"html pre.shiki code .sYZai, html code.shiki .sYZai{--shiki-default:#999999}html pre.shiki code .s61at, html code.shiki .s61at{--shiki-default:#99841877}html pre.shiki code .su6XF, html code.shiki .su6XF{--shiki-default:#998418}html pre.shiki code .sSP4y, html code.shiki .sSP4y{--shiki-default:#B5695977}html pre.shiki code .spphp, html code.shiki .spphp{--shiki-default:#B56959}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s8zF2, html code.shiki .s8zF2{--shiki-default:#A0ADA0}html pre.shiki code .sbBg2, html code.shiki .sbBg2{--shiki-default:#1E754F}html pre.shiki code .sySUi, html code.shiki .sySUi{--shiki-default:#59873A}html pre.shiki code .si04Y, html code.shiki .si04Y{--shiki-default:#AB5959}html pre.shiki code .suHK_, html code.shiki .suHK_{--shiki-default:#393A34}html pre.shiki code .sEi1f, html code.shiki .sEi1f{--shiki-default:#A65E2B}",{"title":266,"searchDepth":20,"depth":20,"links":11913},[11914,11915,11916,11917,11919,11920,11921,11923],{"id":10906,"depth":20,"text":10907},{"id":10924,"depth":20,"text":10925},{"id":11014,"depth":20,"text":11015},{"id":11310,"depth":20,"text":11918},"The up Metric",{"id":11368,"depth":20,"text":11369},{"id":11477,"depth":20,"text":11478},{"id":11800,"depth":20,"text":11922},"The last_seen Pattern",{"id":11890,"depth":20,"text":11891},"2025-01-06","How we migrated from NagiosXI to a modern open-source observability stack, and why getting the foundation right mattered more than I expected.",{},"10 min",{"title":10888,"description":11925},"blog\u002Flgtm-stack\u002Fpart-1",[11931,10959,10949,10979,2164],"Monitoring","\u002Fimages\u002Fthumbnails\u002Flgtm-stack\u002Fpart-1.png","CmT-CKEtQmQSsbi9gf40VyuVdIpBxFfMaNYLuRC23aU",1792163584716]