[{"data":1,"prerenderedAt":8844},["ShallowReactive",2],{"nav-stories":3,"blog-building-an-autoscaler-for-docker-swarm":24},[4,15],{"id":5,"color":6,"extension":7,"image":8,"label":9,"link":6,"meta":10,"order":11,"stem":12,"text":13,"__hash__":14},"stories\u002Fstories\u002F01-traffic.yml",null,"yml","\u002Fimages\u002Fstories\u002Fapp-traffic.png","TRAFFIC",{},1,"stories\u002F01-traffic","74.1 million requests in a single day, peaking at about 8,000 requests per second, with 97.3% of them successful.","ZAWpOSlORUptBXwG92wUBweOVfkhWbeLh2pEzmDrejU",{"id":16,"color":17,"extension":7,"image":6,"label":18,"link":6,"meta":19,"order":20,"stem":21,"text":22,"__hash__":23},"stories\u002Fstories\u002F02-swarmctl.yml","#1d1d1f","SWARMCTL",{},2,"stories\u002F02-swarmctl","swarmctl is the autoscaler I built for Docker Swarm. It scales services on CPU, memory and traffic, and asks OpenStack for new VMs when the cluster is full. Should I open-source it? Tell me what you think. \u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm","e1bgTtDeqxwbYkPP3QR4l79ZvA47Oljb4dfnbh43Ftw",{"id":25,"title":26,"body":27,"category":8827,"date":8828,"description":8829,"extension":8830,"meta":8831,"navigation":425,"path":8832,"readTime":8833,"seo":8834,"stem":8835,"tags":8836,"thumbnail":8842,"__hash__":8843},"blog\u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm\u002Findex.md","Building an Autoscaler for Docker Swarm: Inside swarmctl",{"type":28,"value":29,"toc":8783},"minimark",[30,52,58,63,70,76,83,99,106,131,134,166,169,172,175,177,181,192,195,206,208,212,219,224,227,316,320,327,911,914,928,932,935,943,956,971,978,982,985,1007,1014,1167,1185,1189,1192,1319,1323,1333,1419,1426,1430,1449,1458,1469,1475,1713,1715,1719,1722,1743,1746,1750,1753,1759,2198,2201,2237,2441,2445,2448,2592,2602,2605,2610,2613,2789,2792,2796,2799,2886,2893,2961,2975,2981,2985,2988,3108,3112,3123,3130,3364,3370,3432,3435,3590,3605,3608,3611,3614,3621,3628,3728,3732,3738,3866,3870,3877,3881,4072,4075,4103,4107,4162,4166,4169,4404,4408,4414,4418,4425,4432,4442,4535,4549,4620,4630,4716,4722,4725,4728,4732,4743,4759,4769,4771,4775,4778,4782,4886,4890,4896,5028,5032,5183,5187,5194,5441,5463,5467,5470,5472,5476,5483,5671,5678,5682,5985,5988,5992,5999,6118,6131,6135,6141,6217,6224,6228,6235,6258,6417,6421,6432,6606,6610,6617,6621,6632,6634,6638,6651,6655,6665,6931,6950,6956,7216,7226,7230,7240,7399,7404,7691,7694,7796,7800,7810,8264,8275,8278,8281,8291,8300,8471,8475,8478,8486,8489,8491,8495,8498,8504,8510,8516,8528,8534,8544,8546,8550,8553,8726,8732,8738,8740,8744,8765,8779],[31,32,33],"blockquote",{},[34,35,36,40,41,46,47,51],"p",{},[37,38,39],"strong",{},"Series note:"," this is the deep dive behind the autoscaling section of ",[42,43,45],"a",{"href":44},"\u002Fblog\u002Frunning-production-swarm","Running Production Swarm",", which covers the whole architecture it plugs into. You do not need to read that first, but it explains where ",[48,49,50],"code",{},"swarmctl"," sits: on a manager node, reading cAdvisor and Traefik metrics, scaling services that opt in through labels.",[34,53,54],{},[55,56,57],"em",{},"A deep dive into swarmctl: a lightweight daemon that watches Swarm services, decides when to scale, executes the change, and coordinates node-level scaling with OpenStack.",[59,60,62],"h2",{"id":61},"why-build-this-at-all","Why build this at all?",[34,64,65,66,69],{},"Kubernetes gives you the Horizontal Pod Autoscaler, the Cluster Autoscaler, and a metrics\npipeline out of the box. Docker Swarm gives you ",[48,67,68],{},"docker service scale \u003Cname> \u003Cn>"," and nothing else.\nNo reactive scaling, no load-aware placement, no graceful node drain before a VM is terminated.",[34,71,72,73,75],{},"If you are running Swarm in production (and plenty of people still are, because it is simple,\nstable, and fits on three small VMs without a control-plane tax), you have to build that layer\nyourself. ",[48,74,50],{}," is the layer I built.",[34,77,78,79,82],{},"It does ",[37,80,81],{},"not"," try to be Kubernetes. It is explicitly not:",[84,85,86,93,96],"ul",{},[87,88,89,90],"li",{},"a replacement for ",[48,91,92],{},"docker stack deploy",[87,94,95],{},"a full observability stack (no Prometheus, no Grafana)",[87,97,98],{},"a node provisioner (OpenStack's autoscaling group handles that)",[34,100,101,102,105],{},"It ",[55,103,104],{},"is"," a daemon that:",[107,108,109,112,115,122,125,128],"ol",{},[87,110,111],{},"collects per-service metrics every cycle (CPU, memory, CPU throttling, request rate, latency, errors, connections, pending\u002Frejected task counts),",[87,113,114],{},"resolves a scaling policy per service from labels, templates, and defaults,",[87,116,117,118,121],{},"feeds metrics + policy into a ",[37,119,120],{},"pure decision engine",",",[87,123,124],{},"executes the resulting decision through the Docker API with verification and partial-success handling,",[87,126,127],{},"signals OpenStack to add or remove whole VMs when the cluster itself runs out of room,",[87,129,130],{},"drains nodes gracefully before OpenStack terminates them.",[34,132,133],{},"In this post I walk through the whole thing, with the real code. There are five parts worth\nunderstanding in depth:",[84,135,136,142,148,154,160],{},[87,137,138,141],{},[37,139,140],{},"The collector",": turning raw cAdvisor and Traefik numbers into clean per-service metrics.",[87,143,144,147],{},[37,145,146],{},"The policy layer",": resolver (config → policy) and the engine (metrics + policy → decision).",[87,149,150,153],{},[37,151,152],{},"The executor",": making a decision real, safely.",[87,155,156,159],{},[37,157,158],{},"The watch loop",": the stateful orchestrator that ties it all together.",[87,161,162,165],{},[37,163,164],{},"The OpenStack integration",": scale-out signals, scale-in approval, and graceful drain.",[167,168],"hr",{},[34,170,171],{},"Before the code, here is the control loop in miniature. This is a simplified model, not swarmctl itself: watch how startup lag, cooldowns, and a minimum replica count shape what an autoscaler does with a traffic spike. The rest of the post is about the guards that make the real one behave this way.",[173,174],"swarm-autoscale",{},[167,176],{},[59,178,180],{"id":179},"architecture-at-a-glance","Architecture at a glance",[34,182,183,184,187,188,191],{},"There are three paths through swarmctl, and the diagram below follows each one: the watch-loop cycle that does most of the work, OpenStack asking permission to scale in, and the drain that runs just before a VM is terminated. Press ",[37,185,186],{},"Play"," to watch a path, or ",[37,189,190],{},"Step"," to go one stage at a time. Each stage shows the code that does the work, and you can click any box to jump to the step where it is used.",[193,194],"swarmctl-architecture",{},[34,196,197,198,201,202,205],{},"Two concurrency boundaries matter: the ",[37,199,200],{},"watch loop"," runs on the main thread on a fixed\ninterval, and the ",[37,203,204],{},"hook server"," runs on a daemon thread so OpenStack can call in at any time.\nThey share a small amount of state (last scale-out timestamp per pool) through a lock-protected\nmodule-level dict.",[167,207],{},[59,209,211],{"id":210},"part-1-the-collector","Part 1: The Collector",[34,213,214,215,218],{},"The engine is only as good as its inputs. The collector's job is to produce one clean\n",[48,216,217],{},"ServiceMetrics"," object per enabled service, every cycle, without ever blocking the loop\nfor longer than one interval.",[220,221,223],"h3",{"id":222},"only-manage-what-opts-in","Only manage what opts in",[34,225,226],{},"Services opt in with a Swarm label. Nothing is scaled unless it explicitly asks to be:",[228,229,234],"pre",{"className":230,"code":231,"language":232,"meta":233,"style":233},"language-yaml shiki shiki-themes vitesse-light","deploy:\n  labels:\n    - \"swarmctl.enable=true\"\n    - \"swarmctl.policy=blazor-standard\"   # named template\n    - \"swarmctl.max_replicas=8\"           # override any single field\n    - \"swarmctl.scale_up_cpu_avg=60\"\n","yaml","",[48,235,236,248,255,272,289,304],{"__ignoreMap":233},[237,238,240,244],"span",{"class":239,"line":11},"line",[237,241,243],{"class":242},"su6XF","deploy",[237,245,247],{"class":246},"sYZai",":\n",[237,249,250,253],{"class":239,"line":20},[237,251,252],{"class":242},"  labels",[237,254,247],{"class":246},[237,256,258,261,265,269],{"class":239,"line":257},3,[237,259,260],{"class":246},"    -",[237,262,264],{"class":263},"sSP4y"," \"",[237,266,268],{"class":267},"spphp","swarmctl.enable=true",[237,270,271],{"class":263},"\"\n",[237,273,275,277,279,282,285],{"class":239,"line":274},4,[237,276,260],{"class":246},[237,278,264],{"class":263},[237,280,281],{"class":267},"swarmctl.policy=blazor-standard",[237,283,284],{"class":263},"\"",[237,286,288],{"class":287},"s8zF2","   # named template\n",[237,290,292,294,296,299,301],{"class":239,"line":291},5,[237,293,260],{"class":246},[237,295,264],{"class":263},[237,297,298],{"class":267},"swarmctl.max_replicas=8",[237,300,284],{"class":263},[237,302,303],{"class":287},"           # override any single field\n",[237,305,307,309,311,314],{"class":239,"line":306},6,[237,308,260],{"class":246},[237,310,264],{"class":263},[237,312,313],{"class":267},"swarmctl.scale_up_cpu_avg=60",[237,315,271],{"class":263},[220,317,319],{"id":318},"concurrent-collection-with-a-hard-deadline","Concurrent collection with a hard deadline",[34,321,322,323,326],{},"Collecting stats for a service means hitting cAdvisor on whichever node each task runs on.\nDone serially, total time is ",[48,324,325],{},"O(n_services × latency)",". The collector fans out across a\nthread pool and enforces a deadline so one slow service can never stall the whole loop:",[228,328,332],{"className":329,"code":330,"language":331,"meta":233,"style":233},"language-python shiki shiki-themes vitesse-light","def collect_all(self) -> List[Tuple[ServiceMetrics, Dict[str, str]]]:\n    s = get_settings()\n    self._docker.refresh_infra_maps()\n\n    enabled = self._docker.list_enabled_services()\n    if not enabled:\n        return []\n\n    # Scrape Traefik ONCE per cycle. Calling get_metrics() per service would reset\n    # the rate-state on every call and give zero deltas for all but the first service.\n    traffic_cache = self._docker.traefik.get_metrics()\n    deadline = time.monotonic() + s.collection_timeout\n\n    with ThreadPoolExecutor(max_workers=min(len(enabled), 16),\n                            thread_name_prefix=\"collect\") as pool:\n        future_to_svc = {\n            pool.submit(self._docker.get_service_metrics, svc.id, traffic_cache): (svc, labels)\n            for svc, labels in enabled\n        }\n        remaining = max(0.1, deadline - time.monotonic())\n        done, not_done = _wait(future_to_svc, timeout=remaining)\n\n        if not_done:\n            skipped = [future_to_svc[f][0].name for f in not_done]\n            logger.warning(f\"Collection deadline: {len(not_done)} skipped this cycle: {skipped}\")\n            for fut in not_done:\n                fut.cancel()\n        ...\n","python",[48,333,334,388,402,421,427,448,462,471,476,482,488,514,544,549,589,614,625,681,700,706,739,772,777,788,831,878,892,905],{"__ignoreMap":233},[237,335,336,340,344,347,351,354,357,360,363,366,368,370,372,375,377,380,382,385],{"class":239,"line":11},[237,337,339],{"class":338},"si04Y","def",[237,341,343],{"class":342},"sySUi"," collect_all",[237,345,346],{"class":246},"(",[237,348,350],{"class":349},"suHK_","self",[237,352,353],{"class":246},")",[237,355,356],{"class":246}," ->",[237,358,359],{"class":349}," List",[237,361,362],{"class":246},"[",[237,364,365],{"class":349},"Tuple",[237,367,362],{"class":246},[237,369,217],{"class":349},[237,371,121],{"class":246},[237,373,374],{"class":349}," Dict",[237,376,362],{"class":246},[237,378,379],{"class":242},"str",[237,381,121],{"class":246},[237,383,384],{"class":242}," str",[237,386,387],{"class":246},"]]]:\n",[237,389,390,393,396,399],{"class":239,"line":20},[237,391,392],{"class":349},"    s ",[237,394,395],{"class":246},"=",[237,397,398],{"class":349}," get_settings",[237,400,401],{"class":246},"()\n",[237,403,404,408,411,414,416,419],{"class":239,"line":257},[237,405,407],{"class":406},"sEi1f","    self",[237,409,410],{"class":246},".",[237,412,413],{"class":349},"_docker",[237,415,410],{"class":246},[237,417,418],{"class":349},"refresh_infra_maps",[237,420,401],{"class":246},[237,422,423],{"class":239,"line":274},[237,424,426],{"emptyLinePlaceholder":425},true,"\n",[237,428,429,432,434,437,439,441,443,446],{"class":239,"line":291},[237,430,431],{"class":349},"    enabled ",[237,433,395],{"class":246},[237,435,436],{"class":406}," self",[237,438,410],{"class":246},[237,440,413],{"class":349},[237,442,410],{"class":246},[237,444,445],{"class":349},"list_enabled_services",[237,447,401],{"class":246},[237,449,450,454,457,460],{"class":239,"line":306},[237,451,453],{"class":452},"sbBg2","    if",[237,455,456],{"class":338}," not",[237,458,459],{"class":349}," enabled",[237,461,247],{"class":246},[237,463,465,468],{"class":239,"line":464},7,[237,466,467],{"class":452},"        return",[237,469,470],{"class":246}," []\n",[237,472,474],{"class":239,"line":473},8,[237,475,426],{"emptyLinePlaceholder":425},[237,477,479],{"class":239,"line":478},9,[237,480,481],{"class":287},"    # Scrape Traefik ONCE per cycle. Calling get_metrics() per service would reset\n",[237,483,485],{"class":239,"line":484},10,[237,486,487],{"class":287},"    # the rate-state on every call and give zero deltas for all but the first service.\n",[237,489,491,494,496,498,500,502,504,507,509,512],{"class":239,"line":490},11,[237,492,493],{"class":349},"    traffic_cache ",[237,495,395],{"class":246},[237,497,436],{"class":406},[237,499,410],{"class":246},[237,501,413],{"class":349},[237,503,410],{"class":246},[237,505,506],{"class":349},"traefik",[237,508,410],{"class":246},[237,510,511],{"class":349},"get_metrics",[237,513,401],{"class":246},[237,515,517,520,522,525,527,530,533,536,539,541],{"class":239,"line":516},12,[237,518,519],{"class":349},"    deadline ",[237,521,395],{"class":246},[237,523,524],{"class":349}," time",[237,526,410],{"class":246},[237,528,529],{"class":349},"monotonic",[237,531,532],{"class":246},"()",[237,534,535],{"class":338}," +",[237,537,538],{"class":349}," s",[237,540,410],{"class":246},[237,542,543],{"class":349},"collection_timeout\n",[237,545,547],{"class":239,"line":546},13,[237,548,426],{"emptyLinePlaceholder":425},[237,550,552,555,558,560,564,566,569,571,574,576,579,582,586],{"class":239,"line":551},14,[237,553,554],{"class":452},"    with",[237,556,557],{"class":349}," ThreadPoolExecutor",[237,559,346],{"class":246},[237,561,563],{"class":562},"svycV","max_workers",[237,565,395],{"class":246},[237,567,568],{"class":242},"min",[237,570,346],{"class":246},[237,572,573],{"class":242},"len",[237,575,346],{"class":246},[237,577,578],{"class":349},"enabled",[237,580,581],{"class":246},"),",[237,583,585],{"class":584},"s-TwI"," 16",[237,587,588],{"class":246},"),\n",[237,590,592,595,597,599,602,604,606,609,612],{"class":239,"line":591},15,[237,593,594],{"class":562},"                            thread_name_prefix",[237,596,395],{"class":246},[237,598,284],{"class":263},[237,600,601],{"class":267},"collect",[237,603,284],{"class":263},[237,605,353],{"class":246},[237,607,608],{"class":452}," as",[237,610,611],{"class":349}," pool",[237,613,247],{"class":246},[237,615,617,620,622],{"class":239,"line":616},16,[237,618,619],{"class":349},"        future_to_svc ",[237,621,395],{"class":246},[237,623,624],{"class":246}," {\n",[237,626,628,631,633,636,638,640,642,644,646,649,651,654,656,659,661,664,667,670,673,675,678],{"class":239,"line":627},17,[237,629,630],{"class":349},"            pool",[237,632,410],{"class":246},[237,634,635],{"class":349},"submit",[237,637,346],{"class":246},[237,639,350],{"class":406},[237,641,410],{"class":246},[237,643,413],{"class":349},[237,645,410],{"class":246},[237,647,648],{"class":349},"get_service_metrics",[237,650,121],{"class":246},[237,652,653],{"class":349}," svc",[237,655,410],{"class":246},[237,657,658],{"class":349},"id",[237,660,121],{"class":246},[237,662,663],{"class":349}," traffic_cache",[237,665,666],{"class":246},"):",[237,668,669],{"class":246}," (",[237,671,672],{"class":349},"svc",[237,674,121],{"class":246},[237,676,677],{"class":349}," labels",[237,679,680],{"class":246},")\n",[237,682,684,687,689,691,694,697],{"class":239,"line":683},18,[237,685,686],{"class":452},"            for",[237,688,653],{"class":349},[237,690,121],{"class":246},[237,692,693],{"class":349}," labels ",[237,695,696],{"class":452},"in",[237,698,699],{"class":349}," enabled\n",[237,701,703],{"class":239,"line":702},19,[237,704,705],{"class":246},"        }\n",[237,707,709,712,714,717,719,722,724,727,730,732,734,736],{"class":239,"line":708},20,[237,710,711],{"class":349},"        remaining ",[237,713,395],{"class":246},[237,715,716],{"class":242}," max",[237,718,346],{"class":246},[237,720,721],{"class":584},"0.1",[237,723,121],{"class":246},[237,725,726],{"class":349}," deadline ",[237,728,729],{"class":338},"-",[237,731,524],{"class":349},[237,733,410],{"class":246},[237,735,529],{"class":349},[237,737,738],{"class":246},"())\n",[237,740,742,745,747,750,752,755,757,760,762,765,767,770],{"class":239,"line":741},21,[237,743,744],{"class":349},"        done",[237,746,121],{"class":246},[237,748,749],{"class":349}," not_done ",[237,751,395],{"class":246},[237,753,754],{"class":349}," _wait",[237,756,346],{"class":246},[237,758,759],{"class":349},"future_to_svc",[237,761,121],{"class":246},[237,763,764],{"class":562}," timeout",[237,766,395],{"class":246},[237,768,769],{"class":349},"remaining",[237,771,680],{"class":246},[237,773,775],{"class":239,"line":774},22,[237,776,426],{"emptyLinePlaceholder":425},[237,778,780,783,786],{"class":239,"line":779},23,[237,781,782],{"class":452},"        if",[237,784,785],{"class":349}," not_done",[237,787,247],{"class":246},[237,789,791,794,796,799,801,803,806,809,812,815,818,821,824,826,828],{"class":239,"line":790},24,[237,792,793],{"class":349},"            skipped ",[237,795,395],{"class":246},[237,797,798],{"class":246}," [",[237,800,759],{"class":349},[237,802,362],{"class":246},[237,804,805],{"class":349},"f",[237,807,808],{"class":246},"][",[237,810,811],{"class":584},"0",[237,813,814],{"class":246},"].",[237,816,817],{"class":349},"name ",[237,819,820],{"class":452},"for",[237,822,823],{"class":349}," f ",[237,825,696],{"class":452},[237,827,785],{"class":349},[237,829,830],{"class":246},"]\n",[237,832,834,837,839,842,844,846,849,852,854,856,859,861,864,867,869,872,874,876],{"class":239,"line":833},25,[237,835,836],{"class":349},"            logger",[237,838,410],{"class":246},[237,840,841],{"class":349},"warning",[237,843,346],{"class":246},[237,845,805],{"class":338},[237,847,848],{"class":267},"\"Collection deadline: ",[237,850,851],{"class":406},"{",[237,853,573],{"class":242},[237,855,346],{"class":246},[237,857,858],{"class":349},"not_done",[237,860,353],{"class":246},[237,862,863],{"class":406},"}",[237,865,866],{"class":267}," skipped this cycle: ",[237,868,851],{"class":406},[237,870,871],{"class":349},"skipped",[237,873,863],{"class":406},[237,875,284],{"class":267},[237,877,680],{"class":246},[237,879,881,883,886,888,890],{"class":239,"line":880},26,[237,882,686],{"class":452},[237,884,885],{"class":349}," fut ",[237,887,696],{"class":452},[237,889,785],{"class":349},[237,891,247],{"class":246},[237,893,895,898,900,903],{"class":239,"line":894},27,[237,896,897],{"class":349},"                fut",[237,899,410],{"class":246},[237,901,902],{"class":349},"cancel",[237,904,401],{"class":246},[237,906,908],{"class":239,"line":907},28,[237,909,910],{"class":406},"        ...\n",[34,912,913],{},"Two design decisions worth calling out:",[84,915,916,922],{},[87,917,918,921],{},[37,919,920],{},"Traefik is scraped once per cycle, not once per service."," The scraper keeps per-node\nrate state (previous counter values). If you scraped per service, every call but the first\nwould see a zero time delta and report a zero rate. Scrape once, pass the cache down.",[87,923,924,927],{},[37,925,926],{},"Concurrency is capped at 16."," Beyond that, the Docker daemon socket becomes the\nbottleneck, not the thread count.",[220,929,931],{"id":930},"computing-cpu-the-right-way","Computing CPU the right way",[34,933,934],{},"Docker\u002FcAdvisor expose raw cumulative CPU counters, not percentages. You need two samples and\nthe standard delta formula:",[228,936,941],{"className":937,"code":939,"language":940},[938],"language-text","cpu_delta    = cpu_usage.total      - precpu_usage.total\nsystem_delta = system_cpu_usage     - pre_system_cpu_usage\ncpu_percent  = (cpu_delta \u002F system_delta) * online_cpus * 100\n","text",[48,942,939],{"__ignoreMap":233},[34,944,945,947,948,951,952,955],{},[48,946,50],{}," pulls two samples straight from cAdvisor (",[48,949,950],{},"?count=2",") rather than Docker's own\nstats endpoint, because cAdvisor also exposes ",[37,953,954],{},"CPU throttling",", which is the single most\nuseful signal for a CPU-limited container:",[228,957,959],{"className":329,"code":958,"language":331,"meta":233,"style":233},"# cpu_throttle_pct: % of CFS scheduling periods that were throttled. A container\n#                    showing low raw CPU% but high throttle% is hitting its CPU ceiling.\n",[48,960,961,966],{"__ignoreMap":233},[237,962,963],{"class":239,"line":11},[237,964,965],{"class":287},"# cpu_throttle_pct: % of CFS scheduling periods that were throttled. A container\n",[237,967,968],{"class":239,"line":20},[237,969,970],{"class":287},"#                    showing low raw CPU% but high throttle% is hitting its CPU ceiling.\n",[34,972,973,974,977],{},"A container pinned to ",[48,975,976],{},"0.5"," CPU can sit at \"50% CPU\" forever and look healthy while being\nheavily throttled and serving slow responses. Raw CPU percentage hides that; throttle percentage\nexposes it. This is why the engine has a dedicated throttle trigger.",[220,979,981],{"id":980},"two-memory-denominators","Two memory denominators",[34,983,984],{},"Memory gets reported two ways because there are two meaningful questions:",[84,986,987,995],{},[87,988,989,994],{},[37,990,991],{},[48,992,993],{},"node_mem_pct",": memory as a percentage of the host node's total RAM. Always present,\nalways a consistent denominator across services. This is what the policy engine uses.",[87,996,997,1002,1003,1006],{},[37,998,999],{},[48,1000,1001],{},"limit_mem_pct",": memory as a percentage of the container's own limit. Only present when\na real limit is set; ",[48,1004,1005],{},"None"," otherwise.",[34,1008,1009,1010,1013],{},"The \"no limit set\" case is a trap. cAdvisor reports an unlimited container's limit as a kernel\nsentinel (",[48,1011,1012],{},"9223372036854771712",", i.e. max int64). Dividing by that gives a meaningless\nnear-zero percentage, so the code treats anything over 128 GB as \"no limit\":",[228,1015,1017],{"className":329,"code":1016,"language":331,"meta":233,"style":233},"_MAX_SANE_LIMIT = 128 * 1024 * 1024 * 1024  # 128 GB\nmem_limit_bytes = entry.get(\"spec\", {}).get(\"memory\", {}).get(\"limit\", 0)\nhas_real_limit  = 0 \u003C mem_limit_bytes \u003C _MAX_SANE_LIMIT\nlimit_mem_pct   = (mem_bytes \u002F mem_limit_bytes * 100.0) if has_real_limit else None\n",[48,1018,1019,1047,1109,1130],{"__ignoreMap":233},[237,1020,1021,1024,1027,1030,1033,1036,1038,1040,1042,1044],{"class":239,"line":11},[237,1022,1023],{"class":406},"_MAX_SANE_LIMIT",[237,1025,1026],{"class":246}," =",[237,1028,1029],{"class":584}," 128",[237,1031,1032],{"class":338}," *",[237,1034,1035],{"class":584}," 1024",[237,1037,1032],{"class":338},[237,1039,1035],{"class":584},[237,1041,1032],{"class":338},[237,1043,1035],{"class":584},[237,1045,1046],{"class":287},"  # 128 GB\n",[237,1048,1049,1052,1054,1057,1059,1062,1064,1066,1069,1071,1073,1076,1078,1080,1082,1085,1087,1089,1091,1093,1095,1097,1100,1102,1104,1107],{"class":239,"line":20},[237,1050,1051],{"class":349},"mem_limit_bytes ",[237,1053,395],{"class":246},[237,1055,1056],{"class":349}," entry",[237,1058,410],{"class":246},[237,1060,1061],{"class":349},"get",[237,1063,346],{"class":246},[237,1065,284],{"class":263},[237,1067,1068],{"class":267},"spec",[237,1070,284],{"class":263},[237,1072,121],{"class":246},[237,1074,1075],{"class":246}," {}).",[237,1077,1061],{"class":349},[237,1079,346],{"class":246},[237,1081,284],{"class":263},[237,1083,1084],{"class":267},"memory",[237,1086,284],{"class":263},[237,1088,121],{"class":246},[237,1090,1075],{"class":246},[237,1092,1061],{"class":349},[237,1094,346],{"class":246},[237,1096,284],{"class":263},[237,1098,1099],{"class":267},"limit",[237,1101,284],{"class":263},[237,1103,121],{"class":246},[237,1105,1106],{"class":584}," 0",[237,1108,680],{"class":246},[237,1110,1111,1114,1116,1118,1121,1124,1127],{"class":239,"line":257},[237,1112,1113],{"class":349},"has_real_limit  ",[237,1115,395],{"class":246},[237,1117,1106],{"class":584},[237,1119,1120],{"class":338}," \u003C",[237,1122,1123],{"class":349}," mem_limit_bytes ",[237,1125,1126],{"class":338},"\u003C",[237,1128,1129],{"class":406}," _MAX_SANE_LIMIT\n",[237,1131,1132,1135,1137,1139,1142,1145,1147,1150,1153,1155,1158,1161,1164],{"class":239,"line":274},[237,1133,1134],{"class":349},"limit_mem_pct   ",[237,1136,395],{"class":246},[237,1138,669],{"class":246},[237,1140,1141],{"class":349},"mem_bytes ",[237,1143,1144],{"class":338},"\u002F",[237,1146,1123],{"class":349},[237,1148,1149],{"class":338},"*",[237,1151,1152],{"class":584}," 100.0",[237,1154,353],{"class":246},[237,1156,1157],{"class":452}," if",[237,1159,1160],{"class":349}," has_real_limit ",[237,1162,1163],{"class":452},"else",[237,1165,1166],{"class":452}," None\n",[34,1168,1169,1170,1173,1174,1177,1178,1181,1182,1184],{},"Memory also uses ",[48,1171,1172],{},"working_set"," (not raw ",[48,1175,1176],{},"usage",") to match what ",[48,1179,1180],{},"docker stats"," shows:\n",[48,1183,1172],{}," excludes reclaimable page cache, so you measure what the process actually needs.",[220,1186,1188],{"id":1187},"a-stale-data-guard","A stale-data guard",[34,1190,1191],{},"cAdvisor's default housekeeping interval is 10 seconds. If your watch interval is 5 seconds,\nevery other cycle you'd re-read the exact same sample, compute a zero delta, and report 0% CPU,\nwhich would spuriously suppress scale-up triggers on alternating cycles. The collector caches\nthe last result keyed by timestamp and returns the cache when the sample hasn't advanced:",[228,1193,1195],{"className":329,"code":1194,"language":331,"meta":233,"style":233},"latest_ts = stats_list[-1].get(\"timestamp\", \"\")\ncached_ts, cached_result = self._cadvisor_cache.get(container_id, (\"\", None))\nif latest_ts and latest_ts == cached_ts and cached_result is not None:\n    return cached_result   # don't report a false 0%\n",[48,1196,1197,1234,1277,1308],{"__ignoreMap":233},[237,1198,1199,1202,1204,1207,1209,1211,1214,1216,1218,1220,1222,1225,1227,1229,1232],{"class":239,"line":11},[237,1200,1201],{"class":349},"latest_ts ",[237,1203,395],{"class":246},[237,1205,1206],{"class":349}," stats_list",[237,1208,362],{"class":246},[237,1210,729],{"class":338},[237,1212,1213],{"class":584},"1",[237,1215,814],{"class":246},[237,1217,1061],{"class":349},[237,1219,346],{"class":246},[237,1221,284],{"class":263},[237,1223,1224],{"class":267},"timestamp",[237,1226,284],{"class":263},[237,1228,121],{"class":246},[237,1230,1231],{"class":263}," \"\"",[237,1233,680],{"class":246},[237,1235,1236,1239,1241,1244,1246,1248,1250,1253,1255,1257,1259,1262,1264,1266,1269,1271,1274],{"class":239,"line":20},[237,1237,1238],{"class":349},"cached_ts",[237,1240,121],{"class":246},[237,1242,1243],{"class":349}," cached_result ",[237,1245,395],{"class":246},[237,1247,436],{"class":406},[237,1249,410],{"class":246},[237,1251,1252],{"class":349},"_cadvisor_cache",[237,1254,410],{"class":246},[237,1256,1061],{"class":349},[237,1258,346],{"class":246},[237,1260,1261],{"class":349},"container_id",[237,1263,121],{"class":246},[237,1265,669],{"class":246},[237,1267,1268],{"class":263},"\"\"",[237,1270,121],{"class":246},[237,1272,1273],{"class":452}," None",[237,1275,1276],{"class":246},"))\n",[237,1278,1279,1282,1285,1288,1290,1293,1296,1298,1300,1302,1304,1306],{"class":239,"line":257},[237,1280,1281],{"class":452},"if",[237,1283,1284],{"class":349}," latest_ts ",[237,1286,1287],{"class":338},"and",[237,1289,1284],{"class":349},[237,1291,1292],{"class":338},"==",[237,1294,1295],{"class":349}," cached_ts ",[237,1297,1287],{"class":338},[237,1299,1243],{"class":349},[237,1301,104],{"class":338},[237,1303,456],{"class":338},[237,1305,1273],{"class":452},[237,1307,247],{"class":246},[237,1309,1310,1313,1316],{"class":239,"line":274},[237,1311,1312],{"class":452},"    return",[237,1314,1315],{"class":349}," cached_result   ",[237,1317,1318],{"class":287},"# don't report a false 0%\n",[220,1320,1322],{"id":1321},"replica-state-and-stuck-pending","Replica state and \"stuck pending\"",[34,1324,1325,1326,1329,1330,1332],{},"The collector counts tasks by state and, critically, distinguishes ",[55,1327,1328],{},"why"," a task is pending.\nA task pending because of insufficient cluster capacity is a scale-out signal. A task pending\nbecause of a bad image or a missing secret is a broken deployment that must ",[37,1331,81],{}," trigger\nscaling:",[228,1334,1336],{"className":329,"code":1335,"language":331,"meta":233,"style":233},"_CAPACITY_PATTERNS = (\"no suitable node\", \"insufficient resources\",\n                      \"scheduling constraints not satisfied\")\n_CONFIG_ERROR_PATTERNS = (\"no such image\", \"pull access denied\", \"secret not found\", ...)\n",[48,1337,1338,1366,1378],{"__ignoreMap":233},[237,1339,1340,1343,1345,1347,1349,1352,1354,1356,1358,1361,1363],{"class":239,"line":11},[237,1341,1342],{"class":406},"_CAPACITY_PATTERNS",[237,1344,1026],{"class":246},[237,1346,669],{"class":246},[237,1348,284],{"class":263},[237,1350,1351],{"class":267},"no suitable node",[237,1353,284],{"class":263},[237,1355,121],{"class":246},[237,1357,264],{"class":263},[237,1359,1360],{"class":267},"insufficient resources",[237,1362,284],{"class":263},[237,1364,1365],{"class":246},",\n",[237,1367,1368,1371,1374,1376],{"class":239,"line":20},[237,1369,1370],{"class":263},"                      \"",[237,1372,1373],{"class":267},"scheduling constraints not satisfied",[237,1375,284],{"class":263},[237,1377,680],{"class":246},[237,1379,1380,1383,1385,1387,1389,1392,1394,1396,1398,1401,1403,1405,1407,1410,1412,1414,1417],{"class":239,"line":257},[237,1381,1382],{"class":406},"_CONFIG_ERROR_PATTERNS",[237,1384,1026],{"class":246},[237,1386,669],{"class":246},[237,1388,284],{"class":263},[237,1390,1391],{"class":267},"no such image",[237,1393,284],{"class":263},[237,1395,121],{"class":246},[237,1397,264],{"class":263},[237,1399,1400],{"class":267},"pull access denied",[237,1402,284],{"class":263},[237,1404,121],{"class":246},[237,1406,264],{"class":263},[237,1408,1409],{"class":267},"secret not found",[237,1411,284],{"class":263},[237,1413,121],{"class":246},[237,1415,1416],{"class":406}," ...",[237,1418,680],{"class":246},[34,1420,1421,1422,1425],{},"Only capacity-pending tasks feed ",[48,1423,1424],{},"longest_pending_seconds",", which the engine turns into a\nhost-scale signal.",[220,1427,1429],{"id":1428},"service-name-matching-with-traefik","Service name matching with Traefik",[34,1431,1432,1433,1436,1437,1440,1441,1444,1445,1448],{},"Traefik labels services as ",[48,1434,1435],{},"myapp_blazor@docker","; Docker calls them ",[48,1438,1439],{},"myapp_blazor",". The scraper\nstrips the ",[48,1442,1443],{},"@docker"," \u002F ",[48,1446,1447],{},"@swarm"," provider suffix with a regex, and the actual Traefik service name\nis derived from the service's own Traefik label rather than guessed:",[228,1450,1452],{"className":329,"code":1451,"language":331,"meta":233,"style":233},"# traefik.http.services.\u003Cname>.loadbalancer.server.port  →  \u003Cname>\n",[48,1453,1454],{"__ignoreMap":233},[237,1455,1456],{"class":239,"line":11},[237,1457,1451],{"class":287},[34,1459,1460,1461,1464,1465,1468],{},"Because Traefik can run on several nodes (one per manager in our cluster), the scraper discovers all Traefik\nendpoints and keeps ",[37,1462,1463],{},"per-node"," rate state, summing deltas across nodes each cycle. Counter\nresets (a Traefik container restart) are guarded with ",[48,1466,1467],{},"max(0.0, current - previous)"," so a reset\nnever produces a negative rate.",[34,1470,1471,1472,1474],{},"The output of all this is one ",[48,1473,217],{}," per service:",[228,1476,1478],{"className":329,"code":1477,"language":331,"meta":233,"style":233},"class ServiceMetrics(BaseModel):\n    service: str\n    replicas_running: int\n    replicas_desired: int\n    replicas_pending: int\n    replicas_failed: int\n    replicas_rejected: int = 0\n\n    avg_cpu: float; max_cpu: float\n    avg_throttle: float; max_throttle: float      # the throttle signal\n    avg_mem: float; max_mem: float                 # % of node RAM\n    avg_limit_mem: Optional[float]; max_limit_mem: Optional[float]  # % of container limit\n\n    request_rate: float; avg_response_ms: float; error_rate: float; open_connections: int\n    longest_pending_seconds: int\n    tasks: List[TaskStat]\n",[48,1479,1480,1497,1508,1518,1527,1536,1545,1560,1564,1582,1601,1620,1654,1658,1688,1697],{"__ignoreMap":233},[237,1481,1482,1485,1489,1491,1494],{"class":239,"line":11},[237,1483,1484],{"class":338},"class",[237,1486,1488],{"class":1487},"sUxyF"," ServiceMetrics",[237,1490,346],{"class":246},[237,1492,1493],{"class":342},"BaseModel",[237,1495,1496],{"class":246},"):\n",[237,1498,1499,1502,1505],{"class":239,"line":20},[237,1500,1501],{"class":349},"    service",[237,1503,1504],{"class":246},":",[237,1506,1507],{"class":242}," str\n",[237,1509,1510,1513,1515],{"class":239,"line":257},[237,1511,1512],{"class":349},"    replicas_running",[237,1514,1504],{"class":246},[237,1516,1517],{"class":242}," int\n",[237,1519,1520,1523,1525],{"class":239,"line":274},[237,1521,1522],{"class":349},"    replicas_desired",[237,1524,1504],{"class":246},[237,1526,1517],{"class":242},[237,1528,1529,1532,1534],{"class":239,"line":291},[237,1530,1531],{"class":349},"    replicas_pending",[237,1533,1504],{"class":246},[237,1535,1517],{"class":242},[237,1537,1538,1541,1543],{"class":239,"line":306},[237,1539,1540],{"class":349},"    replicas_failed",[237,1542,1504],{"class":246},[237,1544,1517],{"class":242},[237,1546,1547,1550,1552,1555,1557],{"class":239,"line":464},[237,1548,1549],{"class":349},"    replicas_rejected",[237,1551,1504],{"class":246},[237,1553,1554],{"class":242}," int",[237,1556,1026],{"class":246},[237,1558,1559],{"class":584}," 0\n",[237,1561,1562],{"class":239,"line":473},[237,1563,426],{"emptyLinePlaceholder":425},[237,1565,1566,1569,1571,1574,1577,1579],{"class":239,"line":478},[237,1567,1568],{"class":349},"    avg_cpu",[237,1570,1504],{"class":246},[237,1572,1573],{"class":242}," float",[237,1575,1576],{"class":349},"; max_cpu",[237,1578,1504],{"class":246},[237,1580,1581],{"class":242}," float\n",[237,1583,1584,1587,1589,1591,1594,1596,1598],{"class":239,"line":484},[237,1585,1586],{"class":349},"    avg_throttle",[237,1588,1504],{"class":246},[237,1590,1573],{"class":242},[237,1592,1593],{"class":349},"; max_throttle",[237,1595,1504],{"class":246},[237,1597,1573],{"class":242},[237,1599,1600],{"class":287},"      # the throttle signal\n",[237,1602,1603,1606,1608,1610,1613,1615,1617],{"class":239,"line":490},[237,1604,1605],{"class":349},"    avg_mem",[237,1607,1504],{"class":246},[237,1609,1573],{"class":242},[237,1611,1612],{"class":349},"; max_mem",[237,1614,1504],{"class":246},[237,1616,1573],{"class":242},[237,1618,1619],{"class":287},"                 # % of node RAM\n",[237,1621,1622,1625,1627,1630,1632,1635,1638,1641,1643,1645,1647,1649,1651],{"class":239,"line":516},[237,1623,1624],{"class":349},"    avg_limit_mem",[237,1626,1504],{"class":246},[237,1628,1629],{"class":349}," Optional",[237,1631,362],{"class":246},[237,1633,1634],{"class":242},"float",[237,1636,1637],{"class":246},"]",[237,1639,1640],{"class":349},"; max_limit_mem",[237,1642,1504],{"class":246},[237,1644,1629],{"class":349},[237,1646,362],{"class":246},[237,1648,1634],{"class":242},[237,1650,1637],{"class":246},[237,1652,1653],{"class":287},"  # % of container limit\n",[237,1655,1656],{"class":239,"line":546},[237,1657,426],{"emptyLinePlaceholder":425},[237,1659,1660,1663,1665,1667,1670,1672,1674,1677,1679,1681,1684,1686],{"class":239,"line":551},[237,1661,1662],{"class":349},"    request_rate",[237,1664,1504],{"class":246},[237,1666,1573],{"class":242},[237,1668,1669],{"class":349},"; avg_response_ms",[237,1671,1504],{"class":246},[237,1673,1573],{"class":242},[237,1675,1676],{"class":349},"; error_rate",[237,1678,1504],{"class":246},[237,1680,1573],{"class":242},[237,1682,1683],{"class":349},"; open_connections",[237,1685,1504],{"class":246},[237,1687,1517],{"class":242},[237,1689,1690,1693,1695],{"class":239,"line":591},[237,1691,1692],{"class":349},"    longest_pending_seconds",[237,1694,1504],{"class":246},[237,1696,1517],{"class":242},[237,1698,1699,1702,1704,1706,1708,1711],{"class":239,"line":616},[237,1700,1701],{"class":349},"    tasks",[237,1703,1504],{"class":246},[237,1705,359],{"class":349},[237,1707,362],{"class":246},[237,1709,1710],{"class":349},"TaskStat",[237,1712,830],{"class":246},[167,1714],{},[59,1716,1718],{"id":1717},"part-2-the-policy-layer","Part 2: The Policy Layer",[34,1720,1721],{},"The policy layer has two halves that must stay strictly separated:",[84,1723,1724,1733],{},[87,1725,1726,1729,1730,410],{},[37,1727,1728],{},"The resolver"," does I\u002FO. It reads config, merges labels and templates, reloads when the\nconfig file changes. It produces a ",[48,1731,1732],{},"ResolvedPolicy",[87,1734,1735,1738,1739,1742],{},[37,1736,1737],{},"The engine"," does logic. It takes ",[48,1740,1741],{},"(metrics, policy, timers)"," and returns a decision. No\nI\u002FO, no clock reads, no randomness. Same inputs → same outputs, every time.",[34,1744,1745],{},"That separation is what makes the engine fully unit-testable without mocking anything, and it's\nwhy the test file for it is ~28 KB on its own.",[220,1747,1749],{"id":1748},"the-resolver-three-layers-highest-wins","The resolver: three layers, highest wins",[34,1751,1752],{},"Every policy field is resolved from three layers, in priority order:",[228,1754,1757],{"className":1755,"code":1756,"language":940},[938],"1. per-service label   (swarmctl.scale_up_cpu_avg=60)   highest\n2. named template      (swarmctl.policy=blazor-standard) middle\n3. global default      (from config.json)                lowest, always present\n",[48,1758,1756],{"__ignoreMap":233},[228,1760,1762],{"className":329,"code":1761,"language":331,"meta":233,"style":233},"def resolve(self, service, labels) -> ResolvedPolicy | None:\n    self._reload_if_changed()                 # hot-reload config.json on mtime change\n    resolved = self.config[\"defaults\"].copy()  # layer 3\n    resolved[\"service\"] = service.name\n    resolved[\"enabled\"] = True\n\n    template = labels.get(\"swarmctl.policy\")   # layer 2\n    if template and template in self.config[\"templates\"]:\n        resolved.update(self.config[\"templates\"][template])\n\n    for label_key, (field, typ) in label_map.items():   # layer 1\n        if label_key in labels:\n            try:\n                val = labels[label_key]\n                resolved[field] = (val.lower() == \"true\") if typ is bool else typ(val)\n            except (ValueError, AttributeError):\n                pass   # invalid label value → silently fall through to template\u002Fdefault\n\n    policy = ResolvedPolicy(**resolved)\n    return policy if self._validate(policy) else None\n",[48,1763,1764,1798,1812,1845,1870,1889,1893,1920,1951,1985,1989,2028,2041,2048,2064,2122,2139,2147,2151,2171],{"__ignoreMap":233},[237,1765,1766,1768,1771,1773,1775,1777,1780,1782,1784,1786,1788,1791,1794,1796],{"class":239,"line":11},[237,1767,339],{"class":338},[237,1769,1770],{"class":342}," resolve",[237,1772,346],{"class":246},[237,1774,350],{"class":349},[237,1776,121],{"class":246},[237,1778,1779],{"class":349}," service",[237,1781,121],{"class":246},[237,1783,677],{"class":349},[237,1785,353],{"class":246},[237,1787,356],{"class":246},[237,1789,1790],{"class":349}," ResolvedPolicy ",[237,1792,1793],{"class":338},"|",[237,1795,1273],{"class":452},[237,1797,247],{"class":246},[237,1799,1800,1802,1804,1807,1809],{"class":239,"line":20},[237,1801,407],{"class":406},[237,1803,410],{"class":246},[237,1805,1806],{"class":349},"_reload_if_changed",[237,1808,532],{"class":246},[237,1810,1811],{"class":287},"                 # hot-reload config.json on mtime change\n",[237,1813,1814,1817,1819,1821,1823,1826,1828,1830,1833,1835,1837,1840,1842],{"class":239,"line":257},[237,1815,1816],{"class":349},"    resolved ",[237,1818,395],{"class":246},[237,1820,436],{"class":406},[237,1822,410],{"class":246},[237,1824,1825],{"class":349},"config",[237,1827,362],{"class":246},[237,1829,284],{"class":263},[237,1831,1832],{"class":267},"defaults",[237,1834,284],{"class":263},[237,1836,814],{"class":246},[237,1838,1839],{"class":349},"copy",[237,1841,532],{"class":246},[237,1843,1844],{"class":287},"  # layer 3\n",[237,1846,1847,1850,1852,1854,1857,1859,1861,1863,1865,1867],{"class":239,"line":274},[237,1848,1849],{"class":349},"    resolved",[237,1851,362],{"class":246},[237,1853,284],{"class":263},[237,1855,1856],{"class":267},"service",[237,1858,284],{"class":263},[237,1860,1637],{"class":246},[237,1862,1026],{"class":246},[237,1864,1779],{"class":349},[237,1866,410],{"class":246},[237,1868,1869],{"class":349},"name\n",[237,1871,1872,1874,1876,1878,1880,1882,1884,1886],{"class":239,"line":291},[237,1873,1849],{"class":349},[237,1875,362],{"class":246},[237,1877,284],{"class":263},[237,1879,578],{"class":267},[237,1881,284],{"class":263},[237,1883,1637],{"class":246},[237,1885,1026],{"class":246},[237,1887,1888],{"class":452}," True\n",[237,1890,1891],{"class":239,"line":306},[237,1892,426],{"emptyLinePlaceholder":425},[237,1894,1895,1898,1900,1902,1904,1906,1908,1910,1913,1915,1917],{"class":239,"line":464},[237,1896,1897],{"class":349},"    template ",[237,1899,395],{"class":246},[237,1901,677],{"class":349},[237,1903,410],{"class":246},[237,1905,1061],{"class":349},[237,1907,346],{"class":246},[237,1909,284],{"class":263},[237,1911,1912],{"class":267},"swarmctl.policy",[237,1914,284],{"class":263},[237,1916,353],{"class":246},[237,1918,1919],{"class":287},"   # layer 2\n",[237,1921,1922,1924,1927,1929,1931,1933,1935,1937,1939,1941,1943,1946,1948],{"class":239,"line":473},[237,1923,453],{"class":452},[237,1925,1926],{"class":349}," template ",[237,1928,1287],{"class":338},[237,1930,1926],{"class":349},[237,1932,696],{"class":338},[237,1934,436],{"class":406},[237,1936,410],{"class":246},[237,1938,1825],{"class":349},[237,1940,362],{"class":246},[237,1942,284],{"class":263},[237,1944,1945],{"class":267},"templates",[237,1947,284],{"class":263},[237,1949,1950],{"class":246},"]:\n",[237,1952,1953,1956,1958,1961,1963,1965,1967,1969,1971,1973,1975,1977,1979,1982],{"class":239,"line":478},[237,1954,1955],{"class":349},"        resolved",[237,1957,410],{"class":246},[237,1959,1960],{"class":349},"update",[237,1962,346],{"class":246},[237,1964,350],{"class":406},[237,1966,410],{"class":246},[237,1968,1825],{"class":349},[237,1970,362],{"class":246},[237,1972,284],{"class":263},[237,1974,1945],{"class":267},[237,1976,284],{"class":263},[237,1978,808],{"class":246},[237,1980,1981],{"class":349},"template",[237,1983,1984],{"class":246},"])\n",[237,1986,1987],{"class":239,"line":484},[237,1988,426],{"emptyLinePlaceholder":425},[237,1990,1991,1994,1997,1999,2001,2004,2006,2009,2011,2014,2017,2019,2022,2025],{"class":239,"line":490},[237,1992,1993],{"class":452},"    for",[237,1995,1996],{"class":349}," label_key",[237,1998,121],{"class":246},[237,2000,669],{"class":246},[237,2002,2003],{"class":349},"field",[237,2005,121],{"class":246},[237,2007,2008],{"class":349}," typ",[237,2010,353],{"class":246},[237,2012,2013],{"class":452}," in",[237,2015,2016],{"class":349}," label_map",[237,2018,410],{"class":246},[237,2020,2021],{"class":349},"items",[237,2023,2024],{"class":246},"():",[237,2026,2027],{"class":287},"   # layer 1\n",[237,2029,2030,2032,2035,2037,2039],{"class":239,"line":516},[237,2031,782],{"class":452},[237,2033,2034],{"class":349}," label_key ",[237,2036,696],{"class":338},[237,2038,677],{"class":349},[237,2040,247],{"class":246},[237,2042,2043,2046],{"class":239,"line":546},[237,2044,2045],{"class":452},"            try",[237,2047,247],{"class":246},[237,2049,2050,2053,2055,2057,2059,2062],{"class":239,"line":551},[237,2051,2052],{"class":349},"                val ",[237,2054,395],{"class":246},[237,2056,677],{"class":349},[237,2058,362],{"class":246},[237,2060,2061],{"class":349},"label_key",[237,2063,830],{"class":246},[237,2065,2066,2069,2071,2073,2075,2077,2079,2082,2084,2087,2089,2092,2094,2097,2099,2101,2103,2106,2108,2111,2114,2116,2118,2120],{"class":239,"line":591},[237,2067,2068],{"class":349},"                resolved",[237,2070,362],{"class":246},[237,2072,2003],{"class":349},[237,2074,1637],{"class":246},[237,2076,1026],{"class":246},[237,2078,669],{"class":246},[237,2080,2081],{"class":349},"val",[237,2083,410],{"class":246},[237,2085,2086],{"class":349},"lower",[237,2088,532],{"class":246},[237,2090,2091],{"class":338}," ==",[237,2093,264],{"class":263},[237,2095,2096],{"class":267},"true",[237,2098,284],{"class":263},[237,2100,353],{"class":246},[237,2102,1157],{"class":452},[237,2104,2105],{"class":349}," typ ",[237,2107,104],{"class":338},[237,2109,2110],{"class":242}," bool",[237,2112,2113],{"class":452}," else",[237,2115,2008],{"class":349},[237,2117,346],{"class":246},[237,2119,2081],{"class":349},[237,2121,680],{"class":246},[237,2123,2124,2127,2129,2132,2134,2137],{"class":239,"line":616},[237,2125,2126],{"class":452},"            except",[237,2128,669],{"class":246},[237,2130,2131],{"class":242},"ValueError",[237,2133,121],{"class":246},[237,2135,2136],{"class":242}," AttributeError",[237,2138,1496],{"class":246},[237,2140,2141,2144],{"class":239,"line":627},[237,2142,2143],{"class":452},"                pass",[237,2145,2146],{"class":287},"   # invalid label value → silently fall through to template\u002Fdefault\n",[237,2148,2149],{"class":239,"line":683},[237,2150,426],{"emptyLinePlaceholder":425},[237,2152,2153,2156,2158,2161,2163,2166,2169],{"class":239,"line":702},[237,2154,2155],{"class":349},"    policy ",[237,2157,395],{"class":246},[237,2159,2160],{"class":349}," ResolvedPolicy",[237,2162,346],{"class":246},[237,2164,2165],{"class":338},"**",[237,2167,2168],{"class":349},"resolved",[237,2170,680],{"class":246},[237,2172,2173,2175,2178,2180,2182,2184,2187,2189,2192,2194,2196],{"class":239,"line":708},[237,2174,1312],{"class":452},[237,2176,2177],{"class":349}," policy ",[237,2179,1281],{"class":452},[237,2181,436],{"class":406},[237,2183,410],{"class":246},[237,2185,2186],{"class":349},"_validate",[237,2188,346],{"class":246},[237,2190,2191],{"class":349},"policy",[237,2193,353],{"class":246},[237,2195,2113],{"class":452},[237,2197,1166],{"class":452},[34,2199,2200],{},"Note three things:",[84,2202,2203,2217,2227],{},[87,2204,2205,2208,2209,2212,2213,2216],{},[37,2206,2207],{},"Config hot-reload."," ",[48,2210,2211],{},"_reload_if_changed()"," checks the file mtime each cycle, so operators\ncan tune thresholds in ",[48,2214,2215],{},"config.json"," live without restarting the daemon.",[87,2218,2219,2222,2223,2226],{},[37,2220,2221],{},"Invalid labels fail silently."," A typo'd ",[48,2224,2225],{},"scale_up_cpu_avg=sixty"," doesn't crash the loop or\ndisable the service: it just falls through to the next layer.",[87,2228,2229,2232,2233,2236],{},[37,2230,2231],{},"Validation can veto a service."," If the resolved policy is incoherent, the service is\n",[55,2234,2235],{},"skipped for the cycle"," rather than scaled with nonsense thresholds:",[228,2238,2240],{"className":329,"code":2239,"language":331,"meta":233,"style":233},"def _validate(self, policy) -> bool:\n    if policy.min_replicas \u003C 1: return False\n    if policy.min_replicas > policy.max_replicas: return False\n    if policy.scale_down_cpu >= policy.scale_up_cpu_avg: return False   # no sane band\n    if policy.scale_down_mem >= policy.scale_up_mem_avg: return False\n    if policy.cooldown_after_up \u003C= 0 or policy.cooldown_after_down \u003C= 0: return False\n    if policy.scale_down_response_ms > policy.scale_up_response_ms: return False\n    return True\n",[48,2241,2242,2266,2290,2316,2347,2373,2409,2435],{"__ignoreMap":233},[237,2243,2244,2246,2249,2251,2253,2255,2258,2260,2262,2264],{"class":239,"line":11},[237,2245,339],{"class":338},[237,2247,2248],{"class":342}," _validate",[237,2250,346],{"class":246},[237,2252,350],{"class":349},[237,2254,121],{"class":246},[237,2256,2257],{"class":349}," policy",[237,2259,353],{"class":246},[237,2261,356],{"class":246},[237,2263,2110],{"class":242},[237,2265,247],{"class":246},[237,2267,2268,2270,2272,2274,2277,2279,2282,2284,2287],{"class":239,"line":20},[237,2269,453],{"class":452},[237,2271,2257],{"class":349},[237,2273,410],{"class":246},[237,2275,2276],{"class":349},"min_replicas ",[237,2278,1126],{"class":338},[237,2280,2281],{"class":584}," 1",[237,2283,1504],{"class":246},[237,2285,2286],{"class":452}," return",[237,2288,2289],{"class":452}," False\n",[237,2291,2292,2294,2296,2298,2300,2303,2305,2307,2310,2312,2314],{"class":239,"line":257},[237,2293,453],{"class":452},[237,2295,2257],{"class":349},[237,2297,410],{"class":246},[237,2299,2276],{"class":349},[237,2301,2302],{"class":338},">",[237,2304,2257],{"class":349},[237,2306,410],{"class":246},[237,2308,2309],{"class":349},"max_replicas",[237,2311,1504],{"class":246},[237,2313,2286],{"class":452},[237,2315,2289],{"class":452},[237,2317,2318,2320,2322,2324,2327,2330,2332,2334,2337,2339,2341,2344],{"class":239,"line":274},[237,2319,453],{"class":452},[237,2321,2257],{"class":349},[237,2323,410],{"class":246},[237,2325,2326],{"class":349},"scale_down_cpu ",[237,2328,2329],{"class":338},">=",[237,2331,2257],{"class":349},[237,2333,410],{"class":246},[237,2335,2336],{"class":349},"scale_up_cpu_avg",[237,2338,1504],{"class":246},[237,2340,2286],{"class":452},[237,2342,2343],{"class":452}," False",[237,2345,2346],{"class":287},"   # no sane band\n",[237,2348,2349,2351,2353,2355,2358,2360,2362,2364,2367,2369,2371],{"class":239,"line":291},[237,2350,453],{"class":452},[237,2352,2257],{"class":349},[237,2354,410],{"class":246},[237,2356,2357],{"class":349},"scale_down_mem ",[237,2359,2329],{"class":338},[237,2361,2257],{"class":349},[237,2363,410],{"class":246},[237,2365,2366],{"class":349},"scale_up_mem_avg",[237,2368,1504],{"class":246},[237,2370,2286],{"class":452},[237,2372,2289],{"class":452},[237,2374,2375,2377,2379,2381,2384,2387,2389,2392,2394,2396,2399,2401,2403,2405,2407],{"class":239,"line":306},[237,2376,453],{"class":452},[237,2378,2257],{"class":349},[237,2380,410],{"class":246},[237,2382,2383],{"class":349},"cooldown_after_up ",[237,2385,2386],{"class":338},"\u003C=",[237,2388,1106],{"class":584},[237,2390,2391],{"class":338}," or",[237,2393,2257],{"class":349},[237,2395,410],{"class":246},[237,2397,2398],{"class":349},"cooldown_after_down ",[237,2400,2386],{"class":338},[237,2402,1106],{"class":584},[237,2404,1504],{"class":246},[237,2406,2286],{"class":452},[237,2408,2289],{"class":452},[237,2410,2411,2413,2415,2417,2420,2422,2424,2426,2429,2431,2433],{"class":239,"line":464},[237,2412,453],{"class":452},[237,2414,2257],{"class":349},[237,2416,410],{"class":246},[237,2418,2419],{"class":349},"scale_down_response_ms ",[237,2421,2302],{"class":338},[237,2423,2257],{"class":349},[237,2425,410],{"class":246},[237,2427,2428],{"class":349},"scale_up_response_ms",[237,2430,1504],{"class":246},[237,2432,2286],{"class":452},[237,2434,2289],{"class":452},[237,2436,2437,2439],{"class":239,"line":473},[237,2438,1312],{"class":452},[237,2440,1888],{"class":452},[220,2442,2444],{"id":2443},"the-engine-the-decision-pipeline","The engine: the decision pipeline",[34,2446,2447],{},"The engine signature tells you everything about what state lives outside it:",[228,2449,2451],{"className":329,"code":2450,"language":331,"meta":233,"style":233},"def evaluate(\n    self,\n    metrics: ServiceMetrics,\n    policy: ResolvedPolicy,\n    time_since_last_action: int,        # from watch loop\n    last_scale_action: ScalingAction,   # from watch loop\n    elapsed_scale_up_seconds: int,      # from watch loop (duration timer)\n    elapsed_scale_down_seconds: int,    # from watch loop (duration timer)\n    suppressed_triggers: frozenset = frozenset(),  # per-trigger cooldown\n) -> Tuple[ScalingDecision, bool]:      # (decision, host_scale_signal)\n",[48,2452,2453,2463,2469,2480,2491,2505,2520,2534,2548,2568],{"__ignoreMap":233},[237,2454,2455,2457,2460],{"class":239,"line":11},[237,2456,339],{"class":338},[237,2458,2459],{"class":342}," evaluate",[237,2461,2462],{"class":246},"(\n",[237,2464,2465,2467],{"class":239,"line":20},[237,2466,407],{"class":349},[237,2468,1365],{"class":246},[237,2470,2471,2474,2476,2478],{"class":239,"line":257},[237,2472,2473],{"class":349},"    metrics",[237,2475,1504],{"class":246},[237,2477,1488],{"class":349},[237,2479,1365],{"class":246},[237,2481,2482,2485,2487,2489],{"class":239,"line":274},[237,2483,2484],{"class":349},"    policy",[237,2486,1504],{"class":246},[237,2488,2160],{"class":349},[237,2490,1365],{"class":246},[237,2492,2493,2496,2498,2500,2502],{"class":239,"line":291},[237,2494,2495],{"class":349},"    time_since_last_action",[237,2497,1504],{"class":246},[237,2499,1554],{"class":242},[237,2501,121],{"class":246},[237,2503,2504],{"class":287},"        # from watch loop\n",[237,2506,2507,2510,2512,2515,2517],{"class":239,"line":306},[237,2508,2509],{"class":349},"    last_scale_action",[237,2511,1504],{"class":246},[237,2513,2514],{"class":349}," ScalingAction",[237,2516,121],{"class":246},[237,2518,2519],{"class":287},"   # from watch loop\n",[237,2521,2522,2525,2527,2529,2531],{"class":239,"line":464},[237,2523,2524],{"class":349},"    elapsed_scale_up_seconds",[237,2526,1504],{"class":246},[237,2528,1554],{"class":242},[237,2530,121],{"class":246},[237,2532,2533],{"class":287},"      # from watch loop (duration timer)\n",[237,2535,2536,2539,2541,2543,2545],{"class":239,"line":473},[237,2537,2538],{"class":349},"    elapsed_scale_down_seconds",[237,2540,1504],{"class":246},[237,2542,1554],{"class":242},[237,2544,121],{"class":246},[237,2546,2547],{"class":287},"    # from watch loop (duration timer)\n",[237,2549,2550,2553,2555,2558,2560,2562,2565],{"class":239,"line":478},[237,2551,2552],{"class":349},"    suppressed_triggers",[237,2554,1504],{"class":246},[237,2556,2557],{"class":242}," frozenset",[237,2559,1026],{"class":246},[237,2561,2557],{"class":242},[237,2563,2564],{"class":246},"(),",[237,2566,2567],{"class":287},"  # per-trigger cooldown\n",[237,2569,2570,2572,2574,2577,2579,2582,2584,2586,2589],{"class":239,"line":484},[237,2571,353],{"class":246},[237,2573,356],{"class":246},[237,2575,2576],{"class":349}," Tuple",[237,2578,362],{"class":246},[237,2580,2581],{"class":349},"ScalingDecision",[237,2583,121],{"class":246},[237,2585,2110],{"class":242},[237,2587,2588],{"class":246},"]:",[237,2590,2591],{"class":287},"      # (decision, host_scale_signal)\n",[34,2593,2594,2595,2598,2599,410],{},"The engine never reads the clock. All timing is passed in. That means a test can simulate\n\"it's been 119 seconds since the last scale-up\" by passing ",[48,2596,2597],{},"119",": no sleeping, no mocking\n",[48,2600,2601],{},"time.time()",[34,2603,2604],{},"The pipeline runs in a strict order, short-circuiting as early as possible.",[2606,2607,2609],"h4",{"id":2608},"step-1-guard-checks","Step 1: Guard checks",[34,2611,2612],{},"Never scale a service that is stopped or crash-looping:",[228,2614,2616],{"className":329,"code":2615,"language":331,"meta":233,"style":233},"if not policy.enabled:\n    return self._none(svc, \"policy disabled\", BLOCKED_DISABLED), False\nif metrics.replicas_running == 0 and metrics.replicas_desired == 0:\n    return self._none(svc, \"service stopped\", BLOCKED_SERVICE_STOPPED), False\nif metrics.replicas_running == 0 and metrics.replicas_desired > 0:\n    return self._none(svc, \"crash loop: 0 running, desired > 0\", BLOCKED_CRASH_LOOP), False\n",[48,2617,2618,2632,2665,2697,2729,2757],{"__ignoreMap":233},[237,2619,2620,2622,2624,2626,2628,2630],{"class":239,"line":11},[237,2621,1281],{"class":452},[237,2623,456],{"class":338},[237,2625,2257],{"class":349},[237,2627,410],{"class":246},[237,2629,578],{"class":349},[237,2631,247],{"class":246},[237,2633,2634,2636,2638,2640,2643,2645,2647,2649,2651,2654,2656,2658,2661,2663],{"class":239,"line":20},[237,2635,1312],{"class":452},[237,2637,436],{"class":406},[237,2639,410],{"class":246},[237,2641,2642],{"class":349},"_none",[237,2644,346],{"class":246},[237,2646,672],{"class":349},[237,2648,121],{"class":246},[237,2650,264],{"class":263},[237,2652,2653],{"class":267},"policy disabled",[237,2655,284],{"class":263},[237,2657,121],{"class":246},[237,2659,2660],{"class":406}," BLOCKED_DISABLED",[237,2662,581],{"class":246},[237,2664,2289],{"class":452},[237,2666,2667,2669,2672,2674,2677,2679,2681,2684,2686,2688,2691,2693,2695],{"class":239,"line":257},[237,2668,1281],{"class":452},[237,2670,2671],{"class":349}," metrics",[237,2673,410],{"class":246},[237,2675,2676],{"class":349},"replicas_running ",[237,2678,1292],{"class":338},[237,2680,1106],{"class":584},[237,2682,2683],{"class":338}," and",[237,2685,2671],{"class":349},[237,2687,410],{"class":246},[237,2689,2690],{"class":349},"replicas_desired ",[237,2692,1292],{"class":338},[237,2694,1106],{"class":584},[237,2696,247],{"class":246},[237,2698,2699,2701,2703,2705,2707,2709,2711,2713,2715,2718,2720,2722,2725,2727],{"class":239,"line":274},[237,2700,1312],{"class":452},[237,2702,436],{"class":406},[237,2704,410],{"class":246},[237,2706,2642],{"class":349},[237,2708,346],{"class":246},[237,2710,672],{"class":349},[237,2712,121],{"class":246},[237,2714,264],{"class":263},[237,2716,2717],{"class":267},"service stopped",[237,2719,284],{"class":263},[237,2721,121],{"class":246},[237,2723,2724],{"class":406}," BLOCKED_SERVICE_STOPPED",[237,2726,581],{"class":246},[237,2728,2289],{"class":452},[237,2730,2731,2733,2735,2737,2739,2741,2743,2745,2747,2749,2751,2753,2755],{"class":239,"line":291},[237,2732,1281],{"class":452},[237,2734,2671],{"class":349},[237,2736,410],{"class":246},[237,2738,2676],{"class":349},[237,2740,1292],{"class":338},[237,2742,1106],{"class":584},[237,2744,2683],{"class":338},[237,2746,2671],{"class":349},[237,2748,410],{"class":246},[237,2750,2690],{"class":349},[237,2752,2302],{"class":338},[237,2754,1106],{"class":584},[237,2756,247],{"class":246},[237,2758,2759,2761,2763,2765,2767,2769,2771,2773,2775,2778,2780,2782,2785,2787],{"class":239,"line":306},[237,2760,1312],{"class":452},[237,2762,436],{"class":406},[237,2764,410],{"class":246},[237,2766,2642],{"class":349},[237,2768,346],{"class":246},[237,2770,672],{"class":349},[237,2772,121],{"class":246},[237,2774,264],{"class":263},[237,2776,2777],{"class":267},"crash loop: 0 running, desired > 0",[237,2779,284],{"class":263},[237,2781,121],{"class":246},[237,2783,2784],{"class":406}," BLOCKED_CRASH_LOOP",[237,2786,581],{"class":246},[237,2788,2289],{"class":452},[34,2790,2791],{},"Scaling a crash-looping service up just creates more crashing containers. The engine refuses.",[2606,2793,2795],{"id":2794},"step-2-emergency-check-bypasses-cooldown","Step 2: Emergency check (bypasses cooldown)",[34,2797,2798],{},"Hard ceilings fire regardless of cooldown. A container at 96% max CPU can't wait two minutes:",[228,2800,2802],{"className":329,"code":2801,"language":331,"meta":233,"style":233},"if metrics.max_cpu > self._EMERGENCY_CPU_HARD_CEILING:   # e.g. 95%\n    emergency_trigger_names.add(TRIGGER_MAX_CPU)\nif metrics.max_mem > self._EMERGENCY_MEM_HARD_CEILING:   # e.g. 92%\n    emergency_trigger_names.add(TRIGGER_MAX_MEM)\n",[48,2803,2804,2829,2846,2871],{"__ignoreMap":233},[237,2805,2806,2808,2810,2812,2815,2817,2819,2821,2824,2826],{"class":239,"line":11},[237,2807,1281],{"class":452},[237,2809,2671],{"class":349},[237,2811,410],{"class":246},[237,2813,2814],{"class":349},"max_cpu ",[237,2816,2302],{"class":338},[237,2818,436],{"class":406},[237,2820,410],{"class":246},[237,2822,2823],{"class":406},"_EMERGENCY_CPU_HARD_CEILING",[237,2825,1504],{"class":246},[237,2827,2828],{"class":287},"   # e.g. 95%\n",[237,2830,2831,2834,2836,2839,2841,2844],{"class":239,"line":20},[237,2832,2833],{"class":349},"    emergency_trigger_names",[237,2835,410],{"class":246},[237,2837,2838],{"class":349},"add",[237,2840,346],{"class":246},[237,2842,2843],{"class":406},"TRIGGER_MAX_CPU",[237,2845,680],{"class":246},[237,2847,2848,2850,2852,2854,2857,2859,2861,2863,2866,2868],{"class":239,"line":257},[237,2849,1281],{"class":452},[237,2851,2671],{"class":349},[237,2853,410],{"class":246},[237,2855,2856],{"class":349},"max_mem ",[237,2858,2302],{"class":338},[237,2860,436],{"class":406},[237,2862,410],{"class":246},[237,2864,2865],{"class":406},"_EMERGENCY_MEM_HARD_CEILING",[237,2867,1504],{"class":246},[237,2869,2870],{"class":287},"   # e.g. 92%\n",[237,2872,2873,2875,2877,2879,2881,2884],{"class":239,"line":274},[237,2874,2833],{"class":349},[237,2876,410],{"class":246},[237,2878,2838],{"class":349},[237,2880,346],{"class":246},[237,2882,2883],{"class":406},"TRIGGER_MAX_MEM",[237,2885,680],{"class":246},[34,2887,2888,2889,2892],{},"The rejected-tasks case is special and important: it is the ",[37,2890,2891],{},"cluster-full"," signal:",[228,2894,2896],{"className":329,"code":2895,"language":331,"meta":233,"style":233},"if metrics.replicas_rejected > 0:\n    return ScalingDecision(..., action=NONE, blocked_by=BLOCKED_CLUSTER_FULL), True\n    #                                 ^^^^                                    ^^^^\n    #       Don't try to add containers: there's no room.    But DO raise host_scale_signal.\n",[48,2897,2898,2915,2951,2956],{"__ignoreMap":233},[237,2899,2900,2902,2904,2906,2909,2911,2913],{"class":239,"line":11},[237,2901,1281],{"class":452},[237,2903,2671],{"class":349},[237,2905,410],{"class":246},[237,2907,2908],{"class":349},"replicas_rejected ",[237,2910,2302],{"class":338},[237,2912,1106],{"class":584},[237,2914,247],{"class":246},[237,2916,2917,2919,2922,2924,2927,2929,2932,2934,2937,2939,2942,2944,2947,2949],{"class":239,"line":20},[237,2918,1312],{"class":452},[237,2920,2921],{"class":349}," ScalingDecision",[237,2923,346],{"class":246},[237,2925,2926],{"class":406},"...",[237,2928,121],{"class":246},[237,2930,2931],{"class":562}," action",[237,2933,395],{"class":246},[237,2935,2936],{"class":406},"NONE",[237,2938,121],{"class":246},[237,2940,2941],{"class":562}," blocked_by",[237,2943,395],{"class":246},[237,2945,2946],{"class":406},"BLOCKED_CLUSTER_FULL",[237,2948,581],{"class":246},[237,2950,1888],{"class":452},[237,2952,2953],{"class":239,"line":257},[237,2954,2955],{"class":287},"    #                                 ^^^^                                    ^^^^\n",[237,2957,2958],{"class":239,"line":274},[237,2959,2960],{"class":287},"    #       Don't try to add containers: there's no room.    But DO raise host_scale_signal.\n",[34,2962,2963,2964,2967,2968,2970,2971,2974],{},"When tasks are being ",[55,2965,2966],{},"rejected",", adding more replicas is pointless: the cluster has no capacity.\nSo the engine returns ",[48,2969,2936],{}," for the service but sets ",[48,2972,2973],{},"host_scale_signal = True",", telling the\nwatch loop to ask OpenStack for a new VM instead.",[34,2976,2977,2978,2980],{},"If an emergency fires and there ",[55,2979,104],{}," room, it scales by the emergency step (3) immediately,\nwaiving cooldown.",[2606,2982,2984],{"id":2983},"step-3-cooldown-check","Step 3: Cooldown check",[34,2986,2987],{},"After an ordinary scale, respect the configured quiet period so the system can stabilize before\nreacting again:",[228,2989,2991],{"className":329,"code":2990,"language":331,"meta":233,"style":233},"if last_scale_action == SCALE_UP and time_since_last_action \u003C policy.cooldown_after_up:\n    return self._none(svc, f\"cooldown after scale-up ...\", BLOCKED_COOLDOWN_UP), False\nif last_scale_action == SCALE_DOWN and time_since_last_action \u003C policy.cooldown_after_down:\n    return self._none(svc, f\"cooldown after scale-down ...\", BLOCKED_COOLDOWN_DOWN), False\n",[48,2992,2993,3021,3052,3078],{"__ignoreMap":233},[237,2994,2995,2997,3000,3002,3005,3007,3010,3012,3014,3016,3019],{"class":239,"line":11},[237,2996,1281],{"class":452},[237,2998,2999],{"class":349}," last_scale_action ",[237,3001,1292],{"class":338},[237,3003,3004],{"class":406}," SCALE_UP",[237,3006,2683],{"class":338},[237,3008,3009],{"class":349}," time_since_last_action ",[237,3011,1126],{"class":338},[237,3013,2257],{"class":349},[237,3015,410],{"class":246},[237,3017,3018],{"class":349},"cooldown_after_up",[237,3020,247],{"class":246},[237,3022,3023,3025,3027,3029,3031,3033,3035,3037,3040,3043,3045,3048,3050],{"class":239,"line":20},[237,3024,1312],{"class":452},[237,3026,436],{"class":406},[237,3028,410],{"class":246},[237,3030,2642],{"class":349},[237,3032,346],{"class":246},[237,3034,672],{"class":349},[237,3036,121],{"class":246},[237,3038,3039],{"class":338}," f",[237,3041,3042],{"class":267},"\"cooldown after scale-up ...\"",[237,3044,121],{"class":246},[237,3046,3047],{"class":406}," BLOCKED_COOLDOWN_UP",[237,3049,581],{"class":246},[237,3051,2289],{"class":452},[237,3053,3054,3056,3058,3060,3063,3065,3067,3069,3071,3073,3076],{"class":239,"line":257},[237,3055,1281],{"class":452},[237,3057,2999],{"class":349},[237,3059,1292],{"class":338},[237,3061,3062],{"class":406}," SCALE_DOWN",[237,3064,2683],{"class":338},[237,3066,3009],{"class":349},[237,3068,1126],{"class":338},[237,3070,2257],{"class":349},[237,3072,410],{"class":246},[237,3074,3075],{"class":349},"cooldown_after_down",[237,3077,247],{"class":246},[237,3079,3080,3082,3084,3086,3088,3090,3092,3094,3096,3099,3101,3104,3106],{"class":239,"line":274},[237,3081,1312],{"class":452},[237,3083,436],{"class":406},[237,3085,410],{"class":246},[237,3087,2642],{"class":349},[237,3089,346],{"class":246},[237,3091,672],{"class":349},[237,3093,121],{"class":246},[237,3095,3039],{"class":338},[237,3097,3098],{"class":267},"\"cooldown after scale-down ...\"",[237,3100,121],{"class":246},[237,3102,3103],{"class":406}," BLOCKED_COOLDOWN_DOWN",[237,3105,581],{"class":246},[237,3107,2289],{"class":452},[2606,3109,3111],{"id":3110},"steps-4-5-weighted-multi-trigger-scoring","Steps 4 & 5: Weighted multi-trigger scoring",[34,3113,3114,3115,3118,3119,3122],{},"This is the heart of scale-up. Each trigger is evaluated independently; ",[37,3116,3117],{},"any single one firing\nis enough to scale up",". But the engine also computes a weighted ",[55,3120,3121],{},"score"," that drives how\naggressively to scale (by 1, 2, or 3 replicas).",[34,3124,3125,3126,3129],{},"Each trigger contributes ",[48,3127,3128],{},"overage × weight",", where overage is how far past the threshold the\nmetric is, normalized to the headroom between threshold and ceiling:",[228,3131,3133],{"className":329,"code":3132,"language":331,"meta":233,"style":233},"def _check(trigger, condition, weight, overage):\n    if not condition:\n        return\n    if trigger in suppressed_triggers:   # per-trigger cooldown\n        suppressed.append(trigger)\n    else:\n        fired.append(trigger)\n        weighted_overage_sum += overage * weight\n\n# T1 avg_cpu   weight 0.30\n_check(TRIGGER_AVG_CPU, metrics.avg_cpu > policy.scale_up_cpu_avg, 0.30,\n       (metrics.avg_cpu - policy.scale_up_cpu_avg) \u002F (100.0 - policy.scale_up_cpu_avg))\n# T2 max_cpu   weight 0.30\n# T3 avg_mem   weight 0.30   (same as CPU: sustained high memory should reach MODERATE)\n# T4 max_mem   weight 0.25   (slightly lower: one hot container shouldn't alone drive emergency)\n# T5 cpu_throttle weight 0.30 (fires even when raw CPU% looks low)\n# T7 connections weight 0.10  (overage capped at 1.0, a 10x spike shouldn't alone hit EMERGENCY)\n# T8 request_rate weight 0.15 (only when a rate threshold is explicitly configured)\n",[48,3134,3135,3164,3174,3179,3196,3212,3219,3234,3250,3254,3259,3293,3334,3339,3344,3349,3354,3359],{"__ignoreMap":233},[237,3136,3137,3139,3142,3144,3147,3149,3152,3154,3157,3159,3162],{"class":239,"line":11},[237,3138,339],{"class":338},[237,3140,3141],{"class":342}," _check",[237,3143,346],{"class":246},[237,3145,3146],{"class":349},"trigger",[237,3148,121],{"class":246},[237,3150,3151],{"class":349}," condition",[237,3153,121],{"class":246},[237,3155,3156],{"class":349}," weight",[237,3158,121],{"class":246},[237,3160,3161],{"class":349}," overage",[237,3163,1496],{"class":246},[237,3165,3166,3168,3170,3172],{"class":239,"line":20},[237,3167,453],{"class":452},[237,3169,456],{"class":338},[237,3171,3151],{"class":349},[237,3173,247],{"class":246},[237,3175,3176],{"class":239,"line":257},[237,3177,3178],{"class":452},"        return\n",[237,3180,3181,3183,3186,3188,3191,3193],{"class":239,"line":274},[237,3182,453],{"class":452},[237,3184,3185],{"class":349}," trigger ",[237,3187,696],{"class":338},[237,3189,3190],{"class":349}," suppressed_triggers",[237,3192,1504],{"class":246},[237,3194,3195],{"class":287},"   # per-trigger cooldown\n",[237,3197,3198,3201,3203,3206,3208,3210],{"class":239,"line":291},[237,3199,3200],{"class":349},"        suppressed",[237,3202,410],{"class":246},[237,3204,3205],{"class":349},"append",[237,3207,346],{"class":246},[237,3209,3146],{"class":349},[237,3211,680],{"class":246},[237,3213,3214,3217],{"class":239,"line":306},[237,3215,3216],{"class":452},"    else",[237,3218,247],{"class":246},[237,3220,3221,3224,3226,3228,3230,3232],{"class":239,"line":464},[237,3222,3223],{"class":349},"        fired",[237,3225,410],{"class":246},[237,3227,3205],{"class":349},[237,3229,346],{"class":246},[237,3231,3146],{"class":349},[237,3233,680],{"class":246},[237,3235,3236,3239,3242,3245,3247],{"class":239,"line":473},[237,3237,3238],{"class":349},"        weighted_overage_sum ",[237,3240,3241],{"class":246},"+=",[237,3243,3244],{"class":349}," overage ",[237,3246,1149],{"class":338},[237,3248,3249],{"class":349}," weight\n",[237,3251,3252],{"class":239,"line":478},[237,3253,426],{"emptyLinePlaceholder":425},[237,3255,3256],{"class":239,"line":484},[237,3257,3258],{"class":287},"# T1 avg_cpu   weight 0.30\n",[237,3260,3261,3264,3266,3269,3271,3273,3275,3278,3280,3282,3284,3286,3288,3291],{"class":239,"line":490},[237,3262,3263],{"class":349},"_check",[237,3265,346],{"class":246},[237,3267,3268],{"class":406},"TRIGGER_AVG_CPU",[237,3270,121],{"class":246},[237,3272,2671],{"class":349},[237,3274,410],{"class":246},[237,3276,3277],{"class":349},"avg_cpu ",[237,3279,2302],{"class":338},[237,3281,2257],{"class":349},[237,3283,410],{"class":246},[237,3285,2336],{"class":349},[237,3287,121],{"class":246},[237,3289,3290],{"class":584}," 0.30",[237,3292,1365],{"class":246},[237,3294,3295,3298,3301,3303,3305,3307,3309,3311,3313,3315,3318,3320,3323,3326,3328,3330,3332],{"class":239,"line":516},[237,3296,3297],{"class":246},"       (",[237,3299,3300],{"class":349},"metrics",[237,3302,410],{"class":246},[237,3304,3277],{"class":349},[237,3306,729],{"class":338},[237,3308,2257],{"class":349},[237,3310,410],{"class":246},[237,3312,2336],{"class":349},[237,3314,353],{"class":246},[237,3316,3317],{"class":338}," \u002F",[237,3319,669],{"class":246},[237,3321,3322],{"class":584},"100.0",[237,3324,3325],{"class":338}," -",[237,3327,2257],{"class":349},[237,3329,410],{"class":246},[237,3331,2336],{"class":349},[237,3333,1276],{"class":246},[237,3335,3336],{"class":239,"line":546},[237,3337,3338],{"class":287},"# T2 max_cpu   weight 0.30\n",[237,3340,3341],{"class":239,"line":551},[237,3342,3343],{"class":287},"# T3 avg_mem   weight 0.30   (same as CPU: sustained high memory should reach MODERATE)\n",[237,3345,3346],{"class":239,"line":591},[237,3347,3348],{"class":287},"# T4 max_mem   weight 0.25   (slightly lower: one hot container shouldn't alone drive emergency)\n",[237,3350,3351],{"class":239,"line":616},[237,3352,3353],{"class":287},"# T5 cpu_throttle weight 0.30 (fires even when raw CPU% looks low)\n",[237,3355,3356],{"class":239,"line":627},[237,3357,3358],{"class":287},"# T7 connections weight 0.10  (overage capped at 1.0, a 10x spike shouldn't alone hit EMERGENCY)\n",[237,3360,3361],{"class":239,"line":683},[237,3362,3363],{"class":287},"# T8 request_rate weight 0.15 (only when a rate threshold is explicitly configured)\n",[34,3365,3366,3367,3369],{},"The pending-stuck trigger is a special case: it fires, adds its weight, ",[37,3368,1287],{}," raises the\nhost-scale signal, because stuck tasks mean the cluster is out of room:",[228,3371,3373],{"className":329,"code":3372,"language":331,"meta":233,"style":233},"if metrics.longest_pending_seconds > policy.pending_stuck_seconds:\n    fired.append(TRIGGER_PENDING_STUCK)\n    weighted_overage_sum += 0.30\n    host_scale_signal = True\n",[48,3374,3375,3397,3413,3423],{"__ignoreMap":233},[237,3376,3377,3379,3381,3383,3386,3388,3390,3392,3395],{"class":239,"line":11},[237,3378,1281],{"class":452},[237,3380,2671],{"class":349},[237,3382,410],{"class":246},[237,3384,3385],{"class":349},"longest_pending_seconds ",[237,3387,2302],{"class":338},[237,3389,2257],{"class":349},[237,3391,410],{"class":246},[237,3393,3394],{"class":349},"pending_stuck_seconds",[237,3396,247],{"class":246},[237,3398,3399,3402,3404,3406,3408,3411],{"class":239,"line":20},[237,3400,3401],{"class":349},"    fired",[237,3403,410],{"class":246},[237,3405,3205],{"class":349},[237,3407,346],{"class":246},[237,3409,3410],{"class":406},"TRIGGER_PENDING_STUCK",[237,3412,680],{"class":246},[237,3414,3415,3418,3420],{"class":239,"line":257},[237,3416,3417],{"class":349},"    weighted_overage_sum ",[237,3419,3241],{"class":246},[237,3421,3422],{"class":584}," 0.30\n",[237,3424,3425,3428,3430],{"class":239,"line":274},[237,3426,3427],{"class":349},"    host_scale_signal ",[237,3429,395],{"class":246},[237,3431,1888],{"class":452},[34,3433,3434],{},"The score has a built-in floor, so any fired trigger yields at least a NORMAL scale:",[228,3436,3438],{"className":329,"code":3437,"language":331,"meta":233,"style":233},"score = min(1.0, self._SCORE_BASE + weighted_overage_sum)   # SCORE_BASE = 0.30\n\nurgency, step = Urgency.NORMAL, 1\nif score >= self._SCORE_EMERGENCY_THRESHOLD:   # 0.75 → EMERGENCY, step 3\n    urgency, step = Urgency.EMERGENCY, 3\nelif score >= self._SCORE_MODERATE_THRESHOLD:  # 0.45 → MODERATE, step 2\n    urgency, step = Urgency.MODERATE, 2\n",[48,3439,3440,3474,3478,3503,3524,3547,3568],{"__ignoreMap":233},[237,3441,3442,3445,3447,3450,3452,3455,3457,3459,3461,3464,3466,3469,3471],{"class":239,"line":11},[237,3443,3444],{"class":349},"score ",[237,3446,395],{"class":246},[237,3448,3449],{"class":242}," min",[237,3451,346],{"class":246},[237,3453,3454],{"class":584},"1.0",[237,3456,121],{"class":246},[237,3458,436],{"class":406},[237,3460,410],{"class":246},[237,3462,3463],{"class":406},"_SCORE_BASE",[237,3465,535],{"class":338},[237,3467,3468],{"class":349}," weighted_overage_sum",[237,3470,353],{"class":246},[237,3472,3473],{"class":287},"   # SCORE_BASE = 0.30\n",[237,3475,3476],{"class":239,"line":20},[237,3477,426],{"emptyLinePlaceholder":425},[237,3479,3480,3483,3485,3488,3490,3493,3495,3498,3500],{"class":239,"line":257},[237,3481,3482],{"class":349},"urgency",[237,3484,121],{"class":246},[237,3486,3487],{"class":349}," step ",[237,3489,395],{"class":246},[237,3491,3492],{"class":349}," Urgency",[237,3494,410],{"class":246},[237,3496,3497],{"class":406},"NORMAL",[237,3499,121],{"class":246},[237,3501,3502],{"class":584}," 1\n",[237,3504,3505,3507,3510,3512,3514,3516,3519,3521],{"class":239,"line":274},[237,3506,1281],{"class":452},[237,3508,3509],{"class":349}," score ",[237,3511,2329],{"class":338},[237,3513,436],{"class":406},[237,3515,410],{"class":246},[237,3517,3518],{"class":406},"_SCORE_EMERGENCY_THRESHOLD",[237,3520,1504],{"class":246},[237,3522,3523],{"class":287},"   # 0.75 → EMERGENCY, step 3\n",[237,3525,3526,3529,3531,3533,3535,3537,3539,3542,3544],{"class":239,"line":291},[237,3527,3528],{"class":349},"    urgency",[237,3530,121],{"class":246},[237,3532,3487],{"class":349},[237,3534,395],{"class":246},[237,3536,3492],{"class":349},[237,3538,410],{"class":246},[237,3540,3541],{"class":406},"EMERGENCY",[237,3543,121],{"class":246},[237,3545,3546],{"class":584}," 3\n",[237,3548,3549,3552,3554,3556,3558,3560,3563,3565],{"class":239,"line":306},[237,3550,3551],{"class":452},"elif",[237,3553,3509],{"class":349},[237,3555,2329],{"class":338},[237,3557,436],{"class":406},[237,3559,410],{"class":246},[237,3561,3562],{"class":406},"_SCORE_MODERATE_THRESHOLD",[237,3564,1504],{"class":246},[237,3566,3567],{"class":287},"  # 0.45 → MODERATE, step 2\n",[237,3569,3570,3572,3574,3576,3578,3580,3582,3585,3587],{"class":239,"line":464},[237,3571,3528],{"class":349},[237,3573,121],{"class":246},[237,3575,3487],{"class":349},[237,3577,395],{"class":246},[237,3579,3492],{"class":349},[237,3581,410],{"class":246},[237,3583,3584],{"class":406},"MODERATE",[237,3586,121],{"class":246},[237,3588,3589],{"class":584}," 2\n",[34,3591,3592,3593,3596,3597,3600,3601,3604],{},"The defaults (tunable in config): a floor of ",[48,3594,3595],{},"0.30"," so any single fired trigger yields at least a\nNORMAL +1; ",[48,3598,3599],{},"0.45"," crosses into MODERATE +2 (one trigger well above threshold, or two triggers\nfiring together); ",[48,3602,3603],{},"0.75"," into EMERGENCY +3 (several triggers blown out at once). The combined\ntrigger weights sum to roughly 1.80, so the score saturates well before every trigger has to fire.",[34,3606,3607],{},"In short, barely over one threshold gives +1, and multiple thresholds blown out give +3. The weighting means a\nservice that is simultaneously CPU-bound, memory-bound, and throttled escalates fast, while a\nservice that merely has slightly elevated connections nudges up by one.",[34,3609,3610],{},"Play with the numbers yourself. Slide each metric past its threshold (the small tick on each slider) and watch the score and the step size change. The \"throttled, CPU looks fine\" preset is the case that makes the throttle trigger worth having.",[3612,3613],"swarm-scoring",{},[2606,3615,3617,3618],{"id":3616},"step-55-scale-up-must-be-sustained","Step 5.5: Scale-up must be ",[55,3619,3620],{},"sustained",[34,3622,3623,3624,3627],{},"A single spiky cycle shouldn't scale you up. Scale-up conditions must hold for a configured\nduration, ",[37,3625,3626],{},"except"," pending-stuck, which bypasses the timer because waiting would only make a\ncapacity shortage worse:",[228,3629,3631],{"className":329,"code":3630,"language":331,"meta":233,"style":233},"duration_waived = TRIGGER_PENDING_STUCK in fired\nif not duration_waived and elapsed_scale_up_seconds \u003C policy.scale_up_duration:\n    return self._none(svc, f\"...met for {elapsed_scale_up_seconds}s, need {policy.scale_up_duration}s\",\n                      BLOCKED_SCALE_UP_NOT_SUSTAINED), False\n",[48,3632,3633,3648,3673,3719],{"__ignoreMap":233},[237,3634,3635,3638,3640,3643,3645],{"class":239,"line":11},[237,3636,3637],{"class":349},"duration_waived ",[237,3639,395],{"class":246},[237,3641,3642],{"class":406}," TRIGGER_PENDING_STUCK",[237,3644,2013],{"class":338},[237,3646,3647],{"class":349}," fired\n",[237,3649,3650,3652,3654,3657,3659,3662,3664,3666,3668,3671],{"class":239,"line":20},[237,3651,1281],{"class":452},[237,3653,456],{"class":338},[237,3655,3656],{"class":349}," duration_waived ",[237,3658,1287],{"class":338},[237,3660,3661],{"class":349}," elapsed_scale_up_seconds ",[237,3663,1126],{"class":338},[237,3665,2257],{"class":349},[237,3667,410],{"class":246},[237,3669,3670],{"class":349},"scale_up_duration",[237,3672,247],{"class":246},[237,3674,3675,3677,3679,3681,3683,3685,3687,3689,3691,3694,3696,3699,3701,3704,3706,3708,3710,3712,3714,3717],{"class":239,"line":257},[237,3676,1312],{"class":452},[237,3678,436],{"class":406},[237,3680,410],{"class":246},[237,3682,2642],{"class":349},[237,3684,346],{"class":246},[237,3686,672],{"class":349},[237,3688,121],{"class":246},[237,3690,3039],{"class":338},[237,3692,3693],{"class":267},"\"...met for ",[237,3695,851],{"class":406},[237,3697,3698],{"class":349},"elapsed_scale_up_seconds",[237,3700,863],{"class":406},[237,3702,3703],{"class":267},"s, need ",[237,3705,851],{"class":406},[237,3707,2191],{"class":349},[237,3709,410],{"class":246},[237,3711,3670],{"class":349},[237,3713,863],{"class":406},[237,3715,3716],{"class":267},"s\"",[237,3718,1365],{"class":246},[237,3720,3721,3724,3726],{"class":239,"line":274},[237,3722,3723],{"class":406},"                      BLOCKED_SCALE_UP_NOT_SUSTAINED",[237,3725,581],{"class":246},[237,3727,2289],{"class":452},[2606,3729,3731],{"id":3730},"step-6-scale-up-bounds","Step 6: Scale-up bounds",[34,3733,3734,3735,3737],{},"If already at ",[48,3736,2309],{},", don't scale, but raise the host signal, because the service wants\nmore and the only way to give it more is a bigger cluster:",[228,3739,3741],{"className":329,"code":3740,"language":331,"meta":233,"style":233},"if current_total >= policy.max_replicas:\n    return ScalingDecision(..., action=NONE, blocked_by=BLOCKED_AT_MAX_REPLICAS), True   # host signal\nif current_total + step > policy.max_replicas:\n    step = policy.max_replicas - current_total\nif current_total + step == policy.max_replicas:\n    host_scale_signal = True   # hitting the ceiling this cycle: warn OpenStack early\n",[48,3742,3743,3760,3795,3816,3835,3855],{"__ignoreMap":233},[237,3744,3745,3747,3750,3752,3754,3756,3758],{"class":239,"line":11},[237,3746,1281],{"class":452},[237,3748,3749],{"class":349}," current_total ",[237,3751,2329],{"class":338},[237,3753,2257],{"class":349},[237,3755,410],{"class":246},[237,3757,2309],{"class":349},[237,3759,247],{"class":246},[237,3761,3762,3764,3766,3768,3770,3772,3774,3776,3778,3780,3782,3784,3787,3789,3792],{"class":239,"line":20},[237,3763,1312],{"class":452},[237,3765,2921],{"class":349},[237,3767,346],{"class":246},[237,3769,2926],{"class":406},[237,3771,121],{"class":246},[237,3773,2931],{"class":562},[237,3775,395],{"class":246},[237,3777,2936],{"class":406},[237,3779,121],{"class":246},[237,3781,2941],{"class":562},[237,3783,395],{"class":246},[237,3785,3786],{"class":406},"BLOCKED_AT_MAX_REPLICAS",[237,3788,581],{"class":246},[237,3790,3791],{"class":452}," True",[237,3793,3794],{"class":287},"   # host signal\n",[237,3796,3797,3799,3801,3804,3806,3808,3810,3812,3814],{"class":239,"line":257},[237,3798,1281],{"class":452},[237,3800,3749],{"class":349},[237,3802,3803],{"class":338},"+",[237,3805,3487],{"class":349},[237,3807,2302],{"class":338},[237,3809,2257],{"class":349},[237,3811,410],{"class":246},[237,3813,2309],{"class":349},[237,3815,247],{"class":246},[237,3817,3818,3821,3823,3825,3827,3830,3832],{"class":239,"line":274},[237,3819,3820],{"class":349},"    step ",[237,3822,395],{"class":246},[237,3824,2257],{"class":349},[237,3826,410],{"class":246},[237,3828,3829],{"class":349},"max_replicas ",[237,3831,729],{"class":338},[237,3833,3834],{"class":349}," current_total\n",[237,3836,3837,3839,3841,3843,3845,3847,3849,3851,3853],{"class":239,"line":291},[237,3838,1281],{"class":452},[237,3840,3749],{"class":349},[237,3842,3803],{"class":338},[237,3844,3487],{"class":349},[237,3846,1292],{"class":338},[237,3848,2257],{"class":349},[237,3850,410],{"class":246},[237,3852,2309],{"class":349},[237,3854,247],{"class":246},[237,3856,3857,3859,3861,3863],{"class":239,"line":306},[237,3858,3427],{"class":349},[237,3860,395],{"class":246},[237,3862,3791],{"class":452},[237,3864,3865],{"class":287},"   # hitting the ceiling this cycle: warn OpenStack early\n",[220,3867,3869],{"id":3868},"scale-down-conservative-by-design","Scale-down: conservative by design",[34,3871,3872,3873,3876],{},"Scale-up is aggressive (any trigger fires). Scale-down is the opposite: ",[37,3874,3875],{},"every"," condition must\nbe true at once, stay true for a full duration, and then survive a simulation.",[2606,3878,3880],{"id":3879},"step-7-all-conditions-simultaneously","Step 7: All conditions, simultaneously",[228,3882,3884],{"className":329,"code":3883,"language":331,"meta":233,"style":233},"response_ms_ok = metrics.traffic is None or metrics.avg_response_ms \u003C policy.scale_down_response_ms\ndown_eligible = (\n    metrics.avg_cpu \u003C policy.scale_down_cpu\n    and metrics.max_cpu \u003C policy.scale_up_cpu_max      # one hot replica blocks scale-down\n    and metrics.avg_mem \u003C policy.scale_down_mem\n    and metrics.open_connections \u003C policy.scale_down_connections\n    and response_ms_ok\n    and metrics.replicas_running > policy.min_replicas\n    and metrics.replicas_pending == 0\n    and metrics.replicas_failed == 0\n)\n",[48,3885,3886,3922,3932,3949,3972,3992,4012,4019,4038,4053,4068],{"__ignoreMap":233},[237,3887,3888,3891,3893,3895,3897,3900,3902,3904,3906,3908,3910,3913,3915,3917,3919],{"class":239,"line":11},[237,3889,3890],{"class":349},"response_ms_ok ",[237,3892,395],{"class":246},[237,3894,2671],{"class":349},[237,3896,410],{"class":246},[237,3898,3899],{"class":349},"traffic ",[237,3901,104],{"class":338},[237,3903,1273],{"class":452},[237,3905,2391],{"class":338},[237,3907,2671],{"class":349},[237,3909,410],{"class":246},[237,3911,3912],{"class":349},"avg_response_ms ",[237,3914,1126],{"class":338},[237,3916,2257],{"class":349},[237,3918,410],{"class":246},[237,3920,3921],{"class":349},"scale_down_response_ms\n",[237,3923,3924,3927,3929],{"class":239,"line":20},[237,3925,3926],{"class":349},"down_eligible ",[237,3928,395],{"class":246},[237,3930,3931],{"class":246}," (\n",[237,3933,3934,3936,3938,3940,3942,3944,3946],{"class":239,"line":257},[237,3935,2473],{"class":349},[237,3937,410],{"class":246},[237,3939,3277],{"class":349},[237,3941,1126],{"class":338},[237,3943,2257],{"class":349},[237,3945,410],{"class":246},[237,3947,3948],{"class":349},"scale_down_cpu\n",[237,3950,3951,3954,3956,3958,3960,3962,3964,3966,3969],{"class":239,"line":274},[237,3952,3953],{"class":338},"    and",[237,3955,2671],{"class":349},[237,3957,410],{"class":246},[237,3959,2814],{"class":349},[237,3961,1126],{"class":338},[237,3963,2257],{"class":349},[237,3965,410],{"class":246},[237,3967,3968],{"class":349},"scale_up_cpu_max      ",[237,3970,3971],{"class":287},"# one hot replica blocks scale-down\n",[237,3973,3974,3976,3978,3980,3983,3985,3987,3989],{"class":239,"line":291},[237,3975,3953],{"class":338},[237,3977,2671],{"class":349},[237,3979,410],{"class":246},[237,3981,3982],{"class":349},"avg_mem ",[237,3984,1126],{"class":338},[237,3986,2257],{"class":349},[237,3988,410],{"class":246},[237,3990,3991],{"class":349},"scale_down_mem\n",[237,3993,3994,3996,3998,4000,4003,4005,4007,4009],{"class":239,"line":306},[237,3995,3953],{"class":338},[237,3997,2671],{"class":349},[237,3999,410],{"class":246},[237,4001,4002],{"class":349},"open_connections ",[237,4004,1126],{"class":338},[237,4006,2257],{"class":349},[237,4008,410],{"class":246},[237,4010,4011],{"class":349},"scale_down_connections\n",[237,4013,4014,4016],{"class":239,"line":464},[237,4015,3953],{"class":338},[237,4017,4018],{"class":349}," response_ms_ok\n",[237,4020,4021,4023,4025,4027,4029,4031,4033,4035],{"class":239,"line":473},[237,4022,3953],{"class":338},[237,4024,2671],{"class":349},[237,4026,410],{"class":246},[237,4028,2676],{"class":349},[237,4030,2302],{"class":338},[237,4032,2257],{"class":349},[237,4034,410],{"class":246},[237,4036,4037],{"class":349},"min_replicas\n",[237,4039,4040,4042,4044,4046,4049,4051],{"class":239,"line":478},[237,4041,3953],{"class":338},[237,4043,2671],{"class":349},[237,4045,410],{"class":246},[237,4047,4048],{"class":349},"replicas_pending ",[237,4050,1292],{"class":338},[237,4052,1559],{"class":584},[237,4054,4055,4057,4059,4061,4064,4066],{"class":239,"line":484},[237,4056,3953],{"class":338},[237,4058,2671],{"class":349},[237,4060,410],{"class":246},[237,4062,4063],{"class":349},"replicas_failed ",[237,4065,1292],{"class":338},[237,4067,1559],{"class":584},[237,4069,4070],{"class":239,"line":490},[237,4071,680],{"class":246},[34,4073,4074],{},"Two subtleties:",[84,4076,4077,4091],{},[87,4078,4079,4082,4083,4086,4087,4090],{},[37,4080,4081],{},"Response time is only enforced when Traefik data exists."," A non-HTTP worker reports 0ms,\nwhich would always satisfy ",[48,4084,4085],{},"\u003C scale_down_response_ms"," and bias every worker toward scale-down.\nGuarding on ",[48,4088,4089],{},"metrics.traffic is None"," prevents that false signal.",[87,4092,4093,4102],{},[37,4094,4095,4098,4099,410],{},[48,4096,4097],{},"max_cpu"," is checked here, not just ",[48,4100,4101],{},"avg_cpu"," One hot replica among several idle ones can\nhide in the average. Checking the max means a single busy container blocks removal.",[2606,4104,4106],{"id":4105},"step-8-scale-down-must-also-be-sustained","Step 8: Scale-down must also be sustained",[228,4108,4110],{"className":329,"code":4109,"language":331,"meta":233,"style":233},"if elapsed_scale_down_seconds \u003C policy.scale_down_duration:\n    return self._none(svc, \"...conditions met for Xs, need Ys\", BLOCKED_SCALE_DOWN_NOT_SUSTAINED), False\n",[48,4111,4112,4130],{"__ignoreMap":233},[237,4113,4114,4116,4119,4121,4123,4125,4128],{"class":239,"line":11},[237,4115,1281],{"class":452},[237,4117,4118],{"class":349}," elapsed_scale_down_seconds ",[237,4120,1126],{"class":338},[237,4122,2257],{"class":349},[237,4124,410],{"class":246},[237,4126,4127],{"class":349},"scale_down_duration",[237,4129,247],{"class":246},[237,4131,4132,4134,4136,4138,4140,4142,4144,4146,4148,4151,4153,4155,4158,4160],{"class":239,"line":20},[237,4133,1312],{"class":452},[237,4135,436],{"class":406},[237,4137,410],{"class":246},[237,4139,2642],{"class":349},[237,4141,346],{"class":246},[237,4143,672],{"class":349},[237,4145,121],{"class":246},[237,4147,264],{"class":263},[237,4149,4150],{"class":267},"...conditions met for Xs, need Ys",[237,4152,284],{"class":263},[237,4154,121],{"class":246},[237,4156,4157],{"class":406}," BLOCKED_SCALE_DOWN_NOT_SUSTAINED",[237,4159,581],{"class":246},[237,4161,2289],{"class":452},[2606,4163,4165],{"id":4164},"step-9-proportional-step","Step 9: Proportional step",[34,4167,4168],{},"Scale-up steps by urgency; scale-down normally steps by one. But if a service is deeply idle,\nremoving one replica per cooldown period is painfully slow. So the step is proportional to how\nfar below the threshold the load is, hard-capped so you can never halve capacity in one move:",[228,4170,4172],{"className":329,"code":4171,"language":331,"meta":233,"style":233},"step = 1\nif n > policy.min_replicas + 1 and policy.scale_down_cpu > 0 and policy.scale_down_mem > 0:\n    cpu_r  = metrics.avg_cpu \u002F policy.scale_down_cpu\n    mem_r  = metrics.avg_mem \u002F policy.scale_down_mem\n    load_r = max(cpu_r, mem_r, 0.0)\n    target = max(policy.min_replicas, math.ceil(load_r * n))\n    step   = max(1, min(n - target, n - policy.min_replicas))\n    step   = min(step, self._SCALE_DOWN_MAX_STEP)   # never remove too many at once\n",[48,4173,4174,4183,4228,4249,4270,4296,4336,4377],{"__ignoreMap":233},[237,4175,4176,4179,4181],{"class":239,"line":11},[237,4177,4178],{"class":349},"step ",[237,4180,395],{"class":246},[237,4182,3502],{"class":584},[237,4184,4185,4187,4190,4192,4194,4196,4198,4200,4202,4204,4206,4208,4210,4212,4214,4216,4218,4220,4222,4224,4226],{"class":239,"line":20},[237,4186,1281],{"class":452},[237,4188,4189],{"class":349}," n ",[237,4191,2302],{"class":338},[237,4193,2257],{"class":349},[237,4195,410],{"class":246},[237,4197,2276],{"class":349},[237,4199,3803],{"class":338},[237,4201,2281],{"class":584},[237,4203,2683],{"class":338},[237,4205,2257],{"class":349},[237,4207,410],{"class":246},[237,4209,2326],{"class":349},[237,4211,2302],{"class":338},[237,4213,1106],{"class":584},[237,4215,2683],{"class":338},[237,4217,2257],{"class":349},[237,4219,410],{"class":246},[237,4221,2357],{"class":349},[237,4223,2302],{"class":338},[237,4225,1106],{"class":584},[237,4227,247],{"class":246},[237,4229,4230,4233,4235,4237,4239,4241,4243,4245,4247],{"class":239,"line":257},[237,4231,4232],{"class":349},"    cpu_r  ",[237,4234,395],{"class":246},[237,4236,2671],{"class":349},[237,4238,410],{"class":246},[237,4240,3277],{"class":349},[237,4242,1144],{"class":338},[237,4244,2257],{"class":349},[237,4246,410],{"class":246},[237,4248,3948],{"class":349},[237,4250,4251,4254,4256,4258,4260,4262,4264,4266,4268],{"class":239,"line":274},[237,4252,4253],{"class":349},"    mem_r  ",[237,4255,395],{"class":246},[237,4257,2671],{"class":349},[237,4259,410],{"class":246},[237,4261,3982],{"class":349},[237,4263,1144],{"class":338},[237,4265,2257],{"class":349},[237,4267,410],{"class":246},[237,4269,3991],{"class":349},[237,4271,4272,4275,4277,4279,4281,4284,4286,4289,4291,4294],{"class":239,"line":291},[237,4273,4274],{"class":349},"    load_r ",[237,4276,395],{"class":246},[237,4278,716],{"class":242},[237,4280,346],{"class":246},[237,4282,4283],{"class":349},"cpu_r",[237,4285,121],{"class":246},[237,4287,4288],{"class":349}," mem_r",[237,4290,121],{"class":246},[237,4292,4293],{"class":584}," 0.0",[237,4295,680],{"class":246},[237,4297,4298,4301,4303,4305,4307,4309,4311,4314,4316,4319,4321,4324,4326,4329,4331,4334],{"class":239,"line":306},[237,4299,4300],{"class":349},"    target ",[237,4302,395],{"class":246},[237,4304,716],{"class":242},[237,4306,346],{"class":246},[237,4308,2191],{"class":349},[237,4310,410],{"class":246},[237,4312,4313],{"class":349},"min_replicas",[237,4315,121],{"class":246},[237,4317,4318],{"class":349}," math",[237,4320,410],{"class":246},[237,4322,4323],{"class":349},"ceil",[237,4325,346],{"class":246},[237,4327,4328],{"class":349},"load_r ",[237,4330,1149],{"class":338},[237,4332,4333],{"class":349}," n",[237,4335,1276],{"class":246},[237,4337,4338,4341,4343,4345,4347,4349,4351,4353,4355,4358,4360,4363,4365,4367,4369,4371,4373,4375],{"class":239,"line":464},[237,4339,4340],{"class":349},"    step   ",[237,4342,395],{"class":246},[237,4344,716],{"class":242},[237,4346,346],{"class":246},[237,4348,1213],{"class":584},[237,4350,121],{"class":246},[237,4352,3449],{"class":242},[237,4354,346],{"class":246},[237,4356,4357],{"class":349},"n ",[237,4359,729],{"class":338},[237,4361,4362],{"class":349}," target",[237,4364,121],{"class":246},[237,4366,4189],{"class":349},[237,4368,729],{"class":338},[237,4370,2257],{"class":349},[237,4372,410],{"class":246},[237,4374,4313],{"class":349},[237,4376,1276],{"class":246},[237,4378,4379,4381,4383,4385,4387,4390,4392,4394,4396,4399,4401],{"class":239,"line":473},[237,4380,4340],{"class":349},[237,4382,395],{"class":246},[237,4384,3449],{"class":242},[237,4386,346],{"class":246},[237,4388,4389],{"class":349},"step",[237,4391,121],{"class":246},[237,4393,436],{"class":406},[237,4395,410],{"class":246},[237,4397,4398],{"class":406},"_SCALE_DOWN_MAX_STEP",[237,4400,353],{"class":246},[237,4402,4403],{"class":287},"   # never remove too many at once\n",[2606,4405,4407],{"id":4406},"step-10-bounds-check","Step 10: Bounds check",[34,4409,4410,4411,4413],{},"Clamp the step so the service can never drop below ",[48,4412,4313],{},", and skip the scale-down entirely if there is nothing left to remove.",[2606,4415,4417],{"id":4416},"step-11-the-removal-simulation","Step 11: The removal simulation",[34,4419,4420,4421,4424],{},"Before removing a replica, the engine ",[55,4422,4423],{},"simulates"," what happens to the survivors. This is the\nguard that stops a classic oscillation: scale down → survivors overload → immediate scale up →\ncooldown → scale down → ...",[34,4426,4427,4428,4431],{},"The crucial insight is that ",[37,4429,4430],{},"different metrics redistribute differently",", and our first\nimplementation treated them all the same, which was wrong.",[34,4433,4434,4437,4438,4441],{},[37,4435,4436],{},"CPU redistributes proportionally."," Remove a replica and the same request volume lands on\nfewer containers, so projecting ",[48,4439,4440],{},"cpu × n \u002F (n-step)"," is valid:",[228,4443,4445],{"className":329,"code":4444,"language":331,"meta":233,"style":233},"projected_cpu = (metrics.avg_cpu * n) \u002F projected_n\nif projected_cpu > policy.scale_up_cpu_avg * self._SIMULATION_SAFETY_MARGIN:   # 85%\n    return self._none(svc, \"simulation failed: projected avg cpu ...\", BLOCKED_SIM_FAILED_CPU), False\n",[48,4446,4447,4473,4503],{"__ignoreMap":233},[237,4448,4449,4452,4454,4456,4458,4460,4462,4464,4466,4468,4470],{"class":239,"line":11},[237,4450,4451],{"class":349},"projected_cpu ",[237,4453,395],{"class":246},[237,4455,669],{"class":246},[237,4457,3300],{"class":349},[237,4459,410],{"class":246},[237,4461,3277],{"class":349},[237,4463,1149],{"class":338},[237,4465,4333],{"class":349},[237,4467,353],{"class":246},[237,4469,3317],{"class":338},[237,4471,4472],{"class":349}," projected_n\n",[237,4474,4475,4477,4480,4482,4484,4486,4489,4491,4493,4495,4498,4500],{"class":239,"line":20},[237,4476,1281],{"class":452},[237,4478,4479],{"class":349}," projected_cpu ",[237,4481,2302],{"class":338},[237,4483,2257],{"class":349},[237,4485,410],{"class":246},[237,4487,4488],{"class":349},"scale_up_cpu_avg ",[237,4490,1149],{"class":338},[237,4492,436],{"class":406},[237,4494,410],{"class":246},[237,4496,4497],{"class":406},"_SIMULATION_SAFETY_MARGIN",[237,4499,1504],{"class":246},[237,4501,4502],{"class":287},"   # 85%\n",[237,4504,4505,4507,4509,4511,4513,4515,4517,4519,4521,4524,4526,4528,4531,4533],{"class":239,"line":257},[237,4506,1312],{"class":452},[237,4508,436],{"class":406},[237,4510,410],{"class":246},[237,4512,2642],{"class":349},[237,4514,346],{"class":246},[237,4516,672],{"class":349},[237,4518,121],{"class":246},[237,4520,264],{"class":263},[237,4522,4523],{"class":267},"simulation failed: projected avg cpu ...",[237,4525,284],{"class":263},[237,4527,121],{"class":246},[237,4529,4530],{"class":406}," BLOCKED_SIM_FAILED_CPU",[237,4532,581],{"class":246},[237,4534,2289],{"class":452},[34,4536,4537,4540,4541,4544,4545,4548],{},[37,4538,4539],{},"Memory does NOT redistribute."," A container's heap, page cache, and mmapped files are\nprocess-private: they don't transfer to siblings when you remove one. The old formula\n",[48,4542,4543],{},"avg_mem × n \u002F (n-step)"," treated memory like a liquid and made scale-down nearly impossible for\nany memory-heavy service. The fix is to check that ",[55,4546,4547],{},"current"," memory already has headroom below\nthe scale-up ceiling:",[228,4550,4552],{"className":329,"code":4551,"language":331,"meta":233,"style":233},"# memory is process-private, not redistributed\nif metrics.avg_mem > policy.scale_up_mem_avg * self._SIMULATION_SAFETY_MARGIN:\n    return self._none(svc, \"simulation failed: current avg mem ...\", BLOCKED_SIM_FAILED_MEM), False\n",[48,4553,4554,4559,4588],{"__ignoreMap":233},[237,4555,4556],{"class":239,"line":11},[237,4557,4558],{"class":287},"# memory is process-private, not redistributed\n",[237,4560,4561,4563,4565,4567,4569,4571,4573,4575,4578,4580,4582,4584,4586],{"class":239,"line":20},[237,4562,1281],{"class":452},[237,4564,2671],{"class":349},[237,4566,410],{"class":246},[237,4568,3982],{"class":349},[237,4570,2302],{"class":338},[237,4572,2257],{"class":349},[237,4574,410],{"class":246},[237,4576,4577],{"class":349},"scale_up_mem_avg ",[237,4579,1149],{"class":338},[237,4581,436],{"class":406},[237,4583,410],{"class":246},[237,4585,4497],{"class":406},[237,4587,247],{"class":246},[237,4589,4590,4592,4594,4596,4598,4600,4602,4604,4606,4609,4611,4613,4616,4618],{"class":239,"line":257},[237,4591,1312],{"class":452},[237,4593,436],{"class":406},[237,4595,410],{"class":246},[237,4597,2642],{"class":349},[237,4599,346],{"class":246},[237,4601,672],{"class":349},[237,4603,121],{"class":246},[237,4605,264],{"class":263},[237,4607,4608],{"class":267},"simulation failed: current avg mem ...",[237,4610,284],{"class":263},[237,4612,121],{"class":246},[237,4614,4615],{"class":406}," BLOCKED_SIM_FAILED_MEM",[237,4617,581],{"class":246},[237,4619,2289],{"class":452},[34,4621,4622,4625,4626,4629],{},[37,4623,4624],{},"Response time is non-linear."," Latency is governed by queue depth and upstream bottlenecks\n(DB, cache, network), not simply by ",[48,4627,4628],{},"1\u002Fn"," servers. Projecting it proportionally is meaningless:\nif a database is the bottleneck, removing a replica doesn't change it. So again the engine checks\nfor genuine current headroom rather than projecting:",[228,4631,4633],{"className":329,"code":4632,"language":331,"meta":233,"style":233},"if metrics.traffic is not None and \\\n   metrics.avg_response_ms > policy.scale_up_response_ms * self._SIMULATION_SAFETY_MARGIN:\n    return self._none(svc, \"simulation failed: current response ...\", BLOCKED_SIM_FAILED_RESPONSE), False\n",[48,4634,4635,4656,4684],{"__ignoreMap":233},[237,4636,4637,4639,4641,4643,4645,4647,4649,4651,4653],{"class":239,"line":11},[237,4638,1281],{"class":452},[237,4640,2671],{"class":349},[237,4642,410],{"class":246},[237,4644,3899],{"class":349},[237,4646,104],{"class":338},[237,4648,456],{"class":338},[237,4650,1273],{"class":452},[237,4652,2683],{"class":338},[237,4654,4655],{"class":246}," \\\n",[237,4657,4658,4661,4663,4665,4667,4669,4671,4674,4676,4678,4680,4682],{"class":239,"line":20},[237,4659,4660],{"class":349},"   metrics",[237,4662,410],{"class":246},[237,4664,3912],{"class":349},[237,4666,2302],{"class":338},[237,4668,2257],{"class":349},[237,4670,410],{"class":246},[237,4672,4673],{"class":349},"scale_up_response_ms ",[237,4675,1149],{"class":338},[237,4677,436],{"class":406},[237,4679,410],{"class":246},[237,4681,4497],{"class":406},[237,4683,247],{"class":246},[237,4685,4686,4688,4690,4692,4694,4696,4698,4700,4702,4705,4707,4709,4712,4714],{"class":239,"line":257},[237,4687,1312],{"class":452},[237,4689,436],{"class":406},[237,4691,410],{"class":246},[237,4693,2642],{"class":349},[237,4695,346],{"class":246},[237,4697,672],{"class":349},[237,4699,121],{"class":246},[237,4701,264],{"class":263},[237,4703,4704],{"class":267},"simulation failed: current response ...",[237,4706,284],{"class":263},[237,4708,121],{"class":246},[237,4710,4711],{"class":406}," BLOCKED_SIM_FAILED_RESPONSE",[237,4713,581],{"class":246},[237,4715,2289],{"class":452},[34,4717,4718,4719,410],{},"If every projection passes, the decision is finally a ",[48,4720,4721],{},"SCALE_DOWN",[34,4723,4724],{},"Try it. The default preset is a service that looks idle enough to shrink but would overload the survivors. Then try the memory-heavy preset, where the old formula would have wrongly refused.",[4726,4727],"swarm-scale-down-sim",{},[220,4729,4731],{"id":4730},"the-host_scale_signal","The host_scale_signal",[34,4733,4734,4735,4738,4739,4742],{},"Threaded through the whole engine is a second return value: ",[48,4736,4737],{},"host_scale_signal",". It is set\n",[48,4740,4741],{},"True"," whenever the service wants capacity the cluster can't give:",[84,4744,4745,4751,4754],{},[87,4746,4747,4748,581],{},"tasks are being rejected (",[48,4749,4750],{},"replicas_rejected > 0",[87,4752,4753],{},"tasks are stuck pending past the threshold,",[87,4755,4756,4757,410],{},"triggers are firing but the service is already at ",[48,4758,2309],{},[34,4760,4761,4762,4765,4766,4768],{},"This signal is deliberately ",[37,4763,4764],{},"separate"," from the ",[48,4767,2581],{},". The decision is about\ncontainers; the signal is about VMs. The watch loop reads the signal and forwards it to the\nOpenStack ASG.",[167,4770],{},[59,4772,4774],{"id":4773},"part-3-the-executor","Part 3: The Executor",[34,4776,4777],{},"The executor has exactly one job: make a decision real, verify it, and report honestly. No\ndecisions of its own, no retries.",[220,4779,4781],{"id":4780},"calculate-the-target-then-skip-if-its-a-no-op","Calculate the target, then skip if it's a no-op",[228,4783,4785],{"className":329,"code":4784,"language":331,"meta":233,"style":233},"if decision.action == ScalingAction.NONE:\n    return None   # the watch loop handles NONE logging; executor never touches Docker for NONE\n\ntarget = self._calculate_target(decision, metrics, policy)\nif target == metrics.replicas_running + metrics.replicas_pending:\n    return None   # nothing to do\n",[48,4786,4787,4809,4818,4822,4851,4877],{"__ignoreMap":233},[237,4788,4789,4791,4794,4796,4799,4801,4803,4805,4807],{"class":239,"line":11},[237,4790,1281],{"class":452},[237,4792,4793],{"class":349}," decision",[237,4795,410],{"class":246},[237,4797,4798],{"class":349},"action ",[237,4800,1292],{"class":338},[237,4802,2514],{"class":349},[237,4804,410],{"class":246},[237,4806,2936],{"class":406},[237,4808,247],{"class":246},[237,4810,4811,4813,4815],{"class":239,"line":20},[237,4812,1312],{"class":452},[237,4814,1273],{"class":452},[237,4816,4817],{"class":287},"   # the watch loop handles NONE logging; executor never touches Docker for NONE\n",[237,4819,4820],{"class":239,"line":257},[237,4821,426],{"emptyLinePlaceholder":425},[237,4823,4824,4827,4829,4831,4833,4836,4838,4841,4843,4845,4847,4849],{"class":239,"line":274},[237,4825,4826],{"class":349},"target ",[237,4828,395],{"class":246},[237,4830,436],{"class":406},[237,4832,410],{"class":246},[237,4834,4835],{"class":349},"_calculate_target",[237,4837,346],{"class":246},[237,4839,4840],{"class":349},"decision",[237,4842,121],{"class":246},[237,4844,2671],{"class":349},[237,4846,121],{"class":246},[237,4848,2257],{"class":349},[237,4850,680],{"class":246},[237,4852,4853,4855,4858,4860,4862,4864,4866,4868,4870,4872,4875],{"class":239,"line":291},[237,4854,1281],{"class":452},[237,4856,4857],{"class":349}," target ",[237,4859,1292],{"class":338},[237,4861,2671],{"class":349},[237,4863,410],{"class":246},[237,4865,2676],{"class":349},[237,4867,3803],{"class":338},[237,4869,2671],{"class":349},[237,4871,410],{"class":246},[237,4873,4874],{"class":349},"replicas_pending",[237,4876,247],{"class":246},[237,4878,4879,4881,4883],{"class":239,"line":306},[237,4880,1312],{"class":452},[237,4882,1273],{"class":452},[237,4884,4885],{"class":287},"   # nothing to do\n",[220,4887,4889],{"id":4888},"dont-fight-an-active-deployment","Don't fight an active deployment",[34,4891,4892,4893,4895],{},"A subtle production hazard: if an operator is mid ",[48,4894,92],{}," (a rolling update),\nscaling the service at the same time corrupts the rollout. The executor checks for an in-progress\nupdate and backs off, unless the decision is a forced ghost-recovery:",[228,4897,4899],{"className":329,"code":4898,"language":331,"meta":233,"style":233},"update_status = service.attrs.get(\"UpdateStatus\", {})\nis_updating   = update_status.get(\"State\") == \"updating\"\nif is_updating and not decision.force:\n    return self._result(..., success=False, error=\"active_deployment\")\n",[48,4900,4901,4933,4967,4987],{"__ignoreMap":233},[237,4902,4903,4906,4908,4910,4912,4915,4917,4919,4921,4923,4926,4928,4930],{"class":239,"line":11},[237,4904,4905],{"class":349},"update_status ",[237,4907,395],{"class":246},[237,4909,1779],{"class":349},[237,4911,410],{"class":246},[237,4913,4914],{"class":349},"attrs",[237,4916,410],{"class":246},[237,4918,1061],{"class":349},[237,4920,346],{"class":246},[237,4922,284],{"class":263},[237,4924,4925],{"class":267},"UpdateStatus",[237,4927,284],{"class":263},[237,4929,121],{"class":246},[237,4931,4932],{"class":246}," {})\n",[237,4934,4935,4938,4940,4943,4945,4947,4949,4951,4954,4956,4958,4960,4962,4965],{"class":239,"line":20},[237,4936,4937],{"class":349},"is_updating   ",[237,4939,395],{"class":246},[237,4941,4942],{"class":349}," update_status",[237,4944,410],{"class":246},[237,4946,1061],{"class":349},[237,4948,346],{"class":246},[237,4950,284],{"class":263},[237,4952,4953],{"class":267},"State",[237,4955,284],{"class":263},[237,4957,353],{"class":246},[237,4959,2091],{"class":338},[237,4961,264],{"class":263},[237,4963,4964],{"class":267},"updating",[237,4966,271],{"class":263},[237,4968,4969,4971,4974,4976,4978,4980,4982,4985],{"class":239,"line":257},[237,4970,1281],{"class":452},[237,4972,4973],{"class":349}," is_updating ",[237,4975,1287],{"class":338},[237,4977,456],{"class":338},[237,4979,4793],{"class":349},[237,4981,410],{"class":246},[237,4983,4984],{"class":349},"force",[237,4986,247],{"class":246},[237,4988,4989,4991,4993,4995,4998,5000,5002,5004,5007,5009,5012,5014,5017,5019,5021,5024,5026],{"class":239,"line":274},[237,4990,1312],{"class":452},[237,4992,436],{"class":406},[237,4994,410],{"class":246},[237,4996,4997],{"class":349},"_result",[237,4999,346],{"class":246},[237,5001,2926],{"class":406},[237,5003,121],{"class":246},[237,5005,5006],{"class":562}," success",[237,5008,395],{"class":246},[237,5010,5011],{"class":452},"False",[237,5013,121],{"class":246},[237,5015,5016],{"class":562}," error",[237,5018,395],{"class":246},[237,5020,284],{"class":263},[237,5022,5023],{"class":267},"active_deployment",[237,5025,284],{"class":263},[237,5027,680],{"class":246},[220,5029,5031],{"id":5030},"scale-and-classify-failures-precisely","Scale and classify failures precisely",[228,5033,5035],{"className":329,"code":5034,"language":331,"meta":233,"style":233},"try:\n    service = self.client.services.get(metrics.service)\n    service.scale(target)\nexcept docker.errors.NotFound:\n    return self._result(..., error=EXEC_ERROR_NOT_FOUND)    # service removed between eval & exec\nexcept Exception as exc:\n    return self._result(..., error=str(exc))\n",[48,5036,5037,5044,5077,5093,5112,5140,5154],{"__ignoreMap":233},[237,5038,5039,5042],{"class":239,"line":11},[237,5040,5041],{"class":452},"try",[237,5043,247],{"class":246},[237,5045,5046,5049,5051,5053,5055,5058,5060,5063,5065,5067,5069,5071,5073,5075],{"class":239,"line":20},[237,5047,5048],{"class":349},"    service ",[237,5050,395],{"class":246},[237,5052,436],{"class":406},[237,5054,410],{"class":246},[237,5056,5057],{"class":349},"client",[237,5059,410],{"class":246},[237,5061,5062],{"class":349},"services",[237,5064,410],{"class":246},[237,5066,1061],{"class":349},[237,5068,346],{"class":246},[237,5070,3300],{"class":349},[237,5072,410],{"class":246},[237,5074,1856],{"class":349},[237,5076,680],{"class":246},[237,5078,5079,5081,5083,5086,5088,5091],{"class":239,"line":257},[237,5080,1501],{"class":349},[237,5082,410],{"class":246},[237,5084,5085],{"class":349},"scale",[237,5087,346],{"class":246},[237,5089,5090],{"class":349},"target",[237,5092,680],{"class":246},[237,5094,5095,5097,5100,5102,5105,5107,5110],{"class":239,"line":274},[237,5096,3626],{"class":452},[237,5098,5099],{"class":349}," docker",[237,5101,410],{"class":246},[237,5103,5104],{"class":349},"errors",[237,5106,410],{"class":246},[237,5108,5109],{"class":349},"NotFound",[237,5111,247],{"class":246},[237,5113,5114,5116,5118,5120,5122,5124,5126,5128,5130,5132,5135,5137],{"class":239,"line":291},[237,5115,1312],{"class":452},[237,5117,436],{"class":406},[237,5119,410],{"class":246},[237,5121,4997],{"class":349},[237,5123,346],{"class":246},[237,5125,2926],{"class":406},[237,5127,121],{"class":246},[237,5129,5016],{"class":562},[237,5131,395],{"class":246},[237,5133,5134],{"class":406},"EXEC_ERROR_NOT_FOUND",[237,5136,353],{"class":246},[237,5138,5139],{"class":287},"    # service removed between eval & exec\n",[237,5141,5142,5144,5147,5149,5152],{"class":239,"line":306},[237,5143,3626],{"class":452},[237,5145,5146],{"class":242}," Exception",[237,5148,608],{"class":452},[237,5150,5151],{"class":349}," exc",[237,5153,247],{"class":246},[237,5155,5156,5158,5160,5162,5164,5166,5168,5170,5172,5174,5176,5178,5181],{"class":239,"line":464},[237,5157,1312],{"class":452},[237,5159,436],{"class":406},[237,5161,410],{"class":246},[237,5163,4997],{"class":349},[237,5165,346],{"class":246},[237,5167,2926],{"class":406},[237,5169,121],{"class":246},[237,5171,5016],{"class":562},[237,5173,395],{"class":246},[237,5175,379],{"class":242},[237,5177,346],{"class":246},[237,5179,5180],{"class":349},"exc",[237,5182,1276],{"class":246},[220,5184,5186],{"id":5185},"verify-by-polling-partial-success-is-a-real-outcome","Verify by polling: partial success is a real outcome",[34,5188,5189,5190,5193],{},"Calling ",[48,5191,5192],{},"scale()"," returns immediately; it does not mean the replicas are running. The executor\npolls until the running count matches the target, or a rejection appears, or it times out:",[228,5195,5197],{"className":329,"code":5196,"language":331,"meta":233,"style":233},"while time.time() \u003C deadline:\n    time.sleep(_POLL_INTERVAL)\n    running, rejected = self._count_task_states(service_name)\n\n    if rejected > 0:\n        return self._result(..., replicas_after=running, success=False, error=EXEC_ERROR_REJECTED)\n    if running == target:\n        return self._result(..., replicas_after=running, success=True)\n\n# timeout: record what actually happened, not what we asked for\nrunning, _ = self._count_task_states(service_name)\nreturn self._result(..., replicas_after=running, success=False, error=EXEC_ERROR_TIMEOUT)\n",[48,5198,5199,5220,5237,5263,5267,5279,5322,5335,5367,5371,5376,5399],{"__ignoreMap":233},[237,5200,5201,5204,5206,5208,5211,5213,5215,5218],{"class":239,"line":11},[237,5202,5203],{"class":452},"while",[237,5205,524],{"class":349},[237,5207,410],{"class":246},[237,5209,5210],{"class":349},"time",[237,5212,532],{"class":246},[237,5214,1120],{"class":338},[237,5216,5217],{"class":349}," deadline",[237,5219,247],{"class":246},[237,5221,5222,5225,5227,5230,5232,5235],{"class":239,"line":20},[237,5223,5224],{"class":349},"    time",[237,5226,410],{"class":246},[237,5228,5229],{"class":349},"sleep",[237,5231,346],{"class":246},[237,5233,5234],{"class":406},"_POLL_INTERVAL",[237,5236,680],{"class":246},[237,5238,5239,5242,5244,5247,5249,5251,5253,5256,5258,5261],{"class":239,"line":257},[237,5240,5241],{"class":349},"    running",[237,5243,121],{"class":246},[237,5245,5246],{"class":349}," rejected ",[237,5248,395],{"class":246},[237,5250,436],{"class":406},[237,5252,410],{"class":246},[237,5254,5255],{"class":349},"_count_task_states",[237,5257,346],{"class":246},[237,5259,5260],{"class":349},"service_name",[237,5262,680],{"class":246},[237,5264,5265],{"class":239,"line":274},[237,5266,426],{"emptyLinePlaceholder":425},[237,5268,5269,5271,5273,5275,5277],{"class":239,"line":291},[237,5270,453],{"class":452},[237,5272,5246],{"class":349},[237,5274,2302],{"class":338},[237,5276,1106],{"class":584},[237,5278,247],{"class":246},[237,5280,5281,5283,5285,5287,5289,5291,5293,5295,5298,5300,5303,5305,5307,5309,5311,5313,5315,5317,5320],{"class":239,"line":306},[237,5282,467],{"class":452},[237,5284,436],{"class":406},[237,5286,410],{"class":246},[237,5288,4997],{"class":349},[237,5290,346],{"class":246},[237,5292,2926],{"class":406},[237,5294,121],{"class":246},[237,5296,5297],{"class":562}," replicas_after",[237,5299,395],{"class":246},[237,5301,5302],{"class":349},"running",[237,5304,121],{"class":246},[237,5306,5006],{"class":562},[237,5308,395],{"class":246},[237,5310,5011],{"class":452},[237,5312,121],{"class":246},[237,5314,5016],{"class":562},[237,5316,395],{"class":246},[237,5318,5319],{"class":406},"EXEC_ERROR_REJECTED",[237,5321,680],{"class":246},[237,5323,5324,5326,5329,5331,5333],{"class":239,"line":464},[237,5325,453],{"class":452},[237,5327,5328],{"class":349}," running ",[237,5330,1292],{"class":338},[237,5332,4362],{"class":349},[237,5334,247],{"class":246},[237,5336,5337,5339,5341,5343,5345,5347,5349,5351,5353,5355,5357,5359,5361,5363,5365],{"class":239,"line":473},[237,5338,467],{"class":452},[237,5340,436],{"class":406},[237,5342,410],{"class":246},[237,5344,4997],{"class":349},[237,5346,346],{"class":246},[237,5348,2926],{"class":406},[237,5350,121],{"class":246},[237,5352,5297],{"class":562},[237,5354,395],{"class":246},[237,5356,5302],{"class":349},[237,5358,121],{"class":246},[237,5360,5006],{"class":562},[237,5362,395],{"class":246},[237,5364,4741],{"class":452},[237,5366,680],{"class":246},[237,5368,5369],{"class":239,"line":478},[237,5370,426],{"emptyLinePlaceholder":425},[237,5372,5373],{"class":239,"line":484},[237,5374,5375],{"class":287},"# timeout: record what actually happened, not what we asked for\n",[237,5377,5378,5380,5382,5385,5387,5389,5391,5393,5395,5397],{"class":239,"line":490},[237,5379,5302],{"class":349},[237,5381,121],{"class":246},[237,5383,5384],{"class":349}," _ ",[237,5386,395],{"class":246},[237,5388,436],{"class":406},[237,5390,410],{"class":246},[237,5392,5255],{"class":349},[237,5394,346],{"class":246},[237,5396,5260],{"class":349},[237,5398,680],{"class":246},[237,5400,5401,5404,5406,5408,5410,5412,5414,5416,5418,5420,5422,5424,5426,5428,5430,5432,5434,5436,5439],{"class":239,"line":516},[237,5402,5403],{"class":452},"return",[237,5405,436],{"class":406},[237,5407,410],{"class":246},[237,5409,4997],{"class":349},[237,5411,346],{"class":246},[237,5413,2926],{"class":406},[237,5415,121],{"class":246},[237,5417,5297],{"class":562},[237,5419,395],{"class":246},[237,5421,5302],{"class":349},[237,5423,121],{"class":246},[237,5425,5006],{"class":562},[237,5427,395],{"class":246},[237,5429,5011],{"class":452},[237,5431,121],{"class":246},[237,5433,5016],{"class":562},[237,5435,395],{"class":246},[237,5437,5438],{"class":406},"EXEC_ERROR_TIMEOUT",[237,5440,680],{"class":246},[34,5442,5443,5444,5447,5448,5451,5452,5451,5455,5458,5459,5462],{},"The ",[48,5445,5446],{},"ExecutionResult"," records ",[48,5449,5450],{},"replicas_before",", ",[48,5453,5454],{},"replicas_after",[48,5456,5457],{},"requested_change",", and\n",[48,5460,5461],{},"actual_change"," separately, so a scale from 3→5 that only reached 4 is recorded faithfully as a\npartial success rather than a lie.",[220,5464,5466],{"id":5465},"no-retries-on-purpose","No retries, on purpose",[34,5468,5469],{},"The executor never retries. If a scale fails, the watch loop will re-evaluate next cycle, and if\nthe service still needs scaling the engine will fire again. Retrying inside the executor would\nrisk double-scaling during a transient blip. Let the loop's natural cadence be the retry.",[167,5471],{},[59,5473,5475],{"id":5474},"part-4-the-watch-loop","Part 4: The Watch Loop",[34,5477,5478,5479,5482],{},"The engine is pure, the executor is stateless, so ",[37,5480,5481],{},"all"," mutable state lives in the watch loop.\nIt owns the cooldown timestamps, the duration timers, the per-trigger cooldowns, and the\nrecord of what each service's target replica count should be.",[228,5484,5486],{"className":329,"code":5485,"language":331,"meta":233,"style":233},"last_scale_time:   Dict[str, float]            = {}\nlast_scale_action: Dict[str, ScalingAction]    = {}\nlast_trigger_time: Dict[str, Dict[str, float]] = {}   # svc -> {trigger -> ts}\nscale_up_start:    Dict[str, Optional[float]]  = {}    # duration timer\nscale_down_start:  Dict[str, Optional[float]]  = {}    # duration timer\ntarget_replicas:   Dict[str, int]              = audit.get_last_targets()   # hydrated from DB\n",[48,5487,5488,5514,5538,5574,5606,5636],{"__ignoreMap":233},[237,5489,5490,5493,5495,5498,5500,5502,5504,5506,5508,5511],{"class":239,"line":11},[237,5491,5492],{"class":349},"last_scale_time",[237,5494,1504],{"class":246},[237,5496,5497],{"class":349},"   Dict",[237,5499,362],{"class":246},[237,5501,379],{"class":242},[237,5503,121],{"class":246},[237,5505,1573],{"class":242},[237,5507,1637],{"class":246},[237,5509,5510],{"class":246},"            =",[237,5512,5513],{"class":246}," {}\n",[237,5515,5516,5519,5521,5523,5525,5527,5529,5531,5533,5536],{"class":239,"line":20},[237,5517,5518],{"class":349},"last_scale_action",[237,5520,1504],{"class":246},[237,5522,374],{"class":349},[237,5524,362],{"class":246},[237,5526,379],{"class":242},[237,5528,121],{"class":246},[237,5530,2514],{"class":349},[237,5532,1637],{"class":246},[237,5534,5535],{"class":246},"    =",[237,5537,5513],{"class":246},[237,5539,5540,5543,5545,5547,5549,5551,5553,5555,5557,5559,5561,5563,5566,5568,5571],{"class":239,"line":257},[237,5541,5542],{"class":349},"last_trigger_time",[237,5544,1504],{"class":246},[237,5546,374],{"class":349},[237,5548,362],{"class":246},[237,5550,379],{"class":242},[237,5552,121],{"class":246},[237,5554,374],{"class":349},[237,5556,362],{"class":246},[237,5558,379],{"class":242},[237,5560,121],{"class":246},[237,5562,1573],{"class":242},[237,5564,5565],{"class":246},"]]",[237,5567,1026],{"class":246},[237,5569,5570],{"class":246}," {}",[237,5572,5573],{"class":287},"   # svc -> {trigger -> ts}\n",[237,5575,5576,5579,5581,5584,5586,5588,5590,5592,5594,5596,5598,5601,5603],{"class":239,"line":274},[237,5577,5578],{"class":349},"scale_up_start",[237,5580,1504],{"class":246},[237,5582,5583],{"class":349},"    Dict",[237,5585,362],{"class":246},[237,5587,379],{"class":242},[237,5589,121],{"class":246},[237,5591,1629],{"class":349},[237,5593,362],{"class":246},[237,5595,1634],{"class":242},[237,5597,5565],{"class":246},[237,5599,5600],{"class":246},"  =",[237,5602,5570],{"class":246},[237,5604,5605],{"class":287},"    # duration timer\n",[237,5607,5608,5611,5613,5616,5618,5620,5622,5624,5626,5628,5630,5632,5634],{"class":239,"line":291},[237,5609,5610],{"class":349},"scale_down_start",[237,5612,1504],{"class":246},[237,5614,5615],{"class":349},"  Dict",[237,5617,362],{"class":246},[237,5619,379],{"class":242},[237,5621,121],{"class":246},[237,5623,1629],{"class":349},[237,5625,362],{"class":246},[237,5627,1634],{"class":242},[237,5629,5565],{"class":246},[237,5631,5600],{"class":246},[237,5633,5570],{"class":246},[237,5635,5605],{"class":287},[237,5637,5638,5641,5643,5645,5647,5649,5651,5653,5655,5658,5661,5663,5666,5668],{"class":239,"line":306},[237,5639,5640],{"class":349},"target_replicas",[237,5642,1504],{"class":246},[237,5644,5497],{"class":349},[237,5646,362],{"class":246},[237,5648,379],{"class":242},[237,5650,121],{"class":246},[237,5652,1554],{"class":242},[237,5654,1637],{"class":246},[237,5656,5657],{"class":246},"              =",[237,5659,5660],{"class":349}," audit",[237,5662,410],{"class":246},[237,5664,5665],{"class":349},"get_last_targets",[237,5667,532],{"class":246},[237,5669,5670],{"class":287},"   # hydrated from DB\n",[34,5672,5673,5674,5677],{},"That last line matters: the loop ",[37,5675,5676],{},"hydrates its target state from the audit log on startup",", so\na restart doesn't forget what each service's intended replica count was. Cooldown and duration\ntimers do reset on restart (the first cycle after a restart is treated as a warm-up), but the\nintended targets survive.",[220,5679,5681],{"id":5680},"the-cycle-in-three-phases","The cycle, in three phases",[228,5683,5685],{"className":329,"code":5684,"language":331,"meta":233,"style":233},"# Phase 1: evaluate every service (no blocking I\u002FO)\nfor sm, labels in all_service_data:\n    policy = policy_svc.resolve(sm.info, labels)\n    if policy is None:\n        continue\n    ... compute time_since, elapsed_up, elapsed_down, suppressed triggers ...\n    decision, host_signal = policy_svc.evaluate(sm, policy, time_since, last_action,\n                                                elapsed_up, elapsed_down, suppressed)\n    evals.append(_Eval(sm, labels, policy, decision, host_signal))\n\n# Phase 2: execute all non-NONE decisions in PARALLEL\nactive = [e for e in evals if e.decision.action != NONE]\nfutures = {_exec_pool.submit(_exec_one, ev): ev for ev in active}\n# total blocking time is max(individual poll_timeout), not the sum\n\n# Phase 3: process results, update state, write audit, update the live table\n",[48,5686,5687,5692,5710,5740,5752,5757,5783,5822,5838,5875,5879,5884,5927,5971,5976,5980],{"__ignoreMap":233},[237,5688,5689],{"class":239,"line":11},[237,5690,5691],{"class":287},"# Phase 1: evaluate every service (no blocking I\u002FO)\n",[237,5693,5694,5696,5699,5701,5703,5705,5708],{"class":239,"line":20},[237,5695,820],{"class":452},[237,5697,5698],{"class":349}," sm",[237,5700,121],{"class":246},[237,5702,693],{"class":349},[237,5704,696],{"class":452},[237,5706,5707],{"class":349}," all_service_data",[237,5709,247],{"class":246},[237,5711,5712,5714,5716,5719,5721,5724,5726,5729,5731,5734,5736,5738],{"class":239,"line":257},[237,5713,2155],{"class":349},[237,5715,395],{"class":246},[237,5717,5718],{"class":349}," policy_svc",[237,5720,410],{"class":246},[237,5722,5723],{"class":349},"resolve",[237,5725,346],{"class":246},[237,5727,5728],{"class":349},"sm",[237,5730,410],{"class":246},[237,5732,5733],{"class":349},"info",[237,5735,121],{"class":246},[237,5737,677],{"class":349},[237,5739,680],{"class":246},[237,5741,5742,5744,5746,5748,5750],{"class":239,"line":274},[237,5743,453],{"class":452},[237,5745,2177],{"class":349},[237,5747,104],{"class":338},[237,5749,1273],{"class":452},[237,5751,247],{"class":246},[237,5753,5754],{"class":239,"line":291},[237,5755,5756],{"class":452},"        continue\n",[237,5758,5759,5762,5765,5767,5770,5772,5775,5777,5780],{"class":239,"line":306},[237,5760,5761],{"class":406},"    ...",[237,5763,5764],{"class":349}," compute time_since",[237,5766,121],{"class":246},[237,5768,5769],{"class":349}," elapsed_up",[237,5771,121],{"class":246},[237,5773,5774],{"class":349}," elapsed_down",[237,5776,121],{"class":246},[237,5778,5779],{"class":349}," suppressed triggers ",[237,5781,5782],{"class":406},"...\n",[237,5784,5785,5788,5790,5793,5795,5797,5799,5802,5804,5806,5808,5810,5812,5815,5817,5820],{"class":239,"line":464},[237,5786,5787],{"class":349},"    decision",[237,5789,121],{"class":246},[237,5791,5792],{"class":349}," host_signal ",[237,5794,395],{"class":246},[237,5796,5718],{"class":349},[237,5798,410],{"class":246},[237,5800,5801],{"class":349},"evaluate",[237,5803,346],{"class":246},[237,5805,5728],{"class":349},[237,5807,121],{"class":246},[237,5809,2257],{"class":349},[237,5811,121],{"class":246},[237,5813,5814],{"class":349}," time_since",[237,5816,121],{"class":246},[237,5818,5819],{"class":349}," last_action",[237,5821,1365],{"class":246},[237,5823,5824,5827,5829,5831,5833,5836],{"class":239,"line":473},[237,5825,5826],{"class":349},"                                                elapsed_up",[237,5828,121],{"class":246},[237,5830,5774],{"class":349},[237,5832,121],{"class":246},[237,5834,5835],{"class":349}," suppressed",[237,5837,680],{"class":246},[237,5839,5840,5843,5845,5847,5849,5852,5854,5856,5858,5860,5862,5864,5866,5868,5870,5873],{"class":239,"line":478},[237,5841,5842],{"class":349},"    evals",[237,5844,410],{"class":246},[237,5846,3205],{"class":349},[237,5848,346],{"class":246},[237,5850,5851],{"class":349},"_Eval",[237,5853,346],{"class":246},[237,5855,5728],{"class":349},[237,5857,121],{"class":246},[237,5859,677],{"class":349},[237,5861,121],{"class":246},[237,5863,2257],{"class":349},[237,5865,121],{"class":246},[237,5867,4793],{"class":349},[237,5869,121],{"class":246},[237,5871,5872],{"class":349}," host_signal",[237,5874,1276],{"class":246},[237,5876,5877],{"class":239,"line":484},[237,5878,426],{"emptyLinePlaceholder":425},[237,5880,5881],{"class":239,"line":490},[237,5882,5883],{"class":287},"# Phase 2: execute all non-NONE decisions in PARALLEL\n",[237,5885,5886,5889,5891,5893,5896,5898,5901,5903,5906,5908,5911,5913,5915,5917,5919,5922,5925],{"class":239,"line":516},[237,5887,5888],{"class":349},"active ",[237,5890,395],{"class":246},[237,5892,798],{"class":246},[237,5894,5895],{"class":349},"e ",[237,5897,820],{"class":452},[237,5899,5900],{"class":349}," e ",[237,5902,696],{"class":452},[237,5904,5905],{"class":349}," evals ",[237,5907,1281],{"class":452},[237,5909,5910],{"class":349}," e",[237,5912,410],{"class":246},[237,5914,4840],{"class":349},[237,5916,410],{"class":246},[237,5918,4798],{"class":349},[237,5920,5921],{"class":338},"!=",[237,5923,5924],{"class":406}," NONE",[237,5926,830],{"class":246},[237,5928,5929,5932,5934,5937,5940,5942,5944,5946,5949,5951,5954,5956,5959,5961,5963,5965,5968],{"class":239,"line":546},[237,5930,5931],{"class":349},"futures ",[237,5933,395],{"class":246},[237,5935,5936],{"class":246}," {",[237,5938,5939],{"class":349},"_exec_pool",[237,5941,410],{"class":246},[237,5943,635],{"class":349},[237,5945,346],{"class":246},[237,5947,5948],{"class":349},"_exec_one",[237,5950,121],{"class":246},[237,5952,5953],{"class":349}," ev",[237,5955,666],{"class":246},[237,5957,5958],{"class":349}," ev ",[237,5960,820],{"class":452},[237,5962,5958],{"class":349},[237,5964,696],{"class":452},[237,5966,5967],{"class":349}," active",[237,5969,5970],{"class":246},"}\n",[237,5972,5973],{"class":239,"line":551},[237,5974,5975],{"class":287},"# total blocking time is max(individual poll_timeout), not the sum\n",[237,5977,5978],{"class":239,"line":591},[237,5979,426],{"emptyLinePlaceholder":425},[237,5981,5982],{"class":239,"line":616},[237,5983,5984],{"class":287},"# Phase 3: process results, update state, write audit, update the live table\n",[34,5986,5987],{},"Separating evaluation (pure, instant) from execution (slow, I\u002FO-bound) lets the loop fire all\nscale operations concurrently. Ten services that each need scaling cost one poll-timeout, not ten.",[220,5989,5991],{"id":5990},"managing-the-duration-timers","Managing the duration timers",[34,5993,5994,5995,5998],{},"The engine is told ",[55,5996,5997],{},"how long"," conditions have held, but it's the loop that measures it. Each\ncycle it asks the policy's own condition helpers whether the up and down conditions are currently met,\nand starts or resets a timer accordingly:",[228,6000,6002],{"className":329,"code":6001,"language":331,"meta":233,"style":233},"if policy_svc.up_conditions_met(sm, policy):\n    if scale_up_start.get(svc) is None:\n        scale_up_start[svc] = now\n    elapsed_up = int(now - scale_up_start[svc])\nelse:\n    scale_up_start[svc] = None\n    elapsed_up = 0\n",[48,6003,6004,6025,6049,6065,6089,6095,6110],{"__ignoreMap":233},[237,6005,6006,6008,6010,6012,6015,6017,6019,6021,6023],{"class":239,"line":11},[237,6007,1281],{"class":452},[237,6009,5718],{"class":349},[237,6011,410],{"class":246},[237,6013,6014],{"class":349},"up_conditions_met",[237,6016,346],{"class":246},[237,6018,5728],{"class":349},[237,6020,121],{"class":246},[237,6022,2257],{"class":349},[237,6024,1496],{"class":246},[237,6026,6027,6029,6032,6034,6036,6038,6040,6042,6045,6047],{"class":239,"line":20},[237,6028,453],{"class":452},[237,6030,6031],{"class":349}," scale_up_start",[237,6033,410],{"class":246},[237,6035,1061],{"class":349},[237,6037,346],{"class":246},[237,6039,672],{"class":349},[237,6041,353],{"class":246},[237,6043,6044],{"class":338}," is",[237,6046,1273],{"class":452},[237,6048,247],{"class":246},[237,6050,6051,6054,6056,6058,6060,6062],{"class":239,"line":257},[237,6052,6053],{"class":349},"        scale_up_start",[237,6055,362],{"class":246},[237,6057,672],{"class":349},[237,6059,1637],{"class":246},[237,6061,1026],{"class":246},[237,6063,6064],{"class":349}," now\n",[237,6066,6067,6070,6072,6074,6076,6079,6081,6083,6085,6087],{"class":239,"line":274},[237,6068,6069],{"class":349},"    elapsed_up ",[237,6071,395],{"class":246},[237,6073,1554],{"class":242},[237,6075,346],{"class":246},[237,6077,6078],{"class":349},"now ",[237,6080,729],{"class":338},[237,6082,6031],{"class":349},[237,6084,362],{"class":246},[237,6086,672],{"class":349},[237,6088,1984],{"class":246},[237,6090,6091,6093],{"class":239,"line":291},[237,6092,1163],{"class":452},[237,6094,247],{"class":246},[237,6096,6097,6100,6102,6104,6106,6108],{"class":239,"line":306},[237,6098,6099],{"class":349},"    scale_up_start",[237,6101,362],{"class":246},[237,6103,672],{"class":349},[237,6105,1637],{"class":246},[237,6107,1026],{"class":246},[237,6109,1166],{"class":452},[237,6111,6112,6114,6116],{"class":239,"line":464},[237,6113,6069],{"class":349},[237,6115,395],{"class":246},[237,6117,1559],{"class":584},[34,6119,6120,6121,1444,6123,6126,6127,6130],{},"Those ",[48,6122,6014],{},[48,6124,6125],{},"down_conditions_met"," helpers live on the ",[48,6128,6129],{},"PolicyService"," as the\nsingle source of truth, so the loop can't drift out of sync with the engine's own thresholds.",[220,6132,6134],{"id":6133},"per-trigger-cooldown","Per-trigger cooldown",[34,6136,6137,6138,6140],{},"The ordinary cooldown blocks ",[55,6139,5481],{}," scaling for a period. But imagine a service whose CPU is\nchronically a hair over threshold: it would scale +1, cool down, scale +1 again, forever, from\nthe same unresolved cause. Per-trigger cooldown suppresses an individual trigger for a while\nafter it caused a scale, while leaving the other triggers free to fire:",[228,6142,6144],{"className":329,"code":6143,"language":331,"meta":233,"style":233},"suppressed = frozenset(\n    t for t, ts in last_trigger_time.get(svc, {}).items()\n    if now - ts \u003C policy.trigger_cooldown_seconds\n)\n",[48,6145,6146,6157,6193,6213],{"__ignoreMap":233},[237,6147,6148,6151,6153,6155],{"class":239,"line":11},[237,6149,6150],{"class":349},"suppressed ",[237,6152,395],{"class":246},[237,6154,2557],{"class":242},[237,6156,2462],{"class":246},[237,6158,6159,6162,6164,6167,6169,6172,6174,6177,6179,6181,6183,6185,6187,6189,6191],{"class":239,"line":20},[237,6160,6161],{"class":349},"    t ",[237,6163,820],{"class":452},[237,6165,6166],{"class":349}," t",[237,6168,121],{"class":246},[237,6170,6171],{"class":349}," ts ",[237,6173,696],{"class":452},[237,6175,6176],{"class":349}," last_trigger_time",[237,6178,410],{"class":246},[237,6180,1061],{"class":349},[237,6182,346],{"class":246},[237,6184,672],{"class":349},[237,6186,121],{"class":246},[237,6188,1075],{"class":246},[237,6190,2021],{"class":349},[237,6192,401],{"class":246},[237,6194,6195,6197,6200,6202,6204,6206,6208,6210],{"class":239,"line":257},[237,6196,453],{"class":452},[237,6198,6199],{"class":349}," now ",[237,6201,729],{"class":338},[237,6203,6171],{"class":349},[237,6205,1126],{"class":338},[237,6207,2257],{"class":349},[237,6209,410],{"class":246},[237,6211,6212],{"class":349},"trigger_cooldown_seconds\n",[237,6214,6215],{"class":239,"line":274},[237,6216,680],{"class":246},[34,6218,6219,6220,6223],{},"Genuine emergencies (urgency EMERGENCY) are never suppressed: a real CPU fire always gets\nthrough. ",[48,6221,6222],{},"pending_stuck"," is never suppressed either, since it drives capacity signaling.",[220,6225,6227],{"id":6226},"reconciliation-and-manual-changes","Reconciliation and manual changes",[34,6229,6230,6231,6234],{},"The loop doesn't assume it's the only actor. An operator might run ",[48,6232,6233],{},"docker service scale","\nby hand. On each cycle the loop compares Swarm's actual desired count to its own remembered\ntarget and reacts intelligently:",[84,6236,6237,6247],{},[87,6238,6239,6246],{},[37,6240,6241,6242,6245],{},"Swarm has ",[55,6243,6244],{},"more"," than expected"," → an intentional manual scale-up. Accept it silently and\nadopt the new number as the target. Don't fight the operator.",[87,6248,6249,6254,6255,6257],{},[37,6250,6241,6251,6245],{},[55,6252,6253],{},"fewer"," → could be a crash, or a manual scale-down. Run the policy\nfirst. If load actually justifies the old higher count (or the service is below\n",[48,6256,4313],{},"), reconcile back up. Otherwise accept the operator's lower number.",[228,6259,6261],{"className":329,"code":6260,"language":331,"meta":233,"style":233},"if current_swarm_desired > known_target:\n    target_replicas[svc] = current_swarm_desired          # accept manual scale-up\nelse:\n    probe_decision, _ = policy_svc.evaluate(... )\n    if probe_decision.action == SCALE_UP or below_min:\n        reconcile_decision = ScalingDecision(action=SCALE_UP, by=diff, urgency=EMERGENCY, ...)\n    else:\n        target_replicas[svc] = current_swarm_desired       # accept manual scale-down\n",[48,6262,6263,6277,6296,6302,6326,6348,6392,6398],{"__ignoreMap":233},[237,6264,6265,6267,6270,6272,6275],{"class":239,"line":11},[237,6266,1281],{"class":452},[237,6268,6269],{"class":349}," current_swarm_desired ",[237,6271,2302],{"class":338},[237,6273,6274],{"class":349}," known_target",[237,6276,247],{"class":246},[237,6278,6279,6282,6284,6286,6288,6290,6293],{"class":239,"line":20},[237,6280,6281],{"class":349},"    target_replicas",[237,6283,362],{"class":246},[237,6285,672],{"class":349},[237,6287,1637],{"class":246},[237,6289,1026],{"class":246},[237,6291,6292],{"class":349}," current_swarm_desired          ",[237,6294,6295],{"class":287},"# accept manual scale-up\n",[237,6297,6298,6300],{"class":239,"line":257},[237,6299,1163],{"class":452},[237,6301,247],{"class":246},[237,6303,6304,6307,6309,6311,6313,6315,6317,6319,6321,6323],{"class":239,"line":274},[237,6305,6306],{"class":349},"    probe_decision",[237,6308,121],{"class":246},[237,6310,5384],{"class":349},[237,6312,395],{"class":246},[237,6314,5718],{"class":349},[237,6316,410],{"class":246},[237,6318,5801],{"class":349},[237,6320,346],{"class":246},[237,6322,2926],{"class":406},[237,6324,6325],{"class":246}," )\n",[237,6327,6328,6330,6333,6335,6337,6339,6341,6343,6346],{"class":239,"line":291},[237,6329,453],{"class":452},[237,6331,6332],{"class":349}," probe_decision",[237,6334,410],{"class":246},[237,6336,4798],{"class":349},[237,6338,1292],{"class":338},[237,6340,3004],{"class":406},[237,6342,2391],{"class":338},[237,6344,6345],{"class":349}," below_min",[237,6347,247],{"class":246},[237,6349,6350,6353,6355,6357,6359,6362,6364,6367,6369,6372,6374,6377,6379,6382,6384,6386,6388,6390],{"class":239,"line":306},[237,6351,6352],{"class":349},"        reconcile_decision ",[237,6354,395],{"class":246},[237,6356,2921],{"class":349},[237,6358,346],{"class":246},[237,6360,6361],{"class":562},"action",[237,6363,395],{"class":246},[237,6365,6366],{"class":406},"SCALE_UP",[237,6368,121],{"class":246},[237,6370,6371],{"class":562}," by",[237,6373,395],{"class":246},[237,6375,6376],{"class":349},"diff",[237,6378,121],{"class":246},[237,6380,6381],{"class":562}," urgency",[237,6383,395],{"class":246},[237,6385,3541],{"class":406},[237,6387,121],{"class":246},[237,6389,1416],{"class":406},[237,6391,680],{"class":246},[237,6393,6394,6396],{"class":239,"line":464},[237,6395,3216],{"class":452},[237,6397,247],{"class":246},[237,6399,6400,6403,6405,6407,6409,6411,6414],{"class":239,"line":473},[237,6401,6402],{"class":349},"        target_replicas",[237,6404,362],{"class":246},[237,6406,672],{"class":349},[237,6408,1637],{"class":246},[237,6410,1026],{"class":246},[237,6412,6413],{"class":349}," current_swarm_desired       ",[237,6415,6416],{"class":287},"# accept manual scale-down\n",[220,6418,6420],{"id":6419},"ghost-task-recovery","Ghost-task recovery",[34,6422,6423,6424,6427,6428,6431],{},"Sometimes a service ends up with ",[48,6425,6426],{},"desired > running",", a task stuck pending, and ",[37,6429,6430],{},"no"," real load\nto justify it: the classic \"ghost update\" where a new task spec can never schedule (bad\nconstraint, image that vanished). Scaling up won't help; the right move is to clear the\nunreachable pending task. The loop waits a configurable number of cycles (so slow image pulls\nand genuine rolling updates resolve themselves first), then force-scales down to clear the ghost:",[228,6433,6435],{"className":329,"code":6434,"language":331,"meta":233,"style":233},"if sm.replicas_desired > sm.replicas_running and sm.replicas_pending > 0 and load_is_idle:\n    stuck_desired_cycles[svc] += 1\n    if stuck_desired_cycles[svc] >= _s.ghost_recovery_min_cycles:\n        recovery = ScalingDecision(action=SCALE_DOWN,\n                                   by=sm.replicas_desired - sm.replicas_running,\n                                   force=True, ...)   # force bypasses the active-deployment guard\n        executor.execute(recovery, sm, ev.policy)\n",[48,6436,6437,6474,6490,6516,6535,6559,6577],{"__ignoreMap":233},[237,6438,6439,6441,6443,6445,6447,6449,6451,6453,6455,6457,6459,6461,6463,6465,6467,6469,6472],{"class":239,"line":11},[237,6440,1281],{"class":452},[237,6442,5698],{"class":349},[237,6444,410],{"class":246},[237,6446,2690],{"class":349},[237,6448,2302],{"class":338},[237,6450,5698],{"class":349},[237,6452,410],{"class":246},[237,6454,2676],{"class":349},[237,6456,1287],{"class":338},[237,6458,5698],{"class":349},[237,6460,410],{"class":246},[237,6462,4048],{"class":349},[237,6464,2302],{"class":338},[237,6466,1106],{"class":584},[237,6468,2683],{"class":338},[237,6470,6471],{"class":349}," load_is_idle",[237,6473,247],{"class":246},[237,6475,6476,6479,6481,6483,6485,6488],{"class":239,"line":20},[237,6477,6478],{"class":349},"    stuck_desired_cycles",[237,6480,362],{"class":246},[237,6482,672],{"class":349},[237,6484,1637],{"class":246},[237,6486,6487],{"class":246}," +=",[237,6489,3502],{"class":584},[237,6491,6492,6494,6497,6499,6501,6503,6506,6509,6511,6514],{"class":239,"line":257},[237,6493,453],{"class":452},[237,6495,6496],{"class":349}," stuck_desired_cycles",[237,6498,362],{"class":246},[237,6500,672],{"class":349},[237,6502,1637],{"class":246},[237,6504,6505],{"class":338}," >=",[237,6507,6508],{"class":349}," _s",[237,6510,410],{"class":246},[237,6512,6513],{"class":349},"ghost_recovery_min_cycles",[237,6515,247],{"class":246},[237,6517,6518,6521,6523,6525,6527,6529,6531,6533],{"class":239,"line":274},[237,6519,6520],{"class":349},"        recovery ",[237,6522,395],{"class":246},[237,6524,2921],{"class":349},[237,6526,346],{"class":246},[237,6528,6361],{"class":562},[237,6530,395],{"class":246},[237,6532,4721],{"class":406},[237,6534,1365],{"class":246},[237,6536,6537,6540,6542,6544,6546,6548,6550,6552,6554,6557],{"class":239,"line":291},[237,6538,6539],{"class":562},"                                   by",[237,6541,395],{"class":246},[237,6543,5728],{"class":349},[237,6545,410],{"class":246},[237,6547,2690],{"class":349},[237,6549,729],{"class":338},[237,6551,5698],{"class":349},[237,6553,410],{"class":246},[237,6555,6556],{"class":349},"replicas_running",[237,6558,1365],{"class":246},[237,6560,6561,6564,6566,6568,6570,6572,6574],{"class":239,"line":306},[237,6562,6563],{"class":562},"                                   force",[237,6565,395],{"class":246},[237,6567,4741],{"class":452},[237,6569,121],{"class":246},[237,6571,1416],{"class":406},[237,6573,353],{"class":246},[237,6575,6576],{"class":287},"   # force bypasses the active-deployment guard\n",[237,6578,6579,6582,6584,6587,6589,6592,6594,6596,6598,6600,6602,6604],{"class":239,"line":464},[237,6580,6581],{"class":349},"        executor",[237,6583,410],{"class":246},[237,6585,6586],{"class":349},"execute",[237,6588,346],{"class":246},[237,6590,6591],{"class":349},"recovery",[237,6593,121],{"class":246},[237,6595,5698],{"class":349},[237,6597,121],{"class":246},[237,6599,5953],{"class":349},[237,6601,410],{"class":246},[237,6603,2191],{"class":349},[237,6605,680],{"class":246},[220,6607,6609],{"id":6608},"writing-the-audit-trail","Writing the audit trail",[34,6611,6612,6613,6616],{},"Every evaluation is written to SQLite, ",[37,6614,6615],{},"including NONE decisions",". A NONE with a nonzero\nscore (\"almost fired\") is exactly the data you want when tuning thresholds later. The table is\nappend-only and indexed by service, timestamp, and action.",[220,6618,6620],{"id":6619},"ephemeral-node-cleanup","Ephemeral node cleanup",[34,6622,6623,6624,6627,6628,6631],{},"As a housekeeping task, the loop watches nodes labeled ",[48,6625,6626],{},"lifecycle=ephemeral"," (VMs added by the\nASG). If one stays ",[48,6629,6630],{},"down"," for enough consecutive cycles (~24h worth), it removes it from the\nswarm so the member list doesn't fill with dead OpenStack instances.",[167,6633],{},[59,6635,6637],{"id":6636},"part-5-the-openstack-integration","Part 5: The OpenStack Integration",[34,6639,6640,6641,6643,6644,6647,6648,6650],{},"This is what turns a ",[55,6642,1856],{}," autoscaler into a ",[55,6645,6646],{},"cluster"," autoscaler. ",[48,6649,50],{}," scales replicas;\nOpenStack's Heat autoscaling group (ASG) scales VMs. They talk through three channels.",[220,6652,6654],{"id":6653},"scale-out-swarmctl-asks-openstack-for-a-vm","Scale-out: swarmctl asks OpenStack for a VM",[34,6656,6657,6658,6660,6661,6664],{},"When the engine raises ",[48,6659,4737],{}," because the cluster is out of room, the watch loop\ncounts how many consecutive cycles the demand has persisted, ",[37,6662,6663],{},"per pool",", and only then fires an\nHTTP webhook at Heat:",[228,6666,6668],{"className":329,"code":6667,"language":331,"meta":233,"style":233},"for ev in evals:\n    pool = ev.labels.get(\"swarmctl.pool\") or _s.DEFAULT_POOL\n    key  = (pool, ev.sm.service)\n    is_pending_stuck = ev.sm.longest_pending_seconds > ev.policy.pending_stuck_seconds\n    if is_pending_stuck and ev.decision.action == ScalingAction.SCALE_UP:\n        svc_stuck_cycles[key] = svc_stuck_cycles.get(key, 0) + 1\n    else:\n        svc_stuck_cycles.pop(key, None)\n\n# aggregate the worst stuck-count per pool, then signal each pool over the threshold\nfor pool, max_stuck in pool_max_stuck.items():\n    if max_stuck >= _s.asg_pending_cycles:\n        asg.trigger_scale_out(pool)\n",[48,6669,6670,6683,6721,6747,6777,6806,6841,6847,6866,6870,6875,6898,6915],{"__ignoreMap":233},[237,6671,6672,6674,6676,6678,6681],{"class":239,"line":11},[237,6673,820],{"class":452},[237,6675,5958],{"class":349},[237,6677,696],{"class":452},[237,6679,6680],{"class":349}," evals",[237,6682,247],{"class":246},[237,6684,6685,6688,6690,6692,6694,6697,6699,6701,6703,6705,6708,6710,6712,6714,6716,6718],{"class":239,"line":20},[237,6686,6687],{"class":349},"    pool ",[237,6689,395],{"class":246},[237,6691,5953],{"class":349},[237,6693,410],{"class":246},[237,6695,6696],{"class":349},"labels",[237,6698,410],{"class":246},[237,6700,1061],{"class":349},[237,6702,346],{"class":246},[237,6704,284],{"class":263},[237,6706,6707],{"class":267},"swarmctl.pool",[237,6709,284],{"class":263},[237,6711,353],{"class":246},[237,6713,2391],{"class":338},[237,6715,6508],{"class":349},[237,6717,410],{"class":246},[237,6719,6720],{"class":406},"DEFAULT_POOL\n",[237,6722,6723,6726,6728,6730,6733,6735,6737,6739,6741,6743,6745],{"class":239,"line":257},[237,6724,6725],{"class":349},"    key  ",[237,6727,395],{"class":246},[237,6729,669],{"class":246},[237,6731,6732],{"class":349},"pool",[237,6734,121],{"class":246},[237,6736,5953],{"class":349},[237,6738,410],{"class":246},[237,6740,5728],{"class":349},[237,6742,410],{"class":246},[237,6744,1856],{"class":349},[237,6746,680],{"class":246},[237,6748,6749,6752,6754,6756,6758,6760,6762,6764,6766,6768,6770,6772,6774],{"class":239,"line":274},[237,6750,6751],{"class":349},"    is_pending_stuck ",[237,6753,395],{"class":246},[237,6755,5953],{"class":349},[237,6757,410],{"class":246},[237,6759,5728],{"class":349},[237,6761,410],{"class":246},[237,6763,3385],{"class":349},[237,6765,2302],{"class":338},[237,6767,5953],{"class":349},[237,6769,410],{"class":246},[237,6771,2191],{"class":349},[237,6773,410],{"class":246},[237,6775,6776],{"class":349},"pending_stuck_seconds\n",[237,6778,6779,6781,6784,6786,6788,6790,6792,6794,6796,6798,6800,6802,6804],{"class":239,"line":291},[237,6780,453],{"class":452},[237,6782,6783],{"class":349}," is_pending_stuck ",[237,6785,1287],{"class":338},[237,6787,5953],{"class":349},[237,6789,410],{"class":246},[237,6791,4840],{"class":349},[237,6793,410],{"class":246},[237,6795,4798],{"class":349},[237,6797,1292],{"class":338},[237,6799,2514],{"class":349},[237,6801,410],{"class":246},[237,6803,6366],{"class":406},[237,6805,247],{"class":246},[237,6807,6808,6811,6813,6816,6818,6820,6823,6825,6827,6829,6831,6833,6835,6837,6839],{"class":239,"line":306},[237,6809,6810],{"class":349},"        svc_stuck_cycles",[237,6812,362],{"class":246},[237,6814,6815],{"class":349},"key",[237,6817,1637],{"class":246},[237,6819,1026],{"class":246},[237,6821,6822],{"class":349}," svc_stuck_cycles",[237,6824,410],{"class":246},[237,6826,1061],{"class":349},[237,6828,346],{"class":246},[237,6830,6815],{"class":349},[237,6832,121],{"class":246},[237,6834,1106],{"class":584},[237,6836,353],{"class":246},[237,6838,535],{"class":338},[237,6840,3502],{"class":584},[237,6842,6843,6845],{"class":239,"line":464},[237,6844,3216],{"class":452},[237,6846,247],{"class":246},[237,6848,6849,6851,6853,6856,6858,6860,6862,6864],{"class":239,"line":473},[237,6850,6810],{"class":349},[237,6852,410],{"class":246},[237,6854,6855],{"class":349},"pop",[237,6857,346],{"class":246},[237,6859,6815],{"class":349},[237,6861,121],{"class":246},[237,6863,1273],{"class":452},[237,6865,680],{"class":246},[237,6867,6868],{"class":239,"line":478},[237,6869,426],{"emptyLinePlaceholder":425},[237,6871,6872],{"class":239,"line":484},[237,6873,6874],{"class":287},"# aggregate the worst stuck-count per pool, then signal each pool over the threshold\n",[237,6876,6877,6879,6881,6883,6886,6888,6891,6893,6895],{"class":239,"line":490},[237,6878,820],{"class":452},[237,6880,611],{"class":349},[237,6882,121],{"class":246},[237,6884,6885],{"class":349}," max_stuck ",[237,6887,696],{"class":452},[237,6889,6890],{"class":349}," pool_max_stuck",[237,6892,410],{"class":246},[237,6894,2021],{"class":349},[237,6896,6897],{"class":246},"():\n",[237,6899,6900,6902,6904,6906,6908,6910,6913],{"class":239,"line":516},[237,6901,453],{"class":452},[237,6903,6885],{"class":349},[237,6905,2329],{"class":338},[237,6907,6508],{"class":349},[237,6909,410],{"class":246},[237,6911,6912],{"class":349},"asg_pending_cycles",[237,6914,247],{"class":246},[237,6916,6917,6920,6922,6925,6927,6929],{"class":239,"line":546},[237,6918,6919],{"class":349},"        asg",[237,6921,410],{"class":246},[237,6923,6924],{"class":349},"trigger_scale_out",[237,6926,346],{"class":246},[237,6928,6732],{"class":349},[237,6930,680],{"class":246},[34,6932,5443,6933,6935,6936,6939,6940,6943,6944,6947,6948,410],{},[37,6934,6732],{}," concept matters. A cluster can have several ASG pools (say ",[48,6937,6938],{},"default"," and ",[48,6941,6942],{},"gpu","). A\nservice is routed to a pool by its Swarm placement constraint\n(",[48,6945,6946],{},"node.labels.pool == gpu","), the same value Swarm itself uses to schedule it, so routing can't\ndrift from reality. A stuck GPU service must grow the GPU pool, not some unrelated one. Scale-out\ndemand is therefore tracked and aggregated ",[37,6949,6663],{},[34,6951,6952,6955],{},[48,6953,6954],{},"AsgService.trigger_scale_out"," enforces a per-pool cooldown (one new VM takes time to boot and\njoin before its effect shows up) and records the scale-out timestamp, which, as you will see, the\nscale-in path reads to avoid flapping:",[228,6957,6959],{"className":329,"code":6958,"language":331,"meta":233,"style":233},"def trigger_scale_out(self, pool=None) -> bool:\n    cfg = s.get_pool(pool)\n    if cfg is None or not cfg.scale_out_url:\n        return False\n    if (now - self._last_trigger.get(pool, 0.0)) \u003C cfg.cooldown:\n        return False   # a VM is already on its way\n    response = httpx.post(cfg.scale_out_url, timeout=10.0)\n    if response.status_code in (200, 201, 202, 204):\n        self._last_trigger[pool] = now\n        with _ts_lock:\n            _last_scale_out_ts[pool] = now   # cross-thread handoff to scale-in logic\n        return True\n",[48,6960,6961,6988,7008,7033,7039,7082,7091,7126,7162,7181,7191,7210],{"__ignoreMap":233},[237,6962,6963,6965,6968,6970,6972,6974,6976,6978,6980,6982,6984,6986],{"class":239,"line":11},[237,6964,339],{"class":338},[237,6966,6967],{"class":342}," trigger_scale_out",[237,6969,346],{"class":246},[237,6971,350],{"class":349},[237,6973,121],{"class":246},[237,6975,611],{"class":349},[237,6977,395],{"class":338},[237,6979,1005],{"class":452},[237,6981,353],{"class":246},[237,6983,356],{"class":246},[237,6985,2110],{"class":242},[237,6987,247],{"class":246},[237,6989,6990,6993,6995,6997,6999,7002,7004,7006],{"class":239,"line":20},[237,6991,6992],{"class":349},"    cfg ",[237,6994,395],{"class":246},[237,6996,538],{"class":349},[237,6998,410],{"class":246},[237,7000,7001],{"class":349},"get_pool",[237,7003,346],{"class":246},[237,7005,6732],{"class":349},[237,7007,680],{"class":246},[237,7009,7010,7012,7015,7017,7019,7021,7023,7026,7028,7031],{"class":239,"line":257},[237,7011,453],{"class":452},[237,7013,7014],{"class":349}," cfg ",[237,7016,104],{"class":338},[237,7018,1273],{"class":452},[237,7020,2391],{"class":338},[237,7022,456],{"class":338},[237,7024,7025],{"class":349}," cfg",[237,7027,410],{"class":246},[237,7029,7030],{"class":349},"scale_out_url",[237,7032,247],{"class":246},[237,7034,7035,7037],{"class":239,"line":274},[237,7036,467],{"class":452},[237,7038,2289],{"class":452},[237,7040,7041,7043,7045,7047,7049,7051,7053,7056,7058,7060,7062,7064,7066,7068,7071,7073,7075,7077,7080],{"class":239,"line":291},[237,7042,453],{"class":452},[237,7044,669],{"class":246},[237,7046,6078],{"class":349},[237,7048,729],{"class":338},[237,7050,436],{"class":406},[237,7052,410],{"class":246},[237,7054,7055],{"class":349},"_last_trigger",[237,7057,410],{"class":246},[237,7059,1061],{"class":349},[237,7061,346],{"class":246},[237,7063,6732],{"class":349},[237,7065,121],{"class":246},[237,7067,4293],{"class":584},[237,7069,7070],{"class":246},"))",[237,7072,1120],{"class":338},[237,7074,7025],{"class":349},[237,7076,410],{"class":246},[237,7078,7079],{"class":349},"cooldown",[237,7081,247],{"class":246},[237,7083,7084,7086,7088],{"class":239,"line":306},[237,7085,467],{"class":452},[237,7087,2343],{"class":452},[237,7089,7090],{"class":287},"   # a VM is already on its way\n",[237,7092,7093,7096,7098,7101,7103,7106,7108,7111,7113,7115,7117,7119,7121,7124],{"class":239,"line":464},[237,7094,7095],{"class":349},"    response ",[237,7097,395],{"class":246},[237,7099,7100],{"class":349}," httpx",[237,7102,410],{"class":246},[237,7104,7105],{"class":349},"post",[237,7107,346],{"class":246},[237,7109,7110],{"class":349},"cfg",[237,7112,410],{"class":246},[237,7114,7030],{"class":349},[237,7116,121],{"class":246},[237,7118,764],{"class":562},[237,7120,395],{"class":246},[237,7122,7123],{"class":584},"10.0",[237,7125,680],{"class":246},[237,7127,7128,7130,7133,7135,7138,7140,7142,7145,7147,7150,7152,7155,7157,7160],{"class":239,"line":473},[237,7129,453],{"class":452},[237,7131,7132],{"class":349}," response",[237,7134,410],{"class":246},[237,7136,7137],{"class":349},"status_code ",[237,7139,696],{"class":338},[237,7141,669],{"class":246},[237,7143,7144],{"class":584},"200",[237,7146,121],{"class":246},[237,7148,7149],{"class":584}," 201",[237,7151,121],{"class":246},[237,7153,7154],{"class":584}," 202",[237,7156,121],{"class":246},[237,7158,7159],{"class":584}," 204",[237,7161,1496],{"class":246},[237,7163,7164,7167,7169,7171,7173,7175,7177,7179],{"class":239,"line":478},[237,7165,7166],{"class":406},"        self",[237,7168,410],{"class":246},[237,7170,7055],{"class":349},[237,7172,362],{"class":246},[237,7174,6732],{"class":349},[237,7176,1637],{"class":246},[237,7178,1026],{"class":246},[237,7180,6064],{"class":349},[237,7182,7183,7186,7189],{"class":239,"line":484},[237,7184,7185],{"class":452},"        with",[237,7187,7188],{"class":349}," _ts_lock",[237,7190,247],{"class":246},[237,7192,7193,7196,7198,7200,7202,7204,7207],{"class":239,"line":490},[237,7194,7195],{"class":349},"            _last_scale_out_ts",[237,7197,362],{"class":246},[237,7199,6732],{"class":349},[237,7201,1637],{"class":246},[237,7203,1026],{"class":246},[237,7205,7206],{"class":349}," now   ",[237,7208,7209],{"class":287},"# cross-thread handoff to scale-in logic\n",[237,7211,7212,7214],{"class":239,"line":516},[237,7213,467],{"class":452},[237,7215,1888],{"class":452},[34,7217,7218,7219,7222,7223,7225],{},"When the new VM boots, cloud-init runs ",[48,7220,7221],{},"docker swarm join",", swarmctl sees the new node next\ncycle, the stuck tasks schedule onto it, and ",[48,7224,4737],{}," goes quiet.",[220,7227,7229],{"id":7228},"scale-in-openstack-asks-swarmctl-for-permission","Scale-in: OpenStack asks swarmctl for permission",[34,7231,7232,7233,7236,7237,7239],{},"The reverse direction is more dangerous. OpenStack's Aodh ",[48,7234,7235],{},"cpu_low"," alarm wants to remove a VM.\nBut Aodh only sees aggregate cluster CPU: it has no idea whether removing that node would strand\nservices. So instead of pointing the alarm straight at Heat, it points at ",[37,7238,50],{},", which\napproves or denies:",[228,7241,7243],{"className":329,"code":7242,"language":331,"meta":233,"style":233},"@router.post(\"\u002Fscale-in-request\")\nasync def scale_in_request(pool: str = \"default\", ...):\n    result = await asyncio.to_thread(service.evaluate_scale_in, pool)\n    if result[\"approved\"]:\n        await asyncio.to_thread(AsgService().trigger_scale_in, pool)\n    return ScaleInResponse(**result)   # always 200 so Aodh never retries\n",[48,7244,7245,7268,7301,7334,7352,7380],{"__ignoreMap":233},[237,7246,7247,7250,7253,7255,7257,7259,7261,7264,7266],{"class":239,"line":11},[237,7248,7249],{"class":246},"@",[237,7251,7252],{"class":342},"router",[237,7254,410],{"class":246},[237,7256,7105],{"class":342},[237,7258,346],{"class":246},[237,7260,284],{"class":263},[237,7262,7263],{"class":267},"\u002Fscale-in-request",[237,7265,284],{"class":263},[237,7267,680],{"class":246},[237,7269,7270,7273,7276,7279,7281,7283,7285,7287,7289,7291,7293,7295,7297,7299],{"class":239,"line":20},[237,7271,7272],{"class":338},"async",[237,7274,7275],{"class":338}," def",[237,7277,7278],{"class":342}," scale_in_request",[237,7280,346],{"class":246},[237,7282,6732],{"class":349},[237,7284,1504],{"class":246},[237,7286,384],{"class":242},[237,7288,1026],{"class":246},[237,7290,264],{"class":263},[237,7292,6938],{"class":267},[237,7294,284],{"class":263},[237,7296,121],{"class":246},[237,7298,1416],{"class":349},[237,7300,1496],{"class":246},[237,7302,7303,7306,7308,7311,7314,7316,7319,7321,7323,7325,7328,7330,7332],{"class":239,"line":257},[237,7304,7305],{"class":349},"    result ",[237,7307,395],{"class":246},[237,7309,7310],{"class":452}," await",[237,7312,7313],{"class":349}," asyncio",[237,7315,410],{"class":246},[237,7317,7318],{"class":349},"to_thread",[237,7320,346],{"class":246},[237,7322,1856],{"class":349},[237,7324,410],{"class":246},[237,7326,7327],{"class":349},"evaluate_scale_in",[237,7329,121],{"class":246},[237,7331,611],{"class":349},[237,7333,680],{"class":246},[237,7335,7336,7338,7341,7343,7345,7348,7350],{"class":239,"line":274},[237,7337,453],{"class":452},[237,7339,7340],{"class":349}," result",[237,7342,362],{"class":246},[237,7344,284],{"class":263},[237,7346,7347],{"class":267},"approved",[237,7349,284],{"class":263},[237,7351,1950],{"class":246},[237,7353,7354,7357,7359,7361,7363,7365,7368,7371,7374,7376,7378],{"class":239,"line":291},[237,7355,7356],{"class":452},"        await",[237,7358,7313],{"class":349},[237,7360,410],{"class":246},[237,7362,7318],{"class":349},[237,7364,346],{"class":246},[237,7366,7367],{"class":349},"AsgService",[237,7369,7370],{"class":246},"().",[237,7372,7373],{"class":349},"trigger_scale_in",[237,7375,121],{"class":246},[237,7377,611],{"class":349},[237,7379,680],{"class":246},[237,7381,7382,7384,7387,7389,7391,7394,7396],{"class":239,"line":306},[237,7383,1312],{"class":452},[237,7385,7386],{"class":349}," ScaleInResponse",[237,7388,346],{"class":246},[237,7390,2165],{"class":338},[237,7392,7393],{"class":349},"result",[237,7395,353],{"class":246},[237,7397,7398],{"class":287},"   # always 200 so Aodh never retries\n",[34,7400,7401,7403],{},[48,7402,7327],{}," runs a sequence of safety checks, all scoped to the pool that fired:",[228,7405,7407],{"className":329,"code":7406,"language":331,"meta":233,"style":233},"# 1. Post-scale-out cooldown. If this pool just gained a VM, don't immediately remove one,\n#    otherwise Aodh sees the fresh node idle, approves removal, and could delete the ORIGINAL\n#    production node, briefly dropping to zero replicas (1 → 0 → 1).\nlast_out = _asg_mod.last_scale_out_ts(pool)\nif (now - last_out) \u003C cooldown:\n    return {\"approved\": False, \"reason\": \"post-scale-out cooldown\"}\n\n# 2. Scale-in's own cooldown.\n# 3. Minimum node count: strict \u003C=, so a single-node pool can never be scaled to zero.\nif len(active_workers) \u003C= min_nodes:\n    return {\"approved\": False, \"reason\": \"at minimum node count\"}\n\n# 4. Any pending task in THIS pool blocks removal: the cluster is already short on capacity.\nfor svc in services_in_pool:\n    if svc has a pending task:\n        return {\"approved\": False, \"reason\": f\"{svc.name} has pending tasks\"}\n\nreturn {\"approved\": True, \"reason\": \"safe\"}\n",[48,7408,7409,7414,7419,7424,7445,7467,7503,7507,7512,7517,7539,7574,7578,7583,7597,7606,7652,7656],{"__ignoreMap":233},[237,7410,7411],{"class":239,"line":11},[237,7412,7413],{"class":287},"# 1. Post-scale-out cooldown. If this pool just gained a VM, don't immediately remove one,\n",[237,7415,7416],{"class":239,"line":20},[237,7417,7418],{"class":287},"#    otherwise Aodh sees the fresh node idle, approves removal, and could delete the ORIGINAL\n",[237,7420,7421],{"class":239,"line":257},[237,7422,7423],{"class":287},"#    production node, briefly dropping to zero replicas (1 → 0 → 1).\n",[237,7425,7426,7429,7431,7434,7436,7439,7441,7443],{"class":239,"line":274},[237,7427,7428],{"class":349},"last_out ",[237,7430,395],{"class":246},[237,7432,7433],{"class":349}," _asg_mod",[237,7435,410],{"class":246},[237,7437,7438],{"class":349},"last_scale_out_ts",[237,7440,346],{"class":246},[237,7442,6732],{"class":349},[237,7444,680],{"class":246},[237,7446,7447,7449,7451,7453,7455,7458,7460,7462,7465],{"class":239,"line":291},[237,7448,1281],{"class":452},[237,7450,669],{"class":246},[237,7452,6078],{"class":349},[237,7454,729],{"class":338},[237,7456,7457],{"class":349}," last_out",[237,7459,353],{"class":246},[237,7461,1120],{"class":338},[237,7463,7464],{"class":349}," cooldown",[237,7466,247],{"class":246},[237,7468,7469,7471,7473,7475,7477,7479,7481,7483,7485,7487,7490,7492,7494,7496,7499,7501],{"class":239,"line":306},[237,7470,1312],{"class":452},[237,7472,5936],{"class":246},[237,7474,284],{"class":263},[237,7476,7347],{"class":267},[237,7478,284],{"class":263},[237,7480,1504],{"class":246},[237,7482,2343],{"class":452},[237,7484,121],{"class":246},[237,7486,264],{"class":263},[237,7488,7489],{"class":267},"reason",[237,7491,284],{"class":263},[237,7493,1504],{"class":246},[237,7495,264],{"class":263},[237,7497,7498],{"class":267},"post-scale-out cooldown",[237,7500,284],{"class":263},[237,7502,5970],{"class":246},[237,7504,7505],{"class":239,"line":464},[237,7506,426],{"emptyLinePlaceholder":425},[237,7508,7509],{"class":239,"line":473},[237,7510,7511],{"class":287},"# 2. Scale-in's own cooldown.\n",[237,7513,7514],{"class":239,"line":478},[237,7515,7516],{"class":287},"# 3. Minimum node count: strict \u003C=, so a single-node pool can never be scaled to zero.\n",[237,7518,7519,7521,7524,7526,7529,7531,7534,7537],{"class":239,"line":484},[237,7520,1281],{"class":452},[237,7522,7523],{"class":242}," len",[237,7525,346],{"class":246},[237,7527,7528],{"class":349},"active_workers",[237,7530,353],{"class":246},[237,7532,7533],{"class":338}," \u003C=",[237,7535,7536],{"class":349}," min_nodes",[237,7538,247],{"class":246},[237,7540,7541,7543,7545,7547,7549,7551,7553,7555,7557,7559,7561,7563,7565,7567,7570,7572],{"class":239,"line":490},[237,7542,1312],{"class":452},[237,7544,5936],{"class":246},[237,7546,284],{"class":263},[237,7548,7347],{"class":267},[237,7550,284],{"class":263},[237,7552,1504],{"class":246},[237,7554,2343],{"class":452},[237,7556,121],{"class":246},[237,7558,264],{"class":263},[237,7560,7489],{"class":267},[237,7562,284],{"class":263},[237,7564,1504],{"class":246},[237,7566,264],{"class":263},[237,7568,7569],{"class":267},"at minimum node count",[237,7571,284],{"class":263},[237,7573,5970],{"class":246},[237,7575,7576],{"class":239,"line":516},[237,7577,426],{"emptyLinePlaceholder":425},[237,7579,7580],{"class":239,"line":546},[237,7581,7582],{"class":287},"# 4. Any pending task in THIS pool blocks removal: the cluster is already short on capacity.\n",[237,7584,7585,7587,7590,7592,7595],{"class":239,"line":551},[237,7586,820],{"class":452},[237,7588,7589],{"class":349}," svc ",[237,7591,696],{"class":452},[237,7593,7594],{"class":349}," services_in_pool",[237,7596,247],{"class":246},[237,7598,7599,7601,7604],{"class":239,"line":591},[237,7600,453],{"class":452},[237,7602,7603],{"class":349}," svc has a pending task",[237,7605,247],{"class":246},[237,7607,7608,7610,7612,7614,7616,7618,7620,7622,7624,7626,7628,7630,7632,7634,7636,7638,7640,7642,7645,7647,7650],{"class":239,"line":616},[237,7609,467],{"class":452},[237,7611,5936],{"class":246},[237,7613,284],{"class":263},[237,7615,7347],{"class":267},[237,7617,284],{"class":263},[237,7619,1504],{"class":246},[237,7621,2343],{"class":452},[237,7623,121],{"class":246},[237,7625,264],{"class":263},[237,7627,7489],{"class":267},[237,7629,284],{"class":263},[237,7631,1504],{"class":246},[237,7633,3039],{"class":338},[237,7635,284],{"class":267},[237,7637,851],{"class":406},[237,7639,672],{"class":349},[237,7641,410],{"class":246},[237,7643,7644],{"class":349},"name",[237,7646,863],{"class":406},[237,7648,7649],{"class":267}," has pending tasks\"",[237,7651,5970],{"class":246},[237,7653,7654],{"class":239,"line":627},[237,7655,426],{"emptyLinePlaceholder":425},[237,7657,7658,7660,7662,7664,7666,7668,7670,7672,7674,7676,7678,7680,7682,7684,7687,7689],{"class":239,"line":683},[237,7659,5403],{"class":452},[237,7661,5936],{"class":246},[237,7663,284],{"class":263},[237,7665,7347],{"class":267},[237,7667,284],{"class":263},[237,7669,1504],{"class":246},[237,7671,3791],{"class":452},[237,7673,121],{"class":246},[237,7675,264],{"class":263},[237,7677,7489],{"class":267},[237,7679,284],{"class":263},[237,7681,1504],{"class":246},[237,7683,264],{"class":263},[237,7685,7686],{"class":267},"safe",[237,7688,284],{"class":263},[237,7690,5970],{"class":246},[34,7692,7693],{},"The post-scale-out cooldown is the subtle one, and it needs cross-thread state: the watch loop\n(main thread) records the last scale-out timestamp per pool, and the hook server (daemon thread)\nreads it. That handoff goes through a lock-protected module dict:",[228,7695,7697],{"className":329,"code":7696,"language":331,"meta":233,"style":233},"_ts_lock = threading.Lock()\n_last_scale_out_ts: dict[str, float] = {}\n\ndef last_scale_out_ts(pool: str) -> float:\n    with _ts_lock:\n        return _last_scale_out_ts.get(pool, 0.0)\n",[48,7698,7699,7716,7740,7744,7767,7775],{"__ignoreMap":233},[237,7700,7701,7704,7706,7709,7711,7714],{"class":239,"line":11},[237,7702,7703],{"class":349},"_ts_lock ",[237,7705,395],{"class":246},[237,7707,7708],{"class":349}," threading",[237,7710,410],{"class":246},[237,7712,7713],{"class":349},"Lock",[237,7715,401],{"class":246},[237,7717,7718,7721,7723,7726,7728,7730,7732,7734,7736,7738],{"class":239,"line":20},[237,7719,7720],{"class":349},"_last_scale_out_ts",[237,7722,1504],{"class":246},[237,7724,7725],{"class":349}," dict",[237,7727,362],{"class":246},[237,7729,379],{"class":242},[237,7731,121],{"class":246},[237,7733,1573],{"class":242},[237,7735,1637],{"class":246},[237,7737,1026],{"class":246},[237,7739,5513],{"class":246},[237,7741,7742],{"class":239,"line":257},[237,7743,426],{"emptyLinePlaceholder":425},[237,7745,7746,7748,7751,7753,7755,7757,7759,7761,7763,7765],{"class":239,"line":274},[237,7747,339],{"class":338},[237,7749,7750],{"class":342}," last_scale_out_ts",[237,7752,346],{"class":246},[237,7754,6732],{"class":349},[237,7756,1504],{"class":246},[237,7758,384],{"class":242},[237,7760,353],{"class":246},[237,7762,356],{"class":246},[237,7764,1573],{"class":242},[237,7766,247],{"class":246},[237,7768,7769,7771,7773],{"class":239,"line":291},[237,7770,554],{"class":452},[237,7772,7188],{"class":349},[237,7774,247],{"class":246},[237,7776,7777,7779,7782,7784,7786,7788,7790,7792,7794],{"class":239,"line":306},[237,7778,467],{"class":452},[237,7780,7781],{"class":349}," _last_scale_out_ts",[237,7783,410],{"class":246},[237,7785,1061],{"class":349},[237,7787,346],{"class":246},[237,7789,6732],{"class":349},[237,7791,121],{"class":246},[237,7793,4293],{"class":584},[237,7795,680],{"class":246},[220,7797,7799],{"id":7798},"pre-terminate-drain-a-node-gracefully","Pre-terminate: drain a node gracefully",[34,7801,7802,7803,2208,7806,7809],{},"Once a scale-in is approved and Heat picks a victim VM, the VM's own systemd shutdown hook calls\n",[48,7804,7805],{},"POST \u002Fhooks\u002Fpre-terminate",[37,7807,7808],{},"before"," Docker stops. This is the graceful-drain path, and it is\nthe most involved piece of orchestration in the system:",[228,7811,7813],{"className":329,"code":7812,"language":331,"meta":233,"style":233},"def handle(self, node_name, instance_id) -> dict:\n    node = self._find_node(node_name)\n    if node is None:\n        return {\"status\": \"approve\", \"reason\": \"node not found in swarm\"}\n\n    running_tasks = self._tasks_on_node(node.id)\n    if running_tasks:\n        # 1. Pause the node so Swarm schedules nothing new onto a dying host.\n        if not self._set_availability(node, AVAIL_PAUSE):\n            return {\"status\": \"reject\", \"reason\": \"failed to pause node\"}\n\n        # 2. Drain: scale affected services DOWN by their task count on this node,\n        #    so Swarm reschedules those replicas elsewhere.\n        adjustments = self._compute_adjustments(node.id, running_tasks)\n        self._apply_scale_adjustments(adjustments, scale_down=True)\n\n        # 3. Wait (bounded) for tasks to actually leave the node.\n        drained = self._wait_for_drain(node.id)\n\n        # 4. Scale back UP to the original counts: the new replicas land on OTHER nodes.\n        self._apply_scale_adjustments(adjustments, scale_down=False)\n\n        # 5. Write REBASELINE events so the watch loop treats the restored counts as the\n        #    authoritative target, instead of trying to \"reconcile\" the drain-time dip.\n        self._write_rebaseline_events(adjustments)\n\n        self._set_availability(self._client.nodes.get(node.id), AVAIL_ACTIVE)\n\n    # 6. Finally, simulate removing the node for real.\n    return self._simulate_removal(node.id)\n",[48,7814,7815,7844,7865,7878,7919,7923,7948,7957,7962,7986,8027,8031,8036,8041,8069,8094,8098,8103,8127,8131,8136,8158,8162,8167,8172,8187,8191,8232,8236,8242],{"__ignoreMap":233},[237,7816,7817,7819,7822,7824,7826,7828,7831,7833,7836,7838,7840,7842],{"class":239,"line":11},[237,7818,339],{"class":338},[237,7820,7821],{"class":342}," handle",[237,7823,346],{"class":246},[237,7825,350],{"class":349},[237,7827,121],{"class":246},[237,7829,7830],{"class":349}," node_name",[237,7832,121],{"class":246},[237,7834,7835],{"class":349}," instance_id",[237,7837,353],{"class":246},[237,7839,356],{"class":246},[237,7841,7725],{"class":242},[237,7843,247],{"class":246},[237,7845,7846,7849,7851,7853,7855,7858,7860,7863],{"class":239,"line":20},[237,7847,7848],{"class":349},"    node ",[237,7850,395],{"class":246},[237,7852,436],{"class":406},[237,7854,410],{"class":246},[237,7856,7857],{"class":349},"_find_node",[237,7859,346],{"class":246},[237,7861,7862],{"class":349},"node_name",[237,7864,680],{"class":246},[237,7866,7867,7869,7872,7874,7876],{"class":239,"line":257},[237,7868,453],{"class":452},[237,7870,7871],{"class":349}," node ",[237,7873,104],{"class":338},[237,7875,1273],{"class":452},[237,7877,247],{"class":246},[237,7879,7880,7882,7884,7886,7889,7891,7893,7895,7898,7900,7902,7904,7906,7908,7910,7912,7915,7917],{"class":239,"line":274},[237,7881,467],{"class":452},[237,7883,5936],{"class":246},[237,7885,284],{"class":263},[237,7887,7888],{"class":267},"status",[237,7890,284],{"class":263},[237,7892,1504],{"class":246},[237,7894,264],{"class":263},[237,7896,7897],{"class":267},"approve",[237,7899,284],{"class":263},[237,7901,121],{"class":246},[237,7903,264],{"class":263},[237,7905,7489],{"class":267},[237,7907,284],{"class":263},[237,7909,1504],{"class":246},[237,7911,264],{"class":263},[237,7913,7914],{"class":267},"node not found in swarm",[237,7916,284],{"class":263},[237,7918,5970],{"class":246},[237,7920,7921],{"class":239,"line":291},[237,7922,426],{"emptyLinePlaceholder":425},[237,7924,7925,7928,7930,7932,7934,7937,7939,7942,7944,7946],{"class":239,"line":306},[237,7926,7927],{"class":349},"    running_tasks ",[237,7929,395],{"class":246},[237,7931,436],{"class":406},[237,7933,410],{"class":246},[237,7935,7936],{"class":349},"_tasks_on_node",[237,7938,346],{"class":246},[237,7940,7941],{"class":349},"node",[237,7943,410],{"class":246},[237,7945,658],{"class":349},[237,7947,680],{"class":246},[237,7949,7950,7952,7955],{"class":239,"line":464},[237,7951,453],{"class":452},[237,7953,7954],{"class":349}," running_tasks",[237,7956,247],{"class":246},[237,7958,7959],{"class":239,"line":473},[237,7960,7961],{"class":287},"        # 1. Pause the node so Swarm schedules nothing new onto a dying host.\n",[237,7963,7964,7966,7968,7970,7972,7975,7977,7979,7981,7984],{"class":239,"line":478},[237,7965,782],{"class":452},[237,7967,456],{"class":338},[237,7969,436],{"class":406},[237,7971,410],{"class":246},[237,7973,7974],{"class":349},"_set_availability",[237,7976,346],{"class":246},[237,7978,7941],{"class":349},[237,7980,121],{"class":246},[237,7982,7983],{"class":406}," AVAIL_PAUSE",[237,7985,1496],{"class":246},[237,7987,7988,7991,7993,7995,7997,7999,8001,8003,8006,8008,8010,8012,8014,8016,8018,8020,8023,8025],{"class":239,"line":484},[237,7989,7990],{"class":452},"            return",[237,7992,5936],{"class":246},[237,7994,284],{"class":263},[237,7996,7888],{"class":267},[237,7998,284],{"class":263},[237,8000,1504],{"class":246},[237,8002,264],{"class":263},[237,8004,8005],{"class":267},"reject",[237,8007,284],{"class":263},[237,8009,121],{"class":246},[237,8011,264],{"class":263},[237,8013,7489],{"class":267},[237,8015,284],{"class":263},[237,8017,1504],{"class":246},[237,8019,264],{"class":263},[237,8021,8022],{"class":267},"failed to pause node",[237,8024,284],{"class":263},[237,8026,5970],{"class":246},[237,8028,8029],{"class":239,"line":490},[237,8030,426],{"emptyLinePlaceholder":425},[237,8032,8033],{"class":239,"line":516},[237,8034,8035],{"class":287},"        # 2. Drain: scale affected services DOWN by their task count on this node,\n",[237,8037,8038],{"class":239,"line":546},[237,8039,8040],{"class":287},"        #    so Swarm reschedules those replicas elsewhere.\n",[237,8042,8043,8046,8048,8050,8052,8055,8057,8059,8061,8063,8065,8067],{"class":239,"line":551},[237,8044,8045],{"class":349},"        adjustments ",[237,8047,395],{"class":246},[237,8049,436],{"class":406},[237,8051,410],{"class":246},[237,8053,8054],{"class":349},"_compute_adjustments",[237,8056,346],{"class":246},[237,8058,7941],{"class":349},[237,8060,410],{"class":246},[237,8062,658],{"class":349},[237,8064,121],{"class":246},[237,8066,7954],{"class":349},[237,8068,680],{"class":246},[237,8070,8071,8073,8075,8078,8080,8083,8085,8088,8090,8092],{"class":239,"line":591},[237,8072,7166],{"class":406},[237,8074,410],{"class":246},[237,8076,8077],{"class":349},"_apply_scale_adjustments",[237,8079,346],{"class":246},[237,8081,8082],{"class":349},"adjustments",[237,8084,121],{"class":246},[237,8086,8087],{"class":562}," scale_down",[237,8089,395],{"class":246},[237,8091,4741],{"class":452},[237,8093,680],{"class":246},[237,8095,8096],{"class":239,"line":616},[237,8097,426],{"emptyLinePlaceholder":425},[237,8099,8100],{"class":239,"line":627},[237,8101,8102],{"class":287},"        # 3. Wait (bounded) for tasks to actually leave the node.\n",[237,8104,8105,8108,8110,8112,8114,8117,8119,8121,8123,8125],{"class":239,"line":683},[237,8106,8107],{"class":349},"        drained ",[237,8109,395],{"class":246},[237,8111,436],{"class":406},[237,8113,410],{"class":246},[237,8115,8116],{"class":349},"_wait_for_drain",[237,8118,346],{"class":246},[237,8120,7941],{"class":349},[237,8122,410],{"class":246},[237,8124,658],{"class":349},[237,8126,680],{"class":246},[237,8128,8129],{"class":239,"line":702},[237,8130,426],{"emptyLinePlaceholder":425},[237,8132,8133],{"class":239,"line":708},[237,8134,8135],{"class":287},"        # 4. Scale back UP to the original counts: the new replicas land on OTHER nodes.\n",[237,8137,8138,8140,8142,8144,8146,8148,8150,8152,8154,8156],{"class":239,"line":741},[237,8139,7166],{"class":406},[237,8141,410],{"class":246},[237,8143,8077],{"class":349},[237,8145,346],{"class":246},[237,8147,8082],{"class":349},[237,8149,121],{"class":246},[237,8151,8087],{"class":562},[237,8153,395],{"class":246},[237,8155,5011],{"class":452},[237,8157,680],{"class":246},[237,8159,8160],{"class":239,"line":774},[237,8161,426],{"emptyLinePlaceholder":425},[237,8163,8164],{"class":239,"line":779},[237,8165,8166],{"class":287},"        # 5. Write REBASELINE events so the watch loop treats the restored counts as the\n",[237,8168,8169],{"class":239,"line":790},[237,8170,8171],{"class":287},"        #    authoritative target, instead of trying to \"reconcile\" the drain-time dip.\n",[237,8173,8174,8176,8178,8181,8183,8185],{"class":239,"line":833},[237,8175,7166],{"class":406},[237,8177,410],{"class":246},[237,8179,8180],{"class":349},"_write_rebaseline_events",[237,8182,346],{"class":246},[237,8184,8082],{"class":349},[237,8186,680],{"class":246},[237,8188,8189],{"class":239,"line":880},[237,8190,426],{"emptyLinePlaceholder":425},[237,8192,8193,8195,8197,8199,8201,8203,8205,8208,8210,8213,8215,8217,8219,8221,8223,8225,8227,8230],{"class":239,"line":894},[237,8194,7166],{"class":406},[237,8196,410],{"class":246},[237,8198,7974],{"class":349},[237,8200,346],{"class":246},[237,8202,350],{"class":406},[237,8204,410],{"class":246},[237,8206,8207],{"class":349},"_client",[237,8209,410],{"class":246},[237,8211,8212],{"class":349},"nodes",[237,8214,410],{"class":246},[237,8216,1061],{"class":349},[237,8218,346],{"class":246},[237,8220,7941],{"class":349},[237,8222,410],{"class":246},[237,8224,658],{"class":349},[237,8226,581],{"class":246},[237,8228,8229],{"class":406}," AVAIL_ACTIVE",[237,8231,680],{"class":246},[237,8233,8234],{"class":239,"line":907},[237,8235,426],{"emptyLinePlaceholder":425},[237,8237,8239],{"class":239,"line":8238},29,[237,8240,8241],{"class":287},"    # 6. Finally, simulate removing the node for real.\n",[237,8243,8245,8247,8249,8251,8254,8256,8258,8260,8262],{"class":239,"line":8244},30,[237,8246,1312],{"class":452},[237,8248,436],{"class":406},[237,8250,410],{"class":246},[237,8252,8253],{"class":349},"_simulate_removal",[237,8255,346],{"class":246},[237,8257,7941],{"class":349},[237,8259,410],{"class":246},[237,8261,658],{"class":349},[237,8263,680],{"class":246},[34,8265,8266,8267,8270,8271,8274],{},"The drain is a ",[55,8268,8269],{},"temporary"," scale-down-then-up dance: you shrink each service by exactly the\nnumber of its tasks on the doomed node, let Swarm move them, then grow back so the replacements\nschedule onto healthy nodes. If the drain times out, swarmctl restores the counts and ",[37,8272,8273],{},"rejects","\nthe termination: better to keep a node than to lose replicas.",[34,8276,8277],{},"Here is the whole sequence, including the final simulation that can still say no:",[8279,8280],"swarm-drain",{},[34,8282,5443,8283,8286,8287,8290],{},[37,8284,8285],{},"rebaseline"," step solves a nasty interaction with the watch loop: during the drain the loop\nwould see service counts dip and might try to \"fix\" them. Writing a ",[48,8288,8289],{},"REBASELINE"," audit event\ntells the loop \"this new count is intentional, adopt it as the target,\" so the two subsystems\ndon't fight.",[34,8292,8293,8294,8296,8297,8299],{},"Finally, ",[48,8295,8253],{}," is the last safety gate: it projects every surviving node's memory\nif this node's load is redistributed, and ",[37,8298,8273],{}," the termination if any node would exceed\nthe rebound threshold:",[228,8301,8303],{"className":329,"code":8302,"language":331,"meta":233,"style":233},"threshold = self._settings.hook_rebound_threshold   # e.g. 65%\nfor nid, used_mb in node_used_mem.items():\n    pct = (used_mb * 1024 * 1024) \u002F node_total_mem[nid]\n    if pct > threshold:\n        return {\"status\": \"reject\", \"reason\": HOOK_REASON_REBOUND}\nreturn {\"status\": \"approve\", \"reason\": \"safe\"}\n",[48,8304,8305,8327,8350,8384,8398,8433],{"__ignoreMap":233},[237,8306,8307,8310,8312,8314,8316,8319,8321,8324],{"class":239,"line":11},[237,8308,8309],{"class":349},"threshold ",[237,8311,395],{"class":246},[237,8313,436],{"class":406},[237,8315,410],{"class":246},[237,8317,8318],{"class":349},"_settings",[237,8320,410],{"class":246},[237,8322,8323],{"class":349},"hook_rebound_threshold   ",[237,8325,8326],{"class":287},"# e.g. 65%\n",[237,8328,8329,8331,8334,8336,8339,8341,8344,8346,8348],{"class":239,"line":20},[237,8330,820],{"class":452},[237,8332,8333],{"class":349}," nid",[237,8335,121],{"class":246},[237,8337,8338],{"class":349}," used_mb ",[237,8340,696],{"class":452},[237,8342,8343],{"class":349}," node_used_mem",[237,8345,410],{"class":246},[237,8347,2021],{"class":349},[237,8349,6897],{"class":246},[237,8351,8352,8355,8357,8359,8362,8364,8366,8368,8370,8372,8374,8377,8379,8382],{"class":239,"line":257},[237,8353,8354],{"class":349},"    pct ",[237,8356,395],{"class":246},[237,8358,669],{"class":246},[237,8360,8361],{"class":349},"used_mb ",[237,8363,1149],{"class":338},[237,8365,1035],{"class":584},[237,8367,1032],{"class":338},[237,8369,1035],{"class":584},[237,8371,353],{"class":246},[237,8373,3317],{"class":338},[237,8375,8376],{"class":349}," node_total_mem",[237,8378,362],{"class":246},[237,8380,8381],{"class":349},"nid",[237,8383,830],{"class":246},[237,8385,8386,8388,8391,8393,8396],{"class":239,"line":274},[237,8387,453],{"class":452},[237,8389,8390],{"class":349}," pct ",[237,8392,2302],{"class":338},[237,8394,8395],{"class":349}," threshold",[237,8397,247],{"class":246},[237,8399,8400,8402,8404,8406,8408,8410,8412,8414,8416,8418,8420,8422,8424,8426,8428,8431],{"class":239,"line":291},[237,8401,467],{"class":452},[237,8403,5936],{"class":246},[237,8405,284],{"class":263},[237,8407,7888],{"class":267},[237,8409,284],{"class":263},[237,8411,1504],{"class":246},[237,8413,264],{"class":263},[237,8415,8005],{"class":267},[237,8417,284],{"class":263},[237,8419,121],{"class":246},[237,8421,264],{"class":263},[237,8423,7489],{"class":267},[237,8425,284],{"class":263},[237,8427,1504],{"class":246},[237,8429,8430],{"class":406}," HOOK_REASON_REBOUND",[237,8432,5970],{"class":246},[237,8434,8435,8437,8439,8441,8443,8445,8447,8449,8451,8453,8455,8457,8459,8461,8463,8465,8467,8469],{"class":239,"line":306},[237,8436,5403],{"class":452},[237,8438,5936],{"class":246},[237,8440,284],{"class":263},[237,8442,7888],{"class":267},[237,8444,284],{"class":263},[237,8446,1504],{"class":246},[237,8448,264],{"class":263},[237,8450,7897],{"class":267},[237,8452,284],{"class":263},[237,8454,121],{"class":246},[237,8456,264],{"class":263},[237,8458,7489],{"class":267},[237,8460,284],{"class":263},[237,8462,1504],{"class":246},[237,8464,264],{"class":263},[237,8466,7686],{"class":267},[237,8468,284],{"class":263},[237,8470,5970],{"class":246},[220,8472,8474],{"id":8473},"the-one-rule-openstack-must-obey","The one rule OpenStack must obey",[34,8476,8477],{},"The webhook can only ever return approve or reject. The critical operational contract is on the\nOpenStack side:",[31,8479,8480],{},[34,8481,8482,8485],{},[37,8483,8484],{},"OpenStack must be configured to reject a termination by default if the webhook is\nunreachable."," A node must never be removed without positive approval.",[34,8487,8488],{},"Fail-safe, not fail-open. If swarmctl is down, nodes stay up: a slightly-too-big cluster is\nalways cheaper than an outage.",[167,8490],{},[59,8492,8494],{"id":8493},"design-principles-that-run-through-the-whole-thing","Design principles that run through the whole thing",[34,8496,8497],{},"A few ideas show up again and again, and they're worth stating plainly because they're what make\nthe system trustworthy:",[34,8499,8500,8503],{},[37,8501,8502],{},"The engine is pure."," No I\u002FO, no clock, no randomness. Every bit of timing is injected by the\nwatch loop. This is why it can be exhaustively unit-tested, and why you can reason about its\nbehavior by reading one function.",[34,8505,8506,8509],{},[37,8507,8508],{},"Scale up fast, scale down slow."," Up: any single trigger fires, step 1 to 3 by urgency. Down:\nevery condition true, sustained for a full duration, and a per-metric simulation must pass. The\ncost of under-provisioning (dropped requests) is far higher than the cost of a spare replica.",[34,8511,8512,8515],{},[37,8513,8514],{},"Different metrics behave differently."," CPU redistributes across replicas; memory is\nprocess-private; latency is non-linear and often upstream-bound. The simulation treats each\ncorrectly instead of applying one proportional formula to all three, a bug our first version\nactually had.",[34,8517,8518,8521,8522,8524,8525,8527],{},[37,8519,8520],{},"Separate \"scale containers\" from \"scale VMs\"."," The ",[48,8523,2581],{}," and the\n",[48,8526,4737],{}," are two independent outputs. One drives the Docker API; the other drives\nOpenStack. Keeping them separate keeps each layer simple.",[34,8529,8530,8533],{},[37,8531,8532],{},"Fail safe."," Crash-looping services aren't scaled. Broken deployments (bad image, missing\nsecret) don't trigger capacity signals. Node termination defaults to reject. Partial successes\nare recorded as partial, not as success.",[34,8535,8536,8539,8540,8543],{},[37,8537,8538],{},"Every decision is auditable."," Every cycle, every service, every NONE-with-a-score lands in\nSQLite. When you later ask \"why did it scale at 14:03?\" or \"why ",[55,8541,8542],{},"didn't"," it scale?\", the answer\nis a query away.",[167,8545],{},[59,8547,8549],{"id":8548},"running-it","Running it",[34,8551,8552],{},"Run it as a Swarm service on a manager node, with the Docker socket mounted and the hook port exposed:",[228,8554,8556],{"className":230,"code":8555,"language":232,"meta":233,"style":233},"services:\n  swarmctl:\n    image: yourrepo\u002Fswarmctl:latest\n    command: [\"swarmctl\", \"watch\", \"--interval\", \"30\"]\n    deploy:\n      replicas: 1\n      placement:\n        constraints: [\"node.role == manager\"]\n      restart_policy:\n        condition: any\n    volumes:\n      - \u002Fvar\u002Frun\u002Fdocker.sock:\u002Fvar\u002Frun\u002Fdocker.sock:ro\n      - swarmctl_data:\u002Fdata\n    ports:\n      - \"8889:8889\"   # OpenStack lifecycle hook\n",[48,8557,8558,8564,8571,8581,8625,8632,8641,8648,8666,8673,8683,8690,8698,8705,8712],{"__ignoreMap":233},[237,8559,8560,8562],{"class":239,"line":11},[237,8561,5062],{"class":242},[237,8563,247],{"class":246},[237,8565,8566,8569],{"class":239,"line":20},[237,8567,8568],{"class":242},"  swarmctl",[237,8570,247],{"class":246},[237,8572,8573,8576,8578],{"class":239,"line":257},[237,8574,8575],{"class":242},"    image",[237,8577,1504],{"class":246},[237,8579,8580],{"class":267}," yourrepo\u002Fswarmctl:latest\n",[237,8582,8583,8586,8588,8590,8592,8594,8596,8598,8600,8603,8605,8607,8609,8612,8614,8616,8618,8621,8623],{"class":239,"line":274},[237,8584,8585],{"class":242},"    command",[237,8587,1504],{"class":246},[237,8589,798],{"class":246},[237,8591,284],{"class":263},[237,8593,50],{"class":267},[237,8595,284],{"class":263},[237,8597,121],{"class":246},[237,8599,264],{"class":263},[237,8601,8602],{"class":267},"watch",[237,8604,284],{"class":263},[237,8606,121],{"class":246},[237,8608,264],{"class":263},[237,8610,8611],{"class":267},"--interval",[237,8613,284],{"class":263},[237,8615,121],{"class":246},[237,8617,264],{"class":263},[237,8619,8620],{"class":267},"30",[237,8622,284],{"class":263},[237,8624,830],{"class":246},[237,8626,8627,8630],{"class":239,"line":291},[237,8628,8629],{"class":242},"    deploy",[237,8631,247],{"class":246},[237,8633,8634,8637,8639],{"class":239,"line":306},[237,8635,8636],{"class":242},"      replicas",[237,8638,1504],{"class":246},[237,8640,3502],{"class":584},[237,8642,8643,8646],{"class":239,"line":464},[237,8644,8645],{"class":242},"      placement",[237,8647,247],{"class":246},[237,8649,8650,8653,8655,8657,8659,8662,8664],{"class":239,"line":473},[237,8651,8652],{"class":242},"        constraints",[237,8654,1504],{"class":246},[237,8656,798],{"class":246},[237,8658,284],{"class":263},[237,8660,8661],{"class":267},"node.role == manager",[237,8663,284],{"class":263},[237,8665,830],{"class":246},[237,8667,8668,8671],{"class":239,"line":478},[237,8669,8670],{"class":242},"      restart_policy",[237,8672,247],{"class":246},[237,8674,8675,8678,8680],{"class":239,"line":484},[237,8676,8677],{"class":242},"        condition",[237,8679,1504],{"class":246},[237,8681,8682],{"class":267}," any\n",[237,8684,8685,8688],{"class":239,"line":490},[237,8686,8687],{"class":242},"    volumes",[237,8689,247],{"class":246},[237,8691,8692,8695],{"class":239,"line":516},[237,8693,8694],{"class":246},"      -",[237,8696,8697],{"class":267}," \u002Fvar\u002Frun\u002Fdocker.sock:\u002Fvar\u002Frun\u002Fdocker.sock:ro\n",[237,8699,8700,8702],{"class":239,"line":546},[237,8701,8694],{"class":246},[237,8703,8704],{"class":267}," swarmctl_data:\u002Fdata\n",[237,8706,8707,8710],{"class":239,"line":551},[237,8708,8709],{"class":242},"    ports",[237,8711,247],{"class":246},[237,8713,8714,8716,8718,8721,8723],{"class":239,"line":591},[237,8715,8694],{"class":246},[237,8717,264],{"class":263},[237,8719,8720],{"class":267},"8889:8889",[237,8722,284],{"class":263},[237,8724,8725],{"class":287},"   # OpenStack lifecycle hook\n",[34,8727,8728,8729,8731],{},"Then watch it work: ",[48,8730,50],{}," prints a live Rich table every cycle showing each service's\nreplicas, CPU avg\u002Fmax, throttle, memory, request rate, latency, error rate, the decision taken,\nand a ⚠ marker when the host-scale signal is raised.",[34,8733,8734,8735,8737],{},"Opt a service in with a single label (",[48,8736,268],{},"), optionally pick a template, and\nlet the loop do the rest.",[167,8739],{},[59,8741,8743],{"id":8742},"closing-thought","Closing thought",[34,8745,8746,8747,8749,8750,8752,8753,8755,8756,8758,8759,8761,8762,8764],{},"An autoscaler is mostly a study in restraint. The hard part isn't deciding to scale up when CPU\nis high: that's one ",[48,8748,1281],{},". The hard part is ",[55,8751,81],{}," scaling down into an oscillation, ",[55,8754,81],{}," fighting\nan operator's manual change, ",[55,8757,81],{}," killing a node that would strand a service, ",[55,8760,81],{}," reacting to a\nsingle spiky cycle, and ",[55,8763,81],{}," trusting a proportional formula that happens to be wrong for memory.",[34,8766,8767,8768,8770,8771,8774,8775,8778],{},"Every guard, cooldown, duration timer, simulation, and reconciliation branch in ",[48,8769,50],{}," exists\nbecause our naive first version did the wrong thing in production. That's the real content of this post:\nnot that you ",[55,8772,8773],{},"can"," call ",[48,8776,8777],{},"service.scale()",", but all the things you have to check before you dare to.",[8780,8781,8782],"style",{},"html pre.shiki code .su6XF, html code.shiki .su6XF{--shiki-default:#998418}html pre.shiki code .sYZai, html code.shiki .sYZai{--shiki-default:#999999}html pre.shiki code .sSP4y, html code.shiki .sSP4y{--shiki-default:#B5695977}html pre.shiki code .spphp, html code.shiki .spphp{--shiki-default:#B56959}html pre.shiki code .s8zF2, html code.shiki .s8zF2{--shiki-default:#A0ADA0}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .si04Y, html code.shiki .si04Y{--shiki-default:#AB5959}html pre.shiki code .sySUi, html code.shiki .sySUi{--shiki-default:#59873A}html pre.shiki code .suHK_, html code.shiki .suHK_{--shiki-default:#393A34}html pre.shiki code .sEi1f, html code.shiki .sEi1f{--shiki-default:#A65E2B}html pre.shiki code .sbBg2, html code.shiki .sbBg2{--shiki-default:#1E754F}html pre.shiki code .svycV, html code.shiki .svycV{--shiki-default:#B07D48}html pre.shiki code .s-TwI, html code.shiki .s-TwI{--shiki-default:#2F798A}html pre.shiki code .sUxyF, html code.shiki .sUxyF{--shiki-default:#2E8F82}",{"title":233,"searchDepth":20,"depth":20,"links":8784},[8785,8786,8787,8796,8802,8809,8818,8824,8825,8826],{"id":61,"depth":20,"text":62},{"id":179,"depth":20,"text":180},{"id":210,"depth":20,"text":211,"children":8788},[8789,8790,8791,8792,8793,8794,8795],{"id":222,"depth":257,"text":223},{"id":318,"depth":257,"text":319},{"id":930,"depth":257,"text":931},{"id":980,"depth":257,"text":981},{"id":1187,"depth":257,"text":1188},{"id":1321,"depth":257,"text":1322},{"id":1428,"depth":257,"text":1429},{"id":1717,"depth":20,"text":1718,"children":8797},[8798,8799,8800,8801],{"id":1748,"depth":257,"text":1749},{"id":2443,"depth":257,"text":2444},{"id":3868,"depth":257,"text":3869},{"id":4730,"depth":257,"text":4731},{"id":4773,"depth":20,"text":4774,"children":8803},[8804,8805,8806,8807,8808],{"id":4780,"depth":257,"text":4781},{"id":4888,"depth":257,"text":4889},{"id":5030,"depth":257,"text":5031},{"id":5185,"depth":257,"text":5186},{"id":5465,"depth":257,"text":5466},{"id":5474,"depth":20,"text":5475,"children":8810},[8811,8812,8813,8814,8815,8816,8817],{"id":5680,"depth":257,"text":5681},{"id":5990,"depth":257,"text":5991},{"id":6133,"depth":257,"text":6134},{"id":6226,"depth":257,"text":6227},{"id":6419,"depth":257,"text":6420},{"id":6608,"depth":257,"text":6609},{"id":6619,"depth":257,"text":6620},{"id":6636,"depth":20,"text":6637,"children":8819},[8820,8821,8822,8823],{"id":6653,"depth":257,"text":6654},{"id":7228,"depth":257,"text":7229},{"id":7798,"depth":257,"text":7799},{"id":8473,"depth":257,"text":8474},{"id":8493,"depth":20,"text":8494},{"id":8548,"depth":20,"text":8549},{"id":8742,"depth":20,"text":8743},"Blog","2025-12-08","Swarm gives you `docker service scale` and nothing else. This is how I built swarmctl, a daemon that collects metrics, decides when to scale, executes safely, and coordinates VM scaling with OpenStack.","md",{},"\u002Fblog\u002Fbuilding-an-autoscaler-for-docker-swarm","25 min",{"title":26,"description":8829},"blog\u002Fbuilding-an-autoscaler-for-docker-swarm\u002Findex",[8837,8838,8839,8840,8841],"Docker Swarm","Autoscaling","Python","OpenStack","DevOps","\u002Fimages\u002Fthumbnails\u002Fbuilding-an-autoscaler-for-docker-swarm\u002Fthumbnail.png","z-YSfLtRDHGY3g0RNQ1OHCalUJvSBOuT5sEtuFG57sU",1792163584689]