Building a CloudTrail for OpenStack: Why We Chose Loki Over Hermes, Elasticsearch, and Everything Else

The moment that started this

Partway through an incident-response session on our OpenStack cluster, I realized a VM had been deleted in one project. The cluster is managed by our team, and almost everyone has access to provision VMs in that project. We had no idea who deleted it, when, or why. The only thing we knew was that the VM was gone.

If we had had an audit trail, it would have taken seconds to find out who performed the action, instead of asking around and going back and forth. When I build applications, I always include an audit trail that records who did what and when. I needed something like CloudTrail for OpenStack, and I realized there is no built-in equivalent of AWS CloudTrail in OpenStack. That is a problem, because auditing is a critical requirement for many organizations.

This is the full story: what already exists in the OpenStack ecosystem, why none of it was quite right for us, what I actually built, and the real issues I hit along the way.

Part 1: What already exists

The standard: CADF

The foundation everyone builds on is CADF (Cloud Auditing Data Federation), a DMTF standard that defines a normalized event format answering the seven Ws of auditing: what, when, who, on what, where, from where, and to where. OpenStack's pycadf library implements the data model, and keystonemiddleware ships a WSGI filter that uses it to turn API requests into CADF events.

Option A: The built-in keystonemiddleware audit filter

This is what ships with OpenStack itself, and what Canonical's and Red Hat's OpenStack distributions document as the way to get CADF auditing. You drop a filter into a service's api-paste.ini and point it at a mapping file. The mapping files are bundled with pycadf, one per service: we found glance_api_audit_map.conf, nova_api_audit_map.conf, heat_api_audit_map.conf and others already sitting in the container image, so no extra install was needed. Once configured, the filter starts emitting events via keystonemiddleware.audit:filter_factory or pycadf.middleware.audit:filter_factory.

The problem: there is an open, actively tracked issue stating that the JSON emitted by this exact path isn't actually CADF-conformant. It uses snake_case keys instead of the spec's camelCase, it is missing typeURI and eventType, it has non-standard outcome values, and it hard-codes domain: "default". One finding states plainly: "SIEM CADF parsers cannot ingest the events without translation layers, despite claims of CADF compliance." Canonical's own documented, supported path has this gap.

Option B: SAP's `openstack-audit-middleware`

This is a from-scratch rewrite, explicitly built because the original has a specific, documented weakness: it doesn't properly populate the CADF event's target object. Which resource was actually affected ends up buried in the requestPath attribute or not available at all, according to the project's own README. SAP's version fixes this: the target resource (its type, its ID, and its name) is a first-class, structured part of every event.

We checked the docs and found that it has narrower service coverage. It supports Nova, Neutron, Cinder, Designate, Manila, Glance, Barbican, Ironic and Octavia, and notably not Heat, which matters to us because we have built custom tooling around Heat. The built-in path (Option A), and both Red Hat's and Canonical's distributions, do cover Heat.

Option C: Hermes (SAP), the full product

Hermes is the closest thing to "OpenStack CloudTrail as a product" that exists. The full pipeline looks like this: CADF middleware → RabbitMQ → Hermes-ETL (consumes the queue) → Logstash (enriches with project and domain metadata, and drops noise) → OpenSearch (storage) → Hermes itself, a Go API service that exposes a REST API scoped per project via Keystone RBAC tokens. Optionally, an Elektra module (SAP's own Horizon alternative) renders it as a browsable trail.

Hermes is built for a specific shape of problem: a multi-tenant public cloud, where external paying customers each need self-service access to only their own audit trail, enforced by a dedicated API layer that checks Keystone tokens on every request. That is not our shape. We are a single organization auditing our own private cluster, and the people who would query this are us, the admins, not external customers who need cryptographically enforced isolation from each other.

What the enterprise vendors actually ship

Canonical: the built-in keystonemiddleware path (Option A above), auto-enabled for most services. There is no bundled collection, storage or query layer, and you are expected to wire your own SIEM or logging tool to consume the notifications.
Red Hat OpenStack Platform: the same CADF and keystonemiddleware foundation, documented for Keystone, Nova, Glance, Cinder, Neutron, Heat, Octavia and Ironic. It offers the same two delivery options we had to choose between ourselves: route to "messaging infrastructure or log files". There is no bundled storage or query layer here either.
Mirantis (MOSK + StackLight): the most complete bundled story of the three. CADF events go either to the notifications.info RabbitMQ queue or to plain log output, Fluentd collects them, and OpenSearch stores them, with Mirantis's own dashboard UI on top. This is structurally identical to what we ended up building, with the same message-queue-or-log fork and the same "collector ships to a search backend" shape. The differences are OpenSearch where we used Loki, and Fluentd where we wrote a custom consumer. Mirantis also documents piping these same CADF events onward via Fluentd to external SIEMs (Splunk, ArcSight, QRadar) using Fluentd's Splunk HEC plugin and similar, which is worth knowing if you ever need to feed this into a bigger corporate SIEM rather than your own dashboard.

The cautionary tale: Panko

OpenStack already tried to build a native event-storage-and-query service once: Panko, Ceilometer's event-storage REST API. It is now officially deprecated and unmaintained, which is a useful data point. Across the entire OpenStack ecosystem, including the commercial vendors, the pattern is the same: emit CADF and bring your own collection and storage. That is the go-to approach when you want something like CloudTrail or an audit trail, and nothing more than that.

Part 2: What we actually needed

Three constraints shaped every decision from here:

A central monitoring stack. We already have a central monitoring stack that monitors hosts, services, and applications. It is comprehensive and has some AI features, and we wanted to leverage it. (It is the stack described in my monitoring series.)
A customized UI. Our OpenStack UI is highly customized, and it is easy to add another service, which we will call Footprint: a thin API on top of our existing monitoring data storage that takes all queries and returns the right format.
It must not touch the control-plane RabbitMQ. We have fought RabbitMQ repeatedly: nofile and ulimit exhaustion causing emfile crashes, and stream-queue corruption requiring manual Erlang mnesia:dirty_delete surgery to recover. Audit traffic sharing that broker is not something we want to risk, even in a separate vhost, because a vhost is only logical isolation, not physical (same Erlang VM, same file descriptors, same memory).

Part 3: What we built

The middleware: choosing SAP's version, and actually reading it first

Given Option A's documented non-conformance bug, we went with SAP's openstack-audit-middleware. Before installing it anywhere, we cloned it locally and read the actual source, not just the README.

What we found was legitimate, unremarkable WSGI middleware. There are no eval, exec or subprocess calls, and nothing reaches outside the request and response objects already in process. The request-processing path is wrapped in a _log_and_ignore_error decorator, so any exception in the audit-parsing logic is caught and logged, and never propagated back to break a real API call. This matters, because a bad audit event should never break an OpenStack component.

We found two non-malicious risks, which come with trying to get as close to CloudTrail as possible:

No tamper-evidence of any kind. Anyone with RabbitMQ or storage admin access can alter or delete entries without a trace.
Payload recording is a genuine data-leak trap if enabled. record_payloads defaults to False, but if you turn it on, full request and response JSON bodies get attached to events verbatim unless you configure per-resource exclude lists. Passwords, user data and secrets are all fair game if you are not careful. We left this off.

Installing it: the gotchas

glance-api-paste.ini isn't managed by kolla-ansible at all. We assumed the usual /etc/kolla/config/<service>/ override mechanism would reach it, but it doesn't. The kolla-ansible config.json for glance-api lists exactly the .conf, cache and policy files it templates, and paste.ini isn't one of them. We edited it directly in the running container for the pilot.
But .conf files absolutely are reconciled on every container restart, not just on a full redeploy. Kolla's kolla_set_configs entrypoint deletes and re-copies them from its own source of truth on every start. A manual edit to glance-api.conf did not survive, so we had to redo that specific change through the real /etc/kolla/config/glance/glance-api.conf override path and run kolla-ansible reconfigure --tags glance for it to persist.

Proving the shape before building the real pipeline

Before wiring up a dedicated RabbitMQ at all, we pointed the middleware's notifier at its simplest option, the log driver, which just writes the event straight into the service's own log file. The goal was purely to see the real wire format cheaply. That is how we got a genuine captured event:

{
  "message_id": "1c8b9f39-e702-4e3f-a58f-ad10bf50212d",
  "publisher_id": "glance-api",
  "event_type": "audit.cadf",
  "priority": "INFO",
  "payload": {
    "typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
    "eventType": "activity",
    "action": "read/list",
    "outcome": "success",
    "initiator": {
      "name": "admin",
      "project_id": "a794045dc022430498bb216ad2c0d79d",
      "host": {"address": "10.200.3.100", "agent": "openstacksdk/4.10.0 ..."}
    },
    "target": {"typeURI": "storage/image/images", "name": "glance"},
    "reason": {"reasonType": "HTTP", "reasonCode": "200"},
    "requestPath": "/v2/images"
  }
}

Who, what, when, outcome, source: the whole shape was confirmed. Only once that was verified did we move to the production path: driver = messagingv2 with a dedicated transport_url, which publishes this exact same JSON structure onto a message queue instead of a log file. It never touches the service's own log at all, so no per-service log parsing or filtering is needed downstream.

Why Loki, not Elasticsearch

This was the decision we spent the most time on, and it came down to three things.

1. The math on the control plane didn't work for Elasticsearch. Each control node had 31GB of total RAM but only about 9GB available at the time we checked. The rest was genuinely in use, not just reclaimable cache. Elasticsearch's JVM heap is reserved at startup and held for its exclusive use regardless of what else on the node needs memory, and the standard guidance is to never give it more than about 50% of a node's RAM, with the rest needed for the OS page cache to perform decently. Squeezing even a modest instance onto that node risked the Linux OOM killer taking out a random OpenStack component instead of the thing actually causing the pressure. This cluster also has an existing, documented pattern of exactly this failure mode (a recurring Cinder and Valkey AOF-corruption incident, tracked separately, with the same root cause class: resource contention on shared infrastructure).

2. We didn't need to provision anything new. We already have a central production Prometheus, Loki, Tempo and Alloy monitoring stack on infrastructure genuinely external to the OpenStack cluster itself. It even runs HA Loki, with a multi-replica read, write and backend topology, a memberlist gossip ring, and MinIO (S3-compatible) backing storage. Building a second storage system when a working one already existed, just to avoid using it, would have been the wrong kind of thoroughness.

3. The architectural fit is genuinely better, not just "available." Loki indexes on a small set of labels and treats the rest of each entry as compressed, searchable-but-not-indexed text. CADF events are exactly this shape: a handful of genuinely low-cardinality dimensions (service, action, outcome, project_id) worth indexing, and a pile of per-event detail (target.id, initiator.id, the full request path) that should stay queryable but never become index labels. That is Loki's whole design point. Elasticsearch's full-document indexing solves a different problem.

What we explicitly didn't need, as a result: no Logstash (its job in the Hermes pipeline is mostly compensating for Option A's weak target and project population, and SAP's middleware already puts clean structured fields directly in the payload at the source), no Elasticsearch or OpenSearch, and no Hermes.

The consumer

We had proved that the events are logged to a file, and we had provisioned a dedicated RabbitMQ for the audit events. The next step was to consume the events from the queue and ship them to Loki. We wrote a small consumer service that connects to the dedicated RabbitMQ, listens for messages, and forwards them to Loki using its HTTP API. The consumer is lightweight and focused solely on translating the CADF JSON into a format suitable for Loki.

The query layer: a thin API, not a direct Skyline-to-Loki connection

The last real design decision: should Skyline query Loki directly, or should there be a small service in between?

We went with a thin API in front of Loki, for one specific reason: project-scoping correctness. The Loki instance has no native per-tenant isolation at all, so whatever service queries it is solely responsible for getting the project filter right, on every request, forever. Putting that logic in one small, narrow, easily tested service means it only has to be correct in one place, and not re-derived every time a new audit panel is added to a larger dashboard codebase. The service just trusts the already-authenticated project_id that Skyline's own backend hands it, and turns it into a LogQL label filter. It is not a Hermes-style product: no Postgres, no independent token validation, just translation and project-scoping in front of Loki's query API.

Part 4: What we proved, and what's still open

The full pipeline (middleware → dedicated RabbitMQ → consumer → Loki) is proven end to end with real traffic against one service, Glance. We started with a single service to prove the approach.

A real captured event in Loki showed exactly the label hygiene we had planned for. Only action (truncated to its primary verb), outcome, project_id and service were promoted to labels. The full CADF JSON was preserved as the log line body, and nothing high-cardinality (initiator.id, target.id, message_id, requestPath) was anywhere near the index.

The build also added two smart touches we hadn't specified. One is a service_name label matching this Loki's existing label vocabulary, so audit data joins cleanly with everything else already in there. The other is a source: "openstack-audit" label, which turns out to be exactly the selector needed for a Loki retention-stream override. Audit data almost certainly needs to outlive this instance's default 14-day retention, and now there is a clean, unambiguous way to say so.

Still open:

No Heat coverage. The middleware's supported-service list doesn't include it, so autoscaling activity stays outside this system regardless of anything else we do here. Since Heat is a critical component for us, we will have to decide whether to contribute support for it to SAP's project.
project_name isn't in the event at all, only project_id. Keystone's authtoken middleware already sets an X-Project-Name header on every authenticated request, the same way it sets X-User-Name, which is read, but the middleware's OpenStackResource class has no slot for it. That is another small, legitimate contribution-back candidate if we ever want it at the event level. More likely, Skyline or the consumer will resolve project_id to a name with an API call to OpenStack, since Skyline needs that mapping for everything else it displays anyway.

Part 5: Honestly, how close is this to CloudTrail?

The shape is right: who, what, when, where, outcome and target, proven working end to end with real traffic. What is genuinely missing, measured honestly against AWS's actual product:

No tamper-evidence. CloudTrail's S3-backed storage with integrity digests means logs can't be silently altered after the fact. Ours has no such guarantee. This matters if audit trails need to be defensible evidence and not just operational visibility.
No automated analysis. There is no Athena equivalent for ad-hoc SQL, no EventBridge-style real-time reaction to specific API calls, and no GuardDuty-style anomaly detection on the trail itself. We have just LogQL queries. But with Loki we can do this the same way we set alerts for errors in application logs: we can set alerts for specific events in the audit trail.
No self-monitoring. If someone kills the consumer or the dedicated RabbitMQ, nothing specifically notices. In AWS, disabling CloudTrail is itself a loggable, alertable event.

But, and this is the actual point of surveying Canonical, Red Hat, and Mirantis above, none of them solve these gaps either. Every commercial OpenStack vendor converges on the same tier: emit CADF, ship it somewhere queryable, and build your own dashboard on top. What we built isn't a lesser version of some existing, solved product. It is at parity with what the big vendors actually ship, and, given Option A's documented CADF non-conformance bug, arguably more standards-correct than Canonical's own default path.