AlmaForge Storage Guide

AlmaForge starts with SQLite cluster state and local audit and recording storage for a single-host evaluation. No external storage service is needed for the Standalone guide. See SQLite and Local files to customize these defaults.

For production, choose shared backends before adding Auth replicas. The StorageConfig resource configures three kinds of data. Its name selects the purpose, and the driver field under spec selects the backend.

Cluster state. Every resource the cluster knows about. Users, roles, connectors, tokens, registered nodes and databases, certificate authorities, locks, access requests, session metadata. One backend. Required. Configured as StorageConfig with metadata.name: state.
Audit events. Structured JSON records of what happened. Logins, role changes, session starts and stops, lock events, command executions. Configured as StorageConfig with metadata.name: audit (the default, a local file sink materialized automatically when nothing else is set) or any name starting with audit- (named sinks for fan-out, where every emitted event goes to every audit-* instance).
Session recordings. Raw terminal and Kubernetes session streams. One destination. Configured as StorageConfig with metadata.name: recordings.

A StorageConfig resource looks like this:

state.yaml:

examples/storage/overview/state.yaml
# Minimal shared state backend using DynamoDB.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: state
spec:
  dynamo:
    region: us-east-1
    tableName: cluster-state

Set exactly one driver field under spec in each resource. In this example, dynamo selects DynamoDB for cluster state.

Bootstrap-only

StorageConfig is bootstrap-only. Place its manifests in /etc/almaforge/almad.d on each Auth host before startup. To change storage configuration, update those files on every Auth host and restart the Auth Service pool. See Infrastructure as Code for file ownership and configuration loading.

The Auth Service must know its storage endpoints before connecting to cluster state. Audit and recording destinations follow the same startup configuration model. Changing a destination does not migrate data already stored in the previous backend. Plan backups and recovery before a storage cutover.

You can read the active configuration any time:

Terminal
alma get storageconfig -o yamlalma get storageconfig/state -o yamlalma get storageconfig/audit-primary -o yamlalma get storageconfig/recordings -o yaml

alma get reports the active configuration with sensitive credentials redacted.

The CLI can read StorageConfig, but cannot apply, edit, or delete it.

Cluster state

The state backend is the cluster's source of truth. Every Auth Service instance reads and writes through it. There is exactly one StorageConfig with metadata.name: state, and every Auth instance must point at the same backend with the same parameters.

Driver field	Use when	HA story	Page
`spec.sqlite`	dev, evaluation, single-node	none	SQLite
`spec.postgres`	self-hosted HA, Azure, bare metal	replica + failover you run	PostgreSQL
`spec.dynamo`	AWS deployments	AWS-managed	DynamoDB
`spec.etcd`	VM HA with local quorum, or shops with existing etcd	self-hosted quorum	etcd
`spec.firestore`	GCP deployments	GCP-managed	Firestore

SQLite supports one Auth instance with local state. For multiple Auth instances, use a shared backend and point every instance at the same store. Separate SQLite files do not form one HA cluster, and sharing a SQLite directory between Auth processes is unsupported.

VM HA state topology

VM deployments usually start with three control-plane VMs running Auth and Proxy. Choose where cluster state lives before you write the VM runbook or build firewall rules. The Linux deployment guide uses an external etcd backend by default. You can run those etcd members on separate hosts or colocate them with the control plane, provided you budget disk performance and failure domains for both services.

Storage backend	Use when	Topology effect
etcd	You operate an etcd quorum for cluster state.	Run three members on separate hosts, or colocate one member on each cluster VM with sufficient resources.
PostgreSQL	You already run a managed or self-hosted HA PostgreSQL service.	Remove colocated etcd and depend on PostgreSQL availability.
DynamoDB	The cluster runs in AWS and you want managed state storage.	Remove colocated etcd and depend on DynamoDB IAM, region, and table capacity.
Firestore	The cluster runs in GCP and you want managed state storage.	Remove colocated etcd and depend on Firestore IAM and project availability.

The Auth and Proxy VM shape remains the same unless you intentionally choose a different architecture. PostgreSQL, DynamoDB, and Firestore replace only the state-backend part of the default topology.

Audit events

Audit events are sent to every StorageConfig instance named audit or matching audit-*. AlmaForge materializes a default audit instance with a local file driver when the operator configures nothing, so an audit destination is configured out of the box and visible in alma get storageconfig. Override that default by authoring your own audit instance, or add audit-<suffix> instances for fan-out to multiple destinations. Typical fan-out is one primary external store, one local file backup, and a third for a SIEM pipeline. Monitor audit delivery failures on every deployment.

Driver field	Target	Page
`spec.dynamo`	DynamoDB (use a different `tableName` than the state instance)	DynamoDB
`spec.postgres`	PostgreSQL (use a different database than the state instance)	PostgreSQL
`spec.firestore`	Firestore (use a different `collectionName` than the state instance)	Firestore
`spec.file`	Local files on each Auth host	Local files
`spec.stdout`	The Auth Service's own standard output	Standard output
`spec.athena`	AWS Athena over S3 + SNS + SQS + Glue	not currently documented

A typical fan-out config has two StorageConfig instances: one for the primary external store, one for a local file backup that survives a state-backend outage.

audit-fanout.yaml:

examples/storage/overview/audit-fanout.yaml
# Primary DynamoDB audit storage plus a local investigation copy.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: audit-primary
spec:
  dynamo:
    region: us-east-1
    tableName: cluster-audit
---
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: audit-local
spec:
  file:
    path: /var/lib/almaforge/log/audit

Audit searches use the first configured destination that supports the search operation. Search results are not merged across destinations, and an error from that destination does not trigger a search of the next one. Put the destination you want to search first in the bootstrap configuration.

Retention lives on the driver block as auditRetentionPeriod. Only the drivers whose backend implements retention in code honor it: dynamo (per-item TTL), postgres (background sweeper), and firestore (background purge). athena delegates retention to S3 lifecycle rules and file rotates without deleting, so the field is rejected on those drivers and on non-audit-* instances.

Standard output

stdout writes each audit event to the Auth Service's own standard output and takes no settings at all. Select it with stdout: {} on an audit or audit-<suffix> instance.

Use it for development and smoke tests, where watching events scroll past beats wiring a real sink, and on a host whose log collector already ships the process output. It stores nothing itself, so pair it with a durable sink for anything you have to be able to search later.

Session recordings

Session recordings land in one destination, configured as StorageConfig with metadata.name: recordings. If you need a second copy, configure replication or a backup process in the storage service. Lifecycle rules control retention or storage class, not a second independent recording destination.

Driver field	Target	Page
`spec.s3`	Amazon S3 or any S3-compatible object store	S3
`spec.gcs`	Google Cloud Storage	GCS
`spec.azure`	Azure Blob Storage	Azure Blob
`spec.file`	local directory on the Auth host	Local files

Recording storage (where the files land) is independent of recording mode (where and how the stream is captured). Mode lives in the RecordingConfig resource and takes values node, proxy, node-sync, proxy-sync, or off.

Pick a set

Most operators run matched sets so they do not cross cloud boundaries.

Environment	State	Audit primary	Recordings
AWS	DynamoDB	DynamoDB (second table)	S3
GCP	Firestore	Firestore (second collection)	GCS
Azure	PostgreSQL (Azure AD)	PostgreSQL (second database)	Azure Blob
Self-hosted, bare metal	PostgreSQL	PostgreSQL (second database)	S3-compatible
On-prem Kubernetes with existing etcd	etcd	PostgreSQL or Local files	S3-compatible
Dev and evaluation	SQLite	Local files	Local files

The driver pool is shared across instance names. A dynamo driver block looks the same whether it is configuring name: state or name: audit-primary. The file driver works for both audit and recordings. There is no per-instance schema duplication.

A complete bootstrap example

A small AWS deployment using DynamoDB for state, two audit-* sinks (DynamoDB primary plus local file backup), and S3 for recordings:

production.yaml:

examples/storage/overview/production.yaml
# Shared state, audit fan-out, and S3 recordings for a production cluster.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: state
spec:
  dynamo:
    region: us-east-1
    tableName: cluster-state
    billingMode: PAY_PER_REQUEST
    continuousBackups: true
---
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: audit-primary
spec:
  dynamo:
    region: us-east-1
    tableName: cluster-audit
---
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: audit-local
spec:
  file:
    path: /var/lib/almaforge/log/audit
---
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: recordings
spec:
  s3:
    bucket: almaforge-recordings
    region: us-east-1
    path: records

Place this in the Auth Service bootstrap file on every Auth host (the file must be byte-identical across hosts), then start the Auth Service. After startup, verify with alma get storageconfig.

URI format

Every driver documented below can also be expressed as a single URI string. The same URI grammar drives every env-var and CLI-flag bootstrap. The URI scheme picks the driver, and the host and path carry the primary value, such as a table name, bucket, file path, or database.table for Athena. Per-driver knobs travel either in the URI query string or, for PostgreSQL retention knobs and Azure container names, in the URI fragment after #. Each backend page below shows where its knobs live.

Canonical query keys use snake_case. The camelCase form used in StorageConfig resources is accepted as an alias. Durations use values such as 500ms, 10s, or 72h. An invalid query value stops configuration loading and reports the field instead of using a default.

A few representative examples:

text
sqlite:///var/lib/almaforge/backend
postgres://alma:[email protected]/almaforge?sslmode=verify-full#retention_period=72h
dynamodb://cluster-events?region=us-east-1&audit_retention_period=720h
file:///var/log/almaforge/audit
s3://almaforge-recordings/sessions?region=us-east-1&sse_kms_key=arn:aws:kms:::key/abc

The full grammar for each scheme, including every query knob, every alias, and the host/path/fragment shape, lives on the corresponding backend page below. Two schemes have no page of their own: athena://<database>.<table> is parsed but not yet documented, and stdout:// takes nothing after the scheme and selects the sink described under Standard output.

Next steps

High availability. How the storage backends fit into a production topology.
Break-glass access. What to do when the state backend is down.

Bootstrap-only​

Cluster state​

VM HA state topology​

Audit events​

Standard output​

Session recordings​

Pick a set​

A complete bootstrap example​

URI format​

Next steps​