Skip to main content

AlmaForge Cluster on Kubernetes

This guide builds a production high-availability deployment of AlmaForge on Kubernetes. Three replicas of one almad StatefulSet run Auth and Proxy in the same process behind one external LoadBalancer Service on TCP/443. cert-manager issues the public TLS leaf and trust-manager distributes the cluster CA. The first administrator signs in through your existing SSO provider.

Looking for the 10-minute eval?

Try AlmaForge with the Quick Start demo.

Topology​

One almad StatefulSet with three replicas sits behind a mandatory external LoadBalancer Service on TCP/443. Each pod runs Auth and Proxy in the same process. cert-manager issues the public TLS leaf, and etcd holds cluster state.

Kubernetes HA topology with external storage

  • almad StatefulSet (3 replicas). Each replica runs both Auth and Proxy in a single almad process. All replicas are equal peers. The first to start initializes cluster state in etcd. Subsequent replicas detect existing state and join as additional peers. No replica is "primary". topologySpreadConstraints on the StatefulSet favor spreading the replicas across zones. The example uses ScheduleAnyway, so verify actual pod placement before relying on zone-failure tolerance.
  • External LoadBalancer. A Service of type LoadBalancer forwards TCP/443 to each pod's 8443 listener without decrypting. AlmaForge terminates TLS on the pod so the cert chain stays end-to-end inside the cluster.
  • etcd cluster. Three members give a quorum of two (see How replicas coordinate). Every replica reads and writes the same etcd. The examples/kubernetes/etcd/ bundle runs etcd in the same Kubernetes cluster for evaluation. Production deployments typically use externally managed etcd.
  • Audit events stream to an external database (PostgreSQL, DynamoDB, Firestore, or Athena) so they survive any pod loss.
  • Session recordings stream to external object storage (S3, GCS, or Azure Blob). Choose both per cloud in Storage backends below.

Provision infrastructure​

Review the baseline System Requirements before you plan the deployment. You build the platform pieces (Kubernetes cluster, cert-manager, DNS, TLS, etcd, storage backends) before installing AlmaForge. These pieces are typically owned by your platform, networking, and SRE teams. Hand the cluster and the public hostname off to whoever runs AlmaForge and they can take it from Provision AlmaForge.

The default path runs the example kustomize overlay shipped with this guide. Treat the overlay as a starting point you fork or patch from your own GitOps repository.

Kubernetes cluster​

Run on any Kubernetes 1.27 or later cluster, on x86_64 or arm64. Verify with kubectl version. Each almad pod needs roughly 1 vCPU and 1 GiB to start, scaling with concurrent sessions and registered targets. See Scaling for sizing signals.

The overlay creates a dedicated namespace (almaforge-cluster by default) and a ServiceAccount named almaforge. The cluster RBAC grants the system:auth-delegator ClusterRole so the Auth service of each replica can validate in-cluster agents' ServiceAccount JWTs through the TokenReview API. No shared secret or CA pin is required for in-cluster agent joins.

Firewalls​

Open the following ports for this Kubernetes HA layout. Only TCP/443 on the external LoadBalancer needs to be reachable by clients.

PortDescription / purpose
TCP/443Client-reachable on the external LoadBalancer for users, agents, automation, and alma. The LoadBalancer forwards TCP/443 to each almad pod's 8443 listener.
TCP/8443Pod ALMA_LISTEN for the public Proxy multiplexer. Private to the LoadBalancer and to in-cluster agents that resolve the almaforge headless Service.
TCP/2379Private etcd client API used by the Auth service of each pod. Keep inside the cluster network or private VPC.
TCP/2380Private etcd peer replication when you run the example etcd StatefulSet.
TCP/3000Private diagnostics endpoint for /healthz and /readyz when ALMA_DIAG_LISTEN is enabled. Reachable from kubelet probes only.
TCP/3081Private Prometheus metrics endpoint when ALMA_METRICS_LISTEN is enabled. Reachable from your monitoring namespace only.
Outbound connectionsAllow the configured backend ports, image-registry HTTPS from nodes, OIDC discovery, signing-key and token endpoints from Auth pods, and any cert-manager issuer API. See egress requirements.

If your cluster uses NetworkPolicy, allow the LoadBalancer to reach pods on 8443, pods to reach etcd, and monitoring systems to reach the diagnostics and metrics ports. The bundle ships a baseline networkpolicy.yaml commented out in kustomization.yaml. Uncomment it after adjusting to your CNI and monitoring layout.

cert-manager and trust-manager​

AlmaForge reads its public TLS leaf from a Secret that cert-manager issues and renews. trust-manager distributes the cluster CA to every namespace as a ConfigMap named almaforge-ca so any pod that needs to verify the cluster CA can mount it.

The examples/kubernetes/certmanager/ bundle installs both with Helm for evaluation. In production, install them through your platform team's standard channel (Helm, ArgoCD, or your cluster add-on catalog):

Terminal
make -C examples/kubernetes/certmanager setupkubectl -n cert-manager get podsNAME                              READY   STATUScert-manager-xxx                  1/1     Runningcert-manager-cainjector-xxx       1/1     Runningcert-manager-webhook-xxx          1/1     Runningtrust-manager-xxx                 1/1     Running

cert-manager 1.13 or later is required. Wait for every pod above to reach Running before continuing.

DNS and TLS​

AlmaForge needs a stable public hostname and a TLS certificate that covers it. Both are upstream of the deployment: pick the hostname before you bring any pod online, because it ends up in the TLS certificate, the pod's advertise address, and every join token. Changing it later means re-issuing certificates and re-joining agents.

The minimum DNS records:

RecordPurpose
almaforge.example.com A / AAAAPublic address of the almaforge-lb Service.
*.almaforge.example.com A / AAAAWildcard subdomain for applications.

You can only set the records after the external Service exists and has an external IP. See Step 7.

The TLS certificate must include:

  • CN=almaforge.example.com
  • SAN: almaforge.example.com
  • SAN: *.almaforge.example.com

Use one of:

  • Organization-issued from your internal CA. Create a ClusterIssuer backed by it and have it issue the almaforge-web Certificate referenced by the bundle. Recommended for production. Workstations need to trust the internal CA before Step 8 sign-in. The Troubleshooting section has the per-OS commands.
  • Self-signed for short-lived evaluation. The examples/kubernetes/certmanager/ca.yaml bundle ships a self-signed root, an issuing intermediate, and three ClusterIssuers for this case. Same workstation trust step applies.

The leaf is declared in certificate.yaml:

examples/kubernetes/cluster/certificate.yaml
# Web TLS leaf for the almad StatefulSet, issued by the almaforge-web
# ClusterIssuer that the ../certmanager bundle owns. The cluster CA and
# the trust-manager Bundle live in that bundle, so this namespace can be
# torn down and recreated without losing the CA or its trust
# distribution.
#
# dnsNames must include the public hostname clients dial through the
# external LoadBalancer (the value of ALMA_ADVERTISE in almad.env).
# Edit 'almaforge.example.com' below to match your real hostname before
# first apply, alongside the same change in etc/almaforge/almad.env.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: almaforge-web
spec:
  secretName: almaforge-web
  issuerRef:
    name: almaforge-web
    kind: ClusterIssuer
  commonName: almaforge.example.com
  dnsNames:
    - almaforge.example.com
    - almaforge.almaforge-cluster.svc
    - almaforge.almaforge-cluster.svc.cluster.local
    - '*.almaforge-cluster.svc'
    - '*.almaforge-cluster.svc.cluster.local'
  duration: 2160h
  renewBefore: 720h
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always

cert-manager writes the certificate and key into the Secret almaforge-web in the almaforge-cluster namespace. The almad StatefulSet mounts that Secret at /etc/almaforge/tls.crt and /etc/almaforge/tls.key (the same paths the Linux deployment uses). Before applying the overlay, edit certificate.yaml (or patch it from your overlay) so that spec.commonName and spec.dnsNames match your public hostname and wildcard subdomain. The duration (90-day leaf) and renewBefore (30 days) values come from the example bundle.

For public-CA certificates (Let's Encrypt, ZeroSSL), use a DNS-01 solver. HTTP-01 cannot complete the challenge inside the cluster before the public hostname resolves. The DNS-01 solver runs against your DNS provider's API (Route53, Cloud DNS, Azure DNS, Cloudflare). See your provider's cert-manager solver docs.

Storage backends​

Choose storage before applying the overlay. The deployment needs three storage decisions:

  • Cluster state: resources, identities, certificate authorities, locks, access requests, and registered targets.
  • Audit events: structured records of logins, role changes, locks, session starts, and other control-plane actions.
  • Session recordings: terminal and Kubernetes session streams.

The Storage Guide is the canonical place to choose those backends. This Kubernetes runbook uses etcd for cluster state because it matches the HA shape used by the Linux guide. Pick audit and recording backends from the Storage Guide before first start.

Provision the etcd cluster​

Every almad replica reads and writes the same etcd cluster. Three members give a quorum of two. One etcd member can be down and write availability stays.

The examples/kubernetes/etcd/ bundle deploys a 3-replica etcd StatefulSet in the etcd namespace for evaluation:

Terminal
cd examples/kubernetes/etcd && make setupkubectl -n etcd get podsNAME     READY   STATUSetcd-0   1/1     Runningetcd-1   1/1     Runningetcd-2   1/1     Running

The bundle uses one PersistentVolumeClaim per replica at 256Mi on the cluster's default StorageClass. Resize the PVC and pin a specific StorageClass before any non-throwaway use.

For production, run etcd outside the Kubernetes cluster, enable etcd mTLS, and replace the state drop-in with the mTLS shape from the etcd Storage Guide. That page is the canonical home for the certificate profile, peer mTLS layout, and capacity planning.

The example bundle ships starting points for audit and recordings:

For cloud Kubernetes, prefer attaching cloud identity to the almaforge ServiceAccount over static credentials in the manifest: IRSA on EKS, Workload Identity on GKE, or Azure Workload Identity on AKS. Add the provider-specific annotation or label to the ServiceAccount in your overlay before first apply.

SSO​

AlmaForge authenticates operators through your existing OIDC provider, such as Google Workspace, Okta, Entra ID, Auth0, Keycloak, or any other compliant IdP. Provision the OIDC application on the IdP side now so the credentials are ready when you fill in the connector drop-in.

You need three values from the IdP:

ValueGoes intoNotes
Issuer URLspec.issuerURL in 10-sso.yamlThe IdP's OIDC discovery base. Examples: https://accounts.google.com, https://<tenant>.okta.com, https://login.microsoftonline.com/<tenant-id>/v2.0.
Client IDALMA_SSO_CLIENT_ID in the oidc-client SecretPublic identifier for the OIDC app.
Client secretALMA_SSO_CLIENT_SECRET in the oidc-client SecretTreat as a secret. The connector reads it with ${env:ALMA_SSO_CLIENT_SECRET}.

Register the redirect URI on the IdP side using the public hostname you configured in DNS and TLS: https://almaforge.example.com/api/v1/oidc/callback.

AlmaForge derives the same URL from ALMA_ADVERTISE at runtime. A mismatch between what you register on the IdP and what ALMA_ADVERTISE produces surfaces at sign-in as redirect_uri_mismatch.

See the OIDC connector guide for full claim mapping, scopes, PKCE, and provider-specific setup. Per-IdP walk-throughs: Google, Okta, Microsoft.

Provision AlmaForge​

The full deployment is one kustomize overlay. Fork it, edit the env file and drop-ins for your environment, and apply.

Step 1. Fork the overlay​

Fork the shipped overlay into your GitOps repository, or reference it as a base from your own kustomization.yaml. In the commands below, replace <your-overlay> with that forked overlay path.

kustomization.yaml:

examples/kubernetes/cluster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: almaforge-cluster

resources:
  - namespace.yaml
  - serviceaccount.yaml
  - role.yaml
  - rolebinding.yaml
  - clusterrolebinding.yaml
  - certificate.yaml
  - service.yaml
  - service-lb.yaml
  - statefulset.yaml
  - poddisruptionbudget.yaml
  # Uncomment for cluster network isolation. See networkpolicy.yaml.
  # - networkpolicy.yaml
  # Uncomment with prometheus-operator installed. See podmonitor.yaml.
  # - podmonitor.yaml

configMapGenerator:
  - name: auth-config
    files:
      - 00-cluster-name.yaml=etc/almaforge/almad.d/00-cluster-name.yaml
      - 01-state-storage.yaml=etc/almaforge/almad.d/01-state-storage.yaml
      - 02-audit-storage.yaml=etc/almaforge/almad.d/02-audit-storage.yaml
      - 03-recording-storage.yaml=etc/almaforge/almad.d/03-recording-storage.yaml
      - 10-sso.yaml=etc/almaforge/almad.d/10-sso.yaml
    options:
      disableNameSuffixHash: true

secretGenerator:
  # almad-env carries listener addresses, advertise hostnames, and
  # (when present) ALMA_SSO_CLIENT_ID/SECRET. Stored as a Secret so
  # it can be managed by the same Secret pipeline as oidc-client
  # (SealedSecrets, External Secrets, sops, ...).
  - name: almad-env
    envs:
      - etc/almaforge/almad.env
    options:
      disableNameSuffixHash: true

The overlay packs one ConfigMap and one Secret:

  • auth-config ConfigMap from the files under almad.d/: ClusterName, StorageConfig, OIDCConnector, and any other bootstrap resources the cluster loads at first start.
  • almad-env Secret from almad.env, every operator-tunable env var, in the same KEY=VALUE shape as /etc/almaforge/almad.env on a Linux host. Stored as a Secret so the same pipeline that manages the OIDC oidc-client Secret (SealedSecrets, External Secrets Operator, sops) covers cluster config too.

The shipped statefulset.yaml pulls the container image with the latest tag. The current release is 2026.1.26. Pin it in your overlay (kustomize edit set image quay.io/almaforge/almaforge:2026.1.26) or replace latest directly so an unattended rollout cannot ride a newer image you have not validated.

Step 2. Edit almad.env​

almad.env:

examples/kubernetes/cluster/etc/almaforge/almad.env
# Environment file for an AlmaForge cluster pod.
# Runs both Auth and Proxy services in one process. Mirrors the Linux
# almad.env. kustomize packs this file into the almad-env ConfigMap
# and the StatefulSet pulls it in via envFrom. Edit ALMA_ADVERTISE
# for your public hostname before first apply.

ALMA_AUTH_SERVICE=true
ALMA_PROXY_SERVICE=true

# The cluster pod serves no SSH sessions.
ALMA_SSH_SERVICE=false

# Public Proxy listener inside the pod. The external 'almaforge-lb'
# Service forwards TCP/443 to this port. The container runs as
# non-root, so the listener stays on an unprivileged port. The Linux
# equivalent binds 443 directly.
ALMA_LISTEN=0.0.0.0:8443

# Public hostname clients connect to. Must match the TLS cert and the
# DNS A/AAAA pointing at the external Service.
ALMA_ADVERTISE=almaforge.example.com:443

# Public TLS certificate and key, mounted from the almaforge-web
# Secret that cert-manager populates. Same paths as the Linux layout.
ALMA_TLS_CERT=/etc/almaforge/tls.crt
ALMA_TLS_KEY=/etc/almaforge/tls.key

# Private diagnostics for kubelet probes and operator checks.
ALMA_DIAG_LISTEN=0.0.0.0:3000

# Private Prometheus metrics endpoint. Scrape from monitoring only.
ALMA_METRICS_LISTEN=0.0.0.0:3081

# Static labels attached to this cluster node in inventory and audit.
ALMA_LABELS=env=prod,team=platform

# Trust the cluster CA bundle distributed by trust-manager.
SSL_CERT_FILE=/etc/ssl/almaforge/ca-bundle.pem

# ALMA_SSO_CLIENT_ID and ALMA_SSO_CLIENT_SECRET are interpolated by
# the OIDC connector under almad.d/. They come from the
# operator-created 'oidc-client' Secret (envFrom on the StatefulSet),
# not from this ConfigMap. See README.md.

The only field you must change before first apply is ALMA_ADVERTISE. Set it to the public hostname from DNS and TLS. Everything else is a sensible default for a 3-node HA cluster. Edit the file in your overlay and re-apply to change it, then roll the StatefulSet. Values supplied through envFrom are read when a container starts, so updating the Secret alone does not change the running process environment.

Step 3. Pin the cluster name​

ClusterName is a free-form identifier the cluster carries forever. Pick any stable name. The public hostname is the common convention, because it shows up in alma login --proxy=<hostname> and in issued certificates, but anything stable works (prod, almaforge-corp, cluster-1). Once the first pod finishes initializing cluster state in etcd, the value becomes immutable. Skipping this file locks the cluster name to the first pod's hostname.

almad.d/00-cluster-name.yaml:

examples/kubernetes/cluster/etc/almaforge/almad.d/00-cluster-name.yaml
# Cluster identity. Set this before first apply. The values become
# immutable once the first pod initializes cluster state in etcd.
# Changing them later requires re-bootstrapping the cluster.
# Loaded into the auth-config ConfigMap by kustomize.
apiVersion: almaforge.com/v1
kind: ClusterName
metadata:
  name: cluster-name
spec:
  # Public cluster hostname shown to users and embedded in issued certs.
  clusterName: almaforge.example.com

The shipped kustomization.yaml already references this file under configMapGenerator.

Step 4. Add storage drop-ins​

AlmaForge reads bootstrap resources from /etc/almaforge/almad.d at process start. The shipped state drop-in points at the in-cluster etcd from Provision the etcd cluster:

almad.d/01-state-storage.yaml:

examples/kubernetes/cluster/etc/almaforge/almad.d/01-state-storage.yaml
# Cluster state backend: in-cluster etcd shipped under
# examples/kubernetes/etcd/. For production, run etcd outside the
# Kubernetes cluster, swap spec.etcd for the mTLS shape from
# storage/etcd.mdx (or spec.dynamo / spec.firestore). See
# storage/index.mdx for the matrix.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: state
spec:
  etcd:
    # Client URLs for every etcd member. The Service DNS names below
    # resolve to the StatefulSet pods in the etcd namespace.
    peers:
      - http://etcd-0.etcd.etcd.svc:2379
      - http://etcd-1.etcd.etcd.svc:2379
      - http://etcd-2.etcd.etcd.svc:2379
    # Key prefix that keeps AlmaForge data separate from other etcd users.
    prefix: /almaforge/
    # insecure: true is intentional for this evaluation layout. The
    # in-cluster etcd Service listens on cluster-internal addresses
    # only and traffic never leaves the pod network. For external
    # etcd, drop this line and provide tlsCAFile/tlsCertFile/tlsKeyFile.
    insecure: true

For an external managed etcd cluster, replace this file with the mTLS shape from the etcd Storage Guide and mount the CA and client key material as a Secret into the StatefulSet at the paths the StorageConfig references.

Add the audit and recording drop-ins you chose in Storage backends alongside the state drop-in. Numeric prefixes organize the files for readers. Resource loading handles bootstrap dependencies independently of those prefixes:

text
examples/kubernetes/cluster/etc/almaforge/almad.d/
|-- 00-cluster-name.yaml
|-- 01-state-storage.yaml
|-- 02-audit-storage.yaml
`-- 03-recording-storage.yaml

Add the new files to the configMapGenerator block in your overlay's kustomization.yaml so kustomize repacks the auth-config ConfigMap.

For cloud-managed object storage, attach the bucket identity to the almaforge ServiceAccount before first apply rather than embedding credentials in the manifest:

PlatformServiceAccount metadata
EKS IRSAAnnotation eks.amazonaws.com/role-arn with the IAM role ARN.
GKE Workload IdentityAnnotation iam.gke.io/gcp-service-account with the Google service account email.
AKS Workload IdentityAnnotation azure.workload.identity/client-id with the managed identity client ID on the ServiceAccount. Put label azure.workload.identity/use=true on the StatefulSet's pod template.

For the AKS identity binding and pod label, follow the Microsoft Entra Workload ID guide.

Use the StorageConfig reference for the full schema.

Step 5. Add the OIDC connector​

A fresh cluster has no administrator session yet, so you cannot bootstrap the first connector with alma apply -f. Add the connector as a startup drop-in before first apply. When the first replica comes up, the connector is already in place and the first administrator can sign in immediately.

almad.d/10-sso.yaml:

examples/kubernetes/cluster/etc/almaforge/almad.d/10-sso.yaml
# OIDC connector loaded from almad.d at startup. Replace the
# claim/value pair with the SSO claim that should grant the first
# administrator access.
#
# Loaded into the auth-config ConfigMap by kustomize. Replace the
# issuerURL placeholder below with your IdP's OIDC discovery base.
# ${env:ALMA_SSO_CLIENT_ID} and ${env:ALMA_SSO_CLIENT_SECRET} are
# interpolated from the operator-created oidc-client Secret (mounted
# via envFrom on the StatefulSet).
apiVersion: almaforge.com/v1
kind: OIDCConnector
metadata:
  name: corp
spec:
  issuerURL: <paste-from-your-IdP>
  clientID: ${env:ALMA_SSO_CLIENT_ID} # read from environment (oidc-client Secret)
  clientSecret: ${env:ALMA_SSO_CLIENT_SECRET} # read from environment (oidc-client Secret)
  scope:
    - openid
    - email
    - profile
    # Uncomment for Microsoft Entra, Okta, and other providers.
    # Google rejects this scope. Use googleAdminEmail and
    # googleServiceAccount instead.
    # https://almaforge.com/sso/google/
    # - groups
  claimsToRoles:
    # Map IdP groups to AlmaForge roles. Every claimsToRoles entry
    # is evaluated on every sign-in, and the user receives the union
    # of all matched roles.
    #
    # `claim` is the OIDC claim name as your IdP emits it. `groups`
    # is the convention for Okta, Entra ID, Keycloak, and Auth0.
    # If your IdP does not emit groups, map `email` or another claim
    # that your IdP includes in the ID token.
    #
    # `value` has to match what the provider actually sends, which is
    # often not the name you see in its admin console:
    #
    #   Okta              group-name
    #   Microsoft Entra   GUID of the group
    #   Google            [email protected]
    #
    # `roles` is a list of AlmaForge role names. Built-in roles are
    # `admin` (everything), `auditor` (read-only plus audit),
    # `access` (use resources), `member` (see and request access),
    # and `reviewer` (review access requests). Define custom roles as
    # additional drop-ins under almad.d/.
    - claim: groups
      value: admins
      roles:
        - admin
    - claim: groups
      value: auditors
      roles:
        - auditor

Add the file to the configMapGenerator block in your overlay's kustomization.yaml. Set spec.issuerURL to the value from SSO.

claimsToRoles is the bridge between your IdP and AlmaForge RBAC. The example maps two IdP groups to built-in roles: admins to admin (every resource, including the roles and connectors that decide who else has access) and auditors to auditor (read-only plus audit). The other built-in roles are member, access, and reviewer. See Built-in roles. Replace admins and auditors with the group names your IdP actually emits.

The connector reads ${env:ALMA_SSO_CLIENT_ID} and ${env:ALMA_SSO_CLIENT_SECRET} from the pod environment. The StatefulSet pulls them from a Secret named oidc-client via an optional envFrom. Create it before first apply (the bundle ships namespace.yaml, so create the namespace first):

Terminal
kubectl apply -f <your-overlay>/namespace.yamlkubectl -n almaforge-cluster create secret generic oidc-client \    --from-literal=ALMA_SSO_CLIENT_ID='<client-id>' \    --from-literal=ALMA_SSO_CLIENT_SECRET='<client-secret>'

For GitOps deployments, manage oidc-client through your team's standard Secret operator (SealedSecrets, External Secrets Operator, sops). The same applies to the almad-env Secret generated from almad.env.

The connector's metadata.name is corp. It shows up as the choice on the browser sign-in page in Step 8.

Step 6. Pre-apply check​

Verify no placeholders remain in the rendered overlay before applying:

Terminal
kubectl kustomize <your-overlay> \    | grep -E 'example\.com|<paste-from'

The command should return no matches. Any hit means a placeholder slipped through and the daemon will fail to start or sign-in will fail later.

Step 7. Apply the overlay​

Sanity-check the overlay before applying it:

Terminal
kubectl apply --dry-run=client -k <your-overlay> -o name

The output should include the namespace, almaforge ServiceAccount, RBAC, the almad-env Secret and auth-config ConfigMap, the almaforge-web Certificate, the almaforge headless Service, the almaforge-lb LoadBalancer Service, the almad StatefulSet, and the almad PodDisruptionBudget.

Then apply it:

Terminal
kubectl apply -k <your-overlay>

The first pod to become ready initializes cluster state in etcd: the cluster identity from 00-cluster-name.yaml, the certificate authorities, and the rest of the drop-in resources. No operator session exists yet. That comes in Step 8.

Watch the StatefulSet to ready:

Terminal
kubectl -n almaforge-cluster rollout status statefulset/almad --timeout=5mkubectl -n almaforge-cluster get podsNAME      READY   STATUS    RESTARTSalmad-0   1/1     Running   0almad-1   1/1     Running   0almad-2   1/1     Running   0

After the almaforge-lb Service has an external IP, point the DNS A and AAAA records for almaforge.example.com and *.almaforge.example.com at it.

Step 8. Sign in and verify the cluster​

The cluster is up with the OIDCConnector loaded from the ConfigMap. Sign in as the first administrator and verify the cluster before you wire any resources to it.

From a workstation with alma installed and on a network that can reach the public hostname:

First, make sure the workstation trusts the cluster's CA. The alma binary uses the OS trust store. For an internal CA you used in DNS and TLS, install it once per workstation. For the bundled self-signed eval root, extract it from the almaforge-ca ConfigMap:

Terminal
kubectl -n almaforge-cluster get configmap almaforge-ca -o jsonpath='{.data.ca-bundle\.pem}' > ca.crtsudo cp ca.crt /usr/local/share/ca-certificates/almaforge-ca.crtsudo update-ca-certificates

Equivalent commands exist for Red Hat (update-ca-trust) and macOS (security add-trusted-cert). Without this, alma login fails with "certificate signed by unknown authority". Install the CA before continuing.

Then log in:

Terminal
alma login --proxy=almaforge.example.com:443

A browser opens to the cluster's login page. The corp connector you added in Step 5 is the only choice on a fresh cluster, so it auto-selects. Complete the IdP flow in the browser and AlmaForge maps the returned claim to the admin role per claimsToRoles. On headless workstations pass --no-browser. alma prints a URL you open on any other device.

Terminal
alma clusteralma get oidcconnectoralma get rolealma status

All four should return without auth errors. If they do, the cluster is live: every pod is healthy, the external Service is forwarding, and your first admin can manage the cluster end-to-end.

Connect resources​

The cluster is now ready to host infrastructure access. Add agents for each kind of target:

Each agent is another almad process with the same binary and different service flags. It joins the cluster using credentials from its own Token resource and exposes one or more targets through the cluster Proxy layer.

For agents running in the same Kubernetes cluster as this control plane, the in-cluster Kubernetes join method lets the agent prove its identity with its ServiceAccount JWT through TokenReview. No shared secret or CA pin is required.

Operations​

After the cluster is running, the Operations guide covers the day-2 work:

  • Routine maintenance. Rolling upgrades (kubectl -n almaforge-cluster set image statefulset/almad almad=quay.io/almaforge/almaforge:<tag> then kubectl rollout status), certificate renewal, and scaling. cert-manager renews the TLS Secret, but this overlay mounts the certificate and key using subPath. Roll the StatefulSet after renewal so every pod reads the new files. Three almad replicas provide maintenance headroom. The Auth replica count is independent of the etcd member count, provided the state backend supports it.
  • Observability. Prometheus scrape on :3081 via the podmonitor.yaml template shipped with the bundle (commented out in kustomization.yaml). kubelet probes on :3000. See Monitoring for scrape targets and alert recipes.
  • Incident response. Break-glass access when SSO is down and access incident playbooks.
  • Data lifecycle. etcd snapshots (see the etcd Storage Guide), S3 lifecycle for recordings, audit-table retention, and full Backups.

Access-request approvals integrate with Slack, PagerDuty, and Jira so requests, approvals, and notifications stay in the tools your team already uses.

Troubleshooting​

LoadBalancer Service stuck in <pending>​

Terminal
kubectl -n almaforge-cluster get svc almaforge-lbNAME           TYPE           EXTERNAL-IP   PORT(S)almaforge-lb   LoadBalancer   <pending>     443:30000/TCP

The cluster has no LoadBalancer controller. On managed cloud Kubernetes (EKS, GKE, AKS), the controller is built in. On bare metal or kind/minikube, install MetalLB or the equivalent first. On managed cloud where the controller exists but the LB stays pending, check kubectl describe svc almaforge-lb for events: quota exhaustion, missing subnet tag, or restricted service account often surface there.

almad fails to start with cannot connect to <state-backend>​

The pod cannot reach the state backend on the addresses in 01-state-storage.yaml. Common causes:

  • Wrong endpoint addresses, or the backend is unhealthy.
  • For in-cluster etcd, the etcd namespace is not running yet. Bring up examples/kubernetes/etcd/ first.
  • For external etcd, the pod's client cert is not signed by the backend CA, or the CA Secret mounted into the StatefulSet does not match what the state drop-in references.
  • A NetworkPolicy is blocking egress from almaforge-cluster to the etcd namespace or external endpoint.

For in-cluster etcd, verify from inside a pod:

Terminal
kubectl -n almaforge-cluster exec -it almad-0 -- \    /bin/sh -c 'wget -qO- http://etcd-0.etcd.etcd.svc:2379/health'{"health":"true"}

For external backends, use the vendor's health-check tool against the endpoints in 01-state-storage.yaml.

Sign-in fails with redirect_uri_mismatch​

The IdP's registered redirect URI does not match what almad produces from ALMA_ADVERTISE at runtime. The cluster generates https://<ALMA_ADVERTISE>/api/v1/oidc/callback. Register that exact URL on the IdP side. Trailing slash, port presence, and http vs https must all match. See SSO for the redirect URI registration step.

Listing connectors when locked out​

From a host with kubectl access to the cluster, run the local admin identity inside a pod:

Terminal
kubectl -n almaforge-cluster exec -it almad-0 -- \    alma get oidcconnector

The local admin inside the pod can read cluster resources without an SSO session. Use this to verify the connector exists, then edit the drop-in and roll the StatefulSet. For broader recovery procedures when SSO is down, see Break-glass.

StatefulSet stuck at 0/N ready​

The most common cause is a missing or misconfigured drop-in. Verify the ConfigMap and the pod logs:

Terminal
kubectl -n almaforge-cluster describe configmap auth-configkubectl -n almaforge-cluster logs almad-0

Look for "failed to read config" or "cannot connect to etcd" in the logs. If a file listed under configMapGenerator is missing on disk or carries a typo, the process exits immediately. Confirm every referenced drop-in exists before re-applying.

To test etcd connectivity from inside the cluster:

Terminal
kubectl -n almaforge-cluster exec -it almad-0 -- \    /bin/sh -c 'wget -qO- http://etcd-0.etcd.etcd.svc:2379/health'{"health":"true"}

cert-manager Certificate stuck in Issuing​

Terminal
kubectl -n almaforge-cluster describe certificate almaforge-web

Common causes:

  • The almaforge-web ClusterIssuer is not ready. Inspect it: kubectl describe clusterissuer almaforge-web.
  • The ClusterIssuer's backing CA Secret does not exist in the cert-manager namespace.
  • For DNS-01 challenges, the DNS record has not propagated yet. cert-manager retries automatically. Wait and re-check.

alma fails with certificate signed by unknown authority​

The workstation does not trust the CA that signed the cluster's TLS certificate. Install the cluster CA into the OS trust store. The CA cert is in the almaforge-ca ConfigMap that trust-manager replicates to every namespace:

Terminal
kubectl -n almaforge-cluster get configmap almaforge-ca -o jsonpath='{.data.ca-bundle\.pem}' > ca.crtsudo cp ca.crt /usr/local/share/ca-certificates/almaforge-ca.crtsudo update-ca-certificates

Equivalent commands: update-ca-trust on Red Hat, security add-trusted-cert on macOS.

alma cluster shows a hostname instead of your domain​

The cluster name is the first pod's hostname. You skipped 00-cluster-name.yaml before first apply, so the cluster adopted the first pod's hostname and ClusterName is now immutable. There is no in-place rename. A different cluster name is not itself a TLS endpoint error. If the name must change, plan a new-cluster cutover and preserve backups before decommissioning the existing cluster. Include 00-cluster-name.yaml in the replacement's configMapGenerator before first startup.

The sign-in page shows no connector to pick​

The sign-in page renders one button per active connector, so an empty page means the cluster holds none. List them:

Terminal
alma get oidcconnector

If the connector is missing, the OIDCConnector drop-in was either not in the ConfigMap at startup or was not added to configMapGenerator. Add it and re-apply:

Terminal
kubectl apply -k <your-overlay>

almad re-reads /etc/almaforge/almad.d and applies the connector without a restart after the kubelet updates the mounted ConfigMap. Propagation time depends on the kubelet configuration. Watch for it:

Terminal
kubectl -n almaforge-cluster logs -f statefulset/almad | grep Reloaded

Login succeeds but alma get role returns forbidden​

The assigned roles do not grant the requested read operation. Inspect the current identity before changing the connector:

Terminal
alma status

status shows the resolved roles for the current session. Confirm that claimsToRoles maps the intended IdP claim (often email or groups) to the appropriate administrator role. Sign in again after a mapping change so the new identity contains the updated roles.

References​

  • Declarative Configuration: full schema for StorageConfig, OIDCConnector, Token, Role, and target kinds.
  • Environment Configuration: every supported ALMA_* variable.
  • CLI Reference: alma and almad subcommands and flags.
  • Storage Guide: backend selection, etcd cluster layout, PostgreSQL sizing, DynamoDB IAM, audit and recording retention.
  • Networking Guide: load balancer reference, advertise vs listen, and wildcard DNS.
  • High Availability: the architecture model this guide implements.
  • CLI Setup: installing alma on operator workstations and OS trust-store commands for the cluster CA.
  • Backups: restore drills and capacity planning.
  • Break-glass: recovery procedures when SSO is broken.
  • Upgrades: rolling upgrade flow and release-notes links.
  • Monitoring: Prometheus scrape targets and alert recipes.
  • CA Rotation: cluster CA rotation procedure.
  • Hardening Guide: production-grade filesystem modes, mTLS, secrets, session limits, and Web UI hardening.
  • Support plans: support plans, response times, severity levels, and professional services.