4. Solution Strategy

1. GitOps as the operating model

Every desired state of the cluster is a file in this repository ([CT-004 GitOps single source of truth ]). Argo CD is the reconciliation engine: it watches this repository and applies Application resources with automated: {prune: true, selfHeal: true}, so that

  • a merged commit is applied to the cluster automatically, and

  • manual, undocumented changes on the cluster are reverted on the next reconciliation loop.

This directly serves the [CO-001 Small Team ] constraint: a small team can operate a large set of add-ons because "how do I change X" always has the same answer — edit a file, open a PR, merge.

2. App-of-apps and multi-source pattern

The repository follows a two-level Argo CD app-of-apps pattern:

  1. pandur/argocd/resources/auto-applications.yml is an ApplicationSet that creates one root Application per GitOps repository (this repository plus sibling repositories such as gitops-paladins-inn), each pointed at that repository’s pandur/apps directory.

  2. Each file below pandur/apps/*.yml is itself a parent Application that reconciles one building block’s apps/ directory (e.g. pandur/security/apps), which in turn contains the leaf Application resources that install a single Helm chart or a directory of raw manifests.

Leaf applications that install a third-party Helm chart use Argo CD’s multi-source feature: one source pins the remote chart (repoURL + chart + targetRevision), and a second source with ref: config points at this same Git repository so that helm.valueFiles can reference $config/pandur/<component>/config/<file>.yml. This keeps chart version pinning and organization-specific values cleanly separated while still rendering a single Helm release.

Raw Kubernetes resources that do not come from a Helm chart (secrets, cluster issuers, storage classes, RBAC bindings, …​) live in a component’s resources/ directory and are reconciled by their own small Argo CD Application (e.g. security-resources, observability-resources).

3. Infrastructure vs. workload GitOps boundary

Cluster creation itself (Hetzner network, NAT gateway, control-plane and worker nodes) is not managed by Argo CD — see 7. Deployment View. It is a documented, mostly one-off bootstrap using Cluster API/CAPH and OpenTofu. Only once the nodes exist and a workload kubeconfig is available does GitOps take over: Argo CD, and everything below pandur/, is installed onto the already-running cluster.

This separation keeps "did my node come up" (infrastructure) and "did my add-on reconcile" (GitOps) as two independent failure domains, each with its own tooling.

4. Single Sign On

The cluster holds no local user database. Every login — to the Kubernetes API itself, to Argo CD, and to observability tooling (Grafana, Headlamp) — is delegated via OpenID Connect to the organization’s Keycloak realm k8s ([CT-005 OpenID Connect only ]).

  • The Kubernetes API server is configured with --oidc-issuer-url, --oidc-client-id=k8s-pandur, --oidc-username-prefix=oidc: and --oidc-groups-prefix=oidc:, so that kubectl users authenticate via kubectl oidc-login and are mapped to Kubernetes RBAC purely through Keycloak group membership (e.g. ClusterRoleBinding on group oidc:cluster-admin).

  • Argo CD and Grafana are each configured with their own confidential OIDC client against the same Keycloak realm.

  • Headlamp reuses the public, PKCE-based k8s-pandur client so that a user’s Headlamp session enforces exactly the same RBAC as a kubectl session — no separate authorization mapping is required.

See Single Sign-On for details.

5. Data protection and backup

Stateful workloads run on PostgreSQL clusters managed by CloudNativePG. The Barman Cloud Plugin performs continuous WAL archiving and periodic base backups to S3-compatible object storage, so that any CNPG-managed cluster (including Keycloak’s own database) can be restored after a lost volume or a lost node. Persistent data that is not database-shaped (e.g. long-term file storage) is provisioned via csi-driver-smb onto Hetzner Robot/hidrive CIFS shares, which are independent of the cluster’s own block storage.

See Data Protection for details.

6. High availability

Nodes are spread so that control-plane and worker failures are tolerated ([RE01 24/7 Availability] in 10. Quality Requirements): the KubeadmControlPlane runs multiple replicas behind the CAPH-managed control-plane load balancer, MachineHealthCheck plus HCloudRemediationTemplate replace unhealthy machines automatically, and components that support it (Keycloak, Traefik) are run with more than one replica.