> ## Documentation Index
> Fetch the complete documentation index at: https://docs.4minds.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS Marketplace: Helm Deployment

> Deploy the 4MINDS AI Platform to your own Kubernetes cluster with a single Helm chart, delivered via AWS Marketplace.

Deploy the 4MINDS platform to your own Kubernetes cluster with a single
`helm install`. The umbrella chart carries every service and datastore, and its
own hooks do all in-cluster orchestration (OpenBao init/unseal, DB migrations,
Kafka topics), so no external installer binary is required.

You configure the deployment through one values file: copy the
[values template](#values-template) at the bottom of this page to a file of your
choice (this guide calls it `my-values.yaml`, but the name is arbitrary) and
fill in your hostname, inference endpoints, and options.

<Note>
  The prerequisite commands below use Amazon EKS as the worked example
  (`eksctl`, EBS CSI, IRSA). On another Kubernetes distribution, substitute the
  equivalent steps: a default StorageClass, a CSI driver, an ingress controller,
  and (for KMS auto-unseal) a cloud KMS key + workload identity.
</Note>

## What you'll do

1. **Prerequisites (Steps 0-7):** confirm your IAM permissions (Step 0), then
   prepare the cluster and platform plumbing the chart does *not* create
   (cluster/OIDC, storage, ingress, namespace, TLS, image pull, and, only for
   production KMS unseal, a KMS key + IAM role).
2. **Configure:** copy the [values template](#values-template) from this page into your own values file and fill it in.
3. **Install:** log in to the Marketplace ECR, pull the chart, and `helm install` with your values.
4. **Verify:** confirm pods are Running and the UI answers.

Sections after that cover upgrades, uninstall, and troubleshooting.

## Before you start

**Required CLI tools.** Install any that are missing before you begin (each link
points to its official install guide):

* [`aws`](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html): the AWS CLI. Authenticate it to the account that owns the cluster.
* [`kubectl`](https://kubernetes.io/docs/tasks/tools/): the Kubernetes CLI.
* [`helm`](https://helm.sh/docs/intro/install/): v3.8+ (OCI registry support required).
* [`eksctl`](https://eksctl.io/installation/): only for the Amazon EKS example steps.
* [`openssl`](https://www.openssl.org/source/): only if you want a self-signed test cert.

Verify each is on your `PATH` with `<tool> version` (e.g. `aws --version`,
`helm version`) before continuing.

**Set these environment variables.** Every command below references them:

```bash theme={null}
export CLUSTER=<your-cluster>          # e.g. 4minds-prod
export REGION=<your-region>            # e.g. us-east-1
export HOSTNAME_FQDN=<your-host>       # e.g. 4minds.your-company.com
export NAMESPACE=4minds
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
```

Then confirm none are empty:

```bash theme={null}
echo "$CLUSTER $REGION $HOSTNAME_FQDN $NAMESPACE $ACCOUNT"
```

<Tip>
  **Copy-paste tip.** If a multi-line command (with a trailing `\`) breaks in
  your shell, paste it as a single line instead.
</Tip>

## Prerequisites

### Step 0: IAM permissions for the deployer

**What this does:** confirms the IAM identity (user or role) you run these steps
as has every permission the prerequisites need, so you never hit an
`AccessDenied` mid-deployment. Sort this out before anything else.

Across the steps below you (or `eksctl` on your behalf) create an EKS cluster,
CloudFormation stacks, IAM roles/policies, an OIDC provider, EC2/networking
resources, an EBS CSI addon, a KMS key, and you authenticate to the Marketplace
ECR. That is a broad, privileged set of actions.

**Simplest path (recommended for a one-time deploy):** run as an identity with
the AWS-managed policies below attached. This is the least-friction option and is
what most first deployments use:

| Managed policy                       | Covers                                                   |
| ------------------------------------ | -------------------------------------------------------- |
| `AmazonEKSClusterPolicy`             | EKS cluster operations                                   |
| `AmazonEC2FullAccess`                | VPC/subnets/security groups/instances for the node group |
| `AWSCloudFormationFullAccess`        | the stacks `eksctl` creates for the cluster and IRSA     |
| `IAMFullAccess`                      | create the OIDC provider, IRSA roles, and the KMS policy |
| `AWSKeyManagementServicePowerUser`   | create the KMS key + alias (KMS unseal, Step 7 only)     |
| `AmazonEC2ContainerRegistryReadOnly` | `helm registry login` + pull from the Marketplace ECR    |

<Note>
  `eksctl` drives everything through CloudFormation, which in turn creates IAM
  roles. There is no smaller managed policy that covers cluster creation
  end-to-end. This is why broad IAM/CloudFormation access is needed for the
  initial provisioning.
</Note>

**Least-privilege path (locked-down orgs):** if org policy forbids the broad
managed policies, attach a custom policy granting exactly these actions. Scope
`Resource` down to your account/cluster where you can:

| Service actions                                                                                                                               | Used by                                            |
| --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| `sts:GetCallerIdentity`                                                                                                                       | reading `$ACCOUNT` (Before you start)              |
| `eks:CreateCluster`, `eks:DescribeCluster`, `eks:CreateAddon`, `eks:DescribeAddon`, `eks:CreateNodegroup`, `eks:*`                            | Steps 1-3 (cluster, OIDC, EBS addon)               |
| `cloudformation:CreateStack`, `cloudformation:DescribeStacks`, `cloudformation:*`                                                             | every `eksctl` command                             |
| `ec2:*` (or scoped VPC/subnet/SG/instance/EIP actions)                                                                                        | Step 1 node group + networking                     |
| `iam:CreateOpenIDConnectProvider`, `iam:CreateRole`, `iam:CreatePolicy`, `iam:AttachRolePolicy`, `iam:PassRole`, `iam:GetRole`, `iam:TagRole` | Steps 1, 2, 7 (OIDC, IRSA roles, KMS policy)       |
| `kms:CreateKey`, `kms:CreateAlias`                                                                                                            | Step 7 (KMS auto-unseal; skip if `seal.mode: lab`) |
| `ecr:GetAuthorizationToken`, `ecr:BatchGetImage`, `ecr:GetDownloadUrlForLayer`, `ecr:BatchCheckLayerAvailability`                             | Install (Marketplace ECR login + `helm pull`)      |
| `autoscaling:*`                                                                                                                               | node group scaling resources `eksctl` provisions   |

<Note>
  **Marketplace subscription is separate from IAM.** Before pulling the chart,
  the account must be subscribed to the 4MINDS product on AWS Marketplace.
  Managing that subscription needs `aws-marketplace:Subscribe` /
  `aws-marketplace:ViewSubscriptions` (or Marketplace console access) — an
  account/billing permission, not something the deploy identity uses at runtime.
</Note>

<Note>
  **`kubectl` / `helm` permissions are Kubernetes RBAC, not IAM.** As the cluster
  creator you are automatically `system:masters` (cluster-admin), so every
  `kubectl` and `helm` step in this guide just works. If someone *else* deploys
  into a cluster they did not create, grant them cluster-admin (or map their IAM
  identity in the `aws-auth` configmap / an EKS access entry).
</Note>

### Step 1: EKS cluster with OIDC

**What this does:** creates (or reuses) the Kubernetes cluster. `--with-oidc` is
the only hard requirement; IRSA (used for KMS auto-unseal and the EBS driver)
needs it.

Skip the `create cluster` command if you already have a cluster; use your own
version / instance type / node count.

```bash theme={null}
export K8S_VERSION=<your-k8s-version>      # e.g. 1.31
export NODE_TYPE=<your-instance-type>      # e.g. m5.2xlarge
export NODE_COUNT=<your-node-count>        # e.g. 3

eksctl create cluster --name "$CLUSTER" --region "$REGION" \
  --version "$K8S_VERSION" --node-type "$NODE_TYPE" --nodes "$NODE_COUNT" --with-oidc

aws eks update-kubeconfig --name "$CLUSTER" --region "$REGION"
kubectl get nodes                          # all should be Ready
```

**If the cluster already exists,** just make sure the OIDC provider is
associated:

```bash theme={null}
eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER" --region "$REGION" --approve
```

<Note>
  **Sizing (guidance, not a requirement).** The full platform runs many services
  plus stateful datastores. A reasonable starting point for the application tier
  is **3 × m5.2xlarge** (8 vCPU / 32 GiB each); scale to your workload.

  **Inference is separate.** This chart does NOT run models; it points at
  OpenAI-compatible endpoints you set in `my-values.yaml` (`llm.*`,
  `mlai.embedding`, `symi-gateway.config`, `wren-ai`). Those can be a managed
  service or GPU nodes in this same cluster (add a `g5`/`p4` node group and point
  the endpoints at the in-cluster services).
</Note>

**Models and GPU sizing.** The endpoints above serve the models the platform
uses. A typical deployment runs these:

| Model                         | Role                                                                 | Configured via                        |
| ----------------------------- | -------------------------------------------------------------------- | ------------------------------------- |
| GPT-OSS-120B                  | General LLM (chat history, summarization, wren-ai text-to-SQL, mlai) | `llm.*`, `wren-ai.config.llmEndpoint` |
| Qwen 3.6 35B FP8              | SYMI assistant                                                       | `symi-gateway.config.llm`             |
| BAAI/bge-m3                   | Embeddings                                                           | `mlai.embedding`                      |
| MS-MARCO cross-encoder        | Reranker                                                             | `llm.crossEncoderEndpoint`            |
| numind/NuMarkdown-8B-Thinking | Vision OCR                                                           | `mlai.visionOcr`                      |
| Qwen/Qwen2.5-VL-32B-Instruct  | Vision-language (VLM)                                                | `mlai.vlm`                            |

GPU memory is dominated by the two largest models: GPT-OSS-120B (a sparse
MoE model, \~63 GB of weights) and Qwen 3.6 35B FP8. The rest are small by
comparison. To serve \~100 concurrent users, plan for 2 × NVIDIA RTX PRO
6000 (96 GB each) for inference without fine-tuning. If fine-tuning workloads
must run alongside inference, the requirement increases to 3 × 96 GB RTX PRO
6000\.

### Step 2: Default StorageClass + EBS CSI driver

**What this does:** gives stateful services (PostgreSQL, Redis, Kafka, MinIO,
Qdrant, OpenBao, mlai) a default StorageClass backed by a working CSI driver.
OpenBao persists its vault data on a 2Gi PVC, so this is required even for
`seal.mode: lab`: without persistence, a pod restart re-initializes the vault
and drops the seeded secrets.

```bash theme={null}
# 2a. Mark gp2 (or gp3) as the default StorageClass.
kubectl annotate storageclass gp2 \
  storageclass.kubernetes.io/is-default-class=true --overwrite
kubectl get storageclass                   # exactly one shows "(default)"

# 2b. Install the EBS CSI driver (IRSA role + managed addon).
eksctl create iamserviceaccount --name ebs-csi-controller-sa --namespace kube-system \
  --cluster "$CLUSTER" --region "$REGION" --role-name "${CLUSTER}-ebs-csi" \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve --role-only
eksctl create addon --name aws-ebs-csi-driver --cluster "$CLUSTER" --region "$REGION" \
  --service-account-role-arn "arn:aws:iam::${ACCOUNT}:role/${CLUSTER}-ebs-csi" --force
kubectl -n kube-system rollout status deploy/ebs-csi-controller --timeout=180s
```

### Step 3: ingress-nginx controller

**What this does:** installs the NGINX ingress controller. The chart creates an
`Ingress` of class `nginx`; the controller itself is a prerequisite.

```bash theme={null}
# 3a. Install the controller (as a LoadBalancer).
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.service.type=LoadBalancer
kubectl -n ingress-nginx rollout status deploy/ingress-nginx-controller --timeout=180s

# 3b. Read the load-balancer hostname, then point your DNS record for
#     $HOSTNAME_FQDN at it (CNAME). For a quick local test you can instead map
#     the LB's IP to $HOSTNAME_FQDN in /etc/hosts.
kubectl -n ingress-nginx get svc ingress-nginx-controller \
  -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'; echo
```

### Step 4: Namespace

**What this does:** creates the namespace everything installs into.

```bash theme={null}
kubectl create namespace "$NAMESPACE"
```

### Step 5: TLS secret for your hostname

**What this does:** provides the certificate the Ingress uses to terminate TLS.
The Ingress reads the secret named by `frontend-backend.ingress.tlsSecretName`
(default `frontend-tls`). The host and TLS entry derive from `global.hostname`
automatically, so you only create the secret here.

**Option A: you already have a cert** (ACM-issued, Let's Encrypt, etc.):

```bash theme={null}
kubectl -n "$NAMESPACE" create secret tls frontend-tls \
  --cert=/path/to/tls.crt --key=/path/to/tls.key
```

**Option B: self-signed** (testing/PoC only; browsers will warn):

```bash theme={null}
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt \
  -subj "/CN=$HOSTNAME_FQDN" -addext "subjectAltName=DNS:$HOSTNAME_FQDN"
kubectl -n "$NAMESPACE" create secret tls frontend-tls --cert=tls.crt --key=tls.key
```

<Warning>
  **Trust the self-signed cert on every machine that opens the UI, otherwise the
  app loads but is unusable.**

  The 4MINDS frontend opens a WebSocket (`wss://$HOSTNAME_FQDN`) to the
  backend. Unlike a normal page load, most browsers will not carry a page-level
  "proceed anyway" exception over to a WebSocket: the `wss://` TLS handshake to an
  untrusted cert simply fails and the connection is dropped, so the UI renders but
  stays disconnected (live updates, chat, and streaming never arrive).

  Clicking **Advanced → Proceed** on the page warning is not reliable across
  browsers (Safari, in particular, does not reuse that exception for `wss://`).
  The only dependable fix is to add the cert to the operating system's trust store
  on each client machine, then fully restart the browser.
</Warning>

Add `tls.crt` to the OS trust store on each client machine (restart the browser
afterwards):

**macOS**: add to the System keychain and mark it trusted for SSL:

```bash theme={null}
# imports into the System keychain and marks it always-trusted for SSL
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain tls.crt
```

Or via **Keychain Access.app**: drag `tls.crt` into the **System** keychain,
double-click it, expand **Trust**, and set **When using this certificate: Always
Trust**.

**Linux** (Debian/Ubuntu):

```bash theme={null}
sudo cp tls.crt /usr/local/share/ca-certificates/4minds.crt
sudo update-ca-certificates
```

On RHEL/Fedora:

```bash theme={null}
sudo cp tls.crt /etc/pki/ca-trust/source/anchors/4minds.crt && sudo update-ca-trust
```

**Windows** (PowerShell as Administrator):

```powershell theme={null}
Import-Certificate -FilePath tls.crt -CertStoreLocation Cert:\LocalMachine\Root
```

<Note>
  Firefox keeps its own trust store. Even after the OS import, add the cert
  under **Settings → Privacy & Security → View Certificates → Authorities →
  Import**.
</Note>

<Tip>
  For anything beyond a quick PoC, use a real/trusted certificate (Option A: ACM
  or Let's Encrypt). With a trusted cert, no client-side trust step is needed at
  all and `wss://` works out of the box.
</Tip>

<Note>
  Using a different secret name? Set `frontend-backend.ingress.tlsSecretName`
  in `my-values.yaml` to match.
</Note>

### Step 6: Image pull (usually nothing to do)

**What this does:** lets the cluster pull the images. On AWS Marketplace the
images live in the Marketplace ECR and your EKS node IAM role pulls them
automatically (attach `AmazonEC2ContainerRegistryReadOnly` to the node role if
it isn't already). In that case leave `global.imagePullSecrets: []`.

Only if you mirror the images into your own private registry do you create a
pull secret and list its name under `global.imagePullSecrets`.

### Step 7: OpenBao KMS auto-unseal (production only)

**What this does:** sets up AWS KMS + an IRSA role so OpenBao auto-unseals
without in-cluster keys. Skip this entire step if you use `seal.mode: lab`
(Shamir keys stored in-cluster, fine for test/PoC).

```bash theme={null}
# 7a. Create the KMS key + alias.
KEY_ARN=$(aws kms create-key --description "4minds-openbao-${CLUSTER}" \
  --region "$REGION" --query KeyMetadata.Arn --output text)
aws kms create-alias --alias-name "alias/4minds-openbao-${CLUSTER}" \
  --target-key-id "$KEY_ARN" --region "$REGION"

# 7b. Create a least-privilege IAM policy scoped to just that key.
aws iam create-policy --policy-name "4minds-openbao-kms-${CLUSTER}" \
  --policy-document "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"kms:Encrypt\",\"kms:Decrypt\",\"kms:DescribeKey\",\"kms:GenerateDataKey\"],\"Resource\":\"${KEY_ARN}\"}]}"

# 7c. Create the IRSA ROLE ONLY (--role-only). The chart creates and owns the
#     `platform-openbao` ServiceAccount, so do NOT let eksctl create one too.
eksctl create iamserviceaccount \
  --name platform-openbao --namespace "$NAMESPACE" \
  --cluster "$CLUSTER" --region "$REGION" \
  --role-name "4minds-openbao-irsa-${CLUSTER}" \
  --attach-policy-arn "arn:aws:iam::${ACCOUNT}:policy/4minds-openbao-kms-${CLUSTER}" \
  --role-only --approve
```

Then set these in `my-values.yaml`. The chart binds `roleArn` onto the OpenBao
ServiceAccount as `eks.amazonaws.com/role-arn` automatically (no manual
annotation, survives `helm upgrade`):

```yaml theme={null}
openbao:
  seal:
    mode: kms
    kms:
      provider: aws
      region: <your-region>
      keyId: alias/4minds-openbao-<your-cluster>
      roleArn: arn:aws:iam::<account>:role/4minds-openbao-irsa-<your-cluster>
```

<Note>
  **Azure / GCP:** set `provider: azure` (`tenantId`/`vaultName`/`keyName`) or
  `provider: gcp` (`projectId`/`locationId`/`keyRing`/`cryptoKey`) and bind the
  pod identity via `openbao.serviceAccount.annotations` (Workload Identity).
</Note>

### Prerequisites summary

| # | Prerequisite                     | Why                                                                 |
| - | -------------------------------- | ------------------------------------------------------------------- |
| 0 | IAM permissions for the deployer | create the cluster/roles/KMS + pull from ECR without `AccessDenied` |
| 1 | EKS cluster + OIDC               | run the platform; OIDC needed for IRSA                              |
| 2 | Default StorageClass + EBS CSI   | stateful services (PVCs)                                            |
| 3 | ingress-nginx controller         | serves the Ingress the chart creates                                |
| 4 | Namespace `4minds`               | where everything installs                                           |
| 5 | `frontend-tls` secret            | TLS termination for your hostname                                   |
| 6 | Node IAM ECR pull                | pull the images (usually automatic on Marketplace)                  |
| 7 | KMS key + IRSA role              | OpenBao auto-unseal (only for `seal.mode: kms`)                     |

## Configure

**What this does:** creates your deployment's values file. Copy the
[values template](#values-template) from the bottom of this page into a file
(this guide calls it `my-values.yaml`, but the name is arbitrary) and fill in the
required fields: hostname, email, LLM/embedding endpoints, SYMI, wren-ai, seal
mode, and any SSO/integrations you use. Everything else has a sensible default.

You don't need the chart on disk yet; you pull it in the [Install](#install)
step. Just save your filled-in values file somewhere and pass its path to
`helm install` with `-f`.

<Warning>
  Use the [values template on this page](#values-template), **not** the
  `values-customer-template.yaml` bundled inside the pulled chart. The bundled file
  is out of date; the template on this page is the current, supported one.
</Warning>

### (Optional) External S3 instead of the bundled MinIO

By default object storage is the in-cluster MinIO and you set nothing. To use
AWS S3 (or any S3-compatible endpoint) instead, add to `my-values.yaml`:

```yaml theme={null}
storage:
  endpoint: "https://s3.us-east-1.amazonaws.com"   # empty = bundled MinIO
  region: us-east-1
  bucketName: 4minds-uploads
secrets:                    # keep ONE secrets: block; merge with OAuth secrets
  s3AccessKey: "<access-key>"
  s3SecretKey: "<secret-key>"
```

Leave `storage.endpoint` / `secrets.s3*` empty to keep the bundled MinIO. If you
also set OAuth client secrets, put the S3 keys under the same `secrets:`
block; a second `secrets:` key silently overrides the first (YAML has no merge).
For Azure Blob, set `storage.useAzure: "true"` and `secrets.azureStorageAccountKey`
in that same block.

## Install

**What this does:** pulls the chart from the AWS Marketplace ECR and deploys the
platform with your values. The chart's hooks then run automatically (OpenBao
init+unseal, backend-secret seeding, Kafka topics, the mlai schema migration).
No manual steps.

**1. Log in to the Marketplace ECR.** After subscribing to the product on AWS
Marketplace, authenticate Helm to the Marketplace registry:

```bash theme={null}
aws ecr get-login-password --region "$REGION" \
  | helm registry login --username AWS --password-stdin \
    709825985650.dkr.ecr.us-east-1.amazonaws.com
```

<Note>
  The Marketplace ECR host is fixed at `709825985650.dkr.ecr.us-east-1.amazonaws.com`
  regardless of `$REGION` — always use `us-east-1` in that URL. `--region "$REGION"`
  above only tells the AWS CLI where to fetch the auth token.
</Note>

**2. Pull and unpack the chart.** Download the chart into an empty directory and
extract it:

```bash theme={null}
mkdir awsmp-chart && cd awsmp-chart
helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/4minds-ai/4minds-chart --version 3.3.0
tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete   # extract, then remove the .tgz
```

**3. Install with your values.** Point `-f` at the `my-values.yaml` you filled in
during [Configure](#configure) (use its full path if it lives outside this
directory). The namespace was already created in [Step 4](#step-4-namespace), so
this uses `--namespace` (not `--create-namespace`):

```bash theme={null}
helm install 4minds ./* \
  --namespace "$NAMESPACE" \
  --set global.awsmpServiceAccountName=backend-service \
  -f /path/to/my-values.yaml
kubectl -n "$NAMESPACE" get pods -w        # watch until all are Running/Completed
```

<Note>
  The `--set global.awsmpServiceAccountName=backend-service` flag is required for
  AWS Marketplace metered billing. Keep it **and** pass `-f my-values.yaml`;
  without your values file the platform comes up unconfigured (no hostname, no
  inference endpoints).
</Note>

## Verify

**What this does:** confirms the platform is up and serving.

```bash theme={null}
kubectl -n "$NAMESPACE" get pods                       # all 1/1 Running
kubectl -n "$NAMESPACE" get ingress 4minds-ingress     # ADDRESS = your LB
curl -k https://$HOSTNAME_FQDN/                        # returns the 4MINDS frontend
```

Then open `https://$HOSTNAME_FQDN` and sign up (email + password works out of the
box; SSO only if you configured a provider).

## Upgrades

To upgrade, pull the newer chart version the same way as in [Install](#install)
(log in to the ECR, `helm pull` with the new `--version`, unpack), then run the
upgrade from the unpacked chart directory with the same flags:

```bash theme={null}
helm upgrade 4minds ./* \
  --namespace "$NAMESPACE" \
  --set global.awsmpServiceAccountName=backend-service \
  -f /path/to/my-values.yaml
```

Hooks are idempotent (OpenBao "already initialized", mlai migration re-verify).
Auto-generated secrets (PG/Redis/MinIO/Fernet/Tier/SYMI) are preserved across
upgrades via `helm.sh/resource-policy: keep`.

## Uninstall

```bash theme={null}
helm uninstall 4minds -n "$NAMESPACE"

# PVCs and the generated-secrets Secret are retained by design. To wipe fully:
kubectl -n "$NAMESPACE" delete pvc --all
# platform-openbao-keys only exists in seal.mode: lab (Shamir); in kms mode there
# are no unseal keys to delete, so --ignore-not-found keeps this harmless.
kubectl -n "$NAMESPACE" delete secret 4minds-generated-secrets platform-openbao-keys --ignore-not-found
```

<Warning>
  Deleting PVCs destroys all data (Postgres, Qdrant, MinIO, etc.).
</Warning>

## Troubleshooting

| Symptom                                   | Likely cause / fix                                                                                            |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| Pods `Pending` on PVC                     | No default StorageClass / EBS CSI not ready (Step 2)                                                          |
| `ImagePullBackOff`                        | Node IAM lacks ECR pull, or wrong `imagePullSecrets` (Step 6)                                                 |
| Ingress has no ADDRESS                    | ingress-nginx controller not installed (Step 3)                                                               |
| App pods `CrashLoopBackOff` early on      | OpenBao not unsealed yet; `kubectl logs job/4minds-openbao-bootstrap`                                         |
| SYMI: "No API key for provider …"         | `symi-gateway.config.llm.baseUrl` + `llmModel` not set → set your SYMI LLM endpoint                           |
| SYMI UI: "Backend service is unreachable" | leave `symi-gateway.config.apiKey` empty so it shares the backend's generated key                             |
| Dataset stuck "processing" until refresh  | check `kubectl logs deploy/postgres-kafka-bridge` (the `wait-for-tables` initContainer gates trigger install) |

## Values template

This is the current, supported values template; use this one, not the
`values-customer-template.yaml` inside the pulled chart (that bundled copy is
out of date). Copy it into your own values file, fill in the required fields
(hostname, email, inference endpoints), and pass it to `helm install` with `-f`.

```yaml my-values.yaml theme={null}
# =============================================================================
# 4MINDS Platform: Customer Values (helm-native install template)
#
#   helm install 4minds ./* --namespace 4minds \
#     --set global.awsmpServiceAccountName=backend-service \
#     -f /path/to/my-values.yaml     (run from the unpacked chart dir)
#
# Fill in the fields below for YOUR deployment. Everything not listed here has a
# sensible default in the chart; you only override what is specific to you.
#
# LEGEND:  [REQUIRED] must set   ·   [OPTIONAL] set only if you use that feature
#
# Cluster prerequisites (create BEFORE helm install; the chart does not).
# See INSTALL-GUIDE.md "Prerequisites checklist" (steps 1-7) for the commands:
#   • EKS cluster with OIDC (needed for IRSA)
#   • A default StorageClass + EBS CSI driver (EKS gp2/gp3, etc.)
#   • ingress-nginx controller installed
#   • Namespace  4minds  (kubectl create namespace 4minds)
#   • Secret  frontend-tls  (kubernetes.io/tls) for your hostname
#   • Image pull: on AWS Marketplace the EKS node IAM role pulls from the
#     Marketplace ECR, no pull secret needed. Only create one if you mirror
#     images into your own private registry (see global.imagePullSecrets).
#   • (kms seal only) A KMS key + IRSA role; see INSTALL-GUIDE.md prereq 7.
# =============================================================================

global:
  # [REQUIRED] The public DNS name your users reach the platform on.
  # EVERYTHING derives from this: backend API URL, frontend, websocket (wss),
  # symi, CORS origins, cookie domain, and ALL OAuth redirect URIs.
  # Set this correctly and you rarely need to touch any other URL below.
  hostname: "4minds.your-company.com"

  namespace: 4minds

  # Image pull auth:
  # On AWS Marketplace the images live in the Marketplace ECR and your EKS
  # NODES pull them via their IAM role (AmazonEC2ContainerRegistryReadOnly),
  # so NO imagePullSecrets needed. Leave this empty (the chart omits the field).
  #
  # ONLY set this if you mirror the images into your OWN private registry that
  # needs a docker-registry pull secret; then create the secret first and list
  # its name here:
  #   kubectl -n 4minds create secret docker-registry my-pull-secret ...
  imagePullSecrets: []
  # imagePullSecrets:
  #   - name: my-pull-secret

  security:
    allowInsecureImages: true

# -----------------------------------------------------------------------------
# Ingress + TLS  (nginx). The host and the TLS block are derived automatically
# from global.hostname above; you do NOT repeat the hostname here. You only
# provide the TLS secret: create `frontend-tls` (kubernetes.io/tls) for your
# hostname before installing (see INSTALL-GUIDE.md, prereq 5).
#
# Advanced: to serve extra hosts or use a different secret per host, set an
# explicit `tls:` list here; it overrides the auto-derived single-host block.
# -----------------------------------------------------------------------------
frontend-backend:
  ingress:
    enabled: true
    className: nginx
    tlsSecretName: frontend-tls    # host + TLS auto-derive from global.hostname

# -----------------------------------------------------------------------------
# OpenBao seal mode: configure it HERE, under `openbao.seal`.
# (The install/bootstrap reads ONLY `openbao.seal.mode`; there is no separate
#  top-level `seal:` to keep in sync.)
#   lab  = Shamir keys, auto-generated & stored in-cluster (simplest)
#   kms  = cloud KMS auto-unseal (recommended on AWS/GCP/Azure via IRSA/WI)
# -----------------------------------------------------------------------------
openbao:
  seal:
    mode: lab                  # lab | kms
    # [OPTIONAL] Fill only when mode: kms. Set the block for YOUR cloud; the
    # chart renders the matching seal stanza and (AWS) wires IRSA for you.
    # kms:
    #   provider: aws          # aws | azure | gcp
    #   # --- AWS: KMS key + IRSA role (recommended on EKS) ---
    #   region: us-east-1
    #   keyId: alias/4minds-openbao
    #   roleArn: arn:aws:iam::<acct>:role/4minds-openbao-irsa   # chart annotates
    #                                                           # the OpenBao SA
    #   # --- Azure Key Vault: tenantId + vaultName + keyName ---
    #   # --- GCP KMS: projectId + locationId + keyRing + cryptoKey ---
  # OpenBao data is persisted to a 2Gi PVC by default (needs a default
  # StorageClass, INSTALL-GUIDE prereq 2). Override only if needed:
  # storage:
  #   size: 5Gi
  #   storageClass: gp3

# -----------------------------------------------------------------------------
# [REQUIRED] Email: transactional email (signup verification, invites, reset).
#
# Choose the provider that fits YOUR company; all three are first-class:
#   provider: smtp      → any standard SMTP server (Gmail, O365, SES-SMTP, ...)
#   provider: sendgrid  → SendGrid API
#   provider: azure     → Azure Communication Services
#
# 1) set `provider` to your choice
# 2) fill the SAME-named block below (leave the others empty/removed)
# 3) set senderEmail / senderName
#
# If you do NOT want email verification at all (e.g. internal PoC), set
# disableVerification: "true" and you can leave the provider blocks empty.
# -----------------------------------------------------------------------------
email:
  provider: "smtp"                       # smtp | sendgrid | azure
  senderEmail: "noreply@your-company.com"
  senderName: "Your Company"
  disableVerification: "false"           # "true" to skip signup email verification

  # --- Option A: SMTP (standard mail server) ---
  smtp:
    host: "smtp.your-company.com"
    port: "587"
    user: "noreply@your-company.com"
    password: ""
    useTls: "true"

  # --- Option B: SendGrid ---
  sendgrid:
    apiKey: ""

  # --- Option C: Azure Communication Services ---
  azure:
    connectionString: ""

# -----------------------------------------------------------------------------
# [REQUIRED] Embedding endpoint (your BGE-M3 / embedding server). mlai uses it.
# -----------------------------------------------------------------------------
mlai:
  sslVerifyCertificates: "true"   # set "false" only for self-signed inference endpoints
  embedding:
    endpointUrl: "https://embeddings.your-company.com"
    modelName: "BAAI/bge-m3"
    dimension: "1024"
  # [OPTIONAL] Vision OCR / Vision-Language / image-gen endpoints
  visionOcr:
    enabled: "true"
    endpoint: "https://vlm.your-company.com/v1/chat/completions"
  vlm:
    enabled: "true"
    endpoint: "https://vlm.your-company.com/v1/chat/completions"
  flux:
    apiUrl: ""                    # leave empty to disable image generation

# -----------------------------------------------------------------------------
# [REQUIRED] LLM endpoints (your inference stack: vLLM / any OpenAI-compatible)
# -----------------------------------------------------------------------------
llm:
  historyEndpoint:       "https://llm.your-company.com/v1/chat/completions"
  crossEncoderEndpoint:  "https://reranker.your-company.com/rerank"
  summarizationEndpoint: "https://llm.your-company.com/v1/completions"
  models:
    # Map model keys to your served endpoints. gpt_oss_120b is used broadly
    # (mlai watchdog, finetuning registry, wren-ai text-to-SQL), so set it.
    # Unused ones can be left "".
    gpt_oss_120b:    "https://llm.your-company.com/v1"
    gpt_oss_20b:     ""
    qwen25_72b:      ""
    qwen3_30b:       ""
    qwen3_14b:       ""
    gemma3_12b:      ""
    nemotron_3_nano: ""
    llama4_scout:    ""
    minimax_m2:      ""

# -----------------------------------------------------------------------------
# [REQUIRED] Symi gateway LLM (the assistant). Works with ANY OpenAI-compatible
# endpoint (vLLM, your own gateway, etc.).
#   baseUrl     - your OpenAI-compatible chat endpoint (REQUIRED for Symi)
#   llmModel    - the model id served there
#   llmProvider - a free-form internal label (used as "<provider>/<model>");
#                 leave it or set to anything, e.g. "openai", "vllm". Not a vendor.
# apiKey: leave empty → Symi shares the backend's auto-generated SYMI_API_KEY.
# -----------------------------------------------------------------------------
symi-gateway:
  config:
    apiKey: ""
    llm:
      baseUrl:  "https://llm.your-company.com/v1"
      llmModel: "your-model"
      llmProvider: "internal"

# -----------------------------------------------------------------------------
# [REQUIRED] Wren-AI (natural-language → SQL for structured/analytics queries).
# It calls an OpenAI-compatible chat endpoint to generate SQL. Point llmEndpoint
# at your LLM (the OpenAI-compatible BASE url; code appends /chat/completions;
# same endpoint as llm.models.gpt_oss_120b). Empty → text-to-SQL returns 503.
# -----------------------------------------------------------------------------
wren-ai:
  config:
    llmEndpoint: "https://llm.your-company.com/v1"

# -----------------------------------------------------------------------------
# [OPTIONAL] SSO + connector integrations. ALL optional; enable ONLY the ones
# you use. None is required: users can always sign up / sign in with
# email + password.
#
# To enable a provider, UNCOMMENT the `oauth:` block below AND the specific
# provider line(s) you need. Put clientId here; the clientSecret goes in the
# `secrets:` block. redirectUri auto-derives from global.hostname (leave it
# out unless you need a custom one). Register the derived URI in each provider:
#   sign-in:      https://<hostname>/auth/<provider>/callback
#   integrations: https://<hostname>/api/v1/integrations/<provider>/oauth/callback
#
# IMPORTANT: if you uncomment `oauth:`, you MUST uncomment at least one child
# line under it (a bare `oauth:` with no children is invalid YAML/null). If you
# use NO SSO/integrations, leave this whole block commented; the chart's
# defaults apply and everything works.
# -----------------------------------------------------------------------------
# oauth:
#   # --- Sign-in / SSO providers ---
#   microsoft: { clientId: "" }
#   google:    { clientId: "" }
#   github:    { clientId: "" }
#   okta:      { clientId: "" }
#   # --- Data / connector integrations ---
#   sharepoint:  { clientId: "", scopes: "", tenant: "" }
#   office365:   { clientId: "", scopes: "", tenant: "" }
#   googleDrive: { clientId: "" }
#   gsuite:      { clientId: "" }
#   bigquery:    { clientId: "" }
#   vertexai:    { clientId: "" }
#   hubspot:     { clientId: "" }
#   slack:       { clientId: "" }
#   box:         { clientId: "" }
#   dropbox:     { clientId: "" }
#   dropboxWrite: { clientId: "" }
#   salesforce:  { clientId: "" }
#   supabase:    { clientId: "" }
#   servicenow:  {}                     # client_id/secret entered per-user in UI
#   gong:        { clientId: "", apiBaseUrl: "" }
#   databricks:  { clientId: "", accountId: "", m2mClientId: "" }
#   fabric:      { clientId: "" }
#   azureBlob:   { clientId: "" }
#   huggingface: { clientId: "" }
#   foundry:     { clientId: "", scopes: "", tenant: "" }

# -----------------------------------------------------------------------------
# Bundled datastores. Keep enabled unless you point at external services.
# -----------------------------------------------------------------------------
qdrant:
  enabled: true

# -----------------------------------------------------------------------------
# [OPTIONAL] Object storage. By DEFAULT the platform uses the bundled MinIO and
# you set NOTHING here. To use EXTERNAL S3 (AWS S3 or any S3-compatible store),
# uncomment `storage:` below and add s3AccessKey/s3SecretKey to the single
# `secrets:` block further down.
# -----------------------------------------------------------------------------
# storage:
#   endpoint: "https://s3.us-east-1.amazonaws.com"   # empty = bundled MinIO
#   region: us-east-1
#   bucketName: 4minds-uploads
#   useAzure: "false"          # "true" for Azure Blob (set secrets.azureStorageAccountKey)

# -----------------------------------------------------------------------------
# [OPTIONAL] Secrets. ONE block for ALL of them: OAuth client secrets for the
# integrations you enabled, plus external-S3 / Azure storage keys. The
# PG/Redis/MinIO/encryption keys are AUTO-GENERATED by the chart; do NOT set
# them here. For production prefer external-secrets / sealed-secrets over
# plaintext.
#
# IMPORTANT: keep this a SINGLE `secrets:` block. YAML has no key merging; a
# second `secrets:` anywhere in the file silently overrides the first. Uncomment
# `secrets:` + at least one child (a bare `secrets:` is invalid/null YAML).
# -----------------------------------------------------------------------------
# secrets:
#   # --- External object storage (only if storage.endpoint / useAzure set above) ---
#   s3AccessKey: ""                      # external S3 access key (empty = bundled MinIO)
#   s3SecretKey: ""                      # external S3 secret key
#   azureStorageAccountKey: ""           # when storage.useAzure: "true"
#   # --- OAuth client secrets (only for the providers you enabled above) ---
#   microsoftAuthClientSecret: ""
#   googleOauthClientSecret: ""
#   githubOauthClientSecret: ""
#   oktaOauthClientSecret: ""
#   sharepointOauthClientSecret: ""
#   office365OauthClientSecret: ""
#   hubspotAuthClientSecret: ""
#   slackClientSecret: ""
#   boxOauthClientSecret: ""
#   dropboxClientSecret: ""
#   dropboxWriteClientSecret: ""
#   bigqueryOauthClientSecret: ""
#   vertexaiOauthClientSecret: ""
#   salesforceAuthClientSecret: ""
#   supabaseAuthClientSecret: ""
#   gongOauthClientSecret: ""
#   databricksAuthClientSecret: ""
#   databricksM2mClientSecret: ""
#   fabricAuthClientSecret: ""
#   azureBlobOauthClientSecret: ""
#   hfOauthClientSecret: ""
#   hfApiToken: ""                       # HuggingFace API token (private datasets)

# -----------------------------------------------------------------------------
# [OPTIONAL] Images. DEFAULT to the official 4MINDS registry (do NOT override
# for a real install). Only set this if you mirror images into your own
# registry. NOTE: this belongs under the SAME top-level `global:` block at the
# top of this file; move it there rather than adding a second `global:` key.
# -----------------------------------------------------------------------------
# global:
#   imageRegistry: "your-registry.example.com/4minds"
```
