- AWS-level authentication — how 4MINDS discovers which Redshift clusters and Serverless workgroups exist in your account. Uses IAM Role Federation (recommended) or Amazon Cognito, both shared with the rest of 4MINDS’s AWS integrations.
- Database-level authentication — uses temporary Redshift credentials minted from the same AWS credentials used for discovery. Provisioned clusters use
redshift:GetClusterCredentials; Serverless workgroups useredshift-serverless:GetCredentials. No long-lived database password is stored.
Provisioned vs. Serverless
Both types are listed together when you browse Redshift in the import wizard. The difference is handled server-side; you don’t have to pick a mode.
Redshift IAM Permissions Policy
Whichever AWS auth method you use (IAM Role Federation or Cognito), attach this policy to the role. It covers both provisioned and Serverless. If you only use one, you can drop the statements for the other.
Name the policy something memorable like
4MINDS-Redshift-Access — you’ll attach it when creating the IAM role (for Role Federation) or the Cognito authenticated role.
Note:GetClusterCredentialsis authorized against both thedbuser:anddbname:resource ARNs. The wildcards above allow all clusters, DB users, and databases. To restrict, narrow the ARNs (e.g.,arn:aws:redshift:us-east-1:123456789012:dbuser:my-cluster/fourminds_iam_user).
Least-Privilege: Restricting to Specific Resources
To expose only a subset of your Redshift resources, scope the discovery and credential ARNs:Database Access & Grants
The IAM identity 4MINDS assumes must map to a database user that has read access to the tables you want to import.How 4MINDS decides what you can browse: databases, schemas, and tables are listed through Redshift’s grant-aware system views (SVV_ALL_SCHEMAS,SVV_ALL_TABLES,SVV_ALL_COLUMNS). A schema or table appears only if the resolved DB user has been granted access to it — the user does not need to own it. GrantUSAGEon each schema andSELECTon its tables (below) for every schema you want to browse, not justpublic; a schema the user has no grant on is silently omitted from the picker, and a table without aSELECTgrant won’t appear even if its schema does. The same grants make external (Redshift Spectrum) and datashare tables browsable once they’re accessible to the user.
Provisioned clusters — the DB user
For a provisioned cluster, 4MINDS callsGetClusterCredentials with a DB username you configure (see Database User). That user must already exist in the cluster and have SELECT on the target tables. AutoCreate is not used — 4MINDS never creates users in your cluster.
Older Redshift versions: ifThe IAM policy’sGRANT ... FOR TABLES IN SCHEMA(scoped privileges) isn’t available on your cluster, use the equivalent snapshot pattern instead — but remember it only covers objects that exist now, so re-run it (or set default privileges per creating role) as pipelines add tables:
dbuser: ARN must match this username (e.g., .../my-cluster/fourminds_iam_user).
Important — why newly-created tables can go missing (most common cause of “most of my tables don’t show up”):Fix: run
GRANT SELECT ON ALL TABLES IN SCHEMAis a one-time snapshot. It grants access only to the objects that exist at the moment you run it. Any table, view, or materialized view created afterward is not covered and will silently be omitted from the browser until you re-grant. If a pipeline (e.g. dbt, an ETL job, or a nightly build) keeps adding objects to a schema, you’ll see the older objects but not the newer ones.ALTER DEFAULT PRIVILEGESonly applies to objects created by the role that runs it, going forward. Default privileges are scoped to the creating role. If your tables are created by a different role than the one that ran theALTER DEFAULT PRIVILEGESstatement (very common — e.g. tables created by adbt_servicerole while you granted as an admin), the default grant does not apply and new objects stay invisible.ALTER DEFAULT PRIVILEGESas each role that creates objects in the schema. For example, if dbt writes as roledbt_service:To immediately catch up existing objects the read user is still missing, re-runGRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO fourminds_iam_user;once now.
Simpler fix — grant a standing rule for the whole schema. Redshift’s scoped-privileges syntax grantsSELECTon existing and future tables and views in a schema in a single statement, regardless of which role creates them — so it avoids both theALL TABLESsnapshot gap and the creator-scoped limitation ofALTER DEFAULT PRIVILEGES:This is the recommended one-shot fix when a customer reports that only some tables in a schema show up: it backfills the objects the read user is missing today and keeps new objects visible as pipelines add them, without having to runALTER DEFAULT PRIVILEGESper creating role.
Serverless workgroups — IAM-derived user
Redshift ServerlessGetCredentials does not accept a DB username — it derives the database user from the IAM identity making the call (returned as IAMR:<role-name> or IAM:<user-name>). To grant it access, create/grant that exact user in the database:
Tip: If you’re unsure of the exact derived username, run a browse in 4MINDS and check the error message — apermission deniedfailure will name the DB user Redshift resolved, which you can thenGRANTto.
Note: For Serverless,GetCredentialsauto-creates theIAMR:<role>database user on first use, so the explicitCREATE USERabove is optional — grantingSELECTtoPUBLIC(or to the resolved user after a first browse) is sufficient. Pre-creating the user is only needed if you want to grant privileges before the first connection.
Views (regular, late-binding, materialized)
4MINDS lists and imports views alongside tables — regular views, late-binding views (WITH NO SCHEMA BINDING), and materialized views all appear in the browser and import through the same reflect-and-SELECT path a table uses.
Grant behavior (verified on Redshift):
GRANT SELECT FOR TABLES IN SCHEMA <schema>(and the olderGRANT SELECT ON ALL TABLES IN SCHEMA <schema>) cover all three view types as well as tables — no separate grant is required for views or materialized views.- The standing-rule
FOR TABLES IN SCHEMAgrant covers future views and materialized views automatically. If you use the snapshotON ALL TABLESform instead, pair it withALTER DEFAULT PRIVILEGES IN SCHEMA <schema> GRANT SELECT ON TABLES(per creating role) so view-backed datasets stay browsable as they’re added. - Reading a view needs
SELECTon the view itself only — the DB user does not needSELECTon the view’s underlying base tables. This holds for regular, late-binding, and materialized views. - Browsing is consistent with preview: if a view appears in the picker, the resolved user can preview and import it. There’s no “shows in the list but fails to import” gap from grants.
ALL TABLES, remember to include the specific views and materialized views you want browsable — otherwise they’re silently omitted just like an ungranted table:
Connection Method 1: IAM Role Federation (Recommended)
Short-lived credentials minted per request through AWS STS — no long-lived AWS keys stored in 4MINDS.- Complete the one-time AWS setup in AWS Integrations → IAM Role Federation (OIDC provider + IAM role with trust policy).
- Attach the Redshift IAM policy above to the role (Role Federation and Cognito can share the same role — just attach both service policies to it if you use multiple AWS integrations).
- In 4MINDS, open Integrations → Amazon Redshift → click Configure (or Settings if already connected).
- Keep the IAM Role tab selected.
- Paste your IAM Role ARN (e.g.,
arn:aws:iam::123456789012:role/4MINDS-integration-role). - Leave the External ID field blank — it is not supported for IAM Role Federation (see AWS Integrations → Verify the Trust Policy).
- Enter your AWS Region (e.g.,
us-east-1) — this must match the region of the Redshift resources you want to browse. - (Provisioned only) Enter a Database user — see Database User. Leave blank to use the default
fourminds_iam_user. - Click Test Connection — success shows “Connection successful. Found N Redshift cluster(s).”
- Click Save Credentials.
Connection Method 2: Amazon Cognito
Use this if your organization already authenticates against AWS through Cognito User Pools and Identity Pools.- Complete the one-time Cognito setup in AWS Integrations → Amazon Cognito (User Pool, App Client, Identity Pool, authenticated role).
- Attach the Redshift IAM policy above to the Cognito authenticated role.
- In 4MINDS, open Integrations → Amazon Redshift → click Configure.
- Switch to the Cognito tab.
- Fill in User Pool ID, App Client ID, App Client Secret (if configured), Identity Pool ID, Username, and Password.
- Enter your AWS Region.
- (Provisioned only) Enter a Database user if you’re not using the default.
- Click Test Connection, then Save Credentials.
Database User
GetClusterCredentials (provisioned clusters only) authenticates as a specific database user. In the settings modal, the Database user field controls which user 4MINDS requests credentials for:
- Defaults to
fourminds_iam_userif left blank. - The user must already exist in the cluster with
SELECTon the tables you want to import (see Database Access & Grants). - The value is stored once with your connection metadata and reused for every browse and import — you don’t re-enter it per request.
- Ignored for Serverless — Serverless derives the user from the IAM identity, so this field has no effect on workgroups.
4MINDS Fields
Importing Data from Redshift
Once connected, open the Datasets page and click Add Source → Amazon Redshift (or pick Redshift from the data source bar inside an existing dataset). The import wizard walks you through the hierarchy: clusters/workgroups → databases → tables.Step 1 — Pick a Cluster or Workgroup
4MINDS callsredshift:DescribeClusters and redshift-serverless:ListWorkgroups in your configured region and shows everything the role can see. Provisioned clusters show their node type and node count; Serverless workgroups are labeled Serverless with their namespace. Click one to continue.
If the Serverless permissions are missing, workgroups are simply omitted — provisioned clusters still list normally.
Step 2 — Browse Databases
4MINDS mints a short-lived credential (viaGetClusterCredentials for a cluster, or GetCredentials for a workgroup), opens a live SQL connection to the endpoint over SSL, and lists databases through the grant-aware SVV_REDSHIFT_DATABASES catalog. This surfaces every database the resolved DB user can access — including datashare (shared) databases provided by other clusters or accounts — not just local databases. System databases (template0, template1, padb_harvest, rdsadmin, and the non-connectable sys:internal) are filtered out.
Step 3 — Select Tables
Click a database to list its tables and views — local, external/Spectrum, and datashare relations the DB user can access, including regular, late-binding, and materialized views. For each relation the browser shows:- Schema-qualified name (e.g.,
public.customers) - Column count
- On-disk size (when available)
Note: Listing of views (regular, late-binding, and materialized) alongside tables requires the current 4MINDS release. Earlier releases listed base tables only; if views are missing from the browser and you’ve confirmed the grants above, verify you’re on the latest release.
During Import
At import time, 4MINDS runs a bulk fetch on each selected table:- Up to 50,000 rows per table (hard cap — larger tables are truncated and flagged).
- Binary columns (
BYTEA,VARBINARY,BINARY) are dropped from the export rather than base64-encoded. - Complex types (arrays, structured values) are serialized with
json.dumpsinto a single cell. - Timeout is 120 seconds per table.
Dataset Sync
Overview
Beyond one-time imports, 4MINDS can keep a Redshift-backed dataset up to date automatically. When you enable Dataset Sync on a dataset built from Redshift tables, 4MINDS re-exports the configured tables on a schedule — no manual re-imports. Unlike file-based sources (S3, Dropbox, Databricks volumes), Redshift is a table warehouse with no cheap per-table “last modified” signal. So Redshift sync follows the same full-snapshot model as BigQuery rather than incremental file diffing: each sync cycle re-exports the whole table (up to the 50,000-row cap) and replaces the previous snapshot. There is no row-level change detection — every run is a fresh point-in-time copy.How to Set Up Sync
- Import one or more Redshift tables into a 4MINDS dataset (using the import wizard above).
- On the dataset, toggle Dataset Sync on.
- Select a sync frequency (see table below).
- From that point on, 4MINDS re-exports each configured table at the chosen interval.
Sync Frequencies
Because every Redshift sync is a full-table re-export, prefer less frequent intervals for large tables to control export cost and RPU/credit usage. Daily or weekly is a good default for warehouse tables.
How It Works Internally
Full snapshot, every cycle: Because Redshift exposes no per-table modification timestamp, the sync manifest records nomodified_at for Redshift entries. As a result, the writer treats every configured table as changed on every run and re-downloads a fresh snapshot — the same semantic BigQuery uses.
Stable table identity: Each table is tracked by a stable identifier of the form rs-<cluster>-<database>-<schema>-<table> (schema defaults to public). The manifest de-duplicates on this id, so re-running a sync replaces the prior snapshot of a table rather than accumulating duplicates.
Export pipeline: On each cycle, for every configured table 4MINDS mints a short-lived Redshift credential (via GetClusterCredentials for a cluster, or GetCredentials for a workgroup), opens an SSL SQL connection, and exports the table to CSV bytes — up to 50,000 rows (the same cap as one-shot import; larger tables are truncated and flagged). Binary columns are dropped and empty tables are skipped. The CSV is uploaded to the user’s dataset storage and processed through the 4MINDS ETL pipeline.
Concurrent processing: As with other sources, multiple dataset syncs run in parallel, each with its own isolated database session.
Authentication for Automated Sync
Dataset sync runs in the background with no user present, so 4MINDS replays the stored AWS connection to authenticate each cycle — exactly the same two-layer model used for interactive browsing:- The stored auth method (IAM Role Federation or Cognito) is resolved into fresh temporary AWS credentials for the run (STS
AssumeRoleWithWebIdentity, or CognitoGetCredentialsForIdentity). - Those credentials mint a fresh temporary Redshift credential per table (
GetClusterCredentials/GetCredentials).
SELECT on the synced tables (see Database Access & Grants) — a revoked grant surfaces as a skipped table in the sync logs.
Networking Requirements
Redshift clusters and Serverless workgroups live inside a VPC. Only the 4MINDS backend opens SQL connections to Redshift — the browser app (app.4minds.ai) never connects to your database directly, so you only need to allow the backend’s egress IP. There is no frontend IP to allowlist.
For 4MINDS to open a SQL connection, the endpoint must be reachable from the backend’s outbound network:
-
Allow the 4MINDS backend egress IP on the Redshift port (default
5439):Add it as an inbound rule on the security group attached to your cluster or Serverless workgroup. AWS CLI:Or in the console: EC2 → Security Groups → your Redshift SG → Inbound rules → Edit inbound rules → Add rule, then set Type =Custom TCP, Port range =5439, Source =20.7.240.218/32. - Publicly accessible = Yes is the simplest path for temporary testing; for production, prefer a VPC peering / PrivateLink setup with 4MINDS support.
-
SSL — All connections use SSL (
sslmode=require), as required for temporary-credential auth.
Private cluster behind a load balancer (Connection host override)
Endpoint discovery (redshift:DescribeClusters for a provisioned cluster, redshift-serverless:GetWorkgroup for a Serverless workgroup) returns the source’s internal endpoint. If your cluster or workgroup can’t be made publicly accessible (for example, its VPC has no internet gateway) but you expose a reachable path in front of it — such as a Network Load Balancer (NLB) — that internal endpoint won’t resolve or connect from the 4MINDS backend, and Test/browse will time out.
For this case, set the optional Connection host field in the Redshift connection settings to your reachable hostname (e.g. my-cluster-nlb.elb.us-west-2.amazonaws.com). 4MINDS then opens the SQL connection to that host instead of the discovered endpoint, while still minting temporary IAM credentials against your Cluster identifier (provisioned) or Workgroup (Serverless) — so IAM auth is unchanged. This works the same way for provisioned clusters and Serverless workgroups.
Requirements when using the override:
- The host must forward TCP
5439to your cluster or workgroup and present its certificate (SSL is still required). A Network Load Balancer with a TCP listener passes the TLS connection straight through, so the source’s own certificate reaches the backend. - The Cluster identifier / Workgroup, Region, and Database user must still match the real source so credential minting (
GetClusterCredentialsfor a cluster,GetCredentialsfor a workgroup) succeeds. - The 4MINDS backend egress IP (
20.7.240.218/32) must be allowed to reach the override host on port5439. - One override applies per connection; it’s intended for a single private cluster or workgroup reached through a fixed hostname.
Note: If a connection still times out after allowlisting the IP above, the backend egress IP may have changed — contact 4MINDS support to confirm the current address before widening your rules.If the SQL connection times out, the browser shows:
“Connection timed out. The backend may not be able to reach the Redshift endpoint — check VPC / security group rules (inbound port open from the backend’s network).”
Security Model
No database passwords are stored or transmitted. Each SQL operation mints a short-lived credential from the existing AWS credentials, opens a short-lived SQLAlchemy engine over SSL, runs one query, and disposes of the engine before returning.
Testing Your Connection
After saving credentials, the Test Connection button:- Resolves your auth method → AWS credentials (via STS AssumeRoleWithWebIdentity for Role Federation, or Cognito GetCredentialsForIdentity for Cognito).
- Calls
redshift:DescribeClusterswith a 15-second timeout. - Returns “Connection successful. Found N Redshift cluster(s).” on success.
Troubleshooting
Debugging missing tables or schemas
4MINDS populates the picker from Redshift’s grant-awareSVV_ALL_* catalog views, so an object appears only if the DB user 4MINDS connects as has been granted access to it. When a customer connects successfully but sees fewer schemas or tables than expected, work through these checks as the same DB user 4MINDS connects with (fourminds_iam_user unless configured otherwise), in Redshift Query Editor v2, DBeaver, or psql.
Step 1 — Confirm the user and database.
current_user should be the configured DB user, and current_database() the database the connection points at. A wrong user or database alone explains missing objects.
Step 2 — See what this user can see in the schema, by object type. This is what the picker shows:
current_database(), so objects that resolve under a different database name (cross-database references, datashare-provided databases, or an external Glue/awsdatacatalog catalog) won’t appear:
Whole schema missing vs. some tables missing: a schema that’s entirely absent from the picker means the user lacksUSAGEon it (svv_all_schemasonly returns schemas the user can enter). A schema that shows but is missing some tables is aSELECTgap — most often theALL TABLESsnapshot going stale as pipelines add objects. See Database Access & Grants.
Disconnecting
To remove the Redshift integration:- Open Integrations → Amazon Redshift → click Settings.
- Click Disconnect.