Deployment¶

This guide covers setting up the production infrastructure for the Naluma Content Pipeline.

Prerequisites¶

Fly.io CLI installed and authenticated
Neon Postgres project
Cloudflare account (for documentation site)
GitHub repository with Actions enabled

1. Fly.io Setup¶

Initial Setup¶

# From the repository root
fly launch --no-deploy

# Select region: fra (Frankfurt)
# Select machine: shared-cpu-1x, 3072MB

Configure Secrets¶

Set all required environment variables as Fly.io secrets:

fly secrets set \
  ANTHROPIC_API_KEY="sk-ant-..." \
  DATABASE_URL="postgresql+asyncpg://pipeline:password@ep-xxx-pooler.neon.tech/naluma_content?sslmode=require" \
  DATABASE_URL_MIGRATIONS="postgresql+asyncpg://migrations:password@ep-xxx.neon.tech/naluma_content?sslmode=require" \
  WP_BASE_URL="https://yourdomain.com" \
  WP_USERNAME="content-pipeline" \
  WP_APP_PASSWORD="xxxx xxxx xxxx xxxx" \
  PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:pass@ep-xxx.neon.tech/prefect_server?ssl=require" \
  OPENAI_API_KEY="sk-..." \
  NCBI_API_KEY="..." \
  BRAVE_API_KEY="..." \
  R2_ACCOUNT_ID="..." \
  R2_ACCESS_KEY_ID="..." \
  R2_SECRET_ACCESS_KEY="..." \
  R2_BUCKET_NAME="naluma-wordpress" \
  SLACK_WEBHOOK_URL="https://hooks.slack.com/..." \
  LANGFUSE_PUBLIC_KEY="pk-lf-..." \
  LANGFUSE_SECRET_KEY="sk-lf-..." \
  SENTRY_DSN="https://xxx@xxx.ingest.sentry.io/xxx"

Note: PREFECT_API_URL is set in fly.toml [env] (non-secret, points to http://localhost:4200/api). PREFECT_API_DATABASE_CONNECTION_URL contains credentials and must be a Fly.io secret. Use the direct (non-pooler) Neon endpoint.

Deploy¶

# Manual deploy
fly deploy

# Automated: merging to main triggers .github/workflows/deploy.yml (after CI passes)
# Requires FLY_API_TOKEN repository secret in GitHub Settings > Secrets

GitHub Actions Secret¶

Generate a Fly.io deploy token and add it as a GitHub repository secret:

fly tokens create deploy -x 999999h
# Copy the token -> GitHub repo -> Settings -> Secrets -> New: FLY_API_TOKEN

Scaling¶

fly status       # Check current status
fly scale memory 512   # Scale memory if needed
fly logs         # View logs

2. Prefect Server (Self-Hosted)¶

The Prefect server runs co-located with the worker inside the Fly.io container. It uses a dedicated Neon Postgres database (prefect_server) as its state backend.

Architecture¶

On container startup, scripts/start-pipeline.sh: 1. Runs Alembic migrations for the application database 2. Starts the Prefect server in the background (port 4200) 3. Waits for the server health endpoint (30s timeout) 4. Starts the Prefect worker as the foreground process

Database Setup¶

Create the prefect_server database in the same Neon project (see section 4):

CREATE DATABASE prefect_server;

Set the connection URL as a Fly.io secret (use the direct Neon endpoint, not the pooler):

fly secrets set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:pass@ep-xxx.neon.tech/prefect_server?ssl=require"

Deploy Flows¶

After the Fly.io machine is running, register flow deployments:

# Via fly ssh console
fly ssh console -C "uv run prefect deploy --all"

# Or via local proxy
fly proxy 4200:4200 &
PREFECT_API_URL=http://localhost:4200/api prefect deploy --all

This registers 9 deployments across two work pools:

Content Pool:

research-news-scan — weekly Sunday 06:00 UTC
produce-digests — manual trigger
generate-briefs — manual trigger
batch-produce — manual trigger
produce-article — manual trigger
resume-article — manual trigger
publish-articles — manual trigger
republish-articles — manual trigger

Recovery Pool:

recover-stuck-articles — every 30 minutes

Create Work Pools¶

fly ssh console -C "uv run prefect work-pool create content-pool --type process"
fly ssh console -C "uv run prefect work-pool create recovery-pool --type process"

The recovery pool ensures stuck-article recovery is never blocked by production work filling the content pool.

Monitoring¶

The Prefect UI is not exposed externally. To access it:

fly proxy 4200:4200
# Open http://localhost:4200

3. Dashboard (Fly.io + Cloudflare Access)¶

The Streamlit dashboard runs as a separate Fly.io app (naluma-dashboard) using fly.dashboard.toml and Dockerfile.dashboard. Authentication is handled by Cloudflare Access (Zero Trust) with Google + GitHub OAuth.

Setup¶

# Create a second Fly.io app for the dashboard
fly launch --no-deploy --config fly.dashboard.toml

# Select region: fra (Frankfurt)
# Select machine: shared-cpu-1x, 512MB

Custom Domain¶

Add a custom domain so Cloudflare Access can protect it:

fly certs create dashboard.naluma.space -a naluma-dashboard
fly certs show dashboard.naluma.space -a naluma-dashboard

In Cloudflare DNS, add a CNAME record: dashboard.naluma.space → naluma-dashboard.fly.dev (proxied).

Cloudflare Access (Authentication)¶

Go to Cloudflare Zero Trust → Access → Applications → Add Application
Type: Self-hosted, domain: dashboard.naluma.space
Session duration: 7 days
Add policy: Allow — Include: specific email addresses
Identity providers: Google + GitHub (configure OAuth client ID + secret in Cloudflare)
Note the Application Audience (AUD) Tag from the application config — needed for CF_ACCESS_AUD below

Configure Secrets¶

fly secrets set -a naluma-dashboard \
  DATABASE_URL="postgresql+asyncpg://pipeline:password@ep-xxx-pooler.neon.tech/naluma_content?sslmode=require" \
  ANTHROPIC_API_KEY="sk-ant-..." \
  WP_BASE_URL="https://yourdomain.com" \
  WP_USERNAME="content-pipeline" \
  WP_APP_PASSWORD="xxxx xxxx xxxx xxxx" \
  R2_ACCOUNT_ID="..." \
  R2_ACCESS_KEY_ID="..." \
  R2_SECRET_ACCESS_KEY="..." \
  R2_BUCKET_NAME="naluma-wordpress" \
  CF_ACCESS_TEAM_DOMAIN="your-team-name" \
  CF_ACCESS_AUD="your-application-audience-tag"

Admin User Setup¶

Ensure the admin user's email in dashboard_users matches the Google/GitHub account email:

UPDATE dashboard_users SET email = 'your-actual@email.com' WHERE username = 'admin';

New users are created via the User Management page (admin only) with email + role — no passwords needed.

Deploy¶

# Manual deploy
fly deploy --config fly.dashboard.toml

# Automated: deploy.yml deploys both worker and dashboard via matrix strategy (after CI passes)

The dashboard is accessible at https://dashboard.naluma.space.

Local Development¶

For local development without Cloudflare Access:

ENVIRONMENT=local DASHBOARD_DEV_EMAIL=admin@example.com uv run streamlit run dashboard/app.py

4. Neon Database Setup¶

Create Project and Branches¶

Sign in to Neon Console
Create project: naluma_content
The default branch is main (production)
Create a staging branch: Branches > Create Branch from main

Configure IP Allowlist¶

Under Project Settings > IP Allow:

Add Fly.io egress IPs for the fra region
Add your local development IP
Enable Reject connections from unlisted IPs

Create Databases¶

The project uses two databases in the same Neon project:

-- Application database (created automatically with the project)
-- naluma_content

-- Prefect server state backend
CREATE DATABASE prefect_server;

The prefect_server database uses the direct (non-pooler) Neon endpoint because Prefect's internal SQLAlchemy engine uses prepared statements, which are incompatible with PgBouncer.

Create Database Users¶

-- Pipeline user (app runtime) -- no DDL
CREATE USER pipeline WITH PASSWORD 'secure-password-here';
GRANT CONNECT ON DATABASE naluma_content TO pipeline;
GRANT USAGE ON SCHEMA public TO pipeline;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO pipeline;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE ON TABLES TO pipeline;

-- Migrations user (Alembic DDL)
CREATE USER migrations WITH PASSWORD 'secure-password-here';
GRANT CONNECT ON DATABASE naluma_content TO migrations;
GRANT USAGE, CREATE ON SCHEMA public TO migrations;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO migrations;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO migrations;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO migrations;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO migrations;

Connection Strings¶

App: postgresql+asyncpg://pipeline:PASSWORD@ep-xxx.neon.tech/naluma_content?sslmode=require
Migrations: postgresql+asyncpg://migrations:PASSWORD@ep-xxx.neon.tech/naluma_content?sslmode=require
Prefect server (direct endpoint): postgresql+asyncpg://user:PASSWORD@ep-xxx.neon.tech/prefect_server?ssl=require

Use the app URL for DATABASE_URL, the migrations URL for DATABASE_URL_MIGRATIONS, and the Prefect URL for PREFECT_API_DATABASE_CONNECTION_URL.

Staging Reset¶

To reset the staging branch to match production:

Neon Console > Branches: Delete the staging branch
Create Branch from main (instant copy of production data + schema)
Update staging connection strings if endpoint changed

Test Migrations¶

DATABASE_URL_MIGRATIONS="postgresql+asyncpg://migrations:PASSWORD@ep-staging.neon.tech/naluma_content?sslmode=require" \
  uv run alembic upgrade head

5. Documentation Site (Cloudflare Pages)¶

The MkDocs Material documentation site auto-deploys to Cloudflare Pages.

Setup¶

Create a Cloudflare Pages project named naluma-content-automation
Add repository secrets in GitHub:
CLOUDFLARE_API_TOKEN — API token with Pages edit permissions
CLOUDFLARE_ACCOUNT_ID — your Cloudflare account ID

Deploy¶

Automatic on push to main when docs/site/** or mkdocs.yml files change. Manual trigger via workflow_dispatch.

Local Preview¶

uv run mkdocs serve --livereload
# Open http://127.0.0.1:8000