Deployment¶

This guide covers setting up the production infrastructure for the Naluma Content Pipeline.

Prerequisites¶

Fly.io CLI installed and authenticated
Neon Postgres project
Cloudflare account (for documentation site)
GitHub repository with Actions enabled

1. Fly.io Setup¶

Initial Setup¶

# From the repository root
fly launch --no-deploy

# Select region: fra (Frankfurt)
# Select machine: shared-cpu-1x, 1024MB

Configure Secrets¶

Set all required environment variables as Fly.io secrets:

fly secrets set \
  ANTHROPIC_API_KEY="sk-ant-..." \
  DATABASE_URL="postgresql+asyncpg://pipeline:password@ep-xxx.neon.tech/naluma_content?sslmode=require" \
  DATABASE_URL_MIGRATIONS="postgresql+asyncpg://migrations:password@ep-xxx.neon.tech/naluma_content?sslmode=require" \
  WP_BASE_URL="https://yourdomain.com" \
  WP_USERNAME="content-pipeline" \
  WP_APP_PASSWORD="xxxx xxxx xxxx xxxx" \
  PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:pass@ep-xxx.neon.tech/prefect_server?ssl=require" \
  SLACK_WEBHOOK_URL="https://hooks.slack.com/..."

Note: PREFECT_API_URL is set in fly.toml [env] (non-secret, points to http://localhost:4200/api). PREFECT_API_DATABASE_CONNECTION_URL contains credentials and must be a Fly.io secret. Use the direct (non-pooler) Neon endpoint.

Deploy¶

# Manual deploy
fly deploy

# Automated: merging to main triggers .github/workflows/deploy.yml (after CI passes)
# Requires FLY_API_TOKEN repository secret in GitHub Settings > Secrets

GitHub Actions Secret¶

Generate a Fly.io deploy token and add it as a GitHub repository secret:

fly tokens create deploy -x 999999h
# Copy the token -> GitHub repo -> Settings -> Secrets -> New: FLY_API_TOKEN

Scaling¶

fly status       # Check current status
fly scale memory 512   # Scale memory if needed
fly logs         # View logs

2. Prefect Server (Self-Hosted)¶

The Prefect server runs co-located with the worker inside the Fly.io container. It uses a dedicated Neon Postgres database (prefect_server) as its state backend.

Architecture¶

On container startup, scripts/start-worker.sh: 1. Runs Alembic migrations for the application database 2. Starts the Prefect server in the background (port 4200) 3. Waits for the server health endpoint (30s timeout) 4. Starts the Prefect worker as the foreground process

Database Setup¶

Create the prefect_server database in the same Neon project (see section 4):

CREATE DATABASE prefect_server;

Set the connection URL as a Fly.io secret (use the direct Neon endpoint, not the pooler):

fly secrets set PREFECT_API_DATABASE_CONNECTION_URL="postgresql+asyncpg://user:pass@ep-xxx.neon.tech/prefect_server?ssl=require"

Deploy Flows¶

After the Fly.io machine is running, register flow deployments:

# Via fly ssh console
fly ssh console -C "uv run prefect deploy --all"

# Or via local proxy
fly proxy 4200:4200 &
PREFECT_API_URL=http://localhost:4200/api prefect deploy --all

This registers 6 deployments:

research-news-scan — weekly Sunday 06:00 UTC
generate-briefs — manual trigger
batch-produce — manual trigger
produce-article — manual trigger (concurrency: 5)
publish-articles — manual trigger
cornerstone-update — manual trigger

Create Work Pool¶

fly ssh console -C "uv run prefect work-pool create content-pool --type process"

Monitoring¶

The Prefect UI is not exposed externally. To access it:

fly proxy 4200:4200
# Open http://localhost:4200

3. Dashboard (Fly.io)¶

The Streamlit dashboard runs as a separate Fly.io app (naluma-dashboard) using fly.dashboard.toml and Dockerfile.dashboard.

Setup¶

# Create a second Fly.io app for the dashboard
fly launch --no-deploy --config fly.dashboard.toml

# Select region: fra (Frankfurt)
# Select machine: shared-cpu-1x, 256MB

Configure Secrets¶

fly secrets set --config fly.dashboard.toml \
  DATABASE_URL="postgresql+asyncpg://pipeline:password@ep-xxx.neon.tech/naluma_content?sslmode=require" \
  ANTHROPIC_API_KEY="sk-ant-..." \
  WP_BASE_URL="https://yourdomain.com" \
  WP_USERNAME="content-pipeline" \
  WP_APP_PASSWORD="xxxx xxxx xxxx xxxx" \
  DASHBOARD_ADMIN_EMAIL="admin@example.com" \
  DASHBOARD_ADMIN_PASSWORD="secure-password"

Deploy¶

# Manual deploy
fly deploy --config fly.dashboard.toml

# Automated: deploy.yml deploys both worker and dashboard via matrix strategy (after CI passes)

The dashboard is accessible at https://naluma-dashboard.fly.dev.

4. Neon Database Setup¶

Create Project and Branches¶

Sign in to Neon Console
Create project: naluma_content
The default branch is main (production)
Create a staging branch: Branches > Create Branch from main

Configure IP Allowlist¶

Under Project Settings > IP Allow:

Add Fly.io egress IPs for the fra region
Add your local development IP
Enable Reject connections from unlisted IPs

Create Databases¶

The project uses two databases in the same Neon project:

-- Application database (created automatically with the project)
-- naluma_content

-- Prefect server state backend
CREATE DATABASE prefect_server;

The prefect_server database uses the direct (non-pooler) Neon endpoint because Prefect's internal SQLAlchemy engine uses prepared statements, which are incompatible with PgBouncer.

Create Database Users¶

-- Pipeline user (app runtime) -- no DDL
CREATE USER pipeline WITH PASSWORD 'secure-password-here';
GRANT CONNECT ON DATABASE naluma_content TO pipeline;
GRANT USAGE ON SCHEMA public TO pipeline;
GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA public TO pipeline;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE ON TABLES TO pipeline;

-- Migrations user (Alembic DDL)
CREATE USER migrations WITH PASSWORD 'secure-password-here';
GRANT CONNECT ON DATABASE naluma_content TO migrations;
GRANT USAGE, CREATE ON SCHEMA public TO migrations;
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO migrations;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO migrations;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO migrations;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO migrations;

Connection Strings¶

App: postgresql+asyncpg://pipeline:PASSWORD@ep-xxx.neon.tech/naluma_content?sslmode=require
Migrations: postgresql+asyncpg://migrations:PASSWORD@ep-xxx.neon.tech/naluma_content?sslmode=require
Prefect server (direct endpoint): postgresql+asyncpg://user:PASSWORD@ep-xxx.neon.tech/prefect_server?ssl=require

Use the app URL for DATABASE_URL, the migrations URL for DATABASE_URL_MIGRATIONS, and the Prefect URL for PREFECT_API_DATABASE_CONNECTION_URL.

Staging Reset¶

To reset the staging branch to match production:

Neon Console > Branches: Delete the staging branch
Create Branch from main (instant copy of production data + schema)
Update staging connection strings if endpoint changed

Test Migrations¶

DATABASE_URL_MIGRATIONS="postgresql+asyncpg://migrations:PASSWORD@ep-staging.neon.tech/naluma_content?sslmode=require" \
  uv run alembic upgrade head

5. Documentation Site (Cloudflare Pages)¶

The MkDocs Material documentation site auto-deploys to Cloudflare Pages.

Setup¶

Create a Cloudflare Pages project named naluma-content-automation
Add repository secrets in GitHub:
CLOUDFLARE_API_TOKEN — API token with Pages edit permissions
CLOUDFLARE_ACCOUNT_ID — your Cloudflare account ID

Deploy¶

Automatic on push to main when docs/site/** or mkdocs.yml files change. Manual trigger via workflow_dispatch.

Local Preview¶

uv run mkdocs serve --livereload
# Open http://127.0.0.1:8000