Run the Pipeline¶

The pipeline is orchestrated with Prefect. Flows can be triggered manually via the Prefect UI/CLI or run on a schedule.

Prerequisites¶

Database migrations applied (uv run alembic upgrade head)
Environment variables configured (see Configuration)
Prefect worker running (for deployed flows) or local execution

Triggering Flows¶

Via Prefect Cloud UI¶

Navigate to Deployments in the Prefect Cloud dashboard
Select the flow (e.g., batch-produce)
Click Run and provide parameters

Via Prefect CLI¶

# Run a single article through the full pipeline
prefect deployment run 'produce-article/produce-article' \
  --param article_id=<UUID>

# Generate briefs for backlog items
prefect deployment run 'generate-briefs/generate-briefs' \
  --param backlog_item_ids='["<UUID1>", "<UUID2>"]'

# Batch produce multiple articles
prefect deployment run 'batch-produce/batch-produce' \
  --param article_ids='["<UUID1>", "<UUID2>"]' \
  --param concurrency=5

# Publish approved articles to WordPress
prefect deployment run 'publish-articles/publish-articles' \
  --param article_ids='["<UUID1>"]'

Via Python (Local)¶

For development, you can run flows directly:

import asyncio
from src.pipeline.article_flow import produce_article

asyncio.run(produce_article(article_id=uuid))

Flow Parameters¶

`produce_article`¶

Parameter	Type	Description
`article_id`	`UUID`	The article to process (must be in `brief_approved` status)

`generate_briefs`¶

Parameter	Type	Description
`backlog_item_ids`	`list[UUID]`	Backlog items to generate briefs for

`batch_produce`¶

Parameter	Type	Description
`article_ids`	`list[UUID]`	Articles to produce
`concurrency`	`int`	Max parallel articles (default: 5)

`publish_articles`¶

Parameter	Type	Description
`article_ids`	`list[UUID]`	Articles to publish to WordPress
`concurrency`	`int`	Max parallel publishes (default: 5)

`research_news_scan`¶

Parameter	Type	Description
`languages`	`list[str] \| None`	Language codes (default: `["en", "de"]`)

Runs weekly on Sunday at 06:00 UTC. Skips the week if fewer than 3 newsworthy items are found.

Monitoring¶

Prefect Cloud UI — flow run status, logs, task timelines
Dashboard Pipeline page — overview of recent runs, status counts, cost tracking
Slack alerts — failure notifications via webhook (if SLACK_WEBHOOK_URL is set)

Cost Limits¶

Each flow has a per-run cost ceiling:

Flow	Limit (USD)
`produce_article`	$5.00
`batch_produce`	$100.00
`research_news_scan`	$10.00

If a flow exceeds its budget, it raises CostLimitExceeded and stops. See Cost Management for details.