Run Nightly E2E Tests on Staging

Don’t want to set this up manually? Paste a link to this page into a Devin session and ask it to set everything up for you.

This guide walks through managing scheduled automations via the v3 API, which is useful for infrastructure-as-code workflows. You can also create and manage them directly on the Automations page without any API setup.

Set up a service user for API access

Automations created through the API need a service user with the right permissions. You’ll set one up once, then use its API key in all the calls below.

Go to app.devin.ai > Settings > Devin API, open the Service users tab, and click Provision service user
Assign a role that includes the ManageOrgAutomations permission
Save the API key shown after provisioning — it’s only displayed once and you’ll use it as your Bearer token

Your organization ID is shown at the top of the Settings > Devin API page. Enterprise service users whose role includes ViewOrganizations can also call the List Organizations endpoint with their token:

curl "https://api.devin.ai/v3/enterprise/organizations" \
  -H "Authorization: Bearer $DEVIN_API_KEY"

Export both values so the commands in this guide work as-is:

export DEVIN_API_KEY="sk-your-service-user-key"
export ORG_ID="your-org-id"

See the API authentication docs for more on service users and permissions.

Write a playbook for the test run

Before creating the automation, write a playbook that tells Devin exactly how to run your E2E suite and what to do with the results. Go to Settings > Playbooks and create a new playbook — or ask Devin to generate one for you from a description of your test workflow. Here’s an example for a Playwright suite:Note the playbook ID after saving — you’ll reference it in the automation’s prompt. You can find it in the URL when viewing the playbook (app.devin.ai/.../playbooks/{playbook_id}).

Install the Linear integration so Devin can create tickets as part of the playbook. When editing the automation on the Automations page, you can also add a Post to Slack notification (e.g., #qa-results) so your team gets notified automatically. Give Devin read-only access to your staging environment’s secrets (database URLs, API keys) via organization secrets if your tests need them.

Create the nightly automation via the API

Now use the POST /v3/organizations/{org_id}/automations endpoint to register an automation with a schedule:recurring trigger and a start_session action. Reference the playbook in the prompt with an @playbook:{id} token. Sessions started by an automation only get the tools you grant it, so the tools block enables Linear tools and lets Devin post to your #qa-results channel (replace the Slack workspace and channel IDs with your own). This example runs every night at 2 AM UTC:

curl -X POST "https://api.devin.ai/v3/organizations/$ORG_ID/automations" \
  -H "Authorization: Bearer $DEVIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Nightly E2E — staging",
    "run_as": {"type": "organization"},
    "triggers": [{
      "event_type": "schedule:recurring",
      "conditions": {"any": [{"all": [{
        "field": "rrule",
        "operator": "recurrence",
        "value": "FREQ=DAILY;BYHOUR=2;BYMINUTE=0"
      }]}]}
    }],
    "actions": [{
      "type": "start_session",
      "prompt": "Run the nightly E2E test suite against staging. Follow @playbook:your-playbook-id exactly."
    }],
    "tools": {
      "linear_enabled": true,
      "slack_channels": [{"team_id": "T0123456789", "channel_id": "C0123456789"}]
    }
  }'

The response includes an automation_id you’ll use to manage this automation later. Save it:

export AUTOMATION_ID="a1b2c3d4e5f6789012345678"  # automation_id from the response

The rrule condition takes an iCalendar RRULE evaluated in UTC. Some useful alternatives:

Pattern	When it runs
`FREQ=DAILY;BYHOUR=2;BYMINUTE=0`	Every night at 2 AM UTC
`FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR;BYHOUR=2;BYMINUTE=0`	Weeknights only (Mon-Fri)
`FREQ=WEEKLY;BYDAY=MO;BYHOUR=6;BYMINUTE=0`	Every Monday at 6 AM UTC

Why 2 AM? You want tests to run after the last deploy of the day has settled on staging, but early enough that failures are visible when engineers start work. Adjust to match your team’s timezone and deploy cadence.See the Create automation endpoint docs for all available fields.

Verify the first run and tune the prompt

After the automation fires for the first time, check the session to make sure Devin ran the tests correctly and the output matches what you expect.

Open the automation on the Automations page and follow the session link on its Activity tab
Did the Playwright suite execute? Were Linear tickets created for real failures (not flaky tests)?
Check the #qa-results Slack channel for the summary message

Common issues on the first run and how to fix them:

Devin can’t access staging: Add your staging environment variables (like STAGING_API_KEY or DATABASE_URL) as organization secrets so they’re available in every session the automation starts
Too many tickets from flaky tests: Add a retry to your playbook: “Re-run any failing test once before filing a ticket. Only file tickets for tests that fail twice.”
Tests take too long: Scope the suite — e.g., “Only run tests in tests/critical/ and tests/smoke/” — or increase the session timeout

Manage automations as code

Once your nightly run is stable, you’ll want to manage it alongside your other automations — pausing during deploy freezes, updating the prompt when your test suite changes, or spinning up a second automation for a different environment.Pause the automation during a deploy freeze or maintenance window:

curl -X PATCH "https://api.devin.ai/v3/organizations/$ORG_ID/automations/$AUTOMATION_ID" \
  -H "Authorization: Bearer $DEVIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

Re-enable it when the freeze ends:

curl -X PATCH "https://api.devin.ai/v3/organizations/$ORG_ID/automations/$AUTOMATION_ID" \
  -H "Authorization: Bearer $DEVIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

List all automations to audit what’s running:

curl "https://api.devin.ai/v3/organizations/$ORG_ID/automations" \
  -H "Authorization: Bearer $DEVIN_API_KEY"

For teams managing multiple automations, ask Devin to build a CLI that syncs automation definitions from a YAML config file — so you can version-control your automations alongside your test configuration: