Running a pipeline on AWS

Once your AWS account and an S3 bucket are connected, running on AWS is one choice in the normal launcher — everything else is identical to a Carolina Cloud run.

Launch

In the pipeline launcher, the Compute toggle at the top picks the backend:

• Carolina Cloud — the managed fleet (the default).
• AWS — your own account.

Pick AWS and launch as usual. The toggle is only selectable once both setup steps are done; until then it shows what’s still needed and links back to Getting started.

The output location defaults to your linked S3 bucket (s3://your-bucket/runs/<run-name>/), and the pipeline reads and writes entirely within it.

What happens

1. The head starts on Carolina Cloud and begins your run.
2. A worker pool is provisioned in your AWS account — this is the one time cost worth knowing about: a fresh node takes ~90 seconds to boot and join on the first launch. The head waits for it; it isn’t an error.
3. Tasks run on your node(s), reading and writing your S3 bucket in-region.
4. On completion, failure, or cancel, the pool and its nodes are torn down automatically. AWS bills you only for the time the worker nodes ran; Carolina Cloud bills only the small head container, not the workers.

Watching a run

A BYOC run appears in the console like any other — the live DAG and run view work exactly the same. Inputs and outputs are browsable in your linked bucket.

Good to know

• Node sizing is automatic. Carolina Cloud provisions a node large enough to hold the pipeline’s heaviest task and pack many smaller ones alongside it. A node may look over-provisioned early on — that’s intentional; the small tasks fill it as the run progresses.
• Idle workers are reclaimed during the run so capacity recycles for later stages — you don’t pay for workers sitting idle for long.
• One pool per run. Pools aren’t shared between runs; each run gets its own single-tenant compute and gives it back when done.

Troubleshooting

• The AWS toggle is greyed out. One of the two setup steps is incomplete — the toggle names which (compute environment or linked S3 bucket). An org admin completes it under Admin → Link AWS / Storage → Link an S3 bucket.
• A run fails immediately with a parameter error. That’s the pipeline rejecting its inputs (e.g. a genome reference not specified) before any compute — unrelated to AWS. Fix the launch parameters and relaunch.
• A run sits with no tasks starting for a minute or two at the start. Normal first-launch node bring-up (~90s). If it persists far longer, check that the linked bucket’s key still has read/write access.