Storage & buckets

Nextflow reads inputs, keeps intermediates, and publishes outputs to object storage. Here’s how the ccloud executor handles each.

The work directory

Set workDir to an s3:// path on an S3-compatible bucket you own:

workDir = 's3://my-bucket/nf-work/'

This is where Nextflow keeps every task’s intermediate files. It must be S3-compatible — Wasabi, AWS S3, MinIO, Backblaze B2, etc. (gs:// and az:// work dirs aren’t supported). Intermediates are safe to delete after a successful run.

Credentials — the aws { } block

Your storage credentials go in aws { }. They’re independent of your Carolina Cloud token and are never stored by Carolina Cloud — they live only in your head for the run.

aws {
    accessKey = System.getenv('AWS_ACCESS_KEY_ID')
    secretKey = System.getenv('AWS_SECRET_ACCESS_KEY')
    region    = 'us-central-1'
    client { endpoint = 'https://s3.us-central-1.wasabisys.com' }
}

The endpoint is your provider’s — the example above is Wasabi’s us-central-1. For AWS S3, drop the client.endpoint line and set the right region.

Bring your own bucket

When you create a Nextflow head you can either take a Carolina Cloud pipeline bucket — created for you, billed on stored bytes only at a flat $20/TB/month — or bring your own S3-compatible bucket. Ours is the default; untick the Nextflow bucket option in the head-creation form to bring your own, and supply four things:

• the endpoint URL (e.g. https://s3.us-central-1.wasabisys.com)
• an access key and secret key
• the bucket name

We write-probe the bucket with those credentials before the head is created, so a typo, a missing bucket, or read-only keys fail immediately instead of halfway through your first run. The head is then set up exactly as it would be with ours: your bucket as the work directory, your credentials referenced from the environment rather than written into a readable file. They aren’t stored on our side — they live in the head for its lifetime.

Two things to weigh before pointing a pipeline at storage you already have:

• Minimum retention periods. Some providers — Wasabi among them — bill deleted objects for a minimum period of 90 days. Pipeline work dirs churn hard: one run can write and then delete terabytes of intermediates, and you keep paying for them long after they’re gone.
• Metered egress. Staging is read-heavy; a real run can pull 1–2 TB down to workers. Providers that meter egress past an allowance will charge for that on every run.

Neither applies to a Carolina Cloud pipeline bucket — no retention minimum, no egress charge. That’s the reason it exists.

Pipeline bucket pricing: a flat $20/TB/month

A Carolina Cloud pipeline bucket is billed at $20/TB/month on currently-stored bytes, prorated hourly — and that’s the entire bill. There is no minimum retention, no deleted-object charge, no egress fee, no API-request fee. $20/TB/month is what you pay, no matter what.

The rate is higher than our warm storage ($9/TB/month) specifically because it carries zero minimum retention. Pipeline work dirs churn hard — a run writes terabytes of intermediates and deletes them hours later — and under a retention policy like Wasabi’s 90-day minimum, that churn generates deleted-object charges that dwarf the headline per-TB rate. A pipeline bucket absorbs that: the moment you delete an intermediate, you stop paying for it. Combined with free egress on staging-heavy runs that pull 1–2 TB down to workers each time, the flat rate is the predictable — and usually cheaper — option for pipeline workloads.

Use a pipeline bucket for the churn (work dirs, outputs in flight); move results you’ll keep long-term to your warm or cold storage, which are cheaper per stored byte.

Your pipeline bucket is a folder in the head

A Carolina Cloud pipeline bucket is also mounted inside your head container at ~/pipelinebucket, so you can work with it using ordinary shell commands — no S3 client to configure:

ls ~/pipelinebucket/work                 # every task's work directory
cat ~/pipelinebucket/work/ab/cdef*/.command.err
cp ~/pipelinebucket/results/multiqc_report.html .

It’s the same bucket your run reads and writes, so anything a task publishes shows up here as it happens. Your personal and organization storage mount the same way, at ~/personalbucket and ~/orgbucket.

Bring-your-own buckets aren’t mounted — we don’t hold your credentials, so there’s nothing for us to mount with. Use aws s3 from the head with your own keys instead.

Pipeline references pull automatically — even from public AWS

Most nf-core pipelines pull reference data (genomes, indices) from public AWS — e.g. --genome GRCh38 resolves to s3://ngi-igenomes/… on AWS Open Data. That’s a different provider from your (say, Wasabi) work dir.

The executor handles this for you: for each input, it first tries your endpoint; if the bucket isn’t yours, it transparently falls back to fetching anonymously from public AWS. Each reference is downloaded once and cached for the rest of the run.

This is a real advantage — you get cheap object storage for your work dir and transparent access to AWS-hosted references, with no need to copy multi-gigabyte reference bundles into your own bucket first.

Inputs served over HTTP/FTP (GitHub test data, much of EBI/ENA) are handled by the head automatically as well.

Publishing outputs: the one endpoint rule

--outdir (and publishDir) are written by the head, which talks to a single S3 endpoint at a time.

Work dir and output dir must be on the same endpoint

Your workDir and --outdir must be on the same S3 endpoint (same provider and region). They can be different buckets on that endpoint, as long as your credentials reach all of them. You can’t, for example, keep the work dir on Wasabi and publish results to AWS S3 in the same run.

One exception to “different buckets”: the credentials we mint for a Carolina Cloud pipeline bucket are scoped to that single bucket, so if you’re using ours, keep workDir and --outdir inside it. Bring-your-own credentials that span several buckets can spread across them freely.

This is standard Nextflow/nf-amazon behavior, not a Carolina Cloud restriction.

What’s not supported yet

Not supported (today)	Notes
Private third-party buckets	The executor carries only your own credentials, so it can’t read someone else’s private bucket.
Public buckets on non-AWS S3 endpoints	Public references on a non-AWS S3 endpoint (some institutional MinIO/Ceph object stores) aren’t auto-detected. Uncommon for nf-core defaults.
Cross-provider publishing	See the endpoint rule above.
gs:// / az:// work dirs	Google Cloud Storage and Azure work dirs use different Nextflow filesystems.

Staging tooling

You don’t configure this, but for reference: workers stage data with a fast, parallel S3 client that saturates the network link, and it’s provided to every worker automatically — including off-the-shelf container images that don’t ship an S3 CLI. Nothing to set up.