Configuration

Configuration for a ccloud run comes in three layers: what the head sets for you, what you must provide, and what you can optionally tune. This page walks each one, then covers run files, parameters, and the nextflow run command.

The three layers at a glance

Layer	What	Where it lives	You touch it?
1	Executor + access token	Baked into the head (global config)	No — automatic
2	Work dir + storage credentials	Your own config, per run	Yes — required
3	Pool, resource, and cache tuning	Your own config, per run	Optional

Running on your own machine instead of a head? You supply layer 1 yourself too — see Run from your own machine.

Layer 1 — baked into the head (automatic)

The head ships with a global Nextflow config that wires the executor and your access token. It’s applied to every nextflow run automatically — you never write or edit it:

plugins {
    id 'nf-ccloud@1.0.0'
}

process.executor = 'ccloud'

ccloud {
    apiKey   = System.getenv('CCLOUD_API_KEY')   // scoped token, injected for you
    endpoint = System.getenv('CCLOUD_ENDPOINT')  // the Carolina Cloud API
}

That’s the whole of what’s pre-set. Nothing about your data, resources, or pipeline is baked in — that’s all yours.

Layer 2 — what you must provide

Every run needs two things, which you set in your own config — a nextflow.config in your launch directory, or any file you pass with -c. Nextflow merges it with the head’s global config, so you don’t repeat the layer-1 lines.

// Required: the S3-compatible bucket for Nextflow's intermediate work
workDir = 's3://my-bucket/nf-work/'

// Required: credentials for that bucket. Never stored by Carolina Cloud.
aws {
    accessKey = System.getenv('AWS_ACCESS_KEY_ID')
    secretKey = System.getenv('AWS_SECRET_ACCESS_KEY')
    region    = 'us-central-1'
    client { endpoint = 'https://s3.us-central-1.wasabisys.com' }
}

See Storage & buckets for other providers and the work-dir / output-dir endpoint rules.

Layer 3 — optional tuning

Everything here has a default; set only what you want to change, in your own config.

Executor options — ccloud { }

Option	Type	Default	What it does
poolMax	integer	64	Most pool workers this run requests at once.
idleTimeout	duration	5min	How long an idle worker stays warm before it’s reclaimed. Raise (e.g. '20min') to keep the pool warm across bursty pipelines.
defaultMemGib	integer	4	Memory (GiB) for a task that declares no memory.
defaultDiskGib	integer	20	Scratch disk (GiB) for a task that declares no disk.
cacheBudgetGib	integer	20	Per-worker locality-cache budget, on top of scratch disk. Raise for large-file pipelines (WGS). See Resources & scaling.

apiKey and endpoint are also ccloud options, but they’re set for you on a head — leave them. (Running externally? Set them yourself with a pipeline token.)

// Example: keep the pool warm, and cache large intermediates for reuse
ccloud {
    idleTimeout    = '20min'
    cacheBudgetGib = 200
}

Here cacheBudgetGib = 200 raises each worker’s on-disk cache so large intermediate files (e.g. 100–150 GiB BAMs) stay on the worker that produced them and get re-used by the next task instead of re-downloaded from your bucket. See the locality cache for how that works and the pool-size trade-off it involves.

Per-process resources

How big each task’s worker is comes from that process’s Nextflow resource directives — the executor sizes a worker to fit. Set them in your pipeline or in config:

process {
    cpus   = 2          // default for every process
    memory = 8.GB

    withName: 'STAR_ALIGN' {   // override one process by name
        cpus   = 12
        memory = 36.GB
        disk   = 100.GB
    }
}

Undeclared directives fall back to the ccloud defaults above (cpus → 1, memory → defaultMemGib, disk → defaultDiskGib). See Resources & scaling and Nextflow’s process directives.

Run files & parameters

None of this is Carolina Cloud-specific — it’s standard Nextflow, and it’s how you drive any pipeline (nf-core or your own).

Config vs. params — the key distinction

Config (-c files) controls how Nextflow runs — executor, resources, credentials. Parameters (--x / -params-file) are inputs to the pipeline — samplesheets, references, thresholds. Your Carolina Cloud settings are always config; your data is usually params.

Config files and profiles

Nextflow merges config from several sources, later winning: the pipeline’s own nextflow.config, a nextflow.config in your launch directory, and any files passed with -c. That’s why you can keep your Carolina Cloud settings (workDir, creds, tuning) in one file and layer it onto any pipeline:

nextflow run my-pipeline -c ccloud.config -plugins nf-ccloud@1.0.0

Profiles (-profile) are named config bundles a pipeline defines (e.g. test, docker); select one or more. See Nextflow’s configuration and config profiles.

Pipeline parameters

Pipeline parameters are inputs the pipeline itself defines — they vary per pipeline, there’s no fixed list. Pass them on the command line with a double dash, or in bulk from a file:

# individual params on the command line
nextflow run my-pipeline --input samples.csv --outdir s3://my-bucket/out/

# ...or from a YAML/JSON params file
nextflow run my-pipeline -params-file params.yaml

params.yaml

input:  samples.csv
outdir: s3://my-bucket/out/
genome: GRCh38

--outdir is a common convention for where results are published, but the exact parameters belong to the pipeline, not to Carolina Cloud. See Nextflow’s CLI reference for -params-file and pipeline params.

The nextflow run command

NXF_SYNTAX_PARSER=v1 nextflow run <pipeline> \
  -plugins nf-ccloud@1.0.0 \
  -c ccloud.config \
  -profile <profile> \
  -r <version> \
  -params-file params.yaml

The options you’ll reach for most:

Flag	Purpose
-plugins nf-ccloud@1.0.0	Required. Forces the executor plugin to load. Without it, nf-core pipelines’ own plugins{} block shadows ours → the run fails with Unknown executor name: ccloud.
-c <file>	Layer in a config file (your Carolina Cloud settings).
-profile <name>	Select the pipeline’s named config profile(s).
-r <version>	Pin the pipeline revision/tag (for pipelines pulled from Git).
-params-file <file>	Supply pipeline params from YAML/JSON.
--<name> <value>	Set a single pipeline param.
-resume	Reuse cached results from a previous run (same workDir).
-with-report · -with-trace · -with-timeline	Emit an HTML report / task TSV / Gantt timeline.

And one environment variable:

NXF_SYNTAX_PARSER=v1	Selects Nextflow’s legacy config/DSL parser (keeping the modern runtime). Needed for pipeline releases whose config uses older syntax that Nextflow 26’s strict parser rejects. Per-pipeline — omit on newer tags; add it if a run fails to parse.

Full reference: Nextflow’s run command.

Fresh runs vs. -resume

Nextflow reuses cached task results only with -resume (matched against the workDir). For a clean run — or a benchmark — use a fresh workDir and omit -resume, or tasks come back CACHED and don’t actually execute.

Supported Nextflow stack

Validated on Nextflow 26.04.4 with nf-amazon 3.9.2 (both baked into the head). If you run your own Nextflow, keep nf-amazon at the version your Nextflow bundles by default — forcing a mismatched version fails at startup.