Skip to content
Carolina Cloud Docs

Nextflow on Carolina Cloud

Carolina Cloud ships a native Nextflow executor called ccloud. Pipelines written for Nextflow (nf-core/rnaseq, nf-core/sarek, and anything else) run against swappable executors — local, awsbatch, slurm, k8s — and ccloud is ours. You point your pipeline at it, and tasks run on an elastic pool of Carolina Cloud containers instead of a fixed VM or a cloud batch service.

The mental model: head + pool workers

Section titled “The mental model: head + pool workers”

A Nextflow run has two roles, and they run on different machines:

  • The head container is the driver. It runs the nextflow process (the JVM) plus the nf-ccloud plugin. It builds the DAG, submits and monitors tasks, publishes your --outdir, and stages any HTTP/FTP inputs. It does not run your pipeline’s tools — it just orchestrates, so it stays small.
  • Pool workers run the tasks. These are Carolina Cloud containers the executor spins up on demand. Each task (FASTQC, STAR_ALIGN, SALMON_QUANT, …) is dispatched into a worker sized to that task.

You create one head, open its terminal, and run nextflow run … from inside it. Everything else — allocating workers, sizing them, tearing them down, billing — is automatic.

What makes it fast

Section titled “What makes it fast”

Two things no other Nextflow backend does at task granularity:

  • Resize, don’t recreate. Instead of destroying a container when a process finishes and cold-starting a new one for the next stage, the executor resizes the same container in milliseconds (docker update) for the next task. A 16 → 64 → 8 vCPU chain becomes one worker resized twice — no cold starts between stages.
  • Persistent local scratch + locality. Each worker keeps a persistent local-NVMe /scratch and a keep-folder cache across tasks. When a downstream task lands on the worker that already holds its inputs, they’re hard-linked in with zero network transfer instead of re-downloaded from object storage.

The result is a warm pool that’s reused across the whole run — in one benchmark, 65 workers served 214 tasks — with per-task overhead a fraction of a cold-start batch service.

Benchmark

Section titled “Benchmark”

Coming soon..

Why it’s cheaper

Section titled “Why it’s cheaper”
  • You pay for what runs. Pool workers are billed only while they’re doing work (they shrink to an idle floor between tasks and are reaped when the pipeline is idle). No cloud markup on top of raw compute.
  • Cheap object storage without re-hosting references. Your work dir can live on inexpensive storage (e.g. Wasabi) while your pipeline transparently pulls reference data (iGenomes, GATK bundle) from public AWS — no need to copy references into your own bucket. See Storage & buckets.
  • Exact cost per run, at runtime. Because every worker is metered per-run at a flat rate for its current size, you can see what a pipeline cost as it runs — something managed batch services can’t show you inline.
  • No egress fees from network storage. One of our core features, prevents ballooning charges as data is pulled from object storage to compute containers.

What’s supported at launch

Section titled “What’s supported at launch”
  • Multi-host per run. A run’s pool is placed across multiple hosts to maximise capacity, with smart locality logic keeping related tasks and their data together — see Resources & scaling.
  • DSL2 pipelines (Nextflow’s current standard). DSL1 is not supported.
  • S3-compatible work dir. The work dir must be on an S3-compatible endpoint (Wasabi, AWS, MinIO, Backblaze B2, …). gs:///az:// work dirs aren’t supported.

Next steps

Section titled “Next steps”