Common Workflows

These are real-world patterns for using the Carolina Cloud API and CLI to automate common tasks.

Ephemeral compute for batch jobs

Spin up an instance, run a job, tear it down. You only pay for the time the job runs.

#!/bin/bash
set -e

UUID=$(ccloud new container --cpus 16 --ram 64 --disk 200 \
  --flavor genomics --tier high-performance --name batch-job \
  | grep uuid | awk '{print $2}')

echo "Waiting for instance to start..."
sleep 30

ccloud ssh batch-job << 'REMOTE'
  cd /home/ccloud
  wget https://example.com/reference.fa.gz
  bwa mem -t 16 reference.fa reads_R1.fq.gz reads_R2.fq.gz | samtools sort -o aligned.bam
  aws s3 cp aligned.bam s3://my-bucket/results/
REMOTE

ccloud destroy "$UUID"
echo "Done. Instance destroyed."

Scale up, process, scale down

Use live resizing to temporarily increase resources for a heavy computation, then scale back.

INSTANCE_UUID="a1b2c3d4..."

# Scale up for training
curl -X PATCH "https://console.carolinacloud.io/api/instance/$INSTANCE_UUID/" \
  -H "Authorization: Bearer $CCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"n_vcpus": 32, "mem_gib": 128}'

# Run the job...

# Scale back down
curl -X PATCH "https://console.carolinacloud.io/api/instance/$INSTANCE_UUID/" \
  -H "Authorization: Bearer $CCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"n_vcpus": 4, "mem_gib": 16}'

Stop instances overnight

Save money by stopping instances when you are not using them. Storage is preserved.

#!/bin/bash
# Add to crontab: 0 19 * * 1-5 /path/to/stop-instances.sh

INSTANCES=$(curl -s -H "Authorization: Bearer $CCLOUD_API_KEY" \
  https://console.carolinacloud.io/api/instance/ \
  | jq -r '.[] | select(.state == "running") | .uuid')

for uuid in $INSTANCES; do
  curl -s -X POST "https://console.carolinacloud.io/api/instance/$uuid/stop/" \
    -H "Authorization: Bearer $CCLOUD_API_KEY"
  echo "Stopped $uuid"
done

Provision with S3 data pre-loaded

Create an RStudio instance with reference data already on disk when it boots.

curl -X POST https://console.carolinacloud.io/api/instance/ \
  -H "Authorization: Bearer $CCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "resource_type": "container",
    "flavor": "rstudioserver",
    "n_vcpus": 16,
    "mem_gib": 64,
    "disk_size_gib": 500,
    "name": "rstudio-with-data",
    "cpu_tier": "high-performance",
    "aws_access_key_id": "'"$AWS_ACCESS_KEY_ID"'",
    "aws_secret_access_key": "'"$AWS_SECRET_ACCESS_KEY"'",
    "aws_endpoint_url": "https://s3.wasabisys.com",
    "aws_prefill_bucket": "my-bucket/reference-genomes/"
  }'

When the instance is ready, the reference data is in ~/s3-prefill/ and aws s3 is pre-configured.

Poll for instance readiness

Wait for an instance to finish provisioning before proceeding.

#!/bin/bash
UUID="a1b2c3d4..."

while true; do
  STATE=$(curl -s -H "Authorization: Bearer $CCLOUD_API_KEY" \
    "https://console.carolinacloud.io/api/instance/$UUID/" \
    | jq -r '.state')

  if [ "$STATE" = "running" ]; then
    echo "Instance is ready."
    break
  elif [ "$STATE" = "error" ]; then
    echo "Instance failed to provision."
    exit 1
  fi

  echo "State: $STATE. Waiting..."
  sleep 5
done

Open ports for a web service

Deploy a web application on a VM and expose it to the internet.

# Create the VM
ccloud new vm --cpus 4 --ram 8 --disk 50 --name web-server

# Open HTTP and HTTPS ports
curl -X POST "https://console.carolinacloud.io/api/instance/$UUID/add-port/" \
  -H "Authorization: Bearer $CCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"port": 80}'

curl -X POST "https://console.carolinacloud.io/api/instance/$UUID/add-port/" \
  -H "Authorization: Bearer $CCLOUD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"port": 443}'

Your service is now accessible at http://<public_ip> and https://<public_ip>.