Migration

The Migration tool copies data from an existing source into your Carolina Cloud bucket. Two source kinds are supported:

• S3-compatible — AWS S3, Wasabi, Backblaze B2, Cloudflare R2, DigitalOcean Spaces, MinIO, self-hosted Ceph. Anything that speaks the S3 API.
• SFTP — any Linux server reachable over the public internet. Password or SSH-key authentication.

The copy runs entirely on our infrastructure. You fill out a short form, hit Start migration, and close the tab. We’ll keep going overnight, across reconnects, and through the multi-terabyte mark without you doing anything else.

Where to find it

Dashboard sidebar → Storage → Migrate.

What it copies

• S3-compatible sources — a whole bucket, a single prefix inside one, or a single object key.
• SFTP sources — any directory (and its subtree) on a Linux server.
• Up to one job at a time per account. Queueing a second while the first is live returns a 409.

What it can’t copy (yet)

• Objects in AWS Glacier / Deep Archive. Glacier objects need to be restored to standard S3 before they’re readable, and the tool doesn’t yet issue restore requests for you. If you try to migrate a bucket with Glacier objects mixed in, the copy will fail with Object in GLACIER, restore first. For now: restore the objects via the AWS Console or aws s3api restore-object, wait for the restore window to open, then start the migration. First-class Glacier source support is on the roadmap.
• Sources that aren’t S3-compatible or SFTP. Google Cloud Storage’s native API, Azure Blob, and plain FTP aren’t supported. (GCS has an S3 interop mode that works via the custom-endpoint option. For FTP servers, installing OpenSSH on the source is typically a one-liner — apt install openssh-server — and gives you a working SFTP target.)

Step-by-step

1. Choose a source kind

At the top of the Source section there’s a tab strip: S3-compatible or SFTP. Pick the one matching where your data lives.

2a. (S3 sources) Pick a provider and enter credentials

The Provider dropdown has presets for the common cases — AWS S3 (eight regions), Wasabi (two regions), Backblaze B2. Each preset fills in the endpoint URL and the region automatically.

If your source isn’t in the list, choose Custom endpoint… and fill in:

• Endpoint URL — e.g. https://<accountid>.r2.cloudflarestorage.com for Cloudflare R2, https://nyc3.digitaloceanspaces.com for DigitalOcean Spaces, or your MinIO/Ceph hostname.
• Region (optional) — required for AWS, Wasabi, Backblaze B2, and DigitalOcean Spaces, all of which sign requests with SigV4 and reject signatures from the wrong region. Leave blank for Cloudflare R2, MinIO, or anything else that accepts us-east-1 by default.

Then:

• Access key and Secret key — a pair of S3 credentials with at least s3:ListBucket and s3:GetObject on the source bucket. Read-only is sufficient.
• Bucket — the source bucket name.
• Prefix (optional) — a folder (photos/) or a single object key (report.pdf). Leave blank to copy the whole bucket.

The secret key is forwarded once to the worker, then discarded. It’s never written to the database, and the worker wipes its on-disk config when the job finishes.

2b. (SFTP sources) Enter server details and pick auth

Many bio/research customers paste an sftp:// URL from their PI or HPC docs without knowing how to break it into form fields. For sftp://researcher@hpc.lab.edu:2222/datasets/run-2026-05:

Form field	Value
Host	hpc.lab.edu
Port	2222 (leave blank for the default 22)
Username	researcher
Path	/datasets/run-2026-05

Field notes:

• Host — just the hostname or IP. No sftp:// prefix.
• Path — absolute path on the source server, usually starting with /. If your migration says “path not found”, try the home-relative form /home/<your-user>/<folder>. See Chrooted SFTP paths below for the chroot caveat.

Then pick Authentication: Password or SSH key.

Password authentication

Type the password. Same handling as S3 secrets — forwarded once to the worker, never persisted.

Many institutional SSH servers (university clusters, NIH HPCs, anything with a sysadmin) disable password auth at the server level and only allow SSH keys. If your password is correct but the migration fails with “Username or password rejected”, that’s the most likely cause — switch to SSH-key auth.

SSH key authentication

We generate an ED25519 keypair for you and show you a snippet to paste into ~/.ssh/authorized_keys on the source server. Your private key never leaves Carolina Cloud. This is strictly safer than asking you to upload your own private key.

The snippet looks like:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... carolinacloud-migration-<your-user-id>

Steps:

1. On the source server, append the snippet to ~/.ssh/authorized_keys for the user you entered in the Username field. A safe one-liner:

   echo '<snippet>' >> ~/.ssh/authorized_keys
   chmod 600 ~/.ssh/authorized_keys
   chmod 700 ~/.ssh

2. Back in the form, check I pasted this line into authorized_keys.
3. Click Start migration.

Lifecycle:

• On successful migration, we destroy our copy of the private key. The line in your authorized_keys becomes useless — safe to remove. Look for the unique tag carolinacloud-migration-<your-user-id> (one per Carolina Cloud user).
• On failed or cancelled migration, we keep the keypair so retries don’t force you to paste a fresh line.
• After 14 days of disuse (no migrations dispatched with the key), we wipe it automatically. Your next migration generates a fresh pair.

3. Pick the destination

• Bucket — the Carolina Cloud bucket the files will land in. If you belong to an org with a provisioned bucket and you’re an admin or the org owner, both your personal bucket and the org bucket appear. See Who can migrate to the org bucket below.
• Prefix (optional) — auto-mirrors whatever you typed for the S3 source prefix until you edit it (SFTP migrations start with an empty default). Files land at <prefix>/... inside the destination bucket.

Single-file copies

If you’re copying a single S3 object, both source and destination prefix must end without a trailing slash. With a trailing slash the file lands inside a folder named after itself (report.pdf/report.pdf).

4. Start migration

Click Start migration. The button shows Validating source credentials… for a beat while we run a preflight against the source — this catches bad credentials, the wrong endpoint, a missing bucket/path, or an empty source before any worker is even involved.

If the preflight passes, the page swaps into the active-job view.

During a migration

The active-job card shows:

• Elapsed time — ticks live, second by second.
• Progress bar and percentage — based on bytes copied vs. source total size. The first sizing pass takes a beat for S3 and can take a few minutes for large SFTP trees (it walks every file). Until it finishes, the bar is indeterminate and the label reads Calculating total size…
• Data copied — bytes moved out of total bytes.
• Files copied — file count moved out of total files.
• Recent activity — collapsible tail of the worker’s log, refreshing every 30 seconds.

You can close the tab. The migration keeps running. When you come back to the page later (even days later), it remembers your last-viewed job and picks back up where it left off.

Cancelling

Hit Cancel migration to stop a job mid-flight. Anything already copied stays in your destination bucket — cancellation does not undo partial transfers.

When it finishes

You’ll land on one of three terminal cards:

• Migration complete — green check. Shows total bytes/files copied and the wall-clock duration. Files are in your destination bucket and ready to use. If you used SSH-key auth, the card reminds you to remove the corresponding carolinacloud-migration-... line from your source server’s authorized_keys — we’ve already destroyed our copy of the private key.
• Migration failed — red X. Shows the worker’s error message and exit code, plus the tail of the log so you can see what went wrong. Common causes: transient source-side network error, a source object in Glacier, or credentials being changed on the source mid-job.
• Migration cancelled — neutral. Confirms how many bytes/files made it through before you hit cancel.

All three give you a Start another migration button to launch a fresh job.

Common errors

Error	What it means	Fix
Source credentials rejected — InvalidAccessKeyId	The source provider doesn’t recognize the access key.	Most often a provider mismatch (e.g. AWS preset with Wasabi keys). Check the preset, then check for typos/whitespace.
Source credentials rejected — SignatureDoesNotMatch	Access key recognized; secret didn’t match.	Re-paste the secret. Make sure access key and secret are a matching pair.
Source credentials rejected — AccessDenied	Credentials valid but IAM policy doesn’t allow listing the bucket.	Grant s3:ListBucket and s3:GetObject on the source bucket.
Source bucket not found	No bucket by that name lives at that endpoint.	Check spelling. For AWS, make sure you’ve selected the bucket’s actual region.
No objects found at source bucket/prefix	Bucket and credentials work, but the prefix is empty.	Either pick a different prefix or leave it blank.
AuthorizationHeaderMalformed — region ‘X’ is wrong; expecting ‘Y’	The endpoint URL points at one AWS region but you signed for another.	Pick the matching AWS preset, or set the Region field under the custom-endpoint option.
Object in GLACIER, restore first	One or more objects in the prefix are in Glacier / Deep Archive.	Issue a restore via the AWS Console or aws s3api restore-object, wait for it to complete (3–48h depending on tier), then re-run the migration.
Couldn’t find that host (SFTP)	DNS resolution failed for the Host field.	Check the hostname for typos. Should be just the hostname or IP, no sftp:// prefix.
Nothing listening on that port (SFTP)	Server reachable, but isn’t accepting SSH connections on the port.	Confirm the SFTP service is running and the port (default 22) is correct.
SSH key wasn’t accepted (SFTP)	The source server rejected our SSH key.	Make sure the Username is right and that the snippet is in ~/.ssh/authorized_keys for that exact user. Also check ~/.ssh is mode 0700 and authorized_keys is mode 0600 — sshd refuses keys in over-permissive files.
Username or password rejected (SFTP)	SSH server rejected the credentials.	SSH won’t tell us whether the username or password was wrong — that’s by design. Double-check both. If you’re sure the password is right, the server may only accept SSH-key auth; switch the Authentication radio.
Source path doesn’t exist (SFTP)	The Path field doesn’t resolve on the server.	Try the absolute form, e.g. /home/<user>/<folder>. If your SFTP user is chrooted, see Chrooted SFTP paths.
Can’t read that path (SFTP)	We authenticated, but the user lacks read permission on the path.	Grant read access on the source, or use a user that already has it.
You already have a migration in progress	A previous job is still active on your account.	Wait for it to finish, or cancel it.

Performance

Each migration runs with a fixed per-job bandwidth cap. That’s a deliberate ceiling so a single migration can’t starve other customer workloads sharing the worker. A 1 TiB bucket typically finishes in roughly 1.5 hours of wall-clock time, network and source-side throttling permitting. Bigger transfers (10 TiB+) are perfectly fine to leave running overnight or over the weekend. If you require an expedited transfer, please contact support@carolinacloud.io.

SFTP migrations have two source-specific performance caveats — see Many tiny files and Huge single files below.

SFTP source quirks

A handful of SFTP-specific gotchas, most of which come up at HPC clusters and academic research sites.

MFA / 2FA on the source server

We can’t do MFA. There’s no clever workaround — MFA is designed specifically to require a human-present second factor at session establishment, and our migrations run unattended (we copy for hours or days with no human watching).

The standard fix is to ask your sysadmin to create a service account with SSH-key-only authentication that’s PAM-exempt from MFA. This is a routine setup at any institution that has unattended backups, monitoring, or automation — every cluster sysadmin has done it before. Then use that service account’s username in the form.

Kerberos / GSSAPI authentication

Same shape as MFA — we don’t speak Kerberos, and there’s no workaround on our side. Same fix: a service account that doesn’t require Kerberos for SSH login.

VPN-only or jump-host-only access

If your SFTP server is only reachable via institutional VPN, or only through a bastion / jump host, we can’t connect directly — Carolina Cloud’s worker isn’t on your VPN and we don’t support ProxyJump-style ssh tunneling.

Options:

• Ask your sysadmin to expose the source server publicly with our worker’s egress IP whitelisted. Email support for the current IP.
• Stand up a small publicly-reachable jump box that proxies SFTP to the internal server.

Chrooted SFTP paths

If your SFTP user is chrooted (ChrootDirectory set in sshd_config), the Path field is relative to the chroot, not the system root.

To check what your chroot looks like, log in interactively first:

sftp user@host
> pwd
> ls -la /

Whatever ls -la / shows is everything inside the chroot. If a normal sftp session lands you at / and you see a data/ folder there, then /data is the correct Path field value — not /home/<user>/data.

Symbolic links aren’t followed

The SFTP backend lists symlinks but doesn’t follow them. If your source data layout uses symlinks (common in genomics for pointing at shared reference genomes), those files won’t be copied through the symlink. Either resolve the symlinks to real files before migrating, or migrate the shared target directly as a separate job.

Per-user SSH session limits

We open up to 16 parallel SSH connections per migration. Some HPC sshd configs cap MaxSessions at 10 or lower per user. If you see staggered Connection refused errors in the log tail during a migration, your server is hitting that cap.

Options:

• Ask your sysadmin to raise MaxSessions for the migration user.
• Email support — we can dispatch your migration with reduced parallelism.

Sequencer drop folders and live data

Don’t migrate while a sequencing run (Illumina BCL, Nanopore fast5, PacBio HiFi) is still writing. Our copy captures whatever state it sees mid-flight, which can mean truncated or zero-byte files. The safe pattern: wait for the run to finish, then start the migration.

The same applies to any directory being actively written by another process. If file sizes change mid-read, rclone retries — and a busy write loop can spin retries indefinitely.

Many tiny files

SFTP is high-latency per file. Migrating millions of tiny files (10x Genomics raw output, BCL tiles, FAST5 per-read files) can take days regardless of total byte count, because each file pays a full SSH round-trip.

If you can, tar the directory on the source first and migrate the single archive:

tar czf data.tar.gz data/

You can untar it on a Carolina Cloud instance after the migration. The total wall-clock can drop from days to hours.

Huge single files

SFTP doesn’t support multipart download. A single 100 GiB+ file copies in one TCP stream — bandwidth-limited per file, no parallelism to apply. Plan accordingly for very large monolithic files.

Security and privacy

• Source secrets are never persisted. Passwords and S3 secret keys are passed through to the worker for the duration of the job, then wiped from disk.
• SSH private keys are destroyed on success. For SFTP migrations using SSH-key auth, we wipe our copy of the private key as soon as the migration succeeds. The matching authorized_keys line on your server becomes useless at that point — safe to remove.
• The destination is always your Carolina Cloud bucket. Carolina Cloud does not migrate data into a third-party destination; the destination dropdown is restricted to the buckets you own or admin.
• The copy is server-to-server. Bytes flow directly from your source to your destination bucket via our worker. They don’t traverse your laptop or your local internet connection.

Who can migrate to the org bucket

If your account is in an org and the org has a bucket provisioned, both your personal bucket and the org bucket show up in the destination dropdown. Only org admins (the owner and anyone the owner has promoted) can actually dispatch a migration into the org bucket. A regular member who picks the org bucket from the dropdown will get a 403 from the dispatch endpoint.

Members can always migrate into their own personal bucket — no admin role required.

See Organizations → Overview for the full breakdown of org roles.