allstarr/docs/operations/storage.md

Storage operations

This runbook covers the durable storage behavior shipped with Allstarr's new baseline. It is written for the standard Compose deployment, which runs Allstarr with Postgres and Valkey. SQLite is available for an explicitly configured local or custom deployment, but the standard Compose file does not switch between databases.

What Postgres stores

Postgres does not store song audio. Songs stay as normal files in the media folders that Allstarr, Jellyfin, Navidrome, and your own backup tools can reach.

The standard Compose mounts are:

Data	Container path	Default host or volume location
Downloaded songs	/app/downloads	${DOWNLOAD_PATH:-./downloads}
Kept or favorited songs	/app/kept	${KEPT_PATH:-./kept}
Durable database	Postgres data directory	postgres-data named volume
App state and database backup artifacts	/app/state	allstarr-state named volume
Rebuildable app cache	/app/cache	allstarr-cache named volume
Valkey cache data	/data	valkey-data named volume

The database stores application state such as users, backend identities, provider accounts, encrypted secret versions, durable jobs, outbox events, provider health, matches, playlist links, recommendation state, audit events, and the backup catalog. An audio file still belongs in a configured media root. A database row can point at a song. It does not contain the song.

A database backup does not include /app/downloads, /app/kept, the secret key ring, or unrelated files under /app/state. Back up those items separately when they matter to your recovery plan. Valkey and the two cache locations are not authoritative state.

Select one database on purpose

Allstarr accepts only Postgres or Sqlite in Storage:Provider. It does not fall back from an unavailable Postgres database to a new SQLite file, or the other way around. If the selected database is unavailable or has a pending migration, readiness fails and state-changing requests are rejected. A bounded runtime probe keeps checking the selected database after startup. Readiness, mutations, durable jobs, and outbox delivery pause when connectivity or schema compatibility is lost, then resume after the same database returns with the current schema.

Standard Compose: Postgres

The checked-in docker-compose.yml explicitly sets:

Storage__Provider=Postgres
Storage__ConnectionString=Host=postgres;Port=5432;Database=...;Username=...;Include Error Detail=false
Storage__PasswordFile=/run/secrets/postgres_password

Changing an unrelated .env value cannot silently select SQLite. The Postgres password is read from the mounted secret file and is not placed on a process command line.

Local or custom deployment: SQLite

appsettings.json defaults to SQLite for manual development, using /app/state/allstarr.db. Allstarr will not create a missing SQLite file unless you provide a one-shot confirmation file containing the exact text create-new-allstarr-database. This keeps a lost or unmounted SQLite volume from turning into a new empty installation.

For the first local run, select SQLite explicitly, use persistent absolute paths, and create the confirmation file yourself:

database=/absolute/path/to/allstarr.db
confirmation="${database}.create-confirmation"
mkdir -p "$(dirname "$database")"
umask 077
printf '%s\n' 'create-new-allstarr-database' > "$confirmation"

Storage__Provider=Sqlite \
Storage__ConnectionString="Data Source=$database" \
Storage__SqliteBootstrapConfirmationFile="$confirmation" \
Secrets__KeyRingPath=/absolute/path/to/allstarr-keyring.json \
dotnet run --project allstarr/allstarr.csproj

Allstarr deletes the confirmation only after the checked-in migrations finish and the schema passes verification. Do not recreate it unless you have deliberately selected a missing path and want a genuinely new database. An existing SQLite database, including a verified restore, does not need a confirmation file. If the database later disappears, readiness reports sqlite_database_missing and Allstarr opens SQLite in existing-file-only mode so normal requests cannot recreate it.

Storage:RuntimeProbeIntervalSeconds controls the bounded check cadence and defaults to 5 seconds. Storage:RuntimeProbeTimeoutSeconds also defaults to 5 seconds. Storage:PasswordFile is Postgres-only. The standard Compose file still selects Postgres even though the application default is SQLite. There is no supported SQLite Compose overlay in the repository right now.

Fresh standard install

Pre-overhaul runtime state is not imported automatically. Start the new durable baseline as a separate fresh install. This avoids carrying old Redis, cache, mapping, extension, or job formats into the new database and keeps the old stack available for rollback. Stop the old stack before the new deployment can write media, and give each version separate writable download, kept, cache, and managed-library roots. The existing backend library may be mounted read-only for matching, but the two versions must never write the same media roots concurrently.

After the new database is ready and the administrator signs in, the WebUI can preview an uploaded legacy .env. It imports allowlisted non-secret settings and creates eligible shared provider accounts in a disabled state. Deployment values remain a checklist, personal credentials must be reconnected by their owners, and playlist definitions remain ownership/target handoffs. The exact behavior is defined in the legacy .env import contract.

Run these commands from the repository root:

cp .env.example .env
mkdir -p secrets downloads kept
umask 077
openssl rand -base64 32 > secrets/postgres-password.txt
key="$(openssl rand -base64 32)"
printf '{"activeKeyId":"key-1","keys":{"key-1":"%s"}}\n' "$key" > secrets/allstarr-keyring.json
unset key
chmod 600 .env secrets/postgres-password.txt secrets/allstarr-keyring.json

Review .env before startup. In particular, set the backend, media-server URL, image tag, and host paths for DOWNLOAD_PATH and KEPT_PATH. Then validate and start the deployment:

docker compose config --quiet
docker compose pull
docker compose up -d
docker compose ps
curl --fail --silent --show-error http://127.0.0.1:5274/health/ready

Use the configured proxy address and port instead of 127.0.0.1:5274 if you changed them. A healthy response means the selected database, schema, required directories, and required secret key ring passed readiness.

A normal stop or recreation keeps the named volumes:

docker compose down --remove-orphans
docker compose up -d

To discard the new app state and perform another genuinely fresh setup, remove the named volumes too:

docker compose down --volumes --remove-orphans

That command deletes the Postgres, Valkey, app-state, and app-cache named volumes. It does not delete the bind-mounted downloads and kept folders or the files in secrets. Leave those media folders alone if you want to keep the songs. Regenerate .env and both secret files only when you also intend to reset the deployment credentials and encrypted application secrets.

Schema migrations and the migration lock

Storage__AutoMigrate is true in standard Compose unless STORAGE_AUTO_MIGRATE says otherwise. On startup, Allstarr takes a database-specific lock before applying Entity Framework migrations:

• Postgres uses an advisory lock scoped to the current database.
• SQLite uses an exclusive <database>.migration.lock file beside the database.
• The default lock wait is 120 seconds. Storage:MigrationLockTimeoutSeconds accepts 5 through 1800 seconds.

Only the lock holder applies migrations. Other instances wait and do not become ready against a partially migrated schema. If the lock cannot be acquired, readiness reports migration_lock_unavailable.

With automatic migration disabled, Allstarr only checks connectivity and pending migrations. A pending migration leaves it unready with schema_migration_required. The offline storage command deliberately does not provide an unreviewed migration-only shortcut. Keep automatic migration enabled for the checked-in additive migrations, or run a reviewed application startup during a maintenance window.

Startup also compares every applied migration ID with the migrations known to the running image. If the database contains a migration from a newer or otherwise unknown build, Allstarr does not try to migrate over it. Readiness reports schema_version_unsupported. Use the image that owns that schema or restore a verified backup made for the image you are running.

Before an application upgrade, create and copy out a verified database backup. Keep the prior ALLSTARR_IMAGE tag recorded. Then upgrade:

docker compose pull
docker compose up -d
curl --fail --silent --show-error http://127.0.0.1:5274/health/ready

Do not treat a schema down-migration as rollback. Use a database restored into an isolated name together with the compatible application image, as described below.

Create and retain a verified backup

Sign in to the admin UI, open Settings, and select Create database backup. The authenticated admin endpoint is POST /api/admin/storage/backups, but the UI is the supported way to provide the existing admin session.

You can also create the same verified backup while the normal web host is stopped:

docker compose stop allstarr
docker compose run --rm --no-deps allstarr storage backup

The command prints one JSON object containing the artifact path, manifest path, SHA-256, schema version, and creation time. It does not start workers or the HTTP server.

Backup creation is synchronous inside the request even though the endpoint returns an accepted response. Before it records a backup as verified, Allstarr does the following:

• Postgres: creates a custom-format pg_dump, computes SHA-256, and checks the dump catalog with pg_restore --list.
• SQLite: uses SQLite's online backup API, computes SHA-256, runs PRAGMA integrity_check, and confirms that the complete migration history exactly matches the running image.
• Both: writes a versioned neighboring .manifest.json with the artifact name, provider, schema version, application version, checksum, and SecretKeyMaterialIncluded: false.

Restore treats that manifest as the source of truth for the backup, not as a note for people. Missing, repeated, unknown, or incorrectly typed fields are rejected. The provider, artifact filename, checksum, and schema must agree with the requested restore and the running image. The manifest identity, application version, creation time, and secret-material policy are also strictly validated before they enter the backup catalog. A manifest for an older or newer schema is rejected even when its checksum is valid. Keep the artifact and its manifest together.

Artifacts are written under /app/state/backups. Keeping the only backup in the same named volume is not enough. Copy the artifacts off the Docker host or into your normal backup system:

mkdir -p allstarr-backups
docker compose cp allstarr:/app/state/backups/. ./allstarr-backups/

List the in-container artifacts at any time:

docker compose exec -T allstarr find /app/state/backups -maxdepth 1 -type f -print

For an extra Postgres check before restore, choose the dump filename without its directory and run:

DUMP_NAME=allstarr-postgres-20260710T120000Z-replace-this-id.dump

docker compose exec -T -e DUMP_NAME="$DUMP_NAME" allstarr sh -lc '
  set -eu
  cd /app/state/backups
  expected="$(grep Sha256 "$DUMP_NAME.manifest.json" | head -n 1 | cut -d \" -f 4)"
  test "${#expected}" -eq 64
  printf "%s  %s\n" "$expected" "$DUMP_NAME" | sha256sum -c -
  pg_restore --list "$DUMP_NAME" >/dev/null
'

The runtime image contains pg_dump, pg_restore, and psql. The database password stays in /run/secrets/postgres_password and should be passed through PGPASSWORD, never as a command-line argument.

Restore Postgres into an isolated database

Never run pg_restore --clean against the live database. Restore into a new database name, inspect it, and only then switch Allstarr to it. The following example assumes the default Postgres user allstarr. Set DB_USER to the value of POSTGRES_USER if you changed it.

If the selected dump only exists in your off-host copy, copy the dump and its manifest back first:

DUMP_NAME=allstarr-postgres-20260710T120000Z-replace-this-id.dump
docker compose cp "./allstarr-backups/$DUMP_NAME" "allstarr:/app/state/backups/$DUMP_NAME"
docker compose cp "./allstarr-backups/$DUMP_NAME.manifest.json" "allstarr:/app/state/backups/$DUMP_NAME.manifest.json"

Run the checksum and pg_restore --list check from the previous section. Record the SHA-256 emitted by storage backup, stop the normal host, and create a fresh target:

RESTORE_DB=allstarr_restore_20260710
DB_USER=allstarr
DUMP_SHA256=replace-with-the-recorded-64-character-hash

docker compose stop allstarr

docker compose exec -T -e RESTORE_DB="$RESTORE_DB" postgres sh -lc '
  set -eu
  createdb --username "$POSTGRES_USER" "$RESTORE_DB"
'

Use the offline restore command for the actual restore. It receives target credentials through an environment variable created inside the one-off container. The password does not appear in the process command line:

docker compose run --rm --no-deps \
  --entrypoint sh \
  -e DUMP_NAME="$DUMP_NAME" \
  -e DUMP_SHA256="$DUMP_SHA256" \
  -e RESTORE_DB="$RESTORE_DB" \
  -e DB_USER="$DB_USER" \
  allstarr -lc '
    set -eu
    password="$(cat /run/secrets/postgres_password)"
    export ALLSTARR_RESTORE_TARGET="Host=postgres;Port=5432;Database=$RESTORE_DB;Username=$DB_USER;Password=$password;Include Error Detail=false"
    unset password
    exec dotnet allstarr.dll storage restore-postgres \
      --artifact "/app/state/backups/$DUMP_NAME" \
      --sha256 "$DUMP_SHA256" \
      --target-connection-env ALLSTARR_RESTORE_TARGET \
      --confirm-destructive-restore
  '

The command prints "status":"verified" only after it has strictly parsed the manifest, matched it to the artifact and requested SHA-256, checked the Postgres dump catalog, completed pg_restore, opened the target, and confirmed that its complete migration history exactly matches the running image. It records the restore verification result in the backup catalog. Any restore or verification failure returns a nonzero exit code and does not report the target as verified.

Inspect the users, provider accounts, queued jobs, and audit state appropriate to your deployment. Confirm that the matching key-ring file is available before testing an encrypted provider credential.

For cutover, save the current configuration, change POSTGRES_DB in .env to the isolated database name, and recreate the services:

cp .env .env.before-storage-cutover
chmod 600 .env.before-storage-cutover
docker compose up -d
curl --fail --silent --show-error http://127.0.0.1:5274/health/ready

The original database is still present. Do not delete it until the restored deployment has passed your normal backend login, search, playlist, job, and provider-account smoke checks.

To roll back that cutover, restore the saved .env, pin ALLSTARR_IMAGE to the application version compatible with the original database, and recreate the services:

cp .env.before-storage-cutover .env
chmod 600 .env
docker compose up -d
curl --fail --silent --show-error http://127.0.0.1:5274/health/ready

For an application rollback after a forward schema migration, restore the pre-upgrade dump into another new database name and start the prior image against that database. Do not point the prior image at a database already migrated by the newer image unless that exact compatibility has been tested.

Restore SQLite with the offline command

The storage command verifies a SQLite artifact and restores it to an isolated file. It refuses an online restore over the active SQLite path. Stop the normal process first, retain the checksum printed when the backup was created, and choose a different target path:

Storage__Provider=Sqlite \
Storage__ConnectionString='Data Source=/srv/allstarr/current.db' \
dotnet allstarr.dll storage restore-sqlite \
  --artifact /srv/backups/allstarr-sqlite-replace.sqlite \
  --sha256 replace-with-the-recorded-64-character-hash \
  --target /srv/allstarr/restored.db \
  --confirm-target-offline

The neighboring .manifest.json is used by default. Pass --manifest <path> only when it was copied to a different name. A pre-existing target is rejected unless --overwrite is explicit. Allstarr verifies the restored temporary database against its checked-in migrations before moving it into the requested target path. The command reports verified only after that check and records the restore verification result in the backup catalog. After restore, start Allstarr with Storage:ConnectionString pointing at the isolated file, run readiness and smoke checks, and keep the original file for rollback. Do not copy a backup over a live allstarr.db.

Controlled SQLite and Postgres state transfer

Database-native dumps are for restoring the same provider. The code also has a provider-neutral state transfer format for a planned SQLite to Postgres or Postgres to SQLite move. It exports a checksummed zip of durable tables and imports only into a target explicitly confirmed to be empty.

The transfer has these hard rules:

• Writes must be quiesced before export.
• The source database must be ready.
• The target provider is selected explicitly and migrated before rows are imported.
• Import checks the artifact checksum and strictly parses the manifest before touching the target.
• The manifest must name Sqlite or Postgres, the current schema, and the exact Allstarr application version running the import.
• Unknown, repeated, missing, or incorrectly typed manifest fields and archive entries are rejected.
• Manifest provider, schema, creation time, and checksum metadata must agree with the requested artifact.
• Every durable target table must be empty. Import does not treat a database with only health, outbox, audit, or backup rows as empty.
• Encrypted secret bytes are transferred unchanged.
• Encryption key material, song files, caches, and unrelated app-state files are not included.
• A provider change is a cutover, never automatic failover.

The existing executable now exposes that offline export and import path. Stop every normal Allstarr instance that can write the source, then export into the durable state volume:

docker compose stop allstarr
docker compose run --rm --no-deps allstarr storage export \
  --output /app/state/transfers \
  --confirm-writes-stopped

Record the artifact path and SHA-256 from the JSON output. Configure the explicitly selected empty target database, keep normal Allstarr instances stopped, and import:

docker compose run --rm --no-deps allstarr storage import \
  --artifact /app/state/transfers/allstarr-state-replace.zip \
  --sha256 replace-with-the-recorded-64-character-hash \
  --confirm-empty-target

Import verifies the checksum, exact archive shape, manifest fields, source provider, schema, application version, and artifact metadata before it migrates the selected target. It then verifies the migrated schema and rejects the operation if any durable target table already contains rows. Use the same Allstarr image for export and import. Import never changes Storage:Provider for you. Use an explicit Compose override or manual environment for the new target, review docker compose config, and keep the source database unchanged for rollback.

If a transfer is implemented and used later, carry the original key ring through a separate protected channel. Validate encrypted provider credentials, row counts, job recovery, audit history, and /health/ready before cutover. Keep the source database unchanged for rollback.

Key-ring handling

The database contains encrypted provider-secret versions. The 32-byte keys that decrypt them live only in the external JSON key ring mounted at /run/secrets/allstarr_keyring.

• Keep secrets/allstarr-keyring.json owner-readable only. The application rejects a file that is group-readable, group-writable, other-readable, or other-writable on Unix.
• Back it up separately using encrypted, access-controlled storage. Do not place it next to general database dumps.
• A restored database needs the key IDs used by its encrypted secret versions. A healthy database alone does not prove that every provider secret can be opened.
• Never commit the key ring, Postgres password, .env, dumps, or copied state to Git.

The database password secret and encryption key ring solve different problems. Losing the Postgres password blocks database access. Losing a key-ring key permanently blocks decryption of every secret version that names that key, even if the Postgres dump itself restores cleanly.

Rotate active provider secrets

Take a verified database backup and a separate protected copy of the current key ring before rotation. Then add a new base64-encoded 32-byte key to the keys object and set its ID as activeKeyId. Keep the old key in the file. New or replaced secrets use the active key immediately.

Stop every Allstarr process that can write the database and run the bulk rotation command:

docker compose stop allstarr
docker compose run --rm --no-deps allstarr storage rotate-secrets \
  --confirm-writes-stopped

The command opens each active, non-revoked secret with its current key and creates a new encrypted version under the active key. Its JSON result includes the active key ID plus examined, rotated, and already-active counts. A missing old key, an undecryptable value, an unavailable database, or an incompatible schema makes the command fail instead of skipping that reference.

After a successful command, restart Allstarr, check readiness, and test each configured provider account:

docker compose up -d
curl --fail --silent --show-error http://127.0.0.1:5274/health/ready

Do not remove the old key yet. Retired secret versions and older database backups can still name it. Keep it for the full backup and rollback retention window, and take a new verified database backup with a separately protected copy of the new key ring before considering key retirement.

Recovery checklist

Before calling a restore or migration complete, verify all of the following:

1. The artifact checksum and provider-specific integrity check pass.
2. The restored schema has the expected migration history.
3. The matching application image starts and /health/ready succeeds.
4. Backend login resolves the expected Allstarr user and tenant.
5. Provider accounts can open their encrypted credentials without exposing them.
6. Pending and retrying jobs are present once, with no duplicate work introduced by the cutover.
7. Search, streaming, and playlist operations can still reach the bind-mounted song folders.
8. The original database and prior image remain available until the rollback window closes.
9. Database artifacts, media folders, and the key ring each exist in the separate backup location intended for them.