---
status: active
owner: platform
last_reviewed: 2026-09-29
supersedes: []
---
# Disaster recovery — VM loss or full rebuild
<!-- wiki:featured -->

Use this when the Oracle VM is terminated, corrupted, or you are standing up a **new** production host from git + backups.

**Time estimate:** 2–6 hours depending on secrets vault readiness and backup download speed.

## What you can recover without the old VM

| Asset | Source |
|-------|--------|
| All product code & deploy scripts | GitHub `venture-studio` |
| Life graph (repos, decisions, obligations, chat history) | Nightly `astra_backup_*.sql.gz` on Koofr |
| Uploaded documents | `…/backups/astra-uploads/` on Koofr |
| TLS | Re-issue via Let's Encrypt after DNS points to new IP |

## What you cannot recover from git alone

- `.env` / `.env.local` files — restore from [offline vault](./SECRETS_MANAGEMENT.md)
- `rclone` config — reconfigure or restore `~/.config/rclone/rclone.conf` from vault backup
- Anything never backed up to Koofr

## Recovery phases

### Phase 0 — Before you need this (do now)

- [ ] Push all production commits to GitHub
- [ ] Fill [templates/SECRETS_OFFLINE_VAULT.template.md](./templates/SECRETS_OFFLINE_VAULT.template.md) in password manager
- [ ] Confirm nightly backups: `tail -20 /home/ubuntu/backup.log` (on current VM)
- [ ] Run quarterly restore drill — [BACKUP_AND_RESTORE.md](./BACKUP_AND_RESTORE.md)

### Phase 1 — New VM bootstrap

See [templates/NEW_VM_BOOTSTRAP.checklist.md](./templates/NEW_VM_BOOTSTRAP.checklist.md).

Summary:

```bash
# On fresh Ubuntu host
sudo apt update && sudo apt install -y git docker.io docker-compose-plugin nginx certbot \
  python3-pip nodejs npm rclone

git clone git@github.com:vthotac/venture-studio.git ~/studio
cd ~/studio
git checkout <production-branch>   # e.g. main or cursor/litellm-gateway-wire-astra
```

Restore secrets from vault into gitignored `.env` files (never commit).

### Phase 2 — Data plane

```bash
cd ~/studio/products/saas/astra-ai-cos
docker compose up -d postgres   # or full stack per product README
# Apply migrations
bash scripts/apply-all-migrations.sh
```

Restore database from Koofr — full steps in [BACKUP_AND_RESTORE.md](./BACKUP_AND_RESTORE.md#full-production-restore-destructive).

### Phase 3 — Deploy products

```bash
cd ~/studio/products/saas/astra-ai-cos
bash .devops/deploy-backend.sh
bash .devops/deploy-frontend.sh   # includes visual/E2E/a11y smokes when playwright installed
```

Repeat for other products per [DEPLOYMENT.md](./DEPLOYMENT.md).

### Phase 4 — Edge (DNS + TLS + nginx)

1. Point DuckDNS `studio-venky` to new VM public IP (legacy `astra-venky` redirects to studio-venky)
2. `sudo certbot --nginx -d studio-venky.duckdns.org`
3. Sync nginx: `bash dashboard/.devops/sync-nginx.sh` (canonical: `nginx-studio-venky.conf`)
4. Start studio launcher if used: `dashboard/.devops/install-production.sh`

### Phase 5 — Verification

```bash
cd ~/studio
./scripts/status-venture-stack.sh
bash products/saas/astra-ai-cos/.devops/smoke-test.sh
curl -sS -o /dev/null -w '%{http_code}\n' https://studio-venky.duckdns.org/astra/
```

Browser: Home → Needs you → Settings (trust/export) → spot-check one life area.

## Failure modes

| Symptom | Likely cause | Action |
|---------|--------------|--------|
| UI 502 | Frontend systemd down | `sudo systemctl status astra-frontend` |
| API 502 | Backend container down | `docker ps`; `deploy-backend.sh` |
| Empty life areas | DB not restored | Restore from latest `astra_backup_*.sql.gz` |
| Connectors broken | M365 OAuth secrets wrong | Re-enter Azure app creds in `.env` |

## Related

- [BACKUP_AND_RESTORE.md](./BACKUP_AND_RESTORE.md)
- [SECRETS_MANAGEMENT.md](./SECRETS_MANAGEMENT.md)
- [SYSTEM_INVENTORY.md](./SYSTEM_INVENTORY.md)
- Astra-specific drill: [products/saas/astra-ai-cos/docs/ASTRA_RESTORE_RUNBOOK.md](../../products/saas/astra-ai-cos/docs/ASTRA_RESTORE_RUNBOOK.md)