# HE Signs launch and recovery runbook

**Status: locally tested customer-workspace and integration release 0.7; not a public production launch.**

Domain selected by the owner: **hesigns.homeequal.ai**. Changes remain local under the owner's existing instruction. No source has been pushed and no cloud resources, DNS records, real customers, payments or public endpoint have been created by this work.

See `LAUNCH-REVIEW.md` for the current release decision, `PRICING.md` for the selected commercial catalog and `aws/README.md` for the locally prepared dedicated AWS deployment. Public launch and cloud provisioning still require owner approval.

## Runtime separation

- `demo.mjs` remains the local file-backed sender workspace on port 4317. Its capability URL and demo keys are not production authentication.
- `production/server.mjs` serves `/account`, the authenticated `/workspace`, the database-backed v2 API, signer pages, developer center and embedded sessions. It does not expose the demo capability studio or v1 routes.
- `production/server.mjs --worker` drains the PostgreSQL delivery queue. Run it as a separately supervised service.
- Signed records, artifacts, immutable template revisions and outbox events live in PostgreSQL. Temporary integration schemas are isolated from the local workspace.
- The administrator CLI enrolls tenants, first credentials and a first-owner invitation. Personal accounts, verified email, password recovery, TOTP/recovery codes, team roles, session revocation, contacts, templates and the sender workspace are implemented. See `ACCOUNTS-BILLING.md`.
- Optional Stripe Checkout/customer portal, verified queued billing reconciliation, legal holds and delayed deletion, metadata export, Python SDK, historical proof keys, worker heartbeats and protected metrics are implemented and locally exercised.
- `production/local-workspace.mjs` is a separate loopback preview with an authenticated email-capture outbox. It sends no real email and makes no payments. Its private test-account access file is under ignored `.local-data/production-preview`; the preview entry point is excluded from the production image.

## Prepare hosting

Choose the hosting account, deployment region and managed PostgreSQL service. Configure encrypted storage and backups, TLS with certificate verification, firewall controls, a private database network, monitoring and a secret manager. The API needs outbound SMTP and the worker needs outbound HTTPS callbacks. Enforce an egress policy that denies internal networks and cloud metadata endpoints in addition to application URL/DNS checks.

The included Dockerfile and Compose/Caddy configuration describe a single-region API plus worker behind HTTPS. The app runs as a non-root user with a read-only filesystem. The database is external. Docker Desktop has recovered. The 0.7 Linux release now builds and passes non-root, read-only, network-disabled signing/proof, local OCR and OpenSSL seal checks. The final image verification is recorded in `VERIFICATION.md`. Node and Caddy images are pinned by registry digest. Deployment-host image scanning and deployment in the selected environment remain required. Do not copy `.local-data`, local signing keys or real demo data into the image.

## Configure and initialize

From the `he-signs` directory:

```sh
npm ci --ignore-scripts
node production/admin.mjs init-secrets production/secrets
```

The command refuses to overwrite an existing directory. It generates only the service key, link secret and outbox secret-encryption key. It does not generate or take custody of customer authorization private keys. Back up the service secrets in a separately controlled secret manager. Apply host permissions allowing only the app runtime to read them; container UID is 1000. File-based proof-key rotation now preserves historical public keys; KMS/HSM custody and rotation of link/encryption secrets still require additional work.

Copy `production/.env.example` to `production/.env` and fill in managed database and authenticated SMTP settings. Store the SMTP password in a mounted secret file. Production requires verified database TLS and SMTP TLS. Neither `rejectUnauthorized:false` nor `sslmode` URL overrides are accepted. Set `HE_SIGNS_PDF_FONT_FILE` only to a properly licensed font included through a read-only mount. The Linux image includes the open-source DejaVu font package; the AWS configuration selects DejaVu Sans. No licensed Windows font is redistributed.

Migrate using a migration role before starting the app. The runtime database role should have SELECT/INSERT/UPDATE/DELETE on HE Signs tables and sequence usage, without schema administration privileges. Apply `runtime-grants.sql` with psql variables `runtime_role` and `runtime_schema` after migrations. A local integration test exercises these grants with a role denied schema creation. Ensure the role has no inherited DDL permissions through PUBLIC or other roles. For local command execution, use absolute local secret paths in your environment file; `/run/secrets` paths are for the container.

```sh
node --env-file=production/.env production/admin.mjs migrate
node --env-file=production/.env production/admin.mjs create-tenant production/tenant.example.json production/credentials/customer-admin.json
```

Create the private `production/credentials` parent directory first. Replace the sample tenant file with the real company name, exact allowed embed origins and agreed monthly allowance. The administrator credential is written once to the output file. Keep it out of logs and browsers. Use `admin.mjs invite-owner` with a JSON file containing `tenant_id` and `email` to queue the first owner's invitation. They accept, verify their address and use personal access at `/account`. Further onboarding and role changes happen in the authenticated workspace. Public signup is optional and disabled by default.

The intended deployment commands, after approving hosting and DNS, are:

```sh
docker compose -f production/compose.yaml build
docker compose -f production/compose.yaml up -d
```

Point the domain to the selected server and let the edge obtain its certificate. Never publish the development port or PostgreSQL port. Configure rate limiting at the edge: the app's anonymous limiter is per process. Customer API and account limits are enforced in PostgreSQL. The app trusts forwarded client addresses only from explicit proxy IPs in `HE_SIGNS_TRUSTED_PROXIES`. Compose supplies a dedicated edge IP on a configured bridge subnet. Change both defaults if that subnet conflicts with the host. The included Caddy proxy ignores untrusted incoming forwarded headers by default; placing a CDN in front requires an explicit, reviewed trust configuration. [Caddy proxy documentation](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy)

## Billing activation

Configure the optional provider secret/webhook/catalog files, recurring prices, customer portal, tax settings and retry/dunning notifications as described in `ACCOUNTS-BILLING.md`. Owner-only checkout persists idempotency intents; callbacks are verified over raw bytes and deduplicated before a worker reconciles current provider state. No price is assumed from the LOS subscription discussion. The selected $9/$29/$99 catalog and a real Checkout journey passed in the Stripe sandbox; see `STRIPE-SANDBOX-ACCEPTANCE.md`. Live provider configuration, payment/tax policy review, hosted webhook delivery and public activation remain launch gates. No real-money payment was made.

## Readiness and acceptance

`/health/live` checks the API process. `/health/ready` checks schema/database reachability. `/health/delivery` checks worker heartbeat and delivery backlog, returning 503 when attention is needed. `admin.mjs readiness` reports worker count, backlog age, failed jobs and billing review/sync problems. `/internal/metrics` exports aggregate Prometheus gauges only with the configured bearer token; it returns 404 when disabled or unauthorized. `alerts.example.yaml` contains worker/backlog/billing alert rules. Connect them to the chosen monitoring system and support destination. Request logs omit URLs, bodies, email addresses, tokens and API keys.

Before real customers:

1. Run the local checks below, then repeat representative journeys in staging behind real HTTPS.
2. Send one owner-approved harmless agreement through the production SMTP account, confirm inbox receipt, sign on a different device, verify both completion emails and proof download. This turn sent no new real email.
3. Integrate a separately deployed sample customer app using only the published contract/SDK. The current browser test uses an in-repository consumer; it is not an independent third-party validation.
4. Verify secret handling, cross-tenant boundaries, credential revocation, origin restrictions, callback outage/recovery and proof tamper detection.
5. Repeat capacity tests in the deployment environment. The local 40-agreement, five-concurrent-journey check used twelve-page, 8,762-byte PDFs, completed without errors in 2.35 seconds, and measured 584 ms p95 proof-download time. This is a small local workload, not a production capacity/SLA claim. Commit-ordered events serialize an event-stream lock; PDFs are stored inline in PostgreSQL and rendering is not isolated in a resource-limited worker. Large scans, hostile PDFs, sustained load and real provider/network behavior remain untested.
6. Restore a real backup into an isolated recovery environment and verify document hashes, proof signatures, account access and queued job behavior. Record measured recovery time and data-loss window; no production RTO/RPO is established by local tests.
7. Complete an independent application security review and decide which customer jurisdictions and document categories can be supported, with appropriate contracts, privacy/retention handling and legal review. Do not market this as universally compliant, qualified, identity-verified or trustless.

## Backup and recovery

Use managed point-in-time recovery plus scheduled encrypted logical backups. Preserve tenant data, records, artifacts, templates, events, jobs and credential metadata together. Keep service/link/encryption secrets in a separately backed-up secret manager: database backup alone cannot recreate signing links or decrypt queued webhook secrets.

Restore to a **new isolated database**, with API and worker stopped. Validate record/artifact hashes and evidence using the preserved trusted public key. Verify the correct service secrets are available. Inspect leased jobs: they become retryable after lease expiry, so keep the worker off until recipients and callback destinations are confirmed. Restored credentials and sessions may have been revoked after the backup; reconcile revocations, rotate account API keys, invalidate restored embedded sessions and review private-link exposure before any public traffic. Do not silently restore old access to active customers.

Switch traffic only after acceptance checks. Keep the previous database intact for rollback. A backup can contain legally retained agreements and personal data; jurisdiction-specific automatic retention policies and per-tenant regional residency management are not implemented in this release.

The active database now supports explicit agreement deletion with a seven-day cancellation window and legal holds. Reapply later deletion decisions and holds after a restore, before traffic resumes. A restored backup must not silently resurrect signing links or erased documents. The application does not expire managed backups, erase external copies, choose jurisdiction-specific retention periods or verify the physical location of infrastructure. Regional access boundaries require separate operator-provisioned deployments. Do not roll an old 0.5 worker onto a 0.7 database containing account-mail/billing jobs; restore compatible code and data together.

## Proof-key rotation

Generate a new Ed25519 service key in the selected secret manager or a new protected key file. Preserve the previous **public** PEM and include it in the JSON array referenced by `HE_SIGNS_PREVIOUS_PUBLIC_KEYS_FILE`. Deploy the new active private key together with the complete public-key history to every API and worker instance. Keep the link secret unchanged during this operation so existing signing links remain valid.

`/verification-keys` publishes active/retired key IDs. `/verification-key.pem?key_id=...` fetches a specific public key. Proof kits choose the key ID recorded in their evidence; if its key is missing, the kit fails explicitly instead of bundling the wrong key. Old detached signatures remain unchanged. Test both an old and a newly completed agreement, and distribute fingerprints through a separately trusted channel. A compromised key requires incident handling; marking a key retired does not establish that earlier timestamps or signatures are trustworthy.

## Local verification commands

```powershell
$env:HE_SIGNS_TEST_DATABASE_URL='postgresql://hes_test@127.0.0.1:55439/postgres'
$env:HE_SIGNS_TEST_FONT_FILE='C:\Windows\Fonts\arial.ttf'
$env:HE_SIGNS_PG_BIN='C:\Program Files\PostgreSQL\18\bin'
node --test production/integration.test.mjs production/accounts.test.mjs production/billing.test.mjs agreementTemplates.test.mjs fieldPlacement.test.mjs envelopes.test.mjs flow.test.mjs sdk/server.test.mjs
node production/portal-browser-check.mjs C:\path\workspace-result.png
node production/embedded-browser-check.mjs C:\path\embedded-result.png
node production/load-check.mjs C:\path\local-load-report.json
node workspace-browser-check.mjs
node envelope-browser-check.mjs
npm audit --omit=dev
```

The PostgreSQL tests create randomly named schemas and a temporary NOLOGIN role for the documented permissions check; they remove their own test objects afterwards. All messages use test mailers; no customer email is sent. Python SDK checks require Python 3.10+ (`HE_SIGNS_TEST_PYTHON` can select its executable). Supply a licensed font covering Cyrillic for the configured-font test. The font is read for testing and is not redistributed.

## Remaining product and operational work

The locally implemented work now includes transactional signing, queued delivery, personal accounts/MFA/roles and workspace-enforced MFA, customer sender UI, contacts/templates, optional billing reconciliation, usage, embedded sessions, JavaScript/Python/.NET SDKs, delayed deletion/legal holds, metadata and batched proof-kit export, historical verification keys, worker health and recovery checks. Remaining engineering and external launch work must not be represented as delivered:

- Live Google Workspace and SCIM connector acceptance, plus formal lost-authenticator/account recovery operations. Organization-wide MFA enforcement is implemented; it protects sender accounts, not signer identity.
- Hosted sandbox webhook delivery, live provider billing acceptance, tax registrations, invoice/dunning/refund configuration and finance reconciliation with the operator's accounting system. The selected commercial plans passed a Stripe sandbox Checkout; live prices remain unpublished.
- Full sender/admin translation, professional review of the four signing languages and wider script/typography QA; physical regional deployments; jurisdiction-specific retention policies, backup/global-account/external-copy privacy operations. Background exports of up to 10,000 agreements and workspace privacy-case handling are implemented.
- Government identity verification, qualified signatures, long-term validation, notarization and country-specific acceptance. Email codes, optional PDF service seals and RFC3161 support are implemented; actual provider credentials and trusted certificate/authority acceptance remain required.
- Provider activation and independent customer acceptance of the new packet, attachment, delegation, conditional field, English OCR, bulk and .NET capabilities. Current limits and unsupported advanced routing are recorded in CAPABILITIES-0.7.md.
- KMS/HSM integration, link/encryption-secret rotation, append-only external audit storage, isolated PDF processing, hosted recovery drills, sustained/hostile load and independent penetration/accessibility testing, plus support/SLA operations.

Do not price or promise these unfinished capabilities as delivered. The proof architecture gives customers a useful exit path: their signed artifacts can be retained and checked outside HE Signs. Whether that is commercially disruptive still requires independent integrations and customer evidence.

## Validate configuration without launching

For a configuration-only check with placeholders, set `HE_SIGNS_ENV_FILE=.env.example` and run `docker compose -f production/compose.yaml --env-file production/.env.example config --no-env-resolution --quiet` from the HE Signs directory. Unset that override before deployment; the normal default is `production/.env`. Valid configuration does not establish that containers, DNS or the database are running.

## Release 0.7 activation and migration

Read [CAPABILITIES-0.7.md](CAPABILITIES-0.7.md) before rollout. Migrate all new Google, SCIM, email-code, batch, privacy and regional tables, then reapply runtime grants. Set `HE_SIGNS_DATA_REGION` to the actual dedicated deployment region. Existing tenant rows require reviewed location backfill; production refuses null or mismatched tenant regions. API and worker must run matching 0.7 code. Workers now process both delivery and background batch jobs. Monitor pending/failed batches and overdue privacy cases through protected metrics. Mount Google/certificate/TSA/font secrets read-only as documented; no production credentials are bundled.

The OCR child process has page, time and memory limits, but PDF parsing is not an OS-level isolation boundary. Validate the Linux native canvas/WASM dependencies in a rebuilt image and add infrastructure resource limits and a hardened document-processing boundary before hostile public uploads. Run real Google/SCIM and trusted certificate/TSA acceptance after the respective administrators configure them.