Preparing a streaming migration: parallel validation and rollback planning

A successful streaming migration minimizes surprises and makes rollback easy. This guide walks through a parallel (shadow) migration pattern with concrete safeguards, verification steps, and failure modes to expect when you validate a new Nighthawk-based stack beside your current one.

For installation and configuration context, see /server-docs.

Why parallel validation beats a big-bang cutover

• You exercise real ingest, packaging, and playback paths without interrupting existing viewers.
• You can validate security controls (ingest auth, signed playback), recordings, and hooks end-to-end before traffic moves.
• If anything regresses, rollback is just switching the traffic route back to the current stack.

Prerequisites and environment setup

1) Network and ports

• Plan protocol parity: RTMP (encoders), SRT (encoders/contribution), HTTP for HLS, optional WebRTC, and RTSP for cameras if you use them.
• When you run Nighthawk via Docker, the compose configuration exposes configurable endpoints by environment variable with documented defaults: RTMP (1935), SRT (10080/udp), HTTP for HLS/HTTP-FLV (8080), WebRTC UDP (8000), and RTSP (8554). Align firewall and security groups around those bindings.

2) Feature gating awareness

• Certain services are disabled automatically when the licensed feature is absent (e.g., SRT and WebRTC blocks can be turned off by the runtime at startup). Validate that the features you intend to test are licensed and enabled before scheduling validation windows.

3) Decide the control-plane posture for validation

• Protect shadow ingest with a publish password so only your team can push to it.
• Require signed playback on the validation endpoints so only your testers (or your QA harness) can view them.

See /server-docs for the broader configuration and ports overview.

Configure safeguards for a safe shadow run

Enable these controls on the shadow stack before dual-publishing.

• Publish authentication (ingest control)
• Set the publish password environment variable. On startup, the server notes that stream authentication is enabled. Update your encoder stream key to include the password as a query parameter (e.g., mystream?password=YOUR_PASSWORD) so your publishes succeed.

• Signed playback (viewer-side authorization)
• Set the playback secret environment variable. On startup, the server enforces signed playback for HLS/HTTP-FLV and WebRTC. The runtime inserts the required context so playback hooks can validate tokens.
• Generate tokens in your backend (or with the helper) by signing stream key + expiry using HMAC-SHA256. Keep token TTLs short for test endpoints, and make sure your clocks are in sync to avoid premature expiry.

• Event webhooks (observability during testing)
• Configure the webhook URL to receive lifecycle JSON payloads for publish, play, and recording events. Delivery is fire-and-forget with a short timeout, so design your handler to return quickly; slow or unreachable endpoints may drop events during validation.

• Recording upload (shadow archive and parity checks)
• If you plan to compare recordings across stacks, configure the S3 upload variables (bucket and credentials). The uploader performs signed PUTs and retries a limited number of times. If all attempts fail, the local file remains. Optionally enable local deletion after a successful upload if you prefer cloud-only storage.

Dual-publish and shadow validation workflow

1) Stand up the shadow Nighthawk origin with isolated DNS/hostnames and the controls above. Keep it off your production CDN while you validate.

2) Configure encoders to dual-publish the same program feed to both origins (your current origin and the shadow Nighthawk origin). If encoders cannot dual-publish, consider a short scheduled test on a representative channel.

3) Validate playback privately against the shadow endpoints with signed URLs. Exercise HLS players and any WebRTC preview paths you rely on.

4) If you use ABR transcoding, verify the ladder configuration. The implementation supports specific named variants; unknown variants are skipped. If your license or plan doesn’t permit ABR, it will be disabled—validate expectations early.

5) Confirm that event webhooks and recording uploads behave as expected. Account for the webhook delivery model (no retries beyond the short timeout) and object-store availability; failed uploads retain local files by design.

Failure modes to expect (and how to spot them)

• Publish rejections on shadow: The publish password isn’t appended correctly to the encoder’s stream key. Fix the key formatting and retry; the server enforces this only when set.
• Playback 403s on shadow: Your playback URL isn’t signed, the token is expired, or the secret doesn’t match. Check TTL, clock skew, and signing inputs (stream key normalization).
• Missing webhook events: The handler took too long or was unreachable during delivery. Because forwarding is fire-and-forget with a short timeout, add logs on your endpoint and consider an internal queue if you need durable processing.
• Recording not uploaded: Bucket is set but credentials are missing, or object store is transiently unavailable. The runtime disables upload when credentials are absent; the uploader retries a few times and then keeps the local file for manual intervention.
• ABR variants missing: A rung name wasn’t recognized, or feature gating disabled the transcoder. Check logs for “unknown variant” or for ABR being disabled.
• Protocol listeners not reachable: Ports not exposed or blocked. Confirm Docker port mappings and host firewalls match your chosen NH_* port variables.

Cutover and rollback plan

• Prefer a CDN or DNS switch: Wire the new origin into your CDN configuration behind a separate origin entry. Flip the CDN origin on a low-risk window and monitor. Roll back by pointing back to the prior origin.
• Preserve old endpoints for a short overlap: Keep the former origin warm until you finish post-cutover validation (publishing, playback, webhooks, and recordings).
• Keep controls in place: Maintain publish authentication and signed playback on the new stack through the cutover to avoid unauthorized traffic bursts.

Verification checklist (actionable)

• Networking and features
• RTMP, SRT, HTTP(HLS), WebRTC, and RTSP ports are bound as expected and reachable from test clients.
• Desired features (e.g., SRT, WebRTC, ABR) are enabled and not gated off at startup.

• Ingest and playback controls
• Publish password set; encoder stream key includes the password; publish succeeds when correct and is rejected when incorrect.
• Playback secret set; unsigned requests fail; signed requests succeed within TTL.

• Events and recordings
• Webhook endpoint receives publish/play/DVR events and returns quick 2xx responses.
• Recordings appear locally; successful uploads land in object storage; failed uploads remain on disk for review.

• ABR and players
• ABR variants are present in manifests when enabled; unknown rung names are not emitted.
• Players can start, switch renditions, and seek within the DVR/recording context used in tests.

• Cutover/rollback
• CDN/DNS origin switch tested with a tiny audience slice or synthetic traffic.
• Documented rollback instructions (who, what, and where to flip) with access verified.

Where to read next

• Installation, ports, protocols, recording, APIs, and troubleshooting: /server-docs

Design your migration so validation is routine and rollback is a single change. With parallel shadowing, explicit controls, and the checks above, you can move confidently—knowing exactly how to revert if anything diverges from plan.