An observable, recoverable service
The UI can be healthy while MySQL or S3 is unavailable. Separate liveness from readiness and monitor business journeys as well as servers.
01
Monitoring
Build dashboards and alerts from available signals.
- /api/health: Node process liveness
- /api/ready: MySQL + S3 + latency
- Endpoint 4xx/5xx rate and latency
- Age of waiting/running/failed commands
- Issues by queue/status
- Open system logs and webhook failures
02
Backup and restore
MySQL and S3 form one logical recovery point.
- Daily MySQL backup and monthly restore test
- S3 versioning/snapshots and lifecycle policy
- Record business-owned RPO/RTO
- Restore database and bucket to compatible points
- Never delete migration history
- Verify evidence links after restore
03
Incident response
Contain impact first, then preserve evidence.
- Identify project, environment, and correlation/run/issue ID
- Revoke the affected scoped token
- Preserve logs, event time, and deployed revision
- Separate UI/API/MySQL/S3/AI/webhook failure
- Restore or roll back to a known release
- Document root cause and preventive action
04
Maintenance cadence
A simple schedule prevents risk accumulation.
- Weekly: failed tests/webhooks and open logs
- Monthly: restore drill and token/credential audit
- Every release: migration dry run and smoke/UAT
- Quarterly: dependency/image scan and access review
- Every SRS change: reconcile, impact, and revalidate