The Trinity Beast Infrastructure — CloudWatch Dashboard & Alarm Notifications

Live cost intelligence dashboard that combines resource utilization metrics with cost context. Rather than embedding stale dollar figures, it shows the metrics that drive spend — ECS CPU/Memory utilization across all 4 services, Aurora ACU + utilization (with min/max annotations), ElastiCache CPU/memory, Lambda invocations and duration for all 8 functions, Aurora connections, and NAT Gateway/EC2 data transfer. Two cost-context tables explain the monthly baseline by component (with unit costs) and identify the cost levers you can control. Direct links to Cost Explorer, Budgets, and Savings Plans for exact dollar figures.

Replaces: The previous static Trinity-Beast-Cost-Executive-Dashboard and Trinity-Beast-Cost-Detailed-Dashboard (deleted 2026-05-30) — both were 100% hardcoded markdown text frozen at April 2026 figures, referencing services no longer in use.

┌─────────────────────────────────────────────────────────────────────────┐
│                      NOTIFICATION ROUTING                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  CloudWatch Alarm (any of 21 alarms)                                    │
│    ├─→ SNS: tbi-ops-notifications                                       │
│    │     └─→ tbi-ops-notify Lambda                                      │
│    │           └─→ Formatted HTML email (SES)                           │
│    │                 → CoryDeanKalani@CPMP-Site.org                      │
│    │                                                                    │
│    └─→ EventBridge: tbi-ops-alarm-trigger                               │
│          └─→ Step Function: tbi-ops-health-check-heal                   │
│                └─→ Self-heal → verify recovery → notify                 │
│                                                                         │
│  GuardDuty Finding (severity ≥ 7)                                       │
│    └─→ EventBridge: tbi-ops-guardduty-high-finding                      │
│          └─→ tbi-ops-bedrock-analyze Lambda                             │
│                └─→ AI threat assessment → auto-action → notify          │
│                                                                         │
│  Honeypot Hits (every 5 min)                                            │
│    └─→ EventBridge: tbi-ops-honeypot-queue-processor                    │
│          └─→ tbi-ops-honeypot-processor Lambda                          │
│                └─→ WAF IP block → notify                                │
│                                                                         │
│  Bedrock Threat Analysis (every 5 min)                                  │
│    └─→ EventBridge: tbi-ops-bedrock-analyze-schedule                    │
│          └─→ tbi-ops-bedrock-analyze Lambda                             │
│                └─→ Correlate signals → report → notify if HIGH/CRITICAL │
│                                                                         │
│  Support Ticket Submitted                                               │
│    └─→ Application invokes tbi-raima-support Lambda               │
│          └─→ Categorize → draft response → notify                      │
│                                                                         │
│  Daily/Weekly Digest (cron)                                             │
│    └─→ EventBridge: tbi-ops-daily-digest / tbi-ops-weekly-digest        │
│          └─→ tbi-ops-digest Lambda                                      │
│                └─→ Bedrock summary → formatted email                    │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Metrics published by the Live Price Oracle service:

Metrics published by the Live Report Service:

These alarms use CloudWatch Anomaly Detection (machine learning) to learn normal traffic patterns and alert on deviations. They need ~2 weeks to build a baseline.

All 7 functions share the tbi-autonomous-ops-role IAM role. Metrics visible on the Security Dashboard.

Alarms: Trinity-Beast-ALB-UnhealthyTargets, Trinity-Beast-NLB-UnhealthyTargets

1. Check ECS service health in the console — are tasks running or in a crash loop?
2. Review container logs in /aws/ecs/trinity-beast for startup errors or OOM kills
3. Verify target group health check path and expected response code
4. Check if a recent deployment introduced a breaking change
5. If tasks are running but unhealthy, check application health endpoint directly

Alarms: Trinity-Beast-ECS-CPU-High, ECS-CPU-High-Mirror, ECS-CPU-High-LRS

1. Check for a traffic spike — correlate with LPO/LRS request metrics on the Application Dashboard
2. Consider scaling the service — increase desired task count or adjust auto-scaling thresholds
3. Check for runaway goroutines or infinite loops in recent deployments
4. Review Container Insights for per-task CPU breakdown
5. If sustained, evaluate whether the task CPU allocation (vCPU) needs to be increased

Alarms: Trinity-Beast-Main-Service-Count-Low, Mirror-Service-Count-Low, LRS-Service-Count-Low

1. Check ECS service events for task stopped reasons (OOM, exit code, health check failure)
2. Review container logs for the last running task — look for panic, fatal, or OOM messages
3. Check if the ECR image exists and is pullable (image pull failures)
4. Verify the task execution role has required permissions
5. Manually start a new task if the service is not recovering automatically

Alarm: Trinity-Beast-Aurora-CPU-High

1. Check for slow queries — use Performance Insights or pg_stat_statements
2. Verify ACU scaling — is the cluster at max ACU and still under pressure?
3. Check if the nightly sync job is running and creating batch write pressure
4. Look for missing indexes on frequently queried columns
5. Consider increasing the max ACU limit if load is legitimate

Alarm: Trinity-Beast-Aurora-Connections-High

1. Check connection pool settings in the application — are pools sized correctly?
2. Look for connection leaks — connections opened but never returned to the pool
3. Verify that the sync job and Lambda are not opening excessive connections
4. Consider using RDS Proxy if connection pressure is persistent
5. Check if a recent deployment changed pool configuration

Alarms: Trinity-Beast-ElastiCache-CPU-High, ElastiCache-Memory-High, ElastiCache-Connections-High

1. Check for a cache stampede — many cache misses causing simultaneous upstream fetches
2. Review key eviction metrics — if memory is full, keys are being evicted prematurely
3. Check connection pool settings in the LPO service — are connections being reused properly?
4. Look for large keys or hot keys that may be causing uneven load
5. If memory is consistently high, consider scaling to a larger node type or adding shards
6. Review TTL settings — are cached items living too long and consuming memory?

Alarm: Trinity-Beast-S3-Size-Unusual-Growth

1. Check for unexpected uploads — review S3 access logs or CloudTrail for PutObject events
2. Look for log file accumulation — are old log exports or reports piling up?
3. Verify lifecycle policies are in place to expire or transition old objects
4. Check if the LRS report output is being stored without cleanup
5. Review bucket versioning — old versions may be consuming space