The Trinity Beast Infrastructure — CloudWatch Dashboard & Alarm Notifications

Live cost intelligence dashboard that combines resource utilization metrics with cost context. Rather than embedding stale dollar figures, it shows the metrics that drive spend — ECS CPU/Memory utilization across all 4 services, Aurora ACU + utilization (with min/max annotations), ElastiCache Serverless ECPU/storage, Lambda invocations and duration for all 8 functions, Aurora connections, and NAT Gateway/EC2 data transfer. Two cost-context tables explain the monthly baseline by component (with unit costs) and identify the cost levers you can control. Direct links to Cost Explorer, Budgets, and Savings Plans for exact dollar figures.

Replaces: The previous static Trinity-Beast-Cost-Executive-Dashboard and Trinity-Beast-Cost-Detailed-Dashboard (deleted 2026-05-30) — both were 100% hardcoded markdown text frozen at April 2026 figures, referencing services no longer in use.

Every target group behind every load balancer in the account now has its own UnHealthyHostCount alarm — closing the exact gap the previous section describes. All seven follow the same shape: Maximum statistic, 60-second period, 3 evaluation periods, threshold >= 1, TreatMissingData: missing (a healthy target group always publishes a real 0, so silence can only mean the alarm itself is blind — see Section 11).

Every alarm above this point was found by checking whether an existing alarm points at a real resource. That kind of check is structurally unable to see a resource that was never alarmed on at all. A coverage check that instead enumerates what should be watched and diffs against what is found these three gaps on its first run — including one on a paying product.

Both alarms below watch a CloudWatch Logs metric filter on /aws/ecs/trinity-beast, rather than a metric AWS already publishes. Each filter pattern was validated against real log samples via test-metric-filter before the alarm was created, specifically to confirm it matches the failure line and does not match an adjacent, intentional fallback line that merely looks similar.

These two alarms are different in kind from everything else in this section, and they are worth pausing on. They do not watch an AWS service metric — they watch metrics we publish ourselves into the TBI/Security namespace from CloudTrail activity. They answer a question no built-in metric answers: did somebody change something they should not have?

When you attach target-tracking auto-scaling to an ECS service, AWS silently creates a pair of alarms per service to drive it. They carry generated names like TargetTracking-service/trinity-beast-fargate-cluster/trinity-beast-main-service-AlarmHigh-63fe11ef…. You did not write them and you should not edit them — Application Auto Scaling owns them and will recreate them if you interfere.

┌─────────────────────────────────────────────────────────────────────────┐
│                      NOTIFICATION ROUTING                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  CloudWatch Alarm (any of 40 alarms — 39 us-east-2, 1 us-east-1)         │
│    ├─→ SNS: tbi-ops-notifications                                       │
│    │     └─→ tbi-ops-notify Lambda                                      │
│    │           └─→ Formatted HTML email (SES)                           │
│    │                 → CoryDeanKalani@CPMP-Site.org                      │
│    │                                                                    │
│    └─→ EventBridge: tbi-ops-alarm-trigger                               │
│          └─→ Step Function: tbi-ops-health-check-heal                   │
│                └─→ Self-heal → verify recovery → notify                 │
│                                                                         │
│  GuardDuty Finding (severity ≥ 7)                                       │
│    └─→ EventBridge: tbi-ops-guardduty-high-finding                      │
│          └─→ tbi-ops-bedrock-analyze Lambda                             │
│                └─→ AI threat assessment → auto-action → notify          │
│                                                                         │
│  Honeypot Hits (every 5 min)                                            │
│    └─→ EventBridge: tbi-ops-honeypot-queue-processor                    │
│          └─→ tbi-ops-honeypot-processor Lambda                          │
│                └─→ WAF IP block → notify                                │
│                                                                         │
│  Bedrock Threat Analysis (every 5 min)                                  │
│    └─→ EventBridge: tbi-ops-bedrock-analyze-schedule                    │
│          └─→ tbi-ops-bedrock-analyze Lambda                             │
│                └─→ Correlate signals → report → notify if HIGH/CRITICAL │
│                                                                         │
│  Support Ticket Submitted                                               │
│    └─→ Application invokes tbi-rhema-support Lambda               │
│          └─→ Categorize → draft response → notify                      │
│                                                                         │
│  Daily/Weekly Digest (cron)                                             │
│    └─→ EventBridge: tbi-ops-daily-digest / tbi-ops-weekly-digest        │
│          └─→ tbi-ops-digest Lambda                                      │
│                └─→ Bedrock summary → formatted email                    │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Metrics published by the Live Price Oracle service:

Metrics published by the Live Report Service:

These alarms use CloudWatch Anomaly Detection — a machine-learned band of "expected" values — rather than a fixed threshold. Instead of you choosing a number, CloudWatch observes the metric for roughly two weeks, learns its normal rhythm including daily and weekly cycles, and alarms when reality leaves the band.

All 7 functions share the tbi-autonomous-ops-role IAM role. Metrics visible on the Security Dashboard.

Alarms: Trinity-Beast-ALB-UnhealthyTargets, Trinity-Beast-NLB-UnhealthyTargets

1. Check ECS service health in the console — are tasks running or in a crash loop?
2. Review container logs in /aws/ecs/trinity-beast for startup errors or OOM kills
3. Verify target group health check path and expected response code
4. Check if a recent deployment introduced a breaking change
5. If tasks are running but unhealthy, check application health endpoint directly

Alarms: Trinity-Beast-ECS-CPU-High, ECS-CPU-High-Mirror, ECS-CPU-High-LRS

1. Check for a traffic spike — correlate with LPO/LRS request metrics on the Application Dashboard
2. Consider scaling the service — increase desired task count or adjust auto-scaling thresholds
3. Check for runaway goroutines or infinite loops in recent deployments
4. Review Container Insights for per-task CPU breakdown
5. If sustained, evaluate whether the task CPU allocation (vCPU) needs to be increased

Alarms: Trinity-Beast-Main-Service-Count-Low, Mirror-Service-Count-Low, LRS-Service-Count-Low

1. Check ECS service events for task stopped reasons (OOM, exit code, health check failure)
2. Review container logs for the last running task — look for panic, fatal, or OOM messages
3. Check if the ECR image exists and is pullable (image pull failures)
4. Verify the task execution role has required permissions
5. Manually start a new task if the service is not recovering automatically

Alarm: Trinity-Beast-Aurora-CPU-High

1. Check for slow queries — use Performance Insights or pg_stat_statements
2. Verify ACU scaling — is the cluster at max ACU and still under pressure?
3. Check if the nightly sync job is running and creating batch write pressure
4. Look for missing indexes on frequently queried columns
5. Consider increasing the max ACU limit if load is legitimate

Alarm: Trinity-Beast-Aurora-Connections-High

1. Check connection pool settings in the application — are pools sized correctly?
2. Look for connection leaks — connections opened but never returned to the pool
3. Verify that the sync job and Lambda are not opening excessive connections
4. Consider using RDS Proxy if connection pressure is persistent
5. Check if a recent deployment changed pool configuration

Alarms: Trinity-Beast-ElastiCache-CPU-High, ElastiCache-Memory-High, ElastiCache-Throttled, ElastiCache-Connections-High

1. ECPU alarm (ElastiCacheProcessingUnits): Command volume is nearing the 10,000 ECPU/s limit. Check for cache stampedes, hot keys, or a spike in traffic. Raise the ECPU limit in the console if sustained growth justifies it.
2. BytesUsedForCache alarm: Storage is approaching the 5 GB ceiling. Review TTL settings — are cached items living too long? Check for large keys (LRS report caches, search indexes) consuming disproportionate space. Raise the storage limit or prune stale keys.
3. ThrottledCmds alarm: Commands are being rejected because capacity was exceeded. This is the most urgent signal — it means requests are failing. Immediately raise the ECPU or storage limit. Then investigate what caused the spike.
4. Latency alarm (SuccessfulReadRequestLatency): p99 reads are taking longer than expected. This can indicate the Serverless proxy is under load or network path issues. Check VPC flow logs and ensure the security group allows traffic from all ECS subnets.
5. Remember: ElastiCache is a pure cache layer — Aurora is the source of truth. If the cache is under sustained pressure, the system degrades gracefully (slower reads, no data loss). Raising limits is a one-click operation with zero downtime.

Alarm: Trinity-Beast-S3-Size-Unusual-Growth

1. Check for unexpected uploads — review S3 access logs or CloudTrail for PutObject events
2. Look for log file accumulation — are old log exports or reports piling up?
3. Verify lifecycle policies are in place to expire or transition old objects
4. Check if the LRS report output is being stored without cleanup
5. Review bucket versioning — old versions may be consuming space

Symptom: an alarm has never fired, or sits permanently in INSUFFICIENT_DATA

1. Run bash scripts/kcc.sh alarm-audit — it validates every alarm's dimensions against the live resource inventory and names the offender directly
2. Read the alarm's StateReason. If it cites a datapoint from months ago, that timestamp tells you when the alarm went blind
3. Compare each dimension value against reality — aws elbv2 describe-load-balancers, aws rds describe-db-clusters, and so on
4. Correct it with put-metric-alarm, which upserts on alarm name — the same command that creates an alarm also repairs one
5. Choose TreatMissingData deliberately (see Section 5). Prefer missing on health alarms so a future mistake stays visible
6. Expect a [SELF-HEALED] email as the repaired alarm reaches OK for the first time — warn your team before you start

1. An alarm in OK is not evidence of health. It is evidence of one of two very different things, and you cannot tell which by looking at it. Verify dimensions, not states.
2. INSUFFICIENT_DATA is a finding, not a mood. Acceptable for fifteen minutes after a config change. Past that it means the metric never reports, and the fault is in the alarm rather than in the world.
3. Choose TreatMissingData as a decision, never as a default. Ask: if this metric goes silent, do I want to be reassured, alerted, or told plainly that I do not know? All three answers are sometimes right. notBreaching is the most comfortable and the most likely to hide a problem.
4. Muting an alarm silences the notification, not the automation. Before disabling actions, enumerate everything listening to that alarm's state — EventBridge rules included.
5. Anomaly detection needs a stable, unimodal, adequately-sampled metric. Ratios at low volume, bimodal distributions, and pre-launch baselines all defeat it. Where "bad" has an absolute definition — latency, error counts — a static threshold is clearer, cheaper, and needs no training period.
6. Ask whether a metric measures usage or failure. Usage metrics (CPU, storage, ECPU) deserve thresholds at a fraction of their ceiling. Failure metrics (throttled commands, IAM changes, 5xx) deserve thresholds at the first occurrence.
7. Baseline your noise before choosing a threshold. The WAF alarm sits at 300 because normal background scanning is about 33 per period. A threshold chosen without knowing the floor is a guess wearing the costume of a decision.
8. Infrastructure changes orphan alarms. The highest-yield habit in this whole section. After every migration, rename, or replacement, check what referenced the old thing — alarms and dashboard widgets. Seven widgets in this account were still graphing a load balancer deleted in April.
9. Verify a deletion with a filtered query, more than once, with a gap. Especially anomaly detectors, which can be implicitly recreated by anything that requests their band.
10. Documentation drifts toward describing intent. This guide previously listed all four security alarms as live components of the notification pipeline. None of them worked. Nothing forces a document to reconcile itself with reality, so the reconciliation has to be a deliberate act — which is what this revision is.