The Trinity Beast Infrastructure - Technical Specification v16

Executive Summary

The Trinity Beast Infrastructure (TBI) v4.7 is a high-performance, serverless cryptocurrency price oracle and reporting system built on AWS. The architecture features a WebSocket-first price feed system with zero-network hot path reads, delivering 0ms latency for price queries. The infrastructure leverages ECS Fargate with uniformly-sized containers (8 vCPU / 32 GB each), Aurora Serverless v2 with Optimized I/O right-sized to 2–18 ACU, and ElastiCache for Valkey with a pure cache layer rebuilt by the sync job in ~16 seconds. Under load testing (Run 17), the system sustained 746,374 combined RPS across 1.34 billion requests in a 30-minute test with zero degradation — including 369,600 TCP req/sec and 487,900 UDP req/sec with 100% success through all 13 concurrency levels.

LPO TCP Avg Latency

0.3ms

TCP Peak (Direct)

369,600 req/sec

Cache Hit Rate

99%+

Data Retention

93 Days

Architecture

Serverless

Deployment Model

Multi-AZ

v3.9.3 Key Differentiators: The Trinity Beast v3.9.3 features a zero-network hot path where price requests are served directly from an in-process sync.Map populated by persistent WebSocket feeds from 6 exchanges — Coinbase, Gemini, Kraken, Gate.io, Crypto.com, and OKX. No database or cache network calls occur on the hot path. A background worker pool (999 slots) batches Aurora writes via micro-batch smoothing (300 rows/100ms). All three ECS containers are now identically sized at 8 vCPU / 32 GB RAM running APP_REPORT_SERVER, with a 4th container running WEBHOOK_SERVER for outbound price push delivery. Aurora has been right-sized to 2–18 ACU. ElastiCache (cache.r7g.2xlarge, single node, no replica) serves as a pure cache layer — Aurora is the source of truth. A Distributed Adaptive Connection Governor (6000 max concurrent) manages TCP connections across the cluster. 150 price feed assets across 6 exchanges (25 each: Coinbase, Gemini, Kraken, Gate.io, Crypto.com, OKX). Multi-AZ deployment: Main in us-east-2a, Mirror in us-east-2b, LRS in us-east-2c.

1. Architecture Overview

The Trinity Beast Infrastructure v4.7 employs a modern, cloud-native architecture built entirely on AWS managed services. The design centers on a WebSocket-first price feed with a zero-network hot path, uniform container sizing, and dual load balancers (ALB for TCP, NLB for UDP). The system is designed for high availability, sub-millisecond latency, and horizontal scalability.

1.1 High-Level Architecture Diagram

Diagram 1.1 — High-Level Architecture

graph TB
    subgraph Internet["Internet"]
        Client[Client Applications]
        DNS_API[Route 53
api.cpmp-site.org]
        DNS_WWW[Route 53
www.cpmp-site.org]
        DNS_UDP[Route 53
udp.cpmp-site.org]
        WHSub[Webhook Subscribers
UDP + HTTPS Endpoints]
    end
    
    subgraph PriceFeeds["6 Real-Time WebSocket Feeds"]
        CB_WS[Coinbase]
        GM_WS[Gemini]
        KR_WS[Kraken]
        GT_WS[Gate.io]
        BY_WS[Crypto.com]
        OK_WS[OKX]
    end
    
    subgraph CloudFront["☁️ CloudFront CDN"]
        CF[CloudFront Distribution
cpmp-site.org]
    end
    
    subgraph VPC["VPC vpc-03deaddb7083cd59c
CIDR: 10.0.0.0/16"]
        subgraph PublicSubnets["Public Subnets"]
            ALB[Application Load Balancer
Trinity-Beast-TCP-ALB
TCP 80/443 → 8080/9090]
            NLB[Network Load Balancer
Trinity-Beast-UDP-NLB
UDP 2679/2680]
        end
        
        subgraph PrivateSubnets["Private Subnets (Multi-AZ)"]
            subgraph ECS["ECS Fargate Cluster
trinity-beast-fargate-cluster"]
                Main[BeastMain · AZ 2a
APP_REPORT_SERVER
8 vCPU / 32 GB]
                Mirror[BeastMirror · AZ 2b
APP_REPORT_SERVER
8 vCPU / 32 GB]
                LRS[BeastLRS · AZ 2c
APP_REPORT_SERVER
8 vCPU / 32 GB]
                Webhook[BeastWebhook
WEBHOOK_SERVER
8 vCPU / 32 GB]
            end
            
            subgraph DataLayer["Data Layer"]
                Aurora[(Aurora Serverless v2
PostgreSQL 17.7
Optimized I/O
2-18 ACU)]
                Cache[(ElastiCache
Valkey 7.2
cache.r7g.2xlarge
52.8 GB / TLS)]
            end
        end
    end
    
    Client -->|TCP API| DNS_API
    Client -->|Website| DNS_WWW
    Client -->|UDP API| DNS_UDP
    DNS_API -->|Resolve| ALB
    DNS_UDP -->|Resolve| NLB
    DNS_WWW -->|Resolve| CF
    CF -->|S3 Origin| ALB
    ALB -->|TCP 8080/9090| Main
    ALB -->|TCP 8080/9090| Mirror
    ALB -->|TCP 8080/9090| LRS
    NLB -->|UDP 2679/2680| Main
    NLB -->|UDP 2679/2680| Mirror
    NLB -->|UDP 2679/2680| LRS
    CB_WS -.->|WS Push| Main
    CB_WS -.->|WS Push| Mirror
    CB_WS -.->|WS Push| LRS
    CB_WS -.->|WS Push| Webhook
    GM_WS -.->|WS Push| Main
    GM_WS -.->|WS Push| Mirror
    GM_WS -.->|WS Push| LRS
    GM_WS -.->|WS Push| Webhook
    KR_WS -.->|WS Push| Main
    KR_WS -.->|WS Push| Mirror
    KR_WS -.->|WS Push| LRS
    KR_WS -.->|WS Push| Webhook
    GT_WS -.->|WS Push| Main
    GT_WS -.->|WS Push| Mirror
    GT_WS -.->|WS Push| LRS
    GT_WS -.->|WS Push| Webhook
    BY_WS -.->|WS Push| Main
    BY_WS -.->|WS Push| Mirror
    BY_WS -.->|WS Push| LRS
    BY_WS -.->|WS Push| Webhook
    OK_WS -.->|WS Push| Main
    OK_WS -.->|WS Push| Mirror
    OK_WS -.->|WS Push| LRS
    OK_WS -.->|WS Push| Webhook
    Main -->|Batch Write| Aurora
    Mirror -->|Batch Write| Aurora
    LRS -->|Query| Aurora
    Webhook -->|Delivery Log| Aurora
    Main -->|Fallback Cache| Cache
    Mirror -->|Fallback Cache| Cache
    LRS -->|Read| Cache
    Webhook -->|Read Subscriptions| Cache
    Webhook ==>|UDP + HTTPS Push| WHSub
    
    style Client fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0,stroke-width:2px
    style DNS_API fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0,stroke-width:2px
    style DNS_WWW fill:#4a1942,stroke:#f472b6,color:#e2e8f0,stroke-width:2px
    style DNS_UDP fill:#4a2800,stroke:#FF9900,color:#e2e8f0,stroke-width:2px
    style WHSub fill:#134e4a,stroke:#2dd4bf,color:#e2e8f0,stroke-width:2px
    style ALB fill:#4a2800,stroke:#FF9900,color:#e2e8f0,stroke-width:2px
    style NLB fill:#4a2800,stroke:#FF9900,color:#e2e8f0,stroke-width:2px
    style Main fill:#064e3b,stroke:#10b981,color:#e2e8f0,stroke-width:2px
    style Mirror fill:#064e3b,stroke:#10b981,color:#e2e8f0,stroke-width:2px
    style LRS fill:#064e3b,stroke:#10b981,color:#e2e8f0,stroke-width:2px
    style Webhook fill:#134e4a,stroke:#2dd4bf,color:#e2e8f0,stroke-width:2px
    style Aurora fill:#1e3a5f,stroke:#3b82f6,color:#e2e8f0,stroke-width:2px
    style Cache fill:#7f1d1d,stroke:#ef4444,color:#e2e8f0,stroke-width:2px
    style CB_WS fill:#164e63,stroke:#22d3ee,color:#e2e8f0,stroke-width:2px
    style GM_WS fill:#2e1065,stroke:#a78bfa,color:#e2e8f0,stroke-width:2px
    style KR_WS fill:#4a1942,stroke:#f472b6,color:#e2e8f0,stroke-width:2px
    style GT_WS fill:#422006,stroke:#facc15,color:#e2e8f0,stroke-width:2px
    style BY_WS fill:#1a2e05,stroke:#a3e635,color:#e2e8f0,stroke-width:2px
    style OK_WS fill:#431407,stroke:#fb923c,color:#e2e8f0,stroke-width:2px

    %% Client → DNS — white
    linkStyle 0 stroke:#60a5fa,stroke-width:2.5px
    linkStyle 1 stroke:#f472b6,stroke-width:2.5px
    linkStyle 2 stroke:#FF9900,stroke-width:2.5px
    %% DNS → LBs
    linkStyle 3 stroke:#60a5fa,stroke-width:2.5px
    linkStyle 4 stroke:#FF9900,stroke-width:2.5px
    linkStyle 5 stroke:#f472b6,stroke-width:2.5px
    linkStyle 6 stroke:#f472b6,stroke-width:2.5px
    %% ALB → ECS — blue
    linkStyle 7 stroke:#60a5fa,stroke-width:2.5px
    linkStyle 8 stroke:#60a5fa,stroke-width:2.5px
    linkStyle 9 stroke:#60a5fa,stroke-width:2.5px
    %% NLB → ECS — orange
    linkStyle 10 stroke:#FF9900,stroke-width:2.5px
    linkStyle 11 stroke:#FF9900,stroke-width:2.5px
    linkStyle 12 stroke:#FF9900,stroke-width:2.5px
    %% Coinbase WS — cyan (4 targets now)
    linkStyle 13 stroke:#22d3ee,stroke-width:2px
    linkStyle 14 stroke:#22d3ee,stroke-width:2px
    linkStyle 15 stroke:#22d3ee,stroke-width:2px
    linkStyle 16 stroke:#22d3ee,stroke-width:2px
    %% Gemini WS — violet
    linkStyle 17 stroke:#a78bfa,stroke-width:2px
    linkStyle 18 stroke:#a78bfa,stroke-width:2px
    linkStyle 19 stroke:#a78bfa,stroke-width:2px
    linkStyle 20 stroke:#a78bfa,stroke-width:2px
    %% Kraken WS — pink
    linkStyle 21 stroke:#f472b6,stroke-width:2px
    linkStyle 22 stroke:#f472b6,stroke-width:2px
    linkStyle 23 stroke:#f472b6,stroke-width:2px
    linkStyle 24 stroke:#f472b6,stroke-width:2px
    %% Gate.io WS — yellow
    linkStyle 25 stroke:#facc15,stroke-width:2px
    linkStyle 26 stroke:#facc15,stroke-width:2px
    linkStyle 27 stroke:#facc15,stroke-width:2px
    linkStyle 28 stroke:#facc15,stroke-width:2px
    %% Crypto.com WS — lime
    linkStyle 29 stroke:#a3e635,stroke-width:2px
    linkStyle 30 stroke:#a3e635,stroke-width:2px
    linkStyle 31 stroke:#a3e635,stroke-width:2px
    linkStyle 32 stroke:#a3e635,stroke-width:2px
    %% OKX WS — amber
    linkStyle 33 stroke:#fb923c,stroke-width:2px
    linkStyle 34 stroke:#fb923c,stroke-width:2px
    linkStyle 35 stroke:#fb923c,stroke-width:2px
    linkStyle 36 stroke:#fb923c,stroke-width:2px
    %% ECS → Aurora — red
    linkStyle 37 stroke:#f87171,stroke-width:2px
    linkStyle 38 stroke:#f87171,stroke-width:2px
    linkStyle 39 stroke:#f87171,stroke-width:2px
    linkStyle 40 stroke:#f87171,stroke-width:2px
    %% ECS → ElastiCache — green
    linkStyle 41 stroke:#10b981,stroke-width:2px
    linkStyle 42 stroke:#10b981,stroke-width:2px
    linkStyle 43 stroke:#10b981,stroke-width:2px
    linkStyle 44 stroke:#10b981,stroke-width:2px
    %% Webhook → Subscribers — teal (thick)
    linkStyle 45 stroke:#2dd4bf,stroke-width:3px

■ TCP Path ■ UDP Path ■ Website / CDN ■ Webhook Push ■ Coinbase WS ■ Gemini WS ■ Kraken WS ■ Gate.io WS ■ Crypto.com WS ■ OKX WS ■ Aurora ■ ElastiCache

1.2 Request Flow Diagram

v3.9.3 Zero-Network Hot Path: Client → Route 53 → ALB/NLB → ECS Task → local sync.Map (in-process) → Response. No database or cache network calls on the hot path. WebSocket feeds continuously push prices into the local sync.Map. REST fallbacks (one per exchange) activate only if the corresponding WS feed goes stale. Background worker pool batches Aurora writes asynchronously.

Diagram 1.2 — Request Flow (Zero-Network Hot Path)

%%{init: {'sequence': {'mirrorActors': false, 'actorMargin': 30, 'messageFontSize': 13}}}%%
sequenceDiagram
    participant WS as 🔌 WebSocket Feeds
(6 Exchanges: Coinbase,
Gemini, Kraken, Gate.io,
Crypto.com, OKX)
    participant SM as ⚡ Local sync.Map
(In-Process)
    participant C as 👤 Client
    participant LB as 🔀 ALB / NLB
    participant ECS as 🖥️ ECS Task
    participant MDB as 🧠 ElastiCache
    participant AUR as 🗄️ Aurora
    participant REST as 🌐 REST Fallbacks
(All 6 Exchanges)
    
    Note over WS,SM: ⚡ 150 prewarmed assets via 6 WebSockets
    WS->>SM: Push price updates (150 prewarmed assets)
    
    C->>LB: Request /price?asset=BTC
    LB->>ECS: Forward to Task
    ECS->>SM: Read from local sync.Map
    
    alt sync.Map has fresh price (99%+ of requests)
        SM->>ECS: Return price (0ms network)
        ECS->>LB: Response (~1ms P50)
        LB->>C: JSON Response
    else sync.Map miss — check ElastiCache
        ECS->>MDB: Read cached price
        MDB->>ECS: Return price (~0.5ms)
        ECS->>LB: Response
        LB->>C: JSON Response
    else All caches miss (rare)
        ECS->>REST: Fetch from REST fallbacks
        REST->>ECS: Price Data (30-50ms)
        ECS->>MDB: Write-through cache
        ECS->>SM: Update sync.Map
        ECS->>LB: Response
        LB->>C: JSON Response
    end
    
    Note over ECS,AUR: 🔄 Async Background (Worker Pool: 999 slots)
    ECS-->>MDB: Increment usage counter
    ECS-->>AUR: Batched write (300 records/100ms)

■ WebSocket Feeds (6 Exchanges) ■ Client ↔ LB ■ LB ↔ ECS ■ sync.Map (Hot Path) ■ REST Fallback ■ ElastiCache ■ Aurora (Async)

2. Compute Layer

2.1 Amazon ECS Fargate

ECS Fargate Cluster

Container Orchestration

Attribute	Value
Cluster Name	`trinity-beast-fargate-cluster`
Cluster ARN	`arn:aws:ecs:us-east-2:211998422884:cluster/trinity-beast-fargate-cluster`
Region	us-east-2 (Ohio)
Launch Type	Fargate (Serverless)
Total Tasks	3 (1 Main + 1 Mirror + 1 LRS)
Platform Version	LATEST (1.4.0)

Purpose & Function

The ECS Fargate cluster orchestrates all containerized workloads for The Trinity Beast v3.9.3. It manages three identically-sized services (8 vCPU / 32 GB each), each running the unified The Trinity Beast Docker image with SERVER_TYPE=APP_REPORT_SERVER. All containers expose all 6 ports (TCP 8080, TCP 8081 health, TCP 9090, TCP 9091 health, UDP 2679, UDP 2680).

AWS Nitro System: All Fargate tasks run on Nitro-based hosts. Dedicated Nitro Cards offload networking, storage I/O, and instance management to purpose-built hardware — delivering bare-metal-equivalent performance. 100% of each container's 8 vCPUs are available for application workload. The Nitro hypervisor also provides hardware-enforced isolation between containers and the host, ensuring no operator or co-tenant can access running memory or data.

Connections

ALB → Routes TCP traffic (8080/9090) to tasks

NLB → Routes UDP traffic (2679/2680) to tasks

Aurora Serverless → Batched writes via worker pool

ElastiCache → REST fallback cache

WebSocket Feeds → Coinbase and Gemini

CloudWatch → Logs and metrics

ECR → Container images

Performance Specifications

Total vCPU

32 vCPU

Total Memory

128 GB

Network Mode

awsvpc

Max Tasks

Unlimited

2.2 ECS Services

The Trinity Beast Main Service (BeastMain)

ECS Service

Attribute	Value
Service Name	`trinity-beast-main-service`
Service ARN	`arn:aws:ecs:us-east-2:211998422884:service/trinity-beast-fargate-cluster/trinity-beast-main-service`
Task Definition	`trinity-beast-lpo-task:8`
Desired Count	1 task
Server Type	`APP_REPORT_SERVER`
Node Name	BeastMain
`CPU`	8192 (8 `vCPU`)
Memory	32768 MB (32 GB)
Architecture	X86_64 (AMD64)
Ports	TCP 8080, TCP 9090, UDP 2679, UDP 2680

Purpose & Function

Primary LPO service handling cryptocurrency price queries via the zero-network hot path. WebSocket feeds from Coinbase and Gemini push prices into a local sync.Map. Requests are served directly from in-process memory with no database or cache network calls. Handles all 6 ports; SERVER_TYPE=APP_REPORT_SERVER enables both LPO and LRS listeners.

Performance Specifications

TCP Peak (Direct)

369,600 req/sec

Avg Latency (TCP)

0.3ms

UDP Avg Latency

0.2ms

Cache Hit Rate

99%+ (WebSocket-fed)

Environment Variables

SERVER_TYPE=APP_REPORT_SERVER
DB_SECRET_NAME=trinity-beast-secrets
CACHE_URL=trinity-beast-cache.ptsbmm.0001.use2.cache.amazonaws.com
AURORA_URL=vpce-01935c7c9df23d302-30hrzsj5.rds.us-east-2.vpce.amazonaws.com
TCP_PORT_LPO=8080
TCP_PORT_LRS=9090
HEALTH_PORT_LPO=8081
HEALTH_PORT_LRS=9091
UDP_PORT_LPO=2679
UDP_PORT_LRS=2680
CLUSTER_NODE=BeastMain
NODE_ENV=production
AWS_REGION=us-east-2
SQS_USAGE_LOG_QUEUE_URL=https://sqs.us-east-2.amazonaws.com/211998422884/trinity-beast-queued-usage-logs

The Trinity Beast Mirror Service (BeastMirror)

ECS Service

Attribute	Value
Service Name	`trinity-beast-mirror-service`
Service ARN	`arn:aws:ecs:us-east-2:211998422884:service/trinity-beast-fargate-cluster/trinity-beast-mirror-service`
Task Definition	`trinity-beast-mirror-task:7`
Desired Count	1 task
Server Type	`APP_REPORT_SERVER`
Node Name	BeastMirror
`CPU`	8192 (8 `vCPU`)
Memory	32768 MB (32 GB)
Architecture	X86_64 (AMD64)
Ports	TCP 8080, TCP 9090, UDP 2679, UDP 2680

Purpose & Function

Mirror service providing load distribution and high availability. Configured as APP_REPORT_SERVER with the same WebSocket-fed zero-network hot path as BeastMain. Identically sized at 8 vCPU / 32 GB. All 6 ports exposed; SERVER_TYPE determines active listeners.

Performance Specifications

TCP Peak (Direct)

369,600 req/sec

Avg Latency

0.3ms

Cache Hit Rate

99%+ (WebSocket-fed)

Server Errors

Zero

The Trinity Beast LRS Service (BeastLRS)

ECS Service

Attribute	Value
Service Name	`trinity-beast-lrs-service`
Service ARN	`arn:aws:ecs:us-east-2:211998422884:service/trinity-beast-fargate-cluster/trinity-beast-lrs-service`
Task Definition	`trinity-beast-lrs-task:9`
Desired Count	1 task
Server Type	`APP_REPORT_SERVER`
Node Name	BeastLRS
`CPU`	8192 (8 `vCPU`)
Memory	32768 MB (32 GB)
Architecture	X86_64 (AMD64)
Ports	TCP 8080, TCP 9090, UDP 2679, UDP 2680

Purpose & Function

Combined service providing both LPO price queries and LRS reporting capabilities. Configured as APP_REPORT_SERVER with the same 8 vCPU / 32 GB sizing as all other containers. Handles TCP (8080, 9090), dedicated health (8081, 9091), and UDP (2679, 2680) protocols for price and reporting requests.

Performance Specifications

LRS TCP Peak

369,600 req/sec

LRS TCP Avg

4.1ms

LRS UDP Peak

487,900 req/sec

LRS UDP Avg

1.2ms

2.3 Task Definitions

Container Image: All services use the same unified Docker image stored in Amazon ECR. The SERVER_TYPE environment variable determines which listeners start at runtime.
211998422884.dkr.ecr.us-east-2.amazonaws.com/trinity-beast-lpo-server:latest

Task Definition	Family	Revision	CPU	Memory	Platform
LPO Task (BeastMain)	`trinity-beast-lpo-task`	22	8192	`32768 MB`	X86_64/LINUX
Mirror Task (BeastMirror)	`trinity-beast-mirror-task`	20	8192	`32768 MB`	X86_64/LINUX
LRS Task (BeastLRS)	`trinity-beast-lrs-task`	22	8192	`32768 MB`	X86_64/LINUX
Webhook Task (BeastWebhook)	`trinity-beast-webhook-task`	1	8192	`32768 MB`	X86_64/LINUX
Sync Task	`trinity-beast-sync-task`	5	`512`	`1024 MB`	X86_64/LINUX

v4.7 Upgrade — Container Sizing: All four application containers are uniformly sized at 8 vCPU / 32 GB RAM (total: 32 vCPU, 128 GB). Three run as APP_REPORT_SERVER (LPO + LRS), one runs as WEBHOOK_SERVER (outbound price push). Run 17 proved 369,600 TCP req/sec (direct) and 746,374 combined RPS sustained for 30 minutes across 9 containers (72 vCPU, 288 GB). Production runs 4 containers (3 LPO/LRS + 1 Webhook).

IAM Roles

Task Execution Role

ecsTaskExecutionRole

Permissions: ECR pull, CloudWatch logs, Secrets Manager

Task Role

ecsTaskRole

Permissions: Secrets Manager read, CloudWatch logs write, Translate text

3. Data Layer

3.1 Aurora Serverless v2 (PostgreSQL)

Aurora PostgreSQL Cluster

Serverless Database

Attribute	Value
Cluster Identifier	`trinity-beast-aurora-cluster`
Cluster ARN	`arn:aws:rds:us-east-2:211998422884:cluster:trinity-beast-aurora-cluster`
Engine	Aurora PostgreSQL 17.7
Deployment	Serverless v2 with Optimized I/O
Endpoint (Writer)	`trinity-beast-aurora-cluster.cluster-cvg4oeysemon.us-east-2.rds.amazonaws.com`
VPC Endpoint	`vpce-01935c7c9df23d302-30hrzsj5.rds.us-east-2.vpce.amazonaws.com`
Port	`5432`
Multi-AZ	Yes (3 AZs)

Purpose & Function

Primary relational database storing API keys, user accounts, subscription data, application parameters, and transaction logs. Aurora is no longer on the hot path for price requests. The background worker pool batches writes via micro-batch smoothing (300 rows/100ms), and Aurora has been right-sized to 2–18 ACU.

Performance Specifications

ACU Range

2 - 18 ACU

Peak Observed

19.5 ACU (42.9M requests)

Max Connections

5,000+

Storage

Auto-scaling

IOPS

Unlimited (Optimized I/O)

Latency

1-5ms

v3.9.3 Right-Sizing: ACU range set to 2–18. Aurora is no longer on the hot path — the micro-batch flusher writes 300 rows every 100ms, smoothing ACU usage and eliminating spikes. DB connections: 150 per container (450 total).

Optimized I/O Benefits

Aurora Optimized I/O eliminates I/O charges by bundling I/O into the compute pricing. This provides:

Predictable Pricing: No surprise I/O bills during traffic spikes
Better Performance: No throttling based on I/O budget
Efficiency: Up to 40% savings for I/O-intensive workloads
Simplified Pricing: Single per-ACU charge includes compute + I/O

Backup & Recovery

Backup Retention

7 Days

Backup Window

03:00-04:00 UTC

Snapshot Type

Automated + Manual

Point-in-Time Recovery

5-minute granularity

Connections

Worker Pool → Batched usage log writes (500/10s)

ECS Tasks → API key validation

Sync Job → Daily ElastiCache sync

Admin Tools → Management queries

3.2 ElastiCache for Valkey

ElastiCache Cluster

Cache Layer

Attribute	Value
Replication Group	`trinity-beast-cache`
Engine	Valkey 7.2
Node Type	cache.r7g.2xlarge (Graviton3)
Nodes	1 (single node, no replica)
Memory	52.8 GB
Endpoint	`master.trinity-beast-cache.ptsbmm.use2.cache.amazonaws.com:6379`
Port	`6379`
Encryption	TLS (In-transit + At-rest)

Purpose & Function

ElastiCache serves as a pure cache layer for API key lookups, rate limiting state, application parameters, and REST fallback price cache. It is no longer on the hot path for price requests (that role is filled by the in-process sync.Map). Aurora is the source of truth — ElastiCache is rebuilt by the sync job in ~16 seconds.

Performance Specifications

Memory

52.8 GB

vCPU

8 vCPU (Graviton3)

Network Bandwidth

3.75 Gbps baseline / 15 Gbps burst

Latency

<1ms (p99)

Throughput

400,000+ ops/sec

Cache Connections

300 per container

Key Patterns (v4.0)

Key Pattern	Type	Purpose	TTL
`apikey:<key>`	String	API key cache	`24h`
`app:config`	Hash	Application parameters	`None`
`ratelimit:<key>`	String	Shared rate limit state	Window-based
`usage:counter:<key>`	String	Real-time usage counters	Rolling
`price:<ASSET>`	String	Price cache (REST fallback write-through)	Short
`usage_log:<uuid>`	Hash	Full usage log entries	93 days
`usage_logs:index`	Sorted Set	Global timestamp sorted set	93 days
`usage_logs:api_key:<key>`	Sorted Set	Per-key sorted set	93 days
`usage_logs:asset:<ASSET>`	Sorted Set	Per-asset sorted set	93 days
`{adaptive:lpo}:successes`	String	Adaptive governor success counter	`None`
`{adaptive:lpo}:total`	String	Adaptive governor total counter	`None`
`{adaptive:lpo}:throttle`	String	Adaptive governor throttle counter	`None`
`cluster:stats:<node>`	Hash	Per-node runtime telemetry (24 counters)	`10s`
`session:<session_id>`	String	Stripe checkout session deduplication	`24h`
`webhook:<event_id>`	String	Stripe webhook event deduplication	`24h`
`kcc:daily`	String (JSON)	KCC daily dashboard metrics blob	`24h`
`translate:<lang>:<hash>`	String	AWS Translate cached translations (map captions)	30 days
`report_usage:<id>`	Hash	LRS report usage log entries	93 days

Backup & Retention

Snapshot Retention

7 Days

Snapshot Window

02:00-03:00 UTC

Data Retention

93 Days

Sync Frequency

Daily (1 AM EST)

Sync Job

Task: trinity-beast-sync-job:7 (1024 CPU, 2048 MB)
Schedule: Daily at 1 AM EST, incremental sync with high-water mark
Syncs: usage_logs, API keys, app params, and report_usage_logs from Aurora to ElastiCache
Performance: Full historical load completes in ~16 seconds. Incremental syncs under 200ms.
Environment Variables:

Variable	Value	Purpose
`DB_SECRET_NAME`	`trinity-beast-secrets`	Secrets Manager secret for Aurora credentials
`CACHE_URL`	`master.trinity-beast-cache.ptsbmm.use2.cache.amazonaws.com`	ElastiCache (Valkey) endpoint
`AWS_REGION`	`us-east-2`	AWS region for service calls
`ADMIN_KEY`	(from application_parameters)	Admin API key for search index rebuild trigger

Connections

ECS Tasks → API key cache reads

ECS Tasks → Rate limit state

ECS Tasks → REST fallback price write-through

LRS Tasks → Usage report queries (sorted sets)

Sync Job → Daily Aurora sync

4. Network Layer

4.1 Virtual Private Cloud (VPC)

The Trinity Beast VPC

Network Infrastructure

Attribute	Value
VPC ID	`vpc-03deaddb7083cd59c`
CIDR Block	10.0.0.0/16 (65,`536` IPs)
Region	us-east-2 (Ohio)
DNS Hostnames	Enabled
DNS Resolution	Enabled

Subnet Configuration

Subnet ID	AZ	CIDR	Type	Purpose
`subnet-06781ce7266a4b870`	`us-east-2a`	`10.0.1.0/24`	Public	ALB / NLB
`subnet-0e7e032219e0a6956`	`us-east-2b`	`10.0.2.0/24`	Public	ALB / NLB
`subnet-0d77afcde34842b5c`	`us-east-2c`	`10.0.3.0/24`	Public	ALB / NLB
`subnet-0433c423c48c88fa8`	`us-east-2a`	`10.0.11.0/24`	Private	ElastiCache
`subnet-0fb79c9f7d01b8997`	`us-east-2b`	`10.0.12.0/24`	Private	ElastiCache
`subnet-0595eb04f1ba859f0`	`us-east-2c`	`10.0.13.0/24`	Private	ElastiCache

4.2 Application Load Balancer

The Trinity Beast ALB (TCP Traffic)

Load Balancer

Attribute	Value
Name	`Trinity-Beast-TCP-ALB`
`ARN`	`arn:aws:elasticloadbalancing:us-east-2:211998422884:loadbalancer/app/Trinity-Beast-TCP-ALB/fafe772c40914b09`
DNS Name	`Trinity-Beast-TCP-ALB-807913906.us-east-2.elb.amazonaws.com`
Scheme	Internet-facing
IP Address Type	IPv4
Listeners	HTTP:80, HTTPS:`443`
Routing	Path-based routing to TCP 8080 (LPO) and TCP 9090 (LRS)

Performance Specifications

Max Throughput

Unlimited

Concurrent Connections

500,000+

New Connections/sec

100,000+

Latency Overhead

<1ms

Target Groups

Name	Port	Protocol	Health Check
`trinity-beast-fargate-group`	`8080`	`HTTP`	/health on port 8081 (dedicated health server, 30s interval)
`trinity-beast-targets-v3`	`9090`	`HTTP`	/health on port 9091 (dedicated health server, 30s interval)

4.3 Network Load Balancer (UDP)

The Trinity Beast NLB (UDP Traffic)

Network Load Balancer

Attribute	Value
Name	`Trinity-Beast-UDP-NLB`
Scheme	Internet-facing
IP Address Type	IPv4
Listeners	UDP:2679 (LPO), UDP:2680 (LRS)
ECS Integration	Target groups auto-register/deregister ECS tasks

NLB: The NLB provides dedicated UDP load balancing. Target groups are integrated with ECS for automatic task registration and deregistration. UDP 2679 routes to LPO handlers, UDP 2680 routes to LRS handlers across all three LPO/LRS containers. The 4th container (BeastWebhook) does not receive inbound traffic — it is outbound-only.

Target Groups

Name	Port	Protocol	Integration
`trinity-beast-udp-lpo-tg`	`2679`	`UDP`	ECS auto-register/deregister
`trinity-beast-udp-lrs-tg`	`2680`	`UDP`	ECS auto-register/deregister

4.4 Security Groups

Group ID	Name	Purpose	Inbound Rules
`sg-050b617f93b2388f6`	ECS Tasks SG	ECS task security	8080/tcp from 10.0.0.0/16 9090/tcp from 10.0.0.0/16 2679/udp from 10.0.0.0/16 2680/udp from 10.0.0.0/16
`sg-08a14f22df269a909`	ElastiCache SG	ElastiCache access	6379/tcp from sg-050b617f93b2388f6

5. Advanced Features & Optimizations

5.1 Price Architecture (v3.9.3)

WebSocket-First, Zero-Network Hot Path

The v3.9.3 price architecture eliminates all network calls from the hot path. Prices are served from an in-process sync.Map populated by persistent WebSocket feeds:

6 WebSocket Feeds (Primary): Coinbase WS, Gemini WS, Kraken WS, Gate.io WS, Crypto.com WS, OKX WS — persistent, push-based, 0ms network latency, 25 assets each (150 total)
6 REST Fallbacks (Secondary): Coinbase REST, Gemini REST, Kraken REST, Gate.io REST, Crypto.com REST, OKX REST — only used if the corresponding WS feed is stale
Zero-Network Hot Path: request → local sync.Map → response (no DB, no cache network calls)
Background Worker Pool: 999 slots, batched Aurora writes (300 records/100ms via micro-batch smoothing)
Cache Hit Rate: 99%+ (WebSocket-fed sync.Map always has fresh data)

Diagram 5.1 — Price Architecture (WebSocket-First)

graph LR
    subgraph WSFeeds["6 Real-Time WebSocket Feeds (Push)"]
        CB[Coinbase]
        GM[Gemini]
        KR[Kraken]
        GT[Gate.io]
        BY[Crypto.com]
        OK[OKX]
    end
    
    subgraph Container["ECS Container (×4)"]
        SM[sync.Map
In-Process Memory
150 Prewarmed Assets
0ms latency]
        WP[Worker Pool
999 slots
Batch: 300/100ms]
    end
    
    subgraph Fallback["REST Fallbacks (Pull — Rare, per-exchange)"]
        CR[Coinbase REST]
        GR[Gemini REST]
        KRR[Kraken REST]
        GTR[Gate.io REST]
        BYR[Crypto.com REST]
        OKR[OKX REST]
    end
    
    CB -->|Push| SM
    GM -->|Push| SM
    KR -->|Push| SM
    GT -->|Push| SM
    BY -->|Push| SM
    OK -->|Push| SM
    
    CR -.->|Only if WS stale| SM
    GR -.->|Only if WS stale| SM
    KRR -.->|Only if WS stale| SM
    GTR -.->|Only if WS stale| SM
    BYR -.->|Only if WS stale| SM
    OKR -.->|Only if WS stale| SM
    
    SM -->|Async| WP
    WP -->|Batch Write| AUR[(Aurora)]
    
    style SM fill:#10b981,stroke:#1e293b,color:#0f172a
    style WP fill:#60a5fa,stroke:#1e293b,color:#0f172a
    style CB fill:#22d3ee,stroke:#1e293b,color:#0f172a
    style GM fill:#a78bfa,stroke:#1e293b,color:#0f172a
    style KR fill:#f472b6,stroke:#1e293b,color:#0f172a
    style GT fill:#facc15,stroke:#1e293b,color:#0f172a
    style BY fill:#a3e635,stroke:#1e293b,color:#0f172a
    style OK fill:#fb923c,stroke:#1e293b,color:#0f172a
    style CR fill:#94a3b8,stroke:#1e293b,color:#0f172a
    style GR fill:#94a3b8,stroke:#1e293b,color:#0f172a
    style KRR fill:#94a3b8,stroke:#1e293b,color:#0f172a
    style GTR fill:#94a3b8,stroke:#1e293b,color:#0f172a
    style BYR fill:#94a3b8,stroke:#1e293b,color:#0f172a
    style OKR fill:#94a3b8,stroke:#1e293b,color:#0f172a
    style AUR fill:#3b82f6,stroke:#1e293b,color:#0f172a

    %% Coinbase — cyan
    linkStyle 0 stroke:#22d3ee,stroke-width:2.5px
    %% Gemini — violet
    linkStyle 1 stroke:#a78bfa,stroke-width:2.5px
    %% Kraken — pink
    linkStyle 2 stroke:#f472b6,stroke-width:2.5px
    %% Gate.io — yellow
    linkStyle 3 stroke:#facc15,stroke-width:2.5px
    %% Crypto.com — lime
    linkStyle 4 stroke:#a3e635,stroke-width:2.5px
    %% OKX — amber
    linkStyle 5 stroke:#fb923c,stroke-width:2.5px
    %% REST fallbacks — grey
    linkStyle 6 stroke:#94a3b8,stroke-width:1.5px
    linkStyle 7 stroke:#94a3b8,stroke-width:1.5px
    linkStyle 8 stroke:#94a3b8,stroke-width:1.5px
    linkStyle 9 stroke:#94a3b8,stroke-width:1.5px
    linkStyle 10 stroke:#94a3b8,stroke-width:1.5px
    linkStyle 11 stroke:#94a3b8,stroke-width:1.5px
    %% sync.Map → Worker Pool — green
    linkStyle 12 stroke:#10b981,stroke-width:2px
    %% Worker Pool → Aurora — blue
    linkStyle 13 stroke:#3b82f6,stroke-width:2px

5.2 ElastiCache Performance

Sub-Millisecond Latency at Scale

ElastiCache delivers consistent microsecond read and write latency even under heavy load:

P50 Latency: 200-400 microseconds for reads, 300-500 microseconds for writes
P99 Latency: <1 millisecond even at 400,000+ ops/sec
Throughput: 400,000+ operations per second (cache.r7g.2xlarge)
Configuration: Single node, no replica — Aurora is the source of truth

v3.9.3 Role: ElastiCache is a pure cache layer — not on the price hot path. It serves as the cache for API key lookups, rate limiting state, application parameters, and REST fallback write-through cache. Rebuilt by the sync job in ~16 seconds.

5.3 Aurora Optimized I/O Serverless v2

Why Aurora Optimized I/O?

Traditional Aurora charges separately for I/O operations, which can be unpredictable during traffic spikes. Optimized I/O bundles I/O into compute pricing:

No I/O Charges: Unlimited I/O operations included in ACU pricing
Pricing Predictability: Fixed per-ACU rate regardless of I/O volume
Performance: No throttling based on I/O budget
Efficiency: Up to 40% reduction for I/O-intensive workloads

Serverless v2 Auto-Scaling (Right-Sized in v3.9.3)

Aurora Serverless v2 scales compute capacity in 0.5 ACU increments. In v3.9.3, the range has been right-sized to 2–18 ACU:

Right-Sized Range: 2–18 ACU
Instant Scaling: Scales up/down in seconds, not minutes
Fine-Grained: 0.5 ACU increments for precise capacity matching
Efficient: Pay only for capacity used, billed per second
Off Hot Path: Background worker pool batches writes, reducing ACU demand

5.4 AWS Fargate Advantages

Serverless Container Orchestration

Fargate eliminates the need to provision and manage EC2 instances:

No Server Management: AWS manages underlying infrastructure
Uniform Sizing: All 4 containers at 8 vCPU / 32 GB — 3 LPO/LRS + 1 Webhook Push
Automatic Scaling: Scale tasks independently based on demand
Security: Task-level isolation with dedicated ENIs
Resource Efficient: Pay only for vCPU and memory used

Why Fargate for The Trinity Beast v4.7?

Simplified Operations: No patching, no capacity planning, no cluster management
Rapid Deployment: Deploy new versions in seconds
Cost Effective: Serverless model — pay only for actual compute, no idle capacity waste
Elastic: Can scale to 100+ tasks during traffic spikes
NLB Integration: ECS-integrated target groups for automatic UDP task registration

5.5 Optimization Settings

Scaling Note: All components can scale up with simple parameter adjustments—increased capacity is available in less than 5 minutes for Aurora ACU, ECS task count, and ElastiCache node upgrades.

Component	Setting	Current Value	Impact
Aurora	I/O Model	Optimized I/O	40% I/O reduction, unlimited IOPS
Aurora	ACU Range	2–18 ACU	Auto-scales based on demand
ElastiCache	Node Type	`cache.r7g.2xlarge`	52.8 GB memory, 400K+ ops/sec, TLS
`ECS`	Container Sizing	8 `vCPU` / 32 GB (uniform)	All 4 containers uniform (3 LPO/LRS + 1 Webhook), 32 `vCPU` / `128` GB total
Worker Pool	Batch Size	`300` records / 100ms	Reduces Aurora write pressure, keeps ACU low

6. Performance Specifications

6.1 System-Wide Performance Metrics (Run 17 — Tested)

Production vCPU

32 vCPU

Production Memory

128 GB

Cache Memory (ElastiCache)

52.8 GB

Database ACU

2–18 ACU

TCP Peak (Direct)

369,600 req/sec

UDP Peak (Direct)

487,900 req/sec

Combined Sustained

746,374 RPS

Total Requests (30 min)

1.34 Billion

TCP Avg Latency

0.3ms

UDP Avg Latency

0.2ms

Cache Hit Rate

99%+ (WebSocket-fed)

Data Retention

93 Days

Improvement

943× from v1.0

Server Errors

Zero

6.2 Detailed Test Results (Run 17)

Protocol	Service	Peak req/sec	Duration	Avg Latency	Success
TCP (Direct)	`LPO`	`369,600`	Burst (13 levels)	0.3ms	`100`% (L1–L11)
UDP (Direct)	`LPO`	`487,900`	30 min sustained	0.2ms	`100`% (all 13 levels)
TCP (ALB)	`LPO`	35,800	Burst (subscriber path)	—	`100`% (L1–L4)
UDP (NLB)	`LPO`	69,000	Burst (subscriber path)	—	`100`% (L1–L10)

6.3 Latency Breakdown (v4.7 Hot Path)

Operation	Component	Latency	Notes
DNS Resolution	Route 53	<5ms	Cached by client
TLS Handshake	`ALB`	10-20ms	First request only
Load Balancing	ALB / NLB	<1ms	Minimal overhead
Price Lookup	sync.Map (in-process)	~0ms	Zero network — local memory read
JSON Serialization	Application	<1ms	Minimal overhead
Total Hot Path	End-to-end	~1.0ms P50	No DB or cache network calls

v4.7 Performance: The zero-network hot path eliminates the ElastiCache cache lookup and Aurora API key validation from the request path. Price reads go directly to the in-process sync.Map, delivering 0.3ms TCP avg latency and 0.2ms UDP avg latency. Run 17 sustained 746,374 combined RPS across 1.34 billion requests with zero degradation. Cache hit rate: 99%+ thanks to 6 persistent WebSocket feeds.

6.4 Throughput Capacity

Diagram 6.1 — Throughput Capacity

graph LR
    A[369,600 req/sec
TCP Peak Direct] --> B[BeastMain
APP_REPORT_SERVER
8 vCPU / 32 GB]
    A --> C[BeastMirror
APP_REPORT_SERVER
8 vCPU / 32 GB]
    A --> E[BeastLRS
APP_REPORT_SERVER
8 vCPU / 32 GB]
    
    D[487,900 req/sec
UDP Peak · 100% All 13 Levels] --> C
    D --> E
    
    B --> F[99%+ sync.Map Hit
~0ms network]
    B --> G[<1% REST Fallback
WS feeds stale]
    
    H[BeastWebhook
WEBHOOK_SERVER
8 vCPU / 32 GB] -->|UDP + HTTPS Push| I[Webhook Subscribers
Up to 150 assets
3-60s intervals]
    
    style A fill:#FF9900,stroke:#1e293b,color:#0f172a
    style D fill:#60a5fa,stroke:#1e293b,color:#0f172a
    style B fill:#10b981,stroke:#1e293b,color:#0f172a
    style C fill:#10b981,stroke:#1e293b,color:#0f172a
    style E fill:#10b981,stroke:#1e293b,color:#0f172a
    style F fill:#60a5fa,stroke:#1e293b,color:#0f172a
    style G fill:#ef4444,stroke:#1e293b,color:#fff
    style H fill:#2dd4bf,stroke:#1e293b,color:#0f172a
    style I fill:#134e4a,stroke:#2dd4bf,color:#e2e8f0

7. Disaster Recovery & CloudFormation

7.1 CloudFormation Infrastructure as Code

The Trinity Beast Infrastructure v4.7 is defined and deployed using AWS CloudFormation stacks, enabling:

Version Control: All infrastructure changes tracked in Git
Reproducibility: Entire stack can be recreated in minutes
Disaster Recovery: Rapid recovery in alternate region
Testing: Spin up identical staging environments
Rollback: Revert to previous stack versions instantly

7.2 Backup Strategy

Component	Backup Method	Frequency	Retention	RTO	RPO
Aurora	Automated Snapshots	Daily	7 days	<15 min	`5 min`
ElastiCache	Pure cache — rebuilt by sync job	On demand (~16s)	N/A	<1 min	~16 seconds
ECS Tasks	ECR Image Tags	Per deployment	Unlimited	<2 min	0
Configuration	CloudFormation	Per change	Unlimited	<5 min	0

7.3 Disaster Recovery Procedures

Complete Infrastructure Recreation

# 1. Deploy CloudFormation stack
aws cloudformation create-stack \
  --stack-name trinity-beast-production \
  --template-body file://infrastructure.yaml \
  --region us-east-2

# 2. Restore Aurora from snapshot
aws rds restore-db-cluster-from-snapshot \
  --db-cluster-identifier trinity-beast-aurora-cluster-restored \
  --snapshot-identifier latest-snapshot

# 3. Restore ElastiCache from snapshot (if needed)
# Note: ElastiCache is a pure cache layer — rebuilt by sync job in ~16 seconds
# No snapshot restore needed for normal recovery

# 4. Deploy ECS services
aws ecs update-service \
  --cluster trinity-beast-fargate-cluster \
  --service trinity-beast-main-service \
  --force-new-deployment

# Total Recovery Time: ~20 minutes

7.4 High Availability Features

Multi-AZ Deployment: All components span 3 availability zones
Auto-Healing: ECS automatically replaces failed tasks
Health Checks: ALB removes unhealthy targets within 30 seconds
NLB Integration: ECS target groups auto-register/deregister for UDP
Database Failover: Aurora fails over in <30 seconds
Cache Failover: ElastiCache is a pure cache layer — rebuilt by sync job in ~16 seconds if lost
WebSocket Resilience: 6 independent WS feeds (Coinbase, Gemini, Kraken, Gate.io, Crypto.com, OKX) with per-exchange REST fallback if stale

8. Monitoring & Observability

8.1 CloudWatch Metrics

Metric	Source	Threshold	Action
CPU Utilization	ECS Tasks	>80%	Scale up tasks
Memory Utilization	ECS Tasks	>85%	Scale up tasks
Request Count	`ALB`	>50,000/min	Monitor for scaling
Target Response Time	`ALB`	>10ms P50	Investigate performance
Unhealthy Targets	ALB / NLB	>0	Alert operations
Database ACU	Aurora	>5 ACU	Review worker pool batch size
WebSocket Feed Status	Application	Any feed down	Check REST fallback active
Worker Pool Queue	Application	>`400` slots used	Review batch settings

8.2 Logging

Log Group

/aws/ecs/trinity-beast

Retention

30 Days

Log Streams

Per Task

CloudWatch Insights

9. Scaling & Operations

9.1 Architecture Advantages

Serverless Architecture: Pay only for actual compute usage with Aurora Serverless v2 and ECS Fargate
Aurora Right-Sizing: Right-sized ACU to 2–18
Aurora Optimized I/O: Eliminates per-I/O charges, providing predictable resource usage
Uniform Container Sizing: All containers at 8 vCPU / 32 GB
Micro-Batch Smoothing: 300 rows/100ms eliminates Aurora ACU spikes, keeping writes predictable
WebSocket Feeds: Push-based pricing eliminates REST API call overhead on the hot path

9.2 Scaling Flexibility

The infrastructure is designed for rapid scaling with minimal operational overhead:

ECS Tasks: Increase desired count from 3 to 10+ tasks in under 2 minutes
Aurora ACU: Adjust max ACU from 18 to higher for increased database throughput (scales automatically)
ElastiCache: Pure cache layer — upgrade node type or add replicas if needed
ALB: Automatically scales to handle increased traffic with no configuration changes
NLB: Automatically scales UDP capacity with ECS-integrated target groups

Scaling is just a parameter adjustment away—no infrastructure redesign required. All changes can be made via AWS Console, CLI, or CloudFormation updates.

10. Autonomous Operations (AutoOps)

A 5-layer intelligent operations system that monitors, defends, heals, and reports on the TBI infrastructure autonomously. Built on AWS EventBridge, Step Functions, Lambda, and Amazon Bedrock.

10.1 Architecture

Event-Driven Self-Healing: CloudWatch alarms, GuardDuty findings, WAF block events, honeypot hits, and health check failures flow through EventBridge into Step Functions workflows that orchestrate automated responses — restart tasks, block IPs, notify, or escalate.

10.1.1 Notification Routing (Unified)

All notifications flow through a single pipeline — no raw AWS emails.

CloudWatch Alarm (any of 21 alarms)
  ├─→ SNS: tbi-ops-notifications
  │     └─→ tbi-ops-notify Lambda
  │           └─→ Formatted HTML email (SES) → CoryDeanKalani@CPMP-Site.org
  └─→ EventBridge: tbi-ops-alarm-trigger
        └─→ Step Function: tbi-ops-health-check-heal
              └─→ Self-heal → verify → notify

GuardDuty Finding (severity ≥ 7)
  └─→ EventBridge → tbi-ops-bedrock-analyze → threat assessment → notify

Honeypot Hits (every 5 min)
  └─→ EventBridge → tbi-ops-honeypot-processor → WAF block → notify

Support Ticket
  └─→ tbi-raima-support → categorize → draft → notify

Daily/Weekly Digest
  └─→ EventBridge (cron) → tbi-ops-digest → Bedrock summary → notify

Component	Value
Primary SNS Topic	`tbi-ops-notifications` (arn:aws:sns:us-east-2:211998422884:tbi-ops-notifications)
Subscriber	`tbi-ops-notify` Lambda (protocol: lambda)
Email Sender	`CPMP Mission <No-Reply@CPMP-Site.org>` (SES)
Email Recipient	`CoryDeanKalani@CPMP-Site.org`
Severity Levels	[INFO], [WARNING], [CRITICAL], [SELF-HEALED]
Legacy Topic	`Trinity-Beast-Critical-Alerts` — retained for SMS escalation, no alarms attached

10.2 Lambda Functions

Function	Runtime	Memory	Purpose
`tbi-ops-notify`	Go (provided.al2023)	`1770 MB`	Send SNS/SES notifications with severity and context
`tbi-ops-self-heal`	Go (provided.al2023)	`1770 MB`	Restart ECS tasks, force-deploy services, health checks
`tbi-ops-waf-action`	Go (provided.al2023)	`1770 MB`	Block/unblock IPs in WAF IP set
`tbi-ops-honeypot-processor`	Go (provided.al2023)	`1770 MB`	Drain honeypot queue, apply WAF blocks, notify
`tbi-ops-bedrock-analyze`	Go (provided.al2023)	`1770 MB`	AI threat analysis via Bedrock (Claude Sonnet 4)
`tbi-raima-support`	Go (provided.al2023)	`1770 MB`	Raima Support Assistant — auto-categorize tickets, auto-reply in customer's language, knowledge gap detection (Claude Opus 4)
`tbi-ops-digest`	Go (provided.al2023)	`1770 MB`	Generate daily/weekly operational digests + newsletters via Bedrock (Claude Sonnet 4)

Environment Variables (AutoOps Lambdas): All 4 Lambdas that call the admin API read the admin key from an environment variable — no hardcoded keys. Updated via aws lambda update-function-configuration during key rotation (no rebuild needed).

Variable	Used By	Purpose
`ADMIN_KEY`	bedrock-analyze, bedrock-support, honeypot-processor, digest	Authenticates admin API calls (Valkey read/write, key-status). Read from `os.Getenv` at Lambda init.

10.3 Step Functions

State Machine	Trigger	Workflow
`tbi-ops-health-check-heal`	CloudWatch alarm via EventBridge	Check health → identify failed node → restart task → verify recovery → notify

10.4 EventBridge Rules

Rule	Pattern	Target
`tbi-ops-alarm-trigger`	CloudWatch alarm state change	Step Function: health-check-heal
`tbi-ops-honeypot-queue-processor`	Every 5 minutes (rate)	Lambda: honeypot-processor
`tbi-ops-bedrock-analyze-schedule`	Every 5 minutes (rate)	Lambda: bedrock-analyze
`tbi-ops-guardduty-high-finding`	GuardDuty Finding, severity ≥ 7	Lambda: bedrock-analyze
`tbi-ops-daily-digest`	`cron(0 11 * * ? *)` — 6 AM EST daily	Lambda: digest
`tbi-ops-weekly-digest`	`cron(0 12 ? * MON *)` — 7 AM EST Mondays	Lambda: digest

10.5 WAF Integration

IP Set: tbi-autoops-blocked-ips (ID: 8d55de25-8ba5-4982-8c41-f4316c9bd50d)
WAF Rule: AutoOps-BlockedIPs (priority 7 on trinity-beast-api-waf)
IPs are auto-blocked when honeypot traps detect repeat offenders (2+ hits from same IP).

10.6 AI Threat Analysis (Bedrock)

Model: us.anthropic.claude-sonnet-4-6 (inference profile)
Trigger: Scheduled every 5 min + GuardDuty HIGH/CRITICAL findings (severity ≥ 7)
Inputs: WAF blocks, honeypot hits, rate limits, usage anomalies, alarm state
Outputs: Severity assessment, pattern identification, recommendations, safe auto-actions (WAF IP blocks)
Principle: Bedrock is advisory — AI suggests, Step Functions decide, Lambda acts.

10.7 Predictive Operations (Layer 3)

CloudWatch Anomaly Detection — 4 ML-based anomaly detectors learning normal traffic patterns:

TrinityBeast-Anomaly-RequestRate — ALB RequestCount (both directions)
TrinityBeast-Anomaly-Latency — ALB TargetResponseTime (above only)
TrinityBeast-Anomaly-ErrorRate — ALB 5xx Count (above only)
TrinityBeast-Anomaly-CacheHitRate — ElastiCache CacheHitRate (below only)

All alarms feed into tbi-ops-alarm-trigger → Step Function → self-heal workflow.

10.8 Customer Engagement (Layer 4)

Lambda: tbi-raima-support
Model: us.anthropic.claude-sonnet-4-6 (inference profile)
Trigger: Invoked by application on support ticket submission
Actions: Auto-categorize (8 categories), auto-reply in customer's preferred_lang, pre-fetch key diagnostics, store analysis in Valkey, notify Cory
Branding: Responds as "The Trinity Beast AutoOps" — warm, professional, always offers human escalation to Cory Dean Kalani
Flow: Ticket submitted → Lambda invoked (async) → Bedrock categorizes + drafts → auto-reply posted → email sent to customer in their language → Cory notified with internal notes

10.9 Operational Digests (Layer 5)

Lambda: tbi-ops-digest
Model: us.anthropic.claude-sonnet-4-6 (inference profile)
Daily: 6 AM EST — 300-word operational summary
Weekly: Monday 7 AM EST — 500-word "Week in Review" newsletter to subscribers
Context source: Pre-digested text reports from Valkey (report:text:YYYY-MM-DD) — generated nightly by the sync job from HTML session reports
Flow: Gather stats + read text report → Bedrock narrative in subscriber's preferred_lang → store in Valkey → email via tbi-ops-notify

10.10 Design Principles

Self-heal first, notify second — if the system can fix it, fix it and tell Cory after
Escalate fast on unknowns — if the system doesn't know what to do, page immediately
Never suppress critical alerts — even if self-healing, always log and notify on CRITICAL
No raw AWS emails — every notification routes through tbi-ops-notify Lambda for formatted HTML delivery via SES. AWS User Notifications is disabled.
All actions logged — every autonomous action is recorded in Valkey for the daily digest
Cost-conscious — use Bedrock only when pattern correlation is needed, not for simple threshold checks

10.11 Estimated Cost

Service	Purpose	Monthly Cost
EventBridge	Event routing (6 rules)	~$0.50
Step Functions	Workflow orchestration	~$1.00
Lambda (7 functions)	Action execution	~$3–5
Amazon Bedrock	AI analysis + digests + support	~$3–5
CloudWatch Anomaly Detection	4 ML-based detectors	~$3.00
`SNS`	Notifications	~$0.10
Total		~$10–15