The Trinity Beast – AI AutoOps Implementation

1. Overview

The Trinity Beast Infrastructure (TBI) employs a multi-layered pro-active AI and automation system that monitors, defends, heals, and reports on the infrastructure autonomously. Rather than waiting for problems to escalate, the system detects anomalies in real-time, correlates threat signals using AI, and takes corrective action — all before a human needs to intervene.

AWS Services Utilized

2. Full System Diagram

The complete AutoOps pipeline showing all 7 Lambda functions, AI services, EventBridge rules, and data flows — with per-Lambda binary sizes and roles. See the legend below the diagram for common Lambda configuration shared across all functions.

flowchart TB
    subgraph Sources["Event Sources"]
        CWA["☁️ CloudWatch Alarms\n5xx spike · Health fail\nACU ceiling · Anomaly"]
        GD["🛡️ GuardDuty\nVPC flow · CloudTrail\nDNS · Severity ≥ 7"]
        HP["🪤 Honeypot Traps\n12 decoy endpoints\n2-hit auto-block threshold"]
        SCHED["⏱️ Scheduled Events\nEvery 5 min · Daily 6AM\nMonday 7AM EST"]
    end

    subgraph EB["EventBridge (6 Rules)"]
        R1["tbi-ops-alarm-trigger\nCW alarm → Step Function"]
        R2["tbi-ops-honeypot-queue\nEvery 5 min → honeypot-processor"]
        R3["tbi-ops-bedrock-analyze-schedule\nEvery 5 min → bedrock-analyze"]
        R4["tbi-ops-guardduty-high-finding\nGuardDuty severity ≥ 7 → bedrock-analyze"]
        R5["tbi-ops-daily-digest\ncron 0 11 * * ? * → digest"]
        R6["tbi-ops-weekly-digest\ncron 0 12 ? * MON * → digest"]
    end

    subgraph SF["Step Functions"]
        SFN["tbi-ops-health-check-heal\nCheck → Wait 60s → Recheck\n→ Force-Deploy → Verify → Notify"]
    end

    subgraph LambdaLayer["Lambda Functions — Go · provided.al2023 · 1 vCPU · 1770 MB · 60s timeout · Not in VPC"]
        L1["tbi-ops-self-heal\n━━━━━━━━━━━━━━━━━━\n📦 8.5 MB\n🔧 ECS force-deploy\nrestart-task · check-health\nLogs to Valkey"]
        L2["tbi-ops-waf-action\n━━━━━━━━━━━━━━━━━━\n📦 8.2 MB\n🔧 block-ips · unblock-ips\nlist-blocked\nManages WAF IP set"]
        L3["tbi-ops-notify\n━━━━━━━━━━━━━━━━━━\n📦 8.3 MB\n🔧 Branded HTML email\nSES delivery · SNS fallback\n[INFO/WARN/CRIT/HEALED]"]
        L4["tbi-ops-honeypot-processor\n━━━━━━━━━━━━━━━━━━\n📦 8.2 MB\n🔧 Drain autoblock queue\nDeduplicate IPs\nInvoke waf-action + notify"]
        L5["tbi-ops-bedrock-analyze\n━━━━━━━━━━━━━━━━━━\n📦 8.5 MB\n🔧 Gather threat signals\nQuiet check → Bedrock AI\nAuto-act · Store · Notify"]
        L6["tbi-raima-support\n━━━━━━━━━━━━━━━━━━\n📦 8.5 MB\n🔧 Raima Support Assistant\nAI draft response · Knowledge gaps\nPre-fetch diagnostics · Notify"]
        L7["tbi-ops-digest\n━━━━━━━━━━━━━━━━━━\n📦 8.5 MB · ⏱️ 180s timeout\n🔧 Daily 300-word summary\nWeekly newsletter pipeline\nBedrock narrative · Notify"]
    end

    subgraph AI["Amazon Bedrock"]
        Claude["Claude Sonnet 4.6\nus.anthropic.claude-sonnet-4-6\nInference Profile · No setup\nPattern correlation\nThreat assessment\nSupport drafting\nOperational narratives"]
    end

    subgraph WAFDef["WAF Defense"]
        IPSet["tbi-autoops-blocked-ips\nWAF IP Set\nID: 8d55de25-8ba5-4982-8c41-f4316c9bd50d"]
        WRule["AutoOps-BlockedIPs\nPriority 7\ntrinity-beast-api-waf"]
    end

    subgraph NotifyLayer["Notifications"]
        SNS["tbi-ops-notifications\nSNS Topic\nARN: arn:aws:sns:us-east-2:211998422884:..."]
        SES["Amazon SES\nBranded HTML email\nNo-Reply@CPMP-Site.org → Cory"]
    end

    subgraph ValkeyState["Valkey State (ElastiCache)"]
        VK["autoops:threats:daily\nautoops:actions:log\nautoops:self-heal:count\nautoops:digest:daily · weekly\nsupport:ticket:{id}\nhoneypot:autoblock_queue\nreport:text:YYYY-MM-DD"]
    end

    subgraph Anomaly["Layer 3 — Anomaly Detection"]
        AD["4 CloudWatch ML Models\nRequestCount · TargetResponseTime\nHTTPCode_5XX · CacheHitRate\nBand width: 2σ · 3 eval periods"]
    end

    %% Event sources → EventBridge
    CWA -->|"alarm state change"| R1
    CWA -->|"alarm state change"| R3
    GD -->|"severity ≥ 7"| R4
    HP -->|"hit logged to queue"| R2
    SCHED --> R2
    SCHED --> R3
    SCHED --> R5
    SCHED --> R6
    AD -->|"anomaly alarm"| R1

    %% EventBridge → targets
    R1 --> SFN
    R2 --> L4
    R3 --> L5
    R4 --> L5
    R5 --> L7
    R6 --> L7

    %% Step Function → Lambdas
    SFN -->|"check-health / force-deploy"| L1
    SFN -->|"notify result"| L3

    %% Lambda → Lambda
    L4 -->|"invoke sync"| L2
    L4 -->|"invoke async"| L3
    L5 -->|"invoke async"| L3
    L5 -->|"auto-block sync"| L2
    L6 -->|"invoke async"| L3
    L7 -->|"invoke async"| L3

    %% Lambda → AI
    L5 -->|"InvokeModel"| Claude
    L6 -->|"InvokeModel"| Claude
    L7 -->|"InvokeModel"| Claude

    %% Lambda → WAF
    L2 --> IPSet
    IPSet --> WRule

    %% Lambda → Notifications
    L3 --> SNS
    SNS --> SES

    %% Lambda → Valkey
    L1 -->|"log action"| VK
    L4 -->|"drain queue"| VK
    L5 -->|"store threat report"| VK
    L6 -->|"store ticket analysis"| VK
    L7 -->|"store digest"| VK

    %% Styling
    style Sources fill:#1e293b,stroke:#ef4444,color:#e2e8f0
    style EB fill:#1e293b,stroke:#FF9900,color:#e2e8f0
    style SF fill:#1e293b,stroke:#60a5fa,color:#e2e8f0
    style LambdaLayer fill:#1e293b,stroke:#10b981,color:#e2e8f0
    style AI fill:#1e293b,stroke:#a78bfa,color:#e2e8f0
    style WAFDef fill:#1e293b,stroke:#ef4444,color:#e2e8f0
    style NotifyLayer fill:#1e293b,stroke:#f59e0b,color:#e2e8f0
    style ValkeyState fill:#1e293b,stroke:#06b6d4,color:#e2e8f0
    style Anomaly fill:#1e293b,stroke:#818cf8,color:#e2e8f0
        

3. Event-Driven Automation (EventBridge)

Amazon EventBridge is the central nervous system of AutoOps. It receives events from across the infrastructure and routes them to the appropriate automated response.

How It Works

graph LR
    A["CloudWatch Alarm
State: ALARM"] -->|"Event Pattern"| EB["EventBridge
Event Bus"]
    B["Scheduled
rate(5 minutes)"] -->|"Schedule"| EB
    EB -->|"tbi-ops-alarm-trigger"| SF["Step Function
health-check-heal"]
    EB -->|"tbi-ops-honeypot-queue"| L["Lambda
honeypot-processor"]

    style EB fill:#1e293b,stroke:#FF9900,color:#e2e8f0
    style SF fill:#1e293b,stroke:#60a5fa,color:#e2e8f0
    style L fill:#1e293b,stroke:#10b981,color:#e2e8f0
        

Active Rules

Future Rules (Planned)

• Aurora ACU ceiling → alert + auto-scale recommendation
• Certificate expiration → trigger renewal workflow

4. Workflow Orchestration (Step Functions)

AWS Step Functions orchestrate multi-step automated responses that require sequencing, branching, and retry logic. A single Lambda can restart a task — but a Step Function can check health, wait, recheck, decide whether to restart or escalate, verify recovery, and notify.

Active State Machine

Workflow Steps

5. AI Threat Analysis (Amazon Bedrock)

Amazon Bedrock provides the intelligence layer — correlating multiple weak signals into a coherent threat picture that simple threshold-based alarms would miss.

Model Configuration

graph TB
    subgraph Gather["Signal Gathering"]
        S1["WAF blocks (24h)"]
        S2["Honeypot hits"]
        S3["Alarm state"]
        S4["GuardDuty findings"]
        S5["Self-heal count"]
        S6["Recent IPs"]
    end

    subgraph Decide["Quiet Check"]
        QC{"All signals
below threshold?"}
    end

    subgraph Analyze["Bedrock Analysis"]
        BR["Claude Sonnet 4.6
System: Security Analyst
Input: Threat signals JSON
Output: Structured report"]
    end

    subgraph Act["Response Actions"]
        A1["Store report in Valkey"]
        A2["Notify if HIGH/CRITICAL"]
        A3["Execute safe auto-actions"]
    end

    S1 --> QC
    S2 --> QC
    S3 --> QC
    S4 --> QC
    S5 --> QC
    S6 --> QC
    QC -->|"Yes"| Skip["Log quiet · Skip Bedrock"]
    QC -->|"No"| BR
    BR --> A1
    BR --> A2
    BR --> A3

    style Gather fill:#1e293b,stroke:#ef4444,color:#e2e8f0
    style Decide fill:#1e293b,stroke:#f59e0b,color:#e2e8f0
    style Analyze fill:#1e293b,stroke:#a78bfa,color:#e2e8f0
    style Act fill:#1e293b,stroke:#10b981,color:#e2e8f0
        

What Bedrock Analyzes

Output Format

Cost Control

Bedrock is only invoked when signals exceed quiet thresholds (WAF blocks > 50, honeypot hits > 10, any alarm firing, any GuardDuty finding). During quiet periods, the Lambda logs quiet status and exits without calling Bedrock — keeping costs near zero.

6. Self-Healing Lambda Functions

Seven Go Lambda functions form the action layer of AutoOps. Each is a standalone binary, cross-compiled for Linux, running on the provided.al2023 runtime at 1770 MB memory (multiple of 3 — fast execution, no cold-start pain).

Function Inventory

All functions share the same base configuration: Go · provided.al2023 · 1 vCPU · 1770 MB · Not in VPC · IAM role: tbi-autonomous-ops-role.

Inter-Lambda Communication

Shared IAM Role

All 7 functions share tbi-autonomous-ops-role with permissions for: ECS (update-service, stop-task), WAF (update-ip-set), CloudWatch (get-metrics, describe-alarms), Bedrock (invoke-model), SES (send-email), SNS (publish), SQS (receive/delete), Secrets Manager (get-secret), and Lambda (invoke-function on tbi-ops-*).

7. Honeypot Defense System

12 decoy endpoints that no legitimate user would ever access. Any hit reveals a scanner, bot, or attacker probing for vulnerabilities.

Trap Endpoints

How It Works

8. WAF Automation

The WAF layer provides both static protection (managed rule groups) and dynamic, AI-driven blocking via the AutoOps IP set.

WAF Configuration

AutoOps WAF Integration

9. Notification & Alerting

All operational notifications flow through a branded email pipeline. No raw text, no ugly AWS formatting — every alert arrives as a professionally designed HTML email via Amazon SES.

Notification Flow

The tbi-ops-notify Lambda serves as the single gateway for all notifications. It accepts events from two paths:

• Direct invocation — other AutoOps Lambdas (bedrock-analyze, self-heal, digest, honeypot-processor) invoke it directly with structured event payloads
• SNS subscription — CloudWatch alarms and other AWS services publish to the SNS topic, which triggers the Lambda. The Lambda parses the SNS envelope, detects the message type (alarm, generic), and normalizes it into a structured notification.

graph LR
    subgraph Sources["Event Sources"]
        CW["CloudWatch Alarms"]
        BA["tbi-ops-bedrock-analyze"]
        SH["tbi-ops-self-heal"]
        DG["tbi-ops-digest"]
        HP["tbi-ops-honeypot-processor"]
        SP["tbi-raima-support"]
    end

    subgraph Router["SNS Topic"]
        SNS["tbi-ops-notifications"]
    end

    subgraph Notify["Notification Lambda"]
        NL["tbi-ops-notify"]
        Parse["Parse Event Type"]
        Template["Build HTML Email"]
    end

    subgraph Delivery["Email Delivery"]
        SES["Amazon SES"]
        Email["Branded HTML Email
Dark theme · Structured sections
Severity badge · Action cards"]
    end

    subgraph Fallback["Fallback Path"]
        SNSFB["SNS Plain Text
(only if SES fails)"]
    end

    CW -->|"Alarm state change"| SNS
    SNS -->|"Lambda subscription"| NL
    BA -->|"Direct invoke"| NL
    SH -->|"Direct invoke"| NL
    DG -->|"Direct invoke"| NL
    HP -->|"Direct invoke"| NL
    SP -->|"Direct invoke"| NL
    NL --> Parse
    Parse --> Template
    Template --> SES
    SES --> Email
    Template -.->|"SES failure"| SNSFB

    style Sources fill:#1e293b,stroke:#60a5fa,color:#e2e8f0
    style Router fill:#1e293b,stroke:#FF9900,color:#e2e8f0
    style Notify fill:#1e293b,stroke:#10b981,color:#e2e8f0
    style Delivery fill:#1e293b,stroke:#a855f7,color:#e2e8f0
    style Fallback fill:#1e293b,stroke:#64748b,color:#94a3b8

    linkStyle 0 stroke:#FF9900
    linkStyle 1 stroke:#FF9900
    linkStyle 2 stroke:#10b981
    linkStyle 3 stroke:#10b981
    linkStyle 4 stroke:#10b981
    linkStyle 5 stroke:#10b981
    linkStyle 6 stroke:#10b981
    linkStyle 7 stroke:#60a5fa
    linkStyle 8 stroke:#60a5fa
    linkStyle 9 stroke:#a855f7
    linkStyle 10 stroke:#a855f7
    linkStyle 11 stroke:#64748b
        

SNS Topic

Email Delivery

Email Template Features

• Gmail-compatible dark theme — uses bgcolor HTML attributes on every <td> (Gmail strips CSS background properties)
• Solid hex colors only — no rgba() values (Gmail strips them entirely)
• Color-coded severity badge — INFO (blue), WARNING (amber), CRITICAL (red), SELF-HEALED (green)
• Auto-parsed sections — Summary, Patterns, Recommendations detected from detail text and rendered as structured HTML
• Bullet lists — lines starting with • or - render as actual <ul> lists
• Action Taken card — green left-border card showing what the system did
• Attention banner — red banner for CRITICAL events requiring human intervention
• Branded footer — "The Trinity Beast AutoOps · us-east-2 · The Trinity Beast Infrastructure"

Event Parsing

The Lambda auto-detects the incoming event format and normalizes it:

Severity Levels

Design Principles

• Single gateway — all notifications funnel through one Lambda, one template, one brand
• Never lose a notification — if SES fails, falls back to SNS plain text
• No raw AWS emails — SNS topic has zero email subscriptions; only Lambda subscription
• Consistent branding — same dark theme, same footer, same severity colors across all notification types
• Deduplication — Bedrock threat notifications have a 1-hour cooldown per severity level

10. Custom AI Translation Engine (Bedrock)

A custom Bedrock-powered translation engine replaces AWS Translate for all document translation. The engine uses sentinel preprocessing, multi-layer validation, and Step Function orchestration to produce structurally correct translations that preserve code blocks, diagrams, and brand terminology.

Translation Coverage

Architecture

Key Innovations

• Sentinel preprocessing — three-pass system replaces protected content with placeholder tokens before the model sees it. The model cannot corrupt what it never sees. 57 protected brand terms, version numbers, URLs, ARNs, and numeric values are all shielded.
• Multi-layer validation — every translated chunk is validated for tag count integrity, protected term survival, version number preservation, and HTML structure. Failures trigger automatic retries with temperature jitter.
• ECS Fargate worker — no timeout ceiling. Large documents (140+ KB) translate to completion regardless of processing time. The worker runs as a 1 vCPU / 3 GB Fargate task invoked by the Step Function via ecs:runTask.sync.
• Fire-and-forget operation — a single API call translates any document into all 11 languages, deploys to S3, invalidates CloudFront, rebuilds the search index, and emails a summary.

Translation Protection

11. Observability & Monitoring

The monitoring layer feeds events into AutoOps and provides the data that Bedrock analyzes.

CloudWatch Alarms

GuardDuty

Public Infrastructure Endpoint

All monitoring data is exposed via GET /public/infrastructure — a single unauthenticated endpoint that powers the Infrastructure Live page. Cached 60s in Valkey. Includes CloudFront, WAF, SQS, Lambda, Sync, GuardDuty, Alarms, Honeypot, and AutoOps stats.

12. Cost Analysis

The entire AI and automation layer runs for approximately $10–15/month — a remarkably low cost for a fully autonomous, AI-powered operations system running 24/7.

13. Future Layers

All 5 layers of the AutoOps system are fully deployed and operational.

Layer 3: Predictive Operations (CloudWatch Anomaly Detection)

• 4 Anomaly Detectors: Request rate, latency, 5xx error rate, cache hit rate
• 4 Anomaly Alarms: TrinityBeast-Anomaly-RequestRate, TrinityBeast-Anomaly-Latency, TrinityBeast-Anomaly-ErrorRate, TrinityBeast-Anomaly-CacheHitRate
• Band width: 2 standard deviations — tight enough to catch real issues, loose enough to avoid noise
• Evaluation: 3 periods, 2 datapoints to alarm, treat missing as notBreaching
• All alarms → SNS → EventBridge tbi-ops-alarm-trigger → Step Function health-check-heal
• ML models need ~2 weeks to build baseline, then detect deviations from learned patterns

Layer 4: Customer Engagement — Raima (tbi-raima-support)

• Auto-categorize tickets: billing, technical, api-access, rate-limit, feature-request, bug-report, account, general
• Auto-draft responses in customer's preferred_lang via Bedrock
• Pre-fetch diagnostic info (key status, usage, rate limits, last seen) before Cory reads ticket
• Store analysis in Valkey (support:ticket:{id})
• Notify Cory with category, priority, internal notes, and draft response
• Triggered by application when support ticket is submitted

Layer 5: Self-Documenting Infrastructure (tbi-ops-digest)

• Daily digest: 6 AM EST — concise 300-word operational summary
• Weekly digest + newsletter: Monday 7 AM EST — generates subscriber newsletter from 7 days of daily reports
• EventBridge rules: tbi-ops-daily-digest + tbi-ops-weekly-digest
• Gathers stats from /public/infrastructure endpoint
• Generates narrative via Bedrock (Claude Sonnet 4.6)
• Stores in Valkey (autoops:digest:daily, autoops:digest:weekly)
• Sends via tbi-ops-notify (email digest to Cory)

Weekly Newsletter Data Pipeline

The weekly newsletter uses daily operations reports as its primary source of truth — not just a point-in-time snapshot:

1. Session reports generated — Kiro saves HTML reports to ~/daily-reports/ and uploads to s3://trinity-beast-website-east2/daily-reports/
2. Nightly sync job — reads HTML reports from S3, strips tags to plain text, stores in Valkey as report:text:YYYY-MM-DD (30-day TTL). Backfills all dates from manifest that don't already exist.
3. Weekly digest Lambda — on Monday 8 AM EDT, fetches last 7 days of report:text:* from Valkey (up to 12 KB per day)
4. Bedrock generation — daily reports fed as primary context with accuracy guardrails (never fabricate data, never speculate on missing metrics)
5. Save + translate + send — newsletter saved to Aurora, auto-translated to 11 languages, sent to all active subscribers via SES