The Trinity Beast – TBI Translation Engine

1. Why a Custom Translation Engine

The Trinity Beast Infrastructure maintains 44 technical documents, 43 of them translated into 11 languages — 473 translated files in total. The original approach used AWS Translate batch jobs. It worked for simple prose but failed catastrophically on technical documentation.

1.1 Where AWS Translate Fails

AWS Translate is a neural machine translation service optimized for general-purpose text. Technical documentation with embedded code, diagrams, and brand terminology exposes its fundamental limitations:

1.2 The Scale Problem

With 40 documents × 11 languages, every documentation update triggers a translation cascade. Before the custom engine:

• Each document required manual post-processing to fix code blocks, diagrams, and brand names
• A single document update meant re-translating and re-fixing 11 language versions
• No audit trail — local log files only, no provenance tracking
• No retry mechanism — a failed translation required starting over from scratch
• No cost visibility — Bedrock spend was invisible until the monthly bill
• Translation was becoming a full-time job that blocked documentation improvements

1.3 The Solution

A custom Bedrock-powered translation engine that understands the boundary between human language and machine language. The engine uses defense-in-depth across the full pipeline:

• Source validation — catches structural defects, encoding issues, and Mermaid syntax errors before burning any Bedrock tokens. Auto-repairs what it can, rejects early with actionable reports when it can't.
• Language detection — Unicode script analysis and word frequency heuristics (21 languages, no API calls, <10ms). A detection only counts when it is definitive: English by word frequency, Japanese by kana, Korean by Hangul and Thai by script, at confidence 0.8 or higher. Anything else is a script-family guess (Cyrillic, Arabic script, Devanagari, Han) and resolves to auto, so a wrong guess never puts the wrong language in the prompt. A customer-stated or page-declared source always wins, and detection never raises and never rejects a source. No pivot through English required.
• Sentinel preprocessing — replaces protected content with placeholder tokens before the model sees it, then restores them after translation. The model cannot corrupt what it never sees.
• Smart splitting — 24 KB max part size prevents model output truncation. Per-language code tag thresholds handle complex-script sensitivity.
• Multi-layer validation — every translated chunk is validated against the source for structural integrity, protected term preservation, version number survival, and HTML tag count matching. Failures trigger automatic retries with temperature jitter.
• Integrity check with diagram auto-stitch — full-document post-translation repair. Counts Mermaid diagrams, stitches missing ones back from source, repairs broken span/strong/em wrappers.
• Event-driven orchestration — one SQS message per language, picked up by a persistent ECS worker service that scales to one container per language of the job. Retries, deployment, search index rebuilding and notification follow from the worker's events and the LPO's /admin/translate/complete step. Fire-and-forget from the operator's perspective.

2. Architecture

2.1 Pipeline Flow

The translation service is an event-driven pipeline that decouples submission from execution. The operator submits a job; the system handles everything else asynchronously.

flowchart TB
    subgraph IN["Entry Points"]
        A["POST /admin/translate"]
        A2["Customer accept: staged sources, quote job"]
    end
    subgraph LPO["LPO Server (Go)"]
        B["Validate and enqueue: 127-language catalog, up to 12 languages, 42 docs"]
        C["Valkey: live state"]
        D["Aurora: job row inserted before any send"]
    end
    subgraph AQ["AWS Queue"]
        E["SQS Queue: one message per language"]
    end
    subgraph BTS["BeastTranslate — Persistent ECS Service"]
        direction TB
        BT["SQS Long-Poll Loop"]
        BT --> SC{"Scale Check"}
        SC -->|Single lang| BT2["Process In-Place"]
        SC -->|Multi-lang| BT3["Scale Service to N, at most 12"]
        BT3 --> BT4["N Containers Poll Same Queue"]
        BT4 --> BT5["Each Takes 1 Language Message"]
        BT2 --> H00
        BT5 --> H00
        subgraph TI["Translation Intelligence (Python)"]
            direction LR
            H00["Language Runtime: catalog row + profile, 15-min cache, snapshot fallback"]
            H0["Source Validation"]
            H1["Complexity Analysis"]
            H2["Document Preprocessor"]
            H3["Sentinel System — 4 Types"]
            H4["Bedrock — 3-Region Failover"]
            H5["Validator — Hard + Soft Tiers"]
            H6["Integrity Check + Auto-Repair"]
            H00 --> H0 --> H1 --> H2 --> H3 --> H4 --> H5 --> H6
        end
    end
    subgraph DEL["Delivery"]
        direction LR
        I["Worker writes S3"]
        J["CloudFront Invalidation (library docs)"]
        K["POST /admin/translate/complete: state, search rebuild, email, scale-down"]
    end
    A2 -->|calls| A
    A --> B
    B --> C
    B --> D
    B --> E
    E --> BT
    H6 --> I
    I --> J
    I --> K
    style IN fill:#1e293b,stroke:#60a5fa,color:#e2e8f0
    style LPO fill:#1e293b,stroke:#94a3b8,color:#e2e8f0
    style AQ fill:#1e293b,stroke:#a78bfa,color:#e2e8f0
    style BTS fill:#1e293b,stroke:#fb923c,color:#e2e8f0
    style TI fill:#253347,stroke:#fbbf24,color:#e2e8f0
    style DEL fill:#1e293b,stroke:#10b981,color:#e2e8f0
    style A fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style A2 fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style B fill:#334155,stroke:#94a3b8,color:#e2e8f0
    style C fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style D fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style E fill:#2e1065,stroke:#a78bfa,color:#e2e8f0
    style BT fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style SC fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style BT2 fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style BT3 fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style BT4 fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style BT5 fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style H00 fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style H0 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style H1 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style H2 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style H3 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style H4 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style H5 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style H6 fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style I fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style J fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style K fill:#064e3b,stroke:#10b981,color:#e2e8f0
    linkStyle 0,1,2,3,4,5,6 stroke:#fb923c,stroke-width:2px
    linkStyle 7,8,9,10,11,12,13 stroke:#fbbf24,stroke-width:2px
    linkStyle 14,15 stroke:#60a5fa,stroke-width:2px
    linkStyle 16,17 stroke:#10b981,stroke-width:1.5px
    linkStyle 18,19 stroke:#a78bfa,stroke-width:2.5px
    linkStyle 20,21,22 stroke:#10b981,stroke-width:2px
        

2.2 Components

2.3 Why Python (The Only Python in the Fleet)

Every other compute workload in The Trinity Beast Infrastructure is written in Go. The translation worker is the sole exception, and for good reason:

• lxml for HTML parsing — Go's HTML parsers are adequate for simple tasks but lack the XPath and tree-manipulation capabilities needed for sentinel preprocessing on complex nested documents.
• Battle-tested engine — the translator package was developed and debugged over weeks of production use. Rewriting in Go would re-introduce every bug already fixed (56+ smoke test cases).
• Rapid iteration — prompt engineering and validator tuning require fast feedback loops. Python's interpreted nature allows testing changes without compile cycles.
• Separation of concerns — the translation engine is a self-contained package with its own config, prompts, validators, and chunker. Keeping it in Python isolates it from the Go service layer.
• Container image deployment — ships as a Docker image to ECR, runs as an ECS Fargate task with no timeout ceiling. The same image also supports Lambda invocation for smaller documents.

3. Sentinel Preprocessing System

flowchart TD
    A[Source HTML Chunk] --> B[Pass 1: Extract translate=no elements]
    B --> B1b[Pass 1b: Extract class-bearing spans as FULL sentinels]
    B1b --> B2[Pass 2: Extract all text-only spans as FULL sentinels]
    B2 --> B3[Pass 2b: Wrap text-only links as paired sentinels]
    B3 --> B4[Pass 2c: Extract nested-HTML and event-handler links as FULL]
    B4 --> D[Pass 3: Replace bare numbers with numeric sentinels]
    D --> D2[Pass 4: Replace brand terms with TERM sentinels]
    D2 --> D3[Pass 5: Extract email addresses as FULL sentinels]
    D3 --> D4[Pass 6: Extract bare prose URLs as FULL sentinels]
    D4 --> E[Send to Bedrock with sentinel-aware prompt]
    E --> F[Receive translated chunk with sentinels intact]
    F --> G[Deduplicate any model-doubled paired sentinels]
    G --> H[Restore sentinels high-to-low index order]
    H --> H2[Recovery pass: re-inject any dropped FULL/TERM sentinels]
    H2 --> H3[Repair pass: fix dropped span/strong/em tags]
    H3 --> I[Run validators against source + restored output]
    I -->|PASS| J[Accept chunk]
    I -->|FAIL| K{Retries remaining?}
    K -->|Yes| L[Retry with strict prompt + temperature jitter]
    L --> E
    K -->|No| M[Raise TranslationError]

    style A fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style E fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style J fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style M fill:#450a0a,stroke:#ef4444,color:#e2e8f0
        

4. Validator System

Every translated chunk is validated against the source before acceptance. Validators enforce structural integrity and content preservation — if a translation passes all validators, it is guaranteed to be functionally correct (code works, links resolve, diagrams render).

4.1 Validation Checks

4.2 Retry Strategy

When validation fails, the engine retries with two progressive adjustments:

1. Strict prompt activation — adds an explicit warning: "PREVIOUS ATTEMPT FAILED VALIDATION. Be more careful: every protected term and every version number from the input MUST appear unchanged in the output."
2. Temperature jitter — increments temperature by 0.1 per retry (0.0 → 0.1 → 0.2 → 0.3, capped at 0.5). A deterministic temp=0 retry produces the same erroneous output; temperature jitter lets the model take a different sampling path.

Underneath every attempt, each Bedrock call has its own size and its own clock (October 2026). A call asks for an output cap sized from its input: the estimated input tokens × the target language's measured output ratio × 1.5, never below 4,096 and never above 49,152 tokens. Its read timeout is sized from that cap: 60 seconds plus the cap divided by the model's slowest measured speed (35 tokens per second for Qwen, 25 for Claude), rounded up to the minute and never above 900 seconds. A small chunk therefore waits minutes, not a quarter of an hour, before a stalled call is abandoned.

If a reply stops exactly at its cap, the call is repeated once at the full 49,152-token ceiling (CAP-ESCALATED in the log), so a legitimately long translation is never cut short. That escalation happens at most once per chunk: every later attempt of the same chunk starts at the raised cap and is never raised again. The SDK's own retries are switched off, so the validation loop above and the regional failover in §12 are the only retry layers.

Maximum retries: 3, set per language in translation_language_profiles.max_retries and read at job start, with the value in translation-config.json as the fallback when Aurora is unreachable. If all attempts fail, a TranslationError is raised with the chunk index, validator detail, and a preview of the problematic chunk.

4.3 Hard vs Soft Failures

Validators are classified into two tiers based on what they protect:

This tiered approach eliminates the failure mode where a correctly-translated document is rejected because the model dropped a single decorative <span> wrapper during RTL reordering. The content is correct — only the styling wrapper is missing — and the integrity check restores it automatically.

The ValidationReport aggregates all results and exposes:

• .passed — True if zero hard failures
• .hard_failures — list of blocking issues (content-critical tags)
• .soft_failures — list of warnings (decorative tags — repaired post-translation)
• .summary() — human-readable status string

4.4 Post-Translation Integrity Check

After translation completes and chunks are reassembled, a full-document integrity check runs before the S3 write. This is the defense-in-depth layer — it repairs structural drift that the per-chunk validator intentionally allows through (soft failures).

Repair Capabilities

The integrity check only repairs translate="no" elements (where content is byte-for-byte identical between source and output). For translated content that lost its wrapper, the check logs the discrepancy but cannot reliably re-wrap (the content has been translated — matching it to the source wrapper requires semantic understanding).

4.5 Source Document Validation (v2.8)

Before any translation work begins, the source document passes through a validation gate. This catches defects that would cause translation failures or produce broken output — rejecting early saves Bedrock tokens and prevents corrupted translations from reaching S3.

Defect Categories

Validation Flow

1. Size check — reject if > 500,000 characters (chunking becomes unreliable at this size). This is the worker's backstop and it rejects only internal jobs: customer quote jobs run with skip_validation: true, so this check never rejects them, though its repairs still apply. Customer documents are already bounded by the separate 300 KB (307,200-byte) cap, enforced by the LPO at the quote and again at the accept re-fetch. The two limits measure different things and are never the same number.
2. Encoding check — detect and strip BOM markers, null bytes; flag mixed encodings
3. Structural HTML check — scan for unclosed tags; auto-repair up to 5 by appending closing tags at the correct nesting level
4. Mermaid syntax check — validate every <pre class="mermaid"> block has a valid diagram type, balanced brackets, and non-empty content
5. Conflict check — reject if the root <body> or <html> element has translate="no"

Rejection vs Repair

The validator follows a strict philosophy: try to fix it silently, reject early if you can't. Repairable issues (unclosed tags, BOM markers) are fixed in-place — the customer never knows. Unrecoverable issues produce an actionable defect report with the exact location, what's wrong, and how to fix it. For customer quote jobs (skip_validation: true) the repairs run and the rejections do not, because those documents were already checked at the quote and at accept; internal jobs get both.

ValidationResult:
  valid: false
  rejection_reason: "2 unrecoverable defects found"
  defects:
    - severity: error
      category: MERMAID
      location: "Section 5, line 342"
      description: "Empty Mermaid block — no diagram content"
      suggestion: "Add diagram content or remove the empty <pre class='mermaid'> block"
    - severity: error
      category: SIZE
      location: "Document root"
      description: "Document is 626,688 characters (612 KB). Maximum is 500,000 characters (488 KB)."
      suggestion: "Split into multiple documents or remove large embedded assets"

4.6 Diagram Integrity (v2.8)

Mermaid diagrams are code — they must survive translation byte-for-byte. The integrity check (section 4.4) now includes dedicated diagram verification with automatic recovery.

Detection

The integrity check counts Mermaid blocks in the source (<pre class="mermaid">) and compares against the translated output. If any diagrams are missing from the output, the auto-stitch mechanism activates.

Auto-Stitch Recovery

When a diagram is missing from the translated output:

1. Identify which source diagram is absent (by content matching)
2. Extract the full <div class="diagram-wrap"> block from source (includes label + pre)
3. Locate the correct insertion point in the output (same section, same relative position)
4. Inject the source diagram block verbatim — diagrams don't need translation

The stitched diagram is the English version, which is functionally correct — Mermaid syntax is language-independent. The surrounding prose is already translated, so the reader gets translated explanations with a working diagram.

Tag Inventory Integration

The _count_tags function now reports diagram count alongside other structural tags:

Tag Inventory (source → output):
• Trinity-Beast-Performance-Report.html
  IN:  code:75 pre:8 strong:12 em:3 a:6 br:20 diagrams:4
  OUT: code:75 pre:8 strong:12 em:3 a:6 br:20 diagrams:4

If a diagram is lost during translation and auto-stitched back, the final count still matches — the stitch happens before the tag inventory is calculated. A mismatch in the diagrams count after stitching indicates a structural issue that needs manual review.

4.7 Post-Translation Repair Pipeline (v5.0)

Defense-in-depth: even with perfect preprocessing, models occasionally drop or mangle structural elements. The repair pipeline runs after translation and before final validation, catching anything that slipped through.

Code Tag Repair (_repair_code_tags)

Multi-pass repair (up to 6 iterations) that detects dropped <code> wrappers by comparing the translated output against the source document. For each code-tagged value in the source, it searches the output for the bare content and re-wraps it with the original tag. Converges when no more repairs are possible.

The repair function matches content from the source's code tags against the output using exact string matching. It handles both self-closing patterns and open/close pairs. Each pass may reveal new repair opportunities (nested cases), hence the multi-pass design with a convergence check.

lxml Syntax Validation (_tidy_validate_and_repair)

Structural HTML repair using lxml's robust parser. Catches and fixes issues that the model introduces in the HTML structure itself — unclosed tags, nesting violations, mismatched attributes, and malformed output. This is the final safety net before the translated chunk is accepted.

Uses lxml.html.fragment_fromstring() for parsing — no external binary needed (lxml is in the container's requirements.txt). Falls back to BeautifulSoup if lxml encounters a parse failure it cannot recover from.

Output Integrity Check (_check_output_integrity)

Final gate before a translated chunk is accepted. Compares structural tag counts between source and output — every <code>, <pre>, <strong>, <em>, <a>, <br>, and Mermaid diagram must have matching counts. If counts diverge beyond tolerance, the _recover_missing_wrappers() function attempts targeted restoration before rejecting the chunk.

Per-Language Tag Recovery (tag_recovery/ module — v5.1)

Different languages exhibit different tag-drop patterns. Arabic and Urdu (RTL) tend to drop trailing <br> tags because their paragraph flow doesn't rely on line breaks the same way English does. Hindi sometimes splits or merges <strong> spans when the translated phrase length differs from English. CJK languages occasionally omit <br> at list boundaries. Latin/Cyrillic languages rarely have issues.

Rather than applying a one-size-fits-all fixer, the tag recovery system dispatches to per-language Python scripts — one file per supported language, each encoding the known behavioral patterns for that script type:

Flow: After Bedrock returns a translated chunk, the dispatcher (tag_recovery/__init__.py) counts tags in both source and output using common.py utilities. If counts match, it skips. If they differ, it loads the language-specific fixer and applies deterministic repairs:

1. Count — count_tags() counts <br>, <strong>, <em>, <code>, <a>, <span>, <pre>
2. Diff — Negative = missing, Positive = extra, Zero = match
3. Fix — Language-specific: inject missing <br> at sentence boundaries, wrap unwrapped <strong> terms, remove spurious extras
4. Re-count — Confirm fix resolved the diff

All operations use direct string comparison — no regex. The convention: tags are structural, deterministic, and countable. Either the bytes are there or they aren't. Each language file is small, focused, and independently extensible as new patterns emerge.

4.8 Prose-Mode Fallback (v5.2)

When the normal translation path (preprocessor + sentinel + Bedrock) produces catastrophic code tag loss, the system automatically re-translates the affected part through prose mode — a fundamentally different translation strategy that preserves structural elements by design.

Trigger Condition

The hint validation system counts <code> tags in each translated part immediately after Bedrock returns. If the drift exceeds tolerance, the fallback fires:

actual_code = count(<code> tags in translated output)
drift = actual_code - hint_code
tolerance = max(3, hint_code × 0.15)

if abs(drift) > tolerance:
    → FALLBACK: re-translate this part via prose mode

Fallback Flow

1. Detect — Hint validation catches drift beyond tolerance (e.g., expected 15 code tags, got 0)
2. Extract — Raw-extract all <pre translate="no"> blocks from the source part (byte-perfect preservation)
3. Fragment — extract_prose() walks the DOM and extracts only translatable text fragments, leaving all HTML structure untouched
4. Translate — Fragments are sent to Bedrock as a numbered list (plain text, no HTML) — the model cannot damage structure it never sees
5. Rebuild — rebuild_with_translations() inserts translated fragments back into the original DOM skeleton
6. Restore — Pre-extracted blocks are placed back at their original positions
7. Validate — Re-count code tags in the fallback output; accept if within tolerance, keep original if fallback also fails

Why Prose Mode Preserves Code Tags

The normal path sends full HTML to Bedrock — the model must understand and preserve all structural elements while translating. Prose mode inverts this: it never shows the model any HTML. The DOM is the skeleton; only the text content moves through the model. Code tags, pre blocks, spans, and all structural HTML remain in the skeleton untouched — they were never at risk.

Example: Partner-Onboarding × French

Part 3 of this document has density=high, code=15. On the normal path, Qwen consistently dropped all 15 code tags (drift -15). The fallback re-translates through prose mode — 15/15 code tags preserved, validation passes, job succeeds.

5. BeastTranslate — Persistent Worker Architecture

Translation execution is handled by BeastTranslate — a persistent ECS Fargate service (tbi-translate-worker-service) that continuously polls the SQS translation queue for jobs. Unlike the previous Step Function → RunTask model (v3.0), BeastTranslate is always-on — no cold starts, no orchestration overhead, instant job pickup.

5.1 BeastTranslate Service Design (v3.1)

flowchart TD
    SQS["SQS Translation Queue: one message per language"] --> LP["Long-Poll Loop - 20s wait"]
    LP -->|Message received| DM["Deserialize Job Message"]
    DM --> EM{"Multi-Language?"}
    EM -->|Single lang| RT["Process In-Place"]
    EM -->|Multi-lang| BT["Scale Service to N Containers, at most 12"]
    BT --> PAR["N Containers Poll Same Queue"]
    PAR --> EACH["Each Takes 1 Language Message"]
    EACH --> LRT["Load Language Runtime: catalog row + profile, 15-min cache"]
    RT --> LRT
    LRT --> PROC["Translate All Docs for That Language"]
    PROC --> EV["Per-pair events: POST /admin/translate/event/job_id"]
    EV --> CC{"Before publish: job cancelled?"}
    CC -->|No| PUB["Write to S3"]
    CC -->|Yes| CXL["Stop: lang_cancelled, nothing published"]
    CC -->|Customer job, state unknown| UNK["Fail closed: lang_failed cancel-state-unknown"]
    PUB --> LAST{"Last worker of the job?"}
    CXL --> LAST
    UNK --> LAST
    LAST -->|Yes| CMP["POST /admin/translate/complete/job_id"]
    LAST -->|No| DONE
    CMP --> DONE{"More Messages?"}
    DONE -->|Yes| LP
    DONE -->|No - queue empty| IDLE["IDLE - Resume Polling"]
    IDLE --> LP

    subgraph GS["Graceful Shutdown"]
        SIG["SIGTERM Received"] --> FIN["Finish Current Doc"]
        FIN --> CLEANEX["Clean Exit"]
    end

    subgraph VH["Visibility Heartbeat"]
        HB["Every 5 min"] --> EXTVIS["Extend SQS Visibility"]
        EXTVIS --> PREV["Prevent Re-Delivery"]
    end

    style GS fill:#1e293b,stroke:#c084fc,color:#e2e8f0
    style VH fill:#1e293b,stroke:#60a5fa,color:#e2e8f0
    style SQS fill:#2e1065,stroke:#a78bfa,color:#e2e8f0
    style LP fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style DM fill:#7c2d12,stroke:#fb923c,color:#e2e8f0
    style EM fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style RT fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style BT fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style PAR fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style EACH fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style LRT fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style PROC fill:#92400e,stroke:#fbbf24,color:#e2e8f0
    style EV fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style CC fill:#7f1d1d,stroke:#f87171,color:#e2e8f0
    style PUB fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style CXL fill:#3f3f46,stroke:#a1a1aa,color:#e2e8f0
    style UNK fill:#7f1d1d,stroke:#f87171,color:#e2e8f0
    style LAST fill:#1e293b,stroke:#334155,color:#e2e8f0
    style CMP fill:#064e3b,stroke:#10b981,color:#e2e8f0
    style DONE fill:#1e293b,stroke:#334155,color:#e2e8f0
    style IDLE fill:#1e293b,stroke:#334155,color:#e2e8f0
    style SIG fill:#4c1d95,stroke:#c084fc,color:#e2e8f0
    style FIN fill:#4c1d95,stroke:#c084fc,color:#e2e8f0
    style CLEANEX fill:#4c1d95,stroke:#c084fc,color:#e2e8f0
    style HB fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style EXTVIS fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    style PREV fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    linkStyle 0,1 stroke:#a78bfa,stroke-width:2px
    linkStyle 11,12,15,18,20 stroke:#10b981,stroke-width:2px
    linkStyle 14,17 stroke:#f87171,stroke-width:2px
        

Service Specification

How It Works: The Polling Loop

BeastTranslate runs a continuous polling loop. When no jobs exist, it idles — consuming negligible CPU while maintaining a warm connection to SQS. The moment a translation job is submitted, the container picks it up within seconds (no cold start, no orchestration delay).

1. SQS Poll: receive_message(WaitTimeSeconds=20) — blocks for up to 20 seconds waiting for a message
2. Receive: Deserialize job envelope (job_id, docs[], language, options)
3. Process: For each document in the array, run the full translation pipeline (sentinel → Bedrock → validate → integrity check)
4. Heartbeat: Every 5 minutes, extend the SQS message visibility timeout to prevent re-delivery during long-running documents
5. Report: Each pair reports its events to POST /admin/translate/event/{job_id}, and the last worker of the job calls POST /admin/translate/complete/{job_id}
6. Delete: Remove the message from SQS after successful processing
7. Loop: Return to step 1

Auto-Scaling: Demand-Driven Container Management

BeastTranslate scales itself automatically — no manual intervention required. The system determines container count from the job's language list at submission time, then scales back to 1 when all work completes.

How It Determines Scale

When a translation job is submitted via POST /admin/translate, the LPO server inspects the langs array:

1. Count languages: desired = len(langs) — one container per language for full parallelism
2. Cap at 12: txlimits.MaxLangsPerQuote, the same constant that caps a quote, so a 12-language job never leaves its 12th language waiting for a free container
3. Scale up: ecs:UpdateService(desiredCount=N) fires immediately after the job is enqueued to SQS
4. Containers ready: New containers start from ECR image cache in ~30 seconds

How It Determines Scale-Down

When a worker completes a job, the finalize step checks for remaining work before scaling down:

1. Query active queue: GET /admin/translate/queue — checks active and queued job counts
2. If queue empty: ecs:UpdateService(desiredCount=1) — scale back to steady-state
3. If jobs remain: Skip scale-down — let other workers continue processing

This handles back-to-back batch submissions (e.g., 6 docs + 6 docs) correctly — the first batch to finish won't tear down containers while the second is still running.

Graceful Shutdown (SIGTERM)

When ECS sends SIGTERM (scale-down or deployment), the container:

1. Stops polling for new messages immediately
2. Finishes translating the current document (never abandons mid-document)
3. POSTs partial progress back via callback
4. Exits cleanly with code 0

This ensures no work is lost during scale-down events. A partially-completed job resumes from where it left off when the next container picks up remaining messages.

Backward Compatibility

The worker runs in persistent polling mode, continuously polling SQS for translation messages. Each message = one language × all docs for that job. Auto-scaling provides parallelism across languages.

sequenceDiagram
    participant Admin as Operator or Customer accept
    participant LPO as LPO Server
    participant ECS as ECS Service
    participant SQS as SQS Queue
    participant BT1 as BeastTranslate 1
    participant BT2 as BeastTranslate 2-12

    rect rgba(96, 165, 250, 0.14)
    Admin->>LPO: POST /admin/translate (3 docs x 12 langs)
    LPO->>LPO: Insert the job row before any send
    LPO->>SQS: 12 messages, one per language (each carries all 3 docs)
    LPO->>ECS: UpdateService(desiredCount=12)
    LPO->>Admin: 200 OK (job_id, state: queued)
    end

    Note over ECS: Spins up 11 new containers (~30s)
    Note over BT1: Already polling (always-on)
    BT1->>SQS: ReceiveMessage
    SQS-->>BT1: One language message (all 3 docs)
    BT2->>SQS: ReceiveMessage (competing consumers)
    SQS-->>BT2: One language message each

    rect rgba(251, 191, 36, 0.12)
    par 12 containers process in parallel
        BT1->>BT1: es - all 3 docs
        BT2->>BT2: fr, de, ru, ja, zh, ar, hi, ur, pt, it, ko
    end
    end

    BT1->>LPO: Per-pair events (lang_started, lang_succeeded, lang_failed)
    BT2->>LPO: Last worker: POST /admin/translate/complete/job_id
    rect rgba(52, 211, 153, 0.14)
    LPO->>LPO: Final state, search rebuild, queue check
    LPO->>ECS: UpdateService(desiredCount=1) once the queue is empty
    Note over ECS: 11 containers drain gracefully
    alt Customer job
        LPO->>Admin: Customer completion email, failed pairs go to the refund machine
    else Internal job
        LPO->>Admin: Ops SNS notification email
    end
    end
        

5.2 Error Handling and Recovery

5.3 Architecture Simplification (June 2026)

5.4 Self-Healing Sweeper

The sweeper runs automatically on every GET /admin/translate/health call (piggybacked) and is also available as a dedicated POST /admin/translate/sweep endpoint.

It scans all jobs in tx:active (the Valkey SET of active job IDs). For each job older than 15 minutes in queued or running state:

• Checks the Step Function execution status via the recorded ARN
• If FAILED, TIMED_OUT, or ABORTED → marks job as failed, removes from tx:active, updates Aurora with reason
• If no execution ARN recorded (pipe never triggered) → marks as failed
• If Step Function is still RUNNING → leaves it alone

All sweep actions are logged to translation_job_events for audit trail.

The Second Sweeper: Customer Refunds and Stranded Jobs

A second sweeper keeps the promises made to paying customers. It is an LPO timer, tx_failed_refund_sweep, that ticks every 15 minutes (refund.sweep_interval_minutes) and takes a transaction-scoped advisory lock so only one container sweeps at a time. Its step 0 finishes any customer job nobody finalized (TX-FINALIZE-CATCHUP), claiming the job's finalizing phase as a 15-minute lease so a tick that dies is picked up by the next one. The later steps close retry windows, issue refunds that are due, and refund a paid quote whose job never started. Every step is a re-runnable conditional update in Aurora, so a restart never loses or repeats work, and POST /admin/translate/refund-sweep runs one tick on demand.

5.5 Job Phase Transitions

The job state now reflects the exact phase of execution:

This gives real-time visibility into exactly where a job is in the pipeline.

6. Admin API (9 Endpoints)

All endpoints require the X-Admin-Key header. They are served by the LPO server (Go) alongside the existing admin routes.

6.1 Submit Translation Job

POST /admin/translate

Submits a new translation job. Validates inputs, checks cost limits, creates job state in Valkey, inserts the translation_jobs row synchronously (on a detached 60-second context, and under the quote row lock for a customer quote job) before any SQS send, then enqueues one message per language. first_enqueued_at marks the first successful send, which is what makes a row count as started.

// Request
POST /admin/translate
X-Admin-Key: tbcc-admin-...
X-Idempotency-Key: my-unique-key (optional)
Content-Type: application/json

{
  "docs": ["Trinity-Beast-API-Reference.html", "Trinity-Beast-Architecture-Guide.html"],
  "langs": "all",
  "options": {
    "force": false,
    "delta": false,
    "skip_search_rebuild": false,
    "skip_validation": false
  }
}

// Response 200
{
  "status": "✅ [LPO] [us-east-2] [BeastMain] [/admin/translate] [200]",
  "status_code": 200,
  "endpoint": "/admin/translate",
  "cluster_node": "BeastMain",
  "region": "us-east-2",
  "language": "en",
  "timestamp": "2026-05-16T16:42:00Z",
  "data": {
    "job_id": "1747407720-a3f8b2c1d4e5",
    "state": "queued",
    "submitted_at": "2026-05-16T16:42:00Z"
  },
  "error": ""
}

Validation rules:

• docs — required, 1-42 entries (txlimits.MaxDocsPerQuote; more answers 400 translate_max_docs), each must be a valid filename in S3
• langs — "all" (expands to the internal 11) or an array of 1-12 codes, each resolved through the 127-language catalog and its alternate codes (input shape 2-12 characters) and de-duplicated. An unknown code answers 400 translate_langs_invalid
• options.force — bypass known-failure guard and difficulty rejection
• options.delta — skip pairs where the translated file is already newer than the source (saves up to 90% on re-translation)
• Daily dollar spend must be under $600 (autoops:bedrock:spend:daily)
• Daily token usage must be under 50M combined tokens (autoops:bedrock:tokens:input:daily + autoops:bedrock:tokens:output:daily)
• Active jobs must be under 3 (additional jobs queue in SQS)

6.2 Monitoring Endpoints

GET /admin/translate/status/{job_id}

Returns the full job state. Aurora is the primary source — state, timestamps, docs, langs, cost, and Step Function ARN are read from translation_jobs. Real-time per-doc/lang progress is overlaid from Valkey (written per-pair by the worker, too frequent for Aurora writes). If Aurora doesn't have the job yet (async insert still pending), falls back to Valkey.

GET /admin/translate/queue

Lists all pending and active jobs (state in queued or running).

GET /admin/translate/history

Returns the last 50 completed jobs from translation_jobs in Aurora, ordered by submission date descending. Includes state, docs, succeeded/failed pair counts, cost, and reason. Falls back to the Valkey tx:history list if Aurora is unavailable.

GET /admin/translate/health

System health overview:

{
  "status": "✅ [LPO] [us-east-2] [BeastMain] [/admin/translate/health] [200]",
  "status_code": 200,
  "endpoint": "/admin/translate/health",
  "cluster_node": "BeastMain",
  "region": "us-east-2",
  "language": "en",
  "timestamp": "2026-05-16T17:30:00Z",
  "data": {
    "queue_depth": 0,
    "active_jobs": 1,
    "last_completed_at": "2026-05-16T17:30:00Z",
    "last_state": "succeeded",
    "daily_spend_usd": "12.40",
    "daily_spend_limit_usd": "600.00",
    "daily_input_tokens": 284150,
    "daily_output_tokens": 312480,
    "daily_token_limit": 50000000,
    "swept_jobs": 0
  },
  "error": ""
}

6.3 Control Endpoints

POST /admin/translate/cancel/{job_id}

Stops the Step Function execution via StopExecution API. Marks job as cancelled. Returns 409 if already in a terminal state.

POST /admin/translate/retry-failed/{job_id}

Creates a new job from the failed (doc, lang) pairs of a completed-with-partial job. Returns 409 if the original is still running. A retry runs on the same model as the original job. The handler walks retry_of back to the root job, so even a retry of a retry inherits the first job's model, and writes that model to the new translation_jobs row, the Valkey job hash and the queue message. Before this fix the model was in neither the message nor options, which is why inheriting options alone did not carry it; it now travels at the top level of the message and in options.model. A root job with no recorded model retries on Qwen 3 235B, our standard, never on a fallback.

Customer jobs: a retry of a customer job goes through ExecuteCustomerRetry, the same path the customer's Retry button uses. That means one free retry per quote, only the retryable failed pairs, the quote's own agent read from translation_quotes.model / model_by_lang, and translation_jobs.quote_id set on the retry job. An operator can never reopen a pair that was already refunded.

POST /admin/translate/refund-sweep

Runs one tick of the failed-pair refund sweeper (5.4) now. Returns {opened, closed_none, due, issued, failed, errors}; 409 translate_refund_sweep_busy when another container holds the lock, 503 translate_refund_sweep_failed when the sweep couldn't run.

POST /admin/translate/sweep

Manually triggers the self-healing sweeper. Idempotent — safe to call repeatedly.

// Response 200
{
  "status": "✅ [LPO] [us-east-2] [BeastMain] [/admin/translate/sweep] [200]",
  "status_code": 200,
  "endpoint": "/admin/translate/sweep",
  "cluster_node": "BeastMain",
  "region": "us-east-2",
  "language": "en",
  "timestamp": "2026-05-16T18:00:00Z",
  "data": {
    "swept": 2,
    "checked": 5,
    "results": [
      {
        "job_id": "1747407720-a3f8b2c1d4e5",
        "prior_state": "running",
        "submitted_at": "2026-05-16T16:42:00Z",
        "sfn_status": "FAILED",
        "action": "marked_failed"
      }
    ]
  },
  "error": ""
}

6.4 Worker Callback Endpoints

These endpoints are called by the BeastTranslate worker to update Aurora without needing direct database access (the worker runs outside the VPC). Until June 27, 2026 the finalize Lambda called them too; it was deleted with the Step Functions pipeline (5.3).

POST /admin/translate/update/{job_id}

Updates job state, progress, cost, and timing fields. Called by the worker after each (doc, lang) translation.

POST /admin/translate/event/{job_id}

Records a granular event in the translation_job_events table. Used for audit trail — each doc/lang start, success, failure, retry is logged as a separate event. Two new outcomes: lang_rejected with reason: "target_is_source" (the worker found, for certain, that the document is already in the target language; non-retryable, so the pair is refunded rather than retried), and lang_failed with error: "cancel-state-unknown" (a customer job's before-publish cancel check could not be answered, so the worker failed closed and published nothing).

7. Aurora Observability — Source of Truth

Aurora is the authoritative record for all translation job state. Valkey serves one specific role: real-time per-pair progress updates during active execution (written too frequently for Aurora). For everything else — job state, history, cost, audit trail — Aurora is read first.

7.1 translation_jobs Table

One row per job submission. 28 columns covering the full lifecycle. This table is the ground truth for gap analysis, cost reporting, and audit:

7.2 translation_job_events Table

Granular audit log — one row per significant event in a job's lifecycle. Used by the retry-failed handler as the authoritative source of which (doc, lang) pairs failed:

7.3 Read/Write Strategy

The translation system uses a deliberate split between Aurora and Valkey based on access pattern:

Write path

• Submit handler: writes to Valkey synchronously (fast, needed immediately for active job tracking), writes to Aurora in a go func() goroutine (non-blocking — API response returns without waiting)
• Update/Event handlers: always return HTTP 200 regardless of Aurora write outcome — errors are logged but never propagated. The translation pipeline must never fail because a monitoring write was slow.
• Completion handler: POST /admin/translate/complete/{job_id}, called by the last worker, writes the terminal state — Aurora is updated, Valkey is updated, job removed from active set.

Read path

• Status endpoint: reads Aurora first (authoritative state, timestamps, cost). Overlays Valkey progress (real-time per-pair map). Falls back to Valkey if Aurora insert is still pending (race window on submit).
• History endpoint: reads Aurora exclusively — last 50 jobs ordered by submission date. Valkey fallback only if Aurora is unavailable.
• Retry-failed handler: queries translation_job_events for lang_failed events — Aurora is the only reliable source for which pairs failed.

8. Cost Protection

The translation engine calls Bedrock for every chunk of every document in every language, using the customer's chosen agent. Without guardrails, a single typo in a batch submission could trigger hundreds of expensive API calls.

8.1 Three Protection Layers

8.2 Spend Tracking

Two parallel counters track daily usage — a dollar cap and a token cap. Both live in Valkey with 24-hour TTL auto-reset and are checked on every job submission.

Dollar Cap (autoops:bedrock:spend:daily)

• Type: STRING (numeric dollar value as string)
• Updated by: INCRBYFLOAT after every Bedrock invocation in the worker task
• Reset: 24-hour TTL auto-reset — the worker sets EXPIRE autoops:bedrock:spend:daily 86400 after each increment
• Limit: $600/day — checked by submit handler before admitting new jobs

Token Cap (autoops:bedrock:tokens:input:daily + autoops:bedrock:tokens:output:daily)

• Type: STRING (integer token count)
• Updated by: INCRBY after every Bedrock invocation — separate keys for input and output tokens
• Reset: 24-hour TTL auto-reset — same pattern as dollar cap
• Limit: 51M combined tokens/day (controls.daily_token_limit) — model-agnostic secondary guard
• Purpose: Provides a predictable ceiling that does not move when model pricing does. Because it counts tokens rather than dollars, it is the guardrail that actually binds.

Customer price — a flat $3.00 per document-language pair. This is what a customer pays, and it is deliberately simple: one document into one language is one pair at $3.00, so a customer can predict any quote by multiplying pairs × $3.00 without calling the API. The one exception is the Claude Opus 4.8 agent, where the premium model's token cost can exceed $3.00 on a large document; those pairs run the cost-plus calculation below with $3.00 as a floor and are advertised as "From $3.00". Every other agent is a true flat rate — the cost-plus math does not run for them at all.

Internal cost model (what we spend, not what the customer sees): the figures below are how TBI tracks its own cost and margin. They are never serialized to a customer quote — the quote endpoints return price only (price_per_pair, total_pairs, total_price, currency).

• Bedrock cost — token-based: (input_tokens × input_rate) + (output_tokens × output_rate) per language pair
• Infrastructure cost — a duration-based per-pair figure plus a fixed $0.012/job overhead (see the infrastructure cost formula below); covers ECS Fargate compute, S3 storage, CloudFront invalidation, SQS queuing, and orchestration
• Service fee (markup_pct, currently 30%) — applied to the combined Bedrock + infrastructure cost. Feeds the Opus cost-plus path and internal margin accounting
• Per-pair floor ($3.00) — minimum_per_pair_usd × total_pairs. For every non-Opus agent this floor is the price, applied directly
• Opus price = max((Bedrock cost + infra cost) × 1.30, $3.00 × total_pairs) — the only path where the cost-plus total can win

The engine is completely agent-agnostic. Any model accessible through Amazon Bedrock's invoke_model API can serve as a translation agent — whether it uses the Anthropic Messages format or the OpenAI-compatible format. The engine auto-detects the provider and constructs the appropriate request/response envelope. We selected these six agents based on three criteria: (1) quality on multilingual document translation, (2) availability across our failover regions (us-east-2, us-east-1, us-west-2), and (3) distinct cost/quality tradeoffs that let customers choose the right agent for their workload.

Token rates (stored in Aurora translation_parameters, cached in Valkey, refreshed every night from the AWS Price List):

These are the us-east-2 rates the nightly refresh stored on October 10, 2026, the day the Claude agents moved to Haiku 5.5 and Opus 4.8 (rate keys haiku55, sonnet46, opus48). Sonnet 4.6 now carries the geographic rate it is actually billed at, 10% above the global rate the table used to show. Because the refresh runs every night, treat GET /admin/translate/params as the live answer. Speed factor is relative to Sonnet 4.6 (1.0×). Lower means faster — Haiku at 0.5× processes in half the time. The speed factor directly affects the duration-based infrastructure cost: faster agents cost less in compute time per pair.

Infrastructure cost formula (duration-based):

• duration_per_pair = startup_seconds(10) + translatable_bytes × seconds_per_byte(0.0014) × speed_factor
• infra_cost = fixed_job_overhead($0.012) + (total_container_seconds / 60) × ecs_cost_per_minute($0.00112)
• customer_price = max((bedrock_cost + infra_cost) × (1 + markup_pct/100), minimum_per_pair × total_pairs)

Regional failover: If the primary region (us-east-2) returns a timeout or a 5xx, the engine tries the call twice there, five seconds apart, then fails over to the next region in the chain (§12). Anthropic models use cross-region inference profiles (us. prefix) and route automatically. Qwen and DeepSeek fail over from east-2 to west-2. Mistral is available in all 3 regions.

Typical internal cost vs. customer price (per 50 KB document × 1 language, Express): the first two columns are what TBI spends; the last is what the customer pays. The gap between them is the margin that funds the mission — and the reason the price stays a flat, predictable number instead of a few volatile cents.

8.3 Infrastructure Integration

Translation engine metrics are exposed through two public interfaces:

• GET /public/infrastructure — includes a translation section with daily spend, daily limit, active jobs, queue depth, cost-per-pair estimate, and daily token counts (daily_input_tokens, daily_output_tokens, daily_token_limit). Consumed by the daily digest Lambda, nightly sync, and any monitoring dashboard.
• KCC Live Dashboard — a dedicated Translation Engine card displays real-time spend, limit, active/queued jobs, cost-per-pair, and a token usage chart (input + output bars against the 50M daily limit). Auto-refreshes every 30 seconds alongside all other infrastructure panels.
• Infrastructure Live page — the Translation Engine sub-section shows live stats: spend today, tokens in, tokens out, and active jobs — populated from /public/infrastructure every 30 seconds.

9. CLI Compatibility

The existing CLI tool (scripts/kcc_helpers/translate_doc.py) continues to work unchanged. A --remote flag routes through the new service instead of running Bedrock locally:

The --remote flag produces identical terminal output to local mode — same progress bars, same chunk counters, same completion summary. The operator's workflow doesn't change; only the execution path does.

10. Configuration Reference — Protected Terms

All translation behavior is driven by a single config file: scripts/translation-config.json. This is the shared source of truth consumed by both the Python engine and the Go admin API.

10.1 Protected Terms (60 entries)

Brand names, product names, AWS services, exchange names, and acronyms that must never be translated or transliterated:

Cross Power Ministries of Pakistan, The Trinity Beast Infrastructure,
The Trinity Beast, Trinity Beast Command Center, Kiro Command Center,
Cory Dean Kalani, Shafiq Bhatti, BeastWebhook, BeastMirror, BeastMain,
BeastLRS, Claude Sonnet 4.6, Bedrock, ElastiCache, EventBridge,
CloudFront, GuardDuty, CloudWatch, CloudTrail, Step Functions,
Crypto.com, Coinbase, Gate.io, Bitstamp, Kraken, Aurora, Valkey,
Stripe, Kiro, Fargate, PostgreSQL, Lambda, Route 53, AutoOps,
TBCC, CPMP, TBI, KCC, OKX, ECR, ECS, ALB, NLB, WAF, SNS, SQS,
SES, VPC, IAM, S3 ...

Per-Request Protected Terms

In addition to the global protected terms list, you can submit document-specific terms via the protected_terms array in the translation request. This is useful for:

• Proper nouns specific to a document (customer names, project names)
• Technical identifiers not in the global list (new service names, API endpoints)
• Domain-specific terminology that should remain in English

POST /admin/translate
{
  "docs": ["Trinity-Beast-API-Reference.html"],
  "langs": "all",
  "protected_terms": ["MyCustomService", "SpecialEndpoint", "ProjectAlpha"]
}

Per-request terms are merged with the global list for that job only. They do not persist across jobs.

10b. Configuration Reference — Preserve Patterns

10.2 Preserve Patterns

Regex patterns for technical tokens that must survive translation unchanged:

10c. Configuration Reference — Limits

10.3 Limits

Per-language chunk size overrides:

Every catalog language inherits sensible limits. The table above covers our library's 11 languages. Each of the 127 catalog languages resolves its limits by exact code, then its base_lang, then its tuning_lang, then the default. So fa, ps and sd inherit Urdu's tuning, ko inherits Japanese, Cyrillic languages inherit Russian, and a regional variant such as es-MX follows its base language. The tuning_lang column of the catalog decides it, so a new language is tuned the day it is added.

11. Operations Guide

11.1 Submitting a Translation Job

Single document, all languages:

curl -s -X POST \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"docs":["Trinity-Beast-API-Reference.html"],"langs":"all"}' \
  https://api.cpmp-site.org/admin/translate | jq .

Multiple documents, specific languages:

curl -s -X POST \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"docs":["Trinity-Beast-API-Reference.html","Trinity-Beast-Architecture-Guide.html"],"langs":["es","pt","fr"]}' \
  https://api.cpmp-site.org/admin/translate | jq .

With idempotency key (safe to retry):

curl -s -X POST \
  -H "X-Admin-Key: $ADMIN_KEY" \
  -H "X-Idempotency-Key: api-ref-2026-05-16" \
  -H "Content-Type: application/json" \
  -d '{"docs":["Trinity-Beast-API-Reference.html"],"langs":"all"}' \
  https://api.cpmp-site.org/admin/translate | jq .

11.2 Monitoring Progress

# Check job status
curl -s -H "X-Admin-Key: $ADMIN_KEY" \
  https://api.cpmp-site.org/admin/translate/status/{job_id} | jq .

# View queue
curl -s -H "X-Admin-Key: $ADMIN_KEY" \
  https://api.cpmp-site.org/admin/translate/queue | jq .

# System health
curl -s -H "X-Admin-Key: $ADMIN_KEY" \
  https://api.cpmp-site.org/admin/translate/health | jq .

# Recent history
curl -s -H "X-Admin-Key: $ADMIN_KEY" \
  https://api.cpmp-site.org/admin/translate/history | jq .

11.3 Troubleshooting

curl -s -X POST -H "X-Admin-Key: $ADMIN_KEY" \
  https://api.cpmp-site.org/admin/translate/cancel/{job_id}

12. Regional Failover

The translation engine implements automatic regional failover to maintain availability during Bedrock service disruptions. This was added after a us-east-2 outage during development exposed the single-region weakness — better to discover this before customers were affected.

12.1 Failover Chain

When a Bedrock call fails with a service-level error, the engine automatically retries in the next region:

The failover is transparent to the caller — the translation completes successfully as long as at least one region is available. A log message records when a fallback region was used.

12.2 Trigger Conditions

Failover is triggered for service-level errors and timeouts that indicate the region is unavailable or overloaded:

Other errors (validation failures, authentication errors, malformed requests) are not retried — they would fail identically everywhere.

Per-Region Retry with Backoff

Each region gets 2 attempts before the engine moves to the next region. A 5-second backoff between attempts allows transient pressure to clear:

us-east-2 (attempt 1) → timeout → wait 5s →
us-east-2 (attempt 2) → timeout →
us-east-1 (attempt 1) → timeout → wait 5s →
us-east-1 (attempt 2) → timeout →
us-west-2 (attempt 1) → timeout → wait 5s →
us-west-2 (attempt 2) → timeout → FAIL (raise exception)

Total: 6 attempts across 3 regions. In practice, transient spikes clear within 5-10 seconds, so the retry within the same region usually succeeds without needing failover.

12.3 Cost Impact

Regional failover has negligible cost impact:

• Same pricing: Bedrock pricing is identical across all three regions
• No duplicate charges: Failed requests that trigger failover are not billed (no tokens consumed)
• Minimal latency: Cross-region latency adds ~50-100ms per call, unnoticeable in the context of a multi-second translation
• Estimated overhead: ~$0.0005 per document when failover is triggered (additional API call setup cost)

13. Document Preparation Guide

Proper document preparation ensures clean translations with minimal post-processing. This section covers the conventions that help the translation engine produce accurate results.

13.1 Code Tag Usage

The <code translate="no"> tag tells the translation engine to preserve content exactly as written. Use it correctly to avoid formatting artifacts in translated documents.

When to Use Code Tags

Use <code translate="no"> for technical identifiers that would break if translated:

• Service names: tbi-ops-notify, BeastMain
• API endpoints: /admin/translate, /public/infrastructure
• Environment variables: AWS_REGION, ADMIN_KEY
• Function names: _apply_sentinels(), translate()
• File paths: /var/log/app.log, scripts/kcc.sh
• Database columns: job_id, created_at
• Configuration keys: max_retries, request_timeout_seconds
• Command-line flags: --remote, --force

When NOT to Use Code Tags

Do not wrap pure data values in code tags — they should appear as plain text:

• Memory sizes: 1770 MB, 32 GB, 3 GB (not <code>1770 MB</code>)
• Timeouts: 60 seconds, 180s (not <code>60 seconds</code>)
• Percentages: 98.5%, 62% (not <code>98.5%</code>)
• Counts: 11 languages, 6 documents (not <code>11 languages</code>)
• Costs: $600, $1.65 (not <code>$600</code>)
• Version numbers in prose: version 17.7 (not <code>17.7</code>)

Quick Test

Ask yourself: "If I changed this value, would the system break?" If yes, use code tags. If no (it's just a number or measurement), leave it as plain text.

13.2 Protected Terms Submission

For documents with domain-specific terminology not in the global protected terms list, submit additional terms with the translation request:

POST /admin/translate
{
  "docs": ["Customer-Integration-Guide.html"],
  "langs": "all",
  "protected_terms": [
    "CustomerCorp",
    "ProjectPhoenix",
    "DataSync API",
    "IntegrationHub"
  ]
}

These terms are added to the global list for this job only. The engine will:

1. Wrap each term in <span translate="no"> during preprocessing
2. Replace with sentinel tokens before sending to Bedrock
3. Restore the original terms after translation
4. Validate that all terms survived intact

Best Practices for Protected Terms

• Be specific: "DataSync API" is better than "DataSync" (avoids false matches)
• Include variations: If a term appears as both "ProjectPhoenix" and "Project Phoenix", include both
• Case matters: "CustomerCorp" and "customercorp" are treated as different terms
• Don't over-protect: Common English words that translate well don't need protection

13.3 Clarification Workflow

When the translation engine encounters ambiguous content, it may flag it for human review. This happens in the validation phase when:

• A protected term appears to have been partially translated
• A version number format doesn't match the expected pattern
• HTML structure differs significantly between source and output

Flagged content appears in the job status response under the warnings array:

{
  "status": "✅ [LPO] [us-east-2] [BeastMain] [/admin/translate/status/1747407720-a3f8b2c1d4e5] [200]",
  "status_code": 200,
  "endpoint": "/admin/translate/status/1747407720-a3f8b2c1d4e5",
  "cluster_node": "BeastMain",
  "region": "us-east-2",
  "language": "en",
  "timestamp": "2026-05-16T17:45:00Z",
  "data": {
    "job_id": "1747407720-a3f8b2c1d4e5",
    "state": "succeeded",
    "warnings": [
      "chunk 14 (ja): soft failure — protected term 'DataSync' may have been altered",
      "chunk 22 (ar): soft failure — version number format changed from X.Y.Z to X.Y"
    ]
  },
  "error": ""
}

Soft failures don't block the translation — the output is still deployed. Review the warnings and manually verify the flagged sections if needed.

14. Pre-Scan Complexity Analysis

Before translation begins, the engine analyzes each document for complexity factors that may cause validation failures. This pre-scan identifies code-heavy sections and recommends whether to proceed, exercise caution, or split the document.

14.1 Complexity Metrics

The pre-scan calculates a complexity score for each section based on:

Section Thresholds

• High-density section: Complexity score > 15 OR > 10 code tags
• Document split threshold: Total score > 50 OR > 3 high-density sections

14.2 Recommendations

Based on the analysis, the pre-scan returns one of three recommendations:

Pre-Scan Output Example

DOCUMENT TRANSLATION COMPLEXITY ANALYSIS
========================================
Total characters: 81,107
Total sections: 13
Total code tags: 287
Overall complexity score: 415.4
Recommendation: SPLIT

WARNINGS:
  ⚠️  Document has 287 code tags — high validation failure risk
  ⚠️  Section 'step-function' has 51 code tags — consider simplifying
  ⚠️  Section 'observability' has 48 code tags — consider simplifying

HIGH-DENSITY SECTIONS (9):
  • architecture: 11 code tags, score 22.7
  • sentinel-system: 22 code tags, score 34.5
  • step-function: 51 code tags, score 71.1
  ...

SUGGESTED SPLIT: 4 parts
  → Split after 'validators' (After 3 high-density sections)
  → Split after 'observability' (After 3 high-density sections)
  → Split after 'doc-prep' (After 3 high-density sections)

14.3 Document Splitting

When the pre-scan recommends splitting, it suggests natural break points at section boundaries. Options for handling complex documents:

Option 1: Split into Multiple Documents

Create separate HTML files for each part (e.g., Doc-Part1.html, Doc-Part2.html). Each part translates independently with lower failure risk. Link them together with navigation.

Option 2: Simplify High-Density Sections

Reduce code tag density in problematic sections:

• Replace code examples with prose descriptions where possible
• Move detailed code to appendices or separate reference docs
• Use styled spans instead of code tags for visual-only formatting
• Consolidate similar examples into single code blocks

Option 3: Translate in Batches

Submit fewer languages per job (e.g., 3 instead of 11). This reduces concurrent load and allows the model more capacity per translation. Retry failed languages individually.

Per-Language Split Thresholds (v2.5)

Complex scripts (Urdu, Arabic, Hindi) struggle with high tag density even when Latin-script languages handle the same chunk fine. The prescan now applies per-language code tag limits — tighter thresholds for scripts where the model is more likely to drop markup:

Configuration key: max_code_tags_per_part_by_lang in translation-config.json. When the prescan runs for a specific language, it uses that language's threshold to determine split points. A document that translates as one part for Spanish may automatically split into 2-3 parts for Urdu.

14.4 Splitting Safety Valve (v2.8)

Even when code tag density is low, a single part that exceeds the model's effective output window will be silently truncated — sections at the end of the part simply disappear from the output. The safety valve enforces a hard character limit per part regardless of prescan recommendations.

The Problem

The Performance Report (75 KB) has 18 sections with moderate code density. The prescan recommended splitting into 3 parts based on code tag thresholds. But Part 1 was 36 KB of prose-heavy content — well under the code tag limit but far beyond the model's output token budget. The model translated the first ~24 KB faithfully, then its output simply stopped. Sections 7-8 (partner-sustained, udp-engine) vanished without any error signal.

The Fix

# Safety valve: max chars per part (prevents model output truncation)
MAX_CHARS_PER_PART = 24000  # ~6000 tokens, well within max_output_tokens

The splitter now enforces a 24 KB ceiling on every part. If a part exceeds this limit after the prescan-based split, it is further subdivided at the nearest section boundary. This is conservative — Latin scripts could handle ~30 KB, but 24 KB is safe for all languages including RTL and CJK where token efficiency is lower.

Impact

The safety valve only activates when the prescan's code-tag-based splitting produces oversized parts. For most documents, the code density split already keeps parts well under 24 KB.

15. Document-Level Preprocessor

The document-level preprocessor is a critical layer that runs before chunking. It extracts complex HTML elements from the entire document, replacing them with simple Unicode placeholders. After translation, the postprocessor restores the original elements. This eliminates the "model drops tags" failure mode entirely.

15.1 The Problem

The per-chunk sentinel system (Section 3) works well for most documents, but complex documents with many <code>, <strong>, and <em> tags exposed a fundamental limitation:

• By the time chunks are created, they already contain many protected elements
• Each element becomes a sentinel placeholder (__TBP0__, __TBP1__, etc.)
• Chunks with 20+ placeholders overwhelm the model's attention
• The model occasionally drops, duplicates, or merges placeholders during translation
• Validation catches these failures, but retries often produce the same errors

Example failure: A chunk with 27 <code translate="no"> tags consistently failed validation with tag count mismatch (27→23) — the model dropped 4 placeholders despite explicit instructions to preserve them.

15.2 The Solution

Extract ALL problematic elements from the entire document before chunking. The model never sees these elements — only simple Unicode placeholders that it cannot confuse with HTML structure.

Before vs After

15.3 Processing Flow

The preprocessor uses a two-phase extraction model that integrates into the translation pipeline as the first step:

Document → PHASE 1 (Density Lift) → PHASE 2 (Individual Extract) → Chunk → Translate → Reassemble → POSTPROCESS → Output
              ↓                           ↓                                                              ↓
     Lift entire dense containers  Extract remaining code/pre/       Single-pass flat restore
     as [QZ_ZK_NNN] placeholders   strong/em/numeric individually    (no nesting, no iteration)
     Original HTML preserved        Build manifest mapping            from manifest

Phase 1 runs FIRST on the raw DOM. It inspects container elements (tr, li, dd, dt, p, div) and lifts any container whose ratio of protected elements to prose characters exceeds a per-language density threshold. The lifted BLOCK entries store the original HTML byte-for-byte — no nested placeholders inside them.

Phase 2 then runs on whatever Phase 1 didn't lift, extracting individual elements (pre, code, spans, strong/em, a-tags, numeric patterns) the same way as before.

Integration in engine.py

def translate(text, target_lang, mode="html", ...):
    # Build preprocessor config from language profile
    preprocess_config = {"script_family": profile.get("script_family", "latin")}
    
    # Step 1: PREPROCESS — Phase 1 lifts dense blocks, Phase 2 extracts individuals
    simplified_html, manifest = preprocess_for_translation(text, preprocess_config)
    
    # Step 2: CHUNK — Split simplified document
    head, chunks, tail = chunker.split_document(simplified_html, lang=target_lang)
    
    # Step 3: TRANSLATE — Each chunk through Bedrock (per-chunk sentinels still run)
    for chunk in chunks:
        translated = _translate_chunk(chunk, ...)
    
    # Step 4: REASSEMBLE
    reassembled = chunker.reassemble(head, translated_chunks, tail)
    
    # Step 5: POSTPROCESS — Single-pass flat restore (no nesting)
    output = postprocess_translation(reassembled, manifest)

15.4 Element Extraction

The preprocessor extracts elements in a two-phase order. Phase 1 (density lift) runs first on the raw DOM. Phase 2 (individual extraction) runs on whatever Phase 1 didn't lift, processing elements in order of specificity (most specific first) to handle nesting correctly:

Placeholder Format

Placeholders use ASCII square brackets with opaque Z-prefixed type codes. Three previous formats were evaluated and discarded:

• Unicode brackets (⟦TYPE_NNN⟧): models (especially Qwen) corrupt rare Unicode codepoints — U+27E6/U+27E7 became U+27F2 in Arabic output. The model reproduces the inner text but substitutes visually-similar Unicode glyphs for the brackets.
• Double underscores (__TBI_TYPE_NNN__): models interpret these as Python dunder variables or code identifiers and drop them entirely from the translated output.
• Semantic type names ([TBI_CODE_000]): models read "CODE" and hallucinate <code> tags in the output. The type name provides semantic clues that trigger unwanted generation.

The production format uses a two-letter nonsense prefix (QZ) and single-letter opaque type codes that carry no HTML or programming semantics:

• Format: [QZ_ZC_000] (square bracket + QZ prefix + Z-code + 3-digit index)
• Prefix: QZ — not a brand term, not an HTML tag, not a language code. Zero collision risk.
• Z-codes: Opaque type identifiers relatable for debugging but meaningless to the model:

The result: 100% tag preservation across all models and all target languages. The model sees [QZ_ZC_000] and has zero semantic interpretation — it passes the token through unchanged. Postprocessing restores the original HTML from the manifest.

Phase 2 Nested Element Handling

Within Phase 2, the preprocessor handles arbitrary nesting depth by processing innermost elements first. Note: this nesting only occurs for individually-extracted elements — BLOCK entries from Phase 1 are always flat (they contain original HTML, never placeholders).

Source:
<span translate="no"><code translate="no">tbi-ops-notify</code> Lambda</span>

Pass 1: Extract inner code tag
<span translate="no">⟦CODE_001⟧ Lambda</span>

Pass 2: Extract outer span
⟦SPAN_002⟧

Model sees: ⟦SPAN_002⟧ (one token, no nesting)

Sibling Placeholder Awareness

When the preprocessor extracts elements from a container (e.g., a table cell), earlier passes leave placeholder text in the parent. Later passes must not be confused by these sibling placeholders — a <code translate="no"> tag in the same table cell as an already-extracted element is still a valid extraction target.

15.4a Density-Based Block Lifting

Phase 1 of the preprocessor inspects the raw DOM before any individual extraction. It identifies container elements where the ratio of protected elements to prose characters is too high — meaning there's too little translatable text to justify sending the element through the model.

Density Formula

density = protected_elements / max(prose_chars, 1)

Where:

• protected_elements = unique count of <code> + translate="no" elements + class spans + <pre> inside the container
• prose_chars = character count of visible text content (excluding protected element text)
• Container elements checked: tr, li, dd, dt, p, div

The check uses pure DOM element counting — no regex. This makes it deterministic and immune to content patterns that could fool regex-based approaches.

Thresholds by Script Family

Thresholds are per-language via the script_family column in the translation_language_profiles Aurora table:

Guard Rails

• Max prose for lift: 300 characters (configurable per language via Aurora density_max_prose column). Containers with more prose than this are never lifted, regardless of density.
• Min protected elements: 2. Containers with only 1 protected element are never lifted — not worth the overhead.

Aurora Configuration Columns

The translation_language_profiles table stores per-language density configuration:

Example: Endpoint Table Row

<tr>
  <td><span translate="no">GET</span></td>
  <td><code translate="no">/health</code></td>
  <td>LPO server health check</td>
</tr>

Protected elements: 2 (span + code)
Prose chars: 25 ("GET" + "/health" excluded, "LPO server health check" counted)
Density: 2 / 25 = 0.08

→ CJK (threshold 0.03): LIFTED as ⟦BLOCK_NNN⟧ — density 0.08 > 0.03
→ Latin (threshold 0.06): LIFTED as ⟦BLOCK_NNN⟧ — density 0.08 > 0.06
→ If prose were 40+ chars: density drops to 0.05, Latin would KEEP it

Results

Block lifting eliminates the failure mode where code-heavy table rows overwhelm the model's attention. The model never sees these rows — it translates a single ⟦BLOCK_NNN⟧ token (which it passes through unchanged) instead of a complex structure with multiple inline placeholders.

15.5 Restoration

After translation, the postprocessor restores placeholders in a single flat pass in reverse index order (high → low) to prevent prefix collisions. Because Phase 1 (density lift) runs BEFORE Phase 2 (individual extraction), BLOCK entries contain original HTML — never nested placeholders.

Why Single-Pass Works

• BLOCK entries: Always contain original HTML (no placeholders inside). Phase 1 lifts the raw DOM before any extraction happens, so the stored HTML is pristine.
• Phase 2 entries: May nest (e.g., ⟦SPAN_002⟧'s manifest HTML contains ⟦CODE_001⟧). Reverse-order restore handles this correctly — restoring index 2 first produces the text containing index 1, then index 1 is restored in the same pass.
• No iteration needed: A single reverse pass resolves all nesting because higher-index entries always contain lower-index placeholders, never the reverse.
• No orphan tracking: Every placeholder in the translated text maps to exactly one manifest entry. Missing placeholders (model dropped them) are logged but don't break restoration.

Translated output contains: ⟦BLOCK_003⟧ ... ⟦SPAN_002⟧ ... ⟦CODE_001⟧

Single pass (reverse order):
  Restore ⟦BLOCK_003⟧ → original HTML (no further substitution needed inside)
  Restore ⟦SPAN_002⟧ → <span translate="no">⟦CODE_001⟧ Lambda</span>
  Restore ⟦CODE_001⟧ → <code translate="no">tbi-ops-notify</code>

Final:
  <tr><td>...original row...</td></tr>
  <span translate="no"><code translate="no">tbi-ops-notify</code> Lambda</span>
  <code translate="no">tbi-ops-notify</code>

Perfect reconstruction — no nesting loops, no iteration.

Manifest Structure

The manifest maps each placeholder to its original HTML, enabling exact restoration:

{
  "⟦BLOCK_003⟧": {
    "type": "BLOCK",
    "html": "<tr><td><span translate=\"no\">GET</span></td><td><code translate=\"no\">/health</code></td><td>LPO server health check</td></tr>",
    "index": 3
  },
  "⟦SPAN_002⟧": {
    "type": "SPAN",
    "html": "<span translate=\"no\">⟦CODE_001⟧ Lambda</span>",
    "index": 2
  },
  "⟦CODE_001⟧": {
    "type": "CODE",
    "html": "<code translate=\"no\">tbi-ops-notify</code>",
    "index": 1
  }
}

Note: BLOCK entries always store pristine HTML (no ⟦...⟧ tokens inside). Phase 2 entries like SPAN may contain lower-index placeholders from inner elements extracted in a later pass — this is handled naturally by the reverse-order restore.

15.6 Numeric Pattern Extraction

Pass 5 extracts numeric patterns from the text after HTML element extraction. This protects bare numbers in prose that weren't already inside code or span tags. The model cannot convert, localize, or drop what it never sees.

Why Numeric Extraction Matters

When translating to complex scripts (Arabic, Hindi, Urdu), the model occasionally:

• Drops numeric values: "32 GB" becomes just "GB" or disappears entirely
• Localizes units: "MB" becomes "Mo" (French) or "ميغابايت" (Arabic)
• Converts formats: "98.5%" becomes "٩٨٫٥٪" (Arabic numerals)
• Paraphrases: "1770 MB" becomes "approximately 2 GB"

These transformations break technical accuracy. The numeric extraction pass prevents all of them.

Patterns Extracted

Processing Order

Numeric extraction runs after HTML element extraction (Passes 1-4). This means:

• Numbers inside <code> tags are already protected by Pass 2
• Numbers inside translate="no" spans are already protected by Pass 3
• Pass 5 only catches bare numbers in prose that weren't otherwise protected
• No double-extraction — the regex skips content already inside placeholders

Example: Hindi Translation

Source:
"The Lambda uses 1770 MB of memory and achieves 98.5% uptime."

After Pass 5:
"The Lambda uses ⟦MEM_042⟧ of memory and achieves ⟦PCT_043⟧ uptime."

Model translates prose, placeholders survive intact.

After restoration:
"लैम्ब्डा 1770 MB मेमोरी का उपयोग करता है और 98.5% अपटाइम प्राप्त करता है।"

Technical values preserved exactly — no localization, no conversion.

Placeholder Collision Prevention

The numeric extraction pass includes safeguards to prevent extracting numbers that are part of existing placeholder names (e.g., the "001" in ⟦CODE_001⟧):

• Skips matches preceded by an underscore within a placeholder token
• Skips matches followed by the closing bracket ⟧
• Skips matches inside unclosed placeholder brackets

Without these guards, the numeric regex would corrupt placeholder names by extracting their index numbers, producing nested placeholders like ⟦CODE___TBN10__⟧ that the model cannot handle.

15.7 Type D — Table Micro-Translation

Technical documentation is dominated by tables where most columns are technical identifiers (column names, data types, endpoints, parameter keys) and only one or two columns contain translatable prose (descriptions, explanations). Sending these tables through the full translation pipeline is wasteful and risky — the model tends to transliterate technical values or corrupt table structure.

Type D solves this with a surgical approach: extract the entire table before translation, then micro-translate only the declared prose columns via a separate, targeted Bedrock call. The table structure is never modified — only cell text content changes.

HTML Markup

Document authors declare the translation contract directly on the <table> element using two attributes:

Example: Data Dictionary Table

<table data-tx-headers="all" data-tx-cols="4">
  <thead>
    <tr><th>Column</th><th>Data Type</th><th>Default</th><th>Nullable</th><th>Description</th></tr>
  </thead>
  <tbody>
    <tr>
      <td><code>user_id</code></td>
      <td><code>uuid</code></td>
      <td>—</td>
      <td>YES</td>
      <td>Owner of the key; references users table.</td>  <!-- Only this cell gets translated -->
    </tr>
  </tbody>
</table>

Result in Italian: columns 0–3 remain byte-identical English. Column 4 becomes "Proprietario della chiave; fa riferimento alla tabella users."

Processing Pipeline

Type D tables are handled in three stages that bookend the translation process:

1. Phase 0 Extraction — Before any other preprocessing, all tables with data-tx-headers or data-tx-cols attributes are extracted whole and replaced with ⟦TABLE_NNN⟧ placeholders. The model never sees table HTML.
2. Translation — The document is translated normally (prose mode or standard path). Table placeholders pass through untouched.
3. Postprocess + Micro-Translate — After translation, placeholders are restored to their original HTML. Then a targeted Bedrock call translates only the declared columns:
   • Extract text from headers (per data-tx-headers) and body cells (per data-tx-cols)
   • Skip cells with translate="no"
   • Batch all strings into a single Bedrock text-mode call using ⟪SPLIT⟫ delimiters
   • Parse response and patch translations back into the table HTML

Common Patterns

Integration with Prose Mode

Documents with extreme density hints (hint:density=extreme) route through prose mode, which extracts and translates only text fragments from the HTML. Without Type D awareness, prose mode would indiscriminately translate all visible table text — including technical columns that must stay English.

The fix: Phase 0 table extraction runs before prose mode walks the DOM. Prose mode sees only the ⟦TABLE_NNN⟧ placeholder — never the table content. After prose mode rebuilds the document, postprocessing restores the tables and runs micro-translate on the declared columns.

This applies to both the marker-based split path and the auto-split path in handler.py.

Failure Handling

Type D micro-translation is non-fatal. If the Bedrock batch call fails or returns a mismatched fragment count, the tables are restored in their original English — no data corruption, no job failure. The document is delivered with English tables rather than corrupted ones.

16. Notification System

The translation engine sends email notifications via the AutoOps notification pipeline (tbi-ops-notify Lambda → SES). Notifications are consolidated across batch jobs and include detailed per-document breakdowns.

16.1 Email Format

Each notification email includes:

• Batch Summary: Total jobs, documents, languages, succeeded/failed pairs, final state
• Total Time: Formatted as "Xm Ys" (e.g., "7m 12s") for readability
• Per-Document Breakdown: Which languages succeeded and failed for each document
• Deployment Status: CloudFront invalidation count, S3 deployment confirmation
• Search Index Status: Whether the search index was rebuilt successfully
• Error Details: Specific validation failures with chunk/validator information

Example Notification

Subject: [INFO] Translation Complete: 2 docs × 11 langs — 22/22 pairs SUCCEEDED

Batch Summary:
• Jobs: 2
• Documents: 2
• Languages: 11
• Total Pairs: 22
• Succeeded: 22
• Failed: 0
• Final State: SUCCEEDED
• Total Time: 7m 12s

Documents Translated:
• Trinity-Beast-TBI-Translation-Engine.html
  ✓ Succeeded: es, pt, fr, de, ru, hi, ja, zh, ar, ur, it
• Trinity-Beast-Infrastructure-Overview.html
  ✓ Succeeded: es, pt, fr, de, ru, hi, ja, zh, ar, ur, it

Deployment:
• CloudFront Invalidations: 2
• All translated files deployed to S3

Search Index:
• Rebuilt successfully (all 11 languages)

Partial Success Example

Subject: [WARNING] Translation Complete: 1 doc × 11 langs — 10/11 pairs PARTIAL

Documents Translated:
• Complex-Technical-Guide.html
  ✓ Succeeded: es, pt, fr, de, ru, hi, ja, zh, ar, it
  ✗ Failed: ur

Error Details:
• Complex-Technical-Guide.html → ur: chunk 14 failed validation after 3 retries
  check_tag_counts: expected 27 code tags, found 23

16.2 Batch Consolidation

When multiple translation jobs are submitted together (e.g., translating 5 documents), the notification system consolidates them into a single email:

• Deferred sending: Each finalizing job checks if other jobs are still active
• Last job sends: Only the last job to complete sends the consolidated email
• Safety net: If all jobs see each other as "active" (race condition), a 5-second wait and re-check prevents missed notifications
• Fallback: If no jobs are queued and the batch is wrapping up, the current job sends anyway

This prevents notification spam when translating multiple documents — you get one comprehensive email covering the entire batch, not 5 separate emails.

16.3 Document Resolver (v2.5)

When the same document appears in multiple jobs within a batch (e.g., initial run fails Urdu, retry succeeds), the notification resolves duplicate entries into a single final-state view:

• Merge logic: For each document, collect all succeeded and failed languages across all jobs
• Retry overrides failure: If a language appears in both succeeded (retry job) and failed (original job), it's reported as succeeded
• Deduplicated counts: Summary totals (Succeeded, Failed, Total Pairs) are recalculated from the resolved state — not raw aggregation
• Single entry per document: The notification shows each document exactly once with its final language breakdown

Without the resolver, a retry job would show the same document twice — once with the failure and once with the fix — making the notification confusing and the counts misleading.

16.4 Tag Inventory (v2.8)

Every notification includes a Tag Inventory section showing source vs output tag counts per document. This lets you detect at a glance if the model is adding or dropping tags. As of v2.8, the inventory also reports Mermaid diagram counts:

Tag Inventory (source → output):
• Trinity-Beast-Translation-Service.html
  IN:  code:22 pre:5 strong:8 em:2 a:4 br:3 diagrams:1
  OUT: code:22 pre:5 strong:8 em:2 a:4 br:3 diagrams:1

If the model has a bad day and adds a <span> that wasn't in the source, or drops code tags, you'll see the mismatch immediately:

  IN:  code:23 pre:5 strong:8 diagrams:2
  OUT: code:20 pre:4 strong:8 diagrams:1    ← 3 code dropped, 1 diagram lost

Tag counts are logged per-language in Aurora (translation_job_events) with tags_in and tags_out fields. The notification shows the first successful language's counts (source tags are identical across all languages since it's the same source document).

17. Delta Translation (Incremental Updates)

Documents change frequently — a new endpoint, a revised architecture, an updated pricing table. Without delta translation, every edit requires re-translating the entire document across all 11 languages. Delta translation solves this by identifying exactly which sections changed and translating only those, reusing cached translations for everything else.

17.1 Concept

The delta translation system leverages two key properties of the document library:

• S3 versioning: Every document upload creates a new version in S3. Previous versions are retained indefinitely, providing a complete edit history.
• <!-- TBI-CHUNK --> markers: Human-placed section boundaries in the English source that divide documents into logical, independently-translatable sections.

By comparing the current English document against the version that was last translated, the system identifies which sections changed (by content hash) and only sends those to Bedrock. Unchanged sections are pulled directly from the existing translated document. Typical savings: 70–90% on incremental updates.

17.2 S3 Versioning as Diff Source

The website bucket (trinity-beast-website-east2) has versioning enabled. Every aws s3 cp or s3api put-object creates a new version with a unique VersionId. The delta system uses this to:

• List all versions of a document with timestamps and sizes
• Fetch any previous version by VersionId
• Compare current content against the version that was last successfully translated

No separate manifest storage is required — S3 already has the full history. A lightweight metadata file (docs/delta/{doc}.{lang}.json) tracks which VersionId was last translated for each document-language pair.

17.3 Comment Preservation (Sentinel Pass 0)

For delta translation to work, <!-- TBI-CHUNK --> markers must survive the translation round-trip. Previously, Bedrock silently dropped HTML comments during translation. The sentinel system now includes a Pass 0 that protects all HTML comments:

# Pass 0: Before Bedrock sees the chunk
<!-- TBI-CHUNK -->  →  __TBP0__    (sentinel token)
<!-- Section 5 -->  →  __TBP1__    (sentinel token)

# After translation: sentinels restored
__TBP0__  →  <!-- TBI-CHUNK -->
__TBP1__  →  <!-- Section 5 -->

This is implemented as the first pass in _apply_sentinels() in engine.py, before the existing translate="no" element extraction (Pass 1), paired span sentinels (Pass 2), and numeric protection (Pass 3). Comments are treated as Type A (FULL) sentinels — extracted completely and restored verbatim.

17.4 Hash-Based Section Matching

The algorithm is position-independent — sections are matched by content hash, not by index. This means markers can be added, removed, or repositioned between versions without breaking the delta logic.

flowchart TD
    A[Fetch Current English from S3] --> B[Split by TBI-CHUNK markers]
    B --> C[Hash each section SHA-256]
    D[Fetch Previous English version] --> E[Split by TBI-CHUNK markers]
    E --> F[Hash each section]
    C --> G{Compare hashes}
    F --> G
    G -->|Match found| H[Pull from existing translation]
    G -->|No match| I[Send to Bedrock]
    H --> J[Reassemble with TBI-CHUNK markers]
    I --> J
    J --> K[Deploy to S3 + Save metadata]

    style A fill:#1e3a5f,stroke:#60a5fa,color:#e0e0e0
    style D fill:#1e3a5f,stroke:#60a5fa,color:#e0e0e0
    style H fill:#064e3b,stroke:#10b981,color:#e0e0e0
    style I fill:#7c2d12,stroke:#f97316,color:#e0e0e0
    style K fill:#1e3a5f,stroke:#60a5fa,color:#e0e0e0
        

Marker repositioning example:

• Version 1: 5 sections (markers at A, B, C, D)
• Version 2: 6 sections (new marker added — A, B, C, C2, D)
• Sections before and after the new marker still match by hash → reused
• The split section produces two new hashes → both translate fresh
• Result: 4 of 6 sections reused (67% savings) despite marker change

17.5 CLI Commands

Four KCC commands support delta translation and chunk management:

Delta Diff (Analysis Only)

# List available S3 versions
bash scripts/kcc.sh delta-diff Trinity-Beast-API-Reference.html --list-versions

# Compare current vs previous version (auto-detects)
bash scripts/kcc.sh delta-diff Trinity-Beast-API-Reference.html

# Compare against a specific version
bash scripts/kcc.sh delta-diff Trinity-Beast-API-Reference.html --version-id ksYxUBZIUB8Roi2KQYje6ig9R7JesL9z

# Show delta for a specific language
bash scripts/kcc.sh delta-diff Trinity-Beast-API-Reference.html --lang ja

Delta Translate (Incremental Translation — Local CLI)

# Dry run — show what would change without calling Bedrock
bash scripts/kcc.sh delta-translate Trinity-Beast-API-Reference.html es --dry-run

# Translate only changed sections for one language
bash scripts/kcc.sh delta-translate Trinity-Beast-API-Reference.html es

# Translate changed sections for all languages
bash scripts/kcc.sh delta-translate Trinity-Beast-API-Reference.html all

# Force full translation (creates fresh baseline)
bash scripts/kcc.sh delta-translate Trinity-Beast-API-Reference.html all --force

Delta via Remote API (options.delta)

The delta option is also available on POST /admin/translate — the worker skips any language pair where the translated file on S3 is already newer than the source document. No local CLI needed.

# Submit a delta job via the remote API — skips up-to-date pairs automatically
curl -s -X POST -H "X-Admin-Key: $ADMIN_KEY" -H "Content-Type: application/json" \
  -d '{"docs":["Trinity-Beast-API-Reference.html"],"langs":"all","options":{"delta":true}}' \
  https://api.cpmp-site.org/admin/translate | jq .

Delta Validate (Marker Preservation Check)

# Validate TBI-CHUNK markers survived translation for all delta-enabled docs
bash scripts/kcc.sh delta-validate all all

# Validate a specific doc across all languages
bash scripts/kcc.sh delta-validate Trinity-Beast-API-Reference.html all

# Validate a specific doc + language pair
bash scripts/kcc.sh delta-validate Trinity-Beast-API-Reference.html es

Reports pass/fail per doc×lang pair. Exit code 0 if all pass, 1 if any markers were lost. Run after any translation job to confirm Sentinel Pass 0 is working correctly.

Chunk Sizer (Auto-Placement Suggestions)

# Analyze a doc from S3 and suggest TBI-CHUNK marker placement
bash scripts/kcc.sh chunk-size Trinity-Beast-API-Reference.html

# Analyze a local file
bash scripts/kcc.sh chunk-size /path/to/local/doc.html

Scans the document for <section>, <h2>, <h3>, and .category-section boundaries. Reports current chunk sizes (if markers exist), identifies policy violations, and suggests where to insert markers to stay within the 15KB/18KB/12KB policy. Dense sections (high translate="no" density) automatically target the tighter 12KB limit.

17.6 Bootstrap Path

Existing translated documents do not contain <!-- TBI-CHUNK --> markers (they were stripped before the sentinel fix). The bootstrap sequence is:

1. First run (full cost): Use --force to translate the entire document. The sentinel fix preserves markers in the output. Delta metadata is saved to S3.
2. Subsequent runs (delta savings): The tool detects the existing translation has markers, loads metadata to identify the previous English version, and only translates changed sections.

After the bootstrap run, typical savings on incremental updates:

Cost model: At approximately $1.50 per section-language pair, a 9-section document across 11 languages costs ~$148.50 for a full translation. With delta (2 sections changed), the same update costs ~$33 — a 78% reduction.

18. API Error Message Translation

The document translation engine handles HTML docs (Sections 1–17). A separate system handles the translation of API response messages — the short error and status strings returned in UME envelope responses. This section documents how template variables are protected during that process and why the delimiter choice matters.

18.1 Overview

The error_messages Aurora table holds 21 message keys in 12 languages (252 rows total). These messages are served dynamically based on the caller's api_lang preference. Some messages contain template variables that are substituted at runtime with real values (e.g., usage counts, rate limits).

Example message (English):

Monthly query limit reached. You have used [usage] of [limit] allowed queries this month.

At runtime, the Go server substitutes [usage] and [limit] with the user's actual values before returning the response.

18.2 Template Variables — Brackets Over Braces

Convention: All template variables in translatable messages use [square brackets], never {curly braces}.

Why: Curly braces are universally recognized by AI models as structured data delimiters (JSON, format strings, template literals). When a translation model encounters {usage} in a message, it frequently attempts to:

• Fill in the variable with a fabricated value
• Reformat it as a different template syntax (e.g., {{usage}})
• Interpret it as JSON structure and mangle the surrounding text
• Strip it entirely as "non-translatable formatting"

Square brackets do not trigger this behavior. Models treat [usage] as inline notation — similar to citation markers or editorial annotations — and reliably pass it through unchanged across all 11 target languages.

Implementation:

18.3 Translation Workflow

API error messages are translated using the translate-messages KCC command — a lightweight Bedrock pipeline separate from the document translation engine:

1. Source: English messages from Aurora error_messages table (21 keys)
2. Sentinel wrap: Template variables ([usage], [limit]) and brand terms are wrapped in sentinel tokens (⟦T1⟧...⟦Tn⟧) before the model sees them
3. Translation: Bedrock translates the prose around the sentinels. The standing model is Qwen 3 235B, and it is selectable per language via translation_language_profiles.model_override — so do not read a specific model name here as fixed. This line said "Claude Sonnet 4.6" until 2026-08-30, months after Qwen became the standing choice.
4. Restoration: Sentinels are replaced back with the original [variable] text and brand terms
5. Storage: Translated messages are written back to Aurora (one row per language per key)
6. Cache: LPO server loads messages from Valkey (errmsg:{lang}:{key}) on first access

The entire process is idempotent — running it again overwrites with fresh translations. No S3, no CloudFront, no search index rebuild. Just Aurora + Valkey.

18.4 Sentinel Protection for Variables

The translate_messages.py helper uses the same sentinel wrap/unwrap pattern as the document translation engine, but at a much smaller scale (single sentences vs. multi-KB chunks):

# Before translation:
"Monthly query limit reached. You have used [usage] of [limit] allowed queries."
→ "Monthly query limit reached. You have used ⟦T1⟧ of ⟦T2⟧ allowed queries."

# After translation (Spanish):
"Límite de consultas mensual alcanzado. Has usado ⟦T1⟧ de ⟦T2⟧ consultas permitidas."
→ "Límite de consultas mensual alcanzado. Has usado [usage] de [limit] consultas permitidas."

Protected items include:

• Template variables: [usage], [limit]
• Brand terms from translation-config.json (57 terms)
• Technical identifiers that appear in messages (exchange names, tier names)

Result: 21 messages × 11 languages = 231 translations completed with zero failures. Template variables survive intact in every language — the Go server can substitute real values at runtime regardless of the response language.

19. Translation ETA System (v5.2)

The ETA system predicts how long a translation job will take, based on historical per-pair duration data from Aurora. It provides time estimates at three touchpoints: job submission, status polling, and pre-submission planning.

19.1 Historical Calibration

The system maintains per-language speed metrics derived from the last 90 successful translation pairs per language (990 samples total across 11 languages). Calibration data lives in Valkey (tx:eta:calibration hash, 6-hour TTL) and is rebuilt automatically on every job completion.

Calibration Metrics (Per Language)

Observed Speed Factors (June 2026)

All median values are rounded to multiples of 3 — The Trinity Beast convention.

19.2 ETA Formula

For a job with N documents and M languages (running in parallel, one container per language):

per_language_seconds = startup_overhead + (total_chunks × lang_speed_factor)
job_eta = max(per_language_seconds for each language)
range_low = max(startup + total_chunks × p25 for each language)
range_high = max(startup + total_chunks × p75 for each language)

Languages run in parallel (one container each). The wall-clock time equals the slowest language (the bottleneck). The bottleneck language is identified and reported in every ETA response.

19.3 API Endpoints

19.4 Live Progress Tracking

For running jobs, the status endpoint returns an eta object with real-time progress:

{
  "estimated_seconds": 202,
  "estimated_low_seconds": 153,
  "estimated_high_seconds": 246,
  "estimated_formatted": "3m 22s",
  "estimated_range": "2m 33s – 4m 6s",
  "bottleneck_lang": "hi",
  "elapsed_seconds": 118,
  "remaining_seconds": 84,
  "remaining_formatted": "1m 24s",
  "percent_complete": 58
}

Progress is calculated as elapsed / estimated × 100, capped at 99% until the job actually completes. The remaining time decreases linearly — no smoothing or prediction adjustment mid-flight.

19.5 Auto-Calibration

Calibration rebuilds automatically in a background goroutine on every job completion (RebuildETACalibrationBackground() called from TranslateCompleteHandler). This ensures the speed factors reflect the most recent translation performance — model updates, infrastructure changes, or document complexity shifts are absorbed into the next prediction within one job cycle.

20. Integrity Invariants (v5.9)

Everything above describes what the engine does. This section describes what it is not allowed to do — thirty invariants, each one earned by an incident, each one enforced at a named point in the code and pinned by a test that fails if the enforcement moves.

They are documented here rather than left in the codebase because a translation pipeline fails in a particular way: quietly, and in the artifact rather than the process. A job reports succeeded, the email says every pair completed, the cost is right, and the published document is wrong. Every invariant below exists because that happened at least once.

20.1 Why This Section Exists

Three properties of this system make ordinary testing insufficient, and understanding them is the difference between reading the invariant list as bureaucracy and reading it as hard-won architecture.

The model is stochastic, so the same input can fail once and pass on retry. A defect that appears in one language and not the other ten is not evidence of a language-specific bug — it is the normal shape of model variance. That makes reproduction expensive and makes deterministic guards more valuable than diagnostic ones. Wherever a guard can be made exact, it is worth far more than a guard that is merely usually right.

The pipeline repairs as well as validates, and repair can create defects. There are more than twenty repair, recovery and restore functions between the model's output and the published bytes. Roughly half put back exact source bytes, which is always safe. The other half guess where a wrapper belongs, and a wrong guess is indistinguishable from correct output to anything downstream. The standing direction is to preserve deterministic restoration and prefer hard failure with full diagnostics over heuristic repair.

A check is only as good as the string it reads. This is the subtlest of the three and it produced Invariant 12. A gate can be perfectly correct, pass cleanly, and validate nothing — because the artifact it examined is transformed afterwards. Position in the pipeline is part of a check's correctness, not an implementation detail.

20.2 The Thirty Invariants

Each invariant states an obligation, names where it is enforced, and names the test that would catch its removal. The full incident narrative for each lives in INVARIANTS.md alongside the worker source.

Invariant 1 — Chunking must never change the document's text content

Splitting a document into chunks and reassembling it must be a byte-level identity operation on the text. The mechanism that broke this was an unconditional strip of the model's output, which discarded the boundary whitespace a chunk carried at its edges. Two chunks then joined with no space between them, fusing the last word of one to the first word of the next.

Enforced by capturing each chunk's leading and trailing whitespace from the source before translation and reattaching it afterwards, regardless of what the model returned. Pinned by a fuzz harness that walks the full offset space of a chunk boundary with several simulated model behaviours, plus a control proving it detects a deliberately planted defect.

Invariant 2 — A protected term must never end up glued to adjacent text

A brand term that survives translation but arrives fused to the neighbouring word — Valkey followed immediately by a translated word with no space — is preserved by every presence check and broken to every reader. Presence and correct placement are different properties, and a substring check answers only the first.

Enforced by a spacing check that compares the full glued run at each occurrence against the glued shapes the source already had, per occurrence rather than per term. The per-occurrence rule matters: the same short term can be a legitimate compound identifier in one place and ordinary prose in another within the same chunk, and a per-term check lets one occurrence's shape license a false verdict for the other.

Calibrated for Arabic, August 2026. The check extends a term's glued run past any adjacent alphanumeric character, and Arabic letters are alphanumeric — so the conjunction wa-, which Arabic orthography always writes bound to the following word, read as a lost space and hard-failed the chunk. Arabic was consequently the only language unable to publish this very document. The exemption is deliberately narrow: exactly one proclitic conjunction, nothing bound on the right, and Arabic only. Urdu uses the same letter but its conjunction is a free-standing word, so the exemption does not extend there.

Invariant 3 — A strip on model output must not destroy content a later step needs

Invariant 1's root cause generalised. Every call that trims whitespace from raw model output or extracted text was inventoried and classified. Two share the dangerous shape and are safe only because of a call-order dependency rather than by design — which is precisely the kind of safety that breaks silently when someone reorders code.

Enforced by documentation plus tests that pin the call order those two depend on, so a future reordering trips a test rather than shipping a defect. This one is honest about its own limits: it is a snapshot of the call sites that existed when it was written, not a check that re-derives the inventory.

Invariant 4 — A chunk marker must never sit at a structurally unsafe position

A marker placed inside an open element splits the document mid-structure, and the resulting fragments are not independently valid HTML. Several tools place markers automatically, so the constraint has to live where the placement happens rather than in a reviewer's judgement.

Enforced by moving a proposed marker to the nearest safe boundary rather than rejecting it, and by a validator that re-implements the server's exact line-recognition rule. The marker rule is stricter than it looks: a marker must sit alone on its own line. A marker glued to an adjacent element passes a naive search and is invisible to the splitter, so the intended boundary silently disappears into the previous chunk.

Invariant 5 — The corpus's actual state must be measured, not assumed

A job's reported status is a claim about the job, not about the document. The published artifact is the only authority on whether the published artifact is correct, and it lives on the CDN rather than on any developer's disk.

Enforced by audits that read the live CDN and enumerate from the English document set rather than from a language folder — a distinction that matters because the engine writes translations straight to S3, so a local language folder may not even contain the file that is live. Judged on absolute counts rather than ratios, because a share-of-content ratio flags legitimate short samples and produces noise.

Invariant 6 — No extracted placeholder may be permanently lost

Every extracted element must come back, regardless of which extraction mechanism produced it or why the model failed to echo its token. The incident was an entire thirty-row table dropped whole, because its token sat alone at the start of a chunk with almost no surrounding prose to anchor it.

Enforced by a three-tier recovery in the one function all extracted types funnel through: exact match, then fuzzy match for a garbled token, then anchor recovery for a token dropped outright with no trace to match against. Nothing upstream can guarantee a model never drops an isolated token, so recovery belongs at the point where every type converges rather than in a hope about model behaviour.

Invariant 7 — A chunk's output must never collapse into a repeated character

A model that loses its place can fill its entire output budget with one character. Two published Urdu documents each served a solid wall of a single glyph where prose should have been — over seven thousand repetitions in each — and both were reported as successful translations.

Enforced by a generation-time guard that rejects a chunk whose output degenerates into a repeated character or a short cycle, detected on run length rather than script. The script-based instinct is wrong here and the measurement proves it: one live case was thousands of Arabic diacritic marks inside Urdu's own script, which a cross-script check structurally cannot see.

Invariant 8 — A placeholder-shaped literal in the document's own content must never be replaced

Restoration replaces a placeholder token with the element it stands for. Any text that merely looks like a token is therefore indistinguishable from a real one and gets overwritten with unrelated lifted HTML.

This document is the reason the invariant exists. It documents the placeholder format and so contains roughly thirty placeholder-shaped literals. One of them was replaced by a real element, silently, with no model involved — reproduced with the model removed from the loop entirely, which is what established the defect as ours rather than the model's.

Enforced by scanning the source for placeholder-shaped literals before generating any token and starting the index above every literal found. Because a single counter is shared across all placeholder types, token equality requires index equality, so clearing the highest reserved index makes a collision arithmetically impossible.

Invariant 9 — HTML parsing must never alter the case of an SVG attribute name

HTML attribute names are case-insensitive; SVG attribute names are not. A general-purpose HTML parser lower-cases attribute names, which turns an inline diagram's coordinate-system declaration into an attribute browsers ignore. The graphic then loses its ability to scale — visibly broken, and invisible to every tag-count check, because the tag is still there and the attribute is still there.

Enforced by extracting inline vector graphics as raw bytes before the parser can normalise them.

Invariant 10 — Scaffolding removal must remove only scaffolding, from the right elements

Preprocessing enriches the source before translation — wrapping brand terms, marking code elements as non-translatable. Removing that scaffolding afterwards must be bounded by the excess over the authored document, and it must be targeted by content rather than by count.

The defect was an unconditional removal sitting a few lines below a branch that got the same problem right. Because the house convention requires every authored code element to carry the non-translate marker, "remove them all" is precisely backwards for this corpus — almost none of them are scaffolding. On this document the source carried 628 such attributes, preprocessing added 74, and all 702 were destroyed. Every published translation of every document shipped none of them.

Verified by a round trip that is byte-identical to the authored source across all 471 distinct code contents in this document.

Invariant 11 — One extracted element must produce exactly one copy

Restoration must produce a single copy of an element however many times the model echoed its token. The defect was a boolean presence check followed by an unbounded replacement: a boolean cannot distinguish one occurrence from five, and an unbounded replacement expands every one of them into a full copy of the lifted element.

Model doubling is not a hypothesis here. The sentinel system that preceded the current preprocessor carried an explicit pass to collapse duplicated tokens, commented as a bilingual output instinct. That defence was not carried over when content protection moved to the preprocessor, which makes the duplication a regression introduced by that migration rather than a new model behaviour. The current handling deliberately does not inherit the old pass's limitation of only collapsing adjacent duplicates.

Enforced by counting occurrences instead of testing presence: none means missing and still flows to Invariant 6's recovery untouched, one is the normal path, and more than one restores a single copy at the first position, removes the surplus tokens, reports the placeholder and the count, and fails validation.

Known boundary: this guarantees the element is not duplicated. It does not repair duplicated prose around a doubled token — in a bilingual-doubling event the model may emit a whole passage twice, and restoration cannot adjudicate which copy to keep. The warning says so, and the corpus audits are where that surfaces.

Invariant 12 — Something must validate the bytes that actually ship

A gate that validates a string the pipeline then transforms has validated nothing. At least one check must observe the artifact as it will be written.

An audit of gate positions found that none of the full-document gates observed the published bytes. All of them ran at one point, and the output was then reassigned seven more times before the write with nothing validating any of it. Two of those reassignments are content transformations acting on wrappers and non-translate markers — exactly the dimension the earlier gates check. So those gates compared two strings, neither of which shipped, and the shipped artifact was then altered in the very dimension they had just validated.

Enforced by a differential check immediately before the write, which asks only what the final transformations did. That sequence is deterministic code, so its contract is exact — and an exact check is safe there precisely because it cannot be tripped by the stochastic model drift the earlier tolerance-based gate exists to absorb. Four contracts: scaffolding removal may only reduce non-translate markers and never below the authored count; wrapper repair may only add wrappers; path and language rewrites may not change any tag count; and no structural tag may lose or gain count across the sequence.

It fails hard rather than downgrading. The earlier gate's downgrade path exists so stochastic model drift does not discard a paid-for translation — but a finding here cannot come from the model, so it means the engine's own deterministic code misbehaved.

Invariant 13 — No regular expressions anywhere in the engine

Every string operation in the engine uses direct comparison: membership, equality, find(), character walking, or a real DOM parser. Regular expressions are excluded from the engine entirely — not discouraged, not restricted to non-validation use. Excluded, with the same absolute standing that forbidding omitempty has in the platform's response envelope: the construct does not appear in this codebase, and a reviewer who finds one has found a defect rather than a style disagreement.

This is the only invariant on the list that constrains the source rather than the pipeline, and it is here because every other one kept tracing back to it. A pattern matches characters. Every job in this engine — is this term preserved, is this element intact, did this wrapper survive, is this boundary correct — is a question about meaning. Where those two diverge the pattern still returns an answer, and the answer is silently wrong.

Three failures paid for this rule, each one silent:

• An element-extraction pattern used a non-greedy body with a backreference to its own opening tag. On a nested element of the same name it paired the outer opening tag with the inner closing tag, so the captured element was truncated and the remainder was left behind as loose prose for the model to translate, trailing an orphaned closing tag. A pattern cannot count nesting depth. Nothing about that limitation is exotic — the term wrapper produces exactly that nesting for any protected term that is a substring of a longer one, and the current term list contains four such pairs.
• An address-preservation check found a sample address in the raw source, inside an element already marked as non-translatable, and demanded the same address in the translated output. The extraction had done its job correctly; the check did not know that. It failed two languages repeatedly for ten days while nine others passed by coincidence.
• A term-preservation check asked whether a protected term was present. A term fused to the following word contains the term, so the check was satisfied and the fusion validated clean. Thirty-two of forty-three published documents carried that defect before anyone measured for it.

Enforced by a repository audit that walks every engine source file and reports any import of the pattern library or any call into it. It runs as a standing leg of the daily audit sweep and returns a non-zero exit on any finding, so a reintroduction fails the sweep rather than waiting to be noticed. It was verified in both directions before being trusted — clean across the engine, red on a deliberately induced violation, clean again after reverting it.

One declared exception. A test may reinstate a retired pattern as a differential oracle — running the old pattern and the new direct-comparison implementation across the entire corpus and asserting identical output. That is how a conversion earns confidence instead of receiving it, and the conversion that closed out this invariant was proven that way across every English source in the library: identical output, four hundred and twenty wrappers rewritten, zero disagreements. A file claiming the exception must declare it with an explicit marker so the exemption is auditable rather than assumed, and the oracle should be retired once the property it proved can be stated directly.

Invariant 14 — A translated fragment must land in the slot it was translated for

Prose-mode translation sends the model several fragments in one call and gets back a labelled list — fragment 3's translation goes in slot 3, and nothing beyond the model's own label ties the two together. A model that merges two fragments while continuing to count shifts every later fragment one slot into its neighbour's place. The result is well-formed, correctly scripted, and tag-count-identical to the source, because nothing about a shift changes any of the properties those checks measure. It shipped an Arabic section heading reading "the duplicate" and a key card holding the back half of an unrelated sentence, with the job reporting succeeded.

Two deterministic rules, neither needing a model call. Displacement: a unique anchor from fragment N found in a different slot proves the content moved — stated as presence in the wrong place rather than absence from the right one, because absence alone is ambiguous (the model may simply have mangled the identifier) while presence elsewhere is proof. Deletion: a slot left empty while its source fragment was not is content silently dropped, with the writer producing an unremarkable whitespace-only text node.

Enforced by a check run before the DOM is touched, on the model's raw labelled response. The engine's own retry path takes one corrective pass at a higher temperature before treating a persistent finding as a hard failure; a caller-side recovery fallback that keeps the original document when a fix cannot be made declines rather than guesses, honouring its own contract instead of shipping a repair it cannot verify.

Known boundary: a swap between two anchor-less fragments sharing a parent is undetectable by this mechanism — it needs semantic comparison, which means another model call, and that tradeoff was declined rather than hidden. The gap is named rather than closed, and the corpus audits are the backstop that can still catch it after publication.

Invariant 15 — Two individually correct restorations must not touch with no separator

Invariant 11 guarantees one placeholder produces exactly one copy of its element. It says nothing about what happens when two different, individually healthy placeholders happen to land next to each other during restoration. The defect this closes is two distinct product names arriving glued with zero space between them — the same brand term doubled with no separator, from two entirely separate sentences — where neither placeholder was missing, neither was duplicated, and every existing check that looks at one restoration at a time reports it clean.

The model dropped every character of connecting prose between two adjacent-in-output placeholders. The main restoration loop asks "is this token okay" for each entry in isolation and never asks what now sits on either side of a splice it just made. A join is exactly the place that question has to be asked, because a join is the only point where two independently-correct pieces first touch.

Enforced by a guard applied at every splice point in restoration, reused rather than duplicated from the join logic anchor recovery already needed for the same reason. It inserts a single space only when concatenating the two sides bare would present two touching alphanumeric characters to a reader — the same visible-character test, deliberately, that the tail's brand-term glue repair uses, rather than a second independent definition of "glued" that could drift from the first. Japanese and Chinese are exempted, because those scripts legitimately attach a foreign term to adjacent text with no space, and inserting one there would itself be the defect.

Surfaced as a parity-audit regression — three known defects becoming seven overnight — rather than found by reading code, which is the same discipline as Invariant 5: the corpus's actual state is what proved something had changed, not a hunch about what might have.

Invariant 16 — A response cut off at the model's token ceiling must never ship as a completed translation

Every call into Bedrock returns a stop signal distinguishing a response that finished from one cut off for exhausting its output budget. Nothing before this invariant read that signal. A truncated response is real, well-formed text right up to the cutoff — it fails no content check by virtue of being incomplete, because incompleteness is not a property content checks can see. It is a property of how much content exists at all, and a document is missing everything past the cutoff while every tag-count and brand-term check reports what little text arrived as clean.

A published document shipped with its footer and every closing tag cut off entirely. The failing chunk hit the exact same output-token count on all four attempts — a ceiling divided evenly by the retry count, meaning it was not occasionally near the limit, it was pinned to it every single time. The raw response confirmed the real cause directly: the model was cut off mid-reasoning, before writing a single character of the actual translation, and the pipeline shipped what little text existed anyway.

Enforced by reading the provider's real stop signal per model family and treating a truncated attempt as a hard failure that retries through the existing loop rather than falling through to a return — exhausting every retry raises rather than shipping the partial result. An unclosed reasoning block is treated as truncation evidence in its own right, since the step that would otherwise close it deliberately leaves an unclosed one untouched rather than guessing where it should end.

Relationship to Invariant 7: Invariant 7 catches a chunk that degenerates — collapses into a repeated character while still consuming its full budget doing so. This invariant catches a chunk that runs out of budget before finishing, producing clean, non-repeating text that simply stops. Different failure shape, same root category: a completion-time content loss no placeholder-based recovery can see, because there was never a placeholder standing in for the missing content. Nothing was ever generated to recover.

Invariant 17 — A token format the prompt does not name will be rewritten into one it does

Every placeholder or sentinel format that can reach the model must be named explicitly in its instructions, and those instructions must forbid converting one format into another or inventing a format absent from the input. A chunk that returns fewer placeholders of a given format than it was sent is a hard failure that retries.

One extraction pass lifts protected content into a placeholder format the rest of the engine treats as opaque — it neither mints that format nor restores it, so nothing verified the tokens came back. Meanwhile the model's instructions named every other placeholder format unconditionally and said nothing about this one. Given tokens it could see and instructions about formats it could not, the model did the plausible thing: it rewrote what it had into what it had been told about, and in doing so satisfied the instructions' own counting rule. From the model's own point of view, it had complied.

What finally solved it was reading the pipeline directly rather than reasoning about it from a symptom three layers downstream. A neighbouring chunk with more placeholders round-tripped perfectly through identical code, which eliminated density, size and structural imbalance as causes in one comparison. The failing chunk sent fifty of its own placeholders and returned zero of them — while returning over fifty tokens in the format the model had been told about, tokens the engine's own minting function had not created. The engine did not produce them. The model did.

Every retry had failed identically because temperature was zero and the input never changed — a deterministic-input fault is precisely the shape a retry cannot fix, the same lesson this engine already carries from the placeholder-residue invariant's own reverted blocking gate.

Enforced by two changes made together, because neither alone was sufficient. The instructions now name the previously-unmentioned format explicitly, forbid converting between formats, and count survival per format rather than globally — closing the cause. And a chunk returning fewer of that format than it was sent is now a hard failure routed into the existing retry loop — inert whenever none were sent, so ordinary chunks are unaffected — catching any recurrence loudly instead of letting it corrupt a document silently. The prompt fix removes the trigger; the gate makes a future instance loud rather than invisible.

Invariant 18 — No engine-minted placeholder residue may appear in the published bytes

Every manifest-driven recovery in this pipeline — Invariant 6's three-tier recovery, the fuzzy match, the anchor recovery — asks one question: did the element I extracted come back? That is the absence side of the question. Nothing asked the presence side: is there placeholder-shaped residue sitting in what is about to ship. A document could restore every manifest entry, pass every gate above, and still publish garbage in its prose, because none of those gates read for a token shape — they read for tag counts and term presence.

The German copy of this very document once shipped four raw sentinels in a single sentence explaining a different subsystem — the same content class this section now documents, published as literal tokens rather than the product names they stood for. A corpus sweep at the time found leaked tokens in over a tenth of all published pairs, across every one of the eleven languages, concentrated in the languages whose scripts already run densest.

Enforced by a check immediately before the write that compares every engine-minted token shape against the English source: anything the source legitimately carries — this document discusses its own sentinel syntax in prose, so it is allowed to contain the shapes it explains — passes, and anything the shipped bytes carry that the source does not is stripped from the bytes and reported as a loud warning rather than blocking the publish. It does not block, deliberately: the prior version did block, and making roughly two dozen documents permanently unpublishable while calling the failure "recoverable" was itself the defect, once the model was shown to fail the exact same way on retry.

A companion audit sweeps the already-published corpus for the same shapes, because this gate only stops new residue from shipping — it does nothing for what published before it existed. Neither replaces the other, the same relationship the degeneration guard has with its own corpus audit.

Invariant 19 — A chunk without its own document wrapper still gets the document's own size discipline

The chunker's job is to split a document into pieces small enough for reliable translation, and that job does not change just because the piece it was handed is itself already a fragment of a larger one. A caller that hands the chunker a bare slice of body content — no surrounding document — must receive the same size-bounded packing a full document would, not a single unsplit pass-through.

The chunker's entry point locates a document by its opening body tag before deciding how to split it. When that tag is absent — which is exactly the shape of every marker-delimited piece an upstream splitter produces, since a slice of body content never carries its own body tag — the function had one branch left: hand the whole fragment back as a single chunk, no packing applied. That branch was written for what its own comment called an unusual case. It was, in fact, the case every marker-based document hits on every one of its pieces, every time, because the upstream splitter's output is nothing but bare fragments.

The consequence sat quietly underneath a language's own configured size ceiling. A fragment several times over that ceiling went to the model as one uninterrupted chunk, because the layer meant to enforce the ceiling never ran on it — it had already exited through the branch that skips packing entirely. The languages with the tightest ceilings, set tightest because their scripts need the most room per character, were exposed to the largest oversized chunks relative to their own limit.

Enforced by routing the no-wrapper case through the identical marker-check-then-packing logic a full document already receives, rather than exiting early into an unpacked pass-through. A fragment that carries markers still splits on them, deterministically, exactly as before. A fragment that does not is now packed against its target language's real size ceiling, the same discipline every full document already had. Verified against the two live fragments that had been silently exceeding their ceiling by more than double: both now split into multiple correctly-sized pieces, and a fragment already under its ceiling is confirmed to remain a single chunk — the fix adds packing where it was missing without over-splitting where it was never needed.

Invariant 20 — A retry after a specific, identified failure must correct that specific failure, not restate a generic warning

A retry exists to give the model a genuine second chance, and a second chance that repeats the same generic caution the first attempt already had is not a second chance — on a deterministic model call, an unchanged instruction produces the same error again. When a retry is triggered by a hard failure whose cause is already known and already named, the correction offered on that retry must name the same thing.

A chunk that failed on a specific placeholder-survival violation — the model overwriting one token format with another — was, on retry, handed only the pipeline's general-purpose caution about protected terms and version numbers. That caution says nothing about token formats. The retry consumed a full attempt and a full Bedrock call while asking the model to be more careful about a problem it did not have, leaving the actual problem completely uncorrected.

Enforced by carrying the specific hard-failure name forward from one attempt to the next, and selecting a directed correction that restates the exact rule violated whenever that name matches a defect with a known, nameable cause. The general caution remains the default for every other failure shape, so a retry answers what actually went wrong rather than a fixed script of caution that may or may not apply.

Invariant 21 — A protective extraction that hides text from the model must also arrange for that text to be translated

Lifting an element whole to keep the model from mishandling its wrapper is a valid defense against one failure — but if the lifted element's own visible text was the reason to translate it in the first place, the defense has traded one failure for another. An extraction that protects a wrapper by hiding everything inside it, including ordinary prose, must be paired with a step that still translates that prose, or the fix silently becomes the new source of untranslated content.

Short bold and emphasis labels were lifted as complete, single-piece placeholders specifically because the model tended to drop the wrapping tag around them. That defense worked exactly as designed — the wrapper always survived. It also meant the model never received the label's own text as something to translate, so restoration correctly put back exactly what had been lifted: the original English, verbatim, every time. A measurement across a published document found the majority of its bold labels — product tier names, short lead-in phrases — still in English, and every affected label was short enough to match the extraction's own threshold precisely, which is what confirmed the extraction itself as the cause rather than a model shortcoming.

Enforced by the same repair already proven for an identical gap in plain-text links: after restoration, the visible text inside a lifted label is extracted a second time, translated in one small batch call, and patched back into that specific element only. An element whose text comes back unchanged — a protected brand term the model correctly declined to translate — is left exactly as restored, and an element whose only content sits behind its own protection boundary is skipped before any call is made.

Invariant 22 — A part of a split document must carry the same unmatched-tag shape as its own source

A document too large for one model call is split into parts, and a part is almost never internally balanced: a container opened in part 0 may legitimately close in part 6. So "does this part balance" is the wrong question. The right one is whether the part's boundary signature — the list of tags whose partner lives in another part — is identical in the source part and the translated part. A whole-document tag count cannot see this: the counts stay equal while the nesting moves. Measured when it broke, 114 unmatched closers and 109 unmatched openers were dropped across 38 published pairs, and one lost closer ended a layout wrapper 14% into a page so the rest of the document rendered outside it. Every existing gate passed, because the div counts matched the English exactly.

Enforced by encoding each unmatched boundary tag into an inert comment sentinel that carries its own original text, so it survives the model call rather than being hidden from the parser and then handed to the model. Pinned by a same-input, same-position comparison of the two signatures, which has no false-positive mode on a correctly translated part.

Invariant 23 — An element whose prose must stay translatable cannot be lifted whole

Protecting an element by replacing it with a single placeholder guarantees its tag survives and guarantees its text is never translated — the model never sees the words. For a short label that trade is acceptable and is repaired afterwards. For a long emphasised passage it is not: the reader gets a paragraph of English inside an otherwise translated page.

Enforced by protecting a long emphasis wrapper as two independent placeholders, one for the opening tag and one for the closing tag, with the prose left visible between them. Corpus exposure for this shape went from 252 affected elements to zero. Pinned by a threshold measured on authored text rather than on the engine's own tokens — the first version of the measurement counted placeholder characters and picked a boundary that protected the wrong things.

Invariant 24 — A recovered element must never be inserted into markup

When the model drops a placeholder outright, the engine recovers the element by locating a surviving neighbour and inserting the stored content next to it. Locating that neighbour by plain string search over the whole restored document will happily return an offset inside an HTML comment, inside an attribute value, or in the middle of a tag. Reproduced offline at no cost: the anchor was the four-character string 12.2, and the insertion landed inside a comment, so three emphasis labels per language vanished from the page while every tag count still matched.

Enforced by a safe-position gate that refuses any offset not in document text. Unsafe relocations across the corpus went from 44 documents to zero. Pinned by a test asserting no recovered element may be placed inside a comment, an attribute, or a tag.

Invariant 25 — An element's position is its position in the document, not its order of extraction

Rejecting a bad insertion point only moves an element to the next safe wrong place. Closing the harmful shape of Invariant 24 left the position defect intact, and the labels simply moved from a comment into the following heading — visible and wrong, where before they were invisible and wrong. Extraction runs in phases, so a dense block can hold a low index while sitting late in the page: measured, extraction order disagreed with document order for 15,319 of 15,909 tokens across 430 of 443 parts.

Enforced by three changes measured separately, which is the part worth remembering. Assigning a true document-order position and sorting recovery by it moved almost nothing on its own — 611 clean to 612 — and it is the obvious fix. What delivered the improvement was recovering in token space, before substitution, when every anchor is still a unique token rather than content that may appear identically elsewhere. The third was a scope bound plus a requirement that both flanking entries survive, because document order says which entries flank a dropped one but nothing about how far away they are: in one measured case a term inside a title at offset 161 had its next neighbour at offset 5,414, past a stylesheet block and outside the document head.

Invariant 26 — No raw placeholder token may reach a reader through the recovery path

The content-anchored recovery pass runs after substitution and inserted its stored markup verbatim. An entry whose stored content held a nested token therefore had that token inserted with no pass left to resolve it, and the page rendered the literal text of an engine placeholder. Invisible to every gate in the pipeline: the tag counts matched, the placeholder genuinely was restored, and a leaked token is ordinary text rather than markup. Measured corpus-wide, 352 of 15,919 recoveries leaked a token; after the fix, none did.

Enforced by resolving nested tokens before insertion, with a bounded depth so a self-reference cannot loop — and by token-space recovery, which avoids the class by construction. Pinned by a test asserting no recovered element leaks a raw token.

Invariant 27 — A guard that refuses a retry must know when the engine itself changed

The engine records a pair's past failures and refuses to spend tokens repeating one. The question it has to answer is "has anything changed that could make this succeed," and that has two possible answers — the source document, or the engine. It consulted only the first. So immediately after an engine fix, the pairs that most needed re-translating were precisely the ones permanently refused, and silently, because a refusal is terminal.

Enforced by stamping the build time into the image and admitting a retry when either the engine or the source is newer than the recorded failure. Confirmed live: of twelve queued pairs, the seven carrying failure history from a month earlier were all admitted without any manual override. Pinned by a deliberate negative result — the stamp is generated at build time and is absent from a fresh checkout, so the guard warns loudly rather than failing open, and the protection is image-only by design.

Invariant 28 — A model-added closing tag may be dropped only when it is provably surplus

Because a long emphasis wrapper reaches the model as two sentinels rather than as tags (Invariant 23), the model sometimes returns both sentinels and a literal closing tag, having inferred that the prose it just translated was emphasised. After restoration there is one closer too many and the boundary-signature gate correctly refuses to publish. The gate was right; what was missing was a repair for the one shape that is safe to fix.

Enforced by a repair that runs before the gate and only ever removes a tag it can prove is redundant. Four conditions must all hold: the part's unmatched openers already match the source; every extra closer is inline emphasis, never a structural container; the opening-tag count for that name is unchanged from the source; and the signature must equal the source's after the deletion, re-measured rather than assumed. The third condition is the proof — eighteen opens and nineteen closes against a source of eighteen and eighteen leaves the nineteenth closer with nothing it could possibly close, whereas seventeen opens means an opener was lost and the extra closer is its symptom, so the repair declines and the gate still rejects. Pinned by a suite weighted toward the refusal cases, because the naive version of this fix passes a test that only checks the live case and would erase the evidence of a lost opener.

Invariant 29 — Glue must be judged in rendered text, on both sides of the comparison

Invariant 2 compares a term's glued run against the glued shapes the source already had, which is the correct rule. It was applied to raw markup, and that quietly broke it in both directions. A protected term always sits inside its own protective wrapper, so when the model fuses the next word to it the fusion lands after the closing tag, where a scan over markup stops. And an identifier the author fused on purpose — written by hand with a protective wrapper in the middle of it — reads as new glue on the source side.

So the same check simultaneously invented defects and hid them. It reported a comfortable zero from August to October while 125 genuinely glued terms sat on the published corpus, including German text a reader can simply see. The lesson generalises past this engine: a zero from a check that cannot fail is indistinguishable from a zero from a clean corpus. A half-fix is what made the zero look earned — correcting only the source side still measures zero, which is why the gap survived six weeks after the principle was already written down in the code.

Enforced by reducing both sides to the text a reader sees before any comparison: inline elements vanish with no separator, every other element becomes a single space, so a table-cell boundary stays a word break and cell adjacency invents nothing. Measured before the change rather than after: of the repairs the wider scope enables, 98 restore a real lost space and 9 would have split an authored identifier. Pinned by tests in both directions, including the defect itself so that a revert cannot pass silently.

Invariant 30 — An authored identifier must be repaired or refused, never silently mangled

Seeing more is only an improvement if the repair that follows is right. One live finding class is not a lost space at all but a corrupted name: the model translated the tail of an identifier, so TranslateS3AccessRole shipped as TranslateS3AccessRolle in nine languages. Inserting a space there produces TranslateS3AccessRo lle — measured, that is exactly where the repair's authored-prefix walk lands — which trades a loud, catchable defect for a quiet published one.

Enforced by a refusal rule in the full-document repair: a run whose authored identifier is not intact is reported and left alone, so the gate hard-fails the pair instead of publishing a mangled name. The discriminator is containment rather than a shared prefix, and getting that wrong is easy — an authored compound that survives whole with a translated word glued onto its end is repairable, and a prefix-only rule refuses it. The rule deliberately does not live in the shared position-finding helper, whose verdict must stay identical to the gate's so that a correct scaffolding strip can never hard-fail a paid translation.

And the inverse defect has no check at all. An authored identifier can also be split: BedrockCostForTokens publishes as two words in all nine space-delimited languages, and FlushToElastiCache in one. Every existing check passes it by construction — presence is a substring test, which a split satisfies, and the spacing checks only fire when a run is longer than the term, where a split makes it exactly equal. Both identifiers are now protected terms so the model cannot split them again, and one needed its English markup rewrapped because the protection pass matches within a single text node and could not see an identifier divided by a tag. The missing piece is a gate, and its shape is the mirror of the glue check: an authored run present in the source whose characters appear in the output separated by whitespace. It was found by pre-flighting identifiers before a paid re-translation, which is the finding as much as the defect is.

20.2a Pipeline Tracing — Seeing the Path a Document Actually Took

The tooling that finally separated a genuine cause from a plausible-looking symptom didn't exist yet when the search for Invariant 17's fault began. Two build-and-deploy cycles were spent chasing a caption because there was no way to see, cheaply, what any single pipeline step had actually done to the document passing through it — Aurora records what was processed, not what happened at the moment it was processed. This is the instrument that fills that gap, and it exists because reasoning from a symptom cost two deploy cycles before it caught the actual defect.

It is deliberately not a new gate, not a new audit leg, and not anything that runs unless asked. Off means zero work, by design. Every entry point returns immediately when the switch is unset, so a normal translation carries no cost from this existing at all — the exact discipline behind not adding anything that recurs on a clock. Turning it on is a single environment variable, and it works identically whether the worker is running locally or inside the ECS container, so the same switch answers the same question wherever the failure actually lives.

What it reports, per pipeline step: a glue finding before and after the step, computed with the exact same rule and baseline the real gate uses, so a step that introduces glue is named outright instead of inferred from which step ran most recently. A length delta. And counts of the inline elements that carry this defect class most often — a recovery mechanism that re-injects content as an appended pile shows up here as a count that moves without the surrounding prose moving with it, which is precisely how Invariant 17's root cause was finally identified rather than merely located.

Per translated part, before any placeholder restore runs, it also records how many placeholder tokens of each opaque format were sent to the model against how many came back — the single measurement that distinguished a part the model damaged from a part the engine's own recovery later mishandled, which nothing else in the pipeline records at all.

flowchart TD
    P[Pre-build gate: regression suite + engine source audit] --> A
    A[Authored document from S3] --> B["Source validation: repairs always, rejects internal jobs only"]
    B --> C[Snapshot authored baseline]
    C --> D[Enrich: wrap terms, protect code, absolutise paths]
    D --> E[Split on chunk markers]
    E --> F[Per chunk: extract elements to placeholders]
    F --> G[Bedrock translation]
    G --> H[Per chunk: restore placeholders]
    H --> I[Reassemble chunks]
    I --> J[Full-document gates]
    J --> K[Final transformations]
    K --> L[Differential tail gate]
    L --> M[Write to S3]
    M --> N[Out-of-band corpus audits]
    I1[Inv 1 and 3: boundary whitespace] --- E
    I4[Inv 4: safe marker position] --- E
    I8[Inv 8: reserve literal indices] --- F
    I9[Inv 9: SVG bytes before parsing] --- F
    I7[Inv 7: degeneration guard] --- G
    I16[Inv 16: truncation gate] --- G
    I17[Inv 17: placeholder-format survival] --- G
    I6[Inv 6: three-tier recovery] --- H
    I11[Inv 11: one entry one copy] --- H
    I14[Inv 14: fragment identity] --- H
    I15[Inv 15: adjacent-splice glue guard] --- H
    I2["Inv 2: term spacing, skipped for the 8 word_spaced=false languages"] --- J
    X1["Customer job: definitive source equals target, pair skipped, lang_rejected target_is_source"] --- G
    X2["Customer job: fail-closed cancel check before publish"] --- M
    I10[Inv 10: bounded scaffolding removal] --- K
    I18[Inv 18: placeholder-residue strip-and-warn] --- L
    I12[Inv 12: validate shipped bytes] --- L
    I5[Inv 5: measure the corpus] --- N
    I13[Inv 13: no regex in engine source] --- P
    style P fill:#064e3b,stroke:#10b981,color:#e0e0e0
    style A fill:#1e3a5f,stroke:#60a5fa,color:#e0e0e0
    style G fill:#7c2d12,stroke:#f97316,color:#e0e0e0
    style J fill:#3f3f46,stroke:#a1a1aa,color:#e0e0e0
    style K fill:#3f3f46,stroke:#a1a1aa,color:#e0e0e0
    style L fill:#064e3b,stroke:#10b981,color:#e0e0e0
    style M fill:#1e3a5f,stroke:#60a5fa,color:#e0e0e0
    style N fill:#064e3b,stroke:#10b981,color:#e0e0e0
    style X1 fill:#92400e,stroke:#fbbf24,color:#e0e0e0
    style X2 fill:#7f1d1d,stroke:#f87171,color:#e0e0e0
        

The diagram makes one thing plain that a list cannot: the invariants cluster where the artifact changes hands. Extraction, generation, restoration and the final transformation sequence are where content is created, replaced or removed, and that is where every enforcement point sits. The out-of-band audits at the end exist because a pipeline cannot certify its own output — only the published bytes can do that. Generation gained two more clusters since this diagram was first drawn: a truncation gate that reads the model's own stop signal, and a format-survival gate watching a class of placeholder the rest of the pipeline treats as opaque. Restoration gained two of its own: a check that a translated fragment landed in the slot it was translated for, and a guard at every splice point ensuring two individually correct restorations don't touch with no separator between them.

20.3 The Four Content Defect Classes

Content defects sort into four classes, and the distinction is operational rather than academic: each needs a different detector and a different response. A document can be structurally valid, correctly scripted, free of glued terms, and still be wrong in a way that only one of these four finds.

The absence class carries the subtlest trap in the whole detection story. Size must be normalised against each language's own typical expansion, never against English. A language that normally expands to roughly 140 percent of the English source and measures 100 percent is missing content — while looking perfectly healthy to a check that compares against English alone. Comparing against English produced a false all-clear on the first measurement.

And absence has two causes that need opposite responses. A stale revision is fixed by re-translating that document. A term that failed to preserve at otherwise-normal document size is not fixed by re-translation — it recurs, and multi-word terms dominate that bucket. Treating the second as the first burns tokens and changes nothing.

20.4 The Regression Suite and the Pre-Build Gate

Thirty-eight test files sit alongside the worker, and the build refuses to produce an image if any of them fails. That refusal is the point: before it existed, running the suite before a deploy was a discipline someone had to remember rather than a property the tooling enforced. The count has more than doubled since this section was first written — new invariants arrive with their own tests, never as an addition to an existing one, and the suite's growth is the visible record of that discipline holding.

The runner discovers test files by glob rather than from a list. That is the same "derive it, do not hardcode it" discipline applied elsewhere in this fleet's tooling, and it exists because a hand-maintained list of checks has gone stale here more than once. Adding a test file requires no edit to the runner.

Two properties of these tests are worth stating, because they are what makes the suite trustworthy rather than merely present.

Every fix carries a negative control for the case a naive version would get wrong. Not a test proving the original bug is caught — a test proving the plausible wrong fix is rejected. Several of these were written before the corresponding fix and caught a genuine flaw in the first attempt: the scaffolding removal's count-limited draft preserved the right number of attributes on the wrong elements, and only a control written in advance surfaced it.

Several tests assert that a check can fail. They reproduce the pre-fix behaviour and assert it produces the wrong answer. Without that, a green run cannot distinguish "the fix works" from "these assertions are incapable of failing" — and this codebase has a documented history of checks that returned a clean, confident, wrong answer.

20.5 Known Boundaries

Stating what these guarantees do not cover is part of the guarantee. An overstated boundary is how a clean audit becomes a false all-clear.

• The invariants protect elements, not prose. Duplicated or missing prose around a correctly-restored element is outside every gate above. The corpus audits are the only layer that sees it.
• Multi-word protected terms are not reliably preserved at normal document size, and re-translation does not fix that class. It is an open item rather than a solved one.
• Invariant 3 is a snapshot. It inventories the call sites that existed when it was written. A new one added later is not automatically checked.
• The gate-position audit describes the pipeline as it is. It deliberately pins current positions rather than desired ones, because the build refuses to produce an image when the suite fails and a permanently-failing test would block every deploy. A pipeline change makes it fail loudly and demand an update, which is the property it was built for — but it is a tripwire, not a proof of correctness.
• Deployment is not exercise. A gate verified locally and inside the shipped image has not yet run against real work. The first live translation after a deploy is the actual test, and the distinction between "verified" and "exercised in production" is one worth keeping.
• A finding string names a location, not a cause. Invariant 12's differential tail gate reports glue found anywhere across several mutation steps with one shared message, because it is checked once, after all of them. Two real fixes were built and deployed against that message's literal claim before anyone confirmed which step actually introduced the glue — both were genuine defects, and neither was the one this finding was reporting. Section 20.2a's tracing instrument exists specifically to answer "which step, really" before proposing a fix.

Quick Reference