**Section 5 Testing — Where We Are:**

**What We Know Well (Ready to Test):**

1. **Routing Logic** — Language-based, not country-based. We understand `diversion_language`, `country_code`, `market_code` fields. Can validate tasks land in correct queues.

2. **QA Sampling Configuration** — Real config in `extra.custom_dsl.choice_mode`, not top-level `sample_ratio`. Three-layer QA flow confirmed. Can validate configured rates match actual rates.

3. **Template Consistency** — `jimu_id` should be consistent across related queues. We found clone queues with different templates. Can validate.

4. **Decision Tree Mapping** — `decision_tree_name` follows `live_aim_dt_rpf_[policy_code]` pattern. Can verify production matches SOP.

5. **Task Lifecycle** — Status codes, abandonment reasons, linkage via `previous_task_id`. Can trace task flow.

**What We Still Need to Explore:**

1. **VQA/Themis** — Limited reconnaissance. Different data structure (`object_data` vs `incoming_features_extra`). Need to confirm routing fields and table access.

2. **TCS Manager Logs** — Haven't accessed. Needed for decision tree change detection.

3. **DataEngine/Quokka Config** — VQA injection pipeline. Haven't mapped.

---

**Testing Phases:**

**Phase 1: Routing Correctness (Live AIM)**
- Already have leads: US content in non-US queues, language mismatches
- Risk: Misrouted tasks = wrong labelers, wrong language expertise, potential USDS compliance issue
- Test: Quantify routing anomalies across all queues

**Phase 2: Configuration Consistency**
- Template IDs across queues
- Sampling config vs actual sampling rates
- Decision tree versions vs SOP
- Risk: Inconsistent configs = inconsistent labeling quality

**Phase 3: Data Quality Indicators**
- Abandonment patterns (system vs labeler)
- "Cannot Play" rates
- NML rates in visual queues (shouldn't happen per SOP)
- Risk: Elevated rates = upstream injection issues or config problems

**Phase 4: VQA Reconnaissance + Testing**
- Confirm data access
- Map routing fields
- Apply same tests as Live AIM

---

**Immediate Next Step:**

Start Phase 1 — quantify the routing anomalies we already spotted. Run a comprehensive query across all Live AIM Formal queues checking for country/language mismatches that could indicate systematic routing issues.

Want me to draft the Phase 1 test scripts?


**Section 5 Testing Readiness Assessment**

---

**What We Know Well (Ready to Test):**

1. **Routing Logic** — Language-based routing confirmed. Tasks route by `diversion_language`, not `country_code`. We have the field mappings and can validate.

2. **Queue Configuration** — Template IDs (`jimu_id`), sampling configs (`custom_dsl`), decision tree names all accessible in `dim_project.extra`. We can compare against One-Pager requirements.

3. **QA Sampling Architecture** — Three-layer flow confirmed (Labeling → BPOQA → BPOQA Quality Check). Linkage via `previous_task_id`. Strategy trees in JSON, not top-level `sample_ratio`.

4. **Task Data Structure** — `incoming_features_extra` fields mapped for Live AIM. Routing fields, model scores, batch names, tier/priority all understood.

---

**Gaps (Need More Reconnaissance):**

1. **VQA/Themis** — We have queue inventory but haven't deeply explored task-level data. Object_data structure differs from Live AIM. Need to confirm routing fields exist in equivalent form.

2. **Decision Tree Versioning** — We see `decision_tree_name` in tasks but haven't validated against SOP documentation to confirm correct version deployed.

3. **TCS Manager Logs** — Not yet accessed. Needed for decision tree change visibility per Dmitry's request.

---

**Recommended Testing Phases:**

**Phase 1: Configuration Correctness (R11-C01)**
- Start here — lowest complexity, highest certainty
- Compare `dim_project.extra` settings against One-Pager for sampled queues
- Check: template ID consistency, language config, sampling strategy, resource admins
- This establishes baseline: is the system configured as documented?

**Phase 2: Routing Validation (R10-C01)**
- Validate `diversion_language` matches queue language designation
- Check for misroutes (wrong language in queue)
- Quantify the US content in non-US queues observation
- Validate `decision_tree_name` matches `policy_code` mapping

**Phase 3: Injection Data Quality (R08-C02)**
- Abandonment rates by reason (system vs labeler)
- NML (Not My Language) rates — especially Visual queues
- Cannot Play rates
- Score distribution by policy — are thresholds consistent?

**Phase 4: VQA Reconnaissance + Testing**
- Run equivalent scripts on VQA tables
- Validate routing structure exists
- Compare against VQA One-Pager

---

**First Action:**

Start Phase 1. Pull configuration for 3-5 Live AIM Audio queues (ID, AR, EN) and compare against the One-Pager. Document any deviations. This gives you concrete findings quickly while building toward deeper testing.

Let me think through this systematically by reviewing the transcript for our section 5 controls and then mapping what we know against what we need to test.

Let me pull the actual risk and control framework from the transcript to ground this properly.

OK — I don't have the original risk/control framework verbatim from the planning documents since those were in an earlier context. But based on what the transcript references and what we've discussed, let me lay out where we are and propose a phased testing approach.

---

## Readiness Assessment: What We Know vs. What We Need

### What's Confirmed Through Reconnaissance

**Live AIM (strong foundation):**
- Complete table schemas validated (4 tables)
- Routing logic understood (language-based via TikHawk)
- Three-layer QA architecture mapped with task linkage proven
- Sampling configurations extracted for all queues (1%–20%, with daily caps)
- Two injection pathways identified (standard training vs RCA)
- Policy/model coverage documented across 9 policies
- Volume patterns, batch structures, score thresholds all characterized
- Template consistency verified (one exception set: EN clones)

**VQA/Themis (partial foundation):**
- Queue inventory mapped (All Titles + policy-specific + Batch 2)
- Data lives in `dwd_tcs_task_di` (no dedicated injection table)
- `object_data` JSON structure understood from raw records
- Routing confirmed as language-based via Hawk
- Multi-policy labeling structure understood
- Volume patterns characterized (much lower than Live AIM)
- Weekly targets exist for policy-specific queues (2,100/week), absent for All Titles
- QA queues identified (Filipino-QA, Indonesia-QA) but not yet explored

**Still Missing:**
- Historical `dim_project` partition availability (Script U never confirmed)
- Permission table schema (`dwd_tcs_risk_doorgod_kani_permission_df` — Section 6, not 5)
- Themis QA layer details (how do Filipino-QA / Indonesia-QA queues work?)
- DataEngine/Quokka tables (do they exist in Hive?)
- Full strategy tree JSON for non-ID Live AIM queues

---

## Phased Testing Approach

### Phase 1: QA Sampling Configuration & Effectiveness (Highest Risk)

**Why first:** This is the heart of the control environment. If sampling isn't working as designed, labeled data quality is unverified. We already have the configuration data — we just need to test whether reality matches config.

**What we already know that's concerning:**
- Sampling rates vary 1%–20% across Live AIM queues with no documented justification for differentiation
- Some queues (NP) have daily caps disabled
- BPOQA `resource_mode` says "Inhouse" but process docs say BPO QA does the work
- Themis Hausa/Hebrew have 0% sampling on All Titles queues — potentially no QA at all
- VQA policy queues have weekly targets; All Titles queues don't

**Testing actions:**

**(a) Effective sampling rate validation — Live AIM.** For each Formal queue, compare Layer 1 completed volume against Layer 2 BPOQA received volume, by week, across the audit period. We need to check whether the configured rates + daily caps produce the observed BPOQA volume. We've done this for ID Formal (got ~18% effective vs 7% configured). Now scale across all languages.

```sql
-- Layer 1 completed per queue per week
SELECT project_id, project_title, 
    CONCAT(YEAR(date), '-W', LPAD(WEEKOFYEAR(date),2,'0')) AS week,
    COUNT(*) AS completed_tasks
FROM tnscdm.dwd_tcs_task_verify_di
WHERE date BETWEEN '20250101' AND '20251231'
  AND product_desc = 'TikTok'
  AND business_desc IN ('TikTok Video','TikTok Live Offline')
  AND sample_index = 0
  AND project_id IN (SELECT id FROM tnscdm.dim_project 
    WHERE date = '20251231' AND title LIKE '%AIM%Audio%Formal%' 
    AND title NOT LIKE '%BPOQA%' AND status = 3)
GROUP BY project_id, project_title, 
    CONCAT(YEAR(date), '-W', LPAD(WEEKOFYEAR(date),2,'0'))
ORDER BY project_title, week;
```

Then do the same for BPOQA queues and divide. **But first** — we need to confirm historical partition availability. That's the prerequisite.

**(b) Zero-QA queue identification.** Flag any Formal queue where BPOQA received zero tasks during any calendar month in the audit period. Hausa and Hebrew Themis are suspects based on 0% sampling config.

**(c) Strategy tree vs One-Pager comparison.** Extract the full `custom_dsl.choice_mode` for all Formal queues and compare sampling rates + caps against whatever the project requirements document specifies. This requires getting the One-Pager/project requirements from the engagement team if we don't have them.

---

### Phase 2: Routing Integrity (Medium-High Risk)

**Why second:** If tasks route to wrong queues, labelers apply wrong-language expertise, producing garbage training data.

**What we already know:**
- Routing is language-based, confirmed working for the majority of traffic
- Minor `en` language bleed into non-EN queues (~0.1% of volume) — likely RCA tasks
- Country codes are metadata, not routing determinants
- Hawk `split_features` contain country + language

**Testing actions:**

**(a) Language mismatch audit.** For each queue, count tasks where `diversion_language` doesn't match the queue's designated language. We already have Script I results showing this. Scale it across the audit period.

**(b) Confirm en bleed = RCA tasks.** Run Script W (already drafted) to verify the `en` tasks in non-EN queues are all `Live_AIM_RCA_DT` with NULL `source_type`.

**(c) VQA routing verification.** Extract `__hawk.split_features.language` from `object_data` in `dwd_tcs_task_di` for Themis queues and verify language alignment. This is more complex because `object_data` is a massive JSON, but we've seen the structure.

**(d) Template consistency check.** Confirm all production Formal queues use the correct template ID. We've already done this — the EN clone queues with different template are the exception to investigate.

---

### Phase 3: Data Quality Indicators (Medium Risk)

**Why third:** Abandoned tasks, NML rates, and cannot-play rates indicate systemic issues with injection quality or labeler assignment.

**What we already know:**
- ID Formal outcome distribution: 74.7% No Violation, 24.9% Violation, 0.2% NML, 0.2% Cannot Play, 0.07% Abandon
- These are low anomaly rates for a single queue snapshot
- We haven't looked at this across queues or over time

**Testing actions:**

**(a) Abandonment rate by queue and month.** Pull `status = 4` (abandoned) tasks with `abandon_reason` from `dwd_tcs_task_di` across the audit period. Identify queues with elevated abandonment.

**(b) NML rate analysis.** NML (Not My Language) at the labeling outcome level indicates routing failures at the content level — the labeler got content they couldn't understand. Track NML rates by queue over time.

**(c) Cannot-Play rate analysis.** Content that can't be played suggests injection of expired/broken media. Track by queue and batch.

---

### Phase 4: Injection Volume Consistency (Lower Risk for Controls, Important for DR13)

**Why fourth:** DR13 is a data request, not a control test per se, but the volume analysis feeds into understanding whether controls operate over sufficient population.

**What we already know:**
- Live AIM has extreme daily volatility (min 1–3 tasks, max 30,000–48,000)
- No configured injection targets for Live AIM
- Themis policy queues have 2,100/week targets; All Titles don't
- BPOQA queues are more stable due to cap constraints

**Testing actions:** Mostly done through INJ-A through INJ-F. Package the output, compare Themis actuals to 2,100/week targets.

---

## Immediate Next Step: Prerequisite Gate

Before Phase 1 testing begins, we need **one thing confirmed**:

**Can we access audit period data?** Run this now:

```sql
SELECT COUNT(*) AS task_count
FROM tnscdm.dwd_tcs_task_verify_di
WHERE date = '20250615'
  AND product_desc = 'TikTok'
  AND business_desc IN ('TikTok Video','TikTok Live Offline')
LIMIT 1;
```

If this returns data, we're clear to start Phase 1. If it fails or returns zero, we have a data availability problem that changes everything.

**Second prerequisite:** Do you have the project requirements document (One-Pager) that specifies what the sampling rates, templates, and QA configurations *should* be? Without the "intended" to compare against "actual," we can identify inconsistencies across queues but can't definitively say something is misconfigured.