diff --git a/skills/explaining-batch-data-transform/SKILL.md b/skills/explaining-batch-data-transform/SKILL.md index 8695c01..47b4ff8 100644 --- a/skills/explaining-batch-data-transform/SKILL.md +++ b/skills/explaining-batch-data-transform/SKILL.md @@ -48,7 +48,7 @@ Before any further work, validate by running `python bdt_analyze.py summary ` to list them. Then offer the user a choice — *"The payload has 3 definitions: MyTransform, OtherTransform, ThirdTransform. Which do you want me to explain?"* — and invoke subsequent subcommands with `--definition N` (0-indexed). Editor exports and single-definition payloads can be explained directly without this step; running `definitions` on them still succeeds and shows one row, so it is safe to run on any input if you are unsure. +**Multi-definition payloads.** If the user provides a Connect API payload with a `definitions[]` array containing multiple entries, first run `python bdt_analyze.py definitions ` to list them. Then offer the user a choice — *"The payload has 3 definitions: MyTransform, OtherTransform, ThirdTransform. Which do you want me to explain?"* — and invoke subsequent subcommands with `--definition N` (0-indexed). For example: `python bdt_analyze.py summary --definition 1`. All analysis subcommands (summary, sources, outputs, stages, nodes, node, lineage, field-trace, formula) accept the `--definition` flag. Editor exports and single-definition payloads can be explained directly without this step; running `definitions` on them still succeeds and shows one row, so it is safe to run on any input if you are unsure. ## 4. Explanation Flow (Job A — Progressive Disclosure) @@ -86,7 +86,7 @@ When the user asks a follow-up question, route it to the right subcommand using | "What DMOs/DLOs does this read from?" (L4) | `sources ` | | "What fields does OUTPUT N produce?" (L5) | `node OUTPUT_N` | | "What does this formula mean?" (I1) | `formula X` | -| "Why would F be null/zero/unexpected?" (I2) | `field-trace F` **then** `formula ` | +| "Why would F be null/zero/unexpected?" (I2) | `field-trace F` to locate the defining node(s), **then** `formula ` on the formula or computeRelative node that computes the field (not the output mapping — pick the earliest formula/computeRelative node in the field-trace output). | | "What does this filter do?" (I3) | `node X` | After running the subcommand(s), translate the output into plain English. For I1/I2 you **must** consult `references/bdt-function-catalog.md` (and `references/bdt-window-functions.md` if it's a `computeRelative` node) before narrating. @@ -119,7 +119,7 @@ After running the subcommand(s), translate the output into plain English. For I1 |---|---| | Exit 3: `Invalid JSON at line X` | "This file isn't valid JSON at line X. Could you re-export from the BDT viewer?" Then paste the exact error. | | Exit 3: `Expected top-level 'nodes' object` | "This doesn't look like a BDT export — a BDT has a top-level `nodes` object. Is this the right file?" | -| Exit 3: `Cycle or broken reference detected` | "This BDT has a dependency cycle (or a reference to a node that doesn't exist). A valid BDT shouldn't cycle — could you re-export and share? The unresolved nodes are: \." | +| Exit 3: `Cycle or broken reference detected` | "This BDT has a dependency cycle or references a non-existent node. You'll need to edit the BDT in the Data Cloud viewer to fix the circular dependency or broken reference — re-exporting won't help because the cycle exists in the BDT definition itself. The problematic nodes are: \." | | Exit 2: `No node named 'X'` | "I don't see a node named `X` in this BDT. Did you mean one of: \?" | | Exit 2: `Field 'F' is not defined by any node` | "That field isn't defined anywhere in this BDT. Closest matches from the fields I can see: \. Is it possibly a field from the source DMO that's not pulled in?" | | User's BDT uses an action type not in the catalog | "This BDT uses ``, which isn't in my reference material. Here's what its parameters look like: \. I can describe the graph structure around it, but I can't vouch for the exact semantics of this action type." | @@ -158,7 +158,7 @@ Common issues and how to address them: | `ModuleNotFoundError: No module named 'bdt_analyze'` | The script path is wrong. The script lives at `scripts/bdt_analyze.py` inside the skill directory. Run it by absolute path (the Bash tool invocation should include the full path from the repo root). | | BDT file path contains spaces (e.g. `"default B2C (Prod).json"`) | Quote the path when calling the script: `python bdt_analyze.py summary "default B2C (Prod).json"`. | | Script output is very large for a big BDT (Mode C) | Use `--limit 50` on the `nodes` subcommand, or recommend Mode B (layered) instead of Mode C. | -| Field trace returns no match despite the field clearly appearing in the BDT | The field may only appear qualified (e.g., `SalesOrder.ssot__Id__c`). Try the qualified form. Also try stripping the `__c` suffix. | +| Field trace returns no match despite the field clearly appearing in the BDT | The field may only appear qualified (e.g., `SalesOrder.ssot__Id__c`). Try the qualified form. If still no match, the field may be passed through from a source DMO but not explicitly defined in the BDT — in that case field-trace cannot locate it; narrate "this field originates from the load node's source DMO" instead. Do NOT attempt to trace it by stripping the `__c` suffix — that would risk matching a different field with a similar name. | | User says output "doesn't match what I see in BDT viewer" | Verify the version in the JSON header matches the org's current release. Canonical schema was synced against `core-264`, but older/newer BDTs may use variants not yet in the reference. Flag as "version drift" and narrate best-effort. | ## 11. Example Interactions diff --git a/skills/explaining-batch-data-transform/assets/sample_bdts/append_and_split.json b/skills/explaining-batch-data-transform/assets/sample_bdts/append_and_split.json index 3e79fd7..5e6053f 100644 --- a/skills/explaining-batch-data-transform/assets/sample_bdts/append_and_split.json +++ b/skills/explaining-batch-data-transform/assets/sample_bdts/append_and_split.json @@ -6,7 +6,7 @@ "sources": [], "parameters": { "dataset": {"name": "WebOrders__dlo", "type": "dataLakeObject"}, - "fields": ["OrderId__c", "Amount__c", "Channel__c", "CustomerFullName__c"], + "fields": ["OrderId__c", "Amount__c", "ChannelOrderKey__c"], "sampleDetails": {"type": "TopN", "sortBy": []} } }, @@ -15,7 +15,7 @@ "sources": [], "parameters": { "dataset": {"name": "StoreOrders__dlo", "type": "dataLakeObject"}, - "fields": ["OrderId__c", "Amount__c", "Channel__c", "CustomerFullName__c"], + "fields": ["OrderId__c", "Amount__c", "ChannelOrderKey__c"], "sampleDetails": {"type": "TopN", "sortBy": []} } }, @@ -25,55 +25,53 @@ "parameters": { "allowImplicitDisjointSchema": false, "fieldMappings": [ - {"targetField": "OrderId__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "OrderId__c"}, {"node": "LOAD_STORE_ORDERS", "field": "OrderId__c"}]}, - {"targetField": "Amount__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "Amount__c"}, {"node": "LOAD_STORE_ORDERS", "field": "Amount__c"}]}, - {"targetField": "Channel__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "Channel__c"}, {"node": "LOAD_STORE_ORDERS", "field": "Channel__c"}]}, - {"targetField": "CustomerFullName__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "CustomerFullName__c"}, {"node": "LOAD_STORE_ORDERS", "field": "CustomerFullName__c"}]} + {"targetField": "OrderId__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "OrderId__c"}, {"node": "LOAD_STORE_ORDERS", "field": "OrderId__c"}]}, + {"targetField": "Amount__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "Amount__c"}, {"node": "LOAD_STORE_ORDERS", "field": "Amount__c"}]}, + {"targetField": "ChannelOrderKey__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "ChannelOrderKey__c"}, {"node": "LOAD_STORE_ORDERS", "field": "ChannelOrderKey__c"}]} ] } }, - "SPLIT_CUSTOMER_NAME": { + "SPLIT_CHANNEL_KEY": { "action": "split", "sources": ["APPEND_ALL_ORDERS"], "parameters": { - "sourceField": "CustomerFullName__c", - "delimiter": " ", + "sourceField": "ChannelOrderKey__c", + "delimiter": "-", "targetFields": [ - {"name": "CustomerFirstName__c", "label": "Customer First Name"}, - {"name": "CustomerLastName__c", "label": "Customer Last Name"} + {"name": "Channel__c", "label": "Channel"}, + {"name": "OrderNumber__c", "label": "Order Number"} ] } }, "OUTPUT_ORDERS": { "action": "outputD360", - "sources": ["SPLIT_CUSTOMER_NAME"], + "sources": ["SPLIT_CHANNEL_KEY"], "parameters": { - "name": "OrdersWithCustomerName__dlm", + "name": "OrdersWithChannel__dlm", "type": "dataModelObject", "writeMode": "OVERWRITE", "fieldsMappings": [ - {"sourceField": "OrderId__c", "targetField": "OrderId__c"}, - {"sourceField": "Amount__c", "targetField": "Amount__c"}, - {"sourceField": "Channel__c", "targetField": "Channel__c"}, - {"sourceField": "CustomerFirstName__c", "targetField": "FirstName__c"}, - {"sourceField": "CustomerLastName__c", "targetField": "LastName__c"} + {"sourceField": "OrderId__c", "targetField": "OrderId__c"}, + {"sourceField": "Amount__c", "targetField": "Amount__c"}, + {"sourceField": "Channel__c", "targetField": "Channel__c"}, + {"sourceField": "OrderNumber__c", "targetField": "OrderNumber__c"} ] } } }, "ui": { "nodes": { - "LOAD_WEB_ORDERS": {"label": "Web Orders", "type": "LOAD_DATASET", "top": 100, "left": 100}, - "LOAD_STORE_ORDERS": {"label": "Store Orders", "type": "LOAD_DATASET", "top": 260, "left": 100}, - "APPEND_ALL_ORDERS": {"label": "Union", "type": "APPEND", "top": 180, "left": 260}, - "SPLIT_CUSTOMER_NAME": {"label": "Split full name", "type": "SPLIT", "top": 180, "left": 420}, - "OUTPUT_ORDERS": {"label": "Orders with name", "type": "OUTPUT", "top": 180, "left": 580} + "LOAD_WEB_ORDERS": {"label": "Web Orders", "type": "LOAD_DATASET", "top": 100, "left": 100}, + "LOAD_STORE_ORDERS": {"label": "Store Orders", "type": "LOAD_DATASET", "top": 260, "left": 100}, + "APPEND_ALL_ORDERS": {"label": "Union", "type": "APPEND", "top": 180, "left": 260}, + "SPLIT_CHANNEL_KEY": {"label": "Split channel key", "type": "SPLIT", "top": 180, "left": 420}, + "OUTPUT_ORDERS": {"label": "Orders + channel", "type": "OUTPUT", "top": 180, "left": 580} }, "connectors": [ - {"source": "LOAD_WEB_ORDERS", "target": "APPEND_ALL_ORDERS"}, - {"source": "LOAD_STORE_ORDERS", "target": "APPEND_ALL_ORDERS"}, - {"source": "APPEND_ALL_ORDERS", "target": "SPLIT_CUSTOMER_NAME"}, - {"source": "SPLIT_CUSTOMER_NAME", "target": "OUTPUT_ORDERS"} + {"source": "LOAD_WEB_ORDERS", "target": "APPEND_ALL_ORDERS"}, + {"source": "LOAD_STORE_ORDERS", "target": "APPEND_ALL_ORDERS"}, + {"source": "APPEND_ALL_ORDERS", "target": "SPLIT_CHANNEL_KEY"}, + {"source": "SPLIT_CHANNEL_KEY", "target": "OUTPUT_ORDERS"} ] } } diff --git a/skills/explaining-batch-data-transform/assets/sample_bdts/joins_and_filters.json b/skills/explaining-batch-data-transform/assets/sample_bdts/joins_and_filters.json index 132a355..1db8bae 100644 --- a/skills/explaining-batch-data-transform/assets/sample_bdts/joins_and_filters.json +++ b/skills/explaining-batch-data-transform/assets/sample_bdts/joins_and_filters.json @@ -25,9 +25,10 @@ "parameters": { "filterExpressions": [ {"type": "TEXT", "field": "ssot__Status__c", "operator": "EQUAL", "operands": ["Active"]}, - {"type": "NUMBER", "field": "ssot__GrandTotalAmount__c", "operator": "GREATER_THAN", "operands": ["0"]} + {"type": "NUMBER", "field": "ssot__GrandTotalAmount__c", "operator": "GREATER_THAN", "operands": ["0"]}, + {"type": "NUMBER", "field": "ssot__GrandTotalAmount__c", "operator": "IS_NOT_NULL", "operands": []} ], - "filterBooleanLogic": "1 AND 2" + "filterBooleanLogic": "1 AND 2 AND 3" } }, "JOIN_ORDER_ACCOUNT": { diff --git a/skills/explaining-batch-data-transform/assets/sample_bdts/window_and_aggregate.json b/skills/explaining-batch-data-transform/assets/sample_bdts/window_and_aggregate.json index ff59375..4c9a127 100644 --- a/skills/explaining-batch-data-transform/assets/sample_bdts/window_and_aggregate.json +++ b/skills/explaining-batch-data-transform/assets/sample_bdts/window_and_aggregate.json @@ -23,7 +23,7 @@ "label": "Order Rank", "formulaExpression": "row_number()", "type": "NUMBER", - "businessType": "Number", + "businessType": "NUMBER", "precision": 18, "scale": 0, "defaultValue": "" @@ -31,13 +31,33 @@ ] } }, + "FIRST_ORDER_AMOUNT": { + "action": "formula", + "sources": ["RANK_ORDERS"], + "parameters": { + "expressionType": "SQL", + "fields": [ + { + "name": "FirstOrderAmount__c", + "label": "First Order Amount (per account)", + "formulaExpression": "case when OrderRank__c = 1 then ssot__GrandTotalAmount__c else 0 end", + "type": "NUMBER", + "businessType": "NUMBER", + "precision": 18, + "scale": 2, + "defaultValue": "0" + } + ] + } + }, "AGG_BY_ACCOUNT": { "action": "aggregate", - "sources": ["RANK_ORDERS"], + "sources": ["FIRST_ORDER_AMOUNT"], "parameters": { "groupings": ["ssot__AccountId__c"], "aggregations": [ {"action": "SUM", "name": "TotalAmount__c", "source": "ssot__GrandTotalAmount__c"}, + {"action": "SUM", "name": "FirstOrderTotal__c", "source": "FirstOrderAmount__c"}, {"action": "COUNT", "name": "OrderCount__c", "source": "ssot__Id__c"} ], "nodeType": "STANDARD" @@ -53,6 +73,7 @@ "fieldsMappings": [ {"sourceField": "ssot__AccountId__c", "targetField": "AccountId__c"}, {"sourceField": "TotalAmount__c", "targetField": "TotalAmount__c"}, + {"sourceField": "FirstOrderTotal__c", "targetField": "FirstOrderAmount__c"}, {"sourceField": "OrderCount__c", "targetField": "OrderCount__c"} ] } @@ -60,14 +81,16 @@ }, "ui": { "nodes": { - "LOAD_ORDERS": {"label": "Sales Orders", "type": "LOAD_DATASET", "top": 100, "left": 100}, - "RANK_ORDERS": {"label": "Rank by account", "type": "COMPUTE_RELATIVE", "top": 100, "left": 260}, - "AGG_BY_ACCOUNT": {"label": "Totals per account", "type": "AGGREGATE", "top": 100, "left": 420}, - "OUTPUT_SUMMARY": {"label": "Account Summary", "type": "OUTPUT", "top": 100, "left": 580} + "LOAD_ORDERS": {"label": "Sales Orders", "type": "LOAD_DATASET", "top": 100, "left": 100}, + "RANK_ORDERS": {"label": "Rank by account", "type": "COMPUTE_RELATIVE", "top": 100, "left": 260}, + "FIRST_ORDER_AMOUNT": {"label": "Extract first order amount", "type": "FORMULA", "top": 100, "left": 420}, + "AGG_BY_ACCOUNT": {"label": "Totals per account", "type": "AGGREGATE", "top": 100, "left": 580}, + "OUTPUT_SUMMARY": {"label": "Account Summary", "type": "OUTPUT", "top": 100, "left": 740} }, "connectors": [ {"source": "LOAD_ORDERS", "target": "RANK_ORDERS"}, - {"source": "RANK_ORDERS", "target": "AGG_BY_ACCOUNT"}, + {"source": "RANK_ORDERS", "target": "FIRST_ORDER_AMOUNT"}, + {"source": "FIRST_ORDER_AMOUNT", "target": "AGG_BY_ACCOUNT"}, {"source": "AGG_BY_ACCOUNT", "target": "OUTPUT_SUMMARY"} ] } diff --git a/skills/explaining-batch-data-transform/references/bdt-function-catalog.md b/skills/explaining-batch-data-transform/references/bdt-function-catalog.md index 7d986e9..89906b5 100644 --- a/skills/explaining-batch-data-transform/references/bdt-function-catalog.md +++ b/skills/explaining-batch-data-transform/references/bdt-function-catalog.md @@ -34,7 +34,7 @@ |---|---|---| | `abs(n)` | number → number | Absolute value (strips sign). | | `ceiling(n)` | number → number | Round up, away from zero for negatives. | -| `floor(n)` | number → number | Round toward negative infinity (down on the number line). For negatives, rounds away from zero (e.g., `floor(-2.3) = -3`). | +| `floor(n)` | number → number | Round toward negative infinity. Always rounds down on the number line: `floor(2.7) = 2`, `floor(-2.3) = -3`. | | `exp(n)` | number → number | e raised to n. | | `log(base, n)` | number → number | Logarithm of n in the given base. | | `max(a, b, …)` | number → number | Largest value. | diff --git a/skills/explaining-batch-data-transform/references/bdt-node-catalog.md b/skills/explaining-batch-data-transform/references/bdt-node-catalog.md index a1b4b85..06c981c 100644 --- a/skills/explaining-batch-data-transform/references/bdt-node-catalog.md +++ b/skills/explaining-batch-data-transform/references/bdt-node-catalog.md @@ -211,8 +211,9 @@ semantics — for that, see `computeRelative`. - `label` — user-visible label. - `formulaExpression` — SFSQL (or DCSQL) expression. See `bdt-function-catalog.md`. - `type` — enum `DataType`: `TEXT`, `NUMBER`, `BOOLEAN`, `DATE_ONLY`, `DATETIME`. -- `businessType` — a user-facing business-semantic type name. Enum `BusinessTypeEnum` — - canonical values: +- **`businessType`** — a user-facing business-semantic type name. Enum `BusinessTypeEnum` — + **canonical values** (complete list; matches `BusinessTypeEnum.java` in core-262 as of the + capture date at the top of this file): - `"TEXT"`, `"NUMBER"`, `"BOOLEAN"` - `"EMAIL"`, `"PHONE"`, `"URL"` — text-valued with semantic meaning - `"PERCENT"`, `"CURRENCY"` — number-valued with semantic meaning @@ -222,8 +223,9 @@ semantics — for that, see `computeRelative`. Note: `businessType` values map to underlying `type` (`DataType`) values. E.g., `businessType: "PERCENT"` is stored as `type: "NUMBER"`; `businessType: "DATE"` is stored - as `type: "DATETIME"`; `businessType: "EMAIL"` is stored as `type: "TEXT"`. The canonical - mapping lives in `BusinessTypeEnum.java`. + as `type: "DATETIME"`; `businessType: "EMAIL"` is stored as `type: "TEXT"`. If the skill + ever encounters a `businessType` value outside this list, that is a sign the upstream BDT + schema has evolved — surface the raw value in narration and flag it as undocumented. - `precision` — integer precision (default 10 for numbers; characters for text). - `scale` — decimal places; only for NUMBER. - `defaultValue` — value when the expression yields NULL. diff --git a/skills/explaining-batch-data-transform/scripts/bdt_analyze.py b/skills/explaining-batch-data-transform/scripts/bdt_analyze.py index 919f5ca..235d270 100644 --- a/skills/explaining-batch-data-transform/scripts/bdt_analyze.py +++ b/skills/explaining-batch-data-transform/scripts/bdt_analyze.py @@ -128,7 +128,7 @@ class DataTransform: """Pick which definition the accessor properties (and the rest of the API surface) operate on. Raises `BdtNotFoundError` if `index` is out of range.""" - if not isinstance(index, int) or index < 0 or index >= len(self.definitions): + if type(index) is not int or index < 0 or index >= len(self.definitions): raise BdtNotFoundError( f"Definition index {index} out of range; payload has " f"{len(self.definitions)} definitions " @@ -296,31 +296,32 @@ class DataTransform: def topo_order(self) -> List[str]: """Kahn's algorithm. Raises BdtInputError on cycles.""" - # Adjacency: for each node, its forward edges (consumers) and in-degree + from collections import deque in_degree: Dict[str, int] = {name: 0 for name in self.nodes} forward: Dict[str, List[str]] = {name: [] for name in self.nodes} for name, n in self.nodes.items(): for s in n.sources: if s not in self.nodes: - # Broken reference — counted as in-edge so the dependent node stays blocked - # (we surface broken refs via a separate method; topo should not crash here). + # Broken reference — count as in-edge so the dependent node stays blocked in_degree[name] += 1 continue forward[s].append(name) in_degree[name] += 1 - # Start with all zero-in-degree nodes in deterministic order (sorted) - ready = sorted([name for name, d in in_degree.items() if d == 0]) + # Sort zero-in-degree nodes once for deterministic starting order + ready = deque(sorted([name for name, d in in_degree.items() if d == 0])) order: List[str] = [] while ready: - # Pop deterministically so output is stable across runs - ready.sort() - current = ready.pop(0) + current = ready.popleft() order.append(current) - for consumer in sorted(forward[current]): + # Collect newly-ready nodes, sort them once, append + newly_ready = [] + for consumer in forward[current]: in_degree[consumer] -= 1 if in_degree[consumer] == 0: - ready.append(consumer) + newly_ready.append(consumer) + for node in sorted(newly_ready): + ready.append(node) if len(order) != len(self.nodes): stuck = [n for n in self.nodes if n not in order] @@ -1309,14 +1310,15 @@ def cmd_formula(bdt: "DataTransform", args) -> str: # For each consumed field, find the node that defines it (if any) — give # the LLM ready context for I2-style reasoning. Hoist `bdt.upstream(name)` # out of the loop — it's a pure function of `name` and walks the graph. + # Pre-compute fields_produced for every upstream node once (was O(consumed × upstream)) upstream_names = bdt.upstream(name) + upstream_fields_map = {up_name: fields_produced(bdt.nodes[up_name]) for up_name in upstream_names} upstream_defs = {} for f in consumed: - # Prefer exact match; fall back to unqualified matches = [] for up_name in upstream_names: up = bdt.nodes[up_name] - prod = fields_produced(up) + prod = upstream_fields_map[up_name] if f in prod or f.rsplit(".", 1)[-1] in prod: matches.append({ "node": up_name, @@ -1324,7 +1326,7 @@ def cmd_formula(bdt: "DataTransform", args) -> str: "label": up.ui_label, }) if matches: - upstream_defs[f] = matches[-1] # closest definer (last in topo order) + upstream_defs[f] = matches[-1] payload = { "node": name, diff --git a/skills/explaining-batch-data-transform/tests/test_bdt_analyze.py b/skills/explaining-batch-data-transform/tests/test_bdt_analyze.py index d360685..04572d7 100644 --- a/skills/explaining-batch-data-transform/tests/test_bdt_analyze.py +++ b/skills/explaining-batch-data-transform/tests/test_bdt_analyze.py @@ -38,8 +38,8 @@ class TestBadInput(unittest.TestCase): def test_invalid_json_raises(self): p = FIXTURES / "_tmp_invalid.json" - p.write_text("{not valid json") self.addCleanup(p.unlink, missing_ok=True) + p.write_text("{not valid json") with self.assertRaises(bdt_analyze.BdtInputError) as cm: bdt_analyze.DataTransform.from_path(p) self.assertIn("Invalid JSON", str(cm.exception)) @@ -711,15 +711,16 @@ class TestInputShapeDetection(unittest.TestCase): for subcmd, extra_arg in [("summary", None), ("stages", None), ("nodes", None), ("sources", None), ("outputs", None), ("lineage", "OUTPUT_X"), ("node", "LOAD_X")]: - argv = [subcmd, str(FIXTURES / "api_input_single.json")] - if extra_arg: argv.append(extra_arg) - out = io.StringIO(); err = io.StringIO() - try: - with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): - code = bdt_analyze.main(argv) - except SystemExit as e: - code = e.code - self.assertEqual(code, 0, f"Subcommand {subcmd} failed: {err.getvalue()}") + with self.subTest(subcmd=subcmd): + argv = [subcmd, str(FIXTURES / "api_input_single.json")] + if extra_arg: argv.append(extra_arg) + out = io.StringIO(); err = io.StringIO() + try: + with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err): + code = bdt_analyze.main(argv) + except SystemExit as e: + code = e.code + self.assertEqual(code, 0, f"Subcommand {subcmd} failed: {err.getvalue()}") class TestMultiDefinition(unittest.TestCase):