mirror of
https://github.com/forcedotcom/afv-library.git
synced 2026-08-09 00:42:46 +08:00
fix: address PRizm round-2 review findings from internal port @W-22196528@
Ports the 14 applicable round-2 fixes from the internal PRizm review to the external afv-library source. (plugin.json finding #13 is internal-only and does not apply here.) Accepted fixes: 1 bdt_analyze.py select_definition: isinstance(index, int) -> type(index) is int (reject bool subclass of int). 2 bdt_analyze.py cmd_formula: precompute upstream fields_produced map to eliminate O(consumed x upstream) recomputation. 3 bdt_analyze.py topo_order: replace sort-on-every-iteration with deque-based Kahn (preserves deterministic order). 4 test_bdt_analyze.py test_all_subcommands_work_on_api_input: wrap loop in self.subTest for per-iteration failure reporting. 5 test_bdt_analyze.py test_invalid_json_raises: register addCleanup BEFORE write_text so cleanup runs even if write fails. 6 SKILL.md: remove misleading "strip __c suffix" field-trace troubleshooting advice; replace with source-DMO passthrough guidance. 7 SKILL.md: rewrite cycle error message to direct user to Data Cloud viewer (re-export will not fix a cycle in the BDT definition). 8 SKILL.md: add explicit --definition N flag documentation and list every subcommand that accepts it. 9 SKILL.md: refine I2 routing guidance — pick the earliest formula/ computeRelative node, not the output mapping. 11 bdt-function-catalog.md: clean floor() wording ("always rounds down on the number line"). 12 bdt-node-catalog.md: businessType enum — assert list is canonical/complete; instruct skill to surface+flag any unknown value. 14 append_and_split.json: change split subject from fragile CustomerFullName__c split-on-space to ChannelOrderKey__c split-on-hyphen; avoids name-parsing pitfall (middle names / multi-space). 15 window_and_aggregate.json: use the computed OrderRank__c via a new FIRST_ORDER_AMOUNT formula node (case when rank=1 then amount else 0) feeding AGG_BY_ACCOUNT; illustrates the rank+formula+aggregate idiom. 16 joins_and_filters.json: add explicit IS_NOT_NULL filter expression on GrandTotalAmount__c (defense-in-depth beyond GREATER_THAN 0). Rejected finding (rationale posted as PR comment): 10 bdt-node-catalog.md split section — reviewer claimed split is a pipeline-branching node. Canonical sources (core-262 SplitParametersInputRepresentation.java + Salesforce help DITA c360_a_batch_transform_split.xml) both describe split as a string-splitting operation. Current docs are correct; not changing. Tests: 92/92 passing.
This commit is contained in:
Gaurav Bajpai
parent
db1aca7221
commit
67d919947f
@ -48,7 +48,7 @@ Before any further work, validate by running `python bdt_analyze.py summary <pat
|
||||
|
||||
### 3.1 Multi-definition payloads
|
||||
|
||||
If the user provides a Connect API payload with a `definitions[]` array containing multiple entries, first run `python bdt_analyze.py definitions <path>` to list them. Then offer the user a choice — *"The payload has 3 definitions: MyTransform, OtherTransform, ThirdTransform. Which do you want me to explain?"* — and invoke subsequent subcommands with `--definition N` (0-indexed). Editor exports and single-definition payloads can be explained directly without this step; running `definitions` on them still succeeds and shows one row, so it is safe to run on any input if you are unsure.
|
||||
**Multi-definition payloads.** If the user provides a Connect API payload with a `definitions[]` array containing multiple entries, first run `python bdt_analyze.py definitions <path>` to list them. Then offer the user a choice — *"The payload has 3 definitions: MyTransform, OtherTransform, ThirdTransform. Which do you want me to explain?"* — and invoke subsequent subcommands with `--definition N` (0-indexed). For example: `python bdt_analyze.py summary <path> --definition 1`. All analysis subcommands (summary, sources, outputs, stages, nodes, node, lineage, field-trace, formula) accept the `--definition` flag. Editor exports and single-definition payloads can be explained directly without this step; running `definitions` on them still succeeds and shows one row, so it is safe to run on any input if you are unsure.
|
||||
|
||||
## 4. Explanation Flow (Job A — Progressive Disclosure)
|
||||
|
||||
@ -86,7 +86,7 @@ When the user asks a follow-up question, route it to the right subcommand using
|
||||
| "What DMOs/DLOs does this read from?" (L4) | `sources <path>` |
|
||||
| "What fields does OUTPUT N produce?" (L5) | `node <path> OUTPUT_N` |
|
||||
| "What does this formula mean?" (I1) | `formula <path> X` |
|
||||
| "Why would F be null/zero/unexpected?" (I2) | `field-trace <path> F` **then** `formula <path> <defining-node>` |
|
||||
| "Why would F be null/zero/unexpected?" (I2) | `field-trace <path> F` to locate the defining node(s), **then** `formula <path> <defining-node>` on the formula or computeRelative node that computes the field (not the output mapping — pick the earliest formula/computeRelative node in the field-trace output). |
|
||||
| "What does this filter do?" (I3) | `node <path> X` |
|
||||
|
||||
After running the subcommand(s), translate the output into plain English. For I1/I2 you **must** consult `references/bdt-function-catalog.md` (and `references/bdt-window-functions.md` if it's a `computeRelative` node) before narrating.
|
||||
@ -119,7 +119,7 @@ After running the subcommand(s), translate the output into plain English. For I1
|
||||
|---|---|
|
||||
| Exit 3: `Invalid JSON at line X` | "This file isn't valid JSON at line X. Could you re-export from the BDT viewer?" Then paste the exact error. |
|
||||
| Exit 3: `Expected top-level 'nodes' object` | "This doesn't look like a BDT export — a BDT has a top-level `nodes` object. Is this the right file?" |
|
||||
| Exit 3: `Cycle or broken reference detected` | "This BDT has a dependency cycle (or a reference to a node that doesn't exist). A valid BDT shouldn't cycle — could you re-export and share? The unresolved nodes are: \<list\>." |
|
||||
| Exit 3: `Cycle or broken reference detected` | "This BDT has a dependency cycle or references a non-existent node. You'll need to edit the BDT in the Data Cloud viewer to fix the circular dependency or broken reference — re-exporting won't help because the cycle exists in the BDT definition itself. The problematic nodes are: \<list from stderr\>." |
|
||||
| Exit 2: `No node named 'X'` | "I don't see a node named `X` in this BDT. Did you mean one of: \<list of available names the script printed on stderr\>?" |
|
||||
| Exit 2: `Field 'F' is not defined by any node` | "That field isn't defined anywhere in this BDT. Closest matches from the fields I can see: \<list from stderr\>. Is it possibly a field from the source DMO that's not pulled in?" |
|
||||
| User's BDT uses an action type not in the catalog | "This BDT uses `<action>`, which isn't in my reference material. Here's what its parameters look like: \<raw params\>. I can describe the graph structure around it, but I can't vouch for the exact semantics of this action type." |
|
||||
@ -158,7 +158,7 @@ Common issues and how to address them:
|
||||
| `ModuleNotFoundError: No module named 'bdt_analyze'` | The script path is wrong. The script lives at `scripts/bdt_analyze.py` inside the skill directory. Run it by absolute path (the Bash tool invocation should include the full path from the repo root). |
|
||||
| BDT file path contains spaces (e.g. `"default B2C (Prod).json"`) | Quote the path when calling the script: `python bdt_analyze.py summary "default B2C (Prod).json"`. |
|
||||
| Script output is very large for a big BDT (Mode C) | Use `--limit 50` on the `nodes` subcommand, or recommend Mode B (layered) instead of Mode C. |
|
||||
| Field trace returns no match despite the field clearly appearing in the BDT | The field may only appear qualified (e.g., `SalesOrder.ssot__Id__c`). Try the qualified form. Also try stripping the `__c` suffix. |
|
||||
| Field trace returns no match despite the field clearly appearing in the BDT | The field may only appear qualified (e.g., `SalesOrder.ssot__Id__c`). Try the qualified form. If still no match, the field may be passed through from a source DMO but not explicitly defined in the BDT — in that case field-trace cannot locate it; narrate "this field originates from the load node's source DMO" instead. Do NOT attempt to trace it by stripping the `__c` suffix — that would risk matching a different field with a similar name. |
|
||||
| User says output "doesn't match what I see in BDT viewer" | Verify the version in the JSON header matches the org's current release. Canonical schema was synced against `core-264`, but older/newer BDTs may use variants not yet in the reference. Flag as "version drift" and narrate best-effort. |
|
||||
|
||||
## 11. Example Interactions
|
||||
|
||||
@ -6,7 +6,7 @@
|
||||
"sources": [],
|
||||
"parameters": {
|
||||
"dataset": {"name": "WebOrders__dlo", "type": "dataLakeObject"},
|
||||
"fields": ["OrderId__c", "Amount__c", "Channel__c", "CustomerFullName__c"],
|
||||
"fields": ["OrderId__c", "Amount__c", "ChannelOrderKey__c"],
|
||||
"sampleDetails": {"type": "TopN", "sortBy": []}
|
||||
}
|
||||
},
|
||||
@ -15,7 +15,7 @@
|
||||
"sources": [],
|
||||
"parameters": {
|
||||
"dataset": {"name": "StoreOrders__dlo", "type": "dataLakeObject"},
|
||||
"fields": ["OrderId__c", "Amount__c", "Channel__c", "CustomerFullName__c"],
|
||||
"fields": ["OrderId__c", "Amount__c", "ChannelOrderKey__c"],
|
||||
"sampleDetails": {"type": "TopN", "sortBy": []}
|
||||
}
|
||||
},
|
||||
@ -25,55 +25,53 @@
|
||||
"parameters": {
|
||||
"allowImplicitDisjointSchema": false,
|
||||
"fieldMappings": [
|
||||
{"targetField": "OrderId__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "OrderId__c"}, {"node": "LOAD_STORE_ORDERS", "field": "OrderId__c"}]},
|
||||
{"targetField": "Amount__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "Amount__c"}, {"node": "LOAD_STORE_ORDERS", "field": "Amount__c"}]},
|
||||
{"targetField": "Channel__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "Channel__c"}, {"node": "LOAD_STORE_ORDERS", "field": "Channel__c"}]},
|
||||
{"targetField": "CustomerFullName__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "CustomerFullName__c"}, {"node": "LOAD_STORE_ORDERS", "field": "CustomerFullName__c"}]}
|
||||
{"targetField": "OrderId__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "OrderId__c"}, {"node": "LOAD_STORE_ORDERS", "field": "OrderId__c"}]},
|
||||
{"targetField": "Amount__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "Amount__c"}, {"node": "LOAD_STORE_ORDERS", "field": "Amount__c"}]},
|
||||
{"targetField": "ChannelOrderKey__c", "sources": [{"node": "LOAD_WEB_ORDERS", "field": "ChannelOrderKey__c"}, {"node": "LOAD_STORE_ORDERS", "field": "ChannelOrderKey__c"}]}
|
||||
]
|
||||
}
|
||||
},
|
||||
"SPLIT_CUSTOMER_NAME": {
|
||||
"SPLIT_CHANNEL_KEY": {
|
||||
"action": "split",
|
||||
"sources": ["APPEND_ALL_ORDERS"],
|
||||
"parameters": {
|
||||
"sourceField": "CustomerFullName__c",
|
||||
"delimiter": " ",
|
||||
"sourceField": "ChannelOrderKey__c",
|
||||
"delimiter": "-",
|
||||
"targetFields": [
|
||||
{"name": "CustomerFirstName__c", "label": "Customer First Name"},
|
||||
{"name": "CustomerLastName__c", "label": "Customer Last Name"}
|
||||
{"name": "Channel__c", "label": "Channel"},
|
||||
{"name": "OrderNumber__c", "label": "Order Number"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"OUTPUT_ORDERS": {
|
||||
"action": "outputD360",
|
||||
"sources": ["SPLIT_CUSTOMER_NAME"],
|
||||
"sources": ["SPLIT_CHANNEL_KEY"],
|
||||
"parameters": {
|
||||
"name": "OrdersWithCustomerName__dlm",
|
||||
"name": "OrdersWithChannel__dlm",
|
||||
"type": "dataModelObject",
|
||||
"writeMode": "OVERWRITE",
|
||||
"fieldsMappings": [
|
||||
{"sourceField": "OrderId__c", "targetField": "OrderId__c"},
|
||||
{"sourceField": "Amount__c", "targetField": "Amount__c"},
|
||||
{"sourceField": "Channel__c", "targetField": "Channel__c"},
|
||||
{"sourceField": "CustomerFirstName__c", "targetField": "FirstName__c"},
|
||||
{"sourceField": "CustomerLastName__c", "targetField": "LastName__c"}
|
||||
{"sourceField": "OrderId__c", "targetField": "OrderId__c"},
|
||||
{"sourceField": "Amount__c", "targetField": "Amount__c"},
|
||||
{"sourceField": "Channel__c", "targetField": "Channel__c"},
|
||||
{"sourceField": "OrderNumber__c", "targetField": "OrderNumber__c"}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"ui": {
|
||||
"nodes": {
|
||||
"LOAD_WEB_ORDERS": {"label": "Web Orders", "type": "LOAD_DATASET", "top": 100, "left": 100},
|
||||
"LOAD_STORE_ORDERS": {"label": "Store Orders", "type": "LOAD_DATASET", "top": 260, "left": 100},
|
||||
"APPEND_ALL_ORDERS": {"label": "Union", "type": "APPEND", "top": 180, "left": 260},
|
||||
"SPLIT_CUSTOMER_NAME": {"label": "Split full name", "type": "SPLIT", "top": 180, "left": 420},
|
||||
"OUTPUT_ORDERS": {"label": "Orders with name", "type": "OUTPUT", "top": 180, "left": 580}
|
||||
"LOAD_WEB_ORDERS": {"label": "Web Orders", "type": "LOAD_DATASET", "top": 100, "left": 100},
|
||||
"LOAD_STORE_ORDERS": {"label": "Store Orders", "type": "LOAD_DATASET", "top": 260, "left": 100},
|
||||
"APPEND_ALL_ORDERS": {"label": "Union", "type": "APPEND", "top": 180, "left": 260},
|
||||
"SPLIT_CHANNEL_KEY": {"label": "Split channel key", "type": "SPLIT", "top": 180, "left": 420},
|
||||
"OUTPUT_ORDERS": {"label": "Orders + channel", "type": "OUTPUT", "top": 180, "left": 580}
|
||||
},
|
||||
"connectors": [
|
||||
{"source": "LOAD_WEB_ORDERS", "target": "APPEND_ALL_ORDERS"},
|
||||
{"source": "LOAD_STORE_ORDERS", "target": "APPEND_ALL_ORDERS"},
|
||||
{"source": "APPEND_ALL_ORDERS", "target": "SPLIT_CUSTOMER_NAME"},
|
||||
{"source": "SPLIT_CUSTOMER_NAME", "target": "OUTPUT_ORDERS"}
|
||||
{"source": "LOAD_WEB_ORDERS", "target": "APPEND_ALL_ORDERS"},
|
||||
{"source": "LOAD_STORE_ORDERS", "target": "APPEND_ALL_ORDERS"},
|
||||
{"source": "APPEND_ALL_ORDERS", "target": "SPLIT_CHANNEL_KEY"},
|
||||
{"source": "SPLIT_CHANNEL_KEY", "target": "OUTPUT_ORDERS"}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@ -25,9 +25,10 @@
|
||||
"parameters": {
|
||||
"filterExpressions": [
|
||||
{"type": "TEXT", "field": "ssot__Status__c", "operator": "EQUAL", "operands": ["Active"]},
|
||||
{"type": "NUMBER", "field": "ssot__GrandTotalAmount__c", "operator": "GREATER_THAN", "operands": ["0"]}
|
||||
{"type": "NUMBER", "field": "ssot__GrandTotalAmount__c", "operator": "GREATER_THAN", "operands": ["0"]},
|
||||
{"type": "NUMBER", "field": "ssot__GrandTotalAmount__c", "operator": "IS_NOT_NULL", "operands": []}
|
||||
],
|
||||
"filterBooleanLogic": "1 AND 2"
|
||||
"filterBooleanLogic": "1 AND 2 AND 3"
|
||||
}
|
||||
},
|
||||
"JOIN_ORDER_ACCOUNT": {
|
||||
|
||||
@ -23,7 +23,7 @@
|
||||
"label": "Order Rank",
|
||||
"formulaExpression": "row_number()",
|
||||
"type": "NUMBER",
|
||||
"businessType": "Number",
|
||||
"businessType": "NUMBER",
|
||||
"precision": 18,
|
||||
"scale": 0,
|
||||
"defaultValue": ""
|
||||
@ -31,13 +31,33 @@
|
||||
]
|
||||
}
|
||||
},
|
||||
"FIRST_ORDER_AMOUNT": {
|
||||
"action": "formula",
|
||||
"sources": ["RANK_ORDERS"],
|
||||
"parameters": {
|
||||
"expressionType": "SQL",
|
||||
"fields": [
|
||||
{
|
||||
"name": "FirstOrderAmount__c",
|
||||
"label": "First Order Amount (per account)",
|
||||
"formulaExpression": "case when OrderRank__c = 1 then ssot__GrandTotalAmount__c else 0 end",
|
||||
"type": "NUMBER",
|
||||
"businessType": "NUMBER",
|
||||
"precision": 18,
|
||||
"scale": 2,
|
||||
"defaultValue": "0"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"AGG_BY_ACCOUNT": {
|
||||
"action": "aggregate",
|
||||
"sources": ["RANK_ORDERS"],
|
||||
"sources": ["FIRST_ORDER_AMOUNT"],
|
||||
"parameters": {
|
||||
"groupings": ["ssot__AccountId__c"],
|
||||
"aggregations": [
|
||||
{"action": "SUM", "name": "TotalAmount__c", "source": "ssot__GrandTotalAmount__c"},
|
||||
{"action": "SUM", "name": "FirstOrderTotal__c", "source": "FirstOrderAmount__c"},
|
||||
{"action": "COUNT", "name": "OrderCount__c", "source": "ssot__Id__c"}
|
||||
],
|
||||
"nodeType": "STANDARD"
|
||||
@ -53,6 +73,7 @@
|
||||
"fieldsMappings": [
|
||||
{"sourceField": "ssot__AccountId__c", "targetField": "AccountId__c"},
|
||||
{"sourceField": "TotalAmount__c", "targetField": "TotalAmount__c"},
|
||||
{"sourceField": "FirstOrderTotal__c", "targetField": "FirstOrderAmount__c"},
|
||||
{"sourceField": "OrderCount__c", "targetField": "OrderCount__c"}
|
||||
]
|
||||
}
|
||||
@ -60,14 +81,16 @@
|
||||
},
|
||||
"ui": {
|
||||
"nodes": {
|
||||
"LOAD_ORDERS": {"label": "Sales Orders", "type": "LOAD_DATASET", "top": 100, "left": 100},
|
||||
"RANK_ORDERS": {"label": "Rank by account", "type": "COMPUTE_RELATIVE", "top": 100, "left": 260},
|
||||
"AGG_BY_ACCOUNT": {"label": "Totals per account", "type": "AGGREGATE", "top": 100, "left": 420},
|
||||
"OUTPUT_SUMMARY": {"label": "Account Summary", "type": "OUTPUT", "top": 100, "left": 580}
|
||||
"LOAD_ORDERS": {"label": "Sales Orders", "type": "LOAD_DATASET", "top": 100, "left": 100},
|
||||
"RANK_ORDERS": {"label": "Rank by account", "type": "COMPUTE_RELATIVE", "top": 100, "left": 260},
|
||||
"FIRST_ORDER_AMOUNT": {"label": "Extract first order amount", "type": "FORMULA", "top": 100, "left": 420},
|
||||
"AGG_BY_ACCOUNT": {"label": "Totals per account", "type": "AGGREGATE", "top": 100, "left": 580},
|
||||
"OUTPUT_SUMMARY": {"label": "Account Summary", "type": "OUTPUT", "top": 100, "left": 740}
|
||||
},
|
||||
"connectors": [
|
||||
{"source": "LOAD_ORDERS", "target": "RANK_ORDERS"},
|
||||
{"source": "RANK_ORDERS", "target": "AGG_BY_ACCOUNT"},
|
||||
{"source": "RANK_ORDERS", "target": "FIRST_ORDER_AMOUNT"},
|
||||
{"source": "FIRST_ORDER_AMOUNT", "target": "AGG_BY_ACCOUNT"},
|
||||
{"source": "AGG_BY_ACCOUNT", "target": "OUTPUT_SUMMARY"}
|
||||
]
|
||||
}
|
||||
|
||||
@ -34,7 +34,7 @@
|
||||
|---|---|---|
|
||||
| `abs(n)` | number → number | Absolute value (strips sign). |
|
||||
| `ceiling(n)` | number → number | Round up, away from zero for negatives. |
|
||||
| `floor(n)` | number → number | Round toward negative infinity (down on the number line). For negatives, rounds away from zero (e.g., `floor(-2.3) = -3`). |
|
||||
| `floor(n)` | number → number | Round toward negative infinity. Always rounds down on the number line: `floor(2.7) = 2`, `floor(-2.3) = -3`. |
|
||||
| `exp(n)` | number → number | e raised to n. |
|
||||
| `log(base, n)` | number → number | Logarithm of n in the given base. |
|
||||
| `max(a, b, …)` | number → number | Largest value. |
|
||||
|
||||
@ -211,8 +211,9 @@ semantics — for that, see `computeRelative`.
|
||||
- `label` — user-visible label.
|
||||
- `formulaExpression` — SFSQL (or DCSQL) expression. See `bdt-function-catalog.md`.
|
||||
- `type` — enum `DataType`: `TEXT`, `NUMBER`, `BOOLEAN`, `DATE_ONLY`, `DATETIME`.
|
||||
- `businessType` — a user-facing business-semantic type name. Enum `BusinessTypeEnum` —
|
||||
canonical values:
|
||||
- **`businessType`** — a user-facing business-semantic type name. Enum `BusinessTypeEnum` —
|
||||
**canonical values** (complete list; matches `BusinessTypeEnum.java` in core-262 as of the
|
||||
capture date at the top of this file):
|
||||
- `"TEXT"`, `"NUMBER"`, `"BOOLEAN"`
|
||||
- `"EMAIL"`, `"PHONE"`, `"URL"` — text-valued with semantic meaning
|
||||
- `"PERCENT"`, `"CURRENCY"` — number-valued with semantic meaning
|
||||
@ -222,8 +223,9 @@ semantics — for that, see `computeRelative`.
|
||||
|
||||
Note: `businessType` values map to underlying `type` (`DataType`) values. E.g.,
|
||||
`businessType: "PERCENT"` is stored as `type: "NUMBER"`; `businessType: "DATE"` is stored
|
||||
as `type: "DATETIME"`; `businessType: "EMAIL"` is stored as `type: "TEXT"`. The canonical
|
||||
mapping lives in `BusinessTypeEnum.java`.
|
||||
as `type: "DATETIME"`; `businessType: "EMAIL"` is stored as `type: "TEXT"`. If the skill
|
||||
ever encounters a `businessType` value outside this list, that is a sign the upstream BDT
|
||||
schema has evolved — surface the raw value in narration and flag it as undocumented.
|
||||
- `precision` — integer precision (default 10 for numbers; characters for text).
|
||||
- `scale` — decimal places; only for NUMBER.
|
||||
- `defaultValue` — value when the expression yields NULL.
|
||||
|
||||
@ -128,7 +128,7 @@ class DataTransform:
|
||||
"""Pick which definition the accessor properties (and the rest of the
|
||||
API surface) operate on. Raises `BdtNotFoundError` if `index` is out
|
||||
of range."""
|
||||
if not isinstance(index, int) or index < 0 or index >= len(self.definitions):
|
||||
if type(index) is not int or index < 0 or index >= len(self.definitions):
|
||||
raise BdtNotFoundError(
|
||||
f"Definition index {index} out of range; payload has "
|
||||
f"{len(self.definitions)} definitions "
|
||||
@ -296,31 +296,32 @@ class DataTransform:
|
||||
|
||||
def topo_order(self) -> List[str]:
|
||||
"""Kahn's algorithm. Raises BdtInputError on cycles."""
|
||||
# Adjacency: for each node, its forward edges (consumers) and in-degree
|
||||
from collections import deque
|
||||
in_degree: Dict[str, int] = {name: 0 for name in self.nodes}
|
||||
forward: Dict[str, List[str]] = {name: [] for name in self.nodes}
|
||||
for name, n in self.nodes.items():
|
||||
for s in n.sources:
|
||||
if s not in self.nodes:
|
||||
# Broken reference — counted as in-edge so the dependent node stays blocked
|
||||
# (we surface broken refs via a separate method; topo should not crash here).
|
||||
# Broken reference — count as in-edge so the dependent node stays blocked
|
||||
in_degree[name] += 1
|
||||
continue
|
||||
forward[s].append(name)
|
||||
in_degree[name] += 1
|
||||
|
||||
# Start with all zero-in-degree nodes in deterministic order (sorted)
|
||||
ready = sorted([name for name, d in in_degree.items() if d == 0])
|
||||
# Sort zero-in-degree nodes once for deterministic starting order
|
||||
ready = deque(sorted([name for name, d in in_degree.items() if d == 0]))
|
||||
order: List[str] = []
|
||||
while ready:
|
||||
# Pop deterministically so output is stable across runs
|
||||
ready.sort()
|
||||
current = ready.pop(0)
|
||||
current = ready.popleft()
|
||||
order.append(current)
|
||||
for consumer in sorted(forward[current]):
|
||||
# Collect newly-ready nodes, sort them once, append
|
||||
newly_ready = []
|
||||
for consumer in forward[current]:
|
||||
in_degree[consumer] -= 1
|
||||
if in_degree[consumer] == 0:
|
||||
ready.append(consumer)
|
||||
newly_ready.append(consumer)
|
||||
for node in sorted(newly_ready):
|
||||
ready.append(node)
|
||||
|
||||
if len(order) != len(self.nodes):
|
||||
stuck = [n for n in self.nodes if n not in order]
|
||||
@ -1309,14 +1310,15 @@ def cmd_formula(bdt: "DataTransform", args) -> str:
|
||||
# For each consumed field, find the node that defines it (if any) — give
|
||||
# the LLM ready context for I2-style reasoning. Hoist `bdt.upstream(name)`
|
||||
# out of the loop — it's a pure function of `name` and walks the graph.
|
||||
# Pre-compute fields_produced for every upstream node once (was O(consumed × upstream))
|
||||
upstream_names = bdt.upstream(name)
|
||||
upstream_fields_map = {up_name: fields_produced(bdt.nodes[up_name]) for up_name in upstream_names}
|
||||
upstream_defs = {}
|
||||
for f in consumed:
|
||||
# Prefer exact match; fall back to unqualified
|
||||
matches = []
|
||||
for up_name in upstream_names:
|
||||
up = bdt.nodes[up_name]
|
||||
prod = fields_produced(up)
|
||||
prod = upstream_fields_map[up_name]
|
||||
if f in prod or f.rsplit(".", 1)[-1] in prod:
|
||||
matches.append({
|
||||
"node": up_name,
|
||||
@ -1324,7 +1326,7 @@ def cmd_formula(bdt: "DataTransform", args) -> str:
|
||||
"label": up.ui_label,
|
||||
})
|
||||
if matches:
|
||||
upstream_defs[f] = matches[-1] # closest definer (last in topo order)
|
||||
upstream_defs[f] = matches[-1]
|
||||
|
||||
payload = {
|
||||
"node": name,
|
||||
|
||||
@ -38,8 +38,8 @@ class TestBadInput(unittest.TestCase):
|
||||
|
||||
def test_invalid_json_raises(self):
|
||||
p = FIXTURES / "_tmp_invalid.json"
|
||||
p.write_text("{not valid json")
|
||||
self.addCleanup(p.unlink, missing_ok=True)
|
||||
p.write_text("{not valid json")
|
||||
with self.assertRaises(bdt_analyze.BdtInputError) as cm:
|
||||
bdt_analyze.DataTransform.from_path(p)
|
||||
self.assertIn("Invalid JSON", str(cm.exception))
|
||||
@ -711,15 +711,16 @@ class TestInputShapeDetection(unittest.TestCase):
|
||||
for subcmd, extra_arg in [("summary", None), ("stages", None), ("nodes", None),
|
||||
("sources", None), ("outputs", None),
|
||||
("lineage", "OUTPUT_X"), ("node", "LOAD_X")]:
|
||||
argv = [subcmd, str(FIXTURES / "api_input_single.json")]
|
||||
if extra_arg: argv.append(extra_arg)
|
||||
out = io.StringIO(); err = io.StringIO()
|
||||
try:
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = bdt_analyze.main(argv)
|
||||
except SystemExit as e:
|
||||
code = e.code
|
||||
self.assertEqual(code, 0, f"Subcommand {subcmd} failed: {err.getvalue()}")
|
||||
with self.subTest(subcmd=subcmd):
|
||||
argv = [subcmd, str(FIXTURES / "api_input_single.json")]
|
||||
if extra_arg: argv.append(extra_arg)
|
||||
out = io.StringIO(); err = io.StringIO()
|
||||
try:
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = bdt_analyze.main(argv)
|
||||
except SystemExit as e:
|
||||
code = e.code
|
||||
self.assertEqual(code, 0, f"Subcommand {subcmd} failed: {err.getvalue()}")
|
||||
|
||||
|
||||
class TestMultiDefinition(unittest.TestCase):
|
||||
|
||||
Loading…
Reference in New Issue
Block a user