kris/afv-library
1
0
Fork 0
mirror of https://github.com/forcedotcom/afv-library.git synced 2026-08-09 17:22:00 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
177 Commits 9 Branches 53 Tags 31 MiB
fc4d14bce0
Commit Graph

4 Commits

Author SHA1 Message Date
Gaurav Bajpai
fc4d14bce0
test(bdt): unit tests + adversarial fixtures @W-22196528@ 
Adds the full test suite for bdt_analyze.py: 92 unit tests backed by
9 JSON fixtures (4 normative + 5 adversarial). All tests stdlib-only,
runnable via `python3 -m unittest tests.test_bdt_analyze`.

tests/test_bdt_analyze.py coverage:

DAG primitives:
- roots() / sinks() / topo_order() on minimal + branching graphs.
- upstream() / downstream() reachability correctness.
- Cycle tolerance: topo_order + traversals do not infinite-loop or
  raise when the graph contains a cycle; cycle is reported via
  broken_references-adjacent signals.
- Broken-reference tolerance: nodes referencing non-existent upstreams
  are handled gracefully; broken_references() enumerates them.

CLI subcommands (happy-path + error-path for each):
- summary, sources, outputs, stages, nodes, node, lineage,
  field-trace, formula, definitions — each exercised in both
  human-readable and --json modes.
- Exit-code discipline: unknown node -> exit 2, malformed input ->
  exit 3, success -> exit 0.
- Unknown-action graceful degradation: nodes whose action isn't in
  the catalog still appear in summary/nodes output with a generic
  label rather than crashing.
- Output size budgets: default-mode outputs are asserted to stay
  within configured character caps.

Field discovery:
- fields_produced() / fields_consumed() / _scrape_field_refs()
  heuristics tested against both clean and noisy formula bodies.
- TestFieldTraceFocus — narrowing tests preventing false-positive
  substring hits in formula text.

Dual-shape input:
- TestInputShapeDetection — editor export vs. Connect API
  single-definition auto-detection.
- TestMultiDefinition — Connect API multi-definition payload:
  definitions subcommand, --definition N routing, out-of-range
  index error handling.

Fixtures (tests/fixtures/):
- minimal.json               — smallest valid editor-export BDT.
- window_and_aggregate.json  — window + aggregate composition.
- api_input_single.json      — Connect API single-definition payload.
- api_input_multi.json       — Connect API multi-definition payload.

Adversarial fixtures (tests/fixtures/adversarial/):
- cycle.json           — graph with a cycle; parser must not hang.
- broken_ref.json      — node referencing a non-existent upstream.
- unknown_action.json  — node with an action not in the catalog.
- no_ui.json           — BDT missing the UI layer entirely.
- empty_nodes.json     — valid envelope but zero nodes.

Suite status: Ran 92 tests, OK.

@W-22196528@
 2026-04-23 23:29:05 +05:30
Gaurav Bajpai
9c47f28221
feat(bdt): reference docs + sample BDTs @W-22196528@ 
Adds the curated reference library the skill loads on demand, plus
four synthetic sample BDTs used by docs, tests, and LLM-mode demos.

references/ (4 curated Markdown files):
- bdt-reference.md        — top-level BDT JSON anatomy: envelope,
                            nodes, edges, UI layer, definitions,
                            businessType semantics. Cites the core-262
                            upstream JSON schema and Connect API spec.
- bdt-node-catalog.md     — every node type (DMO Source, DMO Sink,
                            Filter, Join, Union, Aggregate, Window,
                            Formula, Split, Append, etc.) with its
                            required/optional fields and typical
                            usage. Audited against core-262 enums.
- bdt-function-catalog.md — the expression-language function surface
                            (string, numeric, date, conditional,
                            aggregate). Grouped by category with
                            signature + one-line semantics.
- bdt-window-functions.md — windowing operators (ROW_NUMBER, RANK,
                            LEAD/LAG, running aggregates) with PARTITION
                            BY / ORDER BY grammar and gotchas.

assets/sample_bdts/ (4 synthetic, dependency-free BDTs):
- minimal_dmo_to_dmo.json   — smallest valid BDT (1 source, 1 sink).
- joins_and_filters.json    — join + filter composition.
- window_and_aggregate.json — window function + aggregate in one graph.
- append_and_split.json     — append-then-split branching topology.

Grounding rules enforced in this commit:
- Every claim in references/ cites an upstream source (core-262 JSON
  schema, Connect API reference, or the Data Cloud BDT editor spec).
  No speculative content.
- No raw DITA or internal-only documentation is shipped; references
  are synthesized from public-facing material.
- BusinessTypeEnum values use the canonical camelCase casing from
  core-262 (case-cleanup fix included here).
- Sample BDTs are original synthetic fixtures, not redacted customer
  data. Each is small enough to read end-to-end.

@W-22196528@
 2026-04-23 23:29:05 +05:30
Gaurav Bajpai
9f1603ed2f
feat(bdt): Python parser core + CLI subcommands @W-22196528@ 
Adds scripts/bdt_analyze.py — a generic, stdlib-only DAG parser and
query CLI for Salesforce Data Cloud BDT JSON. 1,379 lines.

Parser primitives:
- DataTransform.from_path / from_dict — accepts three input shapes:
  editor export ({version, nodes, ui, ...}), Connect API
  single-definition ({name, label, type, definition: {...}}), and
  Connect API multi-definition ({name, definitions: [{name, label,
  type, definition}, ...]}).
- Node dataclass with ui_label / ui_description fallback resolution.
- roots() / sinks() / topo_order() — Kahn's algorithm with
  deterministic tie-break for reproducible output.
- upstream() / downstream() traversal resilient to broken refs and
  cycles (does not infinite-loop on self-edges or cycles).
- broken_references(), fields_produced(), fields_consumed(),
  _scrape_field_refs() heuristics for field-trace discovery.

CLI (argparse, 10 subcommands, each supports --json for machine
output):
- summary      — node counts, source/sink counts, stage totals.
- sources      — list source nodes (no upstream).
- outputs      — list sink nodes (no downstream).
- stages       — topologically ordered stages.
- nodes        — flat node listing with labels.
- node <name>  — per-node detail (action, inputs, outputs, fields,
                 UI label/description).
- lineage <node>       — upstream + downstream chain from a node.
- field-trace <field>  — which nodes produce/consume a given field.
- formula <node>       — extract formulas/expressions from a node.
- definitions          — lists definitions in multi-definition
                         payloads; every other subcommand accepts
                         --definition N (default 0) to route into a
                         specific definition within the payload.

Error contract:
- Exit 0 on success, 2 on unknown node/field, 3 on malformed input.
- BdtInputError (exit 3) and BdtNotFoundError (exit 2) classes
  centralize error handling so the CLI shell stays thin.
- Field-trace narrowing refinements prevent false positives from
  substring matches in formula bodies.
- Upstream/downstream walkers harden against broken refs discovered
  during internal-BDT audit.

Design invariants:
- Python owns truth (parsing, DAG math, field discovery). LLM owns
  narrative (explaining what the structure means to a user).
- No external dependencies — stdlib only: argparse, json, pathlib,
  re, sys, collections, dataclasses, typing.
- Output size budgets: every subcommand caps its default-mode output
  so summaries fit in a single LLM context window; --json dumps
  everything for agents that need raw data.

@W-22196528@
 2026-04-23 23:29:05 +05:30
Gaurav Bajpai
9be9667779
feat(bdt): scaffold skill directory + SKILL.md body @W-22196528@ 
Scaffolds the explaining-batch-data-transform skill directory and delivers
the full, production-grade SKILL.md body. 194 lines.

Directory layout established (skills/explaining-batch-data-transform/):
- SKILL.md                 — the entry point this commit adds
- scripts/                 — parser lands in commit 2
- references/, assets/     — reference docs + samples land in commit 3
- tests/                   — unit tests + fixtures land in commit 4

SKILL.md frontmatter:
- name, description, tier, and owner fields matching the library's CI
  validator schema.
- Description covers the full production surface: "Explain, audit, and
  trace Salesforce Data Cloud Batch Data Transform (BDT) JSON — sources,
  sinks, stages, lineage, field traces, formulas, multi-definition
  payloads, and dual input shapes (editor export + Connect API)."

SKILL.md body (12 sections):
1. When to use this skill
2. Inputs you accept (editor export, Connect API single-definition,
   Connect API multi-definition payloads)
3. Mode A — structural survey (summary / sources / outputs / stages)
4. Mode B — per-node explanation (nodes / node)
5. Mode C — lineage + field trace
6. Mode D — formulas and windows
7. Question routing table (user phrasing -> CLI subcommand)
8. Multi-definition payload handling (--definition flag semantics)
9. Troubleshooting (malformed JSON, unknown node refs, broken upstream)
10. Non-goals as explicit "Do not" directives (no execution, no schema
    inference beyond what the JSON states, no speculation about runtime)
11. References + sample BDTs index
12. Exit-code + error-class contract surfaced from bdt_analyze.py

Structure passes the repo's skill-validator (npm run validate:skills):
single top-level H1, required sections present, frontmatter schema
aligned, no broken internal links.

@W-22196528@
 2026-04-23 23:29:04 +05:30