afv-library/skills/explaining-batch-data-transform/references/bdt-function-catalog.md
Gaurav Bajpai db1aca7221
fix: address PRizm review findings from internal port PR @W-22196528@
Addresses 4 critical findings surfaced during PRizm code review of the
internal plugin port of this skill (internal PR #19). Applying the same
fixes here keeps the external canonical source and the internal port in
sync.

1. Test cleanup discipline: switch `test_invalid_json_raises` from
   `try/finally` to `self.addCleanup(p.unlink, missing_ok=True)` — the
   unittest-idiomatic way to guarantee temp-file cleanup regardless of
   how the test exits.

2. `floor()` description in bdt-function-catalog.md: the old row was
   self-contradictory ("toward zero" AND "toward next integer up" in
   the same cell). Replace with a single coherent definition: rounds
   toward negative infinity; for negatives rounds away from zero
   (e.g., `floor(-2.3) = -3`).

3. Split-node documentation in bdt-node-catalog.md: the old doc claimed
   `split` routes rows into downstream branches via `branches[]` with
   per-branch predicates. That is not the canonical schema. Per
   `SplitParametersInputRepresentation` in core-262-public, `split` is
   a string-splitting operation: one `sourceField` + `delimiter` →
   N `targetFields` (one row in, one row out; columns added). Rewrote
   the section with the correct parameters, lineage effect, gotchas,
   and a canonical example. Row-routing belongs in `filter` nodes.

4. Sample `assets/sample_bdts/append_and_split.json`: the old sample
   used the invented `branches[]` shape AND routed the same split into
   two downstream outputs that each expected different rows — which is
   not how `split` works. Rewrote the sample so:
     - `appendV2` unions two order sources (unchanged intent).
     - `split` uses canonical `{sourceField, delimiter, targetFields}`
       splitting `CustomerFullName__c` into first + last name columns.
     - One downstream output consumes the new columns (removes the
       fake two-branch fan-out).

Tests: 92/92 passing. Sample parses and runs through `bdt_analyze.py
summary` cleanly (5 nodes: 2 load + 1 appendV2 + 1 split + 1 outputD360).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 10:19:36 +05:30

125 lines
5.7 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# SFSQL Function Catalog — for `formula` reasoning
> **Last synced:** 2026-04-23 from the Data Processing Engine PDF (2025-06-24 version) and
> the BDT help DITA XMLs (`c360_a_batch_transform_numeric.xml`, `_string.xml`,
> `_date_functions.xml`, `_boolean_functions.xml`, `_multivalue_functions.xml`,
> `_additionalfunctions.xml`). Consult this file before interpreting any `formulaExpression`.
## How formulas appear in BDT JSON
```jsonc
"action": "formula",
"parameters": {
"expressionType": "SQL", // or "DCSQL"
"fields": [
{
"name": "OutputField__c",
"label": "Output Label",
"type": "TEXT", // TEXT | NUMBER | BOOLEAN | DATE_ONLY | DATETIME
"businessType": "Text", // user-visible type name
"formulaExpression": "upper(coalesce(SourceField__c, ''))",
"precision": 255,
"scale": 0, // only for NUMBER
"defaultValue": ""
}
]
}
```
## Functions by family
### Math / Numeric
| Function | Signature | Purpose |
|---|---|---|
| `abs(n)` | number → number | Absolute value (strips sign). |
| `ceiling(n)` | number → number | Round up, away from zero for negatives. |
| `floor(n)` | number → number | Round toward negative infinity (down on the number line). For negatives, rounds away from zero (e.g., `floor(-2.3) = -3`). |
| `exp(n)` | number → number | e raised to n. |
| `log(base, n)` | number → number | Logarithm of n in the given base. |
| `max(a, b, …)` | number → number | Largest value. |
| `min(a, b, …)` | number → number | Smallest value. |
| `mod(a, b)` | number → number | Remainder after dividing a by b. |
| `power(a, b)` | number → number | a raised to the power b. |
| `round(n, digits)` | number → number | Round to `digits` places. |
| `sqrt(n)` | number → number | Positive square root. |
| `trunc(n, digits)` | number → number | Truncate to `digits` places (doesn't round). |
### String
| Function | Signature | Purpose |
|---|---|---|
| `begins(s, prefix)` | text → boolean | True if s starts with prefix. |
| `concat(a, b, …)` | text → text | Concatenate. |
| `contains(haystack, needle)` | text → boolean | True if haystack contains needle. |
| `ends(s, suffix)` | text → boolean | True if s ends with suffix. |
| `length(s)` | text → number | Character count. |
| `lower(s)` | text → text | Lowercase (locale-aware if a locale is provided). |
| `ltrim(s)` / `ltrim(s, substring)` | text → text | Remove leading whitespace (or a specific substring). |
| `rtrim(s)` / `rtrim(s, substring)` | text → text | Remove trailing whitespace (or substring). |
| `substitute(s, old, new)` | text → text | Replace old with new in s. |
| `substr(s, start, length)` | text → text | Extract a substring. |
| `text(n)` | number → text | Convert a number to its text form. |
| `trim(s)` / `trim(s, substring)` | text → text | Remove leading + trailing whitespace or substring. |
| `upper(s)` | text → text | Uppercase. |
| `uuid()` | → text | Newly generated unique ID. |
| `value(s)` | text → number | Parse a text representation of a number into a numeric value. |
### Date
| Function | Signature | Purpose |
|---|---|---|
| `adddays(date, n)` | date → date | Add n days. |
| `addmonths(date, n)` | date → date | Add n months. |
| `datediff(start, end)` | date × date → number | Days between two dates. |
| `datetimevalue(text_or_date)` | → datetime | GMT/UTC date+time value. |
| `datevalue(text_or_datetime)` | → date | Extract the date part. |
| `day(date)` | date → number | 1-31. |
| `monthdiff(start, end)` | date × date → number | Months between two dates. |
| `now()` | → datetime | Current moment (UTC). |
| `today()` | → date | Current date. |
| `weekday(date)` | date → number | 1=Sunday, 2=Monday, … 7=Saturday. |
### Boolean / Logical
| Function | Signature | Purpose |
|---|---|---|
| `and(a, b, …)` | bool → bool | True when all are true. |
| `or(a, b, …)` | bool → bool | True when any is true. |
| `if(cond, then_val, else_val)` | → any | Ternary. |
| `case when <cond1> then <val1> else <elseval> end` | — | Multi-branch (SQL CASE). |
| `blankvalue(expr, substitute)` | → any | Substitute when expr is blank. |
| `isblank(expr)` | → bool | True when expr is blank. |
| `isnull(expr)` | → bool | True when expr is null. |
| `nullvalue(expr, substitute)` | → any | Substitute when expr is null; returns expr otherwise. |
### Multivalue
| Function | Signature | Purpose |
|---|---|---|
| `sequence(start, end, step?)` | → array | Array of numbers/dates between start and end (step defaults to 1 / 1 day). |
| `explode(array)` | → row per element | Convert multivalue data into one row per element. **Cannot be nested inside another function.** |
### Additional / Null-handling (commonly seen)
| Function | Signature | Purpose |
|---|---|---|
| `coalesce(a, b, …)` | → any | First non-null argument (SQL standard). |
## Patterns to recognize in narration
- **`coalesce(X, Y)`** — "use X when present, otherwise Y." Very common for default-value
handling.
- **`case when RANK = 1 then X else 0 end`** — the "first-occurrence extract" idiom paired
with a `computeRelative` `row_number()` node.
- **`case when FLAG = 'Y' then 'Y' else 'N' end`** — boolean-like text normalization.
- **`concat(coalesce(A, ''), coalesce(B, ''))`** — safe string concatenation that defaults
NULLs to empty strings (producing a synthetic composite key).
## Sources
- Data Processing Engine reference PDF (captured at `research/bdt-doc-10-data_processing_engine_6-24-2025.pdf.md`)
- BDT help XML: `c360_a_batch_transform_numeric.xml`, `_string.xml`, `_date_functions.xml`,
`_boolean_functions.xml`, `_multivalue_functions.xml`, `_additionalfunctions.xml` (captured
at `research/help-xml/`).