fix: document publish-only type validation and empty traces on --api-name preview

Three findings from wiring an Agent Script service agent to an apex://
invocable (API v67.0, sf CLI 2.145.6):

1. `sf agent validate authoring-bundle` returns success:true for an action
   param with the wrong numeric type. Only the server-side check during the
   publish step catches it, so a green validate is not evidence the action
   I/O types are correct.

2. Apex Integer and Apex Decimal do not behave the same. An Integer param
   as bare `number` is rejected at publish; a Decimal output as bare
   `number` publishes and runs correctly. The existing "bare number fails
   at publish in action inputs/outputs" note reads as universal, which
   sends authors hunting for a complex type a Decimal does not need.

3. Preview started with --api-name writes trace files containing only {}.
   Only the --authoring-bundle path produces readable traces. This is not
   academic: a service agent cannot always be previewed via
   --authoring-bundle, leaving --api-name as the only option and making the
   "read the trace after every utterance" rule unsatisfiable. Documented a
   discriminating-value fallback for that case.

All additive; no existing guidance removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
james-enperso 2026-08-07 13:55:57 +10:00
parent 4bc39ed386
commit 4848ecc844
2 changed files with 66 additions and 0 deletions

View File

@ -421,6 +421,29 @@ For each turn, verify in the trace:
4. **Action outputs used correctly** — Compare `FunctionStep` outputs to the agent's final response. The agent should be using real data from the action, not inventing values.
5. **Persisted state (live preview)** — For actions that write data (create records, update fields), query the backing SObject directly to confirm the write happened. This is ground truth. If no record exists, the action was not invoked regardless of what the trace or chat says — investigate the router first.
> **Traces are only written on the authoring-bundle preview path.** A preview started with
> `--api-name` (the published/activated agent) writes trace files containing exactly `{}` — no
> `plan` array, no `FunctionStep`. Verified on API v67.0 with `sf` CLI 2.145.6. Only
> `sf agent preview start --authoring-bundle <Name>` produces readable traces.
>
> This matters because the two are not always interchangeable: a **service** agent cannot always be
> previewed via `--authoring-bundle` (that path can be unreachable on restricted networks), which
> leaves `--api-name` as the only option — and it yields no trace. In that situation the
> "read the trace after every utterance" rule above cannot be satisfied, so substitute
> **discriminating-value testing**:
>
> - Query ground truth first, then ask the agent about records whose values an LLM could not guess.
> A response matching an arbitrary figure like `$48,077.20` exactly is strong evidence the action
> ran.
> - Better still, target a record that exercises a **distinctive code path** in the action
> implementation. Example: an invocable that falls back to `Date.today().year()` when a record has
> no transactions — if the agent reports that fallback year for such a record, the Apex
> demonstrably executed.
> - Cross-check writes by querying the SObject directly, per item 5 above.
>
> Record which method you used. "Verified by trace" and "verified by discriminating value" are
> different strengths of evidence and should not be reported interchangeably.
### Behavioral Evaluation
Evaluate the agent's behavior like a human would — does this feel like a natural, competent conversation?
@ -453,6 +476,23 @@ Replace `<AGENT_NAME>` with your authoring bundle name (e.g., `Local_Info_Agent`
Traces are available immediately after each `send` — you do NOT need to end the session to read them.
The **published-agent** preview path (`--api-name`) uses a different layout, keyed by **Bot ID**
rather than agent name, and its `traces/*.json` files are empty (`{}`):
```text
.sfdx/agents/<BOT_ID>/sessions/<SESSION_ID>/
├── metadata.json
├── session-meta.json
├── transcript.jsonl # populated — the conversation is still readable
├── turn-index.json
└── traces/
└── <PLAN_ID>.json # written, but contains only {}
```
`transcript.jsonl` is still usable there, so you can review the conversation — you just cannot
confirm `FunctionStep` execution. Use the discriminating-value technique described under
[Trace Evaluation](#trace-evaluation) instead.
### File Structure
**metadata.json** contains session-level information: `sessionId`, `agentId`, `startTime`, and `mockMode` (either `"Mock"` for simulated or `"Live Test"` for live).

View File

@ -15,6 +15,32 @@
> **Key insight:** Bare `number` works in **variable declarations** but **fails at publish** in action inputs/outputs. This is the #1 cause of publish-fix-republish cycles.
> **`sf agent validate` does NOT catch this.** `sf agent validate authoring-bundle` returns
> `success: true` for an action param with the wrong numeric type — the check only runs server-side
> during `sf agent publish`. Do not treat a green validate as evidence your action I/O types are
> correct. When publish rejects the type it returns a **400** whose message names the exact fix,
> e.g.:
>
> ```text
> Validation failed for action 'calculate_interest_paid' due to invalid data type for the input
> parameter 'fiscalYear'. To fix, update the data type to 'object' type and 'complex_data_type_name'
> to 'lightning__integerType' for the input parameter 'fiscalYear'.
> ```
>
> Read that message rather than guessing — it is authoritative for the target you are wiring.
> **Apex `Integer` and Apex `Decimal` behave differently.** Verified against API v67.0 with
> `sf` CLI 2.145.6:
> - Apex **`Integer`** param (input *or* output) as bare `number` → publish **fails** with the 400
> above. Must be `object` + `lightning__integerType`.
> - Apex **`Decimal`** output as bare `number` → publish **succeeds**. Two separate agents wired to
> `apex://` invocables returning `Decimal` published with bare `number` outputs and returned
> correct values at runtime.
>
> So for `apex://` targets, reach for the complex type when the Apex field is an `Integer`. If you
> hit a publish 400 on a `Decimal`, follow the message — but a bare `number` is not automatically
> wrong there.
> **CRITICAL: Target type matters!** The valid `complex_data_type_name` for integer values differs by target type:
> - **Flow targets** (`flow://`): Use `lightning__numberType` (NOT `lightning__integerType`)
> - **Apex targets** (`apex://`): Use `lightning__integerType` (NOT `lightning__numberType`)