kris/afv-library
1
0
Fork 0
mirror of https://github.com/forcedotcom/afv-library.git synced 2026-08-09 00:42:46 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat: add generating-eval-seed-data skill

Browse Source 
New skill that generates minimal seed-data stubs (custom fields, objects,
Apex class stubs) for evaluation datasets in the afv-library. Adapted from
the adk-eval-seed-data-generator with afv-library conventions and all
PR #194 review feedback addressed (scoped allowed-tools, no hardcoded
paths, progressive disclosure, skills referenced by name).
This commit is contained in:
ysachdeva@salesforce.com 2026-05-04 16:52:16 +05:30
parent f5e63fd06a
commit 708e54e6ce
12 changed files with 500 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

163
skills/generating-eval-seed-data/SKILL.md Normal file
View File

@ -0,0 +1,163 @@
---
name: generating-eval-seed-data
description: "Generate minimal seed-data stubs for Salesforce metadata evaluation datasets in the afv-library. Use this skill to create seed-data directories containing lightweight dependency declarations (custom fields, objects, Apex class stubs) that a dataset's gold file references. TRIGGER when: user says generate seed data, create seed-data stubs, populate seed-data, dataset dependencies, gold file dependencies, add supporting metadata for eval, or wants to set up prerequisite objects/fields for a test dataset. Also use when the user mentions seed-data, eval fixtures, stub generation, or asks to fill in the seed-data directory for any skill's tests/evals/ dataset. SKIP when: user wants to generate the gold file itself (use the domain-specific generating skill), wants to run evals (use eval runner tooling), or wants to create a new skill from scratch (use creating-sf-skill)."
license: LICENSE.txt has complete terms
metadata:
version: "1.0"
stage: Draft
related-skills: "creating-sf-skill, generating-custom-field, generating-custom-object, generating-apex"
allowed-tools: Bash(sf project generate, sf project deploy start, cp, rm, mkdir, find, ls) Read Write
---
# Generating Eval Seed Data
Generate seed-data **stubs** — minimal supporting metadata dependencies — for evaluation datasets in the afv-library. Stubs declare the bare-minimum custom objects, fields, relationships, and Apex classes that a dataset's gold file references, just enough so the gold file can be validated in isolation.
## Scope
- **In scope**: Analyzing gold files to identify custom dependencies, generating minimal stub XML/Apex for those dependencies, validating stubs deploy successfully via dry-run, and populating the `seed-data/` directory.
- **Out of scope**: Generating the gold file itself (delegate to the domain-specific skill), creating new eval datasets or prompt.md files (delegate to `creating-sf-skill`), deploying metadata to production orgs.
---
## Required Inputs
Gather before proceeding:
- **Dataset path**: Path to a single dataset (`skills/<name>/tests/evals/<dataset>/`) or a domain path containing multiple datasets. Always ask if not provided.
- **Target org alias**: The Salesforce org alias for dry-run validation (e.g., `myDevOrg`). Ask if not provided.
Defaults unless specified:
- API version: `62.0`
- Stub style: absolute minimum elements per metadata type (see `references/stub-rules.md`)
---
## Workflow
All steps are sequential. Do not skip or reorder.
### Phase 1 — Identify and Read
1. **Identify the dataset(s)**
- If the path contains `tests/evals/<datasetName>` (or has `prompt.md` / `gold/` directly inside), treat as a single dataset.
- Otherwise, look for `tests/evals/` subdirectory. If it exists, list all subdirectories — each is a dataset. Process them all.
- If neither pattern matches, ask the user to clarify.
2. **Read the gold file(s)**
- Look for gold files in `{dataset_path}/gold/`. These are Salesforce metadata XML or Apex files.
- If gold files exist, proceed to step 3.
- If gold files do NOT exist, ask: "This dataset has no gold file. Would you like me to generate one from `prompt.md`?" If yes, read `prompt.md` and generate a plausible gold file, then proceed.
3. **Read stub generation rules** — load `references/stub-rules.md` before analyzing.
### Phase 2 — Analyze and Generate
4. **Analyze dependencies**
- Read all gold files and identify every custom dependency. Look for:
- **Custom fields** (`__c`): referenced in formulas, conditions, assignments, or relationship traversals (`__r.Name` implies a lookup `__c`)
- **Custom objects** (`__c`): any custom object the gold metadata lives on or references via lookups
- **Apex classes**: parent classes, interfaces, or utility classes referenced by gold code
- For each dependency, determine: metadata type, correct API name, minimum required attributes.
- Standard Salesforce objects (Account, Contact, Case, etc.) and their standard fields do NOT need stubs.
5. **Generate stubs**
- Create the `seed-data/` directory structure following the rules in `references/stub-rules.md`.
- Include ONLY the minimum elements per metadata type — no optional attributes.
- For picklists: only include values explicitly referenced in the gold file.
6. **Compare against example** — verify output matches patterns in `examples/stub-examples.md`.
### Phase 3 — Validate
7. **Validate with dry-run deployment**
- Create a temporary SFDX project:
```bash
cd /tmp && sf project generate --name seed-data-validation-$(date +%s) --template empty
```
- Read the temp project's `sfdx-project.json` to resolve the deploy path — do not hardcode `force-app/main/default/`. Extract `packageDirectories[].path` (use the entry with `"default": true`; if none, use the first entry).
- Copy seed-data and gold files into the resolved deploy path:
```bash
cp -r {dataset_path}/seed-data/* {temp_project}/{resolved_path}/
cp -r {dataset_path}/gold/* {temp_project}/{resolved_path}/
```
- Run dry-run:
```bash
sf project deploy start --dry-run -d "{resolved_path}" --target-org {target_org} --test-level NoTestRun --wait 10 --json
```
8. **Auto-fix on failure**
- Parse JSON error output and fix issues (missing fields, invalid types, missing relationships).
- Re-run dry-run after each fix. Max 3 retries.
- If still failing after 3 retries, report remaining errors and ask for guidance.
9. **Copy validated stubs back**
- Replace `{dataset_path}/seed-data/` with the validated versions.
- Only copy back stub files you generated — do NOT copy gold file content into seed-data.
10. **Clean up and report**
- Delete the temporary SFDX project.
- Report: files generated, validation status, any fixes applied.
- For multiple datasets, print a summary table:
| # | Dataset | Stubs Generated | Validation | Notes |
|---|---------|----------------|------------|-------|
| 1 | … | … | … | … |
---
## Rules / Constraints
| Constraint | Rationale |
|-----------|-----------|
| Stubs include ONLY minimum required elements | Optional attributes add noise and can cause unexpected deployment errors |
| Never invent picklist values beyond what gold references | Extra values create false dependencies and mislead evaluators |
| Standard objects/fields never get stubs | They exist in every org; stubs would be redundant and can conflict |
| Always validate via dry-run before finalizing | Catches missing dependencies and malformed XML before the contributor sees them |
| API version defaults to 62.0 | Matches current afv-library convention; override only if gold file specifies otherwise |
| Copy back only stub files, not gold files | Mixing gold content into seed-data corrupts the dataset structure |
| Never hardcode `force-app/main/default/` — always read `sfdx-project.json` | Customers customize the package directory path; hardcoding breaks non-default projects |
| Reference cross-skills by name, never by filesystem path | Skill catalog layout varies across AFV installations; hardcoded paths break portability |
---
## Gotchas
| Issue | Resolution |
|-------|------------|
| Relationship traversal (`__r.Name`) implies a lookup field | Generate a Lookup stub for the corresponding `__c` field |
| Gold file references a field on a standard object | Only generate the custom field stub, not the standard object definition |
| Multiple gold files reference the same custom object | Generate the object stub once; place field stubs under the same object directory |
| Picklist referenced in formula via `ISPICKVAL` | Extract only the specific value string from the formula; do not add other values |
| Gold file has no custom dependencies | Skip stub generation; report "no seed-data needed" |
| Dry-run fails with `DUPLICATE_DEVELOPER_NAME` | A stub conflicts with an existing org object — rename or skip |
---
## Output Expectations
Deliverables:
- Stub metadata files: `{dataset_path}/seed-data/objects/{ObjectName}/fields/{FieldName}.field-meta.xml`
- Stub object definitions: `{dataset_path}/seed-data/objects/{ObjectName}/{ObjectName}.object-meta.xml`
- Stub Apex classes: `{dataset_path}/seed-data/classes/{ClassName}.cls` + `.cls-meta.xml`
- Console report: list of generated files, validation status, fixes applied
---
## Cross-Skill Integration
| Need | Delegate to |
|------|-------------|
| Generate the gold file for a dataset | Domain-specific skill (`generating-validation-rule`, `generating-apex`, etc.) |
| Create a new skill with eval datasets | `creating-sf-skill` |
| Generate a complete custom field (not a stub) | `generating-custom-field` |
| Generate a complete custom object (not a stub) | `generating-custom-object` |
---
## Reference File Index
| File | When to read |
|------|-------------|
| `references/stub-rules.md` | Phase 2, step 3 — before generating any stubs |
| `examples/stub-examples.md` | Phase 2, step 6 — to verify generated output matches expected patterns |

132
skills/generating-eval-seed-data/examples/stub-examples.md Normal file
View File

@ -0,0 +1,132 @@
# Seed-Data Stub Examples
Input/output examples showing gold file analysis and the resulting seed-data stubs.
---
## Example 1: Formula Field with Date and Picklist Dependencies
**Gold file**: `gold/objects/Contract/fields/Payment_Overdue__c.field-meta.xml`
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>Payment_Overdue__c</fullName>
<label>Payment Overdue</label>
<type>Text</type>
<formula>IF(AND(Payment_Due_Date__c &lt; TODAY(), ISPICKVAL(Payment_Status__c, "UNPAID")), "PAYMENT OVERDUE", null)</formula>
<formulaTreatBlanksAs>BlankAsZero</formulaTreatBlanksAs>
<length>20</length>
</CustomField>
```
**Analysis**: Formula references `Payment_Due_Date__c` (Date) and `Payment_Status__c`
(Picklist with value `UNPAID`).
**Generated stubs**:
`seed-data/objects/Contract/fields/Payment_Due_Date__c.field-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>Payment_Due_Date__c</fullName>
<label>Payment Due Date</label>
<type>Date</type>
</CustomField>
```
`seed-data/objects/Contract/fields/Payment_Status__c.field-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>Payment_Status__c</fullName>
<label>Payment Status</label>
<type>Picklist</type>
<valueSet>
<valueSetDefinition>
<value>
<fullName>UNPAID</fullName>
<label>UNPAID</label>
</value>
</valueSetDefinition>
</valueSet>
</CustomField>
```
Only `UNPAID` is included — no other values.
---
## Example 2: Flow Referencing Custom Objects
**Gold file**: A Flow XML that triggers on `Adoption__c` record creation and references
a Lookup to `Animal__c`.
**Generated stubs**:
`seed-data/objects/Adoption__c/Adoption__c.object-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<label>Adoption</label>
<nameField>
<label>Adoption Name</label>
<type>Text</type>
</nameField>
<pluralLabel>Adoptions</pluralLabel>
<sharingModel>ReadWrite</sharingModel>
</CustomObject>
```
`seed-data/objects/Animal__c/Animal__c.object-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<label>Animal</label>
<nameField>
<label>Animal Name</label>
<type>Text</type>
</nameField>
<pluralLabel>Animals</pluralLabel>
<sharingModel>ReadWrite</sharingModel>
</CustomObject>
```
`seed-data/objects/Adoption__c/fields/Animal__c.field-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>Animal__c</fullName>
<label>Animal</label>
<type>Lookup</type>
<referenceTo>Animal__c</referenceTo>
<relationshipName>Adoptions</relationshipName>
</CustomField>
```
---
## Example 3: Apex Class with Parent Class Dependency
**Gold file**: `gold/classes/OrderProcessor.cls` extends `BaseProcessor`.
**Generated stubs**:
`seed-data/classes/BaseProcessor.cls`:
```java
public abstract class BaseProcessor {
}
```
`seed-data/classes/BaseProcessor.cls-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<ApexClass xmlns="http://soap.sforce.com/2006/04/metadata">
<apiVersion>62.0</apiVersion>
<status>Active</status>
</ApexClass>
```
Empty stub — no method bodies or implementations.

147
skills/generating-eval-seed-data/references/stub-rules.md Normal file
View File

@ -0,0 +1,147 @@
# Stub Generation Rules
Rules for generating minimal seed-data stubs. Stubs declare what exists and what type
it is — nothing more.
---
## General Principles
- **Stubs, not full metadata**: generate the absolute minimum XML to declare a field,
object, or class. No optional attributes, no invented values.
- **Only what gold references**: for picklists, include ONLY values explicitly named in
the gold file. For objects, do not add fields beyond what gold uses.
- **No optional attributes**: omit `description`, `trackHistory`, `trackTrending`,
`required`, `externalId`, `inlineHelpText`, `restricted`, `sorted`, `default`,
`relationshipLabel`, and any other non-essential element.
- **Standard objects don't need definitions**: Account, Contact, Opportunity, Case, Lead,
User, and other standard objects already exist. Only stub their custom fields (`__c`).
- **API version**: use `62.0` unless gold specifies otherwise.
---
## Custom Field Stubs
**Path**: `seed-data/objects/{ObjectName}/fields/{FieldApiName}.field-meta.xml`
Include ONLY these elements per field type:
| Field Type | Required Elements |
|-----------|------------------|
| Lookup | `fullName`, `label`, `type`, `referenceTo`, `relationshipName` |
| Master-Detail | `fullName`, `label`, `type`, `referenceTo`, `relationshipName` |
| Picklist | `fullName`, `label`, `type`, `valueSet` (only referenced values) |
| Text | `fullName`, `label`, `type`, `length` |
| Number | `fullName`, `label`, `type`, `precision`, `scale` |
| Currency | `fullName`, `label`, `type`, `precision`, `scale` |
| Percent | `fullName`, `label`, `type`, `precision`, `scale` |
| Date | `fullName`, `label`, `type` |
| DateTime | `fullName`, `label`, `type` |
| Checkbox | `fullName`, `label`, `type`, `defaultValue` |
| Email | `fullName`, `label`, `type` |
| Phone | `fullName`, `label`, `type` |
| Url | `fullName`, `label`, `type` |
| TextArea | `fullName`, `label`, `type` |
| LongTextArea | `fullName`, `label`, `type`, `length`, `visibleLines` |
### Picklist Value Rules
Only include picklist values that appear explicitly in the gold file. Common patterns:
| Gold Pattern | Extract |
|-------------|---------|
| `ISPICKVAL(Status__c, "UNPAID")` | `UNPAID` only |
| `IF(Stage__c = "Closed Won", ...)` | `Closed Won` only |
| `<value><fullName>Active</fullName>...</value>` | `Active` only |
Do NOT add values "for completeness." If the gold formula checks for `"UNPAID"`, do not
add `PAID`, `PARTIALLY_PAID`, or any other values.
---
## Custom Object Stubs
**Path**: `seed-data/objects/{ObjectApiName}/{ObjectApiName}.object-meta.xml`
Include ONLY these elements:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<label>{Object Label}</label>
<nameField>
<label>{Object Label} Name</label>
<type>Text</type>
</nameField>
<pluralLabel>{Object Plural Label}</pluralLabel>
<sharingModel>ReadWrite</sharingModel>
</CustomObject>
```
Do NOT add `description`, `enableActivities`, `enableBulkApi`, `enableHistory`,
`enableReports`, `enableSearch`, or any other optional element.
---
## Apex Class Stubs
**Path**: `seed-data/classes/{ClassName}.cls` + `seed-data/classes/{ClassName}.cls-meta.xml`
The `.cls` file is an empty stub — class signature only:
```java
public class MyUtilClass {
}
```
For interfaces:
```java
public interface MyInterface {
}
```
For abstract classes:
```java
public abstract class MyBaseClass {
}
```
The `.cls-meta.xml`:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<ApexClass xmlns="http://soap.sforce.com/2006/04/metadata">
<apiVersion>62.0</apiVersion>
<status>Active</status>
</ApexClass>
```
Do NOT generate method bodies, business logic, or implementations.
---
## Dependency Detection Patterns
| Gold Pattern | Dependency Type | Stub Needed |
|-------------|----------------|-------------|
| `Payment_Due_Date__c` in formula | Custom field | Date field stub |
| `AssetProvided__r.Name` in formula | Lookup relationship | Lookup field stub for `AssetProvided__c` |
| `ISPICKVAL(Status__c, "X")` | Picklist field | Picklist stub with value `X` |
| `<referenceTo>Animal__c</referenceTo>` | Custom object | Object stub for `Animal__c` |
| `extends BaseHandler` | Apex parent class | Class stub for `BaseHandler` |
| `implements Queueable` | Standard interface | No stub needed (standard) |
| `Account.Name` | Standard object + field | No stub needed |
| `CustomObj__c.Custom_Field__c` | Custom field on custom object | Both object and field stubs |
---
## Common Validation Errors and Fixes
| Error Pattern | Fix |
|--------------|-----|
| `Field does not exist: FieldName__c` | Add the missing field stub |
| `Invalid type: ObjectName__c` | Add the custom object stub |
| `Missing required field: X` | Add the required XML element |
| `Invalid picklist value` | Add the missing value to the picklist stub |
| `Relationship not found` | Add the lookup field that defines the relationship |
| `apiVersion is required` | Add `<apiVersion>62.0</apiVersion>` |

3
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-apex-dependencies/gold/classes/BaseProcessor.cls Normal file
View File

@ -0,0 +1,3 @@
// STUB — review and update expected output before running eval
public abstract class BaseProcessor {
}

6
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-apex-dependencies/gold/classes/BaseProcessor.cls-meta.xml Normal file
View File

@ -0,0 +1,6 @@
<!-- STUB — review and update expected output before running eval -->
<?xml version="1.0" encoding="UTF-8"?>
<ApexClass xmlns="http://soap.sforce.com/2006/04/metadata">
<apiVersion>62.0</apiVersion>
<status>Active</status>
</ApexClass>

1
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-apex-dependencies/prompt.md Normal file
View File

@ -0,0 +1 @@
Generate seed-data for an Apex dataset where the gold file is an OrderProcessor class that extends BaseProcessor and implements Queueable. The dataset path is skills/generating-apex/tests/evals/generating-apex-order-processor/. My target org is myDevOrg.

12
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-flow-custom-objects/gold/objects/Adoption__c/Adoption__c.object-meta.xml Normal file
View File

@ -0,0 +1,12 @@
<!-- STUB — review and update expected output before running eval -->
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<label>Adoption</label>
<nameField>
<label>Adoption Name</label>
<type>Text</type>
</nameField>
<pluralLabel>Adoptions</pluralLabel>
<sharingModel>ReadWrite</sharingModel>
</CustomObject>

12
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-flow-custom-objects/gold/objects/Animal__c/Animal__c.object-meta.xml Normal file
View File

@ -0,0 +1,12 @@
<!-- STUB — review and update expected output before running eval -->
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<label>Animal</label>
<nameField>
<label>Animal Name</label>
<type>Text</type>
</nameField>
<pluralLabel>Animals</pluralLabel>
<sharingModel>ReadWrite</sharingModel>
</CustomObject>

1
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-flow-custom-objects/prompt.md Normal file
View File

@ -0,0 +1 @@
I need seed-data stubs for a Flow dataset. The gold file is a record-triggered Flow on Adoption__c that creates a related record on Animal__c via a Lookup field. Target org is myDevOrg. The dataset is at skills/generating-flow/tests/evals/generating-flow-adoption-trigger/.

7
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-formula-field/gold/objects/Contract/fields/Payment_Due_Date__c.field-meta.xml Normal file
View File

@ -0,0 +1,7 @@
<!-- STUB — review and update expected output before running eval -->
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>Payment_Due_Date__c</fullName>
<label>Payment Due Date</label>
<type>Date</type>
</CustomField>

15
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-formula-field/gold/objects/Contract/fields/Payment_Status__c.field-meta.xml Normal file
View File

@ -0,0 +1,15 @@
<!-- STUB — review and update expected output before running eval -->
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>Payment_Status__c</fullName>
<label>Payment Status</label>
<type>Picklist</type>
<valueSet>
<valueSetDefinition>
<value>
<fullName>UNPAID</fullName>
<label>UNPAID</label>
</value>
</valueSetDefinition>
</valueSet>
</CustomField>

1
skills/generating-eval-seed-data/tests/evals/generating-eval-seed-data-formula-field/prompt.md Normal file
View File

@ -0,0 +1 @@
Generate seed-data for the dataset at skills/generating-validation-rule/tests/evals/generating-validation-rule-formula-deps/. The gold file is a validation rule on Contract that uses a formula referencing Payment_Due_Date__c and ISPICKVAL(Payment_Status__c, "UNPAID"). My target org is myDevOrg.