Add-skill-test-examples

This commit is contained in:
jasmine.kaur 2026-04-21 10:32:25 +05:30
parent 6537cf1a1e
commit e57b0ebd8b
No known key found for this signature in database
GPG Key ID: A037296830C500E7
15 changed files with 733 additions and 0 deletions

239
tests/README.md Normal file
View File

@ -0,0 +1,239 @@
# Testing Guide for `afv-library`
This directory contains all tests for the skills in this repo. Tests live here - never inside `skills/` - so the published package stays clean.
## Quick start
```bash
npm ci
pip install pytest # only if you have Python script-tests
npm run test # run everything (validator + skill-tests + script-tests)
npm run validate:skills # structural validator only
npm run test:skills # skill-tests only (Vitest)
npm run test:scripts # script-tests only (pytest, bash, bats, TypeScript/JS)
```
## Directory structure
When skill owners add tests, the directory grows like this:
```
tests/
├── README.md ← you are here
├── helpers/ ← shared utilities (part of the framework)
│ ├── index.ts
│ ├── parse-skill.ts
│ ├── extract-code-blocks.ts
│ ├── apex-validator.ts
│ ├── xml-validator.ts
│ └── link-checker.ts
├── generating-apex/ ← one skill, one directory
│ └── skill-tests/
│ └── content.test.ts
├── testing-agentforce/ ← skill with scripts
│ ├── skill-tests/
│ │ └── content.test.ts
│ └── script-tests/
│ └── test_run_specs.sh
└── ...
```
### How to organize
The directory name doesn't have to match a skill name - Vitest and the discovery script don't check it. You can organize however makes sense for your team:
- **One directory per skill** (e.g. `tests/generating-apex/`) - good default when one team owns one skill.
- **One directory per team** (e.g. `tests/agentforce/`) - useful when a team owns multiple skills and wants to cover them in fewer test files.
Inside each directory, up to two folders:
| Folder | Purpose | Language | Runner |
| --------------- | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | ----------------------------- |
| `skill-tests/` | Validate SKILL.md content - frontmatter values, code block correctness, asset existence | TypeScript only | Vitest |
| `script-tests/` | Test executable scripts in their native language | Same language as your script (Python, Bash, Bats, TypeScript/JS) | pytest, bash, bats, or vitest |
Most skills only need `skill-tests/`. Add `script-tests/` only if your skill ships scripts in `scripts/`.
## Adding tests for your skill
### Step 1: Create the directory
```bash
mkdir -p tests/<your-directory-name>/skill-tests
```
Use the skill name (e.g. `generating-apex`) or a team name (e.g. `agentforce`) - whichever fits your ownership model.
### Step 2: Write a skill-test
Create a `*.test.ts` file inside `skill-tests/`. The `SKILL` constant controls which skill directory gets read - it's independent of the test directory name.
**Single-skill example** (`tests/generating-apex/skill-tests/content.test.ts`):
```typescript
import { describe, it, expect } from "vitest"
import { readSkillFile, skillHasFile, extractCodeBlocks, hasBalancedBraces } from "../../helpers"
const SKILL = "generating-apex"
describe(`${SKILL}: SKILL.md content`, () => {
const content = readSkillFile(SKILL, "SKILL.md")
it("description mentions the key activation context", () => {
expect(content).toMatch(/Apex/)
})
})
describe(`${SKILL}: code examples`, () => {
const body = readSkillFile(SKILL, "SKILL.md")
const blocks = extractCodeBlocks(body, "apex")
it("has code blocks", () => {
expect(blocks.length).toBeGreaterThan(0)
})
for (const block of blocks) {
it(`block at line ${block.startLine} has balanced braces`, () => {
const result = hasBalancedBraces(block.content)
expect(result.balanced, `open=${result.open} close=${result.close}`).toBe(true)
})
}
})
describe(`${SKILL}: required assets exist`, () => {
const files = ["assets/template.cls", "references/patterns.md"]
for (const file of files) {
it(`${file} exists`, () => {
expect(skillHasFile(SKILL, file)).toBe(true)
})
}
})
```
**Multi-skill example** (`tests/agentforce/skill-tests/content.test.ts`) - one file covering multiple skills owned by the same team:
```typescript
import { describe, it, expect } from "vitest"
import { readSkillFile, extractCodeBlocks, hasBalancedBraces } from "../../helpers"
const SKILLS = [
"developing-agentforce",
"testing-agentforce",
"observing-agentforce",
]
for (const skill of SKILLS) {
describe(`${skill}: SKILL.md content`, () => {
const content = readSkillFile(skill, "SKILL.md")
it("description mentions Agentforce", () => {
expect(content).toMatch(/Agentforce/)
})
})
describe(`${skill}: code examples have balanced braces`, () => {
const body = readSkillFile(skill, "SKILL.md")
const blocks = extractCodeBlocks(body, "yaml")
for (const block of blocks) {
it(`block at line ${block.startLine}`, () => {
const result = hasBalancedBraces(block.content)
expect(result.balanced).toBe(true)
})
}
})
}
```
Vitest discovers any `*.test.ts` file under `tests/*/skill-tests/` automatically - no registration needed.
### Step 3: (Optional) Add script-tests
Only needed if your skill has scripts in `skills/<your-skill>/scripts/`.
```bash
mkdir -p tests/<your-skill-name>/script-tests
```
Add a test file matching the **required** naming convention for your language. Files that don't match these patterns will not be discovered and will silently not run:
| Language | File naming pattern | Runner |
| ------------- | -------------------------- | ------ |
| Python | `test_*.py` or `*_test.py` | pytest |
| Bash | `test_*.sh` or `*_test.sh` | bash |
| Bats | `*.bats` | bats |
| TypeScript/JS | `*.test.ts` or `*.test.js` | vitest |
### Step 4: Verify
```bash
npm run test:skills # should pick up your new skill-test
npm run test:scripts # should pick up your new script-test
```
## Shared helpers
The `tests/helpers/` directory provides reusable utilities so skill owners don't have to rewrite common operations. Import them in any skill-test:
```typescript
import {
readSkillFile,
skillHasFile,
parseSkill,
extractCodeBlocks,
hasBalancedBraces,
hasClassOrInterfaceDeclaration,
containsAnnotation,
isWellFormedXml,
findBrokenLinks,
} from "../../helpers"
```
Browse `tests/helpers/index.ts` to see all available exports. The helpers cover skill file access, code block extraction, Apex structural checks, XML validation, and link checking.
Need a helper that doesn't exist yet? Add it to `tests/helpers/` and export it from `index.ts`.
## How discovery works
**Skill-tests**: Vitest finds all `*.test.{ts,js}` files (configured in `vitest.config.ts`). The `npm run test:skills` command filters to only files with `skill-tests` in the path. No registration needed - drop the file and it runs.
**Script-tests**: The discovery script (`scripts/run-skill-tests.sh`) walks every `tests/*/script-tests/` directory, matches test files by extension, and dispatches to the native runner. If a required runner isn't installed (e.g. pytest), it prints `SKIPPED` instead of failing.
The discovery script matches files to runners as follows:
| Language | Finds files matching | Runs them with |
| ------------- | ------------------------ | -------------- |
| Python | `test_*.py`, `*_test.py` | pytest |
| Bash | `test_*.sh`, `*_test.sh` | bash |
| Bats | `*.bats` | bats |
| TypeScript/JS | `*.test.ts`, `*.test.js` | vitest |
**Naming matters.** If your test file doesn't match the expected pattern, it will silently not run:
| Runs | Does NOT run |
| ------------------ | --------------------------------------------- |
| `test_analyzer.py` | `analyzer_tests.py` |
| `analyzer_test.py` | `test-analyzer.py` (hyphens, not underscores) |
| `test_search.sh` | `search_tests.sh` |
| `content.test.ts` | `content.spec.ts` |
## Prerequisites
- **Node.js** (v22, see `.nvmrc`) + `npm ci`
- **Python 3.9+** and `pip install pytest` - only needed if you have Python script-tests
- No additional tools needed for bash script-tests

View File

@ -0,0 +1,22 @@
import { describe, it, expect } from "vitest"
import fs from "fs"
import path from "path"
const REFS_DIR = path.join(__dirname, "..", "..", "..", "skills", "generating-apex", "references")
describe("generating-apex: reference files are valid", () => {
it("references directory exists and has .cls files", () => {
const files = fs.readdirSync(REFS_DIR).filter((f) => f.endsWith(".cls"))
expect(files.length).toBeGreaterThan(0)
})
it("all reference files are non-empty", () => {
const files = fs.readdirSync(REFS_DIR).filter((f) => f.endsWith(".cls"))
for (const file of files) {
const content = fs.readFileSync(path.join(REFS_DIR, file), "utf8")
expect(content.trim().length, `${file} should not be empty`).toBeGreaterThan(0)
}
})
// TODO: Add more tests - validate class names match filenames, check for ApexDoc, etc.
})

View File

@ -0,0 +1,26 @@
import { describe, it, expect } from "vitest"
import fs from "fs"
import path from "path"
const TEMPLATES_DIR = path.join(__dirname, "..", "..", "..", "skills", "generating-apex", "assets")
describe("generating-apex: template files are valid", () => {
it("all .cls files are non-empty", () => {
const files = fs.readdirSync(TEMPLATES_DIR).filter((f) => f.endsWith(".cls"))
expect(files.length).toBeGreaterThan(0)
for (const file of files) {
const content = fs.readFileSync(path.join(TEMPLATES_DIR, file), "utf8")
expect(content.trim().length, `${file} should not be empty`).toBeGreaterThan(0)
}
})
it("every template has a placeholder token", () => {
const files = fs.readdirSync(TEMPLATES_DIR).filter((f) => f.endsWith(".cls"))
for (const file of files) {
const content = fs.readFileSync(path.join(TEMPLATES_DIR, file), "utf8")
expect(content, `${file} should have a {placeholder}`).toMatch(/\{[A-Z]/)
}
})
// TODO: Add more tests - validate specific templates, check for required methods, etc.
})

View File

@ -0,0 +1,23 @@
import { describe, it, expect } from "vitest"
import { readSkillFile, skillHasFile, extractCodeBlocks, hasBalancedBraces } from "../../helpers"
const SKILL = "generating-apex"
describe(`${SKILL}: Apex template assets`, () => {
it("service.cls template exists", () => {
expect(skillHasFile(SKILL, "assets/service.cls")).toBe(true)
})
it("batch.cls template has balanced braces", () => {
const content = readSkillFile(SKILL, "assets/batch.cls")
const result = hasBalancedBraces(content)
expect(result.balanced, `open=${result.open} close=${result.close}`).toBe(true)
})
it("batch.cls implements Database.Batchable", () => {
const content = readSkillFile(SKILL, "assets/batch.cls")
expect(content).toContain("Database.Batchable")
})
// TODO: Add more tests - validate remaining templates, check code blocks in SKILL.md, etc.
})

View File

@ -0,0 +1,27 @@
/**
* Lightweight structural checks for Apex .cls files.
* These do NOT compile Apex they verify basic structural integrity
* via string matching. Sufficient for catching template corruption.
*/
export function hasClassOrInterfaceDeclaration(content: string): boolean {
return /\b(class|interface)\s+[\w{}\[\]]+/.test(content)
}
export function hasBalancedBraces(content: string): { balanced: boolean; open: number; close: number } {
const open = (content.match(/{/g) || []).length
const close = (content.match(/}/g) || []).length
return { balanced: open === close, open, close }
}
export function containsAnnotation(content: string, annotation: string): boolean {
return content.includes(annotation)
}
export function containsKeyword(content: string, keyword: string): boolean {
return new RegExp(`\\b${keyword}\\b`).test(content)
}
export function hasApexDoc(content: string): boolean {
return content.includes("@param") || content.includes("@return") || content.includes("@description")
}

View File

@ -0,0 +1,42 @@
export interface CodeBlock {
language: string
content: string
startLine: number
}
/**
* Extracts fenced code blocks from markdown content.
* Optionally filters by language tag.
*/
export function extractCodeBlocks(markdown: string, language?: string): CodeBlock[] {
const blocks: CodeBlock[] = []
const lines = markdown.split("\n")
let inBlock = false
let currentLang = ""
let currentContent: string[] = []
let blockStart = 0
for (let i = 0; i < lines.length; i++) {
const line = lines[i]
if (!inBlock && line.match(/^```(\w*)/)) {
inBlock = true
currentLang = line.match(/^```(\w*)/)![1] || ""
currentContent = []
blockStart = i + 1
} else if (inBlock && line.trim() === "```") {
inBlock = false
if (!language || currentLang.toLowerCase() === language.toLowerCase()) {
blocks.push({
language: currentLang,
content: currentContent.join("\n"),
startLine: blockStart,
})
}
} else if (inBlock) {
currentContent.push(line)
}
}
return blocks
}

18
tests/helpers/index.ts Normal file
View File

@ -0,0 +1,18 @@
export { parseSkill, listSkillDirs, getSkillsDir, skillHasFile, readSkillFile } from "./parse-skill"
export type { ParsedSkill } from "./parse-skill"
export { extractCodeBlocks } from "./extract-code-blocks"
export type { CodeBlock } from "./extract-code-blocks"
export { isWellFormedXml } from "./xml-validator"
export {
hasClassOrInterfaceDeclaration,
hasBalancedBraces,
containsAnnotation,
containsKeyword,
hasApexDoc,
} from "./apex-validator"
export { extractRelativeLinks, findBrokenLinks } from "./link-checker"
export type { BrokenLink } from "./link-checker"

View File

@ -0,0 +1,54 @@
import fs from "fs"
import path from "path"
export interface BrokenLink {
link: string
text: string
reason: string
}
/**
* Extracts relative markdown links from content.
* Ignores URLs (http/https), anchors-only (#heading), and mailto links.
*/
export function extractRelativeLinks(content: string): Array<{ text: string; link: string }> {
const linkRegex = /\[([^\]]*)\]\(([^)]+)\)/g
const links: Array<{ text: string; link: string }> = []
let match: RegExpExecArray | null
while ((match = linkRegex.exec(content)) !== null) {
const link = match[2]
if (link.startsWith("http://") || link.startsWith("https://") || link.startsWith("#") || link.startsWith("mailto:")) {
continue
}
const filePath = link.split("#")[0]
if (filePath) {
links.push({ text: match[1], link: filePath })
}
}
return links
}
/**
* Checks all relative markdown links in a SKILL.md resolve to existing files.
*/
export function findBrokenLinks(skillDir: string, content: string): BrokenLink[] {
const links = extractRelativeLinks(content)
const broken: BrokenLink[] = []
for (const { text, link } of links) {
const resolved = path.resolve(skillDir, link)
if (!resolved.startsWith(path.resolve(skillDir))) {
broken.push({ link, text, reason: `Escapes skill directory via ../` })
continue
}
if (!fs.existsSync(resolved)) {
broken.push({ link, text, reason: `File not found: ${link}` })
}
}
return broken
}

View File

@ -0,0 +1,65 @@
import fs from "fs"
import path from "path"
import yaml from "js-yaml"
const SKILLS_DIR = path.join(__dirname, "..", "..", "skills")
export interface ParsedSkill {
dirName: string
dirPath: string
content: string
rawFrontmatter: string | null
frontmatter: Record<string, unknown> | null
body: string
}
export function getSkillsDir(): string {
return SKILLS_DIR
}
export function listSkillDirs(): string[] {
return fs.readdirSync(SKILLS_DIR).filter((entry) => {
const fullPath = path.join(SKILLS_DIR, entry)
return fs.statSync(fullPath).isDirectory() && !entry.startsWith(".")
})
}
export function parseSkill(dirName: string): ParsedSkill {
const dirPath = path.join(SKILLS_DIR, dirName)
const skillMdPath = path.join(dirPath, "SKILL.md")
if (!fs.existsSync(skillMdPath)) {
return { dirName, dirPath, content: "", rawFrontmatter: null, frontmatter: null, body: "" }
}
const content = fs.readFileSync(skillMdPath, "utf8")
const match = content.match(/^---\r?\n([\s\S]*?)\r?\n---\r?\n?/)
if (!match) {
return { dirName, dirPath, content, rawFrontmatter: null, frontmatter: null, body: content }
}
const rawFrontmatter = match[1]
let frontmatter: Record<string, unknown> | null = null
try {
const parsed = yaml.load(rawFrontmatter)
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) {
frontmatter = parsed as Record<string, unknown>
}
} catch {
frontmatter = null
}
const body = content.slice(match[0].length)
return { dirName, dirPath, content, rawFrontmatter, frontmatter, body }
}
export function skillHasFile(dirName: string, relativePath: string): boolean {
return fs.existsSync(path.join(SKILLS_DIR, dirName, relativePath))
}
export function readSkillFile(dirName: string, relativePath: string): string {
return fs.readFileSync(path.join(SKILLS_DIR, dirName, relativePath), "utf8")
}

View File

@ -0,0 +1,54 @@
/**
* Lightweight XML well-formedness check.
* Verifies the string looks like valid XML by checking:
* - Has an opening tag
* - All opened tags are closed (or self-closing)
* - No obvious structural errors
*
* For a full XML parse, use DOMParser or a library.
* This is sufficient for catching common template corruption.
*/
export function isWellFormedXml(xml: string): { valid: boolean; error?: string } {
const trimmed = xml.trim()
if (!trimmed.startsWith("<")) {
return { valid: false, error: "Does not start with <" }
}
const stripped = trimmed.replace(/<!--[\s\S]*?-->/g, "")
const tagStack: string[] = []
const tagRegex = /<\/?([a-zA-Z_][\w:.-]*)[^>]*?\/?>/g
let match: RegExpExecArray | null
while ((match = tagRegex.exec(stripped)) !== null) {
const fullMatch = match[0]
const tagName = match[1]
if (fullMatch.startsWith("<?") || fullMatch.startsWith("<!")) {
continue
}
if (fullMatch.endsWith("/>")) {
continue
}
if (fullMatch.startsWith("</")) {
if (tagStack.length === 0) {
return { valid: false, error: `Closing tag </${tagName}> without matching open tag` }
}
const expected = tagStack.pop()
if (expected !== tagName) {
return { valid: false, error: `Expected closing tag </${expected}>, found </${tagName}>` }
}
} else {
tagStack.push(tagName)
}
}
if (tagStack.length > 0) {
return { valid: false, error: `Unclosed tags: ${tagStack.join(", ")}` }
}
return { valid: true }
}

View File

@ -0,0 +1,46 @@
import sys
import os
sys.path.insert(0, os.path.join(
os.path.dirname(__file__), "..", "..", "..", "skills",
"trigger-refactor-pipeline", "scripts",
))
from analyze_trigger import TriggerAnalyzer
CLEAN_TRIGGER = """\
trigger AccountTrigger on Account (before insert) {
for (Account a : Trigger.new) {
if (a.Name == null) {
a.Name = 'Default';
}
}
}"""
DML_IN_LOOP_TRIGGER = """\
trigger OpportunityTrigger on Opportunity (after update) {
for (Opportunity o : Trigger.new) {
Task t = new Task(WhatId = o.Id, Subject = 'Follow up');
insert t;
}
}"""
class TestTriggerAnalyzer:
def test_initializes_with_name(self):
analyzer = TriggerAnalyzer("MyTrigger")
assert analyzer.trigger_name == "MyTrigger"
def test_clean_trigger_has_no_dml_issues(self):
analyzer = TriggerAnalyzer("AccountTrigger")
analyzer.trigger_body = CLEAN_TRIGGER
analyzer.analyze_dml_in_loops()
assert len(analyzer.issues["dml_in_loops"]) == 0
def test_detects_dml_in_loop(self):
analyzer = TriggerAnalyzer("OpportunityTrigger")
analyzer.trigger_body = DML_IN_LOOP_TRIGGER
analyzer.analyze_dml_in_loops()
assert len(analyzer.issues["dml_in_loops"]) == 1
# TODO: Add more tests - SOQL in loops, bulkification, complexity scoring, etc.

View File

@ -0,0 +1,29 @@
import { describe, it, expect } from "vitest"
import { readSkillFile, skillHasFile, extractCodeBlocks, hasBalancedBraces } from "../../helpers"
const SKILL = "trigger-refactor-pipeline"
describe(`${SKILL}: SKILL.md content`, () => {
const content = readSkillFile(SKILL, "SKILL.md")
it("declares Python 3.9+ compatibility", () => {
expect(content).toMatch(/compatibility:.*Python 3\.9/)
})
it("has Apex code blocks with balanced braces", () => {
const blocks = extractCodeBlocks(content, "apex")
expect(blocks.length).toBeGreaterThan(0)
for (const block of blocks) {
const result = hasBalancedBraces(block.content)
expect(result.balanced, `block at line ${block.startLine}`).toBe(true)
}
})
it("required assets exist", () => {
expect(skillHasFile(SKILL, "scripts/analyze_trigger.py")).toBe(true)
expect(skillHasFile(SKILL, "assets/test_template.apex")).toBe(true)
expect(skillHasFile(SKILL, "references/handler_patterns.md")).toBe(true)
})
// TODO: Add more tests - validate test_template.apex structure, check bash code blocks, etc.
})

View File

@ -0,0 +1,20 @@
#!/usr/bin/env bats
SCRIPT="skills/using-ui-bundle-salesforce-data/scripts/graphql-search.sh"
@test "exits 1 with no arguments" {
run bash "$SCRIPT"
[ "$status" -eq 1 ]
}
@test "shows usage text when called with no arguments" {
run bash "$SCRIPT"
[[ "$output" == *"Usage:"* ]]
}
@test "exits 1 when schema file does not exist" {
run bash "$SCRIPT" -s /nonexistent/schema.graphql Account
[ "$status" -eq 1 ]
}
# TODO: Add more tests - valid schema lookup, entity extraction, multiple entities, etc.

View File

@ -0,0 +1,41 @@
#!/usr/bin/env bash
set -euo pipefail
PASS=0; FAIL=0
SCRIPT="skills/using-ui-bundle-salesforce-data/scripts/graphql-search.sh"
assert_eq() {
local label="$1" expected="$2" actual="$3"
if [[ "$expected" == "$actual" ]]; then
echo "$label"; PASS=$((PASS + 1))
else
echo "$label (expected '$expected', got '$actual')"; FAIL=$((FAIL + 1))
fi
}
assert_contains() {
local label="$1" haystack="$2" needle="$3"
if [[ "$haystack" == *"$needle"* ]]; then
echo "$label"; PASS=$((PASS + 1))
else
echo "$label (expected to contain '$needle')"; FAIL=$((FAIL + 1))
fi
}
echo "--- no arguments: prints usage and exits 1 ---"
rc=0; bash "$SCRIPT" >/dev/null 2>&1 || rc=$?
assert_eq "exits 1 with no args" "1" "$rc"
echo "--- missing schema file: exits 1 ---"
rc=0; bash "$SCRIPT" -s /nonexistent/schema.graphql Account >/dev/null 2>&1 || rc=$?
assert_eq "exits 1 on missing schema" "1" "$rc"
echo "--- no args shows usage text ---"
output=$(bash "$SCRIPT" 2>&1 || true)
assert_contains "shows usage" "$output" "Usage:"
# TODO: Add more tests - valid schema lookup, multiple entities, unknown entity warning, etc.
echo ""
echo "$PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]

View File

@ -0,0 +1,27 @@
import { describe, it, expect } from "vitest"
import { readSkillFile, skillHasFile, extractCodeBlocks, hasBalancedBraces } from "../../helpers"
const SKILL = "using-ui-bundle-salesforce-data"
describe(`${SKILL}: SKILL.md content`, () => {
const content = readSkillFile(SKILL, "SKILL.md")
it("description mentions Salesforce record operations", () => {
expect(content).toMatch(/Salesforce record operation/)
})
it("has GraphQL code blocks with balanced braces", () => {
const blocks = extractCodeBlocks(content, "graphql")
expect(blocks.length).toBeGreaterThan(0)
for (const block of blocks) {
const result = hasBalancedBraces(block.content)
expect(result.balanced, `block at line ${block.startLine}`).toBe(true)
}
})
it("required assets exist", () => {
expect(skillHasFile(SKILL, "scripts/graphql-search.sh")).toBe(true)
})
// TODO: Add more tests - validate TypeScript code blocks, check non-negotiable rules, etc.
})