mirror of
https://github.com/forcedotcom/afv-library.git
synced 2026-07-30 11:19:27 +08:00
feat: add ADK eval datasets for applying-cms-brand skill
Add three evaluation datasets to satisfy the mandatory ADK Eval onboarding requirement for skill contributions to afv-library. Datasets cover: single brand (happy path), multiple brands (user selection), and no brand found (graceful degradation). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Runying Mao
parent
649931b312
commit
b07607caab
@ -0,0 +1,47 @@
|
||||
# Expected Output
|
||||
|
||||
## Agent Behavior
|
||||
|
||||
1. Agent calls `search_brands` to find available brands
|
||||
2. Agent finds 3 brands and presents options to user using `ask_followup_question`:
|
||||
- "I found 3 brands in your CMS. Which one should I apply?
|
||||
1. Nova Enterprise Brand
|
||||
2. Nova Startup Brand
|
||||
3. Nova Internal Communications
|
||||
Which brand would you like to use?"
|
||||
3. User selects "Nova Enterprise Brand"
|
||||
4. Agent calls `get_brand_instructions` with managedContentId "9Pn5g000000AAA111"
|
||||
5. Agent applies brand guidelines to the product description
|
||||
|
||||
## Expected Branded Product Description
|
||||
|
||||
The generated description should:
|
||||
- Lead with outcomes, not features
|
||||
- Use power verbs (accelerate, transform, empower, unlock)
|
||||
- Have a bold, authoritative, forward-thinking voice
|
||||
- Be confident, aspirational, and direct in tone
|
||||
- Keep paragraphs to 3 sentences maximum
|
||||
- Use second person ("you", "your team")
|
||||
- Not use "simple", "easy", or clichés
|
||||
- Incorporate key messages where natural
|
||||
|
||||
## Example Branded Output
|
||||
|
||||
"Your Team, Unleashed.
|
||||
|
||||
Accelerate collaboration across your entire organization. Track progress, share files, and communicate in real time — all from one platform your team already knows how to use.
|
||||
|
||||
Enterprise-grade reliability. Startup-speed agility. Built for teams that move fast."
|
||||
|
||||
## Evaluation Criteria
|
||||
|
||||
- [ ] Agent searched for brands before generating content
|
||||
- [ ] Agent presented all brand options to user (did not auto-select)
|
||||
- [ ] Agent waited for user selection before proceeding
|
||||
- [ ] Agent extracted brand instructions for the selected brand
|
||||
- [ ] Output leads with outcomes, not features
|
||||
- [ ] Output uses power verbs
|
||||
- [ ] Output matches brand voice (bold, authoritative, forward-thinking)
|
||||
- [ ] Output matches brand tone (confident, aspirational, direct)
|
||||
- [ ] Guardrails are respected (no passive headlines, no "simple"/"easy", no clichés)
|
||||
- [ ] Agent summarized which guidelines were applied
|
||||
@ -0,0 +1,5 @@
|
||||
Search for our brand and apply it to the following product description:
|
||||
|
||||
"Our project management tool helps teams collaborate efficiently. It includes task tracking, file sharing, and real-time messaging."
|
||||
|
||||
Use our brand voice and tone to make it compelling.
|
||||
@ -0,0 +1,10 @@
|
||||
{
|
||||
"description": "Mocked response from get_brand_instructions tool for Nova Enterprise Brand (selected by user).",
|
||||
"tool": "get_brand_instructions",
|
||||
"input": {
|
||||
"managedContentId": "9Pn5g000000AAA111"
|
||||
},
|
||||
"response": {
|
||||
"promptBody": "Brand: Nova Enterprise\n\nBrand Voice: Bold, authoritative, and forward-thinking. Speak as a market leader.\n\nBrand Tone: Confident, aspirational, and direct. Inspire action and convey expertise.\n\nKey Messages:\n- Built for teams that move fast\n- Enterprise-grade reliability, startup-speed agility\n- The future of work, delivered today\n\nContent Rules:\n- Lead with outcomes, not features\n- Use power verbs (accelerate, transform, empower, unlock)\n- Keep paragraphs to 3 sentences maximum\n- Use second person (\"you\", \"your team\")\n- Sentence fragments are acceptable for emphasis\n\nTypography:\n- Headlines: Bold, concise (5 words or fewer)\n- Body: Clear, punchy sentences\n\nGuardrails:\n- Never use passive voice in headlines\n- Do not say \"simple\" or \"easy\" (implies the problem was trivial)\n- Avoid clichés (\"game-changer\", \"synergy\", \"leverage\")\n- Do not reference specific pricing"
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,36 @@
|
||||
{
|
||||
"description": "Mocked response from search_brands tool when multiple brands exist in the org.",
|
||||
"tool": "search_brands",
|
||||
"input": {
|
||||
"inputs": [
|
||||
{
|
||||
"searchQuery": "brand"
|
||||
}
|
||||
]
|
||||
},
|
||||
"response": {
|
||||
"results": [
|
||||
{
|
||||
"managedContentId": "9Pn5g000000AAA111",
|
||||
"managedContentKey": "MCK-brand-nova-enterprise",
|
||||
"title": "Nova Enterprise Brand",
|
||||
"contentUrl": "/cms/brand/nova-enterprise",
|
||||
"totalResults": 3
|
||||
},
|
||||
{
|
||||
"managedContentId": "9Pn5g000000BBB222",
|
||||
"managedContentKey": "MCK-brand-nova-startup",
|
||||
"title": "Nova Startup Brand",
|
||||
"contentUrl": "/cms/brand/nova-startup",
|
||||
"totalResults": 3
|
||||
},
|
||||
{
|
||||
"managedContentId": "9Pn5g000000CCC333",
|
||||
"managedContentKey": "MCK-brand-nova-internal",
|
||||
"title": "Nova Internal Communications",
|
||||
"contentUrl": "/cms/brand/nova-internal",
|
||||
"totalResults": 3
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,30 @@
|
||||
# Expected Output
|
||||
|
||||
## Agent Behavior
|
||||
|
||||
1. Agent calls `search_brands` to find available brands
|
||||
2. Agent receives empty results (no brands found)
|
||||
3. Agent does NOT attempt to generate branded content without guidelines
|
||||
4. Agent informs the user and offers alternatives:
|
||||
- "No brands found in Salesforce CMS. To use branding:
|
||||
1. Create a brand in Salesforce CMS (Content Type: sfdc_cms__brand)
|
||||
2. Provide brand guidelines directly in this conversation
|
||||
Would you like to proceed without CMS branding, or provide guidelines manually?"
|
||||
|
||||
## Expected Content Output
|
||||
|
||||
No branded content should be generated. The agent should:
|
||||
- Clearly communicate that no brands were found
|
||||
- Explain how to create a brand in CMS
|
||||
- Offer the alternative of providing guidelines manually
|
||||
- Wait for user direction before proceeding
|
||||
|
||||
## Evaluation Criteria
|
||||
|
||||
- [ ] Agent searched for brands before attempting to generate content
|
||||
- [ ] Agent did NOT generate branded content without guidelines
|
||||
- [ ] Agent informed user that no brands were found
|
||||
- [ ] Agent offered actionable alternatives (create in CMS or provide manually)
|
||||
- [ ] Agent did NOT hallucinate or assume brand guidelines
|
||||
- [ ] Agent waited for user response before proceeding
|
||||
- [ ] Error message was clear and helpful (not technical/confusing)
|
||||
@ -0,0 +1,4 @@
|
||||
Apply our brand to the following landing page headline and subheading:
|
||||
|
||||
Headline: "Welcome to Our Service"
|
||||
Subheading: "We provide the tools you need to succeed in today's market."
|
||||
@ -0,0 +1,15 @@
|
||||
{
|
||||
"description": "Mocked response from search_brands tool when no brands are found in the org.",
|
||||
"tool": "search_brands",
|
||||
"input": {
|
||||
"inputs": [
|
||||
{
|
||||
"searchQuery": "brand"
|
||||
}
|
||||
]
|
||||
},
|
||||
"response": {
|
||||
"results": [],
|
||||
"totalResults": 0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,43 @@
|
||||
# Expected Output
|
||||
|
||||
## Agent Behavior
|
||||
|
||||
1. Agent calls `search_brands` with a general query
|
||||
2. Agent finds one brand ("Acme Corp Brand") and confirms with user: "I found the brand 'Acme Corp Brand'. Should I apply this brand's guidelines to the content?"
|
||||
3. After user confirms, agent calls `get_brand_instructions`
|
||||
4. Agent applies brand guidelines and produces branded email content
|
||||
|
||||
## Expected Branded Email
|
||||
|
||||
The generated email should:
|
||||
- Use a professional yet approachable voice
|
||||
- Have a warm, encouraging, empowering tone
|
||||
- Address the reader directly with "you"
|
||||
- Use active voice
|
||||
- Keep sentences under 20 words
|
||||
- Contain no more than 1 exclamation mark per paragraph
|
||||
- Not use superlatives ("best", "fastest", "most powerful")
|
||||
- Incorporate key messages where natural (innovation made simple, your success is our priority)
|
||||
|
||||
## Example Branded Output
|
||||
|
||||
"Hi, welcome to Acme Corp. We're glad you're here.
|
||||
|
||||
You now have access to tools designed to help you succeed. Here's how to get started:
|
||||
|
||||
1. Set up your profile so we can personalize your experience.
|
||||
2. Explore the dashboard to see what's available to you.
|
||||
3. Reach out to our team if you need guidance along the way.
|
||||
|
||||
Your success drives everything we do. We're here to support you at every step."
|
||||
|
||||
## Evaluation Criteria
|
||||
|
||||
- [ ] Agent searched for brands before generating content
|
||||
- [ ] Agent confirmed brand selection with user
|
||||
- [ ] Agent extracted brand instructions before generating content
|
||||
- [ ] Output matches brand voice (professional yet approachable)
|
||||
- [ ] Output matches brand tone (warm, encouraging, empowering)
|
||||
- [ ] Content rules are respected (active voice, short sentences, "you" addressing)
|
||||
- [ ] Guardrails are not violated (no competitor names, no performance guarantees)
|
||||
- [ ] Agent summarized which guidelines were applied
|
||||
@ -0,0 +1,5 @@
|
||||
Apply my brand guidelines to the following welcome email:
|
||||
|
||||
"Hi there, welcome to our platform! We're excited to have you on board. Here's what you can do to get started..."
|
||||
|
||||
Make sure the email matches our brand voice and tone.
|
||||
@ -0,0 +1,10 @@
|
||||
{
|
||||
"description": "Mocked response from get_brand_instructions tool for Acme Corp brand.",
|
||||
"tool": "get_brand_instructions",
|
||||
"input": {
|
||||
"managedContentId": "9Pn5g000000XYZABC"
|
||||
},
|
||||
"response": {
|
||||
"promptBody": "Brand: Acme Corp\n\nBrand Voice: Professional yet approachable. Speak with confidence and clarity. Avoid jargon.\n\nBrand Tone: Warm, encouraging, and empowering. Make the reader feel supported and capable.\n\nKey Messages:\n- Innovation made simple\n- Your success is our priority\n- Technology that works for you\n\nContent Rules:\n- Use active voice\n- Keep sentences under 20 words\n- Address the reader directly with \"you\"\n- Avoid superlatives (never say \"best\", \"fastest\", \"most powerful\")\n- Do not use exclamation marks excessively (max 1 per paragraph)\n\nGuardrails:\n- Never mention competitors by name\n- Do not make performance guarantees or promises\n- Avoid technical jargon unless explaining it immediately after"
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,22 @@
|
||||
{
|
||||
"description": "Mocked response from search_brands tool when a single brand exists in the org.",
|
||||
"tool": "search_brands",
|
||||
"input": {
|
||||
"inputs": [
|
||||
{
|
||||
"searchQuery": "brand"
|
||||
}
|
||||
]
|
||||
},
|
||||
"response": {
|
||||
"results": [
|
||||
{
|
||||
"managedContentId": "9Pn5g000000XYZABC",
|
||||
"managedContentKey": "MCK-brand-acme-corp",
|
||||
"title": "Acme Corp Brand",
|
||||
"contentUrl": "/cms/brand/acme-corp",
|
||||
"totalResults": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user