mirror of https://github.com/forcedotcom/afv-library.git synced 2026-08-03 05:41:08 +08:00

Chandresh Patel 1d4f1dfd00

modify the skills name from - datacloud_code_extension → developing-datacloud-code-extension and datacloud_schema → getting-datacloud-schema

2026-04-29 09:48:56 -07:00

22 KiB

Raw Blame History

name	description
developing-datacloud-code-extension	Develop and deploy Data Cloud Code Extensions using SF CLI plugin. Use this skill when creating custom Python transformations for Data Cloud, deploying code extensions, or testing data transformations. Supports init, run, scan, and deploy operations.

developing-datacloud-code-extension Skill

Overview

This skill provides a complete workflow for developing, testing, and deploying custom Python code extensions to Salesforce Data Cloud. Code extensions allow you to write Python transformations that read from and write to Data Lake Objects (DLOs) and Data Model Objects (DMOs).

When to Use

User wants to create a new code extension project
User needs to test a code extension locally
User wants to scan code for required permissions
User needs to deploy a code extension to Data Cloud
User is working with Data Cloud transformations
User wants to read/write DLO or DMO data programmatically

Prerequisites Check

Before executing any code extension commands, verify prerequisites:

SF CLI with plugin installed

sf plugins --core | grep data-code-extension

If not installed:

sf plugins install @salesforce/plugin-data-codeextension

Python 3.11
```
python --version  # Should show 3.11.x
```

Data Cloud Custom Code SDK

pip list | grep salesforce-data-customcode

If not installed:

pip install salesforce-data-customcode

Docker running (for deploy only)
```
docker ps
```

Authenticated org

sf org display --target-org <org_alias> --json

Skill Workflow

Phase 1: Initialize Project

Create a new code extension project with scaffolding.

Commands:

For script-based code extensions (batch transformations):

sf data-code-extension script init --package-dir <directory>

For function-based code extensions (real-time):

sf data-code-extension function init --package-dir <directory>

Required Option:

--package-dir, -p - Directory path where the package will be created

Examples:

# Create script project in new directory
sf data-code-extension script init --package-dir ./my-transform

# Create function project in current directory
sf data-code-extension function init --package-dir .

What it creates:

my-transform/              # ← Project root
├── payload/               # ← CRITICAL: This is what --package-dir must point to for deploy
│   ├── entrypoint.py      # Main transformation code
│   ├── requirements.txt   # Python dependencies
│   └── config.json        # Code extension configuration
└── README.md

Directory Context During Workflow

IMPORTANT: Understanding the directory structure is critical for successful deployment.

After running init, your structure looks like:

my-transform/              # ← Project root (run commands from here)
├── payload/               # ← THIS directory contains deployable code
│   ├── entrypoint.py
│   ├── config.json
│   └── requirements.txt
└── README.md

Commands and their directory requirements:

Command	Run From	Path/File Argument
`init`	Parent directory	`<project-name>` or `.`
`scan`	Project root	`./payload/entrypoint.py`
`run`	Project root	`./payload/entrypoint.py`
`deploy`	Project root	`--package-dir ./payload` (REQUIRED)

CRITICAL: The --package-dir argument in deploy command MUST point to the payload directory, not the project root.

Phase 2: Develop Transformation

Edit payload/entrypoint.py with transformation logic.

Script Example (Batch):

from datacustomcode import Client

client = Client()

# Read from DLO
df = client.read_dlo('Employee__dll')

# Transform data (uppercase position field)
df['position_upper'] = df['position'].str.upper()

# Write to output DLO
client.write_to_dlo('Employee_Upper__dll', df, 'overwrite')

Function Example (Real-time):

from datacustomcode import FunctionClient

def transform(event, context):
    client = FunctionClient(context)

    # Process incoming record
    input_data = event['data']

    # Transform
    output = {
        'name': input_data['name'].upper(),
        'status': 'processed'
    }

    return output

Common Operations:

client.read_dlo('DLO_Name__dll') - Read from DLO
client.read_dmo('DMO_Name') - Read from DMO
client.write_to_dlo('DLO_Name__dll', df, 'overwrite') - Write to DLO
client.write_to_dmo('DMO_Name', df, 'upsert') - Write to DMO

Phase 3: Scan for Permissions

Scan the entrypoint file to detect required permissions and generate config.json.

Command:

sf data-code-extension script scan --entrypoint <entrypoint_file>

Example:

# Scan and update config.json
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

What it detects:

Read permissions for DLOs/DMOs
Write permissions for DLOs/DMOs
Python package dependencies
Updates config.json and requirements.txt

Example config.json:

{
  "version": "1.0",
  "permissions": {
    "read": ["Employee__dll"],
    "write": ["Employee_Upper__dll"]
  },
  "resources": {
    "cpu_size": "CPU_2XL"
  }
}

Phase 4: Validate DLO Schema (Pre-Test Check)

CRITICAL: Before running tests locally, validate that all DLOs used in your code exist and have the expected fields.

This prevents runtime errors and ensures your transformation will work with the actual Data Cloud schema.

Step 4a: Extract DLOs from config.json

After scanning, review the generated config.json to identify all DLOs:

cat payload/config.json

Look for DLOs in the permissions section:

{
  "permissions": {
    "read": ["Employee__dll", "Department__dll"],
    "write": ["Employee_Upper__dll"]
  }
}

Step 4b: Validate Each DLO Schema

Use the getting-datacloud-schema skill to verify DLOs exist and check field names.

For each DLO referenced in your code:

Verify DLO exists:

Ask Claude: "Use getting-datacloud-schema skill to check if Employee__dll exists in afvibe"

Or manually:

python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee__dll

Verify field names match:

Compare fields used in your entrypoint.py against the DLO schema:

In your code:
```
df['position_upper'] = df['position'].str.upper()
```
Verify in schema:
- Check that position field exists in Employee__dll
- Check data type is Text
- Verify you have read permissions
Check all DLOs:
- Validate all DLOs in read permissions
- Validate all DLOs in write permissions
- Check field names match exactly (case-sensitive)
- Verify data types are compatible with operations

Step 4c: Validation Checklist

Before proceeding to run, ensure:

All DLOs in config.json exist in target org
All field names used in code exist in DLO schemas
Field data types match your transformation logic
Primary key fields are correctly identified
Write target DLOs are created and accessible

Common Issues to Check:

Issue	Check	Fix
DLO doesn't exist	Use getting-datacloud-schema skill	Create DLO first or update code
Field name typo	Compare code vs. schema	Fix field name in entrypoint.py
Wrong data type	Check schema data type	Update transformation logic
Missing permissions	Check config.json	Re-run scan

Example Validation Workflow:

# 1. Check what DLOs are used
cat payload/config.json

# 2. Validate source DLO exists and get schema
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee__dll

# 3. Verify field 'position' exists in schema output
# Look for: name: position__c (or position)

# 4. Check target DLO exists
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee_Upper__dll

# 5. If all checks pass, proceed to run

Phase 5: Test Locally

After validating DLO schemas, run the code extension locally against your Data Cloud org.

Command:

sf data-code-extension script run --entrypoint <entrypoint_file> --target-org <org_alias> [options]

Options:

--target-org, -o - SF CLI org alias (required)
--config-file, -c - Custom config file path

Example:

# Run with default config (after schema validation)
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe

# Run with custom config
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o afvibe -c custom-config.json

What it does:

Executes transformation locally
Reads/writes data from/to actual Data Cloud org
Shows execution logs and errors
Tests logic before deployment

Monitor output for:

Data read/write operations
Transformation results
Errors or warnings
Execution time

If you get errors:

Re-validate DLO schemas
Check field names are exact matches
Verify data types are compatible
Review error messages for field/DLO issues

Phase 6: Deploy to Data Cloud

Deploy the code extension to Data Cloud for scheduled or on-demand execution.

CRITICAL: You MUST specify --package-dir ./payload to point to the payload directory created by init.

Command:

sf data-code-extension script deploy --target-org <org_alias> --name <name> --package-dir ./payload --package-version <version> --description <description> [options]

Required Options:

--target-org, -o - SF CLI org alias
--name, -n - Name for code extension deployment
--package-dir - Path to payload directory (REQUIRED - must be ./payload when running from project root)
--package-version - Version string (default: 0.0.1)
--description - Description of code extension

Optional Options:

--cpu-size - CPU size: CPU_L, CPU_XL, CPU_2XL (default), CPU_4XL
--function-invoke-opt - Function invoke options (for function type)
--network - Docker network (default: default)

Example from project root:

# Basic deployment (MUST include --package-dir ./payload)
sf data-code-extension script deploy \
  --target-org afvibe \
  --name Employee_Upper \
  --package-version 1.0.0 \
  --description "Uppercase employee positions" \
  --package-dir ./payload

# Full deployment with all options
sf data-code-extension script deploy \
  --target-org afvibe \
  --name Employee_Upper \
  --package-version 1.0.0 \
  --description "Uppercase employee positions" \
  --cpu-size CPU_4XL \
  --package-dir ./payload

Example with full path (if not in project root):

sf data-code-extension script deploy \
  --target-org afvibe \
  --name Employee_Upper \
  --package-version 1.0.0 \
  --description "Uppercase employee positions" \
  --package-dir /full/path/to/my-transform/payload

What it does:

Packages code with dependencies using Docker
Uploads to Data Cloud
Creates deployment record
Makes code available for execution in UI

After deployment:

Navigate to Data Cloud in Salesforce UI
Go to Data Transforms section
Find your deployment by name
Click "Run Now" to execute
Schedule for recurring execution

Complete Workflow Example

Here's a complete end-to-end example for creating an Employee uppercase transformation:

# 1. Create project directory (or use existing directory)
mkdir employee-transform && cd employee-transform

# 2. Initialize script project
sf data-code-extension script init --package-dir .

# 3. Edit payload/entrypoint.py (see code below)

# 4. Scan for permissions
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# 5. Validate DLO schemas (CRITICAL: check before testing)
# Check config.json for DLOs used
cat payload/config.json
# Validate Employee__dll exists and has 'position' field
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee__dll
# Validate Employee_Upper__dll exists (target DLO)
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee_Upper__dll

# 6. Test locally (after DLO validation passes)
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe

# 7. Deploy to Data Cloud (CRITICAL: include --package-dir ./payload)
sf data-code-extension script deploy \
  --target-org afvibe \
  --name Employee_Upper \
  --package-version 1.0.0 \
  --description "Uppercase employee positions" \
  --package-dir ./payload

entrypoint.py code:

from datacustomcode import Client

client = Client()

# Read employee data
employees = client.read_dlo('Employee__dll')

# Transform - uppercase position field
employees['position_upper'] = employees['position'].str.upper()

# Select relevant columns
output = employees[['id', 'name', 'position_upper']]

# Write to output DLO
client.write_to_dlo('Employee_Upper__dll', output, 'overwrite')

print(f"Processed {len(output)} employee records")

Error Handling

Common Issues and Solutions

1. SF CLI Plugin Not Found

Error: command data-code-extension not found

Solution:

sf plugins install @salesforce/plugin-data-codeextension

2. Python SDK Not Installed

Error: datacustomcode CLI not found

Solution:

pip install salesforce-data-customcode
datacustomcode version  # Verify

3. Wrong Python Version

Error: Python version mismatch

Solution:

python --version  # Should show 3.11.x
# Use pyenv to manage Python versions
pyenv install 3.11.0
pyenv local 3.11.0

4. Docker Not Running

Error: Cannot connect to Docker daemon

Solution:

Start Docker Desktop
Or start Docker service: sudo systemctl start docker

5. Org Not Authenticated

Error: No org found for alias 'afvibe'

Solution:

sf org login web --alias afvibe
sf org list  # Verify

6. Config.json Missing

Error: config.json not found

Solution:

sf data-code-extension scan ./payload/entrypoint.py

7. DLO Not Found During Run

Error: DLO 'Employee__dll' not found

Solution:

Verify DLO exists in org (use DLO Schema skill)
Check spelling and suffix (__dll)
Ensure proper read permissions in config.json

8. Permission Denied During Write

Error: Permission denied writing to 'Employee_Upper__dll'

Solution:

Run scan to update permissions: sf data-code-extension script scan ./payload/entrypoint.py
Verify target DLO exists and is writable
Check Data Cloud permissions in org

9. Deploy Fails - Wrong Directory Path

Error: Cannot find entrypoint.py or config.json
Error: No such file or directory: './entrypoint.py'
Error: Deploy failed - invalid path

Solution: CRITICAL: Ensure --package-dir argument points to the payload directory, not the project root.

# WRONG - Missing --package-dir argument
sf data-code-extension script deploy -o afvibe -n MyTransform

# WRONG - Pointing to project root instead of payload
sf data-code-extension script deploy -o afvibe -n MyTransform --package-dir .

# CORRECT - From project root, point to payload directory, includes package-version
sf data-code-extension script deploy -o afvibe -n MyTransform --package-dir ./payload --package-version 1.0.0 --description "Uppercase employee positions" 

# CORRECT - With full path
sf data-code-extension script deploy -o afvibe -n MyTransform --package-version 1.0.0 --package-dir /full/path/to/project/payload --description "Uppercase employee positions"

Check your current directory:

pwd  # Should be in project root
ls   # Should see 'payload' directory
ls payload/  # Should see entrypoint.py, config.json, requirements.txt

Best Practices

Development

Always scan before testing: Run scan after code changes
Test locally first: Use run command before deploying
Use version control: Git commit after each successful test
Version your deployments: Use semantic versioning (1.0.0, 1.1.0, etc.)
Deploy from project root with --package-dir ./payload: ALWAYS specify the payload directory explicitly in deploy commands

Code Organization

Keep entrypoint.py focused: Main transformation logic only
Extract helpers: Create separate modules for reusable functions
Add logging: Use print statements for debugging
Handle errors: Add try/except blocks for data operations

Performance

Choose appropriate CPU size:
- CPU_L: Small datasets (< 1M records)
- CPU_2XL: Medium datasets (1M-10M records)
- CPU_4XL: Large datasets (> 10M records)
Filter early: Read only needed columns/rows
Batch operations: Process data in chunks for large datasets

Security

No hardcoded credentials: Use SF CLI authentication only
Validate input data: Check for nulls and data types
Limit write permissions: Only grant necessary DLO/DMO access

Integration with Other Skills

Use with DLO Schema Skill (CRITICAL for validation):

The getting-datacloud-schema skill is required for validating DLOs before testing code extensions.

Workflow Integration:

1. List all DLOs: "Show me all DLOs in afvibe"
   → Verify target DLOs exist in org

2. Get DLO schema: "What's the schema for Employee__dll?"
   → Validate field names used in code

3. Check field types: Review schema output for data types
   → Ensure transformation logic is compatible

4. Validate before test: Use schema validation BEFORE running locally
   → Prevents runtime errors from missing fields/DLOs

5. Create code extension: "Create a code extension to read Employee__dll"
   → Design transformation based on actual schema

Example Integrated Workflow:

# Step 1: Check what DLOs exist
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe

# Step 2: Get schema for source DLO
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee__dll

# Step 3: Design code extension based on schema
# (Write entrypoint.py using fields from schema)

# Step 4: Scan for permissions
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py

# Step 5: Validate all DLOs referenced in config.json
cat payload/config.json
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee__dll
python3 ~/.a4drules/skills/getting-datacloud-schema/scripts/get_dlo_schema.py afvibe Employee_Upper__dll

# Step 6: Test locally (after validation passes)
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe

# Step 7: Deploy
sf data-code-extension script deploy --target-org afvibe --name Employee_Upper --package-dir ./payload --package-version 1.0.0 --description "Uppercase employee positions"

Use with Datakit Workflow:

1. Create DLO via code extension
2. Map DLO to DMO using datakit workflow
3. Use DMO in segments and activations

Output Interpretation

Scan Output

Scanning ./payload/entrypoint.py...
Found permissions:
  Read: Employee__dll
  Write: Employee_Upper__dll
Found dependencies:
  pandas==2.0.0
  numpy==1.24.0
Updated: payload/config.json
Updated: payload/requirements.txt

Run Output

Reading from Employee__dll...
Records read: 12
Transforming data...
Writing to Employee_Upper__dll...
Records written: 12
Execution completed in 2.3s

Deploy Output

Building Docker image...
Packaging dependencies...
Uploading to Data Cloud...
Deployment 'Employee_Upper' created successfully
ID: 2dgXXXXXXXXXXXXXXX
Version: 1.0.0
Status: ACTIVE

Advanced Usage

Custom Dependencies

Add to requirements.txt:

pandas==2.0.0
numpy==1.24.0
scikit-learn==1.3.0

Multiple DLO Operations

# Read from multiple DLOs
employees = client.read_dlo('Employee__dll')
departments = client.read_dlo('Department__dll')

# Join data
merged = employees.merge(departments, on='dept_id')

# Write to multiple outputs
client.write_to_dlo('Employee_Enriched__dll', merged, 'overwrite')
client.write_to_dmo('EmployeeDMO', merged, 'upsert')

Conditional Logic

# Read data
df = client.read_dlo('Employee__dll')

# Apply transformations based on conditions
df['grade'] = df['position'].apply(lambda x:
    'Senior' if 'Director' in x or 'VP' in x else 'Junior'
)

# Filter and write
senior_employees = df[df['grade'] == 'Senior']
client.write_to_dlo('Senior_Employees__dll', senior_employees, 'overwrite')

Command Reference

Command	Purpose	Required Args
`script init`	Create new script project	--package-dir
`function init`	Create new function project	--package-dir
`script scan`	Generate config	entrypoint file
`script run`	Test locally	entrypoint file, --target-org
`script deploy`	Deploy to Data Cloud	--target-org, --name, --package-dir, --package-version, --description

Resources

SF CLI Plugin: https://github.com/salesforcecli/plugin-data-code-extension
Python SDK: https://github.com/forcedotcom/datacloud-customcode-python-sdk
Data Cloud Docs: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm
Python SDK PyPI: https://pypi.org/project/salesforce-data-customcode/

Notes

Code extensions run in isolated Python 3.11 environment
Docker is required only for deployment, not for local testing
Use SF CLI authentication only (no separate credential files)
Scan command auto-detects permissions from code
Local run uses actual Data Cloud data (not mocked)
Deployments are versioned and can be rolled back in UI

22 KiB Raw Blame History

developing-datacloud-code-extension Skill

Overview

When to Use

Prerequisites Check

Skill Workflow

Phase 1: Initialize Project

Directory Context During Workflow

Phase 2: Develop Transformation

Phase 3: Scan for Permissions

Phase 4: Validate DLO Schema (Pre-Test Check)

Step 4a: Extract DLOs from config.json

Step 4b: Validate Each DLO Schema

Step 4c: Validation Checklist

Phase 5: Test Locally

Phase 6: Deploy to Data Cloud

Complete Workflow Example

Error Handling

Common Issues and Solutions

Best Practices

Development

Code Organization

Performance

Security

Integration with Other Skills

Output Interpretation

Scan Output

Run Output

Deploy Output

Advanced Usage

Custom Dependencies

Multiple DLO Operations

Conditional Logic

Command Reference

Resources

Notes

22 KiB

Raw Blame History