mirror of
https://github.com/forcedotcom/afv-library.git
synced 2026-07-30 03:09:50 +08:00
@W-22471646 : feat: add datacloud code extension and schema skills (#263)
feat: add datacloud code extension and schema skills
This commit is contained in:
parent
07665415cc
commit
3bf93c147d
321
skills/developing-datacloud-code-extension/SKILL.md
Normal file
321
skills/developing-datacloud-code-extension/SKILL.md
Normal file
@ -0,0 +1,321 @@
|
|||||||
|
---
|
||||||
|
name: developing-datacloud-code-extension
|
||||||
|
description: "Develop and deploy Data Cloud Code Extensions using SF CLI plugin. Use this skill when creating custom Python transformations for Data Cloud, deploying code extensions, or testing data transformations. Supports init, run, scan, and deploy operations."
|
||||||
|
metadata:
|
||||||
|
version: "1.0"
|
||||||
|
---
|
||||||
|
|
||||||
|
# developing-datacloud-code-extension Skill
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This skill provides a complete workflow for developing, testing, and deploying custom Python code extensions to Salesforce Data Cloud. Code extensions allow you to write Python transformations that read from and write to Data Lake Objects (DLOs) and Data Model Objects (DMOs).
|
||||||
|
|
||||||
|
## When to Use
|
||||||
|
|
||||||
|
- User wants to create a new code extension project
|
||||||
|
- User needs to test a code extension locally
|
||||||
|
- User wants to scan code for required permissions
|
||||||
|
- User needs to deploy a code extension to Data Cloud
|
||||||
|
- User is working with Data Cloud transformations
|
||||||
|
- User wants to read/write DLO or DMO data programmatically
|
||||||
|
|
||||||
|
## Prerequisites Check
|
||||||
|
|
||||||
|
Before executing any code extension commands, verify prerequisites:
|
||||||
|
|
||||||
|
1. **SF CLI with plugin installed**
|
||||||
|
```bash
|
||||||
|
sf plugins --core | grep data-code-extension
|
||||||
|
```
|
||||||
|
If not installed:
|
||||||
|
```bash
|
||||||
|
sf plugins install @salesforce/plugin-data-codeextension
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Python 3.11**
|
||||||
|
```bash
|
||||||
|
python --version # Should show 3.11.x
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Data Cloud Custom Code SDK**
|
||||||
|
```bash
|
||||||
|
pip list | grep salesforce-data-customcode
|
||||||
|
```
|
||||||
|
If not installed:
|
||||||
|
```bash
|
||||||
|
pip install salesforce-data-customcode
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Docker running** (for deploy only)
|
||||||
|
```bash
|
||||||
|
docker ps
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Authenticated org**
|
||||||
|
```bash
|
||||||
|
sf org display --target-org <org_alias> --json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Skill Workflow
|
||||||
|
|
||||||
|
### Phase 1: Initialize Project
|
||||||
|
|
||||||
|
Create a new code extension project with scaffolding.
|
||||||
|
|
||||||
|
**Commands:**
|
||||||
|
|
||||||
|
For **script-based** code extensions (batch transformations):
|
||||||
|
```bash
|
||||||
|
sf data-code-extension script init --package-dir <directory>
|
||||||
|
```
|
||||||
|
|
||||||
|
For **function-based** code extensions (real-time):
|
||||||
|
```bash
|
||||||
|
sf data-code-extension function init --package-dir <directory>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Required Option:**
|
||||||
|
- `--package-dir, -p` - Directory path where the package will be created
|
||||||
|
|
||||||
|
**What it creates:**
|
||||||
|
```
|
||||||
|
my-transform/ # Project root
|
||||||
|
├── payload/ # CRITICAL: This is what --package-dir must point to for deploy
|
||||||
|
│ ├── entrypoint.py # Main transformation code
|
||||||
|
│ └── config.json # Code extension configuration
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Directory Context During Workflow
|
||||||
|
|
||||||
|
**IMPORTANT:** Understanding the directory structure is critical for successful deployment.
|
||||||
|
|
||||||
|
**Commands and their directory requirements:**
|
||||||
|
|
||||||
|
| Command | Run From | Path/File Argument |
|
||||||
|
|---------|----------|-------------------|
|
||||||
|
| `init` | Parent directory | `<project-name>` or `.` |
|
||||||
|
| `scan` | Project root | `./payload/entrypoint.py` |
|
||||||
|
| `run` | Project root | `./payload/entrypoint.py` |
|
||||||
|
| `deploy` | Project root | `--package-dir ./payload` (**REQUIRED**) |
|
||||||
|
|
||||||
|
**CRITICAL: The `--package-dir` argument in deploy command MUST point to the `payload` directory, not the project root.**
|
||||||
|
|
||||||
|
### Phase 2: Develop Transformation
|
||||||
|
|
||||||
|
Edit `payload/entrypoint.py` with transformation logic.
|
||||||
|
|
||||||
|
**Script Example (Batch):**
|
||||||
|
```python
|
||||||
|
from datacustomcode import Client
|
||||||
|
|
||||||
|
client = Client()
|
||||||
|
|
||||||
|
# Read from DLO
|
||||||
|
df = client.read_dlo('Employee__dll')
|
||||||
|
|
||||||
|
# Transform data (uppercase position field)
|
||||||
|
df['position_upper'] = df['position'].str.upper()
|
||||||
|
|
||||||
|
# Write to output DLO
|
||||||
|
client.write_to_dlo('Employee_Upper__dll', df, 'overwrite')
|
||||||
|
```
|
||||||
|
|
||||||
|
**Function Example (Real-time):**
|
||||||
|
```python
|
||||||
|
from datacustomcode import FunctionClient
|
||||||
|
|
||||||
|
def transform(event, context):
|
||||||
|
client = FunctionClient(context)
|
||||||
|
input_data = event['data']
|
||||||
|
output = {
|
||||||
|
'name': input_data['name'].upper(),
|
||||||
|
'status': 'processed'
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
```
|
||||||
|
|
||||||
|
**Common Operations:**
|
||||||
|
- `client.read_dlo('DLO_Name__dll')` - Read from DLO
|
||||||
|
- `client.read_dmo('DMO_Name')` - Read from DMO
|
||||||
|
- `client.write_to_dlo('DLO_Name__dll', df, 'overwrite')` - Write to DLO
|
||||||
|
- `client.write_to_dmo('DMO_Name', df, 'upsert')` - Write to DMO
|
||||||
|
|
||||||
|
### Phase 3: Scan for Permissions
|
||||||
|
|
||||||
|
Scan the entrypoint file to detect required permissions and generate config.json.
|
||||||
|
|
||||||
|
**Command:**
|
||||||
|
```bash
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
```
|
||||||
|
|
||||||
|
**What it detects:**
|
||||||
|
- Read permissions for DLOs/DMOs
|
||||||
|
- Write permissions for DLOs/DMOs
|
||||||
|
- Python package dependencies
|
||||||
|
- Updates `config.json` and `requirements.txt`
|
||||||
|
|
||||||
|
### Phase 4: Validate DLO Schema (Pre-Test Check)
|
||||||
|
|
||||||
|
**CRITICAL: Before running tests locally, validate that all DLOs used in your code exist and have the expected fields.**
|
||||||
|
|
||||||
|
#### Step 4a: Extract DLOs from config.json
|
||||||
|
|
||||||
|
After scanning, review the generated `config.json` to identify all DLOs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat payload/config.json
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 4b: Validate Each DLO Schema
|
||||||
|
|
||||||
|
**Use the `getting-datacloud-schema` skill to verify DLOs exist and check field names.**
|
||||||
|
|
||||||
|
For each DLO referenced in your code:
|
||||||
|
|
||||||
|
1. **Verify DLO exists:**
|
||||||
|
```bash
|
||||||
|
python3 scripts/get_dlo_schema.py <org_alias> <dlo_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Verify field names match** — compare fields used in your `entrypoint.py` against the DLO schema.
|
||||||
|
|
||||||
|
3. **Check all DLOs:**
|
||||||
|
- Validate all DLOs in `read` permissions
|
||||||
|
- Validate all DLOs in `write` permissions
|
||||||
|
- Check field names match exactly (case-sensitive)
|
||||||
|
- Verify data types are compatible with operations
|
||||||
|
|
||||||
|
#### Step 4c: Validation Checklist
|
||||||
|
|
||||||
|
Before proceeding to run, ensure:
|
||||||
|
|
||||||
|
- [ ] All DLOs in config.json exist in target org
|
||||||
|
- [ ] All field names used in code exist in DLO schemas
|
||||||
|
- [ ] Field data types match your transformation logic
|
||||||
|
- [ ] Primary key fields are correctly identified
|
||||||
|
- [ ] Write target DLOs are created and accessible
|
||||||
|
|
||||||
|
### Phase 5: Test Locally
|
||||||
|
|
||||||
|
After validating DLO schemas, run the code extension locally against your Data Cloud org.
|
||||||
|
|
||||||
|
**Command:**
|
||||||
|
```bash
|
||||||
|
sf data-code-extension script run --entrypoint <entrypoint_file> --target-org <org_alias> [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Options:**
|
||||||
|
- `--target-org, -o` - SF CLI org alias (required)
|
||||||
|
- `--config-file, -c` - Custom config file path
|
||||||
|
|
||||||
|
**If you get errors:**
|
||||||
|
- Re-validate DLO schemas
|
||||||
|
- Check field names are exact matches
|
||||||
|
- Verify data types are compatible
|
||||||
|
- Review error messages for field/DLO issues
|
||||||
|
|
||||||
|
### Phase 6: Deploy to Data Cloud
|
||||||
|
|
||||||
|
Deploy the code extension to Data Cloud for scheduled or on-demand execution.
|
||||||
|
|
||||||
|
**CRITICAL: You MUST specify `--package-dir ./payload` to point to the payload directory created by init.**
|
||||||
|
|
||||||
|
**Command:**
|
||||||
|
```bash
|
||||||
|
sf data-code-extension script deploy --target-org <org_alias> --name <name> --package-dir ./payload --package-version <version> --description <description> [options]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Required Options:**
|
||||||
|
- `--target-org, -o` - SF CLI org alias
|
||||||
|
- `--name, -n` - Name for code extension deployment
|
||||||
|
- `--package-dir` - Path to payload directory (**REQUIRED** - must be `./payload` when running from project root)
|
||||||
|
- `--package-version` - Version string (default: 0.0.1)
|
||||||
|
- `--description` - Description of code extension
|
||||||
|
|
||||||
|
**Optional Options:**
|
||||||
|
- `--cpu-size` - CPU size: CPU_L, CPU_XL, CPU_2XL (default), CPU_4XL
|
||||||
|
- `--function-invoke-opt` - Function invoke options (for function type)
|
||||||
|
- `--network` - Docker network (default: default)
|
||||||
|
|
||||||
|
**After deployment:**
|
||||||
|
- Navigate to Data Cloud in Salesforce UI
|
||||||
|
- Go to Data Transforms section
|
||||||
|
- Find your deployment by name
|
||||||
|
- Click "Run Now" to execute
|
||||||
|
- Schedule for recurring execution
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Common Issues and Solutions
|
||||||
|
|
||||||
|
| Error | Solution |
|
||||||
|
|-------|----------|
|
||||||
|
| `command data-code-extension not found` | `sf plugins install @salesforce/plugin-data-codeextension` |
|
||||||
|
| `datacustomcode CLI not found` | `pip install salesforce-data-customcode` |
|
||||||
|
| `Python version mismatch` | Use pyenv: `pyenv install 3.11.0 && pyenv local 3.11.0` |
|
||||||
|
| `Cannot connect to Docker daemon` | Start Docker Desktop |
|
||||||
|
| `No org found for alias` | `sf org login web --alias <org_alias>` |
|
||||||
|
| `config.json not found` | `sf data-code-extension script scan --entrypoint ./payload/entrypoint.py` |
|
||||||
|
| `DLO not found` | Verify DLO exists (use getting-datacloud-schema skill), check spelling and `__dll` suffix |
|
||||||
|
| `Permission denied writing` | Re-run scan, verify target DLO exists and is writable |
|
||||||
|
| `Deploy fails - wrong directory` | Ensure `--package-dir` points to `payload/` directory, not project root |
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### Development
|
||||||
|
1. Always scan before testing — run scan after code changes
|
||||||
|
2. Test locally first — use `run` command before deploying
|
||||||
|
3. Use version control — git commit after each successful test
|
||||||
|
4. Version your deployments — use semantic versioning (1.0.0, 1.1.0, etc.)
|
||||||
|
5. Deploy from project root with `--package-dir ./payload`
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
- **CPU_L**: Small datasets (< 1M records)
|
||||||
|
- **CPU_2XL**: Medium datasets (1M-10M records)
|
||||||
|
- **CPU_4XL**: Large datasets (> 10M records)
|
||||||
|
|
||||||
|
### Security
|
||||||
|
1. No hardcoded credentials — use SF CLI authentication only
|
||||||
|
2. Validate input data — check for nulls and data types
|
||||||
|
3. Limit write permissions — only grant necessary DLO/DMO access
|
||||||
|
|
||||||
|
## Integration with Other Skills
|
||||||
|
|
||||||
|
**Use with getting-datacloud-schema skill (CRITICAL for validation):**
|
||||||
|
|
||||||
|
The `getting-datacloud-schema` skill is **required** for validating DLOs before testing code extensions.
|
||||||
|
|
||||||
|
**Use with Datakit Workflow:**
|
||||||
|
1. Create DLO via code extension
|
||||||
|
2. Map DLO to DMO using datakit workflow
|
||||||
|
3. Use DMO in segments and activations
|
||||||
|
|
||||||
|
## Command Reference
|
||||||
|
|
||||||
|
| Command | Purpose | Required Args |
|
||||||
|
|---------|---------|---------------|
|
||||||
|
| `script init` | Create new script project | --package-dir |
|
||||||
|
| `function init` | Create new function project | --package-dir |
|
||||||
|
| `script scan` | Generate config | entrypoint file |
|
||||||
|
| `script run` | Test locally | entrypoint file, --target-org |
|
||||||
|
| `script deploy` | Deploy to Data Cloud | --target-org, --name, --package-dir, --package-version, --description |
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
- SF CLI Plugin: https://github.com/salesforcecli/plugin-data-code-extension
|
||||||
|
- Python SDK: https://github.com/forcedotcom/datacloud-customcode-python-sdk
|
||||||
|
- Data Cloud Docs: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm
|
||||||
|
- Python SDK PyPI: https://pypi.org/project/salesforce-data-customcode/
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Code extensions run in isolated Python 3.11 environment
|
||||||
|
- Docker is required only for deployment, not for local testing
|
||||||
|
- Use SF CLI authentication only (no separate credential files)
|
||||||
|
- Scan command auto-detects permissions from code
|
||||||
|
- Local run uses actual Data Cloud data (not mocked)
|
||||||
|
- Deployments are versioned and can be rolled back in UI
|
||||||
193
skills/developing-datacloud-code-extension/references/README.md
Normal file
193
skills/developing-datacloud-code-extension/references/README.md
Normal file
@ -0,0 +1,193 @@
|
|||||||
|
# developing-datacloud-code-extension Skill
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
A skill that provides a complete workflow for developing, testing, and deploying custom Python code extensions to Salesforce Data Cloud using the SF CLI plugin.
|
||||||
|
|
||||||
|
## What It Does
|
||||||
|
|
||||||
|
This skill helps you create Data Cloud Code Extensions through a complete workflow:
|
||||||
|
|
||||||
|
1. **Init** - Create new code extension project with scaffolding
|
||||||
|
2. **Develop** - Write Python transformation logic
|
||||||
|
3. **Scan** - Auto-detect permissions and generate config
|
||||||
|
4. **Run** - Test locally against Data Cloud org
|
||||||
|
5. **Deploy** - Package and deploy to Data Cloud
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
**Initialize a project:**
|
||||||
|
```
|
||||||
|
"Create a new Data Cloud code extension project called employee-transform"
|
||||||
|
"Initialize a code extension to transform employee data"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test locally:**
|
||||||
|
```
|
||||||
|
"Run the code extension in my-transform directory against afvibe org"
|
||||||
|
"Test the entrypoint.py file locally"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Scan for permissions:**
|
||||||
|
```
|
||||||
|
"Scan the entrypoint.py to generate config"
|
||||||
|
"Update permissions in config.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Deploy:**
|
||||||
|
```
|
||||||
|
"Deploy Employee_Upper code extension to afvibe"
|
||||||
|
"Deploy this transform with package-version 1.0.0"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Direct Command Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Initialize project
|
||||||
|
sf data-code-extension script init --package-dir <directory>
|
||||||
|
|
||||||
|
# Scan for permissions
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
|
||||||
|
# Test locally
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org <org_alias>
|
||||||
|
|
||||||
|
# Deploy
|
||||||
|
sf data-code-extension script deploy --target-org <org_alias> --name <name> --package-version <version> --description <description> --package-dir ./payload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
1. **SF CLI with Plugin**
|
||||||
|
```bash
|
||||||
|
sf plugins install @salesforce/plugin-data-codeextension
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Python 3.11**
|
||||||
|
```bash
|
||||||
|
python --version # Must be 3.11.x
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Data Cloud Custom Code SDK**
|
||||||
|
```bash
|
||||||
|
pip install salesforce-data-customcode
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Docker** (for deploy only)
|
||||||
|
- Docker Desktop or equivalent
|
||||||
|
|
||||||
|
5. **Authenticated Org**
|
||||||
|
```bash
|
||||||
|
sf org login web --alias <org_alias>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Complete End-to-End Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Create project
|
||||||
|
mkdir employee-transform && cd employee-transform
|
||||||
|
sf data-code-extension script init --package-dir .
|
||||||
|
|
||||||
|
# 2. Edit payload/entrypoint.py with your transformation
|
||||||
|
|
||||||
|
# 3. Scan for permissions
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
|
||||||
|
# 4. Test locally
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe
|
||||||
|
|
||||||
|
# 5. Deploy (MUST include --package-dir ./payload)
|
||||||
|
sf data-code-extension script deploy \
|
||||||
|
--target-org afvibe \
|
||||||
|
--name Employee_Upper \
|
||||||
|
--package-version 1.0.0 \
|
||||||
|
--description "Uppercase employee positions" \
|
||||||
|
--package-dir ./payload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Transformation
|
||||||
|
|
||||||
|
**Read from DLO, transform, write to DLO:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
from datacustomcode import Client
|
||||||
|
|
||||||
|
client = Client()
|
||||||
|
|
||||||
|
# Read employee data from DLO
|
||||||
|
employees = client.read_dlo('Employee__dll')
|
||||||
|
|
||||||
|
# Transform - uppercase position field
|
||||||
|
employees['position_upper'] = employees['position'].str.upper()
|
||||||
|
|
||||||
|
# Select output columns
|
||||||
|
output = employees[['id', 'name', 'position_upper']]
|
||||||
|
|
||||||
|
# Write to output DLO
|
||||||
|
client.write_to_dlo('Employee_Upper__dll', output, 'overwrite')
|
||||||
|
|
||||||
|
print(f"Processed {len(output)} employee records")
|
||||||
|
```
|
||||||
|
|
||||||
|
## Project Structure
|
||||||
|
|
||||||
|
After `init`, you'll have:
|
||||||
|
|
||||||
|
```
|
||||||
|
my-transform/
|
||||||
|
├── payload/
|
||||||
|
│ ├── entrypoint.py # Your transformation code
|
||||||
|
│ └── config.json # Permissions and configuration
|
||||||
|
├── requirements.txt # Python dependencies
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Operations
|
||||||
|
|
||||||
|
### Read/Write DLOs
|
||||||
|
```python
|
||||||
|
# Read
|
||||||
|
df = client.read_dlo('Employee__dll')
|
||||||
|
|
||||||
|
# Write (modes: 'overwrite', 'append')
|
||||||
|
client.write_to_dlo('Employee_Upper__dll', df, 'overwrite')
|
||||||
|
```
|
||||||
|
|
||||||
|
### Read/Write DMOs
|
||||||
|
```python
|
||||||
|
# Read
|
||||||
|
df = client.read_dmo('EmployeeDMO')
|
||||||
|
|
||||||
|
# Write (modes: 'upsert', 'insert')
|
||||||
|
client.write_to_dmo('EmployeeDMO', df, 'upsert')
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Error | Quick Fix |
|
||||||
|
|-------|-----------|
|
||||||
|
| Plugin not found | `sf plugins install @salesforce/plugin-data-codeextension` |
|
||||||
|
| Python SDK missing | `pip install salesforce-data-customcode` |
|
||||||
|
| Wrong Python version | Use pyenv to install 3.11.0 |
|
||||||
|
| Org not connected | `sf org login web --alias <alias>` |
|
||||||
|
| Config missing | Run scan command |
|
||||||
|
| DLO not found | Check DLO name, use getting-datacloud-schema skill |
|
||||||
|
| Docker error | Start Docker Desktop |
|
||||||
|
|
||||||
|
## CPU Size Selection
|
||||||
|
|
||||||
|
| CPU Size | Use Case | Data Volume |
|
||||||
|
|----------|----------|-------------|
|
||||||
|
| CPU_L | Small datasets | < 1M records |
|
||||||
|
| CPU_XL | Medium datasets | 1M-5M records |
|
||||||
|
| CPU_2XL | Large datasets (default) | 5M-10M records |
|
||||||
|
| CPU_4XL | Very large datasets | > 10M records |
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
- **SF CLI Plugin**: https://github.com/salesforcecli/plugin-data-code-extension
|
||||||
|
- **Python SDK**: https://github.com/forcedotcom/datacloud-customcode-python-sdk
|
||||||
|
- **Data Cloud Docs**: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm
|
||||||
|
- **SDK on PyPI**: https://pypi.org/project/salesforce-data-customcode/
|
||||||
@ -0,0 +1,269 @@
|
|||||||
|
# Data Cloud Code Extension - Quick Reference
|
||||||
|
|
||||||
|
## Command Cheat Sheet
|
||||||
|
|
||||||
|
### Initialize Project
|
||||||
|
```bash
|
||||||
|
# Create script project
|
||||||
|
sf data-code-extension script init --package-dir <directory>
|
||||||
|
|
||||||
|
# Create function project
|
||||||
|
sf data-code-extension function init --package-dir <directory>
|
||||||
|
|
||||||
|
# Examples
|
||||||
|
sf data-code-extension script init --package-dir .
|
||||||
|
sf data-code-extension script init --package-dir my-transform
|
||||||
|
```
|
||||||
|
|
||||||
|
### Scan for Permissions
|
||||||
|
```bash
|
||||||
|
# Basic scan
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
|
||||||
|
# Preview without saving
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py --dry-run
|
||||||
|
|
||||||
|
# Custom config location
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py --config ./custom-config.json
|
||||||
|
|
||||||
|
# Skip requirements.txt
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py --no-requirements
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run Locally
|
||||||
|
```bash
|
||||||
|
# Basic run
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org <org_alias>
|
||||||
|
|
||||||
|
# With custom config
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o <org_alias> -c custom-config.json
|
||||||
|
|
||||||
|
# Examples
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o afvibe
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deploy
|
||||||
|
```bash
|
||||||
|
# Minimal deployment (MUST include --package-dir ./payload)
|
||||||
|
sf data-code-extension script deploy \
|
||||||
|
--target-org <org_alias> \
|
||||||
|
--name <name> \
|
||||||
|
--package-version <version> \
|
||||||
|
--description "<description>" \
|
||||||
|
--package-dir ./payload
|
||||||
|
|
||||||
|
# Full options
|
||||||
|
sf data-code-extension script deploy \
|
||||||
|
--target-org <org_alias> \
|
||||||
|
--name <name> \
|
||||||
|
--package-version <version> \
|
||||||
|
--description "<description>" \
|
||||||
|
--cpu-size <CPU_L|CPU_XL|CPU_2XL|CPU_4XL> \
|
||||||
|
--package-dir ./payload
|
||||||
|
|
||||||
|
# Examples (CRITICAL: Always include --package-dir ./payload)
|
||||||
|
sf data-code-extension script deploy \
|
||||||
|
--target-org afvibe \
|
||||||
|
--name Employee_Upper \
|
||||||
|
--package-version 1.0.0 \
|
||||||
|
--description "Uppercase employee positions" \
|
||||||
|
--package-dir ./payload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Workflows
|
||||||
|
|
||||||
|
### New Project from Scratch
|
||||||
|
```bash
|
||||||
|
# 1. Create directory
|
||||||
|
mkdir my-transform && cd my-transform
|
||||||
|
|
||||||
|
# 2. Initialize
|
||||||
|
sf data-code-extension script init --package-dir .
|
||||||
|
|
||||||
|
# 3. Edit payload/entrypoint.py with your transformation
|
||||||
|
|
||||||
|
# 4. Scan
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
|
||||||
|
# 5. Test
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py --target-org afvibe
|
||||||
|
|
||||||
|
# 6. Deploy (MUST include --package-dir ./payload)
|
||||||
|
sf data-code-extension script deploy \
|
||||||
|
--target-org afvibe \
|
||||||
|
--name MyTransform \
|
||||||
|
--package-version 1.0.0 \
|
||||||
|
--description "My transformation" \
|
||||||
|
--package-dir ./payload
|
||||||
|
```
|
||||||
|
|
||||||
|
### Update Existing Code Extension
|
||||||
|
```bash
|
||||||
|
# 1. Edit payload/entrypoint.py
|
||||||
|
|
||||||
|
# 2. Re-scan
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
|
||||||
|
# 3. Test
|
||||||
|
sf data-code-extension script run --entrypoint ./payload/entrypoint.py -o afvibe
|
||||||
|
|
||||||
|
# 4. Deploy with new version (include --package-dir ./payload)
|
||||||
|
sf data-code-extension script deploy \
|
||||||
|
-o afvibe \
|
||||||
|
-n MyTransform \
|
||||||
|
--package-version 1.1.0 \
|
||||||
|
--description "Updated transformation" \
|
||||||
|
--package-dir ./payload
|
||||||
|
```
|
||||||
|
|
||||||
|
## Python Code Patterns
|
||||||
|
|
||||||
|
### Read/Write DLO
|
||||||
|
```python
|
||||||
|
from datacustomcode import Client
|
||||||
|
|
||||||
|
client = Client()
|
||||||
|
|
||||||
|
# Read
|
||||||
|
df = client.read_dlo('Employee__dll')
|
||||||
|
|
||||||
|
# Transform
|
||||||
|
df['new_field'] = df['old_field'].str.upper()
|
||||||
|
|
||||||
|
# Write (modes: 'overwrite', 'append')
|
||||||
|
client.write_to_dlo('Output__dll', df, 'overwrite')
|
||||||
|
```
|
||||||
|
|
||||||
|
### Read/Write DMO
|
||||||
|
```python
|
||||||
|
# Read
|
||||||
|
df = client.read_dmo('EmployeeDMO')
|
||||||
|
|
||||||
|
# Write (modes: 'upsert', 'insert')
|
||||||
|
client.write_to_dmo('EmployeeDMO', df, 'upsert')
|
||||||
|
```
|
||||||
|
|
||||||
|
### Multiple DLO Operations
|
||||||
|
```python
|
||||||
|
# Read multiple
|
||||||
|
employees = client.read_dlo('Employee__dll')
|
||||||
|
departments = client.read_dlo('Department__dll')
|
||||||
|
|
||||||
|
# Join
|
||||||
|
merged = employees.merge(departments, on='dept_id')
|
||||||
|
|
||||||
|
# Write multiple
|
||||||
|
client.write_to_dlo('Enriched__dll', merged, 'overwrite')
|
||||||
|
client.write_to_dmo('EmployeeDMO', merged, 'upsert')
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Transformations
|
||||||
|
```python
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
# Filter
|
||||||
|
active = df[df['status'] == 'Active']
|
||||||
|
|
||||||
|
# Computed column
|
||||||
|
df['full_name'] = df['first'] + ' ' + df['last']
|
||||||
|
|
||||||
|
# Aggregate
|
||||||
|
summary = df.groupby('dept')['salary'].mean()
|
||||||
|
|
||||||
|
# Conditional
|
||||||
|
df['grade'] = df['position'].apply(
|
||||||
|
lambda x: 'Senior' if 'VP' in x else 'Junior'
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Option Reference
|
||||||
|
|
||||||
|
### --cpu-size
|
||||||
|
- `CPU_L` - Small datasets (< 1M records)
|
||||||
|
- `CPU_XL` - Medium datasets (1M-5M)
|
||||||
|
- `CPU_2XL` - Large datasets (5M-10M) **[default]**
|
||||||
|
- `CPU_4XL` - Very large (> 10M records)
|
||||||
|
|
||||||
|
### Write Modes
|
||||||
|
- `overwrite` - Replace all data
|
||||||
|
- `append` - Add to existing data
|
||||||
|
- `upsert` - Update or insert (DMO only)
|
||||||
|
- `insert` - Insert only (DMO only)
|
||||||
|
|
||||||
|
## Troubleshooting Quick Fixes
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Plugin not found
|
||||||
|
sf plugins install @salesforce/plugin-data-codeextension
|
||||||
|
|
||||||
|
# Python SDK missing
|
||||||
|
pip install salesforce-data-customcode
|
||||||
|
|
||||||
|
# Verify Python version (must be 3.11.x)
|
||||||
|
python --version
|
||||||
|
|
||||||
|
# Org not connected
|
||||||
|
sf org login web --alias <org_alias>
|
||||||
|
|
||||||
|
# Config missing
|
||||||
|
sf data-code-extension script scan --entrypoint ./payload/entrypoint.py
|
||||||
|
|
||||||
|
# Docker not running (for deploy)
|
||||||
|
# Start Docker Desktop
|
||||||
|
```
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
my-project/
|
||||||
|
├── payload/
|
||||||
|
│ ├── entrypoint.py # Main code
|
||||||
|
│ └── config.json # Auto-generated permissions
|
||||||
|
├── requirements.txt # Auto-generated dependencies
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## config.json Format
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"version": "1.0",
|
||||||
|
"permissions": {
|
||||||
|
"read": ["Employee__dll", "Department__dll"],
|
||||||
|
"write": ["Enriched__dll"]
|
||||||
|
},
|
||||||
|
"resources": {
|
||||||
|
"cpu_size": "CPU_2XL"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common Errors
|
||||||
|
|
||||||
|
| Error | Quick Fix |
|
||||||
|
|-------|-----------|
|
||||||
|
| Plugin not found | `sf plugins install @salesforce/plugin-data-codeextension` |
|
||||||
|
| Python SDK missing | `pip install salesforce-data-customcode` |
|
||||||
|
| Wrong Python version | Use pyenv to install 3.11.0 |
|
||||||
|
| Org not connected | `sf org login web --alias <alias>` |
|
||||||
|
| Config missing | Run scan command |
|
||||||
|
| DLO not found | Check DLO name, use getting-datacloud-schema skill |
|
||||||
|
| Docker error | Start Docker Desktop |
|
||||||
|
|
||||||
|
## Deployment Checklist
|
||||||
|
|
||||||
|
- [ ] Code written in entrypoint.py
|
||||||
|
- [ ] Scanned for permissions
|
||||||
|
- [ ] Tested locally
|
||||||
|
- [ ] Version number decided
|
||||||
|
- [ ] Description added
|
||||||
|
- [ ] CPU size chosen
|
||||||
|
- [ ] Docker running
|
||||||
|
- [ ] Org authenticated
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
- Plugin: https://github.com/salesforcecli/plugin-data-code-extension
|
||||||
|
- Python SDK: https://github.com/forcedotcom/datacloud-customcode-python-sdk
|
||||||
|
- Data Cloud Docs: https://help.salesforce.com/s/articleView?id=sf.c360_a_intro.htm
|
||||||
380
skills/getting-datacloud-schema/SKILL.md
Normal file
380
skills/getting-datacloud-schema/SKILL.md
Normal file
@ -0,0 +1,380 @@
|
|||||||
|
---
|
||||||
|
name: getting-datacloud-schema
|
||||||
|
description: "Retrieve Data Lake Object (DLO) and Data Model Object (DMO) schema information from Salesforce Data Cloud using REST APIs. Use this skill when you need to inspect DLO or DMO field definitions, data types, or metadata. Takes org alias and optional DLO/DMO name as parameters."
|
||||||
|
metadata:
|
||||||
|
version: "1.0"
|
||||||
|
---
|
||||||
|
|
||||||
|
# getting-datacloud-schema Skill
|
||||||
|
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This skill retrieves Data Lake Object (DLO) and Data Model Object (DMO) schema information from Salesforce Data Cloud using the SSOT REST API. It can list all DLOs or DMOs in an org, or retrieve detailed schema for a specific DLO or DMO.
|
||||||
|
|
||||||
|
## When to Use
|
||||||
|
|
||||||
|
- User wants to see all DLOs or DMOs in a Data Cloud org
|
||||||
|
- User needs field schema for a specific DLO or DMO
|
||||||
|
- User is exploring Data Cloud data structures
|
||||||
|
- User needs to understand DLO or DMO field types and metadata
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- SF CLI installed and authenticated to target org
|
||||||
|
- Org has Data Cloud enabled
|
||||||
|
- User has appropriate Data Cloud permissions
|
||||||
|
|
||||||
|
## Skill Execution
|
||||||
|
|
||||||
|
### Parameters
|
||||||
|
|
||||||
|
1. **org_alias** (required): The SF CLI org alias (e.g., 'afvibe', 'myorg')
|
||||||
|
2. **dlo_name** (optional): Specific DLO developer name (e.g., 'Employee__dll')
|
||||||
|
3. **dmo_name** (optional): Specific DMO developer name (e.g., 'Individual__dlm')
|
||||||
|
|
||||||
|
### Step 1: Discover Connected Org
|
||||||
|
|
||||||
|
First, run `sf org list` to find out which org is connected and extract the alias to use for all subsequent calls:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sf org list
|
||||||
|
```
|
||||||
|
|
||||||
|
Example output:
|
||||||
|
```
|
||||||
|
┌────┬───────┬──────────────────────────┬────────────────────┬───────────┐
|
||||||
|
│ │ Alias │ Username │ Org Id │ Status │
|
||||||
|
├────┼───────┼──────────────────────────┼────────────────────┼───────────┤
|
||||||
|
│ 🍁 │ myorg │ chandresh@afvidedemo.org │ 00DKZ00000b80NT2AY │ Connected │
|
||||||
|
└────┴───────┴──────────────────────────┴────────────────────┴───────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
Extract the **Alias** value (e.g., `myorg`) from the output and use it as the `<org_alias>` for all subsequent calls. Use `--all` to see expired and deleted scratch orgs as well.
|
||||||
|
|
||||||
|
### Step 2: Validate SF CLI Authentication
|
||||||
|
|
||||||
|
Before making API calls, verify the org is connected:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sf org display --target-org <org_alias> --json
|
||||||
|
```
|
||||||
|
|
||||||
|
If not connected, inform user to run:
|
||||||
|
```bash
|
||||||
|
sf org login web --alias <org_alias>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3a: Execute DLO Schema Script
|
||||||
|
|
||||||
|
The Python scripts are bundled with this skill. They live in the `scripts/` subdirectory of the same directory that contains this SKILL.md file. Use the absolute path to that directory — do NOT use `./scripts/` as that resolves relative to the current working directory, not the skill directory.
|
||||||
|
|
||||||
|
**To list all DLOs:**
|
||||||
|
```bash
|
||||||
|
python3 <skill_dir>/scripts/get_dlo_schema.py <org_alias>
|
||||||
|
```
|
||||||
|
|
||||||
|
**To get specific DLO schema:**
|
||||||
|
```bash
|
||||||
|
python3 <skill_dir>/scripts/get_dlo_schema.py <org_alias> <dlo_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3b: Execute DMO Schema Script
|
||||||
|
|
||||||
|
**To list all DMOs:**
|
||||||
|
```bash
|
||||||
|
python3 <skill_dir>/scripts/get_dmo_schema.py <org_alias>
|
||||||
|
```
|
||||||
|
|
||||||
|
**To get specific DMO schema:**
|
||||||
|
```bash
|
||||||
|
python3 <skill_dir>/scripts/get_dmo_schema.py <org_alias> <dmo_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Present Results
|
||||||
|
|
||||||
|
Parse and present the results in a user-friendly format:
|
||||||
|
|
||||||
|
**For DLO List:**
|
||||||
|
- Show DLO name, label, category, and ID
|
||||||
|
- Indicate total count
|
||||||
|
- Highlight DLOs with data (totalRecords > 0)
|
||||||
|
|
||||||
|
**For DLO Schema:**
|
||||||
|
- Show basic info (name, label, category, status)
|
||||||
|
- List all fields with:
|
||||||
|
- Field name
|
||||||
|
- Data type
|
||||||
|
- Primary key indicator
|
||||||
|
- Nullable status
|
||||||
|
- Highlight custom fields (exclude system fields like DataSource__c, cdp_sys_*)
|
||||||
|
- Show record count if available
|
||||||
|
|
||||||
|
**For DMO List:**
|
||||||
|
- Show DMO name, label, category, and ID
|
||||||
|
- Indicate total count
|
||||||
|
|
||||||
|
**For DMO Schema:**
|
||||||
|
- Show basic info (name, label, category, description)
|
||||||
|
- List all fields with:
|
||||||
|
- Field name
|
||||||
|
- Data type
|
||||||
|
- Primary key indicator
|
||||||
|
- Nullable status
|
||||||
|
- Show dataspace information if available
|
||||||
|
|
||||||
|
### Step 5: Offer Next Steps
|
||||||
|
|
||||||
|
After displaying results, suggest relevant follow-up actions:
|
||||||
|
- Query data from the DLO
|
||||||
|
- Create calculated insights
|
||||||
|
- Build segments
|
||||||
|
- Set up data streams
|
||||||
|
- Create DMO mappings
|
||||||
|
|
||||||
|
## API Endpoints Used
|
||||||
|
|
||||||
|
### List All DLOs
|
||||||
|
```
|
||||||
|
GET /services/data/v64.0/ssot/data-lake-objects
|
||||||
|
```
|
||||||
|
|
||||||
|
Response structure:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dataLakeObjects": [
|
||||||
|
{
|
||||||
|
"name": "Employee__dll",
|
||||||
|
"label": "Employee",
|
||||||
|
"category": "Profile",
|
||||||
|
"id": "1dlXXXXXXXXXXXXXXX",
|
||||||
|
"status": "ACTIVE",
|
||||||
|
"totalRecords": 12,
|
||||||
|
"fields": [...]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"totalSize": 5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get DLO Schema
|
||||||
|
```
|
||||||
|
GET /services/data/v64.0/ssot/data-lake-objects/{dlo_name}
|
||||||
|
```
|
||||||
|
|
||||||
|
Response structure (same as individual object in list response, but wrapped in paginated format).
|
||||||
|
|
||||||
|
### List All DMOs
|
||||||
|
```
|
||||||
|
GET /services/data/v64.0/ssot/data-model-objects
|
||||||
|
```
|
||||||
|
|
||||||
|
Response structure:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"dataModelObjects": [
|
||||||
|
{
|
||||||
|
"name": "Individual__dlm",
|
||||||
|
"label": "Individual",
|
||||||
|
"category": "Profile",
|
||||||
|
"id": "0dmXXXXXXXXXXXXXXX",
|
||||||
|
"fields": [...]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"totalSize": 10
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get DMO Schema
|
||||||
|
```
|
||||||
|
GET /services/data/v64.0/ssot/data-model-objects/{dmo_name}
|
||||||
|
```
|
||||||
|
|
||||||
|
Response structure (same as individual object in list response, but wrapped in paginated format).
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
**Common Issues:**
|
||||||
|
|
||||||
|
1. **Org not connected**
|
||||||
|
- Message: "Org not connected"
|
||||||
|
- Solution: Ask user to authenticate via SF CLI
|
||||||
|
|
||||||
|
2. **DLO not found**
|
||||||
|
- Message: "DLO 'XYZ__dll' not found"
|
||||||
|
- Solution: List all DLOs first to verify name
|
||||||
|
|
||||||
|
5. **DMO not found**
|
||||||
|
- Message: "DMO 'XYZ__dlm' not found"
|
||||||
|
- Solution: List all DMOs first to verify name
|
||||||
|
|
||||||
|
3. **Permission issues**
|
||||||
|
- Message: HTTP 403 errors
|
||||||
|
- Solution: Verify user has Data Cloud permissions
|
||||||
|
|
||||||
|
4. **API version mismatch**
|
||||||
|
- Current: v64.0
|
||||||
|
- Solution: Script can be updated for newer API versions
|
||||||
|
|
||||||
|
## Example Usage
|
||||||
|
|
||||||
|
**Example 1: List all DLOs**
|
||||||
|
```
|
||||||
|
User: "Show me all DLOs in afvibe org"
|
||||||
|
|
||||||
|
Response:
|
||||||
|
1. Run sf org list to discover connected org alias
|
||||||
|
2. Authenticate to afvibe
|
||||||
|
3. Run: python3 <skill_dir>/scripts/get_dlo_schema.py afvibe
|
||||||
|
4. Display formatted list of DLOs
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example 2: Get specific DLO schema**
|
||||||
|
```
|
||||||
|
User: "Get the schema for Employee__dll in afvibe"
|
||||||
|
|
||||||
|
Response:
|
||||||
|
1. Run sf org list to discover connected org alias
|
||||||
|
2. Authenticate to afvibe
|
||||||
|
3. Run: python3 <skill_dir>/scripts/get_dlo_schema.py afvibe Employee__dll
|
||||||
|
4. Display field schema with types and metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example 3: Explore DLOs then get schema**
|
||||||
|
```
|
||||||
|
User: "What DLOs exist in myorg and show me the schema for the Employee one"
|
||||||
|
|
||||||
|
Response:
|
||||||
|
1. Run sf org list to discover connected org alias
|
||||||
|
2. List all DLOs in myorg
|
||||||
|
3. Identify Employee__dll
|
||||||
|
4. Get detailed schema for Employee__dll
|
||||||
|
5. Present both results
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example 4: List all DMOs**
|
||||||
|
```
|
||||||
|
User: "Show me all DMOs in afvibe org"
|
||||||
|
|
||||||
|
Response:
|
||||||
|
1. Run sf org list to discover connected org alias
|
||||||
|
2. Authenticate to afvibe
|
||||||
|
3. Run: python3 <skill_dir>/scripts/get_dmo_schema.py afvibe
|
||||||
|
4. Display formatted list of DMOs
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example 5: Get specific DMO schema**
|
||||||
|
```
|
||||||
|
User: "Get the schema for Individual__dlm in afvibe"
|
||||||
|
|
||||||
|
Response:
|
||||||
|
1. Run sf org list to discover connected org alias
|
||||||
|
2. Authenticate to afvibe
|
||||||
|
3. Run: python3 <skill_dir>/scripts/get_dmo_schema.py afvibe Individual__dlm
|
||||||
|
4. Display field schema with types and metadata
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example 6: Explore DMOs then get schema**
|
||||||
|
```
|
||||||
|
User: "What DMOs exist in myorg and show me the schema for the Individual one"
|
||||||
|
|
||||||
|
Response:
|
||||||
|
1. Run sf org list to discover connected org alias
|
||||||
|
2. List all DMOs in myorg
|
||||||
|
3. Identify Individual__dlm
|
||||||
|
4. Get detailed schema for Individual__dlm
|
||||||
|
5. Present both results
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
### DLO List Output
|
||||||
|
```
|
||||||
|
Found 5 DLOs in org 'afvibe':
|
||||||
|
|
||||||
|
1. DataCustomCodeLogs__dll
|
||||||
|
Label: DataCustomCodeLogs
|
||||||
|
Category: Engagement
|
||||||
|
Records: 233
|
||||||
|
|
||||||
|
2. Employee__dll
|
||||||
|
Label: Employee
|
||||||
|
Category: Profile
|
||||||
|
Records: 12
|
||||||
|
|
||||||
|
[...]
|
||||||
|
```
|
||||||
|
|
||||||
|
### DLO Schema Output
|
||||||
|
```
|
||||||
|
DLO: Employee__dll
|
||||||
|
Label: Employee
|
||||||
|
Category: Profile
|
||||||
|
Status: ACTIVE
|
||||||
|
Records: 12
|
||||||
|
|
||||||
|
Custom Fields:
|
||||||
|
• id__c (Text) - Primary Key
|
||||||
|
• name__c (Text)
|
||||||
|
• position__c (Text)
|
||||||
|
• manager_id__c (Number)
|
||||||
|
|
||||||
|
System Fields:
|
||||||
|
• DataSource__c (Text)
|
||||||
|
• InternalOrganization__c (Text)
|
||||||
|
• cdp_sys_SourceVersion__c (Text)
|
||||||
|
|
||||||
|
Next steps:
|
||||||
|
- Query data: SELECT * FROM Employee__dll LIMIT 10
|
||||||
|
- Create segment based on position field
|
||||||
|
- Set up data stream for real-time updates
|
||||||
|
```
|
||||||
|
|
||||||
|
### DMO List Output
|
||||||
|
```
|
||||||
|
Found 10 DMOs in org 'afvibe':
|
||||||
|
|
||||||
|
1. Individual__dlm
|
||||||
|
Label: Individual
|
||||||
|
Category: Profile
|
||||||
|
|
||||||
|
2. ContactPointEmail__dlm
|
||||||
|
Label: Contact Point Email
|
||||||
|
Category: Profile
|
||||||
|
|
||||||
|
[...]
|
||||||
|
```
|
||||||
|
|
||||||
|
### DMO Schema Output
|
||||||
|
```
|
||||||
|
DMO: Individual__dlm
|
||||||
|
Label: Individual
|
||||||
|
Category: Profile
|
||||||
|
Description: Represents an individual person
|
||||||
|
|
||||||
|
Fields:
|
||||||
|
• Id__c (Text) - Primary Key
|
||||||
|
• FirstName__c (Text)
|
||||||
|
• LastName__c (Text)
|
||||||
|
• BirthDate__c (DateTime)
|
||||||
|
|
||||||
|
Next steps:
|
||||||
|
- Query data: SELECT * FROM Individual__dlm LIMIT 10
|
||||||
|
- View DLO mappings to this DMO
|
||||||
|
- Create calculated insights
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- DLO names always end with `__dll` suffix
|
||||||
|
- DMO names always end with `__dlm` suffix
|
||||||
|
- Field names always end with `__c` suffix
|
||||||
|
- System fields (DataSource__c, KQ_*, cdp_sys_*) are automatically added
|
||||||
|
- Primary key fields are required for DLO and DMO queries
|
||||||
|
- API supports pagination (limit/offset) for large result sets
|
||||||
|
|
||||||
|
## Related Skills
|
||||||
|
|
||||||
|
- **datakit_workflow**: For DMO mapping operations
|
||||||
|
- **datakit_validation**: For validating datakit configurations
|
||||||
|
- Use this skill before creating DMO mappings to understand source DLO structure
|
||||||
191
skills/getting-datacloud-schema/references/README.md
Normal file
191
skills/getting-datacloud-schema/references/README.md
Normal file
@ -0,0 +1,191 @@
|
|||||||
|
# getting-datacloud-schema Skill
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
A skill that retrieves Data Lake Object (DLO) and Data Model Object (DMO) schema information from Salesforce Data Cloud using REST APIs.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
**List all DLOs:**
|
||||||
|
```
|
||||||
|
"Show me all DLOs in afvibe org"
|
||||||
|
"List Data Lake Objects in myorg"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Get specific DLO schema:**
|
||||||
|
```
|
||||||
|
"Get the schema for Employee__dll in afvibe"
|
||||||
|
"What fields does the Employee__dll DLO have in myorg?"
|
||||||
|
```
|
||||||
|
|
||||||
|
**List all DMOs:**
|
||||||
|
```
|
||||||
|
"Show me all DMOs in afvibe org"
|
||||||
|
"List Data Model Objects in myorg"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Get specific DMO schema:**
|
||||||
|
```
|
||||||
|
"Get the schema for Individual__dlm in afvibe"
|
||||||
|
"What fields does the Individual__dlm DMO have in myorg?"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Direct Script Usage
|
||||||
|
|
||||||
|
You can also run the scripts directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# List all DLOs
|
||||||
|
python3 scripts/get_dlo_schema.py <org_alias>
|
||||||
|
|
||||||
|
# Get specific DLO schema
|
||||||
|
python3 scripts/get_dlo_schema.py <org_alias> <dlo_name>
|
||||||
|
|
||||||
|
# List all DMOs
|
||||||
|
python3 scripts/get_dmo_schema.py <org_alias>
|
||||||
|
|
||||||
|
# Get specific DMO schema
|
||||||
|
python3 scripts/get_dmo_schema.py <org_alias> <dmo_name>
|
||||||
|
```
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
```bash
|
||||||
|
# List all DLOs in afvibe org
|
||||||
|
python3 scripts/get_dlo_schema.py afvibe
|
||||||
|
|
||||||
|
# Get Employee__dll schema from afvibe
|
||||||
|
python3 scripts/get_dlo_schema.py afvibe Employee__dll
|
||||||
|
|
||||||
|
# List all DMOs in afvibe org
|
||||||
|
python3 scripts/get_dmo_schema.py afvibe
|
||||||
|
|
||||||
|
# Get Individual__dlm schema from afvibe
|
||||||
|
python3 scripts/get_dmo_schema.py afvibe Individual__dlm
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
1. **SF CLI Installed**
|
||||||
|
```bash
|
||||||
|
sf --version
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Authenticated to Target Org**
|
||||||
|
```bash
|
||||||
|
sf org login web --alias <org_alias>
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Python 3 and Dependencies**
|
||||||
|
```bash
|
||||||
|
pip install requests pyyaml
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Data Cloud Enabled**
|
||||||
|
- Org must have Data Cloud provisioned
|
||||||
|
- User must have Data Cloud permissions
|
||||||
|
|
||||||
|
## What It Does
|
||||||
|
|
||||||
|
### List All DLOs
|
||||||
|
- Calls: `GET /services/data/v64.0/ssot/data-lake-objects`
|
||||||
|
- Returns: All DLOs with name, label, category, ID, record count
|
||||||
|
- Shows paginated results
|
||||||
|
|
||||||
|
### Get DLO Schema
|
||||||
|
- Calls: `GET /services/data/v64.0/ssot/data-lake-objects/{dlo_name}`
|
||||||
|
- Returns: Detailed field schema including field names, data types, primary key indicators, nullable status
|
||||||
|
|
||||||
|
### List All DMOs
|
||||||
|
- Calls: `GET /services/data/v64.0/ssot/data-model-objects`
|
||||||
|
- Returns: All DMOs with name, label, category, ID
|
||||||
|
- Shows paginated results
|
||||||
|
|
||||||
|
### Get DMO Schema
|
||||||
|
- Calls: `GET /services/data/v64.0/ssot/data-model-objects/{dmo_name}`
|
||||||
|
- Returns: Detailed field schema including field names, data types, primary key indicators, nullable status
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
| Endpoint | Method | Purpose |
|
||||||
|
|----------|--------|---------|
|
||||||
|
| `/services/data/v64.0/ssot/data-lake-objects` | GET | List all DLOs |
|
||||||
|
| `/services/data/v64.0/ssot/data-lake-objects/{name}` | GET | Get DLO schema |
|
||||||
|
| `/services/data/v64.0/ssot/data-model-objects` | GET | List all DMOs |
|
||||||
|
| `/services/data/v64.0/ssot/data-model-objects/{name}` | GET | Get DMO schema |
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
### DLO List
|
||||||
|
```
|
||||||
|
Found 5 DLOs in org 'afvibe':
|
||||||
|
|
||||||
|
1. DataCustomCodeLogs__dll
|
||||||
|
Label: DataCustomCodeLogs
|
||||||
|
Category: Engagement
|
||||||
|
Records: 233
|
||||||
|
|
||||||
|
2. Employee__dll
|
||||||
|
Label: Employee
|
||||||
|
Category: Profile
|
||||||
|
Records: 12
|
||||||
|
```
|
||||||
|
|
||||||
|
### DLO Schema
|
||||||
|
```
|
||||||
|
DLO: Employee__dll
|
||||||
|
Label: Employee
|
||||||
|
Category: Profile
|
||||||
|
Status: ACTIVE
|
||||||
|
Records: 12
|
||||||
|
|
||||||
|
Fields (9 total):
|
||||||
|
- id__c (Text) - Primary Key
|
||||||
|
- name__c (Text)
|
||||||
|
- position__c (Text)
|
||||||
|
- manager_id__c (Number)
|
||||||
|
- DataSource__c (Text)
|
||||||
|
[...]
|
||||||
|
```
|
||||||
|
|
||||||
|
### DMO List
|
||||||
|
```
|
||||||
|
Found 10 DMOs in org 'afvibe':
|
||||||
|
|
||||||
|
1. Individual__dlm
|
||||||
|
Label: Individual
|
||||||
|
Category: Profile
|
||||||
|
|
||||||
|
2. ContactPointEmail__dlm
|
||||||
|
Label: Contact Point Email
|
||||||
|
Category: Profile
|
||||||
|
```
|
||||||
|
|
||||||
|
### DMO Schema
|
||||||
|
```
|
||||||
|
DMO: Individual__dlm
|
||||||
|
Label: Individual
|
||||||
|
Category: Profile
|
||||||
|
|
||||||
|
Fields (8 total):
|
||||||
|
- Id__c (Text) - Primary Key
|
||||||
|
- FirstName__c (Text)
|
||||||
|
- LastName__c (Text)
|
||||||
|
- BirthDate__c (DateTime)
|
||||||
|
[...]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
| Issue | Fix |
|
||||||
|
|-------|-----|
|
||||||
|
| Org not connected | `sf org login web --alias <org_alias>` |
|
||||||
|
| Module not found: requests | `pip install requests pyyaml` |
|
||||||
|
| DLO not found | Verify name ends with `__dll`, list all DLOs first |
|
||||||
|
| DMO not found | Verify name ends with `__dlm`, list all DMOs first |
|
||||||
|
| Permission denied | Verify user has Data Cloud permissions |
|
||||||
|
|
||||||
|
## Related Skills
|
||||||
|
|
||||||
|
- **datakit workflow**: For DMO mapping operations
|
||||||
|
- **datakit validation**: For validating datakit configurations
|
||||||
|
- Use this skill before creating DMO mappings to understand source DLO structure
|
||||||
244
skills/getting-datacloud-schema/scripts/get_dlo_schema.py
Executable file
244
skills/getting-datacloud-schema/scripts/get_dlo_schema.py
Executable file
@ -0,0 +1,244 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
List all Data Lake Objects and retrieve schema for one DLO using REST API.
|
||||||
|
Uses SF CLI for authentication.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
def authenticate_to_org(org_alias):
|
||||||
|
"""
|
||||||
|
Authenticate to Salesforce org using SF CLI.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
org_alias: SF CLI org alias (e.g., 'afvibe')
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (instance_url, access_token, username)
|
||||||
|
"""
|
||||||
|
print(f"🔐 Authenticating to Salesforce org '{org_alias}'...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
['sf', 'org', 'display', '--target-org', org_alias, '--json'],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True
|
||||||
|
)
|
||||||
|
|
||||||
|
org_data = json.loads(result.stdout)
|
||||||
|
|
||||||
|
if org_data.get('status') != 0:
|
||||||
|
raise Exception(f"SF CLI returned error: {org_data}")
|
||||||
|
|
||||||
|
org_info = org_data['result']
|
||||||
|
|
||||||
|
if org_info.get('connectedStatus') != 'Connected':
|
||||||
|
raise Exception(f"Org '{org_alias}' is not connected. Run: sf org login web --alias {org_alias}")
|
||||||
|
|
||||||
|
instance_url = org_info['instanceUrl']
|
||||||
|
access_token = org_info['accessToken']
|
||||||
|
username = org_info.get('username', 'Unknown')
|
||||||
|
|
||||||
|
print(f"✅ Authenticated as: {username}")
|
||||||
|
print(f"📍 Instance: {instance_url}\n")
|
||||||
|
|
||||||
|
return instance_url, access_token, username
|
||||||
|
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
raise Exception(f"SF CLI command failed: {e.stderr}")
|
||||||
|
except (json.JSONDecodeError, KeyError) as e:
|
||||||
|
raise Exception(f"Failed to parse SF CLI output: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def list_all_dlos(instance_url, access_token, api_version='v64.0'):
|
||||||
|
"""
|
||||||
|
List all Data Lake Objects using SSOT REST API.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
instance_url: Salesforce instance URL
|
||||||
|
access_token: OAuth access token
|
||||||
|
api_version: API version (default: v64.0)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of DLO dictionaries
|
||||||
|
"""
|
||||||
|
url = f"{instance_url}/services/data/{api_version}/ssot/data-lake-objects"
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': f'Bearer {access_token}',
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Accept': 'application/json'
|
||||||
|
}
|
||||||
|
|
||||||
|
print("📋 Fetching all Data Lake Objects...")
|
||||||
|
response = requests.get(url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise Exception(f"API Error: HTTP {response.status_code}\n{response.text[:500]}")
|
||||||
|
|
||||||
|
response_data = response.json()
|
||||||
|
|
||||||
|
# Extract DLO list from paginated response
|
||||||
|
if isinstance(response_data, dict) and 'dataLakeObjects' in response_data:
|
||||||
|
dlos = response_data['dataLakeObjects']
|
||||||
|
total_size = response_data.get('totalSize', len(dlos))
|
||||||
|
print(f"✅ Found {len(dlos)} DLOs (Total: {total_size})\n")
|
||||||
|
else:
|
||||||
|
# Fallback if response format is different
|
||||||
|
dlos = response_data if isinstance(response_data, list) else []
|
||||||
|
print(f"✅ Found {len(dlos)} DLOs\n")
|
||||||
|
|
||||||
|
return dlos
|
||||||
|
|
||||||
|
|
||||||
|
def get_dlo_schema(instance_url, access_token, dlo_name, api_version='v64.0'):
|
||||||
|
"""
|
||||||
|
Get detailed schema for a specific DLO.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
instance_url: Salesforce instance URL
|
||||||
|
access_token: OAuth access token
|
||||||
|
dlo_name: DLO developer name (e.g., 'Employee__dll')
|
||||||
|
api_version: API version (default: v64.0)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
DLO detail dictionary with full schema
|
||||||
|
"""
|
||||||
|
url = f"{instance_url}/services/data/{api_version}/ssot/data-lake-objects/{dlo_name}"
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': f'Bearer {access_token}',
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Accept': 'application/json'
|
||||||
|
}
|
||||||
|
|
||||||
|
print(f"🔍 Fetching schema for DLO: {dlo_name}...")
|
||||||
|
response = requests.get(url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise Exception(f"API Error: HTTP {response.status_code}\n{response.text[:500]}")
|
||||||
|
|
||||||
|
response_data = response.json()
|
||||||
|
|
||||||
|
# Extract DLO from paginated response
|
||||||
|
if isinstance(response_data, dict) and 'dataLakeObjects' in response_data:
|
||||||
|
dlos = response_data['dataLakeObjects']
|
||||||
|
if dlos:
|
||||||
|
return dlos[0] # Return first (should be only) DLO
|
||||||
|
else:
|
||||||
|
raise Exception(f"DLO '{dlo_name}' not found")
|
||||||
|
else:
|
||||||
|
# Fallback if response format is different
|
||||||
|
return response_data
|
||||||
|
|
||||||
|
|
||||||
|
def display_dlo_list(dlos):
|
||||||
|
"""Display summary of all DLOs."""
|
||||||
|
print("=" * 80)
|
||||||
|
print("📊 DATA LAKE OBJECTS")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
for idx, dlo in enumerate(dlos, 1):
|
||||||
|
print(f"\n{idx}. {dlo.get('name', 'Unknown')}")
|
||||||
|
print(f" Label: {dlo.get('label', 'N/A')}")
|
||||||
|
print(f" Category: {dlo.get('category', 'N/A')}")
|
||||||
|
if 'id' in dlo:
|
||||||
|
print(f" ID: {dlo['id']}")
|
||||||
|
|
||||||
|
|
||||||
|
def display_dlo_schema(dlo_detail):
|
||||||
|
"""Display detailed schema information for a DLO."""
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print(f"🔍 SCHEMA DETAILS FOR: {dlo_detail.get('name')}")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
print(f"\n📝 Basic Information:")
|
||||||
|
print(f" Name: {dlo_detail.get('name')}")
|
||||||
|
print(f" Label: {dlo_detail.get('label')}")
|
||||||
|
print(f" Category: {dlo_detail.get('category')}")
|
||||||
|
print(f" Description: {dlo_detail.get('description', 'N/A')}")
|
||||||
|
|
||||||
|
if 'dataspaceInfo' in dlo_detail:
|
||||||
|
dataspaces = dlo_detail['dataspaceInfo']
|
||||||
|
dataspace_names = [ds.get('name', 'Unknown') for ds in dataspaces]
|
||||||
|
print(f" Dataspaces: {', '.join(dataspace_names)}")
|
||||||
|
|
||||||
|
# Display field schema
|
||||||
|
fields = dlo_detail.get('fields', [])
|
||||||
|
|
||||||
|
if fields:
|
||||||
|
print(f"\n🔧 Fields ({len(fields)} total):")
|
||||||
|
print("-" * 80)
|
||||||
|
|
||||||
|
# Show all fields with detailed info
|
||||||
|
for field in fields:
|
||||||
|
print(f"\n • {field.get('name')}")
|
||||||
|
print(f" Label: {field.get('label', 'N/A')}")
|
||||||
|
print(f" Data Type: {field.get('dataType', 'Unknown')}")
|
||||||
|
print(f" Primary Key: {field.get('isPrimaryKey', False)}")
|
||||||
|
print(f" Nullable: {field.get('isNullable', True)}")
|
||||||
|
|
||||||
|
if 'length' in field:
|
||||||
|
print(f" Length: {field['length']}")
|
||||||
|
if 'precision' in field:
|
||||||
|
print(f" Precision: {field['precision']}")
|
||||||
|
if 'scale' in field:
|
||||||
|
print(f" Scale: {field['scale']}")
|
||||||
|
else:
|
||||||
|
print("\n ⚠️ No fields found in schema")
|
||||||
|
|
||||||
|
# Show full JSON (optional, can be commented out)
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("📄 FULL SCHEMA (JSON):")
|
||||||
|
print("=" * 80)
|
||||||
|
print(json.dumps(dlo_detail, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main execution function."""
|
||||||
|
if len(sys.argv) < 2:
|
||||||
|
print("Usage: python list_dlos_and_schema.py <org_alias> [dlo_name]")
|
||||||
|
print("\nExamples:")
|
||||||
|
print(" python list_dlos_and_schema.py afvibe")
|
||||||
|
print(" python list_dlos_and_schema.py afvibe Employee__dll")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
org_alias = sys.argv[1]
|
||||||
|
specific_dlo = sys.argv[2] if len(sys.argv) > 2 else None
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Step 1: Authenticate
|
||||||
|
instance_url, access_token, username = authenticate_to_org(org_alias)
|
||||||
|
|
||||||
|
# Step 2: List all DLOs
|
||||||
|
dlos = list_all_dlos(instance_url, access_token)
|
||||||
|
display_dlo_list(dlos)
|
||||||
|
|
||||||
|
# Step 3: Get schema for a specific DLO
|
||||||
|
if specific_dlo:
|
||||||
|
# User specified a DLO name
|
||||||
|
dlo_detail = get_dlo_schema(instance_url, access_token, specific_dlo)
|
||||||
|
display_dlo_schema(dlo_detail)
|
||||||
|
elif dlos:
|
||||||
|
# Get schema for the first DLO
|
||||||
|
first_dlo = dlos[0]
|
||||||
|
dlo_name = first_dlo.get('name')
|
||||||
|
dlo_detail = get_dlo_schema(instance_url, access_token, dlo_name)
|
||||||
|
display_dlo_schema(dlo_detail)
|
||||||
|
else:
|
||||||
|
print("\n⚠️ No DLOs found in this org")
|
||||||
|
|
||||||
|
print("\n✅ Done!")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"\n❌ Error: {e}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
233
skills/getting-datacloud-schema/scripts/get_dmo_schema.py
Executable file
233
skills/getting-datacloud-schema/scripts/get_dmo_schema.py
Executable file
@ -0,0 +1,233 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
List all Data Model Objects and retrieve schema for one DMO using REST API.
|
||||||
|
Uses SF CLI for authentication.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import requests
|
||||||
|
|
||||||
|
|
||||||
|
def authenticate_to_org(org_alias):
|
||||||
|
"""
|
||||||
|
Authenticate to Salesforce org using SF CLI.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
org_alias: SF CLI org alias (e.g., 'afvibe')
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (instance_url, access_token, username)
|
||||||
|
"""
|
||||||
|
print(f"🔐 Authenticating to Salesforce org '{org_alias}'...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
['sf', 'org', 'display', '--target-org', org_alias, '--json'],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=True
|
||||||
|
)
|
||||||
|
|
||||||
|
org_data = json.loads(result.stdout)
|
||||||
|
|
||||||
|
if org_data.get('status') != 0:
|
||||||
|
raise Exception(f"SF CLI returned error: {org_data}")
|
||||||
|
|
||||||
|
org_info = org_data['result']
|
||||||
|
|
||||||
|
if org_info.get('connectedStatus') != 'Connected':
|
||||||
|
raise Exception(f"Org '{org_alias}' is not connected. Run: sf org login web --alias {org_alias}")
|
||||||
|
|
||||||
|
instance_url = org_info['instanceUrl']
|
||||||
|
access_token = org_info['accessToken']
|
||||||
|
username = org_info.get('username', 'Unknown')
|
||||||
|
|
||||||
|
print(f"✅ Authenticated as: {username}")
|
||||||
|
print(f"📍 Instance: {instance_url}\n")
|
||||||
|
|
||||||
|
return instance_url, access_token, username
|
||||||
|
|
||||||
|
except subprocess.CalledProcessError as e:
|
||||||
|
raise Exception(f"SF CLI command failed: {e.stderr}")
|
||||||
|
except (json.JSONDecodeError, KeyError) as e:
|
||||||
|
raise Exception(f"Failed to parse SF CLI output: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def list_all_dmos(instance_url, access_token, api_version='v64.0'):
|
||||||
|
"""
|
||||||
|
List all Data Model Objects using SSOT REST API.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
instance_url: Salesforce instance URL
|
||||||
|
access_token: OAuth access token
|
||||||
|
api_version: API version (default: v64.0)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of DMO dictionaries
|
||||||
|
"""
|
||||||
|
url = f"{instance_url}/services/data/{api_version}/ssot/data-model-objects"
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': f'Bearer {access_token}',
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Accept': 'application/json'
|
||||||
|
}
|
||||||
|
|
||||||
|
print("📋 Fetching all Data Model Objects...")
|
||||||
|
response = requests.get(url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise Exception(f"API Error: HTTP {response.status_code}\n{response.text[:500]}")
|
||||||
|
|
||||||
|
response_data = response.json()
|
||||||
|
|
||||||
|
# Extract DMO list from paginated response
|
||||||
|
if isinstance(response_data, dict) and 'dataModelObject' in response_data:
|
||||||
|
dmos = response_data['dataModelObject']
|
||||||
|
total_size = response_data.get('totalSize', len(dmos))
|
||||||
|
print(f"✅ Found {len(dmos)} DMOs (Total: {total_size})\n")
|
||||||
|
else:
|
||||||
|
# Fallback if response format is different
|
||||||
|
dmos = response_data if isinstance(response_data, list) else []
|
||||||
|
print(f"✅ Found {len(dmos)} DMOs\n")
|
||||||
|
|
||||||
|
return dmos
|
||||||
|
|
||||||
|
|
||||||
|
def get_dmo_schema(instance_url, access_token, dmo_name, api_version='v64.0'):
|
||||||
|
"""
|
||||||
|
Get detailed schema for a specific DMO.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
instance_url: Salesforce instance URL
|
||||||
|
access_token: OAuth access token
|
||||||
|
dmo_name: DMO developer name (e.g., 'Individual__dlm')
|
||||||
|
api_version: API version (default: v64.0)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
DMO detail dictionary with full schema
|
||||||
|
"""
|
||||||
|
url = f"{instance_url}/services/data/{api_version}/ssot/data-model-objects/{dmo_name}"
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'Authorization': f'Bearer {access_token}',
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Accept': 'application/json'
|
||||||
|
}
|
||||||
|
|
||||||
|
print(f"🔍 Fetching schema for DMO: {dmo_name}...")
|
||||||
|
response = requests.get(url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise Exception(f"API Error: HTTP {response.status_code}\n{response.text[:500]}")
|
||||||
|
|
||||||
|
response_data = response.json()
|
||||||
|
|
||||||
|
# Single DMO endpoint returns the object directly (not wrapped in an array)
|
||||||
|
return response_data
|
||||||
|
|
||||||
|
|
||||||
|
def display_dmo_list(dmos):
|
||||||
|
"""Display summary of all DMOs."""
|
||||||
|
print("=" * 80)
|
||||||
|
print("📊 DATA MODEL OBJECTS")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
for idx, dmo in enumerate(dmos, 1):
|
||||||
|
print(f"\n{idx}. {dmo.get('name', 'Unknown')}")
|
||||||
|
print(f" Label: {dmo.get('label', 'N/A')}")
|
||||||
|
print(f" Category: {dmo.get('category', 'N/A')}")
|
||||||
|
print(f" Creation Type: {dmo.get('creationType', 'N/A')}")
|
||||||
|
print(f" Data Space: {dmo.get('dataSpaceName', 'N/A')}")
|
||||||
|
|
||||||
|
|
||||||
|
def display_dmo_schema(dmo_detail):
|
||||||
|
"""Display detailed schema information for a DMO."""
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print(f"🔍 SCHEMA DETAILS FOR: {dmo_detail.get('name')}")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
print(f"\n📝 Basic Information:")
|
||||||
|
print(f" Name: {dmo_detail.get('name')}")
|
||||||
|
print(f" Label: {dmo_detail.get('label')}")
|
||||||
|
print(f" Category: {dmo_detail.get('category')}")
|
||||||
|
print(f" Creation Type: {dmo_detail.get('creationType', 'N/A')}")
|
||||||
|
print(f" Data Space: {dmo_detail.get('dataSpaceName', 'N/A')}")
|
||||||
|
|
||||||
|
# Display field schema
|
||||||
|
fields = dmo_detail.get('fields', [])
|
||||||
|
|
||||||
|
if fields:
|
||||||
|
print(f"\n🔧 Fields ({len(fields)} total):")
|
||||||
|
print("-" * 80)
|
||||||
|
|
||||||
|
# Show all fields with detailed info
|
||||||
|
for field in fields:
|
||||||
|
print(f"\n • {field.get('name')}")
|
||||||
|
print(f" Label: {field.get('label', 'N/A')}")
|
||||||
|
print(f" Data Type: {field.get('type', 'Unknown')}")
|
||||||
|
print(f" Primary Key: {field.get('isPrimaryKey', False)}")
|
||||||
|
print(f" Creation Type: {field.get('creationType', 'N/A')}")
|
||||||
|
print(f" Usage Tag: {field.get('usageTag', 'N/A')}")
|
||||||
|
|
||||||
|
if 'length' in field:
|
||||||
|
print(f" Length: {field['length']}")
|
||||||
|
if 'precision' in field:
|
||||||
|
print(f" Precision: {field['precision']}")
|
||||||
|
if 'scale' in field:
|
||||||
|
print(f" Scale: {field['scale']}")
|
||||||
|
else:
|
||||||
|
print("\n ⚠️ No fields found in schema")
|
||||||
|
|
||||||
|
# Show full JSON
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("📄 FULL SCHEMA (JSON):")
|
||||||
|
print("=" * 80)
|
||||||
|
print(json.dumps(dmo_detail, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Main execution function."""
|
||||||
|
if len(sys.argv) < 2:
|
||||||
|
print("Usage: python get_dmo_schema.py <org_alias> [dmo_name]")
|
||||||
|
print("\nExamples:")
|
||||||
|
print(" python get_dmo_schema.py afvibe")
|
||||||
|
print(" python get_dmo_schema.py afvibe Individual__dlm")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
org_alias = sys.argv[1]
|
||||||
|
specific_dmo = sys.argv[2] if len(sys.argv) > 2 else None
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Step 1: Authenticate
|
||||||
|
instance_url, access_token, username = authenticate_to_org(org_alias)
|
||||||
|
|
||||||
|
# Step 2: List all DMOs
|
||||||
|
dmos = list_all_dmos(instance_url, access_token)
|
||||||
|
display_dmo_list(dmos)
|
||||||
|
|
||||||
|
# Step 3: Get schema for a specific DMO
|
||||||
|
if specific_dmo:
|
||||||
|
# User specified a DMO name
|
||||||
|
dmo_detail = get_dmo_schema(instance_url, access_token, specific_dmo)
|
||||||
|
display_dmo_schema(dmo_detail)
|
||||||
|
elif dmos:
|
||||||
|
# Get schema for the first DMO
|
||||||
|
first_dmo = dmos[0]
|
||||||
|
dmo_name = first_dmo.get('name')
|
||||||
|
dmo_detail = get_dmo_schema(instance_url, access_token, dmo_name)
|
||||||
|
display_dmo_schema(dmo_detail)
|
||||||
|
else:
|
||||||
|
print("\n⚠️ No DMOs found in this org")
|
||||||
|
|
||||||
|
print("\n✅ Done!")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"\n❌ Error: {e}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Loading…
Reference in New Issue
Block a user