afv-library/skills/automation-sandbox-post-copy-config-generate/references/source_format_handling.md

Source Format Handling

The customer SOP can arrive in any of the formats below — sometimes several at once. This file gives the exact extraction recipe per format so action records and endpoint values can be located reliably, and explains how to decide whether an embedded image carries data or is illustrative.

Rule of thumb: the action list (which OBM / Remote Site to update) and the endpoint table (the actual URL values) are frequently in different files or different tabs / pages. Always read every source the user supplies before deciding a value is missing. If a value still cannot be located after exhausting every source, skip the entry — never emit empty / placeholder fields.

---

PDF (.pdf)

import pypdf
reader = pypdf.PdfReader(path)
for i, page in enumerate(reader.pages):
    text = page.extract_text()

• Many enterprise SOPs are slide decks exported to PDF — the text layer is usually present, but tables may flatten oddly. Inspect the raw extracted text rather than trusting visual alignment.
• For section navigation, search case-insensitively for: "Post Refresh", "Post-Refresh", "Post Copy", "Post-Copy", "After Refresh", "Update ".
• If extract_text() returns empty or near-empty for a page, the page is likely an embedded image. OCR it:

import pdfplumber, pytesseract
with pdfplumber.open(path) as pdf:
    for page in pdf.pages:
        if not (page.extract_text() or "").strip():
            img = page.to_image(resolution=200).original
            text = pytesseract.image_to_string(img)

• If pytesseract is unavailable, fall back to pdftotext -layout (Poppler) and tesseract CLI.

xlsx (.xlsx)

import openpyxl
wb = openpyxl.load_workbook(path, data_only=True)
for sname in wb.sheetnames:
    ws = wb[sname]
    for row in ws.iter_rows(values_only=False):
        for cell in row:
            ...

• Always read every sheet. Customer planners routinely split the task list and the URL table across tabs (real example: Michelin UAT Refresh Planner — task list in Integration (Indus), URL table in Evolution SFA columns K/L/M).
• Iterate every column up to ws.max_column. Some sheets have data far past column M (e.g., column index 10000+) that the visible layout hides.
• Read with data_only=True so cached formula values are returned.
• Check cell.comment for free-text annotations that often contain the post-refresh value.
• Look for canonical column headers: Outbound Message, Remote Site, Object, Endpoint, Endpoint URL, Prod Endpoint URL, UAT Endpoint URL, Post-refresh URL. The post-refresh value is the one to emit.
• For embedded images on a sheet, use ws._images (or unzip the xlsx and inspect xl/media/) — see the image rules below.

csv (.csv)

import csv
with open(path, newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        ...

• Treat the same way as a single xlsx sheet — look for the canonical column headers above.

JSON (.json)

• If the customer already supplies a JSON document, scan every key for the canonical names (outboundMessage, endpointUrl, object, remoteSiteUrl, label, name). Be tolerant of casing (endpointURL, EndpointUrl, endpoint_url).
• Customer JSON might already be partially in the target shape; do not blindly copy it through — re-validate every entry against assets/json_schema.json and the catalog.

docx (.docx)

from docx import Document
doc = Document(path)
for p in doc.paragraphs:
    ...
for tbl in doc.tables:
    for row in tbl.rows:
        for cell in row.cells:
            ...

• Extract both paragraph text and table cells. Word tables frequently hold the endpoint table.
• Inline images (doc.inline_shapes) follow the image rules below.

Markdown / plain text (.md, .txt)

• Read directly with the Read tool. Tables in pipe-delimited Markdown are easy to parse — split on |.

Pasted excerpts in the conversation

• The user may paste content directly. Treat it as plain text.
• If they paste an image (drag-drop), use the Read tool on the image path and follow the image rules below.

---

Images (.png, .jpg, .jpeg, .tiff, .bmp)

Two kinds of image arrive in customer SOPs:

Kind	Examples	What to do
Data-bearing	screenshot of the Outbound Messages list page showing real Names + Endpoint URLs; captured Remote Site Settings table with real URLs; spreadsheet snippet pasted as a screenshot; configuration page where the endpoint values are visible	OCR and treat the values as authoritative for the entries they cover
Illustrative	"this is what the setup screen looks like" with placeholder values; architecture diagram showing system topology; UI flow chart; redacted/blurred values; screenshot whose surrounding text says "for example" / "as shown below your values will look like this"	Do NOT extract values — the image is documentation, not data

Heuristic — is this image data-bearing?

Yes (use the values), if any of:

• The surrounding paragraph references it as the source of truth: "the values listed below", "as captured in the spreadsheet above", "copy these into the post-refresh setup", "the URLs you copied before the refresh".
• The image shows a Salesforce setup table (Outbound Messages list, Remote Site Settings list, Custom Metadata Type list) with real values for known catalog types.
• Real endpoint URLs are visible (full domain, not redacted, not obviously a placeholder like https://example.com/...).
• The image directly precedes or follows a "Post Refresh Steps" heading and lists the entities named in that step.

No (skip the image), if any of:

• Surrounding text says "for example", "as a sample", "this is what it will look like", "your real values will differ".
• The values shown are placeholders (PROD_URL, <your domain>, username/password, all-x or all-* redaction).
• The image is a flowchart / arch diagram / sequence diagram.
• The image is a UI walkthrough screenshot whose purpose is to show where to click, not what to enter.
• The image is a stock screenshot of the Salesforce Help / Setup interface unrelated to the customer's actual values.

OCR recipe

from PIL import Image
import pytesseract
img = Image.open(image_path)
text = pytesseract.image_to_string(img)

For higher accuracy on tables:

text = pytesseract.image_to_string(img, config="--psm 6")

--psm 6 treats the image as a single uniform block of text (good for tables). Try --psm 4 for column-of-text layouts.

If pytesseract is unavailable, read the image with the Read tool and transcribe the visible values into a temporary text buffer — the multimodal Read reliably picks up table contents in screenshots.

After OCR

• Normalise whitespace, strip trailing zero-width chars.
• Fix common OCR confusions in URLs: 0/O, 1/l/I, :/;, //\, _/—. Cross-check with surrounding text where possible.
• If two URL columns are present (Prod / UAT), use the post-refresh column.
• Treat OCR'd values like any other extracted value — if the text is unreadable or ambiguous, skip the entry and surface it. Do NOT guess to fill a field.

---

Multi-source merge

When the user supplies multiple files:

1. Read every file end-to-end first; build an internal map of "named entity → values found".
2. Identify the action list source (which OBMs / Remote Sites need updating) — usually the file with "Post Refresh Steps".
3. For each named action, look up the value in the map. If found in any source, use it. If not found anywhere, skip and surface.
4. Emit one entry per (configuration, label) pair — do not duplicate when the same OBM appears in multiple sources.

---

Failure modes to surface in the response

When you skip an entry, list it explicitly so the customer can correct the source. Use one of these labels in the response:

• Missing value — the entity is named but no concrete value was found in any supplied source.
• Out of catalog — the action targets a configuration type not yet supported (Custom Labels, Connected Apps, etc.).
• Unreadable image — an image was data-bearing but OCR returned unintelligible output.
• Ambiguous Object (OutboundMessages only) — the OBM name does not reveal the target SObject and no supplemental table provides it.