Education Cloud Course Catalog Migrate

Migrating Course Catalog to Education Cloud

Import course catalog data from multiple sources (CSV, PDF, website) and bulk-create Learning Course records with structure detection and data quality validation.

Scope

• In scope: Multi-source ingestion (CSV, PDF, web), Learning + LearningCourse bulk creation via sObject REST API (Composite batches up to 200), linked course detection (lecture+lab via IsLinkedOnly), credit hour parsing (Duration/DurationUnit), course type mapping, data quality validation, conflict resolution, gap identification/remediation
• Out of scope: Creating individual Learning Course records manually (delegate to metadata generation skills), creating Learning Programs (Learning.Type = 'LearningProgram' + separate child object), creating External Learning (Learning.Type = 'ExternalLearning'), course sections/offerings (CourseOffering object), enrollment/registration workflows (no dedicated skill yet — ask the user), SIS sync (separate integration workflow)

Required Inputs

Gather or infer before proceeding:

• Target org: Active Salesforce org with Education Cloud enabled
• Primary data source: CSV file, PDF document, or website URL with course catalog data
• Business Account: the Academic Operations Business Account to link courses to (e.g., College of Engineering)
• Academic term (optional): context only, not stored on Learning Course

Additional sources (gathered during workflow):

• Supplementary data sources for missing fields (descriptions, types, mappings)
• Lecture-lab mapping files (if multi-component courses detected)
• Conflict resolution decisions (if existing courses found)

Defaults unless specified:

• Course status: Active (IsActive = true on Learning)
• Batch size: 25 records per Composite API call as an agent (API supports 200; tool-call payload limits are the binding constraint — see Step 7)
• Data quality validation: Enabled
• Learning.Type: 'LearningCourse' (required for course catalog entries)

Execution Transport

Resolve the API version first — do not hardcode. GET /services/data/ (no version segment) returns an endpoint map already resolved to the org's max version, not a {version, label, url} list. Pipe it into scripts/resolve_api_version.py to get {"version": "69.0"}; use as vXX.X everywhere below, including the transport probe.

Then probe transport health: GET /services/data/vXX.X/limits via headless-360 dispatch. 2xx → route all calls through it. On absent/4xx/5xx → probe other transports (other Salesforce MCP, custom MCP), use first healthy, announce it. None healthy → stop, ask user to connect one; never fabricate. (SOQL /query is routable on headless-360 — confirmed by live test; /tooling/query and ?fields= are unconfirmed, use describe + direct record reads instead.)

Never fall back to sf CLI or shell out for org credentials — a slow/oversized batch is a batch-size problem (shrink it, step 7), not a transport failure, and CLI shell-out trips an unbypassable approval gate on some surfaces. A version too new for the org returns 404 NOT_FOUND on an otherwise-healthy transport — re-read GET /services/data/, drop to the org's max version, retry, don't switch transport. See references/gotchas.md for both.

Workflow

All steps are sequential. Do not skip or reorder. If blocked, stop and ask for missing context.

Phase 1 — Ingest Primary Source

1. Identify data source type — determine if primary source is CSV, PDF, or website URL. Ask user to provide the source if not already specified.

2. Parse primary source — extract course data:

   • For CSV: Use Read tool to load file, parse columns
   • For PDF: Use Read tool (supports PDF extraction), parse text. Over 20 pages, read in sequential page-range chunks and merge parsed rows before validating completeness — see references/large-catalog-handling.md §1
   • For website: Use WebFetch tool to retrieve HTML, parse course listings

   Look for: course codes (identity — split into SubjectAbbreviation + CourseNumber, see step 8/9), course names (Name), credit hours (Duration + DurationUnit), descriptions (Description), course types (Lecture/Lab/Exam/Drill/Recitation), prerequisites, corequisites, linked course indicators (IsLinkedOnly), delivery modality (if the same course code repeats with different Duration/DurationUnit — see references/large-catalog-handling.md §2).

3. Validate data completeness — pipe the parsed records into the completeness helper:

   python3 scripts/validate_completeness.py --required "CourseCode,Name,Duration" --recommended "Description,CourseType" <<'JSON'
   [ { "CourseCode": "...", "Name": "...", ... }, ... ]
   JSON
   

   The report gives present/missing counts per field. Optional fields (prerequisites, corequisites, linked courses) are not checked — their absence never blocks progress.

   If any required field shows missing > 0, report error and stop. If recommended fields are missing, note the gaps and continue to Phase 2.

Phase 2 — Identify Gaps and Request Additional Sources

4. Report data quality — print a summary table showing which fields are present and which are missing. Example:

   Data Quality Report:
   • Course codes: 120/120 OK
   • Course names: 120/120 OK
   • Credit hours: 120/120 OK
   • Descriptions: 115/120 (5 missing)
   • Course types: 118/120 (2 missing)
   

5. Request supplementary sources — if key fields are missing (descriptions, types), ask user: "Do you have additional sources — a website, department document, or mapping file?"

   • If yes: receive additional source, parse it, and merge data with primary source
   • If no: offer to create courses with placeholder values or stop and wait for complete data

   Repeat this step until all gaps are filled or user confirms to proceed with incomplete data.

   If subject-to-Business-Account mapping is ambiguous, resolve every ambiguous case in one pass (source doc + live public catalog + course level/number patterns) and present them together, not one round at a time (see references/gotchas.md).

6. Detect multi-component courses — pipe the course codes into the lecture/lab pairing helper:

   python3 scripts/detect_linked_courses.py --suffixes "L,LAB" <<'JSON'
   [ "BIO-201", "BIO-201L", ... ]
   JSON
   

   Each {"lecture", "lab"} pair returned is a candidate multi-component course.

   • If any pairs are found, pause and explain: "I found X multi-component (lecture + lab) courses. Structure them as (1) separate standalone courses, or (2) linked courses (IsLinkedOnly flag, students must take both)?"

   If user selects "linked courses", request a mapping file or confirm the naming pattern is reliable.

   Detect modality duplicates — group parsed rows by course code. Same code, different Duration/DurationUnit/Description = a modality variant (e.g. Online vs. On-Campus), not a duplicate — never silently dedupe/merge. Resolve the representation once per run, not per-course. See references/large-catalog-handling.md §2.

Phase 3 — Batch Create Learning and LearningCourse Records

7. Prepare batch creation plan — pipe the parsed course records into the batching helper. The Composite API accepts up to 200 records/call, but an agent's tool-call payload has a smaller practical ceiling (~56KB/150 records). Use --batch-size 25 when executing as an agent (raise toward 200 only if the calling environment confirms it can handle larger payloads):

   python3 scripts/batch_courses.py --batch-size 25 <<'JSON'
   [ { "Name": "...", ... }, ... ]
   JSON
   

   Each element of the returned array is one Composite API call's worth of records. Print: "Starting course catalog import. I'll use Composite API to create records in batches. Total: X courses."

   If a batch still won't inline through a healthy dispatch transport, re-run with a smaller --batch-size (e.g. 10) instead of switching transport. Track dispatch vs. fallback counts and report both in the Phase 5 summary.

   Over 200 courses (8+ batches): disclose total batch count upfront, print the Batch N complete line after every batch (not just at the end), and — only if the surface exposes a parallel-task/subagent tool — follow the fan-out rules in references/large-catalog-handling.md §3-4. Never assume that capability exists; default to sequential batches.

8. Check for existing courses — before creating Batch 1, scope the check to this batch's own course codes, never a global fetch (an unrelated catalog entry already in the org is not a conflict).

   Split this batch's course codes first (scripts/split_course_code.py) — key the check on those identity fields, never CipCode (many orgs leave it null — see references/gotchas.md).

   Headless-360 note: SOQL /query is routable via the headless-360 dispatch tool (confirmed by live test) — query LearningCourse, matching each split {SubjectAbbreviation, CourseNumber} pair exactly (SOQL has no tuple IN, so OR the pairs): SELECT Id, LearningId, Name, SubjectAbbreviation, CourseNumber FROM LearningCourse WHERE (SubjectAbbreviation = 'BIO' AND CourseNumber = '201') OR .... If Ids are unknown and a SOQL lookup isn't available, ask the user to supply them (or a prior import manifest). Never do this via Bash/CLI — see references/gotchas.md.

   If any rows come back (i.e. a SubjectAbbreviation+CourseNumber pair in this batch already exists in the org), pause and explain conflict:

   • Show existing course details vs. new import data side-by-side
   • Check if existing courses have dependencies (course offerings, enrollments)
   • Offer options: (1) Update existing courses, (2) Skip existing courses, (3) Cancel and review manually

   Wait for user decision before proceeding.

9. Create Batch 1 — two-step process for each course:

   Field placement (verified via describe): CipCode, Duration, DurationUnit, ProviderId are createable on Learning ONLY — ProviderId on the child fails with INVALID_FIELD_FOR_INSERT_UPDATE. LearningCourse takes Name, LearningId, CourseType, IsLinkedOnly, SubjectAbbreviation, CourseNumber, Description, CourseLevelDescription, FieldOfStudy — it inherits the account link via LearningId. Confirm with describe before building the body.

   CipCode is opt-in. Federal CIP taxonomy field, not the course's identity — leave unset unless the source supplies a real CIP code and the user confirms.

   Step 1: Create Learning (parent) records via Composite API:

   POST /services/data/vXX.X/composite/sobjects
   {
     "allOrNone": false,
     "records": [
       {
         "attributes": {"type": "Learning", "referenceId": "learning1"},
         "Name": "<course-name>",
         "Type": "LearningCourse",
         "Description": "<description>",
         "ProviderId": "<account-id>",
         "Duration": <hours>,
         "DurationUnit": "Credit Hours",
         "IsActive": true
       },
       ...
     ]
   }
   

   Capture each returned Id (order matches the records array) to use as LearningId in Step 2.

   Step 2: Create LearningCourse (child) records via Composite API:

   POST /services/data/vXX.X/composite/sobjects
   {
     "allOrNone": false,
     "records": [
       {
         "attributes": {"type": "LearningCourse"},
         "Name": "<course-record-name>",
         "LearningId": "<learning-id-from-step-1>",
         "SubjectAbbreviation": "<subject-prefix>",
         "CourseNumber": "<course-number>",
         "CourseType": "<parsed-course-type>",
         "IsLinkedOnly": false
       },
       ...
     ]
   }
   

   Set CourseType from the parsed/mapped course type for that record (see the Gotchas table for value mapping) — do not hard-code it. A lab record must get CourseType: "Lab", not "Lecture".

   CourseType picklist values (verified): Lecture, Lab, Exam, Drill, Recitation.

   restrictedPicklist: true means only today's values are valid to submit, not that the set is closed. If a source value (e.g. DurationUnit = "Program Points") isn't in describe's list, don't silently drop/remap it — offer adding the picklist value via metadata alongside remapping or a placeholder (see references/gotchas.md).

   Print progress: Batch 1 complete — X Learning + X LearningCourse records created.

10. Handle multi-component courses — if user selected "linked courses" in step 6:

    • Create lecture courses with CourseType: "Lecture", IsLinkedOnly = false
    • Create lab courses with CourseType: "Lab", IsLinkedOnly = true
    • Use the detect_linked_courses.py pairs (or a user-supplied mapping file) to determine which labs are linked-only
    • Print: X linked lab courses created (IsLinkedOnly = true)

11. Repeat for remaining batches — continue steps 8-10 for Batch 2, Batch 3, etc. until all courses created.

Phase 4 — Data Quality Validation and Conflict Resolution

12. Scan for incomplete records — after all batches complete, read back each newly created record by the Id captured from its create response:

    GET /services/data/vXX.X/sobjects/Learning/<id>
    GET /services/data/vXX.X/sobjects/LearningCourse/<id>
    

    Headless-360 note: SOQL /query is routable via the headless-360 dispatch tool (confirmed by live test) — a CreatedDate = TODAY query works as an alternative to iterating captured Ids with direct record GETs.

    Run the completeness helper separately per sObject — CourseType exists only on LearningCourse, Description only on Learning; combining both shapes into one call false-positives every parent as missing CourseType:

    python3 scripts/validate_completeness.py --required "Name" --recommended "Description" <<'JSON'
    [ { "Name": "...", "Description": "...", ... }, ... ]   # Learning read-backs only
    JSON
    

    Repeat with --recommended "CourseType" against LearningCourse read-backs only. Identify courses with missing descriptions (Learning report) or missing types (LearningCourse report). Print data quality report (same format as step 4).

13. Detect conflicting data — if multiple sources provided different values for the same course, flag conflicts, e.g. "CHEM-301: website says X, PDF says Y — which should I use?" Wait for user decision, then update the conflicting courses.

14. Remediate gaps — if user provides additional sources to fill gaps, parse them and update the incomplete courses:

    PATCH /services/data/vXX.X/sobjects/Learning/<id>
    {"Description": "..."}
    
    PATCH /services/data/vXX.X/sobjects/LearningCourse/<id>
    {"CourseType": "<parsed-course-type>"}
    

Phase 5 — Summary and Verification

15. Generate import summary — print final report:

    Course catalog import finished!
    Summary:
    • X Learning + X LearningCourse records created
    • Y standalone, Z linked lab courses (IsLinkedOnly)
    • All courses linked to <Business Account name>, set Active
    Data sources used: 1. <primary> 2. <supplementary 1> 3. <supplementary 2>
    

    If any gaps were remediated from a supplementary source (step 5/14 ran), the final report MUST also include, in this order:

    • Data Quality Report — two tables, "Initial data (from <primary source>)" and "After supplementary source (<supplementary source>)", each listing every field from step 4 with its present/total count. This is the before/after pair, not just the final counts.
    • Gap Remediation — grouped by field (e.g. "Courses with missing descriptions (filled from <source>)"), a numbered list of every remediated course with the value it received. One list per remediated field.

16. Offer verification — ask user: "Would you like me to query specific courses to verify the data? For example, I can show you a course with its LearningCourse child record, or check for any remaining data quality issues."

17. Check the running user can actually see what was just created — don't just flag this, check it (see references/gotchas.md for the resolve-running-user + FLS/RecordTypeVisibility pattern), against every object touched (Learning, LearningCourse, Account if Person Accounts came up). Name any gap and its profile, offer the specific fix, and wait for confirmation before writing Profile/PermissionSet metadata. Default-Private OWD/sharing is org-wide, not per-user — flag it, don't offer to change it.

Rules / Constraints

Gotchas

See references/gotchas.md for known failure modes and resolutions (parsing quirks, missing Business Account, picklist mapping, partial Composite failures, and more).

Output Expectations

Deliverables:

• X Learning records created (Type = 'LearningCourse')
• X LearningCourse child records created (LearningId lookups)
• Data quality report showing completeness per field
• Import summary listing all data sources used
• Optional: course verification via direct record reads or SOQL /query (both routable via headless-360)

No files are produced — all output is org records created via sObject REST API.

Cross-Skill Integration

Reference File Index

• references/gotchas.md — known failure modes and their resolutions
• references/gotchas-detail.md — lower-frequency gotchas, linked from gotchas.md
• references/large-catalog-handling.md — rules for >200 courses or >20-page source docs: paginated reads, modality-duplicate detection, checkpointing, gated parallel fan-out (steps 2, 6, 7)
• scripts/resolve_api_version.py — extracts org's max API version from the headless-360 endpoint map (Execution Transport)
• scripts/validate_completeness.py — required/recommended field presence check (steps 3, 12)
• scripts/detect_linked_courses.py — lecture/lab course-code pairing (step 6)
• scripts/detect_modality_variants.py — flags CourseCode rows differing by Duration/DurationUnit/Description, for large-catalog-handling.md §2
• scripts/batch_courses.py — Composite API batch splitting (step 7)
• scripts/split_course_code.py — SubjectAbbreviation/CourseNumber splitting (step 9)