Expert System Engineer

Expert System Engineering

Full lifecycle methodology for engineering knowledge-based expert systems. Upstream phases use CommonKADS for organizational scoping and conceptual modeling. Downstream phases use generalized MYCIN/EMYCIN patterns for rule-based implementation. Follow all phases in order.

For domain terminology, see terminology.md.

Phase 0: Suitability Assessment

Determine whether the problem warrants an expert system before committing to the methodology.

1. Classify the problem type:

   • Analytic (classification, diagnosis, assessment, monitoring) — system interprets existing data
   • Synthetic (configuration, planning, scheduling, design) — system constructs a solution from components
   • Neither — redirect to a different approach

2. Evaluate expert-system fit against alternatives:

   Criterion	Expert System Fit	Alternative
   Domain has identifiable human experts	Required	If no experts → data-driven ML
   Knowledge is heuristic, judgmental	Strong fit	If purely algorithmic → conventional software
   Reasoning must be explainable	Strong fit	If black-box acceptable → ML/statistical
   Domain is bounded and well-scoped	Required	If open-ended → LLM-based approaches
   Decisions involve uncertainty	Strong fit (CF/rules)	If precise probabilities needed → Bayesian networks
   Training data is limited	Advantage over ML	If large labeled datasets exist → ML

3. Make an explicit go/no-go decision. Document the rationale.

Completion gate

• Problem type classified (analytic or synthetic)
• Fit evaluated against at least three alternatives
• Go/no-go decision documented with rationale

Phase 1: Organizational Scoping & Feasibility

Scrutinize the organizational environment, isolate the knowledge bottleneck, and confirm feasibility before technical work begins.

1. Map organizational context (OM-1): Document business mission, drivers, problems, opportunities, and candidate solution directions.

2. Analyze variant aspects (OM-2): Map organizational structure, business processes (activity diagrams), people, resources, and culture/power dynamics.

3. Decompose into tasks (OM-3): Break business processes into discrete tasks. For each, record the performing agent, knowledge assets used, knowledge-intensity (1–5), and business significance.

4. Assess knowledge assets (OM-4): For each asset, evaluate whether it is in the right form, right place, right time, and right quality.

5. Produce feasibility decision (OM-5): Score business, technical, and project feasibility. Identify the specific knowledge bottleneck the system will address.

Completion gate

• Knowledge bottleneck identified and scoped
• Feasibility scored across all three dimensions
• Project focus confirmed (which task, which knowledge asset)

Phase 2: Task Analysis & Knowledge Acquisition

Analyze the target task and begin eliciting domain expertise.

Task Analysis

1. Characterize the task (TM-1): Specify goal, value-add, inputs, outputs, control structure, resources, and quality criteria.

2. Identify knowledge bottlenecks (TM-2): For each knowledge item the task requires, assess nature (heuristic, formal, procedural, tacit), availability, and the specific bottleneck.

3. Model agents (AM-1): For each agent involved, catalogue competencies, responsibilities, communication links, and constraints.

Knowledge Elicitation

4. Select elicitation technique matched to the target knowledge type:

   Target Knowledge	Technique
   Orientation, scope, terminology	Unstructured interview
   Heuristic rules, decision criteria	Structured interview (probes P1–P6)
   Dynamic reasoning, problem-solving trace	Protocol analysis (think-aloud)
   Concept hierarchies, taxonomies	Laddering
   Latent dimensions, attribute discovery	Concept sorting / repertory grids

5. Execute elicitation sessions with recording and realistic problem scenarios.

6. Code transcripts against the knowledge model structure: mark up concepts, attributes, values, inferences, and candidate rules.

7. Validate findings with the domain expert using teach-back.

Completion gate

• Task goal, inputs, outputs, and quality criteria specified
• Knowledge bottlenecks identified with nature and availability
• At least one elicitation session executed and coded
• Findings validated with domain expert

Phase 3: Conceptual Knowledge Modeling

Construct an implementation-independent model of domain, inference, and task knowledge. Model before rules — do not encode rules until this model is stable.

For detailed construction procedures and the full task template catalog, see knowledge-modeling.md.

Domain Layer

1. Build the domain schema: Identify concepts, attributes, value types, relations, and subtype hierarchies. This is the static structure of the domain — independent of any reasoning task.

2. Define rule types: Specify logical dependencies between domain expressions (antecedent → consequent) with cardinality and connection type.

Inference Layer

3. Specify inferences: Define each primitive reasoning step as a declarative specification with named input and output knowledge roles. Inferences are black boxes — they must not contain internal control flow.

4. Map knowledge roles to domain: Connect dynamic roles (run-time inputs/outputs) and static roles (stable knowledge base references) to domain schema elements.

5. Define transfer functions: Specify interaction points with external agents — obtain (system asks), receive (agent volunteers), present (system reports), provide (agent requests).

Task Layer

6. Select a task template matching the problem type:

   • Analytic: Classification, Assessment, Diagnosis, Monitoring
   • Synthetic: Configuration Design, Assignment, Planning, Scheduling

7. Specify task methods: Define how each task decomposes into subtasks and inferences, with explicit control structures (sequence, iteration, selection).

Validation

8. Walk through scenarios: Trace at least 2 realistic cases through the complete model (domain → inference → task) on paper. Verify no missing inferences, unmapped roles, or dead-end paths.

Completion gate

• Domain schema specified (concepts, relations, rule types)
• Inference layer specified (all inferences, knowledge role mappings)
• Task template selected, methods defined with control structures
• At least 2 scenarios traced through the full model without gaps

Phase 4: Knowledge Representation & Rule Engineering

Choose a representation and encode the conceptual model into executable knowledge structures.

For detailed rule engineering patterns (context trees, parameter types, certainty factor calculus, rule syntax, backward chaining mechanics), see rule-engineering.md.

Representation Selection

1. Choose representation based on the knowledge model:

   Knowledge Characteristics	Representation
   Heuristic associations, evidential reasoning	Production rules + certainty factors
   Structured entities with inheritance	Frames / object hierarchies
   Taxonomic relationships, shared vocabulary	Ontologies / semantic networks
   Tabular decision logic, bounded conditions	Decision tables
   Similarity-based reasoning	Case libraries
   Multiple knowledge types	Hybrid combination

Rule-Based Encoding (when production rules selected)

2. Define the domain entity hierarchy: Organize entities into a context tree with parent-child relationships and specify properties for each node.

3. Classify domain parameters: For each attribute, specify value type and bounds, exclusivity (single-valued / multi-valued / binary), and acquisition method (ask user, deduce from rules, or both).

4. Encode production rules: Each rule is a standalone modular conditional — a conjunction of predicate functions over parameters concluding a parameter value with a certainty weight.

5. Assign certainty factors: Attach evidential strength CF ∈ [-1.0, +1.0] to each rule action. Positive values increase belief; negative values increase disbelief.

6. Build knowledge tables: Consolidate repetitive multi-rule mappings into static lookup tables accessed via standardized predicates.

7. Index rules: Compile cross-reference indices — UPDATED-BY (rules concluding about each parameter) and LOOKAHEAD (rules referencing each parameter in their premise).

Completion gate

• Representation selected with rationale
• Domain entity hierarchy defined
• All parameters classified with types and value bounds
• Rules encoded in modular form with certainty factors
• Cross-reference indices compiled

Phase 5: Verification & Validation

Verify the knowledge base for consistency and completeness before deployment.

Static Verification

1. Partition rules into clusters concluding about the same parameter within the same context.

2. Build condition-action matrices: Enumerate all condition value combinations and map to concluded action values.

3. Run consistency checks:

   • Conflict: Identical conditions → contradictory conclusions
   • Redundancy: Identical conditions → identical conclusions
   • Subsumption: Rule A's conditions ⊂ Rule B's conditions, same conclusion

4. Run completeness checks: Identify condition combinations with no rule mapping. Determine if each gap is a genuine missing rule or an impossible domain state.

5. Resolve anomalies with the domain expert: distinguish real bugs from intentional domain shortcuts.

Dynamic Validation

6. Assemble a test case library with known expert-determined outcomes.

7. Run consultation traces and compare outputs against expert determinations.

8. Trace failures: For each discrepancy, unwind the reasoning chain to identify the root cause (missing rule, incorrect CF, wrong inference path, unmapped parameter).

Completion gate

• Zero unresolved rule conflicts
• Missing rule combinations reviewed and resolved
• Test case library exercised with documented results
• Failure root causes identified and addressed

Phase 6: Design, Implementation & Explanation

Map the verified knowledge model into a working system with explanation and maintenance facilities.

Architecture

1. Apply structure-preserving design: The domain/inference/task distinctions from the knowledge model must appear explicitly in the software architecture. Flattening them destroys transparency, maintainability, and explanation capability.

2. Decompose via Model-View-Controller:

   • Model: Knowledge base access, inference execution, task control
   • View: User interface, explanation display
   • Controller: Session management, event dispatch, dialogue flow

3. Design the inference engine:

   • Select chaining strategy (backward / forward / hybrid)
   • Implement rule preview (discard rules with known-false premises before evaluation)
   • Implement threshold guardrails (halt low-yield branches when premise tally ≤ 0.2)
   • Partition self-referencing rules for deferred execution

Explanation Facility

4. Record reasoning trace: Log the full goal stack, rules evaluated, evidence contributions, and conclusions during each consultation (history tree).

5. Support interactive explanation:

   • WHY: Unwind one level up the goal stack — show current goal, parent goal, linking rule, and premise status
   • HOW: Descend one level into a subgoal — show rules evaluated and evidence tallies

6. Support retrospective queries: Parse post-consultation questions to retrieve reasoning traces, rule citations, and data provenance.

Maintenance Loop

7. Interactive knowledge acquisition: When a test case fails, trace the history tree to locate the fault, solicit a new rule from the expert, validate against existing rules (subsumption, conflict, tautology), and rerun the test library.

8. Case-based refinement: Failed cases feed back into elicitation (Phase 2), model refinement (Phase 3), and re-verification (Phase 5).

Completion gate

• Architecture preserves knowledge model structure
• Inference engine handles chaining, pruning, thresholds, and self-referencing rules
• Explanation facility supports WHY, HOW, and retrospective queries
• Knowledge acquisition loop operational for incremental refinement

Worked Examples

For worked examples illustrating each phase (housing eligibility assessment, bacterial infection diagnosis, backward chaining traces, interactive explanation dialogues, and knowledge acquisition sessions), see examples.md.