Liberty91LTD Feedback Loops
Feedback is Phase 6 of the CTI Hyperloop. Without it, the cycle is open-ended and quality degrades over time.
Types of Feedback
1. Consumer Feedback
Intelligence consumers (stakeholders) assess the value of products they receive.
Collection points:
- After every significant product delivery
- During quarterly PIR reviews
- During stakeholder briefings
Questions:
| Question | Response Options |
|---|---|
| Was this intelligence useful for your decision-making? | Very useful / Somewhat useful / Not useful |
| Was it delivered in time to act on? | Yes / Partially / No |
| Was the format appropriate for your needs? | Yes / Too technical / Too high-level / Wrong format |
| Was the confidence assessment helpful? | Yes / Unclear / Not included |
| What should we prioritise next? | Free text |
Actions:
- Useful + timely → Continue current approach, reinforce collection sources
- Useful + late → Improve detection/triage speed, adjust collection frequency
- Not useful → Review PIR alignment, stakeholder needs, product format
- Format wrong → Adjust per stakeholder-management skill
2. Source Quality Tracking
Track the reliability of intelligence sources over time.
For each source used in products:
- Was the information confirmed, partly confirmed, or disproven?
- Did the source's Admiralty rating need adjustment?
- Were there timeliness issues?
Actions:
- Consistently confirmed → Upgrade reliability rating (e.g., C → B)
- Inconsistent → Maintain or downgrade rating
- Repeatedly disproven → Downgrade to D or E
- Feed updates into source-assessment guidance
3. Analyst Retrospectives
After significant investigations or assessments:
Questions:
- What analytical techniques were applied? Were they effective?
- Were key assumptions validated or invalidated?
- Were there intelligence gaps that could have been filled?
- Were alternative hypotheses adequately considered?
- What would you do differently?
Actions:
- Update SOPs with lessons learned
- Adjust technique selection guidance
- Fill identified collection gaps
- Update knowledge cells with validated/invalidated assumptions
4. Detection Effectiveness
For detection rules produced by the platform:
Metrics:
- True positive rate (did the rule catch real threats?)
- False positive rate (did it generate noise?)
- Coverage (did it miss known instances?)
Actions:
- High FP → Refine rule, add exclusions
- Missed detections → Expand rule, add variants
- No hits → Verify rule logic, check data source availability
Feedback Integration
All feedback feeds back into Phase 1 (Planning & Direction):
Consumer says "not useful" → Review and adjust PIRs
Source proves unreliable → Update Admiralty ratings, adjust collection
Analyst retrospective finds gap → Create new collection tasking
Detection rule has high FP → Refine and redeploy
Tracking
The orchestrator maintains feedback state:
- Consumer feedback logged in stakeholder register
- Source quality tracked in knowledge cell Sources & References
- Retrospective outcomes logged in investigation workspace
- Detection effectiveness tracked in detection rule metadata