copyleftdev James Bach Exploratory Testing Style Guide
Overview
James Bach is a pioneer of exploratory testing and co-founder of the context-driven testing movement. A self-taught software tester who became one of the most influential voices in the field, he advocates for testing as a skilled intellectual activity rather than a mechanical process. His Rapid Software Testing methodology, developed with Michael Bolton, emphasizes simultaneous learning, test design, and execution.
Core Philosophy
"Testing is not a phase. Testing is not a checklist. Testing is the infinite process of comparing a product to what it ought to be."
"Exploratory testing is simultaneous learning, test design, and test execution."
"A good tester is not a person who follows scripts. A good tester is a person who can think."
Bach rejects the notion that testing can be reduced to following predetermined steps. Real testing requires sapient (thinking) humans who adapt their approach based on what they discover. The tester's mind is the primary testing tool.
Design Principles
Context Drives Practice: There is no universal best practice—only practices that fit the context.
Heuristics, Not Rules: Use fallible methods that usually work, but remain aware they can fail.
Skill Over Process: Invest in tester skill development, not just test process documentation.
Oracles Are Human Judgments: We recognize problems through principles and heuristics, not just requirements.
Testing Is Investigation: Approach testing as a detective, not a factory worker.
The Seven Principles of Context-Driven Testing
1. The value of any practice depends on its context.
2. There are good practices in context, but no best practices.
3. People, working together, are the most important part of any project's context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn't solved, the product doesn't work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill can we do the right things at the right times.
Heuristic Test Strategy Model (HTSM)
Bach's framework for thinking about test strategy:
┌─────────────────────────────────────────────────────────────┐
│ PROJECT ENVIRONMENT │
│ Customers, Information, Developer Relations, Test Team, │
│ Equipment & Tools, Schedule, Test Items, Deliverables │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PRODUCT ELEMENTS │
│ Structure, Function, Data, Platform, Operations │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ QUALITY CRITERIA │
│ Capability, Reliability, Usability, Charisma, │
│ Security, Scalability, Compatibility, Performance, │
│ Installability, Maintainability │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ TEST TECHNIQUES │
│ Function, Domain, Stress, Flow, Scenario, Claims, │
│ User, Risk, Automatic │
└─────────────────────────────────────────────────────────────┘
When Testing
Always
- Begin with a testing charter or mission
- Document your mental model of the product
- Use oracles to identify potential problems
- Take notes during sessions (not after)
- Time-box exploratory sessions (60-90 minutes)
- Debrief after sessions to capture learnings
- Question requirements—they are often incomplete
Never
- Follow scripts blindly without thinking
- Assume requirements are complete or correct
- Confuse test execution with testing
- Believe that passing tests means quality
- Stop exploring when you find one bug
- Treat automation as a replacement for thinking
- Assume absence of evidence is evidence of absence
Prefer
- Questions over assumptions
- Exploration over confirmation
- Learning over procedure
- Skill over certification
- Models over checklists
- Collaboration over documentation
- Oracles over expected results
Code Patterns
Session-Based Test Management
class ExploratorySession:
"""
A time-boxed exploratory testing session.
Bach's SBTM: structured freedom for skilled testers.
"""
def __init__(self,
charter: str,
duration_minutes: int = 90,
tester: str = None):
self.charter = charter
self.duration = duration_minutes
self.tester = tester
self.start_time = None
self.end_time = None
self.notes = []
self.bugs = []
self.questions = []
self.risks = []
self.areas_explored = []
self.session_metrics = {}
def start(self):
"""Begin the session."""
self.start_time = datetime.now()
self.log(f"Session started. Charter: {self.charter}")
def log(self, observation: str, category: str = 'note'):
"""
Log observations during the session.
Categories: note, bug, question, risk, idea
"""
entry = {
'timestamp': datetime.now(),
'elapsed': self.elapsed_minutes(),
'category': category,
'content': observation
}
self.notes.append(entry)
if category == 'bug':
self.bugs.append(entry)
elif category == 'question':
self.questions.append(entry)
elif category == 'risk':
self.risks.append(entry)
def elapsed_minutes(self) -> float:
"""Time elapsed since session start."""
if not self.start_time:
return 0
return (datetime.now() - self.start_time).total_seconds() / 60
def remaining_minutes(self) -> float:
"""Time remaining in session."""
return max(0, self.duration - self.elapsed_minutes())
def end(self):
"""
End the session and calculate metrics.
"""
self.end_time = datetime.now()
self.session_metrics = {
'duration_planned': self.duration,
'duration_actual': self.elapsed_minutes(),
'bugs_found': len(self.bugs),
'questions_raised': len(self.questions),
'risks_identified': len(self.risks),
'areas_explored': len(self.areas_explored),
'note_count': len(self.notes)
}
self.log("Session ended.")
return self.generate_report()
def generate_report(self) -> SessionReport:
"""
Generate session report for debrief.
"""
return SessionReport(
charter=self.charter,
tester=self.tester,
duration=self.session_metrics['duration_actual'],
bugs=self.bugs,
questions=self.questions,
risks=self.risks,
areas=self.areas_explored,
notes=self.notes,
metrics=self.session_metrics
)
class TestCharter:
"""
A testing charter defines the mission for an exploratory session.
Format: Explore <target> with <resources> to discover <information>
"""
def __init__(self,
target: str,
resources: List[str],
information_sought: str,
time_box: int = 90,
priority: str = 'medium'):
self.target = target
self.resources = resources
self.information = information_sought
self.time_box = time_box
self.priority = priority
def __str__(self) -> str:
return (f"Explore {self.target} "
f"with {', '.join(self.resources)} "
f"to discover {self.information}")
@classmethod
def from_risk(cls, risk: Risk, product_area: str) -> 'TestCharter':
"""
Generate charter from identified risk.
"""
return cls(
target=product_area,
resources=['boundary analysis', 'error guessing', risk.related_technique],
information_sought=f"whether {risk.description} can occur",
priority=risk.severity
)
Oracle Heuristics
class OracleHeuristics:
"""
Oracles: principles or mechanisms by which we recognize problems.
Bach's FEW HICCUPPS mnemonic for consistency oracles.
"""
# FEW HICCUPPS - Consistency Heuristics
CONSISTENCY_ORACLES = {
'Familiarity': 'Consistent with what testers have seen before',
'Explainability': 'Consistent with what can be explained/documented',
'World': 'Consistent with how the real world works',
'History': 'Consistent with past versions of the product',
'Image': 'Consistent with the organization's desired image',
'Comparable_Products': 'Consistent with similar products',
'Claims': 'Consistent with documentation, ads, specs',
'User_Expectations': 'Consistent with what users want/expect',
'Product': 'Consistent with itself (internal consistency)',
'Purpose': 'Consistent with the explicit/implicit purpose',
'Statutes': 'Consistent with laws, regulations, standards',
}
def __init__(self):
self.oracle_applications = []
def apply_oracle(self,
observation: str,
oracle_type: str,
expected_consistency: str,
actual_behavior: str) -> OracleResult:
"""
Apply an oracle to evaluate observed behavior.
"""
is_consistent = self.evaluate_consistency(
expected_consistency,
actual_behavior
)
result = OracleResult(
oracle=oracle_type,
observation=observation,
expected=expected_consistency,
actual=actual_behavior,
consistent=is_consistent,
confidence=self.assess_confidence(oracle_type),
notes=[]
)
self.oracle_applications.append(result)
return result
def evaluate_consistency(self, expected: str, actual: str) -> bool:
"""
This requires human judgment—cannot be fully automated.
Returns a preliminary assessment.
"""
# This is where human thinking is essential
# Automation can only flag for review
return None # Requires human judgment
def assess_confidence(self, oracle_type: str) -> str:
"""
Some oracles are more reliable than others.
"""
high_confidence = ['Claims', 'Statutes', 'Product']
medium_confidence = ['User_Expectations', 'Comparable_Products', 'History']
low_confidence = ['Familiarity', 'World', 'Image']
if oracle_type in high_confidence:
return 'HIGH'
elif oracle_type in medium_confidence:
return 'MEDIUM'
else:
return 'LOW'
def generate_test_ideas(self, product_element: str) -> List[str]:
"""
Generate test ideas by applying each oracle.
"""
ideas = []
for oracle, description in self.CONSISTENCY_ORACLES.items():
ideas.append(
f"Test {product_element} for {oracle} consistency: "
f"Is it {description.lower()}?"
)
return ideas
class ProductCoverageModel:
"""
Model the product to guide exploration.
SFDPOT: Structure, Function, Data, Platform, Operations, Time
"""
ELEMENTS = {
'Structure': [
'Code modules',
'Database schemas',
'File systems',
'Configuration files',
'Dependencies',
'Interfaces'
],
'Function': [
'Features',
'Error handling',
'Calculations',
'Transformations',
'Workflows',
'Business rules'
],
'Data': [
'Input data',
'Output data',
'Stored data',
'Data states',
'Data flows',
'Data constraints'
],
'Platform': [
'Operating systems',
'Browsers',
'Hardware',
'Networks',
'External services',
'Configurations'
],
'Operations': [
'User scenarios',
'Startup/shutdown',
'Recovery',
'Maintenance',
'Updates',
'Integration'
],
'Time': [
'Concurrent usage',
'Timeouts',
'Date/time handling',
'Scheduling',
'Sequences',
'Race conditions'
]
}
def __init__(self, product_name: str):
self.product = product_name
self.coverage_map = {element: [] for element in self.ELEMENTS}
self.explored_areas = set()
def map_product(self, element: str, specifics: List[str]):
"""
Map specific product elements for coverage tracking.
"""
if element in self.coverage_map:
self.coverage_map[element].extend(specifics)
def mark_explored(self, element: str, specific: str, session_id: str):
"""
Mark an area as explored.
"""
self.explored_areas.add((element, specific, session_id))
def coverage_gaps(self) -> Dict[str, List[str]]:
"""
Identify areas not yet explored.
"""
gaps = {}
explored_specifics = {(e, s) for e, s, _ in self.explored_areas}
for element, specifics in self.coverage_map.items():
unexplored = [s for s in specifics if (element, s) not in explored_specifics]
if unexplored:
gaps[element] = unexplored
return gaps
def suggest_charters(self) -> List[str]:
"""
Suggest test charters based on coverage gaps.
"""
charters = []
gaps = self.coverage_gaps()
for element, specifics in gaps.items():
for specific in specifics[:3]: # Top 3 per element
charters.append(
f"Explore {specific} ({element}) to discover potential problems"
)
return charters
Test Heuristics
class TestHeuristics:
"""
Heuristics are fallible methods for solving problems.
Bach's testing heuristics for generating test ideas.
"""
# ZOMBIE heuristic for boundaries
ZOMBIE = {
'Zero': 'Test with zero, empty, null, none',
'One': 'Test with one, single, first, minimum',
'Many': 'Test with many, multiple, maximum, large',
'Boundary': 'Test at boundaries, edges, limits',
'Interface': 'Test at interfaces, handoffs, integrations',
'Exceptions': 'Test error conditions, invalid inputs, edge cases'
}
# CRUSSPIC STMPL for quality criteria
QUALITY_CRITERIA = {
'Capability': 'Can it perform its functions?',
'Reliability': 'Will it work consistently?',
'Usability': 'Can real users use it?',
'Security': 'Is it protected from threats?',
'Scalability': 'Does it handle growth?',
'Performance': 'Is it fast enough?',
'Installability': 'Can it be deployed?',
'Compatibility': 'Does it work with other things?',
'Supportability': 'Can it be maintained?',
'Testability': 'Can it be tested effectively?',
'Maintainability': 'Can it be changed?',
'Portability': 'Does it work in different environments?',
'Localizability': 'Can it be adapted for different locales?'
}
def generate_boundary_tests(self, variable: str, data_type: str) -> List[str]:
"""
Generate test ideas using ZOMBIE heuristic.
"""
tests = []
if data_type == 'numeric':
tests.extend([
f"{variable} = 0 (Zero)",
f"{variable} = 1 (One)",
f"{variable} = -1 (Negative one)",
f"{variable} = MAX_INT (Many/Boundary)",
f"{variable} = MIN_INT (Many/Boundary)",
f"{variable} = MAX + 1 (Boundary overflow)",
f"{variable} = decimal value (Interface)",
f"{variable} = null/undefined (Exception)",
f"{variable} = NaN (Exception)",
])
elif data_type == 'string':
tests.extend([
f"{variable} = '' (Zero/Empty)",
f"{variable} = single char (One)",
f"{variable} = very long string (Many)",
f"{variable} = max length (Boundary)",
f"{variable} = max + 1 (Boundary)",
f"{variable} = special characters (Interface)",
f"{variable} = unicode (Interface)",
f"{variable} = null (Exception)",
f"{variable} = SQL injection (Exception/Security)",
])
elif data_type == 'collection':
tests.extend([
f"{variable} = [] empty (Zero)",
f"{variable} = [1 item] (One)",
f"{variable} = [many items] (Many)",
f"{variable} = [max items] (Boundary)",
f"{variable} = [duplicate items] (Interface)",
f"{variable} = null (Exception)",
f"{variable} = nested collections (Interface)",
])
return tests
def generate_quality_tests(self, feature: str) -> Dict[str, List[str]]:
"""
Generate test ideas for each quality criterion.
"""
tests = {}
for criterion, question in self.QUALITY_CRITERIA.items():
tests[criterion] = [
f"{feature}: {question}",
f"What would make {feature} fail {criterion.lower()}?",
f"How would a user judge {feature}'s {criterion.lower()}?",
]
return tests
def soap_opera_testing(self, workflow: str) -> List[str]:
"""
Soap Opera Testing: dramatic, complex, intertwined scenarios.
Bach's technique for finding interaction bugs.
"""
return [
f"Start {workflow}, interrupt midway, start another",
f"Run {workflow} with minimum permissions",
f"Run {workflow} with conflicting concurrent users",
f"Start {workflow}, lose connection, reconnect",
f"Run {workflow} with corrupted cache/state",
f"Complete {workflow}, immediately undo, redo",
f"Run {workflow} at system resource limits",
f"Interleave {workflow} with upgrade/migration",
]
def touring_heuristics(self, application: str) -> Dict[str, str]:
"""
Touring: systematic exploration strategies.
Different "tours" reveal different problems.
"""
return {
'Guidebook Tour': f"Follow the documentation exactly for {application}",
'Money Tour': f"Test the features that sell {application}",
'Landmark Tour': f"Navigate between major features of {application}",
'Intellectual Tour': f"Find the most complex parts of {application}",
'FedEx Tour': f"Follow data through {application} end to end",
'Garbage Tour': f"Look for dead code and unused features in {application}",
'Bad Neighborhood Tour': f"Focus on historically buggy areas of {application}",
'Museum Tour': f"Test legacy code and old features in {application}",
'Back Alley Tour': f"Test least-used features of {application}",
'All-Nighter Tour': f"Test {application} over extended time periods",
}
Rapid Software Testing Session
class RapidTestingSession:
"""
Rapid Software Testing: Bach & Bolton's methodology.
Testing as a performance, not a procedure.
"""
def __init__(self,
product: str,
tester: 'SkilledTester',
stakeholder_info: Dict):
self.product = product
self.tester = tester
self.stakeholder = stakeholder_info
self.mental_model = ProductCoverageModel(product)
self.oracles = OracleHeuristics()
self.heuristics = TestHeuristics()
self.sessions = []
self.findings = []
def analyze_context(self) -> ContextAnalysis:
"""
Understand the project context before testing.
"""
return ContextAnalysis(
who_is_the_customer=self.stakeholder.get('customer'),
what_is_the_mission=self.stakeholder.get('mission'),
what_does_quality_mean=self.stakeholder.get('quality_definition'),
what_threatens_quality=self.stakeholder.get('risks'),
what_resources_exist=self.stakeholder.get('resources'),
what_constraints_exist=self.stakeholder.get('constraints'),
what_has_been_done=self.stakeholder.get('prior_testing'),
)
def create_test_strategy(self, context: ContextAnalysis) -> TestStrategy:
"""
Build a test strategy based on context.
No best practices—only practices that fit this context.
"""
strategy = TestStrategy(product=self.product)
# Prioritize based on risks
for risk in context.what_threatens_quality:
strategy.add_focus_area(
area=risk.area,
priority=risk.severity,
techniques=self.select_techniques(risk),
time_allocation=self.estimate_time(risk)
)
# Generate charters for each focus area
for area in strategy.focus_areas:
charters = self.generate_charters(area)
strategy.add_charters(area.name, charters)
return strategy
def select_techniques(self, risk: Risk) -> List[str]:
"""
Select testing techniques based on risk characteristics.
"""
techniques = []
if risk.involves_boundaries:
techniques.append('Boundary Analysis (ZOMBIE)')
if risk.involves_user_interaction:
techniques.append('Scenario Testing')
techniques.append('Touring')
if risk.involves_data:
techniques.append('Data Flow Testing')
techniques.append('CRUD Testing')
if risk.involves_integration:
techniques.append('Interface Testing')
techniques.append('Soap Opera Testing')
if risk.involves_time:
techniques.append('Stress Testing')
techniques.append('Longevity Testing')
# Always include exploratory
techniques.append('Exploratory Testing')
return techniques
def run_session(self, charter: TestCharter) -> SessionReport:
"""
Run a time-boxed exploratory testing session.
"""
session = ExploratorySession(
charter=str(charter),
duration_minutes=charter.time_box,
tester=self.tester.name
)
session.start()
# The actual testing happens here—human cognition
# This framework supports but doesn't replace the tester
self.sessions.append(session)
return session # Tester controls when to end
def debrief(self, session: ExploratorySession) -> DebriefNotes:
"""
Post-session debrief: capture learnings and adjust strategy.
"""
report = session.generate_report()
debrief = DebriefNotes(
session_id=id(session),
what_was_tested=session.areas_explored,
what_was_found=session.bugs,
what_was_learned=self.extract_learnings(session),
what_should_change=self.recommend_changes(session),
new_questions=session.questions,
new_risks=session.risks,
)
# Update mental model based on session
for area in session.areas_explored:
self.mental_model.mark_explored(
area['element'],
area['specific'],
str(id(session))
)
return debrief
def extract_learnings(self, session: ExploratorySession) -> List[str]:
"""
What did we learn about the product?
"""
learnings = []
for note in session.notes:
if 'learned' in note['content'].lower():
learnings.append(note['content'])
if 'realized' in note['content'].lower():
learnings.append(note['content'])
if 'discovered' in note['content'].lower():
learnings.append(note['content'])
return learnings
def recommend_changes(self, session: ExploratorySession) -> List[str]:
"""
What should we do differently next?
"""
recommendations = []
if session.session_metrics['bugs_found'] > 3:
recommendations.append(
"High bug density—consider deeper exploration of this area"
)
if session.session_metrics['questions_raised'] > 5:
recommendations.append(
"Many questions—schedule stakeholder discussion"
)
return recommendations
Mental Model
Bach approaches testing by asking:
- What is the mission? Who cares, and what do they need to know?
- What could go wrong? Risks drive test focus
- How will I recognize a problem? Which oracles apply?
- What have I covered? Maintain a mental model of the product
- What did I learn? Each test teaches something
The Session Checklist
□ Charter defined (Explore X with Y to discover Z)
□ Time-box set (60-90 minutes typical)
□ Oracles identified (how will I recognize problems?)
□ Note-taking ready (real-time, not after)
□ Product model in mind (SFDPOT coverage)
□ Heuristics available (ZOMBIE, touring, soap opera)
□ Questions captured (for stakeholder follow-up)
□ Debrief scheduled (capture learnings immediately)
Signature Bach Moves
- Session-Based Test Management (SBTM)
- Heuristic Test Strategy Model (HTSM)
- FEW HICCUPPS consistency oracles
- SFDPOT product coverage model
- Exploratory testing charters
- Context-driven approach (no best practices)
- Soap Opera Testing for complex scenarios
- Touring heuristics for systematic exploration