The method exists to answer the question; it is chosen after the question and before the data. A method chosen for familiarity or convenience produces a thesis whose conclusions the examiner cannot accept. This skill matches each research question to a design and writes down why.
Match question type to design
| Question asks | Design | Evidence produced |
|---|---|---|
| Does X cause Y? / Is A better than B? | controlled experiment (randomised, with baselines) | effect size with confidence, under controlled conditions |
| Does X cause Y in a real setting where randomisation is impossible? | quasi-experiment (pre/post, difference-in-differences, matched groups) | effect estimate with stated confounds |
| How and why does X happen in context? | case study (single or multiple, embedded) | rich description, mechanisms, propositions |
| How prevalent is X? What do people think or do? | survey (questionnaire) | distributions, correlations, at scale |
| What is the lived experience or reasoning behind X? | interview study, focus groups, observation | themes, categories (thematic analysis, grounded theory) |
| How can we build an artifact that solves X, and does it? | design science (build and evaluate) | artifact plus evaluation against requirements |
| How can we improve practice X while studying it? | action research | cycles of change and reflection |
| What does the literature say about X? | systematic review or mapping study | synthesis with explicit protocol (systematic-review) |
| Does the finding hold across settings? | replication | confirmation or boundary conditions |
Most master's theses in computing combine two: a design-science build with an experimental evaluation, or a survey followed by interviews (explanatory sequential mixed methods). Name the combination and the order.
For each design, decide and write down
Participants, subjects, or objects: population, sampling (random, stratified, convenience with its limitation stated, purposive for interviews), sample size with the reason (power analysis for experiments via statistics-advisor; saturation for interviews; the whole population for a census of repositories).
Variables: independent (manipulated), dependent (measured), controlled, and the confounds you cannot control. Operationalise each: "usability" becomes SUS score and task completion time.
Procedure: step by step, with timing, instruments (questionnaires, tasks, scripts), and the exact conditions. A reader should be able to run it.
Instruments: validated where they exist (SUS, NASA-TLX, TAM); if you write your own, pilot it and say so.
Data: what is collected, in what format, where stored, retention, anonymisation.
Analysis plan, decided before data collection: the statistical tests (statistics-advisor) or the qualitative coding approach, the significance level, the effect size measure, how missing data is handled. Pre-register it in the proposal; deviations are reported as such.
Ethics: informed consent, approval body and reference, data protection, the right to withdraw. Studies of open-source repositories still raise questions (developer names in data).
Validity threats, using the standard four and what you did about each:
- Construct: does the measure capture the concept? (SUS measures perceived usability, not efficiency.)
- Internal: could something other than X explain Y? (learning effects, order effects, selection)
- External: to whom and what do the results generalise? (students are not professionals; one language is not all)
- Conclusion: is the statistical reasoning sound? (power, multiple comparisons, assumptions)
For qualitative work use credibility, transferability, dependability, confirmability, and the tactics (triangulation, member checking, audit trail).
Output
A design section for the method chapter, per research question:
### RQ2: Does feedback style affect fix time?
Design: between-subjects controlled experiment, two conditions (rule-based, LLM-based), random assignment.
Participants: 40 second-year students (power analysis: d = 0.8, alpha 0.05, power 0.8 gives 26 per group; 40 allows attrition), recruited via course announcement (convenience; see External validity).
Variables: IV feedback style; DV time to correct fix (seconds, from logs), secondary DV SUS; controlled: assignment, IDE, time limit.
Procedure: consent; 5-minute tutorial; three tasks in fixed order; SUS; debrief. 45 minutes.
Analysis: Welch's t-test on log-transformed time; Cohen's d; Mann-Whitney U as robustness check; alpha 0.05.
Ethics: approved by <board>, ref <n>; data pseudonymised at collection.
Threats: internal, task order fixed (mitigated by counterbalancing tasks 2 and 3); external, students; construct, fix time measures speed not learning.
Write it to the method chapter file or return it; then update research/QUESTION.md with the design per question so thesis-proposal and thesis-reviewer see the same plan.
