Common Methodological and Statistical Issues in Scientific Manuscripts

Common Methodological and Statistical Issues in Scientific Manuscripts

This document catalogs frequent issues encountered during peer review, organized by category. Use this as a reference to identify potential problems and provide constructive feedback.

Statistical Issues

1. P-Value Misuse and Misinterpretation

Common Problems:

• P-hacking (selective reporting of significant results)
• Multiple testing without correction (familywise error rate inflation)
• Interpreting non-significance as proof of no effect
• Focusing exclusively on p-values without effect sizes
• Dichotomizing continuous p-values at arbitrary thresholds (p=0.049 vs p=0.051)
• Confusing statistical significance with biological/clinical significance

How to Identify:

• Suspiciously high proportion of p-values just below 0.05
• Many tests performed but no correction mentioned
• Statements like "no difference was found" from non-significant results
• No effect sizes or confidence intervals reported
• Language suggesting p-values indicate strength of effect

What to Recommend:

• Report effect sizes with confidence intervals
• Apply appropriate multiple testing corrections (Bonferroni, FDR, Holm-Bonferroni)
• Interpret non-significance cautiously (lack of evidence ≠ evidence of lack)
• Pre-register analyses to avoid p-hacking
• Consider equivalence testing for "no difference" claims

2. Inappropriate Statistical Tests

Common Problems:

• Using parametric tests when assumptions are violated (non-normal data, unequal variances)
• Analyzing paired data with unpaired tests
• Using t-tests for multiple groups instead of ANOVA with post-hoc tests
• Treating ordinal data as continuous
• Ignoring repeated measures structure
• Using correlation when regression is more appropriate

How to Identify:

• No mention of assumption checking
• Small sample sizes with parametric tests
• Multiple pairwise t-tests instead of ANOVA
• Likert scales analyzed with t-tests
• Time-series data analyzed without accounting for repeated measures

What to Recommend:

• Check assumptions explicitly (normality tests, Q-Q plots)
• Use non-parametric alternatives when appropriate
• Apply proper corrections for multiple comparisons after ANOVA
• Use mixed-effects models for repeated measures
• Consider ordinal regression for ordinal outcomes

3. Sample Size and Power Issues

Common Problems:

• No sample size justification or power calculation
• Underpowered studies claiming "no effect"
• Post-hoc power calculations (which are uninformative)
• Stopping rules not pre-specified
• Unequal group sizes without justification

How to Identify:

• Small sample sizes (n<30 per group for typical designs)
• No mention of power analysis in methods
• Statements about post-hoc power
• Wide confidence intervals suggesting imprecision
• Claims of "no effect" with large p-values and small n

What to Recommend:

• Conduct a priori power analysis based on expected effect size
• Report achieved power or precision (confidence interval width)
• Acknowledge when studies are underpowered
• Consider effect sizes and confidence intervals for interpretation
• Pre-register sample size and stopping rules

4. Missing Data Problems

Common Problems:

• Complete case analysis without justification (listwise deletion)
• Not reporting extent or pattern of missingness
• Assuming data are missing completely at random (MCAR) without testing
• Inappropriate imputation methods
• Not performing sensitivity analyses

How to Identify:

• Different n values across analyses without explanation
• No discussion of missing data
• Participants "excluded from analysis"
• Simple mean imputation used
• No sensitivity analyses comparing complete vs. imputed data

What to Recommend:

• Report extent and patterns of missingness
• Test MCAR assumption (Little's test)
• Use appropriate methods (multiple imputation, maximum likelihood)
• Perform sensitivity analyses
• Consider intention-to-treat analysis for trials

5. Circular Analysis and Double-Dipping

Common Problems:

• Using the same data for selection and inference
• Defining ROIs based on contrast then testing that contrast in same ROI
• Selecting outliers then testing for differences
• Post-hoc subgroup analyses presented as planned
• HARKing (Hypothesizing After Results are Known)

How to Identify:

• ROIs or features selected based on results
• Unexpected subgroup analyses
• Post-hoc analyses not clearly labeled as exploratory
• No data-independent validation
• Introduction that perfectly predicts findings

What to Recommend:

• Use independent datasets for selection and testing
• Pre-register analyses and hypotheses
• Clearly distinguish confirmatory vs. exploratory analyses
• Use cross-validation or hold-out datasets
• Correct for selection bias

6. Pseudoreplication

Common Problems:

• Technical replicates treated as biological replicates
• Multiple measurements from same subject treated as independent
• Clustered data analyzed without accounting for clustering
• Non-independence in spatial or temporal data

How to Identify:

• n defined as number of measurements rather than biological units
• Multiple cells from same animal counted as independent
• Repeated measures not acknowledged
• No mention of random effects or clustering

What to Recommend:

• Define n as biological replicates (animals, patients, independent samples)
• Use mixed-effects models for nested or clustered data
• Account for repeated measures explicitly
• Average technical replicates before analysis
• Report both technical and biological replication

Experimental Design Issues

7. Lack of Appropriate Controls

Common Problems:

• Missing negative controls
• Missing positive controls for validation
• No vehicle controls for drug studies
• No time-matched controls for longitudinal studies
• No batch controls

How to Identify:

• Methods section lists only experimental groups
• No mention of controls in figures
• Unclear baseline or reference condition
• Cross-batch comparisons without controls

What to Recommend:

• Include negative controls to assess specificity
• Include positive controls to validate methods
• Use vehicle controls matched to experimental treatment
• Include sham surgery controls for surgical interventions
• Include batch controls for cross-batch comparisons

8. Confounding Variables

Common Problems:

• Systematic differences between groups besides intervention
• Batch effects not controlled or corrected
• Order effects in sequential experiments
• Time-of-day effects not controlled
• Experimenter effects not blinded

How to Identify:

• Groups differ in multiple characteristics
• Samples processed in different batches by group
• No randomization of sample order
• No mention of blinding
• Baseline characteristics differ between groups

What to Recommend:

• Randomize experimental units to conditions
• Block on known confounders
• Randomize sample processing order
• Use blinding to minimize bias
• Perform batch correction if needed
• Report and adjust for baseline differences

9. Insufficient Replication

Common Problems:

• Single experiment without replication
• Technical replicates mistaken for biological replication
• Small n justified by "typical for the field"
• No independent validation of key findings
• Cherry-picking representative examples

How to Identify:

• Methods state "experiment performed once"
• n=3 with no justification
• "Representative image shown"
• Key claims based on single experiment
• No validation in independent dataset

What to Recommend:

• Perform independent biological replicates (typically ≥3)
• Validate key findings in independent cohorts
• Report all replicates, not just representative examples
• Conduct power analysis to justify sample size
• Show individual data points, not just summary statistics

Reproducibility Issues

10. Insufficient Methodological Detail

Common Problems:

• Methods not described in sufficient detail for replication
• Key reagents not specified (vendor, catalog number)
• Software versions and parameters not reported
• Antibodies not validated
• Cell line authentication not verified

How to Identify:

• Vague descriptions ("standard protocols were used")
• No information on reagent sources
• Generic software mentioned without versions
• No antibody validation information
• Cell lines not authenticated

What to Recommend:

• Provide detailed protocols or cite specific protocols
• Include reagent vendors, catalog numbers, lot numbers
• Report software versions and all parameters
• Include antibody validation (Western blot, specificity tests)
• Report cell line authentication method (STR profiling)
• Make protocols available (protocols.io, supplementary materials)

11. Data and Code Availability

Common Problems:

• No data availability statement
• "Data available upon request" (often unfulfilled)
• No code provided for computational analyses
• Custom software not made available
• No clear documentation

How to Identify:

• Missing data availability statement
• No repository accession numbers
• Computational methods with no code
• Custom pipelines without access
• No README or documentation

What to Recommend:

• Deposit raw data in appropriate repositories (GEO, SRA, Dryad, Zenodo)
• Share analysis code on GitHub or similar
• Provide clear documentation and README files
• Include requirements.txt or environment files
• Make custom software available with installation instructions
• Use DOIs for permanent data citation

12. Lack of Method Validation

Common Problems:

• New methods not compared to gold standard
• Assays not validated for specificity, sensitivity, linearity
• No spike-in controls
• Cross-reactivity not tested
• Detection limits not established

How to Identify:

• Novel assays presented without validation
• No comparison to existing methods
• No positive/negative controls shown
• Claims of specificity without evidence
• No standard curves or controls

What to Recommend:

• Validate new methods against established approaches
• Show specificity (knockdown/knockout controls)
• Demonstrate linearity and dynamic range
• Include positive and negative controls
• Report limits of detection and quantification
• Show reproducibility across replicates and operators

Interpretation Issues

13. Overstatement of Results

Common Problems:

• Causal language for correlational data
• Mechanistic claims without mechanistic evidence
• Extrapolating beyond data (species, conditions, populations)
• Claiming "first to show" without thorough literature review
• Overgeneralizing from limited samples

How to Identify:

• "X causes Y" from observational data
• Mechanism proposed without direct testing
• Mouse data presented as relevant to humans without caveats
• Claims of novelty with missing citations
• Broad claims from narrow samples

What to Recommend:

• Use appropriate language ("associated with" vs. "caused by")
• Distinguish correlation from causation
• Acknowledge limitations of model systems
• Provide thorough literature context
• Be specific about generalizability
• Propose mechanisms as hypotheses, not conclusions

14. Cherry-Picking and Selective Reporting

Common Problems:

• Reporting only significant results
• Showing "representative" images that may not be typical
• Excluding outliers without justification
• Not reporting negative or contradictory findings
• Switching between different statistical approaches

How to Identify:

• All reported results are significant
• "Representative of 3 experiments" with no quantification
• Data exclusions mentioned in results but not methods
• Supplementary data contradicts main findings
• Multiple analysis approaches with only one reported

What to Recommend:

• Report all planned analyses regardless of outcome
• Quantify and show variability across replicates
• Pre-specify outlier exclusion criteria
• Include negative results
• Pre-register analysis plan
• Report effect sizes and confidence intervals for all comparisons

15. Ignoring Alternative Explanations

Common Problems:

• Preferred explanation presented without considering alternatives
• Contradictory evidence dismissed without discussion
• Off-target effects not considered
• Confounding variables not acknowledged
• Limitations section minimal or absent

How to Identify:

• Single interpretation presented as fact
• Prior contradictory findings not cited or discussed
• No consideration of alternative mechanisms
• No discussion of limitations
• Specificity assumed without controls

What to Recommend:

• Discuss alternative explanations
• Address contradictory findings from literature
• Include appropriate specificity controls
• Acknowledge and discuss limitations thoroughly
• Consider and test alternative hypotheses

Figure and Data Presentation Issues

16. Inappropriate Data Visualization

Common Problems:

• Bar graphs for continuous data (hiding distributions)
• No error bars or error bars not defined
• Truncated y-axes exaggerating differences
• Dual y-axes creating misleading comparisons
• Too many significant figures
• Colors not colorblind-friendly

How to Identify:

• Bar graphs with few data points
• Unclear what error bars represent (SD, SEM, CI?)
• Y-axis doesn't start at zero for ratio/percentage data
• Left and right y-axes with different scales
• Values reported to excessive precision (p=0.04562)
• Red-green color schemes

What to Recommend:

• Show individual data points with scatter/box/violin plots
• Always define error bars (SD, SEM, 95% CI)
• Start y-axis at zero or indicate breaks clearly
• Avoid dual y-axes; use separate panels instead
• Report appropriate significant figures
• Use colorblind-friendly palettes (viridis, colorbrewer)
• Include sample sizes in figure legends

17. Image Manipulation Concerns

Common Problems:

• Excessive contrast/brightness adjustment
• Spliced gels or images without indication
• Duplicated images or panels
• Uneven background in Western blots
• Selective cropping
• Over-processed microscopy images

How to Identify:

• Suspicious patterns or discontinuities
• Very high contrast with no background
• Similar features in different panels
• Straight lines suggesting splicing
• Inconsistent backgrounds
• Loss of detail suggesting over-processing

What to Recommend:

• Apply adjustments uniformly across images
• Indicate spliced gels with dividing lines
• Show full, uncropped images in supplementary materials
• Provide original images if requested
• Follow journal image integrity policies
• Use appropriate image analysis tools

Study Design Issues

18. Poorly Defined Hypotheses and Outcomes

Common Problems:

• No clear hypothesis stated
• Primary outcome not specified
• Multiple outcomes without correction
• Outcomes changed after data collection
• Fishing expeditions presented as hypothesis-driven

How to Identify:

• Introduction doesn't state clear testable hypothesis
• Multiple outcomes with unclear hierarchy
• Outcomes in results don't match those in methods
• Exploratory study presented as confirmatory
• Many tests with no multiple testing correction

What to Recommend:

• State clear, testable hypotheses
• Designate primary and secondary outcomes a priori
• Pre-register studies when possible
• Apply appropriate corrections for multiple outcomes
• Clearly distinguish exploratory from confirmatory analyses
• Report all pre-specified outcomes

19. Baseline Imbalance and Selection Bias

Common Problems:

• Groups differ at baseline
• Selection criteria applied differentially
• Healthy volunteer bias
• Survivorship bias
• Indication bias in observational studies

How to Identify:

• Table 1 shows significant baseline differences
• Inclusion criteria different between groups
• Response rate <50% with no analysis
• Analysis only includes completers
• Groups self-selected rather than randomized

What to Recommend:

• Report baseline characteristics in Table 1
• Use randomization to ensure balance
• Adjust for baseline differences in analysis
• Report response rates and compare responders vs. non-responders
• Consider propensity score matching for observational data
• Use intention-to-treat analysis

20. Temporal and Batch Effects

Common Problems:

• Samples processed in batches by condition
• Temporal trends not accounted for
• Instrument drift over time
• Different operators for different groups
• Reagent lot changes between groups

How to Identify:

• All treatment samples processed on same day
• Controls from different time period
• No mention of batch or time effects
• Different technicians for groups
• Long study duration with no temporal analysis

What to Recommend:

• Randomize samples across batches/time
• Include batch as covariate in analysis
• Perform batch correction (ComBat, limma)
• Include quality control samples across batches
• Report and test for temporal trends
• Balance operators across conditions

Reporting Issues

21. Incomplete Statistical Reporting

Common Problems:

• Test statistics not reported
• Degrees of freedom missing
• Exact p-values replaced with inequalities (p<0.05)
• No confidence intervals
• No effect sizes
• Sample sizes not reported per group

How to Identify:

• Only p-values given with no test statistics
• p-values reported as p<0.05 rather than exact values
• No measures of uncertainty
• Effect magnitude unclear
• n reported for total but not per group

What to Recommend:

• Report complete test statistics (t, F, χ², etc. with df)
• Report exact p-values (except p<0.001)
• Include 95% confidence intervals
• Report effect sizes (Cohen's d, odds ratios, correlation coefficients)
• Report n for each group in every analysis
• Consider CONSORT-style flow diagram

22. Methods-Results Mismatch

Common Problems:

• Methods describe analyses not performed
• Results include analyses not described in methods
• Different sample sizes in methods vs. results
• Methods mention controls not shown
• Statistical methods don't match what was done

How to Identify:

• Analyses in results without methodological description
• Methods describe experiments not in results
• Numbers don't match between sections
• Controls mentioned but not shown
• Different software mentioned than used

What to Recommend:

• Ensure complete concordance between methods and results
• Describe all analyses performed in methods
• Remove methodological descriptions of experiments not performed
• Verify all numbers are consistent
• Update methods to match actual analyses conducted

How to Use This Reference

When reviewing manuscripts:

1. Read through methods and results systematically
2. Check for common issues in each category
3. Note specific problems with evidence
4. Provide constructive suggestions for improvement
5. Distinguish major issues (affect validity) from minor issues (affect clarity)
6. Prioritize reproducibility and transparency

This is not an exhaustive list but covers the most frequently encountered issues. Always consider the specific context and discipline when evaluating potential problems.