Psychology dissertation outlier analysis is the process of finding unusual observations, understanding why they are unusual, and testing whether they materially change a conclusion. It is not a licence to delete inconvenient scores. A defensible analysis separates errors from valid extremes, uses diagnostics suited to the planned model, and reports every consequential decision.
This guide gives you a practical workflow for questionnaire, experimental, correlational, and regression projects. It explains univariate and multivariate checks, influence statistics, robust alternatives, sensitivity analysis, and transparent reporting. Use it alongside your approved protocol, departmental guidance, and supervisor’s advice.
What counts as a Psychology Dissertation outlier in psychology research?
An outlier is an observation that is unusually distant from the pattern expected under a stated reference. That reference matters. A reaction time of 4,000 milliseconds may be extreme in a rapid lexical-decision task but entirely ordinary in a complex reasoning task. A depression score near the top of a validated scale may be rare in a community sample yet plausible in a clinical sample.
The US National Institute of Standards and Technology defines an outlier as an observation lying an abnormal distance from other values and stresses that analysts must first characterise what is normal for the data. Its guidance also warns that unusual points can contain valuable information about data collection or the process being studied. A flag therefore begins an investigation; it does not settle the case.
Table of Contents
Error, interesting case, or influential observation?
A useful first distinction is among three questions:
- Is the value erroneous? Examples include a misplaced decimal, impossible questionnaire code, duplicated participant, corrupted timestamp, or value recorded in the wrong unit.
- Is the value valid but substantively unusual? It may represent a rare participant, a meaningful subgroup, or a phenomenon that the theory should explain.
- Does the observation influence the fitted model? A valid point can strongly change a coefficient, standard error, fitted value, or conclusion. Conversely, an extreme raw score may have little influence.
Aguinis, Gottfredson, and Joo’s methodological review documented many competing definitions, detection techniques, and handling approaches. Their central lesson is that researchers must define what they mean by an outlier, combine diagnostics with substantive reasoning, and disclose their choices. A case should never be removed solely because it makes a result nonsignificant.
Plan outlier decisions before looking at results
Write an outlier plan before testing the main hypotheses, ideally in your psychology dissertation preregistration or analysis protocol. Specify valid ranges, data-quality rules, planned plots, diagnostic measures, thresholds that trigger review, treatment options, and sensitivity analyses. Planning reduces the temptation to select a rule after seeing which version gives the preferred answer.
Thresholds are screening rules, not universal laws. Their behaviour changes with sample size, distribution, number of predictors, model type, and measurement scale. State why a threshold is appropriate and what happens after a case is flagged. A transparent plan might say: “Values outside the instrument’s possible range will be checked against source records and corrected where verifiable. Valid extreme scores will remain in the primary analysis. Cases with substantial regression influence will trigger prespecified sensitivity analyses.”
| Decision stage | Question | Evidence to retain |
|---|---|---|
| Prevention | Can range checks, attention procedures, or calibrated equipment prevent errors? | Survey logic, codebook, device settings, protocol |
| Verification | Is the value impossible, miscoded, duplicated, or inconsistent with a source record? | Audit trail and correction note |
| Characterisation | Is it unusual marginally, conditionally, or multivariately? | Plots and diagnostic output |
| Influence | Does it change estimates, uncertainty, fit, or interpretation? | Influence statistics and case-deletion comparison |
| Accommodation | Can a better model or robust method represent the data without arbitrary deletion? | Model rationale and diagnostics |
| Reporting | Would a reader understand every exclusion and its effect? | Counts, rules, results, sensitivity analysis |
Start with data provenance and visual inspection
Before calculating statistics, preserve the raw data and create a reproducible cleaning copy. Check IDs, units, allowable ranges, reverse-coded items, missing-value codes, duplicate rows, timing variables, condition assignments, and equipment logs. The data collection guide explains how a codebook and quality controls support this audit.
If a raw value is impossible, trace it to the source. Correct it only when the correct value is verifiable. Otherwise mark it missing with a documented reason rather than inventing a replacement. Do not overwrite raw files. If a participant has a valid extreme score, keep their full record linked across variables so that model-based checks remain possible.
Then inspect plots. Histograms, dot plots, box plots, scatterplots, residual plots, and time-order plots reveal different features. Plot within experimental conditions when the design predicts different distributions. A score may look extreme in the pooled sample but ordinary within its assigned condition. For repeated measures, plot participant trajectories because an unusual change may matter more than an unusual level.
NIST recommends combining graphical and quantitative approaches and checking whether the assumed distribution is credible. A single rule applied blindly can confuse skewness, bounded scales, mixture distributions, or genuine subgroup differences with contamination.
Univariate outlier detection
Univariate methods examine one variable at a time. They are useful for catching impossible values and describing marginal extremes, but they do not reveal whether a case is unusual only in combination with other variables.
Box-plot fences and the interquartile range
A conventional box plot flags observations outside Q1 minus 1.5 times the interquartile range or Q3 plus 1.5 times the interquartile range. NIST describes values beyond these inner fences as mild outliers and values beyond three interquartile ranges as extreme. These labels are descriptive. They do not establish error, justify deletion, or account for the expected distribution.
Box plots are convenient for comparing conditions, but bounded or skewed psychological variables can generate many flags. A floor-heavy symptom scale or positively skewed response time distribution may be functioning exactly as expected. Interpret a box plot alongside measurement limits, design, and distribution shape.
Standard scores and their limitations
Rules based on distance from the mean in standard-deviation units are common, yet the mean and standard deviation are themselves affected by extreme values. This creates masking, where an extreme point inflates the standard deviation enough to hide itself or another point. Small samples also restrict the maximum possible z score.
Leys and colleagues recommend median absolute deviation as a more robust alternative for univariate screening because the median and MAD are less distorted by extreme observations. NIST likewise presents a modified z score based on the median and MAD. Whatever rule you use, define the scaling convention and cutoff because software functions can implement MAD differently.
Domain-specific rules
Some variables need theory-led rules. Reaction times may include anticipatory responses, lapses, and right-skewed valid observations. Accuracy proportions are bounded. Clinical measures have instrument-defined ranges. Physiological recordings may contain artefacts identifiable from device quality metrics. Use rules supported by the measurement process, not a generic threshold copied from an unrelated study.
For repeated trials, distinguish a bad trial from a bad participant. Trial-level filtering changes the participant’s summary score; participant-level exclusion changes the sample. Report both counts and denominators. If rules depend on performance, ensure they do not selectively remove a theoretically meaningful group.
Multivariate and model-based diagnostics
A participant can be unremarkable on each variable separately yet have an unusual combination of scores. For example, very high social anxiety and very high approach behaviour may both fall within valid ranges but be rare jointly. Multivariate checks evaluate that joint pattern.
Mahalanobis distance
Mahalanobis distance measures how far a case lies from the multivariate centre while accounting for correlations among variables. It is often compared with a chi-square reference using degrees of freedom equal to the number of variables. This approach assumes the reference model is appropriate and can be unstable when the covariance matrix is poorly estimated, variables are highly collinear, or the sample is small.
Use variables that belong in the stated multivariate question, not every column in the dataset. Standardise or code them consistently, inspect correlations, and document the reference probability used for flagging. A large distance identifies a candidate for review; it does not prove measurement error.
Residuals, leverage, and influence are different
In regression, three concepts answer different questions. A residual outlier has an outcome poorly predicted by the model. A high-leverage case has an unusual configuration of predictor values. An influential case changes an estimate or fitted result materially when included. One observation can have one, two, or all three properties.
- Studentised residuals screen for cases whose outcomes are far from fitted values relative to estimated residual variability.
- Leverage values describe unusual predictor configurations. Typical review rules depend on the number of estimated parameters and sample size.
- Cook’s distance summarises how much the fitted model changes when a case is omitted.
- DFBETAs show how omission changes each individual coefficient.
- DFFITS reflects the change in a case’s fitted value after omission.
No single statistic is sufficient. Verkoeijen and colleagues demonstrate in an open psychology reanalysis that different definitions, diagnostics, and cutoffs can create a multiverse of plausible outlier decisions. They advocate showing how conclusions vary across defensible analyses. This is especially important for interaction effects, which can depend strongly on observations at sparse predictor combinations.
| Planned analysis | Useful checks | Common mistake |
|---|---|---|
| Independent-groups comparison | Plots within groups, valid ranges, residuals, variance pattern | Flagging cases from the pooled distribution only |
| Correlation | Scatterplot, marginal distributions, robust correlation sensitivity | Deleting a point because it weakens the coefficient |
| Linear regression | Residuals, leverage, Cook’s distance, DFBETAs, functional form | Treating any large residual as influential |
| ANOVA | Cell plots, residuals, variance pattern, cell sizes | Using one global cutoff despite condition differences |
| Repeated measures | Participant trajectories, within-person residuals, trial quality | Confusing unusual level with unusual change |
| Multilevel model | Level-specific residuals, cluster influence, random-effect diagnostics | Applying single-level rules to clustered data |
For more model context, see the guides to regression analysis, ANOVA, and correlation analysis.
A defensible outlier analysis workflow
Step 1: Freeze raw data and reproduce cleaning
Keep a read-only raw file. Perform cleaning with syntax or a logged sequence, saving a derived analysis dataset. Record the original value, action, reason, date, and evidence for every correction or exclusion. The SPSS dissertation workflow shows why syntax is safer than undocumented menu clicks.
Step 2: Apply objective validity checks
Confirm instrument bounds, units, trial durations, skip logic, duplicate criteria, consent, eligibility, and condition assignment. Resolve clear errors before statistical screening. If validity cannot be determined, retain the uncertainty and plan analyses that show its possible consequence.
Step 3: Describe distributions in their design context
Produce plots and summaries overall and within meaningful groups. Compare raw and transformed scales where a transformation is theoretically justified. Check whether apparent extremes reflect skewness, ceiling effects, heterogeneity, nonlinearity, or clustering.
Step 4: Run diagnostics matched to the model
Use univariate screens for individual variables and model-specific diagnostics for the inferential analysis. Save case identifiers with the diagnostic output. Inspect several complementary measures rather than hunting for a single decisive cutoff.
Step 5: Investigate flagged cases without using outcomes selectively
Review source records and protocol notes. Ask whether the case belongs to the target population, whether the measurement is credible, and whether the model is misspecified. Do not invent a story from the pattern alone. If identity or clinical details are sensitive, protect confidentiality in notes and reporting.
Step 6: Choose treatment based on cause
Correct verifiable errors. Mark unverifiable impossible values missing. Retain credible observations when the model can represent them. Consider a justified transformation, nonlinear term, appropriate distribution, robust estimator, or model that recognises subgroups. Exclude a participant only under a defensible protocol rule, and preserve the excluded record for audit.
Step 7: Run sensitivity analyses
Compare the primary analysis with at least one defensible alternative when unusual cases could affect conclusions. Examples include all valid cases versus prespecified exclusion, ordinary least squares versus robust regression, Pearson versus Spearman correlation, raw versus justified transformed response, or conventional versus heteroskedasticity-robust uncertainty.
Compare effect direction, magnitude, interval estimates, model fit, and substantive interpretation, not only whether a p value crosses .05. If conclusions diverge, say so. The instability is a result that limits certainty; it is not a nuisance to hide.
When should an outlier be removed?
Removal is usually strongest when a documented error or eligibility violation makes the observation invalid for the stated analysis. Examples include a questionnaire value outside the possible range caused by import corruption, a participant who did not receive the assigned manipulation, or a device recording shown by logs to have failed.
Removal is weak when the only reason is statistical rarity, a large influence statistic, or an unwanted result. A high Cook’s distance does not say that a person should not exist. It says the fitted conclusion depends on that person’s information. Your response may be to investigate, improve the model, use robust methods, report sensitivity, collect more data if feasible, or narrow the claim.
Winsorising, trimming, or transforming also changes data and requires justification. Winsorising replaces extremes with less extreme values; it can reduce influence but conceals observed values if presented carelessly. Trimming discards information. Transformations can align a measurement scale with a plausible model but should not be selected solely to produce significance. Preserve original values and explain all transformations.
Psychology-specific examples
Reaction-time experiment
A student studies attentional bias using many response-time trials per participant. The protocol defines anticipatory responses below a plausible threshold and device timeouts above a maximum as invalid trials. After removing those trials reproducibly, the student inspects each condition’s distribution and participant-level accuracy. Valid slow responses remain right-skewed, so the analysis uses a model suited to that distribution and compares it with a prespecified transformed analysis.
The report distinguishes invalid trials from excluded participants, gives counts by condition, and states whether conclusions change under an alternative trial rule. This avoids treating every slow response as careless behaviour.
Clinical symptom correlation
A student correlates loneliness and depressive symptoms in a community sample. One participant reports scores near both scale maxima. Source checks show valid item responses with no duplicate or straight-line pattern. The point has high leverage but represents a possible high-distress participant within the target population.
The student retains the participant, reports Pearson and rank-based sensitivity estimates, and notes how the magnitude changes. Removing the person because the correlation changes would erase a potentially important part of the population. Ethical procedures for distress and data confidentiality remain separate from statistical treatment; see the psychology dissertation ethics guide.
How to report outlier decisions
APA’s Journal Article Reporting Standards for quantitative research call for clear inclusion and exclusion criteria and transparent description of data diagnostics and analytic decisions. A dissertation should let a reader reconstruct what was screened, why, when, and with what consequence.
| Report element | Include |
|---|---|
| Definition | What “outlier” meant for each variable or model |
| Timing | Whether rules were preregistered or decided after inspection |
| Method | Plots, diagnostic statistics, scaling, cutoffs, and software |
| Case accounting | Numbers of trials, values, participants, or clusters flagged and excluded |
| Reason | Error evidence, eligibility rule, model concern, or robust accommodation |
| Impact | Primary and sensitivity estimates with any interpretive change |
| Reproducibility | Syntax, decision log, and protected raw data location |
A concise results statement could read: “Visual inspection and prespecified modified z-score screening flagged three unusually slow scores. Source records indicated that one reflected a verified unit-conversion error, which was corrected. Two valid observations were retained. Cook’s distance identified one influential case in the regression; the coefficient remained positive but its interval widened when the case was omitted. Both estimates are reported.” Adapt the language to your actual evidence rather than copying a template mechanically.
Place extensive diagnostic plots and case-deletion tables in an appendix when they interrupt the main argument. In the main results, report the decision and its consequence. The results-section guide can help organise that material.
Common outlier-analysis mistakes
- Deleting every point beyond a box-plot whisker.
- Using a normal-theory z rule for a strongly skewed or bounded variable without justification.
- Checking pooled data when conditions have different expected distributions.
- Calling a large residual influential without examining leverage or coefficient change.
- Changing thresholds repeatedly until the preferred hypothesis is supported.
- Reporting only the final cleaned sample and hiding how many observations were removed.
- Using listwise participant deletion when only one trial or value is invalid.
- Forgetting that unusual cases may indicate model misspecification or a meaningful subgroup.
- Presenting a sensitivity analysis as the primary analysis after results are known.
Frequently asked questions
Is a z score above 3 automatically an outlier?
No. A z threshold is a screening convention whose usefulness depends on sample size, distribution, measurement, and purpose. Standard scores can also be distorted by the extremes they are meant to detect. Inspect the raw record, plot, distribution, and analysis context; consider robust alternatives such as a modified z score based on MAD.
Should I delete outliers before running regression?
Not automatically. Verify errors first, then fit the planned model and examine residuals, leverage, Cook’s distance, and coefficient-specific influence. A credible observation may be retained with sensitivity analysis or accommodated through a better model or robust method.
What if the result is significant only after an outlier is removed?
Report both analyses and the prespecified status and justification of the removal. Do not present the favourable version alone. If the observation is valid and the conclusion changes, the evidence is fragile and your interpretation should become more cautious.
Can I use box plots to identify outliers?
Yes, as a descriptive screening tool. Box-plot fences identify values unusual relative to quartiles, but they do not prove error or justify exclusion. Combine them with domain knowledge, source verification, and model-specific diagnostics.
How many sensitivity analyses are enough?
There is no universal number. Choose a small set of defensible alternatives that address the actual uncertainty without creating an uncontrolled search. Explain why each version matters and compare estimates and conclusions across them.
Conclusion
Good psychology dissertation outlier analysis is a documented reasoning process. Preserve raw data, verify provenance, examine distributions in context, match diagnostics to the model, and distinguish extremity from influence. Treat thresholds as prompts for investigation rather than automatic deletion rules.
When credible observations affect the answer, use sensitivity analyses and report the instability honestly. That approach protects participants’ information, strengthens reproducibility, and produces conclusions proportionate to the data. If you need ethical academic support applying these steps to your own approved project, seek guidance that helps you understand and document the analysis while keeping all decisions and authorship your own.
Authoritative references
- American Psychological Association: JARS-Quant reporting standards table
- Aguinis, Gottfredson, and Joo: Best-practice recommendations for outliers
- Leys and colleagues: Median absolute deviation for outlier detection
- NIST/SEMATECH: What are outliers in the data?
- NIST/SEMATECH: Detection of outliers
- Verkoeijen and colleagues: A multiverse outlier analysis in psychology
