Introduction
A sensor may trigger more often in one room than another. A team may report that unusual sounds cluster around a particular time. A graph may show a dramatic spike. The next question is not simply whether the pattern looks unusual, but whether it is stronger than would reasonably occur through chance, measurement variation or the way the data were collected.
Statistical significance is often misunderstood as proof. It is not. Statistics help quantify uncertainty under stated assumptions; they do not independently establish a cause or validate an interpretation.
What a p-value means
A p-value is the probability, assuming a specified statistical model and null hypothesis, of obtaining results at least as extreme as those observed. It is not the probability that the null hypothesis is true, and it is not the probability that an event is paranormal.
A small p-value can indicate that the observed data are difficult to reconcile with a particular null model. But the result depends on whether the model and analysis were appropriate.
Effect size and practical importance
A result can be statistically significant but practically tiny, especially with large datasets. Conversely, a potentially meaningful effect may fail to reach a conventional significance threshold in a small study.
Report effect sizes and uncertainty intervals alongside p-values. An effect size describes the magnitude of a difference or relationship. A confidence interval communicates a range of values compatible with the data and model under the method's assumptions.
Multiple comparisons and selective reporting
If investigators test many rooms, time windows, sensors, frequencies and event definitions, some results may appear unusual by chance. This is the multiple-comparisons problem.
For example, searching hundreds of time windows and reporting only the one with the largest spike exaggerates how surprising the result is. Better approaches include:
- Pre-registering primary outcomes and analysis plans
- Defining event thresholds before inspecting results
- Reporting all tested comparisons
- Using appropriate correction methods where justified
- Replicating the finding in an independent dataset
Corrections such as Bonferroni or false-discovery-rate procedures address particular settings; they are not universal solutions.
Exploratory versus confirmatory analysis
Exploratory analysis is useful for discovering patterns and generating hypotheses. Confirmatory analysis tests a hypothesis specified in advance. Confusing the two creates a risk of overconfidence.
An exploratory result should be labelled as such and tested in a new session or dataset. Reusing the same data to invent and “confirm” a hypothesis does not provide independent confirmation.
AERM statistical workflow
- Define the research question in measurable terms.
- Specify the primary outcome and event definition before analysis.
- Establish baseline conditions and identify missing data.
- Record sample size, duration, sensor sampling rate and exclusions.
- Choose a statistical method appropriate to the data structure.
- Report effect size, uncertainty and assumptions—not only a p-value.
- Disclose all analyses that were performed.
- Replicate important findings using independent sessions or observers.
Example: sensor activations
Suppose a motion sensor activates 12 times during a two-hour session. That count alone has little meaning. Investigators need a baseline activation rate, the sensor's false-trigger behaviour, environmental conditions, event clustering, and a pre-defined rule for what counts as an activation.
If the sensor was adjusted repeatedly until it produced a striking pattern, the final pattern is not an unbiased test. If the same pattern occurs during controlled sessions with no change in occupancy, that is informative—but the result still requires source investigation.
Evidence is broader than significance
Scientific evidence includes measurement validity, design quality, controls, independent replication, plausible mechanisms, data integrity and transparent reporting. A statistically significant difference can be caused by a confounder. A non-significant result does not prove that no effect exists; the study may simply be too small or noisy to detect it.
Conclusion
Statistical methods are valuable when they are used to quantify uncertainty rather than decorate a conclusion. AERM investigations should distinguish exploratory patterns from pre-planned tests, report effect sizes and uncertainty, and make all comparisons visible. A striking graph is a starting point for analysis, not the end of it.
AERM Core Principle: No anomaly is considered paranormal until environmental, psychological, technical, and scientific explanations have been systematically evaluated.
References and further reading
- Wasserstein, R. L. and Lazar, N. A. “The ASA's Statement on p-Values: Context, Process, and Purpose.” The American Statistician.
- Cumming, G. Understanding The New Statistics. Routledge.
- Lakens, D. “Sample Size Justification.” Collabra: Psychology.
- Benjamini, Y. and Hochberg, Y. “Controlling the false discovery rate: a practical and powerful approach to multiple testing.” Journal of the Royal Statistical Society: Series B.
- National Academies, Reproducibility and Replicability in Science.
About AERM
AERM values transparent data handling and independent replication. Statistical analysis is used to test clearly defined claims, not to convert ambiguous observations into certainty.
Educational and safety disclaimer
This article is educational and does not replace statistical consultation for complex studies. Analysis choices should be documented before conclusions are drawn, and personal data should be handled responsibly.