Method

How the measurement works

This page is the part a reviewer would ask for: what is actually measured, what the error term is, which published procedure is implemented, and where the decisions were not obvious. Everything below is implemented in the repository and covered by tests.

1. What an implicit association test measures

The claim an IAT supports is narrow and mechanical: a participant sorts items faster when the two categories sharing a response key are already associated in memory. It does not read minds, does not reveal hidden beliefs, and is not a lie detector. The interesting property is that the participant cannot easily fake it in the direction they want, because the quantity being measured is how fast they can press a key, not what they choose to say.

The instrument is relative by construction. A two-target IAT compares Brand A against Brand B; it cannot tell you whether A is fast or B is slow, and the comparator you pick partly determines the answer. Where there is no fair comparator, the right instrument is a single-category test — a genuinely different algorithm, not a flag on the same one. Both are implemented here.

2. Block structure

Seven blocks, following Greenwald, Nosek & Banaji (2003), with Block 5 lengthened to 30 trials per the 2022 best-practice update, which found that change "often keeps the effect of order of combined tasks to a minimum". Two calibration blocks run first and feed no part of D.

Counterbalancing. Which pairing a participant meets first is drawn from the session seed. This is not optional tidiness: whichever pairing comes second carries a practice advantage, so a fixed order does not add noise, it adds bias in a direction the researcher chose. Which side Brand A starts on is counterbalanced too, which removes a handedness confound from the raw latencies even though it cancels in D.

Trial ordering. Combined blocks alternate a target trial and an attribute trial. Without alternation a participant can batch-process one dimension and the task degrades into two separate sorting tasks. Exemplars are drawn without replacement within a cycle, and no category is allowed to repeat more than three times running — a long run lets someone stop reading and just repeat the last keypress, which produces fast, correct, meaningless trials.

Errors. A wrong key shows a red ✗ and the trial does not advance until the correct key is pressed. The engine records both clocks: time to the first keypress and time to the correct one. That means either published error procedure can be run against the same raw file, and it removes the arbitrary choice between a 600 ms penalty and a 2 SD penalty, because with forced correction the penalty is built into the measurement.

3. Timing, and what its error term actually is

A reaction time is the interval between two physical events: light changing on a display, and a switch closing under a finger. A browser observes neither. It observes when it handed a frame to the compositor and when the operating system delivered an input event. Three techniques close most of the gap.

Stimulus onset — nested requestAnimationFrame

A rAF callback runs before the paint of the frame it belongs to. So the stimulus is written to the DOM inside rAF #1, which means it is painted in that frame, and the timestamp handed to a nested rAF #2 marks the start of the following frame — the first moment the browser can confirm the previous frame went out. Timestamping with performance.now() at the moment textContent is set would be wrong by up to a full frame plus layout time, and wrong in the same direction on every trial.

Response — event.timeStamp, not performance.now()

For trusted events the browser stamps timeStamp when it creates the event, on the same monotonic clock as performance.now(). Reading the clock inside the handler instead measures when the event loop got round to the page, which adds however long the main thread was busy. That gap is real, variable, and measured here per trial: it is reported on the dashboard as event-loop delay, and it is a direct measurement of the error the naive approach would have introduced.

Refresh rate — measured, not assumed

Onset can only be known to within the frame it was painted in. The frame interval is measured at startup from the median of ~90 animation frames, using the median rather than the mean so a single dropped frame does not drag the estimate. On a 60 Hz panel the irreducible onset uncertainty is about ±8 ms; on 120 Hz it is ±4 ms. Assuming 60 Hz would silently change the error term on half of modern laptops.

None of this removes display lag — the panel's own processing delay, typically 5–20 ms and invisible to software. That is a constant offset, so it cancels in any within-participant difference score. It would not cancel in a claim about absolute latency, which is one reason D is defined as a difference in the first place.

4. The D-score

The recommended procedure of Greenwald, Nosek & Banaji (2003, Table 4), with the four steps that re-implementations most often get wrong called out.

The denominator is the thing to get right

The inclusive SD is deliberately not Cohen's pooled-within-condition SD, not the SD of difference scores, not the mean of the two blocks' SDs, and not one block's SD. Because the two blocks have different means, the between-block variance is part of the denominator on purpose — that is what makes D behave like an individual-level effect size. There is a genuine trap in the source: Table 4 uses the word "pooled", which reads like Cohen's; the body text is the unambiguous version, computing the SD "ignoring the condition membership of each score". A unit test in this repository pins the two apart and asserts they differ.

A published disagreement, stated rather than hidden

The 2003 paper orders the steps so the SD is computed before error latencies are replaced. The 2022 best-practice paper says the opposite, and the two most-used reference implementations follow the 2022 wording. Rather than pick a side silently, this is a configuration flag, both paths are tested, and the shipped default uses forced error correction — under which no substitution happens and the question cannot arise. The dashboard reports the alternative scorings side by side so you can see whether the conclusion depends on the choice.

Interpretation bands

0.15 / 0.35 / 0.65 are the Project Implicit reporting convention, not Cohen's d cut-offs, which are 0.20 / 0.50 / 0.80. They describe the size of a latency difference, not its reliability, and this tool labels them accordingly.

5. Inference without assuming normality

Reaction times are strongly right-skewed — approximately ex-Gaussian, a normal convolved with an exponential tail. A t-test on raw latencies assumes a symmetry the data does not have, and the tail is exactly where the interesting variance lives. Everything inferential here is therefore resampling-based:

• Percentile bootstrap for the interval around D, resampling trials with replacement within each block-pairing cell so the design is preserved and the bootstrap estimates error in the latencies rather than in the block structure.
• Permutation test for whether D differs from zero, shuffling pairing labels within each block pair. This is answerable from one participant's data because the trials themselves are the exchangeable units. An add-one correction is applied, because a permutation p-value should never be exactly zero: the observed arrangement is itself one of the possibilities.
• Split-half diagnostic: D recomputed on odd- and even-numbered trials. A wide gap means the estimate is unstable within the session itself, whatever the headline number says.

6. What a single session can and cannot support

This is the limitation that matters most and the one demos usually omit. The IAT's test–retest reliability is around r = .50. Greenwald's own assessment is that this is "not adequate to justify treating a single IAT observation as accurately diagnostic". Cummins & Hussey (2026, Behavior Research Methods) put numbers on it: a D of exactly zero carries a 95% confidence interval of roughly −0.38 to +0.38, the narrowest intervals they observed were wider than the gaps between the conventional labels, and only about half of participants had scores detectably different from zero.

So the instrument is a group-level measure. This tool reports an interval rather than a verdict, refuses to produce a number at all when the exclusion criteria fire, and says on every report that one session is one person on one occasion.

7. The language model, and where its boundary is

The model receives a fixed block of numbers that Python has already computed, tested and quality-flagged. It sees no trial rows and no raw latencies, so it cannot recompute anything, and it is not asked to: even the trivially derivable judgements — does the interval exclude zero, is the result significant, is the session usable — are computed in Python and handed over as a verdict. A wrong claim about a correct number is the one failure a numeric check cannot catch, so the model is never put in a position to make one.

What it returns is then checked. Every number in the generated text is extracted and matched against the payload; if any figure cannot be traced, the draft is rejected, the deterministic summary is published instead, and the rejected draft is shown alongside the reason. The prompt is displayed in full on every report, because if a model writes part of a research report then the instructions it was given are part of the method.

If no model is configured, or the call fails, a rules-based engine produces the same report from the same numbers and the response says which engine wrote it. The product does not break when the model is unavailable.

One thing the model is explicitly not for: generating respondents. An implicit measure is a measurement of human cognitive processing speed. There is no latency to simulate, because there is no cognition being timed.

8. Reproducibility and data

• A session is a (study, seed) pair. The same seed regenerates a byte-identical trial sequence and the same counterbalancing, verified by a test that runs the generator in subprocesses under different hash seeds.
• The design is generated and stored server-side, so the condition a participant was in is a recorded fact rather than something the browser reported.
• Every session exports as trial-level CSV and as a complete JSON record including the design, the counterbalance, the seed and the client metadata.
• Scoring parameters are serialised into every result, so a number always carries the configuration that produced it.

9. From one participant to a panel

The Studio analyses a whole fieldwork batch. The unit of analysis changes from the trial to the participant: each respondent contributes one D per attribute, and every interval and test is over respondents.

Participant screening, in order

Trial-level trimming is separate: the 10,000 ms upper cut-off by default, and no lower trim, because fast trials are signal and the latency screen is what deals with fast people. Both are sliders in the Studio, and the funnel, D-scores and intervals update as they move.

Two families of test

Parametric: each respondent's difference in mean log latency between pairings, tested with a one-sample t-test, and Welch's t-test between segments. Raw latencies are right-skewed (skew ≈ 1.9 in the demonstration panel); their logs are close to symmetric (≈ 0.3–0.45), which is what a t-test needs. Non-parametric: sign-flip permutation (one-sample) and label-shuffle permutation (between segments) on D itself, 5,000 draws each. The Studio switches between them and counts the significance calls that change.

Multiple comparisons

An attribute × segment grid is many tests at once, and at α = .05 an uncorrected grid of twenty null cells produces one false positive on average. Every p-value is reported with a Benjamini–Hochberg q-value within its family (associations; segment differences), and significance flags are set on q < .05. False-discovery-rate control rather than Bonferroni because this is a screening grid: Bonferroni protects against any single false positive at the price of missing real effects, which is the wrong trade for a brand-image read that will be followed up.

One source of numbers

The dashboard, the Excel workbook and the PowerPoint deck call the same function with the same parameters, and the parameters are written into both files. The deck's executive takeaways are written by rules, not a model: a result that fails q < .05 is never given a direction, D is always described as relative, and a significant but negligible gap is called small. Cells under 30 respondents are flagged as low base.

10. Limitations

• Word stimuli only. Production pack testing uses images, which changes onset timing (decode and paint cost) and needs preloading and per-image onset verification.
• Desktop keyboard only. Touch changes the motor component substantially and would need its own calibration.
• Cohort inference runs on per-participant summaries. A trial-level mixed-effects model with crossed random effects for participants and stimuli would use the data more fully and generalise over stimuli as well as people; it, and power analysis for sample planning, are not built.
• The cohort scorer implements the built-in error penalty only. Uploads from tasks without forced error correction are flagged at ingestion rather than silently re-scored.
• The comparison distribution shown at the end is a convenience sample of whoever opened the link. It is not a norm.
• The speeded attribute-association format that commercial platforms mostly sell — one stimulus, many attributes, roughly two minutes — is not implemented. For brand-image mapping under a fifteen-minute session budget it is arguably a better fit than either instrument here.

References

• Greenwald, A. G., Nosek, B. A., & Banaji, M. R. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85(2), 197–216.
• Greenwald, A. G., Brendl, M., Cai, H., et al. (2022). Best research practices for using the Implicit Association Test. Behavior Research Methods, 54, 1161–1180.
• Karpinski, A., & Steinman, R. B. (2006). The Single Category Implicit Association Test as a measure of implicit social cognition. Journal of Personality and Social Psychology, 91(1), 16–32.
• Richetin, J., Costantini, G., Perugini, M., & Schönbrodt, F. (2015). Should we stop looking for a better scoring algorithm for handling Implicit Association Test data? PLOS ONE, 10(6): e0129601.
• Cummins, J., & Hussey, I. (2026). Individual-level confidence intervals for IAT scores. Behavior Research Methods, 58:21.
• Bridges, D., Pitiot, A., MacAskill, M. R., & Peirce, J. W. (2020). The timing mega-study: comparing a range of experiment generators. PeerJ, 8:e9414.
• Anwyl-Irvine, A., Dalmaijer, E. S., Hodges, N., & Evershed, J. K. (2021). Realistic precision and accuracy of online experiment platforms. Behavior Research Methods, 53, 1407–1425.