Building Reproducible Research From Public Reddit Archives

Reproducibility in archive research is harder than in most empirical work, for a reason that is easy to miss: the dataset changes underneath you.

An archive backfills records, updates scores, and improves coverage of past periods. Run the same query six months apart and get different results — not because anything was wrong, but because the archive learned more.

That rules out the usual standard, where publishing a query is enough for someone to check your work. It does not rule out reproducibility; it changes what has to be recorded.

Record the retrieval, not just the query

The query is necessary and not sufficient. What matters is what you had.

At minimum, record which archive, the exact query parameters, when you retrieved it, how many records came back, and whether the retrieval completed or was stopped early.

That last one gets forgotten constantly. A truncated result analysed as complete is the most common irreproducibility in this space, and it leaves no trace unless someone writes it down.

Keep the records

If your conclusions rest on a particular set of records, keep them. Not a summary — the records.

This is uncomfortable, because it means holding a corpus of other people's public writing longer than you need it. It is worth weighing that against the reproducibility gain, and worth being deliberate rather than defaulting either way.

A reasonable middle: keep the records while the work is active and unpublished, keep the query and the metrics permanently, and delete the corpus when the work is done. Anyone checking later can re-run the query, and the metadata tells them whether their result should match yours.

Fix the analysis parameters

Analysis choices change results as much as the data does.

Timezone. Bucketing in local time makes monthly figures depend on who ran the analysis. Fix it to UTC and say so.

Stopword list. Term frequency depends entirely on which words were excluded. Publish the list or name it.

Tokenization. Whether URLs are stripped, whether case is folded, whether hyphenated words split — all change the output.

Exclusions. If you excluded records with unsettled scores, or a high-volume subreddit, say which and why.

None of these are controversial choices. They just have to be stated, because someone repeating your work with different defaults will get different numbers and have no way to know why.

Report the shape of your coverage

A finding based on 1,200 records means something different depending on whether that is most of an account's history or a recent slice.

Report what you loaded, over what period, and how it compares to what appears to exist. If the archive suggests an account has far more records than you retrieved, say so — a reader can then judge whether your sample supports the claim.

Treat archive disagreement as a finding

If two archives return substantially different results for a query central to your work, that is worth reporting rather than resolving.

Picking the source that supports your conclusion is the obvious hazard. Picking the one with more records is subtler and equally arbitrary. Reporting both, and noting they disagree, tells the reader the record is less settled than a single number implies — which is true and useful.

A minimal methods paragraph

Records were retrieved from [archive] on [date] using [query]. [N] records were returned, covering [range]. Retrieval completed / was stopped after [N] requests. Timestamps were bucketed in UTC. Term frequency excluded [stopword list] and records marked deleted or removed. Score statistics excluded [N] records captured within 36 hours of posting, where the archive stores placeholder values. Records are available at [location] / were deleted on [date].

Six sentences. Most archive research includes none of it, which is why most archive research cannot be checked.