We created a Snowden archive containing all the published documents leaked by Edward Snowden.
Background
The previous archives have either shut down years ago or can barely be called archives. On top of this, they are missing documents.
For example, the Snowden archive created by the University of Toronto and Canadian Journalists for Free Expression (CJFE), hosted at https://snowdenarchive.cjfe.org, was launched in March 2015, but the website disappeared around mid-2024. A subsequent website using the same data was set up in mid-2024, hosted at https://snowden.xsurveillance.site (the host snowden.glendon.yorku.ca redirects there). At the time of writing, the website responds, but its usability is nowhere near that of the previous website (and the site seems broken - try opening a subpage from the Archive navigation dropdown).
Another example is the Courage Foundation's Snowden archive, launched in mid-2015 and hosted at https://search.edwardsnowden.com. The website stopped working around late 2021.
Current archives aren't really proper archives per se (websites with search functionality and document metadata), but rather simple compilation lists or GitHub repositories holding the PDF files. The most up-to-date GitHub repository is iamcryptoki/snowden-archive, but like the others, it's missing hundreds of published documents. The ACLU stopped updating their list of published documents back in 2013. The EFF stopped updating theirs in August 2016. The Intercept's own Snowden Archive only holds a handful of documents they have published themselves. Cryptome stopped updating its compilation around late 2016.
So in early 2025, there were no proper archives, and the ones that existed were seriously lacking. This prompted us to create our own Snowden archive. It was quite remarkable that, given how important these documents are, there were no functioning archive websites.
Creating Snowchive
Our goal was simple. We wanted to create an archive holding all published Snowden documents, without a single one missing. The documents should be accompanied by rich metadata and good-quality OCR. Then, we wanted to have an archive website for this data. Everything should be free software, publicly available, and easy to download.
The first thing we did was download all the PDFs from all the existing archives, websites, and news outlets. We then deduplicated the PDFs not just by comparing filenames and hashes, but also by visually inspecting them to see which ones were duplicates. Because people really like renaming downloaded files and editing the PDFs in some way (e.g., by adding their own pages, compressing the PDFs, or adding annotations), the visual check was a must. We gathered PDFs from all the different sources we could. This process of actually getting all the published files was the hardest part of the whole project, along with hunting down the exact news article each document was published with. Sometimes it took hours to find the correct source for just one document.
The most laborious part was the OCR. All the other archives/websites that had text versions used the typical OCR process of just putting the PDF through pdftotext, Tesseract, or a similar tool. This produces extremely bad OCR, especially if the document contains any graphics/images/charts, like this one:
The typical OCR of this document looks something like this:
Compare that to Snowchive's OCR:
We went through each and every published PDF and made the OCR match the PDF as closely as possible. Over 90% of the documents required the OCR to be either written entirely by hand or heavily edited by hand. (In Snowchive, the text versions can hardly be called OCR per se, since they are not just OCR but curated text versions of the documents.) In many cases, Snowchive contains text from the documents that no other archive has ever had in its OCR. We even transcribed each and every screenshot in the documents into readable text. For example, here's a screenshot from Snowchive's website with the split view (the document on the left, and its text version on the right):
The Snowchive project includes a server and the website. It's free software, written in Go with no dependencies (standard library only). It's quite minimal (<10k LOC excluding HTML, CSS, and tests), and running it doesn't require any powerful hardware (512 MB of RAM and 1 CPU core should be more than enough). The website doesn't require any JavaScript, but there are some enhancements you can get by enabling it. We decided to write our own archive server just for these documents, because using existing projects (like Omeka/Datashare/Greenstone) would've required too much modifications in them.
If you don't like the website, you can use your own tools, like grep and jq, locally. The metadata and text versions of all published documents are only around 58 MB (the PDFs and page images are around 18 GB) and can be downloaded separately.
Website and repositories
The Snowchive project is split across one website and two repositories:
Snowchive data
data.snowchive.org/files hosts the full releases, updated monthly, or less often when there is no new data. The last six releases stay online. Each release comes in two packages: full (PDFs, page images, extracted texts and metadata) and pdfs (the same without page images). Each release also includes the source of the server version it was published with.
Every release is signed with our key 3B05 4BC2 E7B2 CC82 5FF2 3E94 108F 97B8 FC5B 2EBC.
Each package also has a torrent file.
The metadata is dedicated to the public domain (CC0 1.0). No rights are claimed in the documents themselves, which are reproduced as published.
Snowchive server
- https://git.disroot.org/libroot/snowchive-server
- https://codeberg.org/libroot/snowchive-server (mirror)
- https://git.libroot.org/libroot/snowchive-server (mirror)
The source code of the server is licensed under the AGPL-3.0.
Snowchive metadata
Only the metadata and extracted texts (~58 MB, no PDFs or images). The same files are in every release package. Metadata: CC0 1.0.
- https://git.disroot.org/libroot/snowchive-meta
- https://codeberg.org/libroot/snowchive-meta (mirror)
- https://git.libroot.org/libroot/snowchive-meta (mirror)
Future
We plan to write a longer and more informative post about creating this project, how we did the curating, which tools we used and how, our custom tools/scripts etc.
In the near-future Snowchive will have Tor onion service, since currently there are none.
We will maintain the project for as long as possible, and we welcome any contributions and feedback. If you are a researcher, journalist, historian, or just someone who is interested in these documents, we hope you find Snowchive useful. Thank you!