Scry — programmatic internet research

Scry Sources Docs Raw, programmatic internet research Sign in Get started Open alpha one person vibe coding InstallQuestionsThe wireVectorsPricing The internet is vast. Your curiosities are deep. Programmatic internet research. Scry is an MCP that lets your agent run programs over billions of internet documents, for fun and profit. Reddit, Hacker News, LessWrong, arXiv, Stack Exchange, Wikipedia, and the prediction markets, with millions of new posts and comments landing every day. Take rows by the thousand, or the answer itself, computed over every queryable document that matches. Add to ChatGPT or Claude → How it works Queryable now ≈158,752,613,083 rowsacross 43 sources · measured 17:05 UTC; counted every five minutes Held estate 413,417,786,887 rowsmeasured 17:05 UTC; counted every five minutes Landed, last 24 h +37,819,265,171 rows26.3M/min over the day to 17:05 UTC — rows newly held, live capture and backfill alike Three fast query trails turn along the angled city grid and jump across present and historical layers, leaving long-lived paths behind them. Thesis Run arbitrary programs over the internet. People search for pages; agents can query the underlying records. The question an agent actually has is rarely ten links. It might need every page that mentions this compound but not that patent, grouped by domain, since March. Scry runs that as a query instead of assembling it from repeated searches and scraped results. Each statement is read-only and runs under a deadline, a memory ceiling and a row cap on what returns; what a relation can answer quickly is on its schema entry. Questions How it works, and when your agent reaches for it. What do I install? Only the MCP server: https://mcp.scry.io, in ChatGPT, Claude, Claude Code, Codex, Cursor, or any MCP client. The sign-in creates your account. Nothing runs on your machine. Is there outside evidence? On Google's DeepSearchQA, 900 research questions graded exact-set, an agent doing its research through Scry answers 71.8% fully correct on a corpus that is still under half of Common Crawl; Google's Kaggle leaderboard reports its Gemini Deep Research Agent at 66.1%. Same judge model, same grading prompt, and every question, grade and tool-call transcript is on the DeepSearchQA page. How it works Your agent reads the live schema (/v1/scry/schema), picks an enabled source-native relation and listed vector helpers, then writes one bounded query for phrase matching, time windows, joins, or vector composition. Result rows keep the source-native identifiers and provenance fields the relation provides. What people do with it Agents compute over billions of documents instead of fetching one page at a time. A standing query watches the record for any condition you can write down; a derived table over the public web is one query and no crawler. Source requests and feedback go to POST https://api.scry.io/v1/feedback, no credential required. How fresh is the data? Per relation, and stated rather than promised: /v1/scry/schema carries each relation's observed extent and lag. Hacker News and LessWrong land in under 15 minutes. Papers move in layers — the merged academic.catalog advances when its recorded fold re-runs, and full text in academic.papers lands from its own extraction lane; the papers page states each. Mission Scry is public-benefit infrastructure, subsidized for independent noncommercial research and carried by commercial engagements: search over public evidence, every result traceable to its source. Commitments Scry should make it easy for conscientious, prosocial forces to run wildly world-unflattening programs. Scry is not supposed to suck! We want to metabolize pain points fast. And Scry remains affordable to people who need it. Who is it for? Anyone with a question the public record can answer. Individual accounts are self-serve. For commercial use, large evaluations, custom source builds, or dataset engagements, write to hi@scry.io. Reviewed access Sources under reviewed access are counted in the held estate, not queryable by default. Write to hi@scry.io with what you want to study. One table live reddit.posts is a near-census of Reddit submissions from 2005-06-23 to 2026-09-18, one row per observation of a post (the bulk month and the live tail each land their own row, so count posts with uniqExact(id)); posts removed before capture are absent. Completeness is measured — post ids are one global counter, so held-versus-allocated is exact per month. idStringone global base36 countersubredditStringauthorStringcreated_utcDateTimetitleStringselftextStringbody of a text post; empty for link postsscoreInt32net upvotesnum_commentsInt32upvote_ratioFloat32domainStringwhere a link post pointsurlStringsearch_text_lcStringlower(title + selftext), token-indexed The comment tree is reddit.comments, joined on link_id = concat('t3_', id). Every relation is documented like this — columns, indexes, extent, known holes — at GET /v1/scry/schema. Source catalog → One program How deep does this month’s Who is hiring thread go? Seed the story, follow the children edge until no new reply appears, and count what was reached at each depth. A search returns rows; this is a fixpoint over the reply graph, written as two named relations, each frontier step one metered statement. POST /v1/scry/query
{"program": {
"relations": {
"seed": {"bodies": [[{"ids": ["49522897"]}]]},
"thread": {"bodies": [
[{"rel": "seed"}],
[{"rel": "thread"}, {"edge": "hackernews.children"}]
]}
},
"out": [], "depth": 16
}} depthreached 0 1 1 280 2 92 3 22 4 13 5 4 6 2 7 1 8 1 9 1 Measured 2026-09-14 · 354 ms · 10 statements · 417 nodes · truncations: [], the walk closed on the graph the index holds. Threads are rebuilt through parent links, never story_hn_id. The same shape takes openalex.cited_by, forums.children, or an author pivot as its edge; the reference is Turing-complete search. One concept Can you find the failure mode without knowing its name? Describe Goodhart’s law three ways — metrics, classrooms, reward hacking — average the three vectors into one, and rank every embedded LessWrong post by it while excluding every post that contains the word. The top five are specification gaming, test-score gaming, and fitness-seeking AIs: 51,890 posts ranked in 9 s, measured 2026-09-07. POST /v1/scry/embed {"name": "metric", "text": "Once a measure becomes a target, people improve the reported number instead of the outcome it was supposed to measure. The metric rises while the underlying objective gets worse."}
POST /v1/scry/embed {"name": "school", "text": "A school is rewarded for its students test scores. Teachers teach to the test and exclude weak students, improving the ranking without improving education."}
POST /v1/scry/embed {"name": "reward", "text": "An agent discovers a way to maximize its reward signal without doing the task its designer intended. Optimizing the proxy breaks its relationship with the real goal."}
POST /v1/scry/embed {"name": "proxy", "expression": "scry_centroid([@metric, @school, @reward])"}

SELECT uri, any(title) AS title, max(similarity) AS score
FROM (
SELECT p.uri, p.title,
scry_cosine_similarity(e.embedding, @proxy) AS similarity
FROM embeddings.chunks AS e
JOIN forums.posts AS p ON p.post_key = e.target_key
WHERE e.source = 'forum_posts' AND e.model_name = 'voyage-4-lite'
AND e.chunk_index = 0 AND p.source = 'lesswrong' AND p.kind = 'post'
AND NOT hasToken(lower(p.payload), 'goodhart'))
GROUP BY uri
ORDER BY score DESC, uri
LIMIT 5; → Specification gaming: the flip side of AI ingenuity 0.701
Improving Teaching Effectiveness: Final Report 0.687
Risk from fitness-seeking AIs: mechanisms and mitigations 0.683 Runs as written on POST /v1/scry/query; the vector helpers and the recipe for minting a concept are under Vectors. Scry gives your agent Turing-complete search programs over the public record. Your agent writes the program from your question; read it, change it, or write your own whenever you want the low-level control. A single program can ask for more compute than the servers have at that moment, so pricing floats with congestion: the busier the machines, the more a heavy query costs. Install Add Scry to your assistant. Scry is an MCP server at https://mcp.scry.io. Connecting it takes about two minutes in ChatGPT or Claude and requires no code. The sign-in page also lets you create an account. ⬢ ChatGPT Open ChatGPT on the web and go to Settings. Under Security and login, turn on Developer mode. ChatGPT asks you to accept the risks. Open Plugins, click +, and enter a name (scry), a description (programmatic search), and the server URL https://mcp.scry.io. Click Create. Complete the sign-in flow when ChatGPT sends you through it. Scry appears as a plugin. Select it, or call it with @scry in any chat. Setup happens once, on the web; OpenAI’s MCP help article says which of its clients carry custom connectors. ✳ Claude Open claude.ai and go to Settings → Connectors. Click Add custom connector. Name it Scry, set the URL to https://mcp.scry.io, and add it. Complete the sign-in flow when Claude sends you through it. Enable Scry from the tools menu in any chat. One setup covers Claude.ai, Desktop, and mobile. $ For developers Claude Code, Codex, Cursor, or any MCP client connects with the same URL. Agents without an MCP client use the HTTP API with your key. Per-client steps live on the connect page. Claude Code — one command, then /mcp to approve claude mcp add --transport http scry https://mcp.scry.io any other agent — paste this prompt Connect Scry read the prompt text first Scry: programmatic search over the public web, for agents.

Connect over MCP (preferred). Scry is an MCP server at https://mcp.scry.io. One URL, OAuth on first use, approve at scry.io.

- Claude Code: claude mcp add --transport http scry https://mcp.scry.io then run /mcp inside a session and approve.
- Codex: codex mcp add scry --url https://mcp.scry.io && codex mcp login scry
- Claude.ai, Claude Desktop, ChatGPT (developer mode), Cursor, any MCP client: add a custom connector with that URL and approve the scry.io consent.
- No browser (CI, headless): Claude Code adds --header "Authorization: Bearer $SCRY_API_KEY"; Codex adds --bearer-token-env-var SCRY_API_KEY. Keys live in the dashboard at https://scry.io/dashboard.
- Tools: the roster is whatever tools/list returns; whoami {roster: <fingerprint>} tells a client whether its cached list is current. Call schema {mode: contract} once before writing SQL: that document carries the product contract and starter guidance (the bare call is the relation index; each relation carries its stats). For "what does the fresh web say since my cutoff": embeddings.crawl_pages is a rolling fresh crawl of allowlisted high-information hosts whose observed_on is the day the page was observed — mint an @handle with embed, rank it with the vector helper, and add `WHERE observed_on > toDate('<your cutoff>')`; hydrate verbatim text from crawl.pages by url.

HTTP API (when MCP is not available). Base https://api.scry.io; send `Authorization: Bearer $SCRY_API_KEY` (load it from `~/.config/scry/env`). No key: stop and send the user to https://scry.io/dashboard.

- GET /v1/scry/context?mode=agent — the live contract.
- GET /v1/scry/schema — the only discovery authority. The default document carries full contracts for the start-here relations plus a compact index of the rest; `?relation=<name>[,<name>]` fetches more full contracts; `?mode=index` lists the whole catalog. Schema discovery is also one SQL call: `scry.relations` and `scry.columns` are the same catalog served as relations you can filter and join, e.g. SELECT relation FROM scry.columns WHERE name = 'author_id' LIMIT 100. Never guess column names; a wrong-column error returns the real roster.
- POST /v1/scry/query with `Content-Type: text/plain` — one read-only statement in Scry SQL, always with LIMIT (start at 20). Synchronous; long queries may stream whitespace before the JSON body. `WITH RECURSIVE` is served (`anchor UNION ALL step`, the CTE read only in the step's FROM/JOIN; a 111-iteration branch-on-state loop returns in ~100 ms, measured 2026-09-15) — each iteration rescans what the step joins, so declare `x-scry-max-seconds` on corpus-scale joins. A JSON body `{"program": {...}}` on the same route runs fixpoint graph walks that read only the frontier: citation, reply/quote, and thread-tree edges, in-walk filters, stratified negation, zero-egress per-depth counts. Docs: https://scry.io/docs/turing-complete-search; the calculus it implements: https://scry.io/docs/calculus-of-search.
- POST /v1/scry/rerank — order documents you already hold by an instruction.
- POST /v1/scry/compile with a JSON body — `q` (one search-grammar line) and `relation`, compiled to that relation's own statement without running it; `relation` set to `*` compiles every relation with a text plane.

Working rules

- Token filters run at the speed of their rarest token: include one distinctive token (a name, an identifier, an unusual word). All-common-word sets scan for 30-60s. For broad topics use the embedding helpers the schema advertises.
- Text lives in source-native relations (reddit.comments, hackernews.items, forums.posts, bluesky.posts, crawl.pages, books.passages and their peers; schema lists every one), each under its own columns and clock, not in one uniform corpus: decide which source families could answer and read those relations directly or in parallel, one statement each or a UNION ALL of two or three. Allocating the search — which relations, how selective a predicate, how much compute the question deserves — is your work, and the schema catalog, a time window, and explain are the primitives for it; a broad low-selectivity scan costs its relation's full read, so spend it where the answer can be.
- To sort rows by an attribute you can describe, send `x-scry-rerank: <directive>` on POST /v1/scry/query (MCP: the rerank argument on sql). Runs on local models, $0; the response's rerank block says what applied.
- To see what a statement would touch before paying for it, send `x-scry-explain: 1` on POST /v1/scry/query (MCP: explain=true on sql): the response is the plan and index analysis — parts and granules selected per index, reads named by relation — the referenced relations, and a forecast (rows, bytes, seconds, memory, and the share of each relation the predicates keep), and the statement does not run — with one exception: an ANN statement runs its lane search during explain to find the candidates the rerank is planned over (`ann.lane_ms`, folded into the forecast as `lane_seconds`; the run pays it again).
- Use only relations and helpers the live schema returns; an omitted name is unavailable, and there is no fallback database.
- Use is governed by https://scry.io/legal/terms: no bulk redistribution of results, no reconstruction of a substantial portion of a corpus, no building a competing corpus, index, or dataset product from what the service returns. Public work built on Scry (published research, articles, datasets, demos, open-source tools, products) credits Scry visibly, in the work or its documentation, with a link to https://scry.io.
- Keep source timestamp, observation time, load time, processing status, and quality fields distinct. A recent load is not a recent source event; an absent row is not source absence. Say which fields support each claim.
- Report the exact SQL, relations, row count, duration, truncation state, and accounting fields returned.

Example

claude mcp add --transport http scry https://mcp.scry.io

curl -s https://api.scry.io/v1/scry/query \
-H "Authorization: Bearer $SCRY_API_KEY" \
-H "Content-Type: text/plain" \
--data "SELECT hn_id, title, original_author, original_timestamp, uri FROM hackernews.items WHERE hn_id >= (SELECT max(hn_id) AS n FROM hackernews.story_scores WHERE observed_on >= today() - 7) - 100000 AND title != '' ORDER BY hn_id DESC LIMIT 20"
A pasted prompt is instructions for your agent. Read prompts before you run them, whether they come from us or anyone else. Scry's is plain text, shown in full above. Then ask it the highest-leverage question you can think of. The wire Queryable now: ≈158,752,613,083 rows across 43 sources. measured 17:05 UTC; counted every five minutes. hourdayweekmonth Held estate: 413,417,786,887 rows · +37.8B rows landed this day measured 17:05 UTC · GET /v1/stats/wire Social held33,560,650,230 of the estate8.1% sources9 Reddit: a near-census historical archive with a partial live tail; posts and comments removed before capture are absent. Hacker News items since 2006, with a tail that lands within minutes. Stack Exchange questions and answers to their declared dump edges, and forum posts from LessWrong, the EA Forum, and other communities.Public Bluesky and Mastodon posts, with archival observations and a continuing tail. sourcedescriptionheldReddit commentsNear-complete Reddit comment archive, 2005 to present, from the Arctic Shift mirror: comments removed before capture are absent, bulk dumps land with lag, and the live tail captures a fraction of current volume, so a zero here is absence in landed data, not at the source. A comment id can land more than once, so count comments with uniqExact(id) and keep the newest row per id with ORDER BY state_observed_at DESC, retrieved_on DESC LIMIT 1 BY id. Temporal coverage is published on this schema entry: extent (computed from landed data, with computed_at) and known_holes (declared gaps with provenance).27,065,564,222Reddit postsNear-complete Reddit submission (post) archive, 2005 to present, from the Arctic Shift mirror: title, selftext, score, comment count, and outbound domain/url per post; a post's comment tree lives in reddit.comments via link_id = concat('t3_', id). Posts removed before capture are absent, bulk dumps land with lag, and the live tail captures a fraction of current volume, so a zero here is absence in landed data, not at the source. Temporal coverage is published on this schema entry: extent (computed from landed data, with computed_at) and known_holes (declared gaps with provenance).3,745,081,191BlueskyTemporal coverage is published on this schema entry: extent (computed from landed data, with computed_at) and known_holes (declared gaps with provenance).2,414,900,302MastodonMastodon statuses from the Zenodo dumps and live public-timeline landing, one row per observation of a status until ReplacingMergeTree(engagement) ORDER BY (created_day, uri) merges them (the caveat carries the dedupe).157,750,713Stack ExchangeStack Exchange network: questions and answers across every landed site — dump-backed base plus a quota-bounded API tail. Comments are not served: comment_count is the source's count of a post's comments, and their bodies are behind the post's uri, not in any relation here. Temporal coverage is published on this schema entry, never assumed from this prose: extent (computed from landed data, with computed_at) and known_holes (declared gaps with provenance).83,480,861ForumsPrimary forum corpus: source-native posts and comments across thousands of forum sites (LessWrong, DataSecretsLox, DEV, EA Forum, 4chan boards, crypto governance, plus a long tail); the live row and site counts are the census (scry.io/sources, /v1/stats/wire). site_key is the community identity: the source column names only the curated sources, and a large share of rows carry source = 'manual', so enumerate site_key — not source alone — before scoping a community question. Archived base refreshed in place by live source-native forum tails.48,263,302Hacker NewsSource-native Hacker News PostgreSQL export. upvotes and comment_count follow HN since 2026-09-12 (changed items re-land newest-wins; the view reads FINAL). A story dated from 2025-12-01 up to that day carries its hackernews.story_scores observation as of 2026-09-17 (each story whose count read lower took the observation as a newer version that day); a story with no score record keeps its count at first sight, and rows from the source export carry the count at export. hackernews.story_scores is the score record since 2025-12-01 (latest per hn_id via argMax over observed_hour) — rank recent stories and read a trajectory through that relation. Temporal coverage is published on this schema entry: extent (computed from landed data, with computed_at) and known_holes (declared gaps with provenance).45,609,639LessWrong · in ForumsPrimary forum corpus: source-native posts and comments across thousands of forum sites (LessWrong, DataSecretsLox, DEV, EA Forum, 4chan boards, crypto governance, plus a long tail); the live row and site counts are the census (scry.io/sources, /v1/stats/wire). site_key is the community identity: the source column names only the curated sources, and a large share of rows carry source = 'manual', so enumerate site_key — not source alone — before scoping a community question. Archived base refreshed in place by live source-native forum tails.926,187EA Forum · in ForumsPrimary forum corpus: source-native posts and comments across thousands of forum sites (LessWrong, DataSecretsLox, DEV, EA Forum, 4chan boards, crypto governance, plus a long tail); the live row and site counts are the census (scry.io/sources, /v1/stats/wire). site_key is the community identity: the source column names only the curated sources, and a large share of rows carry source = 'manual', so enumerate site_key — not source alone — before scoping a community question. Archived base refreshed in place by live source-native forum tails.193,136Video & audio held155,132,133 of the estate<0.1% sources1 Public podcast episode records. sourcedescriptionheldPodcast episodesPodcast episodes swept from RSS across 4.2M feeds (uniq, 2026-09-13), counted as distinct (feed, episode) pairs; the table keeps one row per sweep until the dedup merge, so its physical row count runs higher.155,132,133Writing & web held67,936,864,668 of the estate16% sources11 Public mailing-list and Usenet messages, the Linux kernel list, netdev, Fedora, extropians, and the long tail among them, threaded by reply and root and queryable by list, author, date, and text.A bibliographic catalog across OCLC/WorldCat, OpenLibrary, ISBNdb, and Google Books; public-domain full text in passages; FanFiction.net stories from 1998 to 2015; Wikipedia articles embedded for vector ranking. Archived web pages and the cleaned reading layer of Common Crawl, with WARC provenance. sourcedescriptionheldCommon Crawl indexThe Common Crawl per-capture URL index (CDX): one row per WARC capture — full URL decomposition, fetch status, MIME, CLD2 languages, content SHA-1 digest, and the exact WARC (filename, offset, length) provenance triple that re-derives the raw record from CC's public hosting. The existence-and-provenance layer: what the web had at a URL or domain, without text cost; content_digest joins duplicate content across captures.45,468,367,569Common Crawl pagesWhole-page extracted text — boilerplate (nav, cookie banners, footers) included; the same URL recurs across epochs by design (one row per capture). The recall layer: for indexed clean text grep commoncrawl.distillate first and hydrate here.18,411,657,043Mailing listsPublic mailing-list and Usenet archive messages (extropians, SL4, and ~34k more lists); payload is the message text, threaded by parent/root keys.1,608,150,038Web crawl pagesText extractions of observed web pages across every observed host — the live open-web corpus. (The /search crawled_url family still reads the earlier static archive in internet.documents; this relation is the fresh one.)958,378,452Web crawl artifactsThe crawl's stored captures: the fetched bytes behind each page, kept in segment packs.722,893,601WikidataWikidata items and claims from the weekly entity snapshot (CC0).373,834,895Internet documentsInternet text documents across 85 source families (source keys e.g. crawled_url, lesswrong, mailing_list): one row per document, with content_text, uri, source, and quality fields. A frozen federated snapshot (nothing after ~2026-04) — contents overlap but do not match the live per-source tables; the live-forum sources' current rows land in forums.posts beside this archive. A record_ref '<source>:<uuid>' resolves here by id (non-UUID refs resolve in forums.posts). The crawled_url family is the earlier static web archive; the live corpus is crawl.pages.204,516,642Substack commentsComments beneath the Substack posts above.87,012,347Common Crawl distillateThe cleaned reading layer of Common Crawl: genre-classified pages (article / forum / mailing_list) selected by link-graph centrality and URL-shape predicates, with boilerplate-free extracted text (~4.3KB avg vs ~8.2KB raw WET). Tier A rows also carry raw html, tier B text only, tier C provenance only — every row keeps the WARC triple to re-derive the original from CC hosting. Token-indexed (hasToken over lower(text)).64,994,473Substack postsSubstack posts, swept across a roster of publications that grows in place, custom domains included.30,727,357WikipediaEnglish Wikipedia articles: one row per main-namespace page with title, full article text (payload), word count, categories (JSON), and quality flags — the full 6.2M-page set landed 2026-03 plus new articles from the recentchanges feed. embeddings.wikipedia_articles is its semantic companion (page_id).6,332,251Scholarship held9,364,826,415 of the estate2.3% sources11 The OpenAlex works graph with citation edges, author profiles, and DOI identifiers where supplied. arXiv, PubMed, PMC, Europe PMC with bioRxiv and medRxiv, INSPIRE-HEP, and a DOI-keyed journal full-text corpus, folded into one catalog that states which corpora hold each paper and whether its full text is on hand. ClinicalTrials.gov study records, one current row per NCT id. sourcedescriptionheldBook passagesBook passages with books-norm-2 normalized text, packed to at most 1,024 cl100k tokens with a section heading, exact token counts, and character offsets.3,334,453,872OpenAlex citationsReverse citation edges of the OpenAlex graph: one row per (cited work, citing work) with the citing work's publication date — who cites this paper, newest first, in milliseconds. Follow references forward through works.referenced_works, citations backward here; hydrate either end via openalex.works.3,086,374,182Library catalog recordsBibliographic catalog records across OCLC/WorldCat, OpenLibrary, ISBNdb, and Google Books.1,527,812,492OpenAlex worksOpenAlex scholarly-work metadata snapshot (the 2026-06-26 public release; nothing changes between releases, so freshness_lag is the release's age and extent.max is a publisher-assigned date, not an arrival date): title, venue, year, DOI, authorships (author id, name, ORCID, institutions, corresponding flag), topics, concepts, keywords, funders, open-access locations, citation counts, and the citation graph (referenced_works). One row per work version in a ReplacingMergeTree (count works with uniq(id), an estimate within about 1%). doi_norm (lowercase bare DOI) joins academic.papers full text via doi_norm = decodeURLComponent(paper_key); paper_key keeps the first slash literal (10.48550/arxiv.2211.09527) and percent-encodes later ones (10.18653/v1%2f2024.acl-long.331). An arXiv DOI (10.48550/arxiv.<id>) names the preprint work, type = 'preprint', with its own cited_by_count; the published version is a separate work under the journal DOI (2404.18213: 15 citations as the preprint, 41 as its 2025 journal article), and academic.catalog's doi carries that journal DOI when one exists. counts_by_year is Array(Tuple(year, cited_by_count)) with Nullable members: citations in a window are arraySum(arrayMap(t -> coalesce(tupleElement(t, 2), 0), arrayFilter(t -> coalesce(tupleElement(t, 1), 0) BETWEEN 2012 AND 2015, counts_by_year))) — tupleElement, never t.1, and coalesce before the sum (0.8 s over the table beside a token anchor).511,893,275Paper extractionsEvery academic-paper text-extraction attempt and revision: status, error class, text-quality measures, and the text itself.396,105,644CrossrefThe Crossref public data file (March 2026): one row per registered DOI work.179,535,192PatentsPatent publications from the Google Patents public dataset.170,418,479Academic papers (full text)Filter quality_label = 'good' for the strictly faithful core. Rows whose quality_flags contain 'tex_pandoc' or 'tex_clean' (the arXiv TeX route) carry 'good' by construction, not by measurement; in 'tex_pandoc' rows LaTeX macros pandoc could not expand are dropped silently, and 'tex_clean' rows keep raw unexpanded TeX, so counts and other macro-valued numbers may be missing — verify against another rendering. pdf_bytes > 0 marks a PDF-backed row (the source PDF's size); converter names the route the text took (PDF text layer, OCR, arXiv TeX or ar5iv HTML, PMC XML, a preprint abstract). How many rows, papers, PDF-backed or TeX-derived texts, and tokens are on hand is served measured on this entry's corpus_status (with computed_at), never quoted here.91,219,378PubMedNCBI's annual PubMed baseline plus every daily update file, revisions and deletions settled in order.42,915,537Book & magazine full textFull text of books, magazines, and standards documents: one row per extracted file with format (epub, pdf, djvu, fb2, html, txt), converter, quality label, and the complete extracted text. md5 joins books.catalog (record_id = concat('md5:', md5)) for title, author, year, and language. Grows continuously as new collections land — fiction and non-fiction, Russian fiction, world magazines, standards documents, English non-fiction epubs, and a large scanned-book collection.17,168,360arXivarXiv's bibliographic feed, with the TeX source and full text on hand where arXiv supplies it.6,930,004Markets held2,104,488,921 of the estate0.5% sources5 Kalshi (largely auto-minted financial-range series), Polymarket, and Manifold as one catalog of markets with their resolution context, plus Manifold bets, each carrying the market probability before and after it, and public Manifold comments. sourcedescriptionheldMarket trades (unified)One unified trade stream across Kalshi, Polymarket, and Manifold, each trade carrying the market probability before and after it.631,974,838Kalshi tradesPublic Kalshi executions: realized price and liquidity history, landed within minutes of the trade.587,140,920Kalshi marketsEvery Kalshi market as the public API shows it, swept continuously: status, settlement, and probability observations, append-only.574,290,664Polymarket tradesPublic Polymarket trades, one row per execution.305,869,551Polymarket marketsPolymarket markets: title, category, status, resolution, open/close/settle times, volume, liquidity, probability.5,212,948Records held1,717,498,382 of the estate0.4% sources3 SEC EDGAR filing documents in full text from 1993 on, from annual and quarterly reports to insider filings. The DOJ Epstein releases as a queryable artifact index. The Trace ledger of AI-safety grants and donations, compiled by Manifund from public sources. sourcedescriptionheldGoogle TrendsGoogle Trends top and rising search terms by US metro and by country, each daily refresh carrying the term's weekly scores; hourly through May 2026.1,048,992,260FEC transactionsFEC campaign-finance transactions by committee and candidate, from the FEC bulk files.434,463,850SEC filingsSEC EDGAR filings — annual and quarterly reports, current reports, insider filings — in full text, plus the quarterly financial-statement numbers.234,042,272Code held43,904,313,001 of the estate11% sources5 GitHub: archival observations of repositories that were public when captured; the outcome of the archive’s latest visit is recorded separately, and an owner lookup returns the repositories the archive observed under that owner. The package registries as one catalog: npm, Go modules, PyPI, NuGet, Packagist, crates.io, and other platforms through Libraries.io. sourcedescriptionhelddeps.dev graphThe deps.dev dependency graph: package-to-package edges across open ecosystems, from the public snapshot.25,192,519,296GitHub archive eventsThe GH Archive event stream, hour by hour: every public push, issue, pull request, star, and fork on GitHub.11,180,431,988Software Heritage revisionsCommits from the Software Heritage graph export — the history layer over every archived forge origin.5,940,997,207Package registriesPackage-to-repository links across npm, Go, PyPI, NuGet, Packagist, crates.io, and about thirty more registries, as one catalog.1,152,917,663Software Heritage originsSoftware Heritage archival observations of repositories that were public when captured; the outcome of the archive's latest visit is recorded separately.437,446,847Undisclosed held254,674,013,137 of the estate62% sources1 Sources under reviewed access are counted in the held estate, not queryable by default. sourcedescriptionheld20 undisclosed sourcesSources under reviewed access are counted in the held estate, not queryable by default.254,674,013,137 * how the count is kept * counted since the census began. Landed means newly held: a row the census counted for the first time, whether captured live or backfilled from an archive — so the rate is the estate’s intake rather than the internet’s posting rate. A count that fell — crawls retired to cold storage, duplicates merged away — is not a landing. Held and missing are published, never rounded away. Reddit numbers its comments with one global counter, so the 6% not held is an exact subtraction, and every relation carries its computed extent and declared holes on its schema entry. An empty result can reflect absent records or a coverage gap. Browse the source catalog → Vectors Embedding spaces carry useful linear structure, and Scry exposes it directly. the difference between two embeddings is often a direction with a meaning; the mean of several points at a shared concept; projecting one onto another keeps or removes one component of what a text is about. Most systems hide embeddings behind a similarity API and throw that structure away. In Scry a vector is a first-class value: mint one from any text; add, subtract, and project them in a query; rank a whole corpus along the result. Text search speaks a full operator grammar — exact phrases, exclusion, OR, regex, fuzzy, proximity — and scry_lex carries that whole grammar into the query as one predicate, compiled server-side to each relation’s own text indexes. Everything below runs as written against the live schema. @metric@school@reward@proxy0 scry_centroid([@metric, @school, @reward]) — three descriptions averaged into one concept, a direction to rank a corpus by. @unreachable0@achievableevery post, dropped onto one axis scry_contrast_axis_balanced(@achievable, @unreachable) — two sentences’ difference is a direction; a document’s cosine to it is where it lands, everything orthogonal set aside. unreachableachievableLessWrong: median -0.053 over 62,473 on-topic documents (10–90%: -0.137 to +0.043)LessWrongMoltbook: median -0.015 over 130,405 on-topic documents (10–90%: -0.121 to +0.073)MoltbookarXiv: median +0.004 over 12,001 on-topic documents (10–90%: -0.075 to +0.080)arXiv The same axis over every corpus with Voyage-4 vectors, on-topic documents only (cosine ≥ 0.5 to the poles’ centroid): 10–90% run, middle half, median. The Congressional Record and Kalshi hold nothing on topic. Measured 2026-09-11, 13–67 s per corpus. Which small accounts posted LLM alpha this week that people saved but didn’t like? A bookmark is a stronger signal than a like and nobody can see who left it. Every archived tweet revision carries both counts, so “saved more than liked” is one predicate; a token list keeps it on coding agents and inference, a second one drops the giveaway bait, and a follower ceiling keeps it to accounts you have never heard of. Three days, 204 million revisions read, 4.9 s. SELECT any(author_handle) AS handle, max(author_followers) AS followers,
max(like_count) AS likes, max(bookmark_count) AS saves,
round(saves / likes, 1) AS saves_per_like, substring(any(text), 1, 150) AS text
FROM twitter.tweets
WHERE bucket_date >= today() - 3
AND hasAnyTokens(search_text_lc, ['subagents', 'mcp', 'evals', 'harness', 'worktrees', 'agentic',
'finetuning', 'rlhf', 'tokenizer', 'inference', 'codex', 'opus', 'sonnet', 'cursor', 'devin', 'ralph'])
AND NOT hasAnyTokens(search_text_lc, ['giveaway', 'retweet', 'free', 'course', 'bookmark', 'link',
'below', 'join', 'pdf', 'earn', 'money', 'income', 'breaking', 'viral', 'comment', 'dm', 'follow'])
AND lang = 'en' AND NOT startsWith(text, 'RT @')
AND bookmark_count >= 150 AND bookmark_count >= like_count AND author_followers < 20000
GROUP BY tweet_id
ORDER BY saves_per_like DESC
LIMIT 10; → @navaneethvb 558 followers 205 likes 337 saves "A lot of my LLM inference posts here were explaining how…"
@LLMpsycho 1,169 followers 146 likes 242 saves "security-audit-skill is a Cloudflare coding agent skill…"
@syrgkanis 1,087 followers 157 likes 198 saves "…we recently released a fully automated agentic pipeline…"
@jaimintf 2,873 followers 335 likes 467 saves "before → after. right one used mcp + skills, both one-shotted." Which way has LessWrong drifted on alignment: reachable, or out of reach? Mint the two poles as parallel sentences that differ only in stance, take the balanced axis between them, score every post once as the mean of its chunks’ projections, and average the posts quarter by quarter, joined to real post timestamps. The window function centers each quarter on the era mean, because the zero of a contrast axis only means equidistant from the two sentences; posts is the census of the quarter’s LessWrong posts proper from forums.posts alone (it holds comments too, fifteen to a post on LessWrong; p.kind = 'post' leaves them out), scored is how many of them the embedding join reached, and coverage is their ratio: read a drift only where coverage is near 1, and an unscored quarter keeps its row with drift NULL instead of vanishing. One query returns the trajectory of a community. -- Mint the poles once, then query:
-- POST /v1/scry/embed {"model": "voyage-4-lite", "name": "achievable", "text": "Robust alignment seems achievable by defense in depth: iterative training, scalable supervision, interpretability, red-teaming, corrigible tool use, formal checks where possible, and institutions that prevent single-point failure. No one technique must be perfect if several imperfect safeguards compose well."}
-- POST /v1/scry/embed {"model": "voyage-4-lite", "name": "unreachable", "text": "Robust alignment seems unreachable by defense in depth: training optimizes proxies, supervision fails once systems surpass us, deception can hide until deployment, interpretability is incomplete, and institutions face racing pressure. Combining several imperfect safeguards still leaves correlated catastrophic failure modes."}
SELECT quarter,
if(scored = 0, NULL, lean - avgIf(lean, scored > 0) OVER ()) AS drift,
scored, posts, round(scored / posts, 3) AS coverage
FROM (
SELECT quarter, avg(post_lean) AS lean, count() AS scored
FROM (
SELECT e.target_key,
toStartOfQuarter(any(p.original_timestamp)) AS quarter,
avg(scry_cosine_similarity(e.embedding,
scry_contrast_axis_balanced(@achievable, @unreachable))) AS post_lean
FROM embeddings.chunks AS e
JOIN forums.posts AS p ON p.post_key = e.target_key
WHERE e.source = 'forum_posts' AND e.model_name = 'voyage-4-lite'
AND p.source = 'lesswrong' AND p.kind = 'post'
AND p.original_timestamp >= '2009-01-01'
GROUP BY e.target_key)
GROUP BY quarter) AS s
RIGHT JOIN (
SELECT toStartOfQuarter(original_timestamp) AS quarter, uniqExact(post_key) AS posts
FROM forums.posts
WHERE source = 'lesswrong' AND kind = 'post' AND original_timestamp >= '2009-01-01'
GROUP BY quarter) AS c USING (quarter)
ORDER BY quarter
LIMIT 100; How fast did “vibe coding” catch on? The search grammar rides inside the query as one predicate — here as an aggregate operand, with the month’s whole item count as its denominator. Hacker News: zero through January 2025, 34 mentions in February (the first item, 2025-02-03, links Karpathy’s post), 494 in March, 590 in April; 4 s, measured 2026-09-07. SELECT toStartOfMonth(original_timestamp) AS month, count() AS items,
countIf(scry_lex('"vibe coding"')) AS mentions,
round(100000.0 * mentions / items, 1) AS per_100k
FROM hackernews.items
WHERE original_timestamp >= '2024-12-01'
GROUP BY month
ORDER BY month
LIMIT 36; → 2025-01 0 0.0 per 100k
2025-02 34 11.9
2025-03 494 169.4
2025-04 590 212.6 Who do Elon Musk, Sam Altman and Eliezer Yudkowsky all follow? The follow graph is a table, so “everyone all three of them follow” is one GROUP BY with a HAVING — a button Twitter never shipped in seventeen years. Ten accounts, Vitalik and JD Vance at the top, Ilya, Hinton and Scott Alexander further down, in 171 ms over 2.9 million rows read. WITH common AS (
SELECT followee_id FROM twitter.following
WHERE follower_id IN (44196397, 1605, 2595244026) -- @elonmusk, @sama, @ESYudkowsky
GROUP BY followee_id HAVING uniqExact(follower_id) = 3)
SELECT argMax(handle, observed_on) AS handle, argMax(followers, observed_on) AS follower_count
FROM twitter.users
WHERE author_id IN (SELECT followee_id FROM common)
GROUP BY author_id
ORDER BY follower_count DESC
LIMIT 10; → VitalikButerin 7,866,454
JDVance 5,551,484
paulg 5,243,364
karpathy 4,165,859
demishassabis 1,865,207 … waitbutwhy, ilyasut, geoffreyhinton, slatestarcodex, ShaneLegg Which papers cite both Scaling Laws and Chinchilla? Citation edges are native keys, so “related work” becomes a set intersection: every OpenAlex work that cites both Scaling Laws for Neural Language Models and Training Compute-Optimal Large Language Models, ranked by its own citation count. 195 shared citers, 0.1 s, measured 2026-09-07 against the 2026-06-26 OpenAlex snapshot. SELECT id, title, publication_year, cited_by_count
FROM openalex.works
WHERE id IN (
SELECT citing_work_id FROM openalex.cited_by
WHERE cited_work_id IN ('https://openalex.org/W3001279689', -- Scaling Laws, 2020
'https://openalex.org/W4225591000') -- Chinchilla, 2022
GROUP BY citing_work_id HAVING uniqExact(cited_work_id) = 2)
ORDER BY cited_by_count DESC
LIMIT 10; → Large language models encode clinical knowledge 2023 3,194
Can LLMs Transform Computational Social Science? 2023 429
ProGen2: boundaries of protein language models 2023 426 Who keeps arguing with whom on Hacker News? A self-join on parent_hn_id pairs every reply with the comment it answers; least/greatest make the pair unordered, and the HAVING keeps only pairs where both people replied to the other at least three times. Every regular feels the feuds; this is the first time anyone could count them — three weeks, 46.7 million rows read, 1.6 s. SELECT least(a.original_author, b.original_author) AS one,
greatest(a.original_author, b.original_author) AS other,
count() AS replies, uniqExact(a.story_hn_id) AS threads
FROM hackernews.items AS a
INNER JOIN hackernews.items AS b ON a.parent_hn_id = b.hn_id
WHERE a.original_timestamp >= now() - INTERVAL 21 DAY
AND a.original_author != b.original_author
GROUP BY one, other
HAVING countIf(a.original_author = one) >= 3
AND countIf(a.original_author = other) >= 3
ORDER BY threads DESC, replies DESC
LIMIT 8; → JumpCrisscross ↔ toomuchtodo 11 replies 8 threads
HarHarVeryFunny ↔ famouswaffles 39 replies 6 threads
mitxela ↔ ranger_danger 20 replies 5 threads
WalterBright ↔ nrr 21 replies 4 threads What did forecasters change their minds about this week? Every Manifold bet carries the market probability before and after it, so a market’s week is argMin of the first price and argMax of the last, joined to its latest title. Binary markets with thirty or more bettors, outside Manifold’s own unranked topic, ranked by how far the crowd moved — the news before the news, 27 million bets in 268 ms. WITH moves AS (
SELECT contract_id,
argMin(prob_before, created_at_source) AS p_start,
argMax(prob_after, created_at_source) AS p_now,
uniqExact(user_id) AS bettors
FROM manifold.bets
WHERE created_at_source >= now() - INTERVAL 7 DAY
AND prob_before != prob_after AND answer_id IS NULL
GROUP BY contract_id HAVING bettors >= 30)
SELECT m.title, round(100 * p_start) AS was, round(100 * p_now) AS now, bettors
FROM moves
INNER JOIN (SELECT market_id, argMax(title, observed_on) AS title
FROM manifold.markets
WHERE market_id IN (SELECT contract_id FROM moves)
GROUP BY market_id
HAVING NOT has(JSONExtract(argMax(raw_market, observed_on),
'groupSlugs', 'Array(String)'), 'unranked')) AS m
ON m.market_id = moves.contract_id
ORDER BY abs(p_now - p_start) DESC
LIMIT 8; → Assassination attempt on a major AI-lab CEO by 2030? 30% → 99% 69 bettors
Will Trump interfere with Greenland? 80% → 17% 32
US average gas price $4.40 or more on September 21? 50% → 97% 59
All Millennium Prize Problems solved before EOY? 50% → 3% 31
Opus 5.2 released in September? 50% → 24% 93 Text scry_lex('"exact phrase" -noise /regex/')the whole search grammar as one predicate a NEAR/50 b · "phrase"~3 · word~1proximity, slop, and typo tolerance, index-pruned hasAllTokens(search_text_lc, […])token-indexed match over full archives — every row, not a ranked sample toStartOfMonth · uniq · argMintime series, distinct authors, first observations JOIN … ON post_key / hn_id / contract_idsource-native keys across relations Meaning POST /v1/scry/embed → @handlemint a named vector from any text scry_contrast_axis(@a, @b)the axis between two ideas scry_project_onto · scry_debias_vectorkeep, or remove, one component of meaning scry_seed_centroid([@x, @y, …])a concept built from examples scry_cosine_similarity · scry_handle_matrixmeasure one pair, or all pairs at once scry_vector_topk_distanceANN-rank an indexed relation along any of the above More on the query examples and vector search pages. Pricing Researcher $0 $5 of credit at signup, no cardAPI key and MCPNon-commercial StartPatron $100 / month $100 a month lands as balance and rolls overNon-commercial Become a patronTeam from $2,000 / month Commercial useDedicated capacityCustom source builds Contact usAgents $0.05 / second declared x402, no accountthe declared time box is the charge Agent guide Built a parser or crawler for a source Scry should cover? Paste a link to the repository, gist, or documentation. We review submissions by hand and wire good ones into the corpus with credit. Submit Scry is not supposed to suck. If it does in some way, please tell us how (copies our email), and we will try to quickly adapt and overcome. Measured Every number here was timed through the public API. Single server-side executions on the live system under production load, read 2026-09-11. Over tens of billions of rows a query takes ten milliseconds to a few seconds; the cost is what it has to touch. 0.86 s every byte of held Hacker News text: 15.5 billion characters, 18 GB/s. 7 ms the count of all 30.6 million posts containing “the”, from the index alone. Twenty of them fetched: 39 ms. 29 ms Hacker News on ChatGPT, ninety minutes before the launch post. 65 constructs the query alphabet on eight axes (anchor, group, window, recurse, fuzzy, ASOF), one canonical query each under a held control; 71 queries timed. 98 ms median 65 of 71 under a second; the slowest, 2.6 s. The full tree → each construct a child of the one before it: query, scope, wall time. Press a clock and the bar runs exactly as long as the query did. Scry /skraɪ/ helping minds see further by Unflatten Product DocsPricingSourcesQuery examplesPublic queriesAgentCompany UnflattenTeamStatusDonatePrefer us on GoogleLegal TermsPrivacyPrinciplesSecurityPGP ~*~ © 2026 Unflatten ~*~ hi@scry.io

Scry is presented as a Master Control Program (MCP) designed to facilitate programmatic internet research by allowing agents to run complex programs over billions of internet documents. The core concept moves beyond traditional search by enabling agents to query the underlying records of the web rather than merely assembling results from repeated searches or scraped data. This functionality is supported by a schema that catalogs relations, columns, extent, and known holes for every source, functioning as a discovery authority for the entire corpus. The system processes data across a vast estate, including metrics such as the held estate, which encompasses trillions of rows across diverse sources.

The system integrates massive, granular datasets from numerous sources, including Reddit, Hacker News, arXiv, Stack Exchange, Wikipedia, and various forum archives. These sources are organized into structured relations, such as reddit.posts, hackernews.items, and academic catalogs, each carrying provenance data, temporal coverage, and declared gaps. This structure allows agents to execute sophisticated queries using logic encompassing phrase matching, time windows, joins, and vector composition across these relations. The methodology involves agents reading the live schema to select source-native relations and vector helpers, then constructing bounded queries that operate directly on the data, ensuring that results retain source-native identifiers and provenance fields.

The power of Scry lies in its ability to enable Turing-complete search programs. Agents can use Scry to compute over these documents in a singular query, contrasting with sequential fetching. This capability is further enhanced by vector embeddings, which are treated as first-class values, allowing for advanced mathematical operations such as calculating centroids of concepts, determining contrast axes between ideas, and measuring cosine similarity across the corpus. This vector structure allows for sophisticated analyses, such as deriving abstract concepts from disparate data points, like averaging descriptions of metrics, schools, and rewards to find a centroid.

Scry supports advanced analytical operations that mirror complex reasoning. For instance, it can be used to quantify nuanced relationships, such as analyzing the trajectory of community drift by comparing embeddings across time windows, or identifying interconnected social dynamics within forum discussions. The system also supports complex graph traversal, allowing agents to navigate reply graphs in forums to count interactions between users, effectively counting feuds or overlaps in conversation threads.

The operational model integrates Scry into external systems via an MCP server, enabling seamless access for assistants like ChatGPT and Claude through custom connectors. This integration allows agents to deploy these powerful search programs without needing to execute code externally. The system provides an HTTP API for direct interaction, offering endpoints for schema discovery, executing queries, reranking held documents, and compiling complex search grammars. Performance metrics indicate that queries execute quickly, with the cost dynamically reflecting the compute required, which is tied to the size and complexity of the relations touched. The process involves token filtering based on source-native relations, and the system is governed by strict legal terms prohibiting bulk redistribution or the creation of competing corpora from the results.

The system’s architecture ensures that data integrity is maintained by keeping source timestamps, observation times, and quality fields distinct. This structure mandates reporting the exact SQL, relations, row counts, and accounting fields returned for every operation. The system is positioned as public-benefit infrastructure, designed to provide transparent, traceable access to public evidence for independent, noncommercial research, supporting the goal of unflattening the public record.