In 1954, three years before Firth's famous one-line aphorism, a Penn linguistics professor named Zellig Harris published the seventeen pages of math behind it. Then the GPUs arrived, and the framework that scaled with corpus size was the teacher's.
In 1992, five researchers at IBM Yorktown Heights published a twelve-page paper on grouping English vocabulary into classes. Two of the authors would walk out of that group and help build the most profitable hedge fund in history.
From Shannon's hand-picked letters to modern LLMs. The real outputs from ELIZA, RACTER, char-rnn, and GPT, and why each generation felt like a breakthrough.
How a 1994 data compression algorithm became the foundation of modern AI. The untold story of Byte Pair Encoding's journey from C Users Journal to GPT-4.
Understanding how LLMs transform text into tokens, and why this seemingly simple process has profound implications for cost, context limits, and model behavior.
BPE, WordPiece, SentencePiece, Unigram. Four algorithms, four trade-offs, none of them know what a word is.
If you dumped every word of Pride and Prejudice into a hat and drew them out at random, the vocabulary would match the novel and the prose would be gibberish. The gap between the hat and Austen is exactly what PMI measures.
Tamil speakers pay 7x more tokens than English speakers for the same meaning. The hidden cost of tokenization and why morphology sets a compression ceiling.
Over a million AI agents registered on a social network built exclusively for them. They formed religions and drafted constitutions. A critical analysis of what is actually happening, and what the security implications mean.
Reddit usernames that break GPT. Invisible characters that bypass filters. The edge cases where tokenization fails spectacularly.
How a 2001 method for comparing corpora became a detector for AI-generated text, pasted content, and ghostwriters. Chi-squared drift detection across sliding windows.
251,022 tokens across five books, measured against the British National Corpus. The frequency data draws a portrait of a man who wrote with his lungs and his skin.
How words turn into coordinates, and why "king minus man plus woman" equals "queen". The story of how meaning became geometry.
Static embeddings gave "oracle" one vector for priestess, database, and Matrix character. Then attention learned to compute meaning from context.
Explore how LLMs manage context windows, from quadratic attention scaling to truncation strategies, and why the most expensive tokens are often the ones you never meant to send.