Sources

Taylor's central contribution is the procedure itself: delete every nth word (typically every fifth) from a passage, present the mutilated text to readers, and score their ability to restore the missing tokens. The resulting Cloze score correlates with traditional readability formulas and, critically, treats comprehension as a predictive act. Pages 415-419 of the original Journalism Quarterly article establish the methodology; pages 420-428 validate it against Dale-Chall and Flesch readability measures. The term "cloze" itself is a contraction of Gestalt psychology's "closure," signaling that Taylor saw completion as the fundamental cognitive operation being measured.

For the article's purposes, Taylor establishes the conceptual precedent: probing a language system by withholding information and observing what it restores. The technique is identical to masked-language-modeling; only the language system has changed.

Shannon's 1948 paper introduces the mathematical foundation for quantifying information. Section 6 (pages 393-396) defines entropy H = -sum(p_i log p_i) as the average uncertainty of a random variable, and Section 7 applies it to English text by modeling letter sequences as Markov processes of increasing order. The progression from zero-order to third-order approximations on pages 388-389 shows how each additional context token reduces uncertainty, which is precisely the mechanism a modern language model exploits.

The relevance to prompt engineering is structural rather than historical: entropy gives us the language to say why prompts work at all. A prompt reduces the conditional entropy of the next token, and the Markov-chain figures in Shannon's paper are the first published examples of conditional text generation.

The 1951 paper is more immediately relevant to prompting than the 1948 paper. Shannon estimates the entropy of English by asking human subjects to guess successive letters of a text they have not seen, recording how many guesses each letter requires. The experimental setup on pages 52-55 is functionally identical to autoregressive next-token prediction: condition on prefix, sample candidate, score the candidate against ground truth. Shannon's bounds of roughly 0.6 to 1.3 bits per letter (page 64) remained the standard benchmark for decades.

For the article, this paper is the clearest early example of treating language as a prediction problem that can be measured, not just modeled. The methodological ancestry from Shannon's guessing game to log-probability-based evaluation of language models is direct.

Section 2.1 of the GPT-3 paper defines the three in-context learning regimes (zero-shot, one-shot, few-shot) that became the vocabulary of prompt engineering. Figure 1.2 on page 4 shows the accuracy-versus-parameter-count curves for each regime, establishing that few-shot performance emerges with scale. The evaluation suite in Section 3 covers 42 benchmarks, but the load-bearing claim is architectural: no gradient updates are required to adapt the model to a new task, so the prompt becomes the adaptation mechanism.

This is the paper that turned prompting from a convenience into a discipline. The article's framing of Week 3 as "prompt engineering as craft" rests on the GPT-3 result that prompt quality directly moves measurable benchmarks.

The paper's central experimental result is in Figure 4 (page 5): chain-of-thought prompting provides essentially no benefit for models below roughly 100 billion parameters, then provides a large benefit once models cross that threshold. Table 3 on page 7 reports the GSM8K arithmetic benchmark jumping from 17.9% to 56.9% on PaLM 540B when intermediate reasoning steps are included in the prompt. Section 6 frames the behavior as emergent, not continuous, which is the specific claim the article references.

The methodological contribution matters as much as the accuracy numbers: Wei et al. show that the prompt can elicit reasoning traces that the model is already capable of but does not spontaneously produce. The prompt shapes the output distribution rather than adding new capability.

Section 3 of the InstructGPT paper describes the three-stage training recipe: supervised fine-tuning on human-written demonstrations (SFT), reward model training on pairwise preferences, and proximal policy optimization against the reward model. Figure 1 on page 3 shows that a 1.3-billion-parameter InstructGPT is preferred to the 175-billion-parameter base GPT-3 by human labelers, which is the result that reset industry expectations about what scale alone can provide.

For the article, the relevance is stylistic: modern prompts read as imperative instructions ("summarize," "translate," "explain step by step") because the models were trained on exactly that distribution. The grammar of contemporary prompting is a direct consequence of the InstructGPT training pipeline.