Last updated: Sep 28, 2026

DHG 502 syllabus · Slides · Assignment 1

Week 4 — Collocations

1. What is a collocation?

A collocation is a pair of words that tend to occur near each other, or cooccur, in natural language. 喝–茶 (drink–tea) and 天氣–好 (weather–good) are collocations; 喝–桌子 (drink–table) is not. Firth (1957): You shall know a word by the company it keeps.

The question is how to measure this attraction and tell it apart from chance. The idea is to compare the real text with a text in which the words have been shuffled at random. If two words cooccur in the real text far more often than they would after shuffling, they form a collocation.

For historians this matters because the company a word keeps is shaped by the people who wrote the source. In an official history such as the Mingshi 明史 (History of the Ming, compiled at the Qing court and completed in 1739), the words that cluster around 宦官 (eunuchs), 倭 (Japanese pirates), or the name of a minister tell us how the compilers framed that group or person: their actions, offices, and moral judgments.

2. Observed frequency O and expected frequency E

Fix two words: the target word X (in the slides: 王婆) and the collocate Y (痛苦).

• Observed frequency O: how many times Y actually appears inside a window of k words to the left and right of X.
• Expected frequency E: how many times Y would appear there by chance alone. Analogy: 10 balls, 5 of them red; draw 4 and you expect \(4 \times 0.5 = 2\) red balls.

E takes three steps:

1. The chance that a random word is Y: \(P(Y) = \dfrac{\text{count}(Y)}{N}\), where \(N\) is the number of words in the corpus.
2. The number of slots around X: \(\text{count}(X) \times \text{window size} \times 2\) (left and right).
3. \(E = P(Y) \times \text{number of slots}\).

Example (illustrative numbers):

• Corpus size: \(N = 50{,}000\)
• Target: count(王婆) \(= 200\)
• Collocate: count(痛苦) \(= 50\)
• Window: 3 + 3

Calculation:

• Probability: \(P(\text{痛苦}) = \dfrac{50}{50{,}000} = 0.001\)
• Slots: \(200 \times 3 \times 2 = 1{,}200\)
• Expected: \(E = 0.001 \times 1{,}200 = 1.2\)
• If we observe \(O = 8\), then \(\dfrac{O}{E} = \dfrac{8}{1.2} \approx 6.7\): the pair cooccurs 6.7 times more often than chance predicts.

3. Mutual information (MI) and PPMI

Taking the base-2 logarithm of \(O/E\) gives (pointwise) mutual information, MI:

\[\mathrm{MI} = \log_2 \frac{O}{E}\]

\(\log_2 x\) is the power you raise 2 to in order to get \(x\). So every doubling of the ratio adds 1 to MI, every halving subtracts 1, and \(O = E\) gives MI \(= 0\). In the example: \(\mathrm{MI} = \log_2 6.7 \approx 2.74\).

PPMI (positive pointwise mutual information) sets every negative value to 0:

\[\mathrm{PPMI} = \max\!\left(0,\ \log_2 \frac{O}{E}\right)\]

A negative value means the words repel each other, but such values are unreliable unless the corpus is very large. PPMI is common in word vectors.

4. A problem with simple measures

Evert gives an example from a corpus of about one million bigrams. Here Evert does not use a window but only looks at bigrams: two adjacent words, where the collocate Y is the word directly before the target X. Each target has exactly one such slot, so the number of slots is just count(X), and E becomes:

\[E = P(Y) \times \text{count}(X)\]

Compare two pairs (Y is the first word, X the second; \(N = 1{,}000{,}000\)):

Both have \(O = 10\) and \(E = 1\), so \(O/E = 10\) and \(\mathrm{MI} = \log_2 10 \approx 3.32\): identical scores.

Yet the situations are different. Iliad occurs only 10 times and every one follows the, so \(O = 10\) is the maximum possible. must and also occur 1,000 times each and could have cooccurred far more often. \(O\) and \(E\) ignore everything else in the corpus: we also need to look at the cases where the two words do not cooccur.

5. The contingency table

For two events \(X\) and \(Y\), every case falls into exactly one of four cells:

Contingency tables are everywhere: smoking × lung cancer, handedness × sex, two words × sentences. The question is always the same: are the two properties independent? For a word pair, the observed frequency \(O\) is the top-left cell \(a\).

6. Fisher’s exact test

Background: n choose k

The number of ways to choose \(k\) items out of \(n\), ignoring order, is

\[\binom{n}{k} = \frac{n!}{k!\,(n-k)!}\]

For example, choosing 4 cups out of 8: \(\dbinom{8}{4} = \dfrac{8!}{4!\,4!} = 70\).

The lady tasting tea

Fisher (1935): a lady claims she can tell whether the milk or the tea was poured into the cup first. She gets 8 identical-looking cups (4 milk-first, 4 tea-first) and must pick the 4 milk-first ones. The chance of getting all 4 right by guessing is only \(\dfrac{1}{70} \approx 0.014\), which is very unlikely; the chance of getting 3 right is \(\dfrac{17}{70} \approx 0.243\), which could easily be luck. This is the idea behind Fisher’s exact test: how unlikely is this table if it were all chance?

The probability of one table

With the row and column totals fixed, the probability of one particular table (a, b, c, d) under pure chance is:

\[p = \frac{\dbinom{a+b}{a}\dbinom{c+d}{c}}{\dbinom{N}{a+c}}\qquad (N = a+b+c+d)\]

The test adds up the probability of this table and of every more extreme table.

Example (illustrative numbers): 20 students (10 women, 10 men); 10 of them are studying, and 8 of those are women. “More extreme” means even more studying women (9 or 10). The three tables (all row and column totals fixed):

Adding the three: \(p = \dfrac{2126}{184756} \approx 0.0115 < 0.05\), so the association between sex and studying is significant.

Back to the Iliad / must also

The two pairs have the same \(O\) and \(E\); now look at the full contingency tables. Every item is a bigram (1,000,000 in total); rows = the first word, columns = the second word.

the Iliad (\(a = 10\), count(the) = 100,000, count(Iliad) = 10)

must also (\(a = 10\), count(must) = 1,000, count(also) = 1,000)

The difference is in \(c\): for the Iliad \(c = 0\) (every Iliad follows the); for must also \(c = 990\) (most instances of also have no must before them). Fisher’s test gives:

• the Iliad: \(p \approx 1 \times 10^{-10}\)
• must also: \(p \approx 1 \times 10^{-7}\)

Both are significant, but the Iliad’s \(p\) is 1,000 times smaller. Ranking by p-value puts the Iliad first, which \(O/E\) and MI cannot do. But a p-value answers a particular question, as the next section shows.

7. Statistical significance vs. strength of association

Evert calls all these scores association measures, but they fall into two families that answer different questions:

Two unfair coins

Coin A is flipped 10 times and lands heads 9 times: a strong bias (90% heads), but little evidence (\(p \approx 0.011\)). Coin B is flipped 100,000 times and lands heads 51,000 times: a very weak bias (51% heads), but overwhelming evidence (\(p \approx 10^{-10}\)). Coin B is more significant; coin A is more strongly biased. A p-value measures how sure we are, not how big the effect is. Frequent word pairs behave like coin B, rare pairs like coin A.

Same proportions, bigger corpus

Take (喝, 茶) with a 4 + 4 window, and make the corpus bigger while every proportion stays the same (values from the full contingency table, as computed in qhchina):

The strength measures do not move; the significance measures keep growing. In a corpus the size of the Mingshi (millions of characters), a great many pairs are “significant”, so the p-value alone cannot tell you which collocations matter.

Log-likelihood (G²): a significance measure

Log-likelihood compares each of the four observed cells with the cell we would expect if the two words were unrelated:

\[G^2 = 2 \sum_{\text{4 cells}} O \cdot \ln \frac{O}{E}\]

Each cell contributes more when \(O\) is far from \(E\) and when the counts are large. As a rule of thumb, G² above 3.84 corresponds to \(p < 0.05\) and above 10.83 to \(p < 0.001\). It is a fast approximation to Fisher’s test (Dunning 1993) and answers the same question: how sure, not how strong.

Dice and logDice: strength measures

Dice asks what share of the two words’ occurrences they spend together:

\[\text{Dice} = \frac{2 \cdot O}{R_1 + C_1} \qquad \text{logDice} = 14 + \log_2 \text{Dice}\]

\(R_1\) is the number of slots around the target (row 1 of the table) and \(C_1\) is the frequency of the collocate (column 1). For (喝, 茶): \(\text{Dice} = \dfrac{2 \times 12}{800 + 40} \approx 0.029\) and \(\text{logDice} \approx 8.87\). The maximum is 14 (the words always occur together), and each point lower means half as strong. Because \(N\) does not appear in the formula, logDice can be compared across corpora of different sizes (Rychlý 2008).

The measures can disagree

The significance measures prefer the Iliad: every Iliad follows the, so the evidence is very consistent. logDice prefers must also: the occurs 100,000 times and only 10 of those precede Iliad, so from the point of view of the the tie is weak. There is no single correct measure; decide which question your research needs answered.

In practice

A common workflow: filter by significance, then rank by strength.

1. Keep only pairs that are unlikely to be chance (p-value or G²).
2. Because you test thousands of candidate collocates at once, some will look significant by luck; correct for this with correction="fdr_bh" and filter on the adjusted p-value.
3. Rank what remains by a strength measure such as logDice.
4. Go back to the passages (KWIC concordance). The numbers point; the texts explain.

from qhchina.analytics import find_collocates

df = find_collocates(
    sentences,
    target_words="魏忠贤",
    horizon=5,
    measures=["log_likelihood", "logDice"],
    correction="fdr_bh",
    filters={"max_adjusted_p": 0.05, "min_obs_local": 3},
    sort_by="log_dice",
)

8. Three ways to define “near”

The contingency table stays the same; only the item being counted changes. First decide what an item is, then ask whether each item contains X and whether it contains Y.

Example: finding (好, 天氣) in a small corpus.

• Window: count the words inside the window.
• Sentence: count the sentences that contain both 好 and 天氣 (a sentence with two of them still counts once).
• Grammatical: count only adjective → noun modifiers; a predicative use such as 心情不好 does not count.

In each case \(O\) is the cell \(a\) of that table. Evert’s the Iliad example uses the smallest possible window: only the word immediately before.

9. In-class exercise: collocations in the Mingshi

We use 明史.txt, which is already in your dhg502 repository from the template (classical Chinese, traditional characters, UTF-8). Choose a target: a person (e.g. 魏忠賢, 張居正, 海瑞), a group (宦官, 倭, 流賊), an office (巡撫, 錦衣衛), or a concept (忠, 邊). Keep a short note of why you chose it and what you expect to find.

Step 1: Classical Chinese needs care

Tools such as jieba were trained on modern Chinese. On classical text they often split names and titles incorrectly, and many meaningful classical words are a single character. Two simple fixes help:

• Convert to simplified characters first (jieba works better with them), and write your target word in simplified characters too (張居正 → 张居正), or nothing will be found.
• Give jieba a user dictionary with the names, offices, and terms your question depends on, so that they stay in one piece. Check a sample of the segmented output by eye.

Step 2: Hand it to the coding assistant

Paste the prompts below into OpenCode one at a time, adjusting the file names and the target word. See the find_collocates() documentation and the load_stopwords() documentation for all parameters.

明史.txt is the History of the Ming (classical Chinese, traditional characters, UTF-8).
Please install jieba, qhchina, and opencc if they are not installed yet.

Write a Python script (segment.py) that:
- loads 明史.txt
- converts the whole text to simplified Chinese characters with opencc
- loads a user dictionary for jieba from userdict.txt (one word per line); create this file with the target words I care about: TARGET WORDS, plus related names and official titles
- splits the text into sentences using Chinese sentence-ending punctuation (。！？；)
- tokenizes each sentence into words with jieba, removing punctuation marks
- keeps only sentences with at least 5 words
- saves the result as sentences.txt, one sentence per line, with the words separated by a single space
- if the source marks its chapters (卷), also saves sentence_index.csv with the 卷 number and title for each line of sentences.txt
- prints 20 random segmented sentences that contain one of my target words, so I can check the segmentation by eye

Using sentences.txt, write a Python script (collocates.py) that finds collocates for the target word "TARGET WORD" (in simplified characters) with find_collocates from qhchina.analytics.collocations.
- Remove classical Chinese stopwords with load_stopwords("zh_cl_sim") from qhchina.
- Keep only collocates with at least 2 characters and with min_obs_local of at least 3.
- Add the measures "log_likelihood" and "logDice", use correction="fdr_bh", and keep only collocates with an adjusted p-value below 0.05.

Run it three times: method="window" with horizon=5, method="window" with horizon=10, and method="sentence". Sort each result by log_dice from high to low, save each one to its own CSV in output/ (name the files so I can tell which run is which), and print the top 20 rows of each.

Write a Python script (make_report.py) that reads the CSV files in output/ and builds a single page, output/results.html, with a dropdown to switch between the runs. In each table, let me click a column header to sort by it, so I can compare the ranking by log_likelihood with the ranking by log_dice.

Write a Python script (concordance.py) that uses kwic from qhchina.analytics.collocations to show, for the target word "TARGET WORD" and each of the collocates I list here: COLLOCATE 1, COLLOCATE 2, COLLOCATE 3, up to 10 passages where both occur (horizon=10). If sentence_index.csv exists, add the 卷 number and title to each line. Save the result as output/kwic.html.

Step 3: Observe and interpret

Open output/results.html and compare the runs:

• Can the collocates be grouped? (People, offices, places, actions, punishments, moral judgments …)
• Sort by log_likelihood, then by log_dice. Which collocates move up, which move down, and why? Which of the two rankings is closer to your historical question?
• Compare horizon=5 with horizon=10, and the window with the sentence method. Does the definition of “near” change what you find?
• Read the KWIC passages. Does a collocate mean what you assumed? Is it a real association, an artefact of segmentation, or a formula of the genre (e.g. the standard phrasing of appointments and deaths in the biographies)?
• Whose voice is this? What do the collocates tell you about how the Qing compilers framed your target, and what can they not tell you?

When you are done, ask the coding assistant to commit and push segment.py, collocates.py, make_report.py, concordance.py, userdict.txt, and the output/ folder. The same workflow is the basis of Assignment 1.

10. Recap

1. Collocation: two words that occur near each other more often than chance.
2. Observed \(O\) vs. expected \(E\); the log gives MI, dropping negatives gives PPMI.
3. Contingency table: also records the cases where the words do not cooccur (a, b, c, d).
4. Fisher’s exact test: how unlikely is this table if it were all chance?
5. Significance is not strength: p and G² ask how sure we are; MI and logDice ask how strong the tie is.
6. Three kinds of “near”: window, sentence, grammatical: the same table with different items.

Readings

• Stefan Evert, “Corpora and Collocations” (main reference for this week).
• (optional) Paul Baker, Using Corpora in Discourse Analysis (2006), Chapter 5, “Collocations”.
• (optional) Ted Dunning, “Accurate Methods for the Statistics of Surprise and Coincidence,” Computational Linguistics 19, no. 1 (1993).
• (optional) Pavel Rychlý, “A Lexicographer-Friendly Association Score” (2008).
• qhchina documentation: find_collocates(), load_stopwords().