Last updated: Sep 26, 2026

Week 3: Building a Corpus, Regular Expressions (Regex)

1. What is a regular expression?

A regular expression (regex) is a small pattern language for describing the shape of text, rather than one exact string. Instead of searching for the literal word "1898," you can search for "any four digits," and instead of removing one specific footnote marker, you can remove "any bracketed number." Regex lets a computer find, extract, or clean text based on structure, which is exactly what most historical source-to-data work requires: pulling dates, names, or chapter headings out of OCR'd text; stripping page numbers and footnote markers; splitting a long file into consistent units (chapters, entries, records).

A regex is matched against text left to right. Most characters in a pattern are literal: they match themselves. A small set of characters (. * + ? ^ $ [ ] ( ) | \ { }) are metacharacters; they mean something special (see below). If you want to match one of these characters literally, you escape it with a backslash, e.g. \. matches an actual period rather than "any character."

2. The basics (RegexOne)

These building blocks come from the interactive practice site RegexOne, which covers the same core vocabulary used in Python's re module and in almost every other regex implementation.

Pattern	Meaning	Example
`abc`	literal characters: matches exactly "abc"	`cat` matches "cat" in "concatenate"
`.`	any single character (except a newline)	`c.t` matches "cat," "cot," "c_t"
`[abc]`	a character class: one character out of the set	`[aeiou]` matches any single vowel
`[a-z]`, `[0-9]`	a range inside a character class	`[A-Za-z]` matches any English letter
`[^abc]`	negation: any character not in the set	`[^0-9]` matches anything that is not a digit
`\d`, `\w`, `\s`	shorthand classes: digit, "word" character (letter/digit/underscore), whitespace	`\d{4}` matches a 4-digit year
`?`	the preceding item is optional (0 or 1 time)	`colou?r` matches "color" and "colour"
`*`	0 or more of the preceding item	`ab*c` matches "ac," "abc," "abbbc"
`+`	1 or more of the preceding item	`\d+` matches one or more digits
`{n}`, `{n,}`, `{n,m}`	exactly n / at least n / between n and m repetitions	`\d{3}-\d{4}` matches a phone-style pattern like "555-1234"
`^`, `$`	anchors: start / end of the line (or string)	`^Chapter` matches only if the line begins with "Chapter"
`a\|b`	alternation: matches "a" or "b"	`cat\|dog` matches "cat" or "dog"
`( )`	a group: bundles a sub-pattern, and can be captured for extraction	`(\d{4})-(\d{2})-(\d{2})` captures year, month, day separately

Worked example: to pull dates written as "1898-03-12" out of a messy text file, the pattern \d{4}-\d{2}-\d{2} says "four digits, a dash, two digits, a dash, two digits." It will match that shape wherever it occurs, without you having to know the date in advance. Wrapping each part in parentheses, (\d{4})-(\d{2})-(\d{2}), additionally lets your code pull out the year, month, and day as separate values.

Practice the basics interactively at regexone.com before moving to the Chinese-specific material below.

3. The Chinese addition: why `\w` and `[a-z]` are not enough

The shorthand classes above (\w, [a-z], etc.) are built for the Latin alphabet and digits, so they simply do not match Chinese characters. Working with Chinese-language sources requires a few extra tools, covered in the interactive page Regex ABCs for Chinese (eleven core lessons plus four extras on real-world pitfalls, using examples from 《紅樓夢》 and 《明史》).

Pattern / issue	What it addresses
`[\u4e00-\u9fff]`	Matches one common Chinese character: the main CJK Unified Ideographs Unicode block.
`[\u3400-\u4dbf]`	CJK Extension A: rarer characters that fall outside the main block above (needed for some names and archaic characters).
`\p{Script=Han}`	Matches any Han character across all Unicode blocks and extensions in one shot, so you don't have to remember individual ranges. In Python, use the third-party `regex` module (not the built-in `re`); in JavaScript, add the `u` flag.
Full-width vs. half-width punctuation	Chinese text typically uses full-width punctuation (`，。！？`), which looks similar to but is a different character from ASCII punctuation (`,.!?`). A pattern written only for ASCII punctuation will silently fail to match Chinese-style punctuation.
`\u3000`	The ideographic (full-width) space, a different character from the ordinary ASCII space (`\u0020`). OCR'd or copy-pasted Chinese text often mixes the two, which breaks patterns that only look for a normal space.
Simplified vs. traditional characters	The same word can be two different character strings, e.g. 討論 (traditional) vs. 讨论 (simplified). A pattern or character class built around one will not match the other. Either normalize the text first (e.g. with `opencc`, as in the collocation exercises) or explicitly include both variants in your class/alternation.

Putting several of these together, a pattern for classical-novel chapter headings such as "第一回" or "第一一八回" might look like ^第[一二三四五六七八九十百千0-9]+回: an anchor (^), a literal character (第), a character class of Chinese numerals mixed with plain digits, a quantifier (+), and another literal (回), combining the RegexOne basics with the Chinese-specific character classes above.

Work through the interactive lessons at Regex ABCs for Chinese; each step gives you a real sentence from a classical or modern Chinese text and asks you to write the pattern that isolates a specific piece of it.

4. In-class / take-home exercise

Paste the following into OpenCode, adjusting the file name and the pattern you need for your own source.

Prompt

I have a historical text file, source.txt, that needs cleaning before analysis.

Write a Python script (clean.py) that:
- loads source.txt
- uses a regular expression to find and report all footnote markers of the form [1], [2], [23], etc. (a number in square brackets)
- uses a regular expression to find and report all four-digit years (e.g. 1898)
- removes the footnote markers from the text and saves the cleaned result as source_clean.txt
- prints how many footnote markers and how many years were found

Explain the regex patterns you used, in plain language, in a short comment above each one.

Check the printed counts and the cleaned file, then try adapting the same script to a pattern relevant to your own source (a chapter heading, a date format, a name list, an OCR artifact). This is exactly the kind of "transformation log" step described in this week's readings.

Readings