Last updated: Sep 14, 2026

DHG 502 syllabus

Week 2 — Language, Text, and Encoding

1. Interactive visualization and video

• Character Encoding Explorer: type any text and watch how it becomes a sequence of bytes under UTF-8, UTF-16, GB18030, Big5, and other encodings — and how it is decoded again.
• Byte-Pair Encoding (BPE) training: watch BPE start from characters or bytes, merge frequent pairs, and grow a subword vocabulary.
• 你懂乱码吗？锟斤拷烫烫烫（详解ASCII、Unicode、UTF-32、UTF-8编码）| Mojibake? (林粒粒呀, YouTube; in Chinese): starts from garbled text and walks through ASCII, Unicode, UTF-32, and UTF-8.

2. What is a “text”?

To you, a text is words, sentences, a story. To a computer, a text is only a sequence of numbers. The machine does not “see” Chinese or English — it only distinguishes two states: on and off, written as 0 and 1. Every character, punctuation mark, and emoji must first become a number before it can be stored or sent. When it is read back, the same rule must be used; if the rule is wrong, you see garbled characters (mojibake). That rule is called an encoding. The sections below start from the smallest unit.

3. The bit: the smallest switch

A bit is a switch that can only be 0 or 1. The more bits you have, the more things you can distinguish — each extra bit doubles the number of choices.

The pattern is simple: n bits → 2ⁿ possibilities. Eight bits make one byte, with 256 possible values (0–255). What files store on disk and what travels over the network are these bytes.

A long string of 0s and 1s is hard to read, so the same numbers are often written in hexadecimal (hex): binary uses only 0 and 1 in each place; hex uses 0–9 and A–F, sixteen symbols. Two notations, one number.

4. ASCII: enough for English, at first

Early computers mainly had to handle English. Engineers used 7 bits (128 slots) for a small table called ASCII (American Standard Code for Information Interchange): letters, digits, and common punctuation, each character mapped to a number. The American Standards Association published the first version in 1963; a 1967 revision added lowercase letters and is essentially the 128-character table still used today. The eighth bit is filled with 0, so each ASCII character occupies exactly 1 byte. The three columns below are three writings of the same number.

Those 128 slots filled up quickly. Chinese, Japanese, and emoji are not in the table — ASCII was never meant to hold them. Later, different regions invented their own encodings: mainland China issued GB 2312 in 1980, Taiwan’s Institute for Information Industry defined Big5 in 1984, and GBK (1995) extended the mainland set. Open the same file with the wrong rule and it becomes garbled text.

5. Unicode: a world catalogue of writing

Unicode’s approach is to keep one large table covering almost every writing system, and to give each character a unique number. The idea was sketched in 1987 by engineers at Xerox and Apple (Joe Becker coined the name); the Unicode Consortium was founded in 1991 and published Unicode 1.0 the same year. That number is a code point. It only answers “which number is this character?”; it does not say how that number should be stored in a file.

Code points are conventionally written as U+ followed by hex:

• U+ is a label meaning “this is a Unicode number.”
• The symbols after it are hex — another way of writing the same number as binary.

For example, “A” is decimal 65, hex 41, so it is written U+0041 (the two extra zeros only pad the width). “中” is decimal 20013, hex 4E2D, so it is U+4E2D. Read U+4E2D as: in the Unicode table, the number for “中” is 4E2D.

6. UTF-8: storing the number as bytes

UTF-8 does the next step: it writes the code point as bytes so it can be saved to a file or sent over a network. Ken Thompson and Rob Pike designed it in September 1992 for Bell Labs’ Plan 9 operating system, so that old ASCII files would still be valid; it later became the default encoding of the web. It is variable-length — small numbers take fewer bytes, large numbers take more. English letters still occupy 1 byte (identical to ASCII); Chinese characters usually take 3; a few rare characters and many emoji take 4. English files therefore stay compact, while the same encoding can hold writing from anywhere.

The same character becomes different bytes under different encodings. Use the Character Encoding Explorer above to see those differences.

How the computer knows “how many bytes this character uses”

UTF-8 marks the start of each byte:

• This character uses 1 byte: the byte starts with 0
• 2 bytes: the first starts with 110, the following one with 10
• 3 bytes: the first starts with 1110, the following two with 10
• 4 bytes: the first starts with 11110, the following three with 10

These marks are the leading byte and the continuation bytes. The remaining x positions are filled with the binary digits of the code point.

The “code-point range” column is also U+ plus hex: U+0000–U+007F means numbers 00 through 7F, which is all of ASCII.

Walk through “中”. The same number is written first in decimal, then in hex, then split into 3 bytes (8 bits each):

1. Decimal: 20013
2. hex: 4E2D (hence U+4E2D)
3. 3 bytes: UTF-8 stores this number as 8 + 8 + 8 bits. The 3-byte template is 1110xxxx 10xxxxxx 10xxxxxx; fill in the binary of 4E2D (0100 1110 0010 1101) at the x positions, and you get:

Bold marks are the leading/continuation prefixes; orange bits are filled in from the code point. What the file actually stores is E4 B8 AD.

Two immediate consequences. First, UTF-8 contains all of ASCII: hex 00 through 7F are written identically, so an old English file is already valid UTF-8. Second, the start of each byte says “I begin a character” or “I continue one,” so even if you start reading in the middle of a file, you can still find the boundaries between characters.

7. Chinese text segmentation

English marks word boundaries with spaces, so a computer can split text easily. Written Chinese does not put spaces between words: 「我喜歡吃火鍋」 has to be cut into 「我／喜歡／吃／火鍋」 before word-level analysis makes sense. Segmentation is that process of cutting a string of characters into words. There are several tools (spaCy, HanLP, THUNLP); this course uses jieba, which decides cut points from a dictionary and statistical probabilities. Segmentation results feed directly into every later analysis — word frequencies, collocations, topic models — so it is the first step in processing Chinese text.

8. Bag of Words (BoW)

Treat a text as a “bag”: count how many times each word appears, and ignore order and syntax entirely. 「貓追狗」 and 「狗追貓」 are the same bag: {貓:1, 狗:1, 追:1}. The simplification is crude, but it is already enough for many tasks (comparing topics across documents, or differences in an author’s vocabulary), and it is the basis of vector-space models and topic models.

9. Stopwords

Stopwords are words that appear extremely often but do little to distinguish one text from another: in Chinese, 「的、了、是、在、我、你」; in English, “the, is, of, a.” Before frequency or topic analysis, they are often removed so they do not drown out words that carry more information. There is no universal stopword list; it should follow the corpus and the research question. If you are studying how often 「我」 appears, 「我」 cannot be treated as a stopword and deleted.

10. Zipf’s law

Zipf’s law: in a text that is long enough, rank all words by frequency from high to low. The frequency of the word in rank n is roughly inversely proportional to n — the most frequent word appears about twice as often as the second, three times as often as the third, and so on. A handful of words account for most occurrences, and a long “tail” contains many rare words that appear only once or twice. That is why stopwords dominate a frequency table, and why textual data are so uneven.

11. In-class exercise

From qhchina-data/corpora, pick a novel’s .txt file. Replace [link] in the prompt below with the file URL, then paste it into OpenCode:

Please download this novel as a .txt file [link]

Then write a .py script that:
- uses jieba to segment the text into words
- prints the 10 most frequent words
- saves a simple png bar chart of the 100 most frequent words (columns only, no labels or annotations)

jieba and matplotlib are already installed; do not create a virtual environment.

When it finishes, look at the top 10 words printed in the terminal, then open the .png. The tallest bars on the left are usually stopwords; the rapid drop from left to right is Zipf’s law.

Commit and push when you are done, so the work is preserved — anything not pushed lives only in your Codespace (or on your laptop) and is lost if that environment is deleted.