How automatic text highlighting actually works
There are two completely different ways to shorten a piece of text, and almost every argument about “AI summarisers” comes from mixing them up.
An abstractive system reads your text and writes new sentences about it. That is what a large language model does. An extractive system never writes anything — it scores the sentences you already have and picks the best ones. Automatic highlighting is extractive by definition, because a highlighter that invented new words would have nothing to draw on top of.
Extractive ranking is a solved problem from the early 2000s, it runs in milliseconds on a phone, and it needs no server. Here is how it works, step by step.
Step 1: Splitting the text into sentences
Everything downstream depends on getting this right, and it is much harder than splitting on full stops. “Dr. Chen earned $1.4m in Q3 vs. 2024.” is one sentence containing four full stops, none of which end it.
A workable segmenter treats a full stop as a boundary only when the surrounding context agrees: the next character is whitespace, the next word starts with a capital, and the token before the stop is not a known abbreviation, initial, decimal number or ordinal. Quotation marks and closing brackets have to travel with the sentence they belong to.
Get this wrong and the damage is invisible but severe. A sentence split in half scores as two weak fragments instead of one strong sentence, so the genuinely important line never surfaces.
Step 2: Working out which words carry meaning
Not all words are evidence. In a chapter about photosynthesis, the word “the” tells you nothing and the word “photosynthesis” tells you almost nothing either — it appears everywhere, so it cannot distinguish one sentence from another. The useful words are the ones that are frequent here but rare in general.
That is TF-IDF: term frequency multiplied by inverse document frequency. Term frequency asks how often a word appears in this text. Inverse document frequency discounts words that appear everywhere. Multiply them and you get a score per word that peaks for terms like “chlorophyll” or “stomata” and bottoms out for “the”, “of” and “is”.
A stopword list removes the obvious filler before scoring even begins. Stemming or lemmatising helps too, so that “absorbs”, “absorbed” and “absorption” are counted as evidence for the same concept rather than three unrelated words.
Step 3: Ranking sentences against each other
Now the interesting part. You could just add up the TF-IDF scores in each sentence and take the highest totals, but that rewards long sentences and misses the structure of the argument.
TextRank does something smarter. Build a graph where every sentence is a node, and connect any two sentences with an edge weighted by how much vocabulary they share. Then run PageRank over it — the same algorithm that ranked the early web.
The insight transfers neatly. PageRank calls a web page important when important pages link to it. TextRank calls a sentence important when it overlaps heavily with other sentences that are themselves heavily overlapped. A thesis statement wins because the rest of the text keeps circling back to its vocabulary. A passing aside loses because nothing else in the text refers to it. Crucially this is relative: the same sentence can rank highly in one document and poorly in another, which is exactly what you want.
Step 4: Spotting facts and definitions
Graph ranking finds central sentences, but centrality is not the only reason a line matters. A single statistic can be the most useful sentence in a chapter while being barely connected to anything else.
So a second pass looks for surface patterns that reliably signal a particular kind of content:
Facts and figures
Digits, percentages, currency symbols, units, dates and number words sitting next to a comparative like “rose”, “fell”, “twice” or “compared with”.
Definitions
Constructions such as “X is defined as”, “X refers to”, “X, also known as”, “the term X means”, and a capitalised term immediately followed by a parenthetical gloss.
Conclusions
Discourse markers that authors use to flag their own summaries: “therefore”, “in conclusion”, “the key point is”, “this suggests that”.
Step 5: Throwing away near-duplicates
Without this step the output is embarrassing. Textbooks state the same idea in the introduction, the body and the chapter summary, so the three highest-scoring sentences are frequently three paraphrases of one point — and the second-most-important idea never gets marked at all.
The fix is to select greedily with a penalty. Take the top-scoring sentence, then before accepting the next one, measure its vocabulary overlap against everything already selected and subtract a penalty proportional to that overlap. A sentence that largely repeats an accepted one drops down the list and something new takes its place. This is the same trade-off as Maximal Marginal Relevance: you are balancing relevance against novelty rather than optimising relevance alone.
What this approach genuinely cannot do
Being honest about the ceiling matters, because it tells you when to trust the output.
A statistical ranker measures structure and vocabulary. It does not comprehend meaning. It cannot detect irony, it cannot tell that a sentence is the author describing a view they are about to demolish, and it cannot know that one throwaway clause happens to be the single thing your exam covers.
It is also only as good as its input. Prose where sentences share vocabulary — textbook chapters, reports, news articles, lecture notes — works well. Poetry, dialogue, song lyrics and bullet lists of single words work badly, because the graph has almost no edges to rank.
Treat extractive highlighting as a fast, free first pass that puts your attention in roughly the right place. The judgement call about what actually matters stays with you, which is the honest division of labour.