Tokenise raw text into a lowercased word stream. Keeps intra-word apostrophes (don't, children's) but drops leading/trailing punctuation and pure whitespace. Accented characters are preserved (\p{L}).
Tokenise raw text into a lowercased word stream. Keeps intra-word apostrophes (don't, children's) but drops leading/trailing punctuation and pure whitespace. Accented characters are preserved (\p{L}).