• Dr Cog@lemmy.ml
    link
    fedilink
    arrow-up
    1
    ·
    1 year ago

    Tokens are usually never multiple words. Think of them like “information units”. If you have a plural word “hats”, there are two tokens: the “hat” one and the "s’ that adds more information about it being plural. Combinations of words only really occur for proper names.