Hacker News new | past | comments | ask | show | jobs | submit login

Apologies and it’s slightly lazy of me to ask, but I was under the impression that a Token was basically 4 bytes/characters of text. This seems to be implying that there’s some differentiation between a token and conjunctions/other sort of in between words?



That is correct.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: