MonGauss: What Are Tokens

Source: MonGauss

https://discord.com/channels/1210695141140996147/1242236099045363713

Permission from MonGauss to use for my patreon as a free article

Very important, is what they are!

LLMs don't process language the way we do in words. They process text as tokens, pre-programmed strings of characters that the AI splits words into. On average, a token will contain 3-4 characters of text, and when using common english words you can expect 100 tokens to correspond to roughly 75 words. You can use this tool https://platform.openai.com/tokenizer to find out how many tokens any given set of text is. LLMs have limited memory called the context window. This is the maximum number of tokens that the LLM can keep in its working memory at any given time. The character definition is a permanent set of information that the bot is constantly aware of, and therefore takes up a certain amount of this context window. Free users of CrushOn get 1750 tokens to work with. Premium users get 2750. Deluxe users get 3500. In order to optimize characters, you should try to keep the token count of their Personality + Greeting to under 800 tokens because every token that's used in the definition cannot be used on the conversation's memory.

How many messages do bots actually remember?

Generally, very few. I can guarantee that most of you heavily overestimate how much the AI remembers at any given moment because they are very good at either extrapolating specific info from context or avoiding situations where it comes up at all, but the AI actually forgets things quite fast. For a free user (most users) you have 1750 tokens of memory at base and a 175 token per message limit. That means if you had an empty character and every exchange with the AI was only 1 token, it could remember 1750 messages, but that's not realistic. In a real chat with a character on the low side of, say 400 tokens, you have 1350 left. If your message + the AI's response averages out to around 100 tokens (again, very on the low side), it still only remembers about 13 messages back + forth before forgetting. This is in a heavily idealized scenario.

Why do I need to know how forgetful the AI is?

It's important to understand this so that you don't go into character RPs with unrealistic expectations. Understanding how memory works and having realistic expectations for the AI's capabilities can really cut down on potential disappointment and frustration. It can also prepare you to work around these limitations. If you're planning to do a long-form RP, go in knowing that the AI will forget things and you can come up with ways to subtly remind it of things it should know through narrative tricks or just outright tell it in an OOC note. It also helps me not have to cringe when I see people asking why the AI can't remember something from 40 messages ago.