Token Usage

When you make Gainly API requests, you'll notice that Tokens are consumed and reported under the token_usage parameter in the API responses.

"token_usage": {
    "semantic_tokens": 8,
    "llm_tokens": {
        "llm_output_tokens": 1386,
        "llm_input_tokens": 79,
        "model": "model_2"
    }
}

For endpoints that do not consume tokens, token_usage parameter may still be present in the API response with values of 0.

"token_usage": {
    "semantic_tokens": 0,
    "llm_tokens": {
        "llm_output_tokens": 0,
        "llm_input_tokens": 0,
        "model": null
    }
}

What Are Tokens?¶

Under the hood, Gainly uses various foundation models (FMs) and large language models (LLMs) to implement its AI features.

Basic Unit of Text¶

Tokens are the basic units of text that the model processes. Tokens can be as short as one character or as long as one word or a few words, depending on the language and other factors.

Before the model processes (reads or writes) text, it first tokenizes that text, meaning it breaks down the text into tokens. This is a crucial step as the model doesn't directly understand words or sentences; it processes tokens.

Number of Tokens¶

Generally speaking, in English text, common words and punctuation might each be a single token, while longer or uncommon words might be split into multiple tokens.

For non-Latin script languages, tokenization often occurs at the character level, with each character representing a single token.

Furthermore, the token count for the same text can, and often does, vary widely between models.

Types of Tokens in Gainly¶

Gainly uses the following types of tokens as a part of its AI features:

Semantic Tokens
LLM Tokens:
- LLM Input Tokens
- LLM Output Tokens

Semantic Tokens¶

Semantic tokens are consumed whenever Gainly extracts semantic meaning of text in your API request - such as when you add/update a document, perform an AI search, etc.

The following operations in Gainly consume semantic tokens:

Efficient use of semantic meanings

Operations (such as Find Similar) that merely use previously extracted semantic meanings do not consume any semantic tokens.

LLM Tokens¶

LLM tokens are consumed whenever an LLM is used in response to your API request - such as when you generate an answer using Gainly RAG. There are two types of LLM tokens:

LLM input tokens: The LLM uses these tokens to read the retrieved information.
LLM output tokens: The LLM uses these tokens to generate (write) the answer.

The following operations in Gainly consume LLM tokens:

Token Pricing¶

Visit your Gainly Dashboard for the current pricing of these tokens (listed under Billing).

Token Usage Reporting¶

Whenever you perform an operation in Gainly that uses tokens, it will be reported under the token_usage parameter in the API response.

Visit your Gainly Dashboard for details on your total token usage during each billing period. See API request logs for token usage by each API request.