What exactly is Generative AI doing?
Generative AI is a broad term for technologies that can generate new content across text, images, audio, code, and more in a plausible way. In particular, the central role in text generation is LLM (Large Language Model). Services like ChatGPT produce text by using an LLM to predict the next expected word (more precisely, the next token).
The key is not that it understands meaning, but that it statistically predicts the most natural continuation with high accuracy. However, as scale increases, more abstract patterns are learned, so the result tends to behave as if it understands.
LLM basics: Two phases of learning and inference
1) Training: Learning patterns from large amounts of text
LLMs learn from vast data including internet documents, books, papers, and code to make sentences likely to continue. Typically, they repeatedly perform the next token prediction and adjust model weights to reduce mistakes.
Rough example: The weather tomorrow is ... Next would likely be sunny, rain, or cloudy. They learn probabilities from context, frequency, and surrounding relationships.
2) Inference: Generating the next token for a given input
When a user provides a prompt, the model internally constructs a probability distribution and selects the next token. This process repeats to extend the text. Common generation parameters include temperature and top-p.
- temperature: lower values yield more deterministic outputs (safer); higher values yield more diverse outputs (more varied)
- top-p (nucleus sampling): choose from the smallest set of tokens whose cumulative probability reaches p
What is a token? The finer unit of text than a word
The smallest unit typically handled by LLMs is the token. Tokens are not necessarily whole words; they can be parts of words or symbols, and in Japanese, sequences of characters or subwords. LLMs first break sentences into a token sequence and base their predictions on that sequence.
Why tokens matter
- Cost: API pricing is usually token-based (input + output)
- Length limit (context length): There is a maximum number of tokens that can be processed at once
- Prompt design: The same content can yield different results and costs depending on token efficiency
Practical guidelines in real-world usage
Token counts vary by language and text, but for English a rough rule is around 1 token ≈ 4 characters. In Japanese, segmentation tends to be finer, so the same amount of information can lead to more tokens. It is generally more stable to summarize, organize, then input rather than sending long texts as-is.
What is Transformer? The crucial engine behind LLMs
Many of today’s LLMs are based on a neural network structure called Transformer. The strength of the Transformer lies in its Attention mechanism, which determines where in the text to focus.
Attention: Going to the important parts of the context
Humans do not read every part with the same focus; we often strongly reference the subject, the object, or the preceding conditional. Attention does this mathematically, computing how much each token should refer to other tokens (weights).
Example: In the sentence Tarou gave Hanako a book. He ... attention to the preceding context matters.
Why is Transformer strong? A quick comparison with RNNs
Historically, sequential models like RNNs/LSTMs were mainstream, but as texts grew long, information could fade and parallelization was challenging. The Transformer can look at the entire sequence at once to compute attention, and it is easy to parallelize on GPUs, making it suitable for large-scale training. This ability to scale up directly contributed to performance improvements.
How LLMs compose text (a super simple version)
- Tokenize the input
- Convert each token to a vector (embedding)
- Compute a context-aware representation with the Transformer (Attention shines)
- Obtain the probability of the next token
- Sampling (temperature/top-p, etc.) to choose one
- Append the chosen token to the end and repeat until the required length
Supplementary knowledge to avoid the feeling of understanding: Context length and hallucinations
Context length: AI's working desk size
LLMs have a limit on how many tokens they can reference at once (context length). If you exceed it, old information will not be included and you must summarize first. In practice, it is standard to split long documents by paragraphs or to extract key points first before asking questions.
Hallucination: Why plausible false content can appear
LLMs mainly produce the most natural continuation, so even if there is no supporting evidence in the input, they may fill in the text. This is hallucination. Effective countermeasures include:
- Have the model quote supporting evidence from the text (provide quoted sections)
- State uncertainty when uncertain
- Use RAG (retrieve from search or internal documents before answering)
- Verification (primary sources, calculations, real-world checks)
Practical tips to make prompts effective
1) Start with the role and objective
Setting roles like You are an editor, or You are an SRE, helps stabilize the style of responses.
2) Be specific about constraints
Clarify length, target audience, prohibitions, output format (bullets/table/JSON), etc. This helps align token usage and reduces back-and-forth.
3) Include one example (few-shot)
Providing a single example like a sample input and expected output helps the model grasp the pattern. Transformers are strong with contextual patterns, so exemplification is an effective technique.
Trends ahead: LLMs from intelligent text generation to executing AI
Recently, not only text generation by a standalone LLM but also expansion to tools such as tool use or function calling that call external tools, agents who plan across multiple steps, and RAG for internal data references, i.e., generation to action. The root mechanism remains next-token prediction plus Transformer, but practical usability is greatly affected by surrounding system design.
Summary: Three things to remember
- LLM creates text by continually predicting the next token
- Transformer uses Attention to reference important parts of the context and scale
- Tokens are the on-site units tied directly to cost, context length, and prompt design
Once you grasp the gist of the mechanism, Generative AI shifts from magic to a tool. Start with token and context length intuition, and master it through short instructions and iterative refinement.
