How ChatGPT-style models work.
Tools like ChatGPT, Claude and Gemini seem almost magical. Underneath, the core idea is surprisingly simple: predicting the next word, extremely well. Today you'll understand how these large language models are built and why they behave the way they do.
The core trick: predict the next word
Complete this: "Chai with Osmania ___". You probably thought "biscuits". You predicted the next word from patterns you've seen.
A Large Language Model (LLM) does exactly this, but it learned from a huge amount of text, so it can predict good next words for almost any topic. Repeat "predict the next word" hundreds of times and you get a full answer.
Tokens: how models read
Models don't read whole words. They break text into tokens: small pieces like "Hyder", "abad", "ing". Roughly, 1 token ≈ ¾ of an English word. Tokens matter because:
- Models have a limit on how many tokens they can handle at once (the context window).
- AI services usually charge by the number of tokens.
- Languages like Urdu or Telugu often use more tokens per sentence than English.
How a model is built (simplified)
| Stage | What happens | Analogy |
|---|---|---|
| 1. Pre-training | The model reads a massive amount of text and learns to predict next tokens. It learns grammar, facts, styles and reasoning patterns | A student reading an entire library |
| 2. Fine-tuning | It's trained on examples of good conversations and helpful answers | A teacher showing how to answer questions properly |
| 3. Feedback training | Humans rate answers; the model learns to prefer helpful, honest and safe ones (often called RLHF) | Exam marks and corrections |
| 4. Using it | You type a prompt; it generates a response token by token | Asking the trained student a question |
A basic LLM answers from patterns learned during training. Some apps add web search or documents (you'll see "searched the web" or sources). Without that, the model may not know recent events.
Why might an AI not know about something that happened last week?
Models have a "knowledge cutoff". Tools with search or documents can add newer information.
The context window: short-term memory
Everything in your current chat (your messages, its replies, files you upload) must fit in the context window. That's why in very long chats, older details may get lost, and why giving the right information in your prompt matters so much (Lesson 5).
Temperature: creativity vs consistency
Many AI tools have a setting called temperature:
- Low temperature: more predictable, consistent answers (good for facts, code).
- High temperature: more varied, creative answers (good for brainstorming, stories).
What is an LLM fundamentally trained to do?
Next-token prediction at massive scale is the foundation; fine-tuning shapes it into a helpful assistant.
Your task (10 min)
Ask any AI assistant: "Explain how you generate answers, in simple words for a 15-year-old." Then ask it: "What's your knowledge cutoff, and can you search the web here?" Compare its answers with this lesson.
Lesson 2 complete!
You understand tokens, next-word prediction, training, fine-tuning and context windows. Next: where AI is used in real jobs.
Next: Where AI is used in real jobs →