Under the Hood: How AI Assistants Really Work
Tokens, system prompts, memory, MCP and skills, explained live instead of with slides
Everyone uses AI assistants now, but very few people know what actually happens between typing a question and getting an answer. Is the AI looking things up? Does it remember you? How does it “know” anything about your company?
I wanted to explain this without forty PowerPoint slides, so I built a small tool that shows an AI assistant being put together one layer at a time. In the video I start with a raw language model and keep adding layers until it’s a real assistant working on real IT data. Every step shows exactly what the AI receives, what it costs and how the answer changes.
It starts with tokens
An AI doesn’t read words. It reads tokens: small numbered pieces of text. “Strawberry” isn’t ten letters to a model, which is why it famously struggles to count the r’s. All a language model does is predict the next token, one piece at a time. Temperature only decides how often it picks something other than its top choice.
The raw model is surprisingly dumb
Without anything around it, a model doesn’t know today’s date, doesn’t know who you are and can’t access a single system. It doesn’t even remember your previous message: chat apps fake memory by resending the entire conversation on every turn. Seeing that live is usually the first “aha” moment.
Then we add the layers
System prompt. Every AI product has hidden instructions. Add one sentence with today’s date and the date question is solved. In the video I edit the prompt live and you see the behaviour change instantly.
Memory. When an assistant “remembers” you, it’s usually a set of short notes, extracted from earlier chats, pasted into the prompt. Useful, but it’s just more text: it costs tokens, it can be wrong, and you should be able to see and delete it.
Tools via MCP. The AI can’t actually do anything. It can only ask. With the Model Context Protocol (MCP), the app runs the tool and hands the result back as text. I demo this with the ControlUp MCP server on real endpoint data. One question can easily become three or four API calls behind the scenes.
Skills. This is the layer I find most interesting. The model queried real data correctly and still drew the wrong conclusion, because it didn’t know how that data is stored: values at a different scale, and special “no data” markers that skew an average. A skill file packs that expertise into a few paragraphs that load only when needed. Same model, same tools, same question, and suddenly the answer is right.
What I hope you take away
The model is only one part of an AI assistant. What makes it useful is everything around it: the instructions, the context, the tools and the domain knowledge. That’s also where companies differentiate, and where you should focus if you’re building your own assistant or chatbot. I also cover running multiple models and what to think about around privacy before you connect AI to your own data.
Watch the video above. And if you’re building something yourself, I’d love to hear what you’re working on.
Disclosure: I work at ControlUp, and the MCP demo in this video uses ControlUp’s MCP server. Opinions are my own.