Beyond the Question Bank: Building a Learning Engine

Most exam prep tools are essentially question banks attached to a progress bar. When I started building Cert Buddy™, I made a deliberate choice: I wasn't going to build another quiz app or an exam braindump designed simply to help people cram and pass. Putting questions on a screen, scoring answers, and showing a percentage is relatively straightforward. I was more interested in whether an exam prep product could develop an increasingly useful understanding of the learner and use that understanding to guide what happens next.

That led me to a different set of questions. What can we learn from the topics someone repeatedly struggles with or avoids? What does it mean when they can recognize an answer in a multiple-choice question but struggle to retrieve the same concept from memory? How should recent performance compare with evidence collected several weeks ago? And how much confidence should we place in an 80 percent score based on five questions compared with the same score based on fifty?

Those questions became the foundation for Cert Buddy.

When I use the term learning engine, I mean something fairly specific: a system in which learner interactions continuously contribute evidence to a shared learning profile, and that profile influences what the system measures, recommends, and presents next. A flashcard response, missed question, practice session, or diagnostic result should not disappear when the activity ends. Each interaction should tell the system something about what the learner appears to know, where uncertainty remains, how that understanding may be changing over time, and what type of practice could be most useful next.

That shift in thinking changed how I approached the entire product. Practice questions, flashcards, diagnostics, simulations, review activities, and coaching stopped being isolated features. They became different ways of collecting evidence and responding to it.

Moving Beyond Simple Averages

At the heart of Cert Buddy is a readiness model that evaluates performance across the Microsoft AI-901 exam objectives. Rather than treating readiness as a simple percentage of correct answers, the model considers both demonstrated performance and the strength of the evidence supporting it.

Two variables are particularly important: recency and evidence volume.

An 80 percent accuracy rate after five questions provides some information, but it should not create the same level of confidence as an 80 percent accuracy rate after fifty. Similarly, strong performance demonstrated several weeks ago should not necessarily carry the same weight as knowledge demonstrated recently. Learning changes, memory fades, and additional evidence can either reinforce or challenge what the system previously believed about a learner.

The goal isn't to claim that Cert Buddy can predict with certainty whether someone will pass an exam. It cannot, at least not today. The readiness score is a model based on the evidence the learner has generated. As that evidence grows, the system can become more confident in some areas while becoming less certain in others that haven't been revisited.

One of the interesting observations from multiple rounds of usability testing through Userbrain has been how naturally learners understand the purpose of the readiness concept. Participants frequently interpreted it less as another score to maximize and more as an answer to the question they actually cared about: Am I getting closer to being ready?

That's an important distinction, but it's also something I want to validate more rigorously as the learner population grows. Over time, one of the questions I hope to answer is how closely the model's estimate of readiness corresponds with actual learner outcomes.

Study Activities as Different Forms of Evidence

Personalization also creates an immediate cold-start problem. When someone first joins Cert Buddy, the system knows almost nothing about them. The diagnostic experience creates an initial baseline, but that baseline is only the beginning.

From there, different study activities contribute different kinds of evidence.

Flashcards focus on active retrieval and memory strength through a spaced repetition approach. They can reveal something that multiple-choice questions often cannot: whether the learner can actually bring a concept to mind without being shown several possible answers.

Practice questions measure applied conceptual knowledge and can be targeted toward specific skill areas. The system can also prioritize unseen or less frequently encountered questions, reducing the likelihood that improving scores are simply the result of memorizing familiar items.

Exam simulations provide a different signal by asking learners to work across topics for longer periods under more realistic constraints. That introduces elements such as sustained attention, topic switching, and decision-making under time pressure.

Mistake review treats an incorrect answer as the beginning of a learning loop rather than the end of one. Missed questions remain part of the learner's profile until subsequent activity provides stronger evidence that the underlying concept has been understood. The Mistakes Notebook makes that history visible rather than allowing mistakes to disappear into a session score.

None of these activities is especially novel on its own. The more interesting design challenge is making them contribute to the same evolving picture of the learner.

AI Is Not the Learning Engine

One distinction I've become increasingly deliberate about is separating the learning model from the AI that sits on top of it. The large language model doesn't decide whether someone has mastered a skill. It doesn't calculate readiness, determine how much evidence exists, or decide which learner activity should count toward demonstrated knowledge. Those decisions belong to the learning system itself.

What AI can do well is interpret structured learner context and make it more useful to the person studying.

When Cert Buddy generates coaching, session summaries, or explanations, the model can receive information about the learner's current skill profile, recent performance, previous activity, unresolved mistakes, trends over time, and areas where the system has limited evidence. Instead of asking an LLM to produce generic study advice, the system gives it context about the learner it's actually trying to help.

Conceptually, the flow looks more like this:

Learner Activity
      ↓
Structured Learning Profile
      ↓
Readiness, Trends, and Evidence
      ↓
AI Interpretation and Coaching
      ↓
Learner Action

That separation matters to me.

I don't want an LLM inventing a learner's level of mastery. I want the learning system to establish the evidence, and I want AI to help the learner understand that evidence and act on it.

As the product evolves, AI may take on more sophisticated roles, particularly in conversational learning, explanation, reflection, and transfer activities. But the foundation still needs to be a credible model of what the learner has actually demonstrated.

Connecting Learning Theory to Product Design

I've also recently started graduate study in Learning Design & Technology, which has made this project even more interesting for me personally.

I've spent much of my career working at the intersection of technology, content, learning, and developer experiences. Cert Buddy now gives me a place to explore those ideas at a much more granular level. Concepts such as retrieval practice, spaced repetition, cognitive load, desirable difficulty, feedback loops, and confidence in assessment can move from research papers and course discussions into product decisions that real learners interact with.

That doesn't mean simply taking a learning theory and turning it into a feature. In practice, the questions are more difficult.

How much retrieval difficulty is productive before it becomes frustrating? When should a learner revisit something they previously knew? How should the system distinguish between weak knowledge and insufficient evidence? When should it recommend remediation, and when should it encourage the learner to move on? How much choice should the learner have compared with what the system recommends?

Those are design questions as much as they are technical ones.

Cert Buddy gives me an environment where I can build a hypothesis, put it in front of learners, observe how they respond, and refine it. My graduate studies give me a stronger theoretical foundation for asking those questions. The product gives me a place to test what those ideas actually look like when translated into a working system.

Separating Knowledge from Motivation

Another important design decision was recognizing that engagement and mastery are not the same thing.

Self-study requires persistence. Streaks, achievements, points, milestones, and the social 🔥Hall of Flame can all help make progress visible and give learners reasons to return. But those mechanisms should not be allowed to distort the system's assessment of what someone knows.

That's why Cert Buddy deliberately separates gamification from readiness. Studying every day can extend a streak. Completing activities can earn points. Reaching milestones can unlock achievements. None of those things, by themselves, increase readiness.

Readiness should move because the learner demonstrated knowledge. That constraint seems simple, but I think it's important. Engagement mechanics are there to encourage consistent learning behavior. The readiness model exists to interpret evidence of learning. Combining the two would make both less meaningful.

Building Faster with AI-Assisted Engineering

There is another form of AI involved in Cert Buddy that has nothing to do with learner coaching: the development process itself.

Building a product like this independently means moving constantly between product management, UX research, instructional design, database architecture, analytics, and software engineering. Modern AI-assisted development tools have dramatically compressed the distance between identifying a problem and testing a possible solution.

That has been especially valuable because the product is evolving through rapid experimentation.

When usability testing showed that a simple binary right-or-wrong flashcard interaction didn't capture the nuance I wanted, I could redesign the experience around levels of recall and quickly test the implications throughout the system. When the recommendation logic changes, I can modify the supporting data structures, application logic, and user experience together rather than treating each as a separate multi-week project.

The biggest benefit isn't that AI writes code faster. It's that shorter development loops allow me to spend more time thinking about what the system should do.

That makes it possible to move more quickly between an instructional hypothesis, a product implementation, learner feedback, and another iteration.

The Feedback Loop Is the Product

The part of Cert Buddy I find most interesting isn't any individual feature. It's the loop connecting them.

Learner Activity
      ↓
New Evidence
      ↓
Updated Learning Profile
      ↓
Targeted Recommendation
      ↓
Learner Action
      ↓
New Evidence

Every cycle gives the system another opportunity to refine what it believes about the learner and another opportunity to give the learner a more useful next step.

Ultimately, I want Cert Buddy to answer two questions every time someone returns:

1. How ready am I?

2. What should I do next?

I'm still early in answering those questions well.

The readiness model needs more validation. The recommendation system will continue evolving as more learners use it. Some ideas will work, others will not, and the product will change as I learn more about both the technology and the learning science behind it.

But that's also what makes the project interesting to me.

I didn't set out simply to build a better question bank. I wanted to explore what exam preparation might look like if the product remembered what you did, interpreted that activity as evidence, and continuously adapted what happened next.

That's the learning engine I'm building.

Cert Buddy is an AI-powered exam preparation platform currently focused on the Microsoft AI-901 certification. Learn more at certbuddy.app.