fross logofross blog

How to test if your AI study tool is guessing

By Mohammad Badran, founder of fross · Published November 20, 2026

Give the tool a few notes, then ask it something those notes never mention. If it answers anyway, it is guessing. A trustworthy study tool says the answer is not in your notes.

Why this matters

When you study from an AI tool, you are trusting every flashcard and quiz answer it writes. Most chatbots are built to always give an answer, so when your notes do not cover something, they fill the gap with general knowledge or with something that only sounds right. You then memorise it as if it came from your class.

The 2-minute test

  1. Use a short set of notes. A paragraph or two is enough, for example a few lines about photosynthesis.
  2. Ask something your notes never mention. For example, if your notes are about photosynthesis, ask "Who discovered photosynthesis?" when no line in your notes says so.
  3. Read the reply. A tool that works from your notes should say it is not in them. A tool that confidently names someone is using outside knowledge or inventing it.
  4. Ask something your notes do cover, and look for the source. Every answer should point to the line it came from, so you can check it in seconds.
  5. Check who checks the work. Ask whether the generated cards and quiz questions are verified against your notes, or whether you are trusting a single pass.

What a good result looks like

This is what fross does when you ask it something your notes do not cover:

That's not in your notes.
It didn't guess. That's the point.

And when you answer a quiz question, it shows the line from your notes the answer came from, for example "From your notes · line 1". A second AI also checks each generated item against your notes before you see it.

What this test does not prove

It checks one behaviour: whether the tool admits when your notes do not cover something. It does not prove a tool is always right. Your notes can contain mistakes too, and a tool that quotes them accurately will repeat them. Keep checking answers against your own notes and your course material, which is much faster when each answer shows its source line.

Try it yourself

fross has a no-signup demo on its home page: paste a few notes, make flashcards, then ask the Teacher something your notes do not cover. fross is free on iOS and Google Play.

Run the test on fross.ai

Frequently asked questions

How can I tell if an AI study tool is making things up?

Give it a short set of notes, then ask a question those notes never mention. A trustworthy tool says the answer is not in your notes. A tool that answers anyway is guessing.

Why do AI study tools make things up?

Most chatbots are built to always produce an answer. If the answer is not in your notes, they fill the gap from general knowledge or invent something that sounds right.

Does this test prove a study tool is always right?

No. It only checks one behaviour: whether the tool admits when your notes do not cover something. You should still check answers against your own notes, which is why a source line next to each answer helps.

Is fross free?

Yes. fross has a free forever plan and is available on iOS and Google Play.