Projects / BareBot: Evaluating an AI Agent Inside a Live LMS
BareBot: Evaluating an AI Agent Inside a Live LMS
54 participants ran a 14-task protocol against a domain AI agent embedded in a learning management system. Task success was high — incorrect information was the real story.
Lead: Mapleforge Labs research group
The problem
Universities are embedding AI agents into learning platforms faster than anyone is evaluating them. Vendors report accuracy in the abstract; instructors need to know what happens when real students ask real questions — including whether the agent protects private course data.
Approach
We built BareBot, an LMS-based domain AI agent, and evaluated it with a structured protocol: 54 participants completed 14 tasks (756 attempts), self-reporting success, source citation, and incorrect information per task, with pre/post trust surveys. Boundary probes tested whether the agent would leak class lists or assignment content.
Results
Task success was 86.2%, yet 19.4% of attempts contained incorrect information — and 19.3% of 'successful' attempts did too. Trust rose significantly after use (3.41 → 3.80, p < .001) even as verifiability scored lowest, which is exactly the calibration problem instructors must teach around. Two boundary probes exposed real privacy failure modes that any LMS agent deployment must close.