AI mentorship and human oversight
AI Mentorship at Kabakoo: What the Evidence Shows
Kabakoo tests its AI mentor through controlled experiments and usage data, from WhatsApp engagement gains to tutorial A/B tests and a Calibrate pass rate that stress-tests tone and quality.
Engagement Gains Tied to AI-Mentored Feedback
The introduction of LLM-generated personalized feedback sent via WhatsApp after learners submit selfie-video assignments is associated with an increase of more than 70% month-over-month in learner engagement.
Where Learners Actually Show Up
Over six months, more than 6,000 learners exchanged with the Kabakoo AI mentor via WhatsApp compared to approximately 300 via the app.
Testing the AI Mentor in Practice
During the WhatsApp Activation Sprint test, 531 learners received video material and at least one personalized feedback from Kabakoo's AI Mentor.
Kabakoo ran an A/B test between June 27 and July 16, 2025 with the OSC Numérique cohort, randomly splitting 127 learners into two groups to test personalized AI follow-up.
In a separate A/B test comparing a human-recorded tutorial video to an AI-generated one, 414 learners were assigned to the human video and 413 to the AI video. Among those who read the onboarding message (200 and 202 respectively), the AI video group showed a first key action rate within 24 hours of 14.4% versus 11.0% for the human video group.
Stress-Testing Tone and Quality
Kabakoo reports a Calibrate pass rate of 0.52, used to stress-test the AI Mentor's tone and quality.
Readers interested in the deployment details behind these tests can consult what Kabakoo learned from building its WhatsApp support agent, which covers learner usage patterns, app comparisons, and practical lessons from the rollout.