Idea: skill-based refresher flash-quiz game

• Topics and Question Database:
  • Generate on demand using LLMs based on user-selected topics
  • Ability to ground/reference a document collection
  • Question validation should be tracked, allowing verification and diputes by user, agents, and experts
• Questions and results should support rich types:
  • Text
  • Markdown
  • Mathjax
  • Mermaid
  • Code snippets with syntax highlighting
• Types of questions:
  • Simple (fast-touch interface questions)
    • multiple choice
    • multiple true/false
    • Multi-multi match questions (e.g. “Link the following terms to their definitions”)
  • Free-answer (must include rubric for auto-grading)
    • Coding (free answer + solution code + test cases)
• Adaptive difficulty:
  • Questions should adapt in difficulty based on user performance
  • The quizzer should generate a topic-specific score that is used for further testing, feedback, and dashboarding
  • Topic/difficulty may be visible, may be selectable, depending on system config
  • Answers should be selected from a pool of plausible distractors generated by the LLM for repeatability
  • Answers should list why they are correct/incorrect after each question
• Adaptive difficulty engine:
  • Questions are tagged with various topic tags tied to difficulty ratings
  • Adaptive testing involves building a robust model of the user’s skill levels, refining that model by querying high-uncertianty questions
  • Every e.g. 5 questions, the system should re-evaluate the user’s skill model and find the next best question to ask
  • Modelling method may vary