Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?

Key points

• In LLM-based KT the LLM generates the answer, which we call System-Two; it is either fine-tuned on the target data or reasons and votes over ten samples, which is slow and gives coarse probabilities.1
• We ask whether an off-the-shelf System-One LLM, which returns a probability for a typed question directly in a single pass, can perform KT when few or no learners are logged.1
• On seven datasets, Jev without any data from the target platform reaches a mean AUC of .706, above the best of 28 deep KT models trained on 8 learners (.689) and above System-Two Thinking-KT on all seven datasets (.650) at about 1/100 of its API cost.1
• Adding examples and a similar-learner statistic from the logged learners (JevKT) raises this to .722; JevKT stays significantly ahead of deep KT up to 16 learners and ahead on average up to 64, and supervised KT catches up between 64 and 128 learners.1

Sources (1)

• [1]Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 02:59 AM

  Knowledge tracing (KT) models need many logged learners, so a new course or platform starts without a usable model.

  In LLM-based KT the LLM generates the answer, which we call System-Two; it is either fine-tuned on the target data or reasons and votes over ten samples, which is slow and gives coarse probabilities.

Extractive summary: sentences quoted from the sources.