Law Professor Builds AI ‘Coach’ to Support Students Around the Clock
Professor has developed an artificial intelligence bot that uses curated, course-specific materials to assist students in mastering the applicable legal rules and their application. These digital “coaches” are available 24/7 to assist students in understanding challenging concepts and then to quiz students on their application, providing immediate feedback in a variety of question and answer formats. Thus far, Graves has deployed the concept in his evidence and contracts courses.
Graves uses OpenAI’s private custom GPT feature, which allows him to provide students with an interactive experience that is narrowly tailored to his specific course. Graves accomplishes this with a comprehensive set of instructions (i.e., prompts) telling the custom coach exactly what to do—providing guardrails to keep it focused on the objective of assisting students in this course—and uploading copies of the course text and other key instructional materials that facilitate Retrieval Augmented Generation (RAG). This domain-specific RAG layer increases the accuracy of the coach’s responses in the context of this specific course and dramatically reduces the potential for errors, as compared to a generic Large Language Model (LLM) trained on generic data of varying quality.
This “walled garden” of course-specific material addresses the common issue with LLM AI platforms that indiscriminately draw from all information on the internet.
“The LLMs pick up a good deal of erroneous information from unreliable sources, and they miss a lot of really good information that’s behind firewalls,” Graves says. “The bot has been instructed to respond to students when they ask for answers by walking them through in a Socratic-style dialog much as I might in class or office hours. When assisting students, the coach relies first and foremost on the information uploaded in its RAG layer, not only helping to explain and quiz the students on accurate course doctrine, but pointing students directly to appropriate sources within the course text itself.”
Of course, the key to this approach is a collaborative relationship between Graves and the publisher of his course textbooks. While Graves is a co-author of his contracts textbook, the copyright is held by West Academic (the publisher of both the Learning Evidence and Learning Contracts textbooks used by Graves). Graves worked closely with West Academic in developing an approach that would appropriately protect all copyrighted material uploaded to the coach’s RAG layer, and his use of both Learning Evidence and Learning Contracts is done under license from West Academic.
The use of the primary course text within the RAG layer effectively expands the value of the text far beyond the initially assigned readings. At the core of the coach’s domain-specific content, the textbook continues to anchor the coach’s role in assisting and quizzing students as they better learn to apply that content.
Supplementing the Teacher’s Role
Graves says, “the teacher’s role is not being outsourced to the coach—it is being supplemented in new ways for which narrowly tailored AI is uniquely suited.”
“The Coach does not replace basic course prep or attendance,” he says. “It is purely a supplement to these traditional teaching and learning tools—albeit a very effective one, arguably far more effective than traditional generic study aids or generic LLMs often used by students today. Perhaps most valuable is the coach’s ability to provide students with unlimited opportunities to apply the course material in a variety of assessment formats, all of which are subject to immediate feedback. At the end of the day, this is often the single most effective teaching and learning tool for law students, and the coach provides this tool in a manner that is always available and fully aligned with course content and course outcomes.”
Students access the coach through a dedicated course link, which provides for private interaction between student and coach, unless the student voluntarily decides to share the unique link generated by a specific conversation. The initial privacy of the conversation encourages students to ask questions they might otherwise be uncomfortable raising (the proverbial “dumb question,” which is often anything but).
It also allows students to use the coach in collaborative study sessions or to forward a conversation to Graves for further exploration. This latter feature is particularly useful in terms of quality control of both student prompts and responses by the coach.
“During the past two semesters, I’ve seen a few responses from the coach that could be improved and one blatant error,” Grave says. “However, the vast majority of interactive challenges arose from imperfect student prompts.”
Thus, the students get two additional benefits from using the coach: they learn the importance of effective inputs (prompts) and they learn the importance of verifying outputs.
Continuing to Fine-Tune the Tool
While the evidence and contract coaches have proven very accurate (Graves directly tests them regularly himself, in addition to frequent student feedback), AI remains imperfect, and the professor has continued to “fine-tune” his bots by uploading additional course-specific material based on his own testing and observations of student/coach interactions.
Graves teaches exclusively in the College of Law’s , so the 24/7 availability of his coaches is particularly important to a body of students located around the world.
“This has allowed me to be more efficient and effective with my time while giving our global students a uniquely tailored experience that will help them master course material, while being available at any time that is convenient to them,” he says.
