AI-powered marking #1248
AI-powered marking #1248
Conversation
There was a problem hiding this comment.
Thanks for this feature! Quite a bit of comments. Please look through, resolve them. I do have some clarification comments as well so please answer those as well.
One question I have is when are these comments used? I couldn't find anytime in which the comments are returned/retrieved to/by the FE.
| @doc """ | ||
| Gets an AI comment by ID. | ||
| """ | ||
| def get_ai_comment!(id), do: Repo.get!(AIComment, id) |
There was a problem hiding this comment.
Is there a reason why we use get! instead of get
| Retrieves an AI comment for a specific submission and question. | ||
| Returns `nil` if no comment exists. | ||
| """ | ||
| def get_ai_comments_for_submission(submission_id, question_id) do |
There was a problem hiding this comment.
Naming implies you are getting all AI comments. Also what is the use case for getting only one of the comments?
| """ | ||
| def update_ai_comment(id, attrs) do | ||
| id | ||
| |> get_ai_comment!() |
There was a problem hiding this comment.
This could raise an error which isn't handled (id not found in DB)
| field(:submission_id, :integer) | ||
| field(:question_id, :integer) |
There was a problem hiding this comment.
If these are FKs, you need to specify that in the schema
| answer_query = | ||
| if is_nil(question_id) do | ||
| base_query | ||
| else | ||
| base_query |> where(question_id: ^question_id) | ||
| end | ||
|
|
| @openai_api_url "https://api.openai.com/v1/chat/completions" | ||
| @model "gpt-4o" | ||
| # To set whether LLM grading is enabled across Source Academy | ||
| @default_llm_grading false |
There was a problem hiding this comment.
This should be done in the DBMS as mentioned above
|
|
||
| alias Cadet.{Assessments, AIComments, Courses} | ||
|
|
||
| @openai_api_url "https://api.openai.com/v1/chat/completions" |
There was a problem hiding this comment.
Should this be in the confg file (env variable). @model as well
| answers_json: answers_json, | ||
| response: response, | ||
| error: error, | ||
| inserted_at: NaiveDateTime.utc_now() |
There was a problem hiding this comment.
Should be automatically handled my Ecto
| updated_attrs = Map.merge(Map.from_struct(existing_comment), attrs) | ||
|
|
||
| case AIComments.update_ai_comment(existing_comment.id, updated_attrs) do | ||
| {:ok, updated_comment} -> | ||
| {:ok, updated_comment} | ||
|
|
||
| {:error, changeset} -> | ||
| Logger.error("Failed to update AI comment in database: #{inspect(changeset.errors)}") | ||
| {:error, changeset} |
There was a problem hiding this comment.
Probably need to refactor this log_comment function. It is doing more than just logging.
| llm_prompt = | ||
| answers | ||
| |> List.first() | ||
| |> Map.get(:question) | ||
| |> Map.get(:question) | ||
| |> Map.get("llm_prompt") |
There was a problem hiding this comment.
Is there a reason why we only use the first question's llm_prompt? What about the other questions' prompts
Description
Created a new feature to automate feedback generation of code submissions in Source Academy, with the help of LLMs. This will help save TAs some time in grading code submissions from students!
generate_ai_commentsendpoint to fetch question details and generate AI-generated comments for submissions.save_chosen_commentsendpoint to save multiple chosen comments for a submission and question for logging purposes.save_final_commentendpoint to save the final comment chosen for a submission for logging purposes.ai_comment_logstable to log various data points from inputs, original student's code, outputs generated by LLM, comments chosen, and final comment.AICommentsmodule to handle creation, retrieval, and updates for AI comments, including saving final and chosen comments.AICodeAnalysisControllerto handle AI comment generation, saving final comments, and saving chosen comments.generate_ai_comments,save_final_comment, andsave_chosen_commentsendpoints inAICodeAnalysisControllerTest.Note: This may require changes to the DB diagram in README.md.
Type of Change
Checklist