Skip to content

Latest commit

 

History

History
92 lines (78 loc) · 3.41 KB

README.md

File metadata and controls

92 lines (78 loc) · 3.41 KB

Goal

  • Grade homework on per question basis, and analyze the grade progression per question to explore the relation of program correctness and type-check status.

Approach

  1. For each homework, extract the functions on per question basis for each student submission.
    • Uses ocaml compiler-lib library to generate AST to facilitate extraction of functions on per question basis.
    • Extraction includes dependent functions.
    • Python script bin/question_split.py facilitates connection to ssh remote database, structure outputs, and navigate data processing traffics between Python script and Ocaml script (bin/main.ml)
      • See output project structure in section below.
  2. Run ocaml grader on per question basis. The command follows the pattern learn-ocaml grade --exercises="./exercises/hw1/question1/exercise" --grade-student="./exercises/hw1/question1/student_submissions/stu.ml" --timeout=60 --dump-reports grade_report. The exercises and student submissions can be stored anywhere on the file system provided the correct path is passed to the --exercises and --grade-student flags.
  3. Perform data analysis.

Usage

Python script calls for the execution of the dune project. Before running, set up the environment:

eval $(opam env)

To execute the project, in dune project ocaml/, run:

python3 bin/question_split.py

The resulting files generated will be produced under directory /analysis/out/. See project structure below.

Project structure

bin/

Where the main scripts are located.

  • question_split.py

    • Extracts functions by question.
    • Setup the structure of and organize outputs under analysis/output/.

  • main.ml

    • Process given Ocaml files into ast_out and pretty_ast_out in analysis/out/.

analysis/

Where analysis related data and output are located.

  • info/ (*Do not edit)

    • consentID2021fall.csv: Student IDs of those that approved the usage of their homework data.
    • fq.json: List of function names for related to each question in each homework.

  • out/

    Where the generated output files are located.

    See full structure below :

├── hw1/
│   ├── err.json 
│   ├── exercise/
│   │   ├── description.html
│   │   ├── meta.json
│   │   ├── prelude.ml
│   │   ├── prepare.ml
│   │   ├── solution.ml
│   │   ├── template.ml
│   │   ├── test.ml
│   ├── q1/
│   │   ├── student_submissions/ (only contains q1)
│   │       ├── d54baff9c3e9e5467505601b4b370289/
│   │          ├── Sep_08_2021_03:12:55.ml
│   │       ├── xxxxx/
│   │       ├── ...
│   ├── q2/
│   │   ├── student_submissions/ (only contains q2)
│   │       ├── xxxxx.ml
│   │       ├── ...
│   ├── q3/
│       ├── ...
├── hw2/
│   ├── q1
│   │   ├── exercise
│   │   │   ├── ...
│   │   ├── student_submissions/ (only contains q1)
│   │       ├── xxxxx.ml
            ├── ...

err.json:

  • logs errors occured during AST generation and function parsing

Data analysis

  • Error distribution
    • Homework level
    • Question level
  • How long students worked on a particular problem. Slice by
    • number of submission
    • time
  • Grade progression per question
    • once type checked how quickly a student gets 100