💻 Week 04 - Class Roadmap (90 min)

2024/25 Autumn Term

Welcome to our fourth seminar/lab class of DS101A.

In this class we will discuss a data-driven approach to setting public policy, using the “Ofqual debacle” as a case.

Ofqual’s Direct Centre Performance model

The examination centre provided a list of teacher predicted grades, called ‘centre assessed grades’ (CAGs).

The students were listed in rank order with no ties.

With exams with a large cohort; the previous results of the centre were consulted. For each of the three previous years, the number of students getting each grade (A* to U) is noted. A percentage average is taken.

This distribution is then applied to the current years students-irrespective of their individual CAG.

A further standardisation adjustment could be made on the basis of previous personal historic data.

The formulas

For large schools with \(n\geq 15\)

\[ P_{kj} = (1 - r_j)C_{kj} +r_{j}(C_{kj} + q_{kj} - p_{kj}) \]

For small schools with \(n<15\)

\[ P_{kj}={\text{CAG}} \]

• \(n\) is the number of pupils in the subject being assessed

• \(k\) is a specific grade

• \(j\) indicates the school

• \(C_{kj}\) is the historical grade distribution of grade at the school (centre) over the last three years, 2017-19.

• \(q_{kj}\) is the predicted grade distribution based on the class’s prior attainment at GCSEs. A class with mostly 9s (the top grade) at GCSE will get a lot of predicted A*s; a class with mostly 1s at GCSEs will get a lot of predicted Us.

• \(p_{kj}\) is the predicted grade distribution of the previous years, based on their GCSEs.

  • You need to know that because, if previous years were predicted to do poorly and did well, then this year might do the same.

• \(r_{j}\) is the fraction of pupils in the class where historical data is available.

  • If you can perfectly track down every GCSE result, then it is 1; if you cannot track down any, it is 0.

• \(\text{CAG}\) is the centre assessed grade.

• \(P_{kj}\) is the result, which is the grade distribution for each grade \(k\) at each school \(j\).

Step 01 - Review the problem against the reading preparation (above) (10m)

• Answering questions taken across the class – based on the 📚 class preparation reading materials

Step 02 - The problem and solution design (20m)

• What is the problem of “fairness in exam grading” being described here – can you articulate what that means – and why it requires analytical consideration?

• Do you have any questions about the algorithm as described?

  • Can you identify any design strengths of the Ofqual algorithm?
  • Can you identify any design weaknesses of the Ofqual algorithm?
  • Can you qualify the role of teachers in predicting course grades?
    • What’s good about subjective considerations?
    • What’s problematic with subjective considerations?
    • In your view, does public opinion reflect (i.e. is it consistent with) the strength and weaknesses?

• Discuss with your neighbours

• Share your insights with the wider class

Step 03 - Technical considerations (15m)

• Why do we say this algorithm is linear?
  • Do you know of any more sophisticated approaches to modelling?
  • Would a perfect model resolve the issues identified and highlighted so far?

🍵 Break (~5 min)

Step 04 - Quality (10m)

• Did the algorithm meet the brief?
  • .. to “ensure, as far as is possible, that qualification standards are maintained and the distribution of grades follows a similar profile to that in previous years”
  • Teacher rankings were taken into consideration, but not the teacher-assessed grades submitted by schools and colleges (at least not for classes of sizes higher than 15 students)
• Was the algorithm’s brief appropriate?
• What metrics do we use to assess performance?
• What factors could the algorithm take into account?
  • How should we go about assessment of the “quality of the outcome” for the choices made?

Step 05 - Alternative approaches to the same problem (15m)

• Discuss the approach taken in Scotland and Ireland?
  • Present whatever other, different, kinds of approach you can think of

Step 06 - Government and public policy (15m)

“Algocracy”

• What is an “algocracy”?
  • Algocracy - on Wikipedia
  • What other areas of policy do you know of that are being defined through algorithms?