SQE Quality Assurance Annual Report 2022-23

We have applied our quality assurance (QA) framework to make sure effective controls are in place to manage risks to the quality and standard of the SQE. This also provides the evidence for the assurances which are necessary to maintain confidence in the SQE as a fair and robust assessment of solicitor competence. We do this through:

• regular meetings with Kaplan
• contractual obligations
• systematic monitoring, evaluation and analyses of assessment data
• obtaining evidence of compliance with agreed policies and procedures
• observations of live deliveries and assessor standardisation
• ensuring compliance with the SQE Assessment Regulations.

Our subject matter experts (SMEs) have continued to provide an expert, objective and independent judgement of the assessments. They reviewed a sample of the questions, observed deliveries, attended and observed markers' meetings and the standardisation and calibration of assessors.

The SQE is also subject to the oversight of:

• an Independent Reviewer who provides assurance as to whether the SQE is likely to deliver fair and defensible outcomes which will command public confidence. His oversight over the past year includes reviewing processes, observations of live deliveries, interviews with key members of the Kaplan team, observing training and meetings, attending the assessment board meetings and mitigating circumstances panel.

We have published his latest report [Link] for 2022-23 which covers the performance of SQE processes and outcomes between October 2022 and September 2023. While he identifies areas for improvement, he concludes that '…candidates, stakeholders and the public should have confidence that the SQE outcomes delivered in 2022/23 were fair and reliable.' There are recommendations to address some minor issues and we will provide details of our action against these in next year's report.

• an Independent Psychometrician who provides expert guidance on the psychometric analysis conducted on each SQE. This includes checks for bias, question analysis and the identification of trends over time and that the interpretations and reporting of these analyses are appropriate. She has confirmed that the psychometric analysis conducted for each assessment was fit for purpose for a high-stakes certification assessment.

Candidates are asked to complete a feedback questionnaire after each delivery. This is administered by Kaplan and the findings are shared with the SRA. Candidates' overall satisfaction levels for SQE1 range between 39%-49% and between 39%-62% for SQE2. We have seen an improvement in the candidate feedback on reasonable adjustments provided on the day from 54% to 84%.

The main areas for improvement relate to the booking process and queuing times, the usefulness of sample assessment questions available on the SQE website and the SQE1 assessment environment. Actions taken to address these are summarised below.

The SQE Assessment Regulations set out the obligations of the SQE Assessment Board. This includes meeting after each delivery to agree the pass mark and make other checks on the reliability and fairness of the assessment. It is chaired by an SRA executive director and its members include senior personnel from the SRA and Kaplan, the Independent Psychometrician and the Independent Reviewer (as observer).

In reaching its decision, the Assessment Board will receive:

• a report on delivery and any adverse events
• a report on any allegations of malpractice and improper conduct
• minutes of the meeting of the Mitigating Circumstances Panel
• a statistical and qualitative report containing information on test quality, the profile of the cohort, assessment performance (validity and standard), pass rates and demographic group performance.

We are confident that the evidence we have obtained through these quality assurance mechanisms supports the following:

• the assessments are valid: they test the competences expected of a newly qualified solicitor to the correct standard and they are set in realistic contexts
• each assessment has been constructed according to the assessment blueprint and reflects the SQE assessment specification
• the psychometric analyses tell us that the assessments are reliable; they measure consistently the performance of the candidates
• appropriate methods for setting the pass mark for high stakes professional exams have been applied
• the assessments are fair and free from bias
• the assessments are secure
• risk is appropriately identified and managed
• there is a commitment to continuous improvement and mechanisms are in place to learn from delivery failures and reduce or eliminate the risk that they are repeated.

Annex 1 lists the monitoring activity which we have undertaken during the period of this report and the evidence that we have collected.

In our last report we identified some areas of SQE delivery which required action. The actions taken in those priority areas were:

• SQE assessment disruptions
  • there have been instances where a candidate or a group of candidates have been unable to complete the assessment because of an IT failure.
  • For SQE1, Kaplan has taken steps to better manage the risks associated with this which has culminated in a longer assessment window being available from the January 2024 SQE1 delivery enabling candidates to be rescheduled in the same assessment window.
  • Staggered start times have also been introduced to relieve the pressure of all candidates starting at the same time (subject to quarantine requirements to ensure the integrity of the assessment).
  • We have also seen enhanced monitoring and quality assurance of test centres and an increased focus on training and communication between the venues and Kaplan to anticipate and identify problems at an early stage to minimise the impact on candidates.
• The booking processes
  • Finding solutions to problems caused by high demand for places when the booking window opens has been a priority.
  • Extending the assessment window from one day to five days for each of the SQE1 assessments has provided candidates with more choice.
  • Candidates taking SQE1 in January 2024 will be able to give an early indication that they intend to take SQE2 in April 2024 and indicate their preferred location.
• Information for candidates and training providers
  • Kaplan has added 40 additional sample SQE1 questions to the website and provided some additional materials about question structure.
  • SQE2 sample question material has been updated to include more detailed guidance on indicators of competence. This includes published performance indicators for each skill assessed in SQE2. A video recording of a sample advocacy assessment is now available.
• Differential attainment between candidates from different ethnic groups
  • Our analysis of candidate performance in the SQE continues to indicate a correlation between success and prior educational achievement and socio-economic factors.
  • We have commissioned University of Exeter to look at what may have caused different levels of attainment for ethnic groups in professional assessments. We published the literature review the factors influencing differences in outcomes by ethnicity in legal professional assessments in June 2023 and will publish the next stage of the research outcomes later this year.
• The provision of a spell check function for SQE2 written assessments: work to address this is ongoing.

An error occurred in the marking of a small number of scripts in one of the sixteen stations in the April SQE2 assessment. Kaplan reviewed the 346 scripts which may have been affected by the error and found that 324 scripts had no marking error. They found a marking error in 22 scripts. The findings of the review were reported to the Assessment Board. In only one case did the error impact a student's initial result and once identified, this student was advised of their correct passing result.

This has led to a full review of Kaplan's marking and moderation processes, marker training and earlier checks being made on marking. Whilst this remains an isolated incident, we acknowledge the harm caused. We are satisfied that the review into the error undertaken by Kaplan was thorough, and the actions taken were fair.

We are also satisfied that Kaplan has introduced additional checks to provide us with confidence that this error will not be repeated. Ensuring confidence in the reliability of the marking remains a priority.

Monitoring activity we undertook during the period of this report and the evidence that we have collected.

Assurance on valid assessments

Evidence collected:

• sample of assessments reviewed by SMEs and SRA
• observation by SRA and SMEs at SQE2 oral assessments
• composition of assessment checked by SRA
• report of Independent Reviewer
• psychometric analysis of assessment data.

Assurance on weightings for blueprint and assessment specifications

Each assessment has been constructed according to the weightings within the assessment blueprint for SQE1 and for SQE1 and SQE2 reflect the assessment specifications.

Evidence collected:

• sample of assessments reviewed by SMEs and SRA
• report from Kaplan's Head of Quality on each assessment confirming all processes relating to question writing and constructing the assessment have been followed
• composition of assessments checked by SRA.

Assurance on reliability

Evidence collected:

• Cronbach's alpha has been greater than 0.8 in all assessments in this period. Cronbach's alpha is a measure of test reliability. The gold-standard alpha for high-stakes assessments is 0.8
• SRA external psychometrician checks.

Assurance on fairness

The assessment is fair and free from bias and decisions about candidate performance are fair and methods agreed for setting the pass mark have been applied.

Evidence collected:

• question writing methodology
• assessor recruitment and training
• reasonable adjustments policy – reported against at monthly contract meetings
• SME review of a sample of the questions for each assessment
• Recognised appropriate standard setting methods for high stakes professional assessments applied
• SME and SRA and Independent Psychometrician and Independent Reviewer observations of live delivery of SQE2 oral assessments
• SME and SRA attendance at assessor standardisation and markers' meetings
• SME and SRA and Independent Psychometrician and independent reviewer attendance at Angoff Panel training for SQE1 standard setting
• analyses and evaluation of psychometric data reviewed by Independent Psychometrician and presented to the Assessment Board
• SRA and Independent Psychometrician attendance at mitigating circumstances panel meetings
• Independent Reviewer report
• SRA Independent Psychometrician checks and requests for further analyses where appropriate.

Assurance on assessment security

Evidence collected:

• confirmation from Kaplan's Head of Quality prior to signing off each assessment that all processes relating to training, writing the individual assessments questions and the assessment build have been followed
• confidentiality obligations imposed on all assessors
• conflict of Interests policy and process (reported on in monthly contract meeting).

Assurance on risk

Risk is appropriately and effectively identified and managed.

Evidence collected:

• monthly meetings with Kaplan to check against service levels including those relating to progressing applications for reasonable adjustments, managing complaints and website accessibility
• review of joint risk log at monthly contract meetings
• checking Kaplan's internal audit plans
• monitoring Kaplan's lessons learned log and action plan
• reviewing and monitoring Kaplan's Business Continuity Planning
• Independent Reviewer report.

Assurance on commitment to continuous improvement

Evidence collected:

• lessons learnt log and actions taken are available to SRA
• annual review of processes
• regular stakeholder engagement through focus groups
• qualitative feedback obtained from candidates
• evidence of actions taken in response to issues.