SQE Quality Assurance Annual Report 2021-22

Annex 1 lists the monitoring activity which we have undertaken and the evidence that we have collected. We are confident that this supports the following assurances:

• the assessments are valid: they test the competences expected of a newly qualified solicitor to the correct standard and they are set in realistic contexts
• each assessment has been constructed according to the weightings within the assessment blueprint for SQE1 and for SQE1 and SQE2 reflect the assessment specifications
• the assessments are reliable: they measure consistently the performance of the candidate
• the assessments are fair and free from bias
• decisions about candidate performance are fair and methods agreed for setting the pass mark have been applied
• the assessments are secure
• risk is appropriately and effectively identified and managed.
• effective mechanisms are in place to detect delivery failures and reduce or eliminate the risk that they are repeated.

The Independent Reviewer provides additional assurance as to whether the exam is likely to deliver fair and defensible outcomes which will command public confidence. This is alongside the assurances that we obtain from our SMEs and our continuous monitoring.

His oversight includes (activities may vary from time to time):

• reviewing processes
• observations of live deliveries
• interviews with key members of the Kaplan team
• observing training and meetings
• attending the assessment board and mitigating circumstances panel.

We have published his post assessment reports for SQE1 in November 2021 and SQE2 in April 2022. He has also produced an annual report looking at the first three assessment sittings .

The Independent Reviewer has expressed satisfaction in the validity, fairness and reliability of the assessments. He has ‘observed good levels of planning and preparation and a great deal of care and attention in setting valid and reliable assessment tasks.’

In his report on the November 2021 delivery of SQE1 he states:

'I observed good evidence of a robust lessons learned process being implemented which will make, mainly minor, improvements for future sittings. The technical analysis, which evaluates the questions set and the examination overall, was thorough and provided a wealth of performance information about candidates that was previously not available nationally before the SQE was set up.'

We have appointed an independent psychometrician to provide assurance that the psychometric analyses of the SQE assessments delivered by Kaplan are robust and fit for purpose. This includes checks that the reporting and interpretations of these analyses are appropriate.

She has confirmed that the psychometric analyses conducted for each assessment were fit for purpose for a high-stakes certification assessment. In, particular, in light of the SQE being a new assessment, that several appropriate methods were explored when interrogating the data.

Kaplan's cycle of audit, risk management and lessons learned provides for continuous improvement.

Through Kaplan's QA, our oversight and that of the Independent Reviewer, the following areas have emerged as requiring action:

• SQE assessment venues:
  • we are satisfied that the assessments have generally been delivered to a good standard. We have seen comments in the candidate feedback survey which express general levels of satisfaction with the standard of delivery and the accommodation. However there has been occasion where events have disrupted the delivery of the assessment. We saw that Kaplan were quick to respond and provide solutions. We are pleased to see enhanced monitoring and quality assurance and an increased focus on training and communication between the venues and Kaplan.
• The booking process:
  • The candidate feedback survey recorded low satisfaction rates with the booking process for the April 2022 SQE2. Kaplan is addressing this through additional readiness checks being carried out immediately before the booking windows open.
• Information for candidates and training providers:
  • The candidate feedback survey records positive scores for the information available on the website. However, we and Kaplan note this as an area for some improvement. This is specifically in relation to understanding how better to inform candidates and training providers on how to prepare for the assessments
  • The website currently contains sample questions and answers, and other information about what to expect at the venues on the day of the assessment
  • In addition to Kaplan's stakeholder engagement, we are holding some joint events this year with a focus on this area.
• Differential attainment between candidates from different ethnic groups:
  • Our initial analysis of candidate performance in the SQE shows a correlation between success and prior educational achievement and socio-economic factors. However, early cohorts are likely to be atypical and we are concerned not to draw inferences too early. It is a complex area, and we want to understand more. We have commissioned the University of Exeter to look at what causes different levels of attainment for ethnic groups in professional assessments. This work will be completed by the end of 2023. But we will continue to interrogate the data we receive on candidate performance and seek assurance that all candidates are treated fairly.
• The provision of a spellcheck function for SQE2 written assessments.

Assurance required

The assessments are valid

Evidence:

• Sample of assessments reviewed by SMEs and SRA
• Observation by SRA and SMEs at SQE2 oral assessments
• Composition of assessment checked by SRA
• Report of Independent Reviewer

Assurance required

Each assessment has been constructed according to the weightings within the assessment blueprint for SQE1 and for SQE1 and SQE2 reflect the assessment specifications.

Evidence:

• Sample of assessments reviewed by SMEs and SRA
• Report from Kaplan's Head of Quality on each assessment confirming all processes relating to question writing and constructing the assessment have been followed
• Composition of assessments checked by SRA

Assurance required

The assessments are reliable

Evidence:

• Cronbach's alpha - in SQE1 it has been greater than 0.9 and above 0.8 for SQE2 (Cronbach's alpha is a measure of test reliability and the gold-standard alpha for high-stakes assessments is 0.8)
• SRA external psychometrician checks

Assurance required

The assessment is fair and free from bias and decisions about candidate performance are fair and methods agreed for setting the pass mark have been applied.

Evidence

• Question writing methodology
• Assessor recruitment and training
• Reasonable adjustments policy – reported against at monthly contract meetings
• SME review of a sample of the questions for each assessment
• Standard setting methods applied
• SME and SRA observations of live delivery of SQE2 oral assessments
• SME and SRA attendance at assessor standardisation events and markers' meetings
• SME and SRA attendance at Angoff Panel training for SQE1 standard setting
• Analysis and evaluation of psychometric data presented to the Assessment Board
• SRA attendance at mitigating circumstances panel meetings
• Independent Reviewer Report
• SRA external psychometrician checks

Assurance required

The assessments are secure

Evidence

• Confirmation from Kaplan's Head of Quality prior to signing off each assessment that all processes relating to training, writing the individual assessments questions and the assessment build have been followed
• Confidentiality obligations imposed on all assessors
• Conflict of Interests policy and process (reported on in monthly contract meeting)

Assurance required

Risk is appropriately and effectively identified and managed

Evidence

• Monthly meetings with Kaplan to check against service levels including those relating to progressing applications for reasonable adjustments, managing complaints and website accessibility
• Review of joint risk log at monthly contract meetings
• Checking Kaplan's internal audit plans
• Monitoring Kaplan's lessons learned log and action plan
• Reviewing and monitoring Kaplan's Business Continuity Planning
• Independent Reviewer Report