An evaluation framework for the Solicitors Qualifying Examination

January 2022

Report by Pye Tait Consulting

Executive Summary

Background

The Solicitors Regulation Authority (SRA) protects consumers and the wider public and supports the rule of law and the administration of justice. It does this by overseeing the education and training requirements, licensing individuals and firms, setting the standards of the profession and regulating and enforcing compliance against these standards. An effective system of legal services education and training is therefore vital to ensuring the quality and integrity of services so that the SRA can ensure that consumers and the public are protected and that the rule of law and administration of justice is supported..

The Solicitors Qualifying Examination (SQE), which will assess 'day one' competence, is the basis upon which the standards for practice are to be assessed at the point of qualifying as a solicitor. The introduction of the SQE and other reforms to the qualification process are intended to address the following problems with solicitor admission routes: inconsistent assessment across different routes into the profession; the system being expensive and inflexible; and, barriers to access.

Objective of the reforms

The SRA have defined two core objectives for these reforms:

1. Greater assurance of consistent high standards at the point of admission as a solicitor; and
2. Development of new and diverse pathways to qualifying as a solicitor which: i) respond to the changing legal services market; and ii) promote a diverse profession by removing artificial and unjustifiable barriers.

The evaluation programme

The SRA has planned a programme of evaluation to understand whether this major reform achieves the stated objectives, and if there are any wider impacts or consequences. The evaluation will both inform the ongoing implementation of the reforms by learning lessons from the early stages of implementation and assess the impact of the reforms. The first phase of the evaluation (the scope of this project and report) has involved establishing an evaluation framework and associated tools to underpin further phases of evaluation spread over ten years following implementation of the reforms. This report outlines the evaluation framework.

Evaluation approach

The introduction of the SQE and other related reforms to the qualification process is a complex policy change Therefore a theory-based evaluation approach has been adopted as this helps to understand the changes resulting from SQE’s introduction, why changes have happened, and how they are impacted by contextual factors.

The desired change, referred to as a ‘Theory of Change’, that the programme of evaluation will test, is that: a single assessment gateway for all intending solicitors (the SQE) which the SRA controls, the availability of transparent data from the SQE, the removal of regulatory barriers such as the period of recognised training and the requirements for prescribed study, and expanded availability of training routes are expected to primarily lead to (a) the SRA being better able to assure consistent high standards at the point of admission and (b) the development of new and diverse pathways to qualifying as a solicitor.

This desired change is illustrated by a graphic called a logic model which sets out the:

• Problem that the reforms are designed to address
• Objectives of the reforms
• Resources (inputs) and activities related to the SQE
• Anticipated immediate outputs from the reforms
• Expected, longer-term outcomes and impacts of the reforms.

The evaluation programme will collect data to explore whether the reforms have achieved the expected outputs, outcomes and impacts. The full outputs, outcomes and impacts that the SRA will evaluate are set out in the logic model.

What the SRA will evaluate

In summary, in the immediate post-implementation stage the SRA will evaluate whether:

• the SQE is a fair, valid and reliable assessment
• education and training meets SQE specifications, supports students/trainees and prepares candidates effectively for the SQE
• there is an expanded availability of price points and lower cost routes to admission
• the availability of transparent data gives candidates, training providers and legal services providers better information to help drive competition, driving up quality and reducing cost
• there is an expanded availability of training routes.

In the longer term, the SRA will evaluate whether:

• they have the information they need to assure consistent high standards at the point of admission
• they have confidence that consumers of legal services and the public are better protected by newly qualified solicitors having a consistent level of competence on entry
• candidates have confidence in a fair assessment
• there is confidence within the profession that the SQE provides a robust assurance of consistent high standards
• there is increased availability of cost effective and flexible routes to the profession
• students with protected characteristics and diverse backgrounds have greater access to opportunities which allow them to enter the profession.

It is important to note that:

• The first of the SRA’s two objectives is more directly within the sphere of its influence than the second
• Some of the broader impacts are dependent on wider societal change, such as universities’ selection processes, not solely the introduction of the SQE, and are therefore out of scope of this report.

The logic model has been translated into indicators to help assess the extent to which the reforms meet their objectives. The indicators are the measurable information that can be collected to evidence whether the reforms are achieving the outcomes and impacts as expected. For example, indicators to identify if the availability of transparent data about the effectiveness of training results in prospective candidates being able to select the most suitable courses for their particular circumstances include:

• SQE pass mark and pass rate
• Movement of prices over time
• Candidate satisfaction with the data (extent of use and helpfulness)
• Candidate satisfaction with training
• Number of students by training provider
• Diversity of students
• Changing behaviour by training providers

Such indicators will be collected through several data sources, including data from Kaplan (the single assessment provider), and surveys and depth interviews, for instance with candidates and training providers. The research activities in the planned evaluation programme will comprise these data sources. More detailed information about the indicators and their data sources for the anticipated changes resulting from the introduction of the reforms can be found in Tables 2-4 in the report. The indicators referenced in this report are examples of those included in a supporting indicator framework. This itself is not exhaustive as other indicators might become more relevant over the next ten years, for example as a result of wider changes.

How the SRA will evaluate the impact of the reforms

The SRA has planned a ten-year programme to evaluate whether the SQE and wider reforms are achieving the desired objectives. The SRA and Kaplan have a robust programme of quality assurance in place for ongoing monitoring of the SQE assessments. An independent evaluative study is planned to take place after five years (with an initial consideration of the scope of this after three years) to ascertain whether the SQE is fair, valid and reliable. This would be undertaken by independent assessment experts who would determine the framework for analysis and indicators to be included at that time.

We recommend that the SRA’s programme of evaluative research includes:

• An initial perception study (to be repeated at 3, 5 and 10 years), measuring stakeholders’ views and perceptions of the SQE;
• A specific, initial evaluation of the impact of Qualifying Work Experience (QWE) (to be repeated at 3, 5 and 10 years), informed by an annual survey on QWE;
• An initial evaluation of the market impact of the SQE;
• An evaluation of the SQE reforms three years post-implementation;
• A full evaluation of the SQE reforms five years post-implementation encompassing:
  • updated studies of those comprised in the initial and three-year evaluation an investigation into the impact of the SQE on the profile and background of the profession as well as candidate experiences; and,
  • an independent evaluative study of the fairness, validity and, reliability of the SQE assessments; and,
• A full evaluation of the SQE reforms ten years post-implementation, updating the research encompassed in the five years full evaluation to identify longer-term impacts.

Possible risks and unintended consequences of the reforms have also been considered in this report.

It is recommended that the SRA revisit the evaluation framework (including the Theory of Change, logic model, indicators, research plan as well as risks and unintended consequences of the reforms) on a regular basis with a view to making changes as and when needed, for example adding indicators as new information or data is made available. Such checkpoints will also offer an opportunity to consider any updates to the currently identified risks and unintended consequences.