SRA response to the SQE2 pilot

8 June 2020

Background

In 2021 we plan to introduce a new centralised assessment - the Solicitors Qualifying Examination (SQE) - to make sure all qualifying solicitors meet a consistent standard.

Working with our assessment provider Kaplan, we have run two pilot exercises involving candidates to test whether our proposed assessments - SQE1 and SQE2 - work well. This is our response to Kaplan’s report on the SQE2 pilot results.

We offer a summary of how the SQE2 pilot went, and our response to the key issues it raises. We have also separately set out all our decisions on the final SQE design.

Executive summary

The purpose of the SQE

The two objectives we set out for the SQE are:

1. delivering greater assurance of consistent, high standards at the point of admission. This will protect consumers of legal services from all communities
2. encouraging the development of new and diverse pathways to qualification, which are responsive to the changing legal services market and promote a diverse profession by removing artificial and unjustifiable barriers.

We also set out that the criteria for the final assessment model were that it would be:

• Reliable
• Valid
• Manageable
• Cost effective

SQE2 is the second stage of the SQE assessment and is focused on assessing candidates’ oral and written legal skills.

In December 2019, we ran a pilot for SQE2 which saw 167 candidates completing the practical legal skills assessments across two venues for the oral skills and 29 Pearson VUE venues across England and Wales for the written skills.

This pilot followed our March 2019 pilot for SQE1, which mainly focused on the application of a candidates functioning legal knowledge. The results showed that SQE1 was on course to be a valid, rigorous assessment.

The purpose of the SQE2 pilot was to test whether our proposed legal skills assessment works well in practice. We wanted to know whether it met our criteria of valid, reliable, manageable and cost effective. We also wanted to explore key design questions. In particular we wanted to test whether candidates should all take the same assessments or whether they should be able to choose the legal practice area in which to have their skills assessed.

Our response is informed by:

• the results of the pilot
• a report by our assessment provider, Kaplan,
• a report by the SQE Independent Reviewer and advice from the External Psychometrician - to provide external scrutiny on the pilot and whether it achieved its purpose.

It is also informed by extensive engagement with hundreds of stakeholders, including in-depth discussions, surveys and expert input on our approach to SQE2.

The results of the pilot were as follows:

• They showed that through the type of assessments tasks we tested it is possible to design an assessment of practical legal skills that it is valid, reliable, manageable and cost effective. We are on course for the SQE to meet the rigorous, international high standards expected of a professional assessment, with a focus on consumer protection and fairness.
• The operational aspects of the pilot went well. The SQE Independent Reviewer confirms that the planning, operation and analyses of the pilot, were generally of a high or very high quality.
• More than 80% of candidates said the legal skills questions were set at an appropriate level, the questions were clear, and that they were the type of problems that might be encountered by a 'day one' solicitor¹. Some candidates thought greater clarity on the standard or level expected would be helped by the provision of sample questions and answers. We plan to publish sample questions and answers for SQE2 later this year.
• Candidate numbers were not as high as for the SQE1 pilot - this was not unexpected as SQE2 demanded a greater commitment from candidates. But, taken with nine years of data and experience from the Qualified Lawyers Transfer Scheme (QLTS) covering more than 5,000 legal skills candidate attempts across 18 assessments, the advice of Kaplan's Advisory Board, the SQE Independent Reviewer and the External Psychometrician, as well as views from a wide range of stakeholders the SQE2 pilot provides sufficient evidence to support the decisions about assessment design.

The two key recommendations from the pilot were that:

• Kaplan, the SQE Independent Reviewer and the External Psychometrician recommend that SQE2 should be a uniform examination in which all candidates take the same assessment. The optional models explored in the pilot did not give sufficient confidence that all candidates would be assessed fairly against the same standard.
• Kaplan, the SQE Independent Reviewer and the External Psychometrician recommend that SQE2 should take the form of a single examination consisting of between 15 and 18 tasks (or 'stations') in which candidates skills are sampled across the range of reserved activities and business law and practice. This number of stations would achieve the high standard required to assess candidates competence reliably and precisely.

We have accepted these recommendations. As explained in our summary of the final design document we have done this in the context of our wider SQE work and extensive stakeholder engagement. For instance, talking to hundreds of people, including solicitors and training providers, about the merits of a universal assessment compared to other options. We will publish a final draft SQE2 Assessment Specification in June setting out the detailed station model.

Progress on developing the assessment

We appointed Kaplan as the independent assessment organisation in August 2018 following a rigorous one-year procurement process. The SQE1 pilot took place in March 2019 and we published the pilot findings in July 2019.

Since then we have worked closely with Kaplan - and a range of stakeholders - to seek views on the key design issues of SQE2. We have involved hundreds of stakeholders to get feedback on:

• Assessment model: We have explored three options:
  1. one universal exam which all candidates take
  2. a combination of a compulsory common core which all candidates take plus assessments from a practice area selected by the candidate or
  3. full optionality for candidates.
• Assessment objectives and assessment criteria: We have worked to make sure of the clarity and appropriateness of the draft assessment objectives and criteria.
• Hurdles/compensation: We have tested whether candidates should be required to pass each individual assessment station or if there could be some compensation with one global final pass mark.
• Specific assessment stations: We have looked at the number of stations, the balance of skills and law and the extent of any permitted materials.

Our work on SQE2 included:

• discussing the SQE2 pilot plans with the SQE Reference Group - a group of academics, training providers, firms and representative groups including the Law Society, Junior Lawyers Division and legal sector equality group.
• feedback from education and training providers through direct engagement, meetings, events, and through our dedicated LinkedIn group, which is open to all stakeholders who are interested in the development of the SQE. It currently has more than 1,650 members including training providers and law firms
• six surveys on SQE2 related topics where we received 615 responses
• meetings with representatives from legal sector equality and diversity group, key professional bodies and roundtable meetings with solicitors and law firms
• two webinars relating to the design of SQE2 which have had more than 700 views
• an online survey to all candidates that attended the SQE2 pilot. While 167 candidates completed the pilot, 190 attended at least some element and the candidate survey was sent to all 190. 156 candidates submitted a response (82%). Overall, the responses were positive. In relation to the exam content and structure:
  • 80% agreed or strongly agreed or were neutral that the assessment specification was helpful in telling them what would be assessed (65% agreed or strongly agreed, 15% were neutral)²
  • 87% agreed or strongly agreed or were neutral that the legal skills questions were clear (80% agreed or strongly agreed, 7% were neutral)
  • 84% agreed or strongly agreed or were neutral that the legal skills questions reflected problems that might be encountered by a “day one” solicitor (67% agreed or strongly agreed, 17% were neutral)
  • 83% agreed or strongly agreed or were neutral that the legal skills questions were set at an appropriate level of difficulty (64% agreed or strongly agreed, 19% were neutral)
• In relation to the assessment specification, although 55% agreed or strongly agreed or were neutral that the assessment specification was helpful in indicating the level of difficulty of the examinations (20% agreed or strongly agreed, 35% were neutral), others thought greater clarity on the standard or level expected could be provided, which would be helped by the provision of sample questions and answers. We plan to publish sample questions and answers for SQE2 later this year.

The SQE2 pilot: what we did

• 190 candidates sat the December 2019 SQE2 pilot with 167 completing all elements. Assessments were held across two locations for the oral skills and 29 Pearson VUE venues across England and Wales for the written skills.
• We had two cohorts of candidates:
  • 97 completing candidates took business organisations, rules and procedures as their specialist context and
  • 70 completing candidates took criminal litigation.
• For the purpose of the pilot and in order to explore the three possible assessment models set out above, candidates took:
  • seven skills assessments in a common core
  • seven skills assessments in either business organisations, rules and procedures or criminal litigation.
• In addition, candidates sat a short multiple choice test to provide benchmark information.
• They were provided with legal materials for the skills exercises.
• SQE pilot candidates were selected to be, as far as possible, representative of those who will sit the SQE. Applications were encouraged from minority groups protected under the Equality Act. The analysis of the pilot data shows a similar distribution of demographic characteristics amongst pilot candidates to those taking the LPC.

The pilot – candidate results

• The purpose of the pilot was to test assessment design and processes, not candidate performance. It was not appropriate to set a pass mark since the pilot was not testing candidate performance. And it was not necessary to do so for the purposes of the pilot.
• Overall, Kaplan reported that pilot candidates displayed a wider range of performance than would be expected in a live context, with a longer tail of poor performance but also some extremely good responses. Pilot candidates will tend to be less motivated to prepare well for the pilot assessment than for a live assessment.
• The average score for all candidates on their 14 stations was 55.8% and scores ranged from 12% to 92%. For the seven common core stations, the average score for all candidates was 56.48% and scores ranged from 11% to 91%.
• The pass mark for the SQE will vary between exams, to make sure that the standard of the assessment remains consistent from one sitting to the next. As with SQE1, no pass mark was set for the pilot. However, we did estimate this using the method likely to be used in a live assessment. On these pilot questions the pass mark would have been 62%.
• It can be expected that the performance of candidates will improve significantly for a live licensing assessment as opposed to a pilot. In the live assessment, candidates will be more motivated, and training aligned to the SQE will be available.
• We will send candidates their full marks for the pilot assessments together with an indication of their performance as compared to the cohort.

Our response to the pilot

The results of the pilot were as follows:

• The SQE2 pilot set out to test the model of assessment to be used for the live SQE2. The outcomes of the pilot helpfully contributed to the evidence needed to finalise the SQE2 design.
• It showed it is possible through the type of assessment tasks we piloted to design an assessment of practical legal skills that is, valid, reliable, manageable and cost effective.
• The majority of pilot candidates were on the whole very positive in their feedback.
• The operational aspects of the pilot went well.

There are four key areas where Kaplan have made recommendations for the live assessment:

• Model: The SRA should adopt a uniform design for SQE2. Consumer protection and fairness to candidates mean that all candidates must be assessed to the same standard. In a uniform exam, all candidates take the same examinations and so are demonstrably assessed against the same standard. Other options, which give candidates choices, make it more difficult to deliver a fair assessment, where we can be confident all successful candidates have met the same high standard. We have decided to accept Kaplan’s proposal which is also endorsed by both the SQE Independent Reviewer and the External Psychometrician.
• Number of stations: Kaplan looked at the number of stations (ie separate assessment points (exercises) within the overall assessment) needed for reliability. Their advice is that 15 to 18 stations are needed for reliability so that we can be confident about pass/fail decisions.
• Compensation: A key concern of stakeholders is the extent to which candidates can compensate for poor performance in some areas with higher scores in others. Kaplan therefore used the pilot to look at the extent to which compensation was occurring between skills. Overall, while some compensation occurs, it is not very common and/or marked. On the whole, good candidates tend to do well in all sections and weak candidates tend to do poorly. Therefore, there is no need to set separate pass marks for separate skills.
• Weighting law - skills: Although the SQE2 assessments are tests of candidates’ legal skills, the pilot assessment criteria were weighted 50% application of the law and 50% pure legal skills. Kaplan looked at whether this weighting was appropriate. Based on candidate performance in the assessment, a weighting of 50:50 is most prudent to ensure candidates do not pass with an unacceptably low mark on their ability to apply the law. We will also look at the provision of appropriate legal materials for candidates.

These recommendations are based on findings from the pilot, nine years of data and experience from the QLTS, the advice of Kaplan’s Advisory Board, the SQE Independent Reviewer and the External Psychometrician. They also take into account views from a wide range of stakeholders about the SQE2 pilot. This provides sufficient evidence to support the decisions about assessment design.

We will accept all these recommendations, on the basis that the design delivers an SQE which is valid, reliable, manageable and cost effective.

Next steps – SQE2

We have published information on the final design of the SQE, following our June Board meeting. In addition to our decisions as a result of the SQE1 pilot, it includes our decisions on key issues such as our approach to assessing skills across SQE1 and 2, and options around taking the assessment in Welsh.

It also includes details of next steps, including the process of us applying to the Legal Services Board in the summer for approval of the assessment regulations. Below we have set out the key next steps:

• SQE2 updated draft Assessment Specification: we will publish a draft SQE2 Assessment Specification based on the uniform design model and we will seek stakeholders views on it. This will include holding a webinar and online survey.
• SQE2 final Assessment Specification: we will then publish a final version of the SQE2 Assessment Specification in summer 2020 so that universities and other training providers have the detail they need to plan their SQE2 training. We will also be publishing a suite of sample SQE2 questions and answers later this year.
• Diversity: we will publish a final EDI Impact Assessment, taking into account feedback on the current draft version and an updated report from the Bridge Group. In light of findings of the SQE2 assessments, including the performance by candidates from protected groups, we will continue to monitor and investigate attainment by candidates with protected characteristics once the SQE goes live. We will make sure our quality assurance processes for both SQE1 and SQE2 scrutinise questions during writing and editing, and after their use including looking at whether individual questions are disadvantaging protected groups over and above any general differences in performance. We will commission on research into the causes of any underperformance by minority protected groups with a view to promoting equality of opportunity.
• Operational arrangements: Kaplan will look at making some improvements to candidate instructions for the live assessment, for example greater clarity in candidate joining instructions or on the SQE website regarding the structure of the exam.

Further detail - Equality, diversity and inclusion

Our priority is to make sure the SQE assessment assesses every candidate to a consistent high standard. We need to be confident that every candidate - regardless of background - has a fair attempt at the assessment.

The numbers involved in the SQE pilots mean that we need to be cautious about extrapolating from the pilot to draw definitive conclusions around how candidates from different backgrounds will perform in the SQE. Any analyses must be viewed with caution and regarded as exploratory not only because of the relatively small numbers in the pilot, but also because of the presence of confounding³ variables, the fact that characteristics were self-declared, and, for some analyses, the complexity of the statistical modelling.

The pilot found little or no difference in performance by gender or disability (although the numbers were too small for a meaningful analysis of performance by disability). But it found that there was differential performance by binary ethnicity (white/Black, Asian and minority ethnic). Further analyses suggested educational factors were also important predictors of performance and performance in the pilot multiple choice test was the most important predictor of performance in the legal skills test.

This pattern of differential attainment by ethnicity is consistent with current experience in assessments in the legal sector and more widely. But it is of concern and we plan to commission research to understand better the causes of any attainment gap in the SQE with a view to working to promote equality of opportunity.

There is no evidence that the SQE2 assessment methodology or any assessment processes are unfair. But Kaplan will include in the live assessment a comprehensive package of measures to minimise any risk of unfairness to candidates from protected groups. This will include:

• Appointing external expertise to advise on these issues
• Recruiting a diverse group of assessors
• Diversity training for assessors, markers and question writers
• Training for assessors on how to avoid unconscious bias
• Statistical monitoring of the performance of assessors of live assessments for unconscious bias
• Statistical analysis of individual questions to check if they are disadvantaging protected groups over and above any general differences in performance between those groups
• Statistical monitoring and analysis of the performance of minority protected groups under the Equality Act 2010.

There were no differences in performance by disability in SQE2 skills. However only 12 of the completing candidates declared a disability under the Equality Act 2010 which was too few for any conclusions to be drawn.

10 out of the 167 completing pilot candidates sat with reasonable adjustments. The adjustments included additional time, individual/smaller testing rooms, enlarged font/screen magnifier and stop the clock rest breaks. The most common adjustment was 25% extra time.

We have published an updated EDI Impact Assessment and has sort views on it. We have also asked the Bridge Group⁴, to update their 2017 report on SQE and diversity. We will publish this alongside our final EDI Impact Assessment.

Further detail – assessment model

• As stated, we have decided we should accept the recommendation that we adopt a uniform assessment model, in which all candidates take the same skills assessments set in five areas of practice.
• Although individual firms employ lawyers in specialised roles, practice rights are generalised and allow solicitors to practise across the reserved legal activities. Therefore, we need to test across all the core skills that are required in order to be able to exercise these rights safely and effectively. And we need to test on a fair and consistent basis.
• Consumer protection and fairness to candidates mean that all candidates must be assessed to the same standard. In a uniform exam, all candidates take the same examinations and so are demonstrably assessed against the same standard.
• Kaplan's recommendation, that of their Advisory Board, of the SQE Independent Reviewer and of the External Psychometrician is therefore that a uniform exam is the most defensible assessment design for a national licensing examination and as a single entry point into the solicitors profession. We have accepted this recommendation.
• Kaplan and the SQE Independent Reviewer advised that the evidence from the pilot showed that candidates would find it very difficult to achieve a pass without performing well across all stations. Generally, where a candidate does well in an assessment, they do well across all stations, not in some. Therefore, there should be no minimum pass marks set for separate skills throughout the SQE2 assessment including advocacy. We have decided to accept Kaplan's recommendation. This should be kept under review once the exam is live.
• Reliability is critical in a national licensing exam where consumers must be protected. Kaplan looked at the number of stations needed for adequate reliability. Their advice, and that of their expert psychometricians, the SQE Independent Reviewer and the External Psychometrician is that 15 to 18 stations are needed to achieve sufficient reliability. So, we have decided to accept Kaplan's recommendation. We will publish a draft Assessment Specification in June. This will set out the assessment detail, such as the exact station design and guidance about the legal materials to which candidates can have access in the assessment.