SQE Independent Reviewer Annual Report 2021-22

I have gathered evidence through a mixture of:

• direct observation of a wide range of exam creation and delivery activities
• interviews with key staff at SRA and Kaplan, including senior assessors
• access to management reports and information produced by the SRA and Kaplan
• support and advice from the independent psychometrician.

In order to provide an annual overview, this report is broken down into these key activities which enable the delivery of the SQE exams:

• Exam creation and production
• Exam delivery and assessment
• Candidate services, reasonable adjustments, mitigating circumstances, appeals and SQE in Welsh
• Standard setting, determining the pass mark and issuing results
• Quality assurance.

The key processes for successful exam creation and production are in place. I have defined 'successful' as the assessments will be high quality, can be assessed reliably and are valid.

A crucial aspect of validity is that the assessments require appropriate functioning legal knowledge (FLK). And/or simulate tasks that a day one solicitor needs to know how to do. The SQE assessments are complex and technically demanding to create and deliver, especially given their high-stakes nature and context.

During 2021-22, Kaplan and the SRA have increased their pool of subject matter experts (SMEs). Kaplan has significantly expanded its academic team by appointing qualified solicitors, many with prior and relevant assessment or other academic experience. These have expertise in a range of legal practice areas.

During the same period, the SRA has also appointed and trained SMEs, again all of whom are qualified solicitors. And have good knowledge of the expectations of a day one solicitor.

For both SQE1 and SQE2 exams, SRA and Kaplan have in place appropriate processes to create good assessments. These include the following safeguards:

• Question and task writers with relevant experience and knowledge of their specialist area of law. They also understand what a day one solicitor should be able to do
• Training to make sure new members of staff understand what features define effective assessment design
• Training in the use of language that is inclusive and avoidance of all forms of bias when writing assessments
• Drafting and editing which enables cross-checking and revision and reduces the risk of errors. This includes inappropriate or unrealistic context or too high or too low levels of difficulty when finalising questions or tasks
• The provision of SMEs from SRA provides important external review and feedback
• Clear reference to the assessment specification and care taken to ensure questions or tasks are accessible to candidates regardless of background
• Satisfactory security arrangements when creating, editing, storing and distributing assessment materials.

During 2023 and beyond, Kaplan will need to continue to ensure effective coverage of the FLK, and further, that this is comparable over assessment sittings.

Overall, the exam creation and production processes were effective and this is evidenced by:

1. the psychometric data analyses
2. the effective pass/fail standard setting process that was conducted and
3. candidates generally thought the assessments were fair.

With Kaplan's academic team and the SRA's SMEs relatively new in post, continued investment in the personal development of these colleagues should be planned for 2023. This should include tapping in to appropriate psychometric expertise to enhance knowledge and understanding of the technical/theoretical aspects of assessment design.

Plans are ongoing to achieve a wider and more diverse representation within SQE assessment writers, with the opportunity within SQE2 of some assessors and markers becoming question writers over time.

The SQE comprises SQE1 and SQE2.

1. SQE1 requires candidates to sit two 180 one-mark test questions which require the candidate to select the single best answer out of five possible answers.
2. SQE2 requires four oral assessments, taken at a small number of locations across England and Wales, plus 12 written assessments at Pearson VUE test centres.

During 2021/22, there were two sittings of the SQE1 exam and one of the SQE2 exam and exams took place in 42 countries. For the vast majority of candidates the exam delivery worked effectively, however some candidates faced issues, mainly when using a Pearson VUE test centre.

SQE1 exams enable computer marked candidate responses, thus providing a robust, very effective and highly reliable method for assessing the FLK. SQE2 exams require assessors to use global professional judgement when applying the marking criteria. The processes to achieve this were at least satisfactory for SQE2 but there is opportunity for enhancement with experience.

With the exception of oral exams, all are delivered to candidates on-screen via a Pearson VUE test centre across the UK and worldwide. While in the main this process worked effectively and had been extensively tested in advance, there were some concerns raised by candidates. These were about detailed aspects of functionality of the on-screen system and some issues arose on exam days.

The most serious issue happened at the Hammersmith test centre on 21 July 2022, when there was a significant failure of service as a result of an IT issue. Here of 136 candidates planning to sit their SQE1 (FLK1) exams, 108 were unable to do so. Additionally, 138 candidates were unable to sit SQE1 (FLK2) on 25 July 2022 at the same location, after the decision not to use that centre was taken.

Kaplan had to act quickly to offer suitable alternative arrangements when it became clear they would be unable to sit the FLK1 exam on the scheduled day. Candidates were offered a choice of resitting on a new date shortly after the initial planned date or waiting until a later SQE1 sitting. While most took the former option, others took the latter.

In a small number of cases, because the candidate was based overseas and had travelled to London specially to take the exam, the rearranged date did not work. Kaplan staff worked quickly to find alternative solutions for the candidates affected, treating each candidate with individual care and attention.

Of the candidates affected, the vast majority accepted the alternative arrangement put in place for them. With most going on to complete the exam on the rearranged date and received their result on the date expected. I am satisfied that these candidates had an equivalent assessment outcome to other candidates that did not face disruption. Kaplan covered the additional expenses incurred for the candidates affected and had good processes in place to ensure follow up communications were clear.

Inevitably, this unfortunate issue has led to a thorough review of the circumstances in which it arose between Kaplan and their computer-based testing provider, Pearson VUE. The incident provided a significant test of Kaplan's and the SRA's incident management and business continuity arrangements, which, in general, stood up to that test.

Having interviewed the relevant senior staff at Kaplan and received the incident report, I am satisfied actions have been, or are being, put in place which should significantly reduce the risk of a similar issue arising in the future.

Actions include Kaplan stepping up their monitoring at test centres on the day of exams And working with Pearson VUE to improve the process for capturing and systemically reporting any exam day risks and issues arising. It is critical to the reputation of the SQE, and most importantly to the candidate experience, that these improvements are maintained or further enhanced.

Separate to the Hammersmith test centre incident, issues were raised and feedback received by candidates regarding the functionality of the computer-based testing system. These have been systematically captured by Pearson VUE and Kaplan.

In the main the impact on the candidate raising an issue was not significant, and candidates managed to complete their exam. However the capturing of issues and candidate feedback has allowed for improvement such as better explanation about how to use the system in advance of the exam eg how to use the cut and paste function. This is allowing Kaplan and Pearson VUE to prioritise functionality improvements.

An improvement priority is to offer a spell-checking function when completing written SQE2 exams, which is not currently available on the Pearson Vue test platform. This means markers are having to give candidates the benefit of the doubt when faced with spelling, grammatical and other typographical errors in candidate responses. Without a spell-checking function the written assessments, that form part of SQE2, do not accurately replicate the context within which a day one solicitor would operate and require markers to make difficult judgements when assessing the clarity of responses. Provision of this function would reduce the risk of crediting candidates who cannot communicate at the appropriate competency level.

The majority of SQE assessments are computer-based and delivered via Pearson VUE. However, the four SQE2 oral exams were delivered at test centres located in Cardiff, Manchester and two sites in London, which are fully managed by Kaplan. These assessments are logistically complex, requiring the assessor and candidate to be face-to-face in an appropriately secure and confidential space. Candidates are also quarantined for that assessment task/day.

Despite this complexity, I observed these assessments worked well at the venues I visited - Cardiff, Manchester and Islington in London. I previously went to the second London venue at Euston during the SQE pilot and was impressed by the quality of the accommodation and management of candidates on the day. This continued to be my experience throughout my 2022 visits.

The reception and booking-in of candidates on the day of oral exams worked effectively. And I observed careful candidate identity checks taking place, including additional day of exam checks for one candidate. This candidate had an updated identity document, requiring an additional authentication check to be required during booking-in.

These exams are of a high-stakes nature and public confidence in them is important. Therefore it was reassuring to understand how Kaplan monitor technological and other developments worldwide to make sure that the integrity of the SQE is protected. Kaplan and the SRA know the risks of candidate impersonation or other potential vulnerabilities will require ongoing attention throughout the lifetime of the SQE.

During 2022, I observed several SQE2 written and oral assessor and marker standardisation and calibration meetings. Overall, assessors are well prepared for the difficult task of making sure marking is reliable. For example, that a candidate would receive a very similar assessment outcome regardless of the assessor allocated to review their work.

Kaplan's expanded academic team are a group of highly competent and knowledgeable qualified solicitors, with relevant prior experience and expertise. These staff have benefitted from the experience of other academic colleagues at Kaplan who have experience of overseeing the creation and delivery of QLTS exams.

Kaplan made good progress during 2022 in diversifying the composition of the SQE2 solicitor and actor oral assessors and written markers. There is a good range of ethnic groups, and those with a disability, represented.

I observed, as did SRA's SMEs, when selecting exemplar candidate work for assessor standardisation and calibration purposes, that occasionally more could be presented around (just above or below) the standard considered to be minimally competent or a 'minimal pass' or 'minimal fail'. Doing so helps assessors to become more confident about what is and is not credit worthy. The selection of candidate exemplars process was enhanced for the autumn 2022 SQE2 sitting.

Also, Kaplan has enhanced its approach to assuring that SQE2 written assessors were marking consistently and effectively. After the standardisation and calibration events, and during the live marking period, Kaplan made sure checks were in place. It also continuously review all its processes after each sitting.

During 2023 Kaplan should continue to seek feedback from assessors, their academic and psychometric teams and SRA's SMEs. This will help review the effectiveness of the standardisation and training processes for SQE2 assessors. Kaplan should also review that marking checks are undertaken effectively to assure marking quality, building on experience and psychometric evidence from the assessment outcomes.

It is important Kaplan continues to invest in the growth and development of their academic team, especially those leading calibration and standardisation activities. This should include:

• continued development of technical assessment expertise
• making sure best practice in other high-stakes professional exams which lead to a licence to practise is understood and learned from
• using psychometric expertise and analysis of data relating to SQE exam outcomes.

This helps to inform where improvements to the assessor standardisation, calibration and checking processes may be made.

Kaplan has made a satisfactory start in their provision of services to candidates. Overall, their processes were effective and their approach to obtaining candidate feedback is excellent.

As with other processes and services there was evidence of continuous improvement throughout 2022. For example, learning from some difficulties experienced by candidates when booking for SQE2 exams in the spring, the system was improved for future bookings.

Not unexpectedly when setting up a brand-new exam, the challenges that remain are accurate forecasting of future candidate demand for each SQE sitting and having effective capacity planning in place. Kaplan and the SRA are working closely with training providers to achieve this.

Kaplan has established good web services for candidates and their candidate services team demonstrates a strong commitment to providing a good service. When things go wrong occasionally, as evidenced in the Hammersmith test centre incident, the Kaplan team worked hard to find appropriate solutions. They treated each candidate as an individual and tailored their approach to that individual.

Kaplan has a formal complaints process which appeared to operate effectively, primarily being tested by the Hammersmith incident.

Reasonable adjustments (RAs) are offered to candidates. Across the two SQE1 and one SQE2 sittings (where results were issued during 2022), RA plans were in place for 264 candidates. Kaplan has a robust process in place to ensure that the RA plans agreed are justified.

The most common adjustments were, in order:

• extra time
• sole use of an assessment room
• access to medicine/snacks/water during the assessment.

Other bespoke provisions were also arranged where evidence supported this.

During 2022 there was evidence of continuous improvement, using candidate feedback and internal quality assurance reviews. For example, reducing the time taken to source suitable seats in test centres. There was evidence that the relevant team in Kaplan was careful to address each individual's needs. And effective interaction between the two parties to agree on the nature of any adjustment.

On a very small number of occasions candidates reported the agreed adjustment plan was either not in place when they arrived at the test centre or was not satisfactory. Kaplan has carefully reviewed and responded to such feedback and there is evidence of learning from this.

Overall, the RA process appears to have worked satisfactorily, this includes carefully monitoring outcomes for the overall cohort compared to candidates with adjustments. It is important that monitoring outcomes continues in 2023 as the data builds up and becomes more reliable for sensible comparisons to be made. This helps make sure neither route (with or without RA) appears to be advantaged or disadvantaged.

If a candidate believes they have suffered some disadvantage while taking an exam they may present a mitigating circumstance claim. The majority submitting a claim cited 'a mistake or irregularity in the administration or conduct of the assessment'.

I observed the meetings convened to consider the claims and observed a thorough and painstaking approach. Here each claim is given very careful consideration before being accepted or rejected by the Assessment Board. Where a request was accepted, appropriate actions were taken.

As time goes on, this process is establishing case history for various scenarios that are accepted or rejected. This will be important to reference to maintain consistency over time. As candidates 'run out' of sitting opportunities having not yet passed, there may be an attempt to 'game' the system by submitting spurious mitigating circumstances requests. To maintain the integrity of the process it will need to withstand the risk of the volume of spurious requests delaying processing times. While also protecting the interests of candidates who raise legitimate requests.

I am satisfied that Kaplan will continue to closely monitor and respond to this risk. I have seen evidence of continuous improvement actions taken in the light of the first few rounds of mitigating circumstances received and processed.

Should a candidate wish to, they may appeal the outcome of their assessment on grounds of either:

• mitigating circumstances which could not have been put before the Assessment Board before it made its decision or
• the decision reached by the Assessment Board or the manner in which that decision was reached involved material irregularity and/or was manifestly unreasonable and/or irrational or both.

At the time of writing 27 appeals had been raised in the period, with most rejected. Three were upheld, and one was still being reviewed. From the evidence available, I believe the process and policy was appropriately followed and cases were given full consideration.

After each assessment sitting, Kaplan issues a feedback survey for candidates to share their experience. This wide-ranging survey includes questions about:

• the website
• conduct of the exam
• assessment specification and questions
• where appropriate reasonable adjustments
• overall service.

Candidates that volunteer to leave their details are contacted to discuss their feedback and invited to a focus group. This survey and follow up activities provide excellent feedback, and Kaplan has shown some improvements through 2022. This includes satisfaction improving about reasonable adjustments matching expectations.

This process offers all the Kaplan delivery teams an opportunity to continue to use detailed feedback to prioritise and enable further improvements in 2023. For example, where candidate satisfaction scores dipped about Kaplan's SQE Equality and Quality team's service, actions are planned or have been taken to address concerns raised.

Overall Kaplan does an excellent job of gathering feedback from candidates after each sitting. Their challenge is to prioritise which improvements will provide most benefit to the candidate service overall and for the outcomes to be recognised in future candidate survey responses.

The SQE will be offered in the medium of Welsh in a phased implementation. This started in 2022 with the option of SQE2 oral and written assessment responses to be provided in Welsh by candidates. While there was no take up of this option in 2022, plans continue to have SQE2 written and oral assessments translated into Welsh by October 2023. And finally all SQE1 questions translated by October 2024.

Good progress is being made, including further pilot activity in September 2021 involving translation of a sample of SQE1 questions. However it is difficult to access legal experts, across a range of legal areas, who are also assessment experts and have Welsh language expertise. Therefore, plans continue to be developed to access relevant expertise and to deliver solutions which protect the integrity and accuracy of the assessments in a Welsh context.

Overall, the decision as to where to set the pass mark for each exam went smoothly, strictly adhering to the processes and policies set out in advance. The basis for the processes and policies followed well established standard setting techniques, as appropriate for an assessment like this.

The processes were supported by excellent analysis of the psychometric data and comprehensive reports to support the Assessment Board in determining the pass marks. Overall, the outcomes appear to be fair and defensible.

In preparing for Assessment Board meetings, which I observed, Kaplan provided good management reports which summarised a wide range of psychometric data. The item and station level analyses were very thorough. The associated measures of test reliability suggested very good internal consistency and high reliability for all the SQE1 and SQE2 assessments.

Other key metrics demonstrated reassuring outcomes, such as standard error of measurement, means and standard deviation data for each test and practice area performance.

Several analyses were undertaken to check for the possibility of inaccurate or biased marking and no significant issues were raised. In addition to the expertise within Kaplan, an independent psychometrician, engaged by the SRA, also oversees the collection and analyses of relevant assessment data. Thereby providing an extra layer of expertise and assurance.

The assessment pass marks and pass rates appear appropriate for these high stakes exams. Kaplan conducts very good analyses of outcomes by various protected characteristics. These data will become increasingly important, as the basis to form judgements and make any changes, as they build up over time.

Indeed, at this early stage, it would be inappropriate to draw firm conclusions based on many data sets as they contain low numbers so far. It is important that SRA reinforces to training providers and others using these data the risk of drawing inappropriate conclusions from small data sets.

Having said that, a significant advantage of the introduction of the SQE is increasing transparency about candidate performance. And emerging evidence of candidate performance by practice area is likely to be of some immediate benefit to candidates and those preparing support for them.

Overall, I was reassured that the outcomes when determining the pass marks appear to be fair. As I have reported previously, there was a concerning difference in outcomes by ethnic group, with White candidates generally achieving a higher pass rate than other groups.

I have found no evidence of bias in any process connected with the SQE, indeed, I have observed many measures aimed at eliminating or reducing the risk of bias. The potential for early cohorts to have an atypical demographic make-up should also be recognised.

It should also be noted that demographic data related to each candidate is self-declared. And a substantial minority decline to provide any or some of the data used to populate a wide range of demographic fields requested when booking the exam. From the data available, differences observed between ethnic groups are partly due to differences in educational background.

I have speculated that differences in access to prior support and learning resources may also help to explain why differences in outcome are observed. It is important, therefore, that the work being undertaken by the University of Exeter for the SRA, is completed. This looks to understand the causes of the attainment gaps seen in legal and other professional qualifications and to identify potential actions. After full consideration by the SRA and Kaplan, this may enable enhancements to be made to the assessment process and/or candidates' preparation for the SQE and/or actions for other stakeholders.

It is also important that the SRA and Kaplan encourage candidates to complete all the demographic data fields when booking exams. This helps to enable a fuller picture in order to explore any demographic differences over time as well as to analyse pass rates by protected characteristics.

It may be that the benefits of doing so need further explanation to candidates at the time of booking in order to improve completion rates. Of course, the fact such consistent and high-quality data is emerging for the first time is a huge step in the right direction. Especially when exploring issues related to demographics in academic attainment connected to solicitor qualification.

Kaplan has provided an excellent summary of outcome data related to different demographic profiles in their annual report. Over time this information will become even more important, as the data set increases. And will provide an unprecedented and increasingly reliable basis for analysis into access to solicitor qualification. It is likely that multivariate analysis will become especially informative.

Results were issued for the first SQE1 (November 2021 sitting) in January 2022 and subsequently results from each SQE1 and SQE2 sitting in 2022 were on time.

3,290 individuals received an outcome for at least one part of the SQE assessment in 2022. A website failure caused a delay to issuing results for the first SQE1 exam. But this was resolved on the day and subsequent issuing of results has been as planned.

There is a comprehensive quality assurance protocol in place to support the delivery of the SQE. The exams are high-stakes, and the delivery of SQE2 exams, in particular, is complex and technically challenging.

As would be expected, Kaplan has at least two layers of internal quality assurance. This is supplemented by external assurance from the SRA, their SMEs and the independent psychometrician. My role as the Independent Reviewer provides an external assurance overview.

During the course of 2022, Kaplan was at least satisfactory in the delivery of exam processes which was a significant achievement given the challenge. Effective quality assurance checks are in place. For example, sampling that the reasonable adjustment arrangements are processed and agreed with the candidate in a timely manner and then effectively implemented on site. Checks were undertaken for all key processes.

Both Kaplan and the SRA demonstrated continuous improvements based on learning and committed growing resources to assurance activities. For example:

• improvements to the candidate journey for booking exams
• the use of the SRA's SMEs to review draft assessment materials
• learning from the Hammersmith test centre exam day issue mentioned above.

While an at least satisfactory start has been made to the way quality is monitored, there is absolutely no room for complacency. An ongoing commitment to learning from issues arising and managing risks effectively will need to be maintained.

This includes the SRA and Kaplan continuing to support a culture among staff that continues to encourage openness about concerns or risks. And to get things right first time and learn and adapt if and when issues arise.

I have observed good progress during 2022 and leaders in both organisations will need to keep encouraging this positive working culture. This is especially true if and when issues arise in the future.