Review of first sitting of SQE2

In order to externally assure the preparation for, and conduct of, the SQE2 exam and pass/fail decision making I observed a wide range of activities. These included:

• Interviewing Kaplan staff responsible for planning and delivering activities such as:
• task writing and editing
• website management
• reasonable adjustments
• candidate contact services
• psychometric analysis
• assessor recruitment and training
• logistical arrangements for managing the exams
• Attending and observing key delivery activities such as:
• hearing feedback from SRA subject matter experts on their views of the draft SQE2 assessment tasks
• visiting oral and written assessment venues, where candidates were being assessed
• training and calibration of Kaplan assessors
• the mitigating circumstances panel
• the assessment board (and pre-meeting) held to determine the pass/fail decision.
• Receiving management reports and summaries, risk, issues and lessons learned logs, psychometric analyses and reports
• Topic specific emails and phone/video calls to Kaplan and SRA experts to request additional information to gain fuller understanding of specific issues.

In most cases I provided immediate feedback to Kaplan and/or SRA if I observed good practice or had a question or concern. While concerns were rare, by raising a question or concern immediately it did occasionally allow some follow up or reinforcement of a particular point to happen before a meeting or task was concluded. In all cases Kaplan and SRA have had the opportunity to review and comment on my conclusions and recommendations, to enable a check for factual accuracy. My recommendations for Kaplan and/or SRA to consider are shown below within each section.

The SQE2 exam comprises four oral assessments and 12 written assessments. These assessments are sometimes referred to as assessment stations or OSCEs (objective structured clinical examinations, given their wide use in qualifying professional clinical competence exams). The written assessments were taken by candidates at Pearson Vue assessment centres based across the UK and worldwide. The oral assessments were held face to face in assessment venues in Cardiff, Manchester and London.

I reviewed the process for producing the tasks for the assessments as conducted by Kaplan. This is led by their academic team, some of whom have relevant experience in commissioning, creating and editing similar tasks in the context of Qualifying Lawyers Transfer Scheme (QLTS) exams. Those members of the team new to the process were given theory based and practical training led by those with prior experience. The task writers commissioned are all qualified solicitors with relevant expertise in the aspect of law being assessed.

Each task writer is expected to have a command of the expectations of the threshold standard for a day one qualified solicitor and many have relevant day-to-day experience of working alongside newly qualified solicitors. This gave me confidence that the nature of the tasks was realistic to replicate potential tasks a day one solicitor could be faced with and would be set at the appropriate level of demand.

SRA use subject matter experts (SMEs) to review a large sample of the draft tasks. The SMEs are also all qualified solicitors and each task is reviewed by the SME with relevant knowledge of the law. The SMEs also have relevant experience of working with newly qualified solicitors.

I observed the SRA review session when the SMEs fed back their comments on the draft SQE2 tasks. As part of this review the SMEs are asked to comment on: any errors or factual inaccuracies; level of demand of the task and comparability of oral tasks across sittings. Each candidate is assessed on different oral tasks according to the date on which they sit these exams, each oral task is 'live' for one day only. SMEs also commented on relevance to replicate tasks that a day one solicitor might face.

Overall, the SMEs thought the tasks to have successfully met the above criteria for review. While a small number of minor   points were picked up, the SMEs were positive about the level of demand and about comparability of oral tasks. This latter point is of crucial importance because task writers are required to generate a larger number of oral tasks, than written tasks, to accommodate multiple sittings of the oral exams across a range of dates. On this occasion four sittings were needed to cope with the logistics of the number of candidates taking the SQE2 in the April 2022 series.

I was assured that Kaplan carefully considered all of the points raised and consequently made a small number of minor changes in the final stages of the editing process of the tasks. This activity happened within two months of the exam being taken and both Kaplan and the SRA intend to complete this activity earlier in future SQE2 task setting cycles.

Candidates were required to book a slot for their exam. During the booking process some candidates were impacted by long waiting times to book due to an issue with the set-up of the booking website. Kaplan have already captured these issues as part of their lessons learned process and have implemented improvements prior to the next SQE1 and SQE2 sittings. It is important that these improvements are delivered.

I also recommend that Kaplan review how capacity is made available to ensure there is a balance across sittings, while enabling candidates to have some choice over their preferred date(s) for assessment. This will reduce the risk that one sitting is under or over-subscribed, will help to balance the number of candidates across sittings and prevent the risk that one becomes a self-selecting group. For example, if candidates booking late in the process have only the last sitting available to them. This will enable determining the psychometric analyses used to assist defining the pass/fail boundary easier if each sitting has a randomised, and therefore more likely to be, representative make up of the overall cohort and is better balanced in terms of numbers of candidates taking each sitting.

Kaplan contacted all candidates who had sat SQE1 in November 2021 with a reasonable adjustment plan by the end of November 2021 about a reasonable adjustment plan for SQE2, with an objective to support candidates with their preparation for SQE2. The broader communications plan reiterated the advice to candidates to contact Kaplan as soon as possible about reasonable adjustments, and before the booking window closed, to enable better planning and certainly for candidates. Only 14 of the 107 reasonable adjustment applications that were received, were submitted after the booking window opened. This is evidence of the success of early engagement with candidates, those progressing from SQE1 to SQE2 experienced a proactive approach from the Kaplan E&Q Team to agree their plans.

The SQE2 reasonable adjustment plans were generally more complex compared to SQE1. Some candidates had different plans for the SQE2 written and oral assessments.

Preferred date and test centre location matches were usually offered for candidates taking oral assessments, where a choice between Cardiff, Manchester or London was offered. However, some candidates who requested sitting 1, and had requested London as their preferred test centre, did not get a seat on sitting 1 because they did not apply until after those dates were fully booked and were subsequently offered other dates. This is another reason why Kaplan should review how capacity is made available to candidates.

• More than 100 candidates made reasonable adjustment applications.
• 57 sat the assessments (an additional two were 'no-shows' on the day)
• 48 candidates withdrew or were withdrawn pre booking, for example if they were no longer eligible because they failed SQE1.

Kaplan supported all candidates who requested accommodations about the date and/or time of their assessments to support their observance of Ramadan.

The main reasonable adjustments offered were extra time, in addition other adjustments included:

• own assessment room
• medication and/or snacks/water to be available in the assessment room
• 'zoom' text or adjustable fonts
• extended breaks between assessments
• reader, recorder, scribe or assistive technology.

Overall, the reasonable adjustments process appeared to perform well and improvements were delivered following lessons learned from the first delivery of SQE1.

Prior to the oral exams being sat, actor and solicitor assessors attend a training and calibration meeting. This meeting aims to ensure that all assessors would use their professional judgement to determine a very similar outcome when assessing the competence of a candidate, regardless of which assessor was allocated to that candidate. This calibration activity, as well as setting appropriate assessment tasks, is at the heart of making a success of assessing the SQE2 tasks.

All assessors were expected to complete on-line equality, diversity and inclusion training prior to starting their assessment duties.

On 15 April 2022 I observed the online training and calibration of actor assessors in preparation for the client interview oral exams to be held on 20 April across four venues nationally. The meeting was well led by an experienced and senior member of Kaplan staff. It was made clear to all assessors that consistency of their presentation to each candidate (how they acted). And how they made their assessment decisions was equally important in order to conduct and replicate the assessment task for each candidate.

Kaplan employs a third-party specialist agency to source suitably qualified actors; the agency provides senior actors with assessment experience to specifically focus on consistency of acting during the oral assessments eg consistent emotional and energy levels when interacting with candidates. The agency also sources actors who can faithfully represent the role play scenario eg to replicate the age of the client in each scenario. Kaplan takes responsibility for ensuring the marking is conducted consistently.

The assessment task and scenario required by the actors to role play, as well as marking criteria and administration instructions had been securely sent in advance of the meeting. There was evidence that the actors had familiarised themselves with their 'script' for the role play.

At the start of the meeting, key principles were explained such as marking being based on holistic professional judgement rather than points-based marking and that candidates were being judged against the threshold standard. The tools needed to complete the task were explained as was the approach to marking. A detailed explanation of the marking criteria was given, and a reminder that it was skills and not law that was being assessed for this interviewing task.

A standard setting grade is also provided by each assessor for each candidate, to assist standard setting (and borderline regression in particular), and advice was given on how to decide whether grades should be: pass, marginal pass, marginal fail or fail. The Kaplan leader provided a thorough and clear description of the above points, including a good explanation of the expanded marking guide.

A video recording of four mock candidates responding to the task, with an actor playing the client, was then played. The mock candidates were played by members of Kaplan staff, and security of the exam was maintained during these recordings as only Kaplan staff, the assessors and the SRA had sight of the assessments prior to the exam day. After each recording the actor assessors were given time to individually mark using the marking criteria. These marks were then captured using an on-line record form. The form displayed the marks and standard-setting grade from each actor assessor and some of the actors were then asked to explain how they had come to their initial assessment on certain marking criteria.

With the exception of the last video recording, or exemplar, the actors demonstrated a high degree of consistency in their marking. The final exemplar, presumably chosen to exemplify a potentially complex candidate response, elicited a much wider range of initial marks across the actors. This exemplar illustrated a candidate exhibiting an unusual behavioural manner, which treated the client with sensitivity and respect but lacked control of the discussion, as would be expected of a day one solicitor. This led to a helpful discussion. It enabled actors who had initially provided a grade for each criterion which was an outlier to explain their thinking to the wider group. Enabling the whole group to benefit from hearing individual's views of how the marking criteria should have been applied and these insights brought the group much closer to a consensus. At the end of the discussion the Kaplan leader talked through how he had applied the marking criteria, leaving those in the meeting with further insight about what to do should a similar candidate response present itself in a live context.

Overall, this session appeared to successfully equip the actor assessors with relevant information about how to apply their professional judgement as well as complete the necessary administration.

On 25 April 2002 I observed the online solicitor training and calibration in preparation for the oral exams to be held on 28 April 2022 in London.

The meeting was led by a Kaplan SQE Subject Head, who is also an experienced assessor and qualified solicitor. The advocacy task, marking and administration instructions had been securely provided in advance to all solicitor assessors participating.

The meeting followed a similar format to the actor session reported above. Once again video role plays of mock candidates (drawn from a pool of Kaplan staff) were shown to the group. The mock candidates addressed the task to a judge, played by an experienced solicitor assessor. As with the actor session, the solicitor assessors individually marked each role play and their marks were collected and shared across the wider group. Once more, a discussion across the group enabling a common understanding of how to consistently apply the law and skills-based marking criteria was successfully completed.

I observed a good discussion of the first exemplar, which included a clear explanation of how the assessor guide should be used and not used. For example, reinforcement that the guide was not a checklist, rather it attempted to cover a wide range of valid points a candidate might raise and therefore not everything should be expected to be presented in the live assessment, even from a superior performing candidate.

There was a useful reminder about how cultural differences might lead candidates to present themselves in these face-to-face assessment scenarios, for example making little or no eye contact can be a sign of respect.

Towards the end of the session there was an excellent summary of the key messages.

Overall, I felt the two calibration sessions achieved their overall purpose and would equip the assessors with the necessary tools and information to conduct their work consistently in the live context. I was also provided with access to the EDI training provided to every assessor prior to them starting work in the live context. Kaplan had sought third party support and advice about how best to ensure all assessors carefully consider EDI issues when conducting their work. This training was designed to heighten awareness of the potential for unconscious bias.

In preparing for future calibration training, I recommend Kaplan consider having identical briefing slides, to be used at the start of each session, for activities or instructions that are common. I observed some minor differences to the way these common instructions were presented and the emphasis given, this would be prevented if a common script and slide deck was used.

726 candidates will receive results and one withdrew part way through.

I observed candidates sitting their SQE2 written exams at the Exeter Pearson Vue assessment centre. This included some of the same candidates I had observed earlier when taking their SQE1 exams at the same venue in November 2021. Once more during my visit I observed a consistent and high-quality reception and booking-in process for candidates. I observed day two of the three days of SQE2 written exams and candidates appeared very familiar with the expectations by this stage. I observed that small improvements, based on lessons learned from the SQE1 sitting, such as clarification about candidate access to water and earplugs during the exam were implemented.

All SQE candidates were seated next to each other in the standard spacings, within their individual screened off stations. I did see that one candidate was concerned that a space bar on their keyboard was 'sticky' and this keyboard was immediately replaced, prior to the exam starting. All candidates started the exam at the same time, as scheduled.

While nothing exceptional was observed during my visit, nationally some issues were reported. The main issues arising were concerns raised by the candidates, primarily about information technology issues. The three most widely reported issues concerned: screen resolution, keyboard issues and computer failure. In the vast majority of cases the issues were resolved quickly. In nearly every case the candidate was able to complete their exam as expected, for example by changing the keyboard or moving a candidate to a new assessment station.

Pearson Vue keep a log of every issue raised by any candidate during the exam and report all their findings to Kaplan. This log confirms any resolution action taken during the exam, and their diagnosis of the issue, raised by any candidate.

If any issue raised by a candidate has not been fully resolved on the day of the exam, such issues can be raised by the candidate as a potential mitigating circumstance. If the candidate considers their marks or performance in the exam were materially and adversely affected in any way, by raising a mitigating circumstance claim, the issue can be fully explored. And each mitigating circumstance application is fully considered by a panel, prior to the issue of results. Further details are provided in section 8.

Assessing more than 700 candidates, over three separate days, in multiple locations in the UK and worldwide, using complex computer-based assessments is always likely to lead to a few issues being reported by candidates.

Overall fewer issues (as a proportion) were raised for the operation of the SQE2 written exams compared to the SQE1 exams. While some improvement had been made there is absolutely no room for complacency, because any issues can cause candidates additional stress and pressure, and I was reassured that Pearson Vue and Kaplan are conducting thorough investigations into issues raised. It is important that this path of continuous improvement is followed, each lesson learned is meticulously followed up and further improvement demonstrated in the next set of SQE2 written exams.

On Wednesday 20 April and Thursday 28 April 2022, I attended the Cardiff and Spring House, London oral assessment venues respectively.

Each candidate takes four oral assessments, two assessing advocacy competence and two assessing client interviewing skills. These are assessed by solicitor and actor assessors respectively. While assessments are being conducted, candidates are held in a secure (quarantined) setting, so that there is no risk that any candidate taking an early assessment could share information with other candidates yet to sit that same task.

As these assessments are conducted over several days, four sets (or sittings) of tasks are created so that each is new for that day of assessment.

The Cardiff venue took over a significant proportion of a city centre hotel, where Kaplan had previously piloted some Welsh language exams at the same venue. Kaplan have also invested in their own new oral assessment facilities at their Spring House office in London.

In Cardiff, I observed day two of two days at this location. I was impressed by the welcome in reception for candidates and the clarity for them about where to go and what was required.

I observed the actor and solicitor assessors' summary of key points prior to assessments taking place at the start of the day, as well as consideration of any issues arising after the first assessment took place. Immediately after this, a cross site call was convened each day to ensure any issues arising at any of the sites (Cardiff, Manchester and two sites in London) were addressed and coordinated to give candidates a consistent experience. As each site had candidates being assessed on the same tasks this is a critical step for consistency. While no significant issues arose, this session felt short of time, partly because one of the venues was running a few minutes behind the others. Kaplan already plan to ensure there is more time on the schedule to conduct this important activity at the same time across all oral assessment sites.

At Cardiff I next observed, using the remote CCTV observation room facilities, assessments taking place in real time. All assessments are video and audio recorded and these records are stored and can be reviewed if any issue arises during, or after, the assessment, for example if an unplanned event happened.

Prior to attending Cardiff, I had observed the actor assessors preparing for their assessment activity. This meant I was familiar with the task being asked of candidates and I observed a reassuring consistency of approach across the actor assessors in the live assessment setting. This process is aided by the Lead Assessor for the actors. There is one at each site, and throughout the day this person observes the actors while conducting the assessments to help offer feedback on points of style of presentation, levels of energy and emotion used and overarching consistency.

I observed that a separate room had been set up for candidates with reasonable adjustments, and while I did not observe these assessments taking place, the facility appeared fit for purpose.

I also took the opportunity to review how the (optically read) marksheets, generated by the assessor after each candidate's oral exam to record the outcome, were managed and processed. A thorough and timely booking-in process is conducted so that these key documents are accounted for at every step, with clearly defined individuals responsible for this task.

I also observed how the confidential tasks given to candidates on the day of the exam were stored and processed. Prior to the first live sitting I had recommended that some careful safeguards be established to ensure that the printing, transporting and storage of these materials remain secure and confidential. I was shown where the documents were securely stored and it was explained who had access. These safeguards are an especially important aspect of security in a more public setting such as a city centre hotel.

Overall, the processes observed appeared fit for purpose, however there remains no room for complacency in such a sensitive aspect of the operation of the exam.

At the Kaplan Spring House venue on 28 April my observation activity followed a very similar pattern to that at Cardiff. This time I spent more time observing the solicitor assessors, because I had observed their preparation for standardisation for the assessments prior to the day.

On this occasion I observed the Chief Marshal reminding all participants (Kaplan staff and assessors) of:

• the importance of conducting themselves in consistent way
• how the assessors were carefully segregated from candidates during breaks in the assessments
• where assessors needed to exercise caution when speaking in any areas where candidates (or others) might be found.

As in Cardiff, I observed a reminder of key messages to actor and solicitor assessors. I also observed the solicitor assessors debrief after the first candidate had been assessed by each assessor.

I was reassured at the level of consistency of approach across the Cardiff and Spring House, London venues. I also heard good practice, such as reminders to assessors not to double penalise across different marking categories. There was a good discussion about not comparing candidates while assessing during the day, rather to assess each candidate's response individually on its merit.

I observed candidates being moved around the assessment centre in controlled and secure conditions and saw that candidates had appropriate access to water, food and toilets during the day.

During oral assessments the invigilator will knock on the door of the assessment room one minute before the scheduled time allocated for the assessment, giving candidates a one-minute notice to complete.

Kaplan confirmed that on two occasions a small number of candidates were affected by an error when an invigilator knocked on the door too early. All such cases were raised as an issue by the relevant Chief Marshal and, where appropriate, the candidates affected were considered as part of the mitigating circumstances process. I recommend that should such an error happen again, or indeed any error or issue that potentially affects candidate performance, that the candidate be requested to complete the mitigating circumstances application process. This will ensure each individual candidate's evidence is collected, so there is clarity as to how they felt they were impacted. It is also critical that the additional measures, as planned, to prevent a potential recurrence of this error of knocking on the door too early, are implemented.

Very soon after completing all the SQE2 assessments all candidates were invited to complete an online survey, in order to share their experiences of booking and sitting the exams. Of the 738 candidates who sat, 347 completed the survey of which 208 provided additional comments and 30 left contact details. All 30 candidates who left contact details were contacted individually. A similar survey was conducted after the SQE1 exams, and further surveys are planned, thus allowing Kaplan to track feedback on specific aspects of the exam experience.

Generally, good levels of satisfaction were expressed about the written and oral assessment venues as well as the website and registration process. The lowest levels of satisfaction related to the booking process, and to a lesser extent the SQE2 tasks and assessment specification.

With respect to the questions and assessment specification a minority of candidates reported finding the tasks too challenging or questioned their appropriateness for a day one solicitor. However, compared to SQE1, the SQE2 assessment tasks scored much more highly for satisfaction.

Some of the free text comments were very positive about their experiences, especially when visiting the oral assessment venues. When rating overall satisfaction with the SQE assessment service provided by Kaplan, 75% were neutral or positive.

As a result of the survey Kaplan followed up the survey findings with relevant teams and actions which require attention such as potential for improvement or to provide a response to a candidate have been recorded and taken forward. A summary report of all findings and follow up activities has been then sent to the SRA.

Kaplan has a good process for post exam candidate feedback in place and pay attention to detailed comments received.

Immediately before the pre-assessment board meeting, the mitigating circumstances panel met to consider all claims received from candidates and I observed this meeting. The panel meets to consider all candidate claims which cite an issue regarding one or more of the following:

• Mistake or irregularity in the administration or conduct of the exam
• Evidence of bias in the conduct of the exam
• Illness or other personal circumstance beyond their reasonable control

One candidate submitted a claim citing every point above. All other candidates cited either point 1 or point 3.   

The following outcomes are possible:

• no action, claim rejected
• claim upheld and the SQE2 exam costs refunded and the sitting discounted (as one of the, up to, three allowed SQE2 attempts).

Given the complexity of the assessments, it is not surprising that a range of issues can be reported to be affecting candidate's performance on the day of assessment. Where a known error or issue had arisen, such as where an invigilator knocked on the door of an oral exam too early, this was fully examined using all the evidence available from the Chief Marshal, other Kaplan staff and, where appropriate, the candidate.

The panel undertook a thorough and exhaustive analysis of all the evidence available for each application, taking care to ensure consistency in their decision making where circumstances raised were the same or similar across different applications. As this was the first occasion that mitigating circumstances were considered for SQE2 exams, the panel were careful to ensure guiding principles were both in line with the published policy and could be maintained consistently in future SQE assessment windows, should similar circumstances arise.

Overall, I was impressed by the meticulous way in which each individual application was considered. The panel were mindful of precedents being set, taking a balanced view and carefully documenting the outcome decisions. During discussions it became clear that a small number of candidates did not complete a mitigating circumstances application because they were informed by an invigilator or Marshal that the issue would be raised on their behalf.

I recommend that all Kaplan staff, and those acting on behalf of Kaplan at assessment centres, be reminded that if any candidate raises an issue or concern on the day, they should be reminded to consider if the mitigating circumstance process should be completed by them.

The pass mark and pass rate for SQE2 is determined at the Assessment Board.

Two standard setting processes were considered to help inform the pass mark and consequent pass rate recommendations to the Assessment Board. These were the borderline regression and Hofstee processes and both have been used in similar or equivalent high stakes exams assessing professional competence in the UK and globally. These activities were completed before the pre-assessment board meeting was held.

The Hofstee panel was convened with the objectives of a review of the minimum competence standard and collecting the judgements for standard setting from panel members (ie highest and lowest acceptable potential pass mark and pass rate, based on the knowledge of the cohort). The outcomes of the Hofstee panel provided another useful reference point when recommending the pass mark and pass rate to the Assessment Board.

Prior to the Assessment Board two pre-assessment board meetings were held, these meetings were attended by senior staff in Kaplan and SRA, including those with psychometric expertise. I observed both meetings. The aim was to fully consider all the psychometric evidence about how the SQE2 assessments performed as individual assessment stations, and overall, and to raise queries about the information and recommendations that would be presented to the Assessment Board, should there be any. The data and analyses demonstrated good evidence that the assessments delivered reliable and fair outcomes.

At the pre-assessment board meeting two issues received particular consideration, the high performance of the cohort and the difference in outcomes by ethnicity. With regard to different achievement levels by ethnicity, as was the case with SQE1, analysis suggested that differential attainment by candidate groups was confounded by educational attainment. It was also noted candidates who did well on SQE2 had typically done well on SQE1.

With regard to the performance of the cohort it was noted that the self-declared personal data from the candidates suggested there were a significant proportion of candidates who had completed relevant work experience ie qualified lawyers who had previously completed the QLTS multiple choice test, or candidates who have been working in the legal profession for some time and have not been in a position to qualify under previous routes. (One caveat in the self-declared personal data is a significant minority of candidates 'preferred not to say' on this point). It was also noted that this cohort, uniquely, did not contain any resitting candidates. It is likely that both factors contributed positively to the performance of this cohort.

The pre-assessment board meeting attendees were interested in confirming that the outcomes of the assessment were fair and that any recommendations made to the Assessment Board protected the public interest so that those passing would be fit to practise. This was evidenced by the discussion about the assessment outcomes and standard error of measurement. The discussion and recommendation carefully referenced and followed the published SQE Marking and Standard Setting policy.

The Assessment Board was held on 11 August, it is chaired by SRA's Chief Executive and I observed the meeting. As required of this Board the pass mark and pass rate were determined as were decisions about the outcome of mitigating circumstances claims, malpractice cases and SQE1 appeals .

A thorough discussion of the performance of SQE2 candidates, taking account of the detailed psychometric analyses, took place. Senior staff working on behalf of Kaplan and SRA with relevant expertise provided insightful observation and evidence of how the cohort had performed.

After proper consideration the pass mark and therefore the pass rate recommended by the pre-assessment board meeting were adopted. I was impressed by the quality of the discussion and was confident the outcomes were in line with the SQE Marking and Standard Setting policy and the requirements of the Assessment Board meeting.