INTRODUCTION The Canadian Association of Medical Radiation Technologists (CAMRT) is the certifying organization for Advanced Practice Registered Technologist (Radiation Therapy) (“APRT(T)”) in Canada. The certification process provides a three-phase standardized mechanism to certify practitioners in this emerging role. Following the successful completion of the first two stages – prior learning assessment portfolio and case log – the candidate is eligible to challenge a competency-based oral examination (CBOE). The objective of the CBOE is to evaluate the candidate's competence in solving clinically relevant problems requiring decision-making in their area of specialization in a live, virtual examination. The exam panel is comprised of 4 trained expert examiners (2 radiation oncologists, 1 medical physicist, 1 APRT(T)/expert therapist) and 1 exam moderator. Competence is assessed at a level defined by the standards outlined in the CAMRT APRT(T) Competency Profile. PROGRAM EVALUATION The CAMRT conducts rigorous evaluation to ensure that the CBOE is justifiable and defensible. Since 2017 the CBOE has been administered 15 times with each iteration being evaluated using 1) examination statistics (candidate performance, time spent and examiner scoring concordance, etc) and 2) qualitative methods (examiner and candidate interview post CBOE). Data was collected in an iterative process to inform ongoing exam and process refinement to ensure the reliability and validity of the exam against sound assessment principles and the standards set for APRT(T) practice in Canada. Results are reviewed by the Exam Preparation Committee and recommendations made to address any issues or deficiencies identified. The following modifications have been recommended and implemented: 1) Professional Practice content - Shorten the number of questions, divide the questions delivering half at the start of the exam and half at the end 2) Exam materials - Provide hard copy versions of supporting documentation (patient history, diagnostic imaging and reports, treatment plans) to give candidates the option of reviewing images, data, etc. on monitor or on hard copy; also provides back up in the event of a technological problem 3) ‘Acceptable’ probes - for each question, specific statements or questions have been added that can be used by examiners to guide the candidate down a certain line of thinking that might not have been explored in their primary answer, without overt prompting 4) Score review - Schedule immediately following an exam instead of after the scores have been submitted to the central office. This provides an opportunity to review and resolve large item inter-rater variability while the experience is fresh in the examiners mind. 5) Process for second attempt – Candidates who did not pass on their first try are now only required to complete the portion of the exam where they were unsuccessful, not sit the entire exam again CONCLUSIONS The CBOE model is an established mechanism for assessment, and provides a meaningful capstone to the Canadian APRT(T) certification process. Approached with rigour, it is nonetheless new territory for the CAMRT and many of its stakeholders, and ongoing efforts will be made to balance the need for consistent, valid, and reliable processes with opportunities for refinement.