The Test Agency

The following provides detailed background information regarding the process that was undertaken in the development of the professional certification examinations.

Assessment Strategies Inc. (ASI) of Ottawa has been a leading provider of customized assessment and measurement services in Canada for over 30 years. They specialize in designing, developing and administering credentialing and continuing competence assessment programs, survey instruments, and occupational and competency research.

The Standards for Educational and Psychological Testing provide a foundation for ASI's test development and administration activities which are supplemented with the Guidelines for Educational and Psychological Testing and the NOCA Certification Handbook. In addition, the International Standards Organization (ISO) has developed an international standard for professional certifications, ISO/IEC 17024 Standard, and ASI reviewed the methods adopted by NIDMAR for the development and maintenance of the certification examinations and determined that they meet and in many cases exceed the ISO Standard.

The passing score is determined for each administration of the examinations based on the complexity of the examination. It is possible that one test form might be slightly easier than another and in order to remain fair to all those who sit the examinations, the pass mark may vary from one administration of the examination to another. This is the standard method for setting pass marks on professional certification and licensure examinations. To date, the passing score has ranged between 75% and 82%, and approximately 80% of test writers have passed the examination. The process is designed to produce an outcome that is valid, reliable and defensible.

ASI has the experience and demonstrated competence to develop and deliver high stakes assessment tools that are tailored to each particular field. Its client base includes a range of provincial, national and international clients such as the Board of Canadian Registered Safety Professionals, Canadian Association of Occupational Therapists, Canadian Council of Professional Engineers, Canadian Nursing Association, Canadian Physiotherapy Association, Canadian Tourism Human Resource Council, Certified General Accountants of Canada, Financial Planners Standards Council, Retail Council of Canada, and Royal Canadian Mounted Police.

Overview of the Test Development and Administration Process

To ensure that the certification examinations were developed to achieve their stated purpose, a rigorous test development process was implemented according to professional standards as specified by the Guidelines for Educational and Psychological Testing (CPA, 1986), the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999) as well as the principles supported by the National Organization for Competency Assurance (NOCA, 1996). This test development process is summarized below.

Examination Committees
To ensure that the examinations reflect professional practices, Examination Committees were established whose primary responsibility was to oversee and participate in test activities related to the examination development. Members of the Examination Committees were disability management / return to work professionals and practitioners representing business, labour, private and public insurance and private service providers from all across Canada.

Competency Profile Finalization
A formal review was held of the essential skills and core competencies identified in the Occupational Standards in Disability Management.

Blueprint Development
Examination blueprints outlining the content to be tested were developed by the Examination Committees. A blueprint includes the competencies (i.e. the content domain) that form the basis for test development. It also specifies variables that provide structure for the examination, as well as guidelines and specifications for weighting the competencies to ensure that the examination accurately reflects the domain of disability management / return to work practice.

Item Development
Multiple choice examination items were developed by trained subject matter experts consisting of disability management / return to work professionals and practitioners representing business, labour, private and public insurance, private service providers and representing all regions of Canada. The examination items measure the specified competencies in accordance with the guidelines identified in the examination blueprint. Item writing session participants were trained by an expert in the field of test development on how to develop effective four-option multiple choice items. After a comprehensive training and review of the principle elements of item construction, participants developed multiple choice items under the guidance of the testing expert. To demonstrate the appropriateness of an item, each item was referenced by one current disability management practice textbook.

Editorial Review
All test items were submitted to an editorial review to ensure consistency of language, terminology and grammar.

Item Review
An independent review took place of all items by experts in the field as well as during a piloting of the examinations.

Test Form Development and Validation
The multiple choice examinations were compiled from items in the item bank in accordance with the blueprint specifications. Final approval of the multiple choice examinations was given after the Examination Committees reviewed the examinations.

Standard Setting
The Examination Committees established the standards using the modified Angoff procedure. This is a professionally accepted and widely used method for setting the standard for criterion-referenced examinations.

Translation
Upon finalization of the English version of the examinations, they were translated by an accredited professional translator and reviewed by a second accredited professional translator whose primary responsibility includes validating translated examinations. The translation was further validated by bilingual subject matter experts.

To date the process as outlined above has included the involvement of over 80 professional practitioners in the field of disability management / return to work representing all stakeholder groups from across Canada who were trained in the requirements of the blueprint development, item generation, item review, editorial review, standard setting, test validation, etc. As the process is a continuing one, the numbers of individuals who will continue to be involved in the development of these certification examinations will continue to grow.

Field Test
The certification examinations consist of 300 multiple-choice questions presented as two separate test booklets. A 3.5-hour testing period is allotted to complete each test booklet. The multiple choice examinations were administered in March 2003 after which information will be collected, both qualitative and quantitative, and will be analyzed as appropriate.