Assessing the assessments: Development of a tool to evaluate assessment items in chemistry according to learning outcomes

Abstract

Higher education in Australia is in a phase of rapid change due to significant regulatory changes, with new standards currently being implemented for registration of institutions and accreditation of degrees. Over the past five years the Australian chemistry community has come to a consensus on common Chemistry Threshold Learning Outcomes (CTLOs) that every Bachelor level chemistry graduate from an Australian university will have attained. The CTLOs will inform the standards used to accredit institutions and degrees. Building upon this, the Royal Australian Chemical Institute (RACI), the professional body for chemists in Australia, has changed its accreditation process for chemistry degree programs and now uses these CTLOs as the basis for accreditation. Therefore, it is paramount to ensure that assessment items used allow students to demonstrate attainment of the CTLOs for a chemistry major. The “Assessing the Assessments” project has used an iterative process to develop an evaluation framework to assist academic staff at tertiary institutions to determine the alignment of their assessment items with the CTLOs. In conjunction with professional development workshops in which colleagues explore the alignment of assessment items with the CTLOs, a sophisticated tool has been developed which can be used to evaluate assessment items. The tool yields ratings for both engagement with and assessment of each CTLO within the assessment task evaluated, highlighting areas of potential improvement in current assessment practices. Comparison of self-evaluations of tasks submitted to the project by academic staff with evaluations conducted by the project team shows that in the majority of cases, faculty over-estimate the ability of their assessment items to confirm achievement of CTLOs. Recommendations to increase the coverage of CTLOs through changes to assessment procedures are presented. Through the development of the framework, difficulties with interpretation and application of some of the CTLOs have been elucidated.