Online assessment research

Schools embrace tailored tests to suit students' talents

Could the one-size-fits-all test become a thing of the past? A growing number of schools are moving away from standardised testing, instead using assessments tailored to students' abilities.

Read the full article (The Age, 31 May 2015)

Research and trialling

ACARA is conducting a broad research program to inform work towards the transition from paper-based to computer-based assessments. The purpose of the research is to provide evidence-based information to education ministers and the broader education community about the possibility of delivering the NAPLAN in a computer-based, or ‘online’, environment.

The research program consists of three elements: a pilot research study, a trialling study and a development study. Data from the trialling study is now being analysed and the development study is underway.

Research findings will be made available progressively.
For more information about ACARA’s research into online assessment, contact [email protected]

Development study

In August 2014 ACARA will conduct a development study to further trial the tailored test design in a sample of schools. Tailored tests provide a multi-branching test consisting of interlocking sets of questions (referred to as ‘testlets’). The development study has been informed by the pilot research study and trialling study, and will finalise the targeting and measurement aspects of the tailored test design.

The development study will also investigate how testlets are trialled for the new design and how the testlets are used to construct NAPLAN online tests. Fully branching tailored tests in reading and numeracy will be trialled as well as new options for language conventions.

Research will also continue into the delivery of technically enhanced items and adjustments for students with disability.

Previous research studies

Pilot research study

From September to November 2012, ACARA conducted a pilot research study into online assessment. Approximately 125 schools across Australia participated in the study. The study used literacy and numeracy questions from NAPLAN, and questions from NAP sample assessments in civics and citizenship (NAP–CC).

The pilot research study comprised both a mode effect study and a cognitive interview study:

  • The mode effect study investigated how the performance of items (questions) is affected by the mode in which the test is delivered; that is, whether items perform differently in paper-based and computer-based tests.

  • In some schools, researchers conducted cognitive interviews with students. This was a small qualitative study intended to gauge levels of student engagement with the computer-based assessments.

Trialling study

In August and September 2013, ACARA trialled online delivery of the tailored test design. Students answered an initial set of questions and were then directed to the subsequent sets of questions based on the accuracy of their responses. Students with a high number of questions correct were directed to more challenging questions. Students who had a lower level of achievement in the initial set of questions were directed to questions that were less challenging. Preliminary results show that the tailored test provides better assessment of individual student achievement relative to ability.

More than 250 schools across Australia volunteered to participate in the study. The study included remote schools as well as a trial with a small number of home-based and geographically remote students. Approximately 23 000 tests were delivered to students in Years 3, 5, 7 and 9 in reading, numeracy and writing. The research study also investigated the cognitive and behavioural engagement of students with the tailored test and the effectiveness of the proposed test design in better targeting underperforming students. ACARA is now analysing the test data and a range of other information collected, including observations by invigilators and technical issues experienced in schools.

ACARA’s research has also investigated autoscoring technology and technically enhanced items that are designed to assess particular aspects of literacy and numeracy aligned with the Australian Curriculum.

Further information on the proposed move to delivering NAPLAN online can be found on Online assessment page of the NAP website.