ETS research finds college students fall short in
demonstrating ICT literacy:

National Policy Council to create national standards

January 2007

by Irvin R. Katz
Educational Testing Service (ETS) released preliminary research that is consistent with a suspicion long held by those in academia: while students may be tech savvy when it comes to entertainment, they may not have the critical thinking skills to perform the kinds of information management and research tasks necessary for academic success. The research, gathered from 6,300 students who took ETS’s ICT Literacy Assessment in 2006, suggests that many of the students who participated lack some of the information and communication technology (ICT) literacy skills expected for college-level work: on average, students earned only about half the points that they could have earned on the test.

College faculty and administrators help develop the assessment
ETS developed the ICT Literacy Assessment under the guidance of college librarians, faculty, and administrators, who recognized a gap between students’ ICT literacy and the level of ICT skills that colleges and companies expected them to have, but were not able to create a reliable, valid, cost-effective assessment to measure the breadth or depth of the problem.

Working from the ACRL “Information literacy competency standards for higher education” and other key documents, our higher education partners identified seven ICT literacy skill areas that the assessment should measure: defining, accessing, managing, evaluating, integrating, creating, and communicating information.

Based on these skill areas and related performance indicators, ETS created an online environment in which students interact with simulated software to solve information problems, such as searching a library database, integrating information in a spreadsheet, and judging the reliability of information found in an Internet search. The assessment measures students’ critical thinking skills—not their knowledge of technology—but students must be able to use both the tools (such as a mouse) and generic applications (such as a word processing program) in order to complete the tasks.

Two assessments serve colleges and high schools
ETS currently offers two levels of the ICT Literacy Assessment: Core and Advanced. Each measures the same ICT literacy construct covering the seven skill areas, and each consists of 14 four-minute tasks that target one skill area each and one 15-minute task that targets two skill areas. The ICT Literacy Assessment–Advanced Level targets students completing a two-year degree or technical program or rising juniors at a four-year institution. The Core Level assessment (designed to be easier) targets high school seniors and students in their first year of post-secondary education.

The preliminary findings were compiled from college students and high school seniors who took either the Core or the Advanced level of the test.

Findings show a need for improvement
By spring 2006, more than 6,300 students at 63 institutions had taken the ICT Literacy Assessment, with institutions selecting students to test based on the institution’s assessment goals. Some chose to test students enrolled in a particular course, some recruited a random sample, and some issued an open invitation and offered gift certificates or other incentives.

Because the data is not representative of all U.S. institutions or all higher education students, ETS urges caution in using these results to generalize to the greater population of college-age students. Even so, the preliminary results are rather revealing about the ICT literacy skills of participating students.

During a task in which students evaluated a set of Web sites:
   • Only 52 percent judged the objectivity of the sites correctly
   • 65 percent judged the authority correctly
   • 72 percent judged the timeliness correctly
   • Overall, only 49 percent of test-takers identified the one Web site that met all criteria

When selecting a research statement for a class assignment:
   • Only 44 percent identified a statement that captured the demands of the assignment
   • 48 percent picked a reasonable, but too broad, statement
   • 8 percent picked statements that did not address the assignment

When asked to narrow an overly broad search:
   • Only 35 percent selected the correct revision
   • 35 percent selected a revision that only marginally narrowed the search results

Other results suggest that these students’ ICT literacy need further development:
   • In a Web search task, only 40 percent entered multiple search terms to narrow the results
   • When constructing a presentation slide designed to persuade, only 12 percent used only those points directly related to the argument
   • Only a few test takers accurately adapted existing material for a new audience
   • When searching a large database, only 50 percent of test takers used a strategy that minimized irrelevant results

Of course, there were some positive findings, as well. For example, test takers appeared to recognize that .edu and .gov sites are less likely to contain biased material than .com sites. Eighty percent of test takers correctly completed an organizational chart based on e-mailed personnel information. Most test takers correctly categorized e-mails and files into folders. And when presented with an unclear assignment, 70 percent of test takers selected the best question to help clarify the assignment.

Colleges use findings to guide decisions
Participating institutions are using the preliminary results to help stakeholders prioritize their educational goals for the ICT literacy of their student population—in addition to helping guide the instruction of the individuals who took the test.

Alexius Macklin, associate professor of library science at Purdue University, said that the preliminary research findings illustrate that most students do not have the ICT literacy skills needed to complete college-level assignments efficiently. “The reality is that when you give students a research assignment, they go straight to Google without any thought to their actual research question or the information need,” Macklin says. “They draw information from questionable resources because they don’t know the difference between information they find from an ad or a biased source, and that which they find on an authoritative, timely, objective site. The preliminary research from ETS shows us that a majority of our students are not ICT literate enough to succeed academically . . . they do not currently have the skills to analyze and synthesize information into something manageable and useful for their needs.”

While the initial snapshot of students’ ICT literacy skills is bleak, Macklin says, the good news is that it is possible to teach these skills. “The preliminary findings from ETS show us that institutions need to consider how to better integrate ICT literacy skills into and across the curricula. It may require initiating an ICT literacy initiative or allocating resources differently. It’s important to help our students better evaluate, manage and communicate information so that they can succeed in school, at work and in life. And now we know that the results are measurable.”

As more institutions administer the assessment, further research will clarify ICT literacy performance at individual institutions and across the United States.

National Policy Council to set benchmarks
In October, the National Forum on Information Literacy announced the formation of the National ICT Literacy Policy Council, which is charged with providing leadership in creating national standards for ICT literacy. Lizabeth (Betsy) Wilson, dean of university libraries, University of Washington, and former ACRL president, is one of the leaders who has committed to serving on this council. The council’s chief work will be to review current ICT literacy and information literacy assessments and standards documents, determine the number of assessment levels, name those levels (such as advanced proficiency, acceptable proficiency, and minimal proficiency), and provide descriptions for each level. In short, the council will determine what students should know and be able to achieve at each ICT literacy level.

The council will also recruit panels of experts to review the ETS ICT Literacy Assessment to recommend cut points for each of the performance levels. These standards should help educators determine which students meet the ICT literacy standards and which may need additional instruction or remediation. The council will review these recommended cut points to modify and/or accept them as appropriately reflecting national ICT literacy standards.

As a first step toward assessing ICT literacy skills nationally, the preliminary findings provide information to colleges and high schools about the ICT literacy of the students who participated, as well as to individual students about their own performance. With more data and further research, ETS hopes to continue to support the educational community dedicated to improving the ICT literacy of students.

ICT Literacy Assessment resources on ETS.org

Homepage: www.ets.org/ictliteracy
Preliminary findings: www.ets.org/ictliteracy/prelimfindings.html

About the Author
Irvin R. Katz is senior research scientist for the Educational Testing Service, e-mail: ikatz@ets.org

© 2007 Irvin R. Katz