Increasingly, students are using the Internet to obtain information about both general and academic topics. Along with this trend there is a growing concern about the dubious nature of online information, and users’ ability to validate or evaluate this information. Research shows that students are frequently deceived when viewing online content and are not able to judge the validity of a website.

In a previous post, I shared insight into developing a criteria for evaluation of online information. This criteria provides an opportunity to identify and evaluate the credibility and relevance of online information. The challenge is that individuals are not able to accurately judge the validity of a website, even when given explicit procedures to do so. Assessments of critical online information literacies provide guidance as we explore these spaces. In the development and implementation of assessments, I believe that we can more granularly examine the individual elements that exist as we critical evaluate information online.

This will post will share some of my earlier research on critical evaluation of online information as well as the instrument I developed to measure these skills. I’ll share some of the insight and tools I employed as I created, validated, this instrument. This instrument has been utilized in classrooms from K up through higher education. I’m sharing this to help you think about assessments you can develop in your own classroom, focusing on your own content.

What are we measuring?

When we create a test, or an assessment, we’re actively trying to measure specific attributes or characteristics of a person. In educational psychology, we call these attributes constructs. Constructs are mental models or explanations that we’ve created to explain human behaviors. Keep in mind that measuring physical attributes (e.g., height, weight) is often much more difficult that measuring mental attributes or characteristics (e.g., motivation, creativity, social skills). That is why it’s extremely important to be as granular and specific as to exactly what you’re measuring. We also want to base the measurements of these constructs on theory and research.

To address these concerns, I’ll share the ways in which I’ve defined critical evaluation of online information, and the literature I’ve used to frame my thinking. This gets quite detailed, but I want you to think through all of the discrete skills we utilize as we critical evaluate online information. I also included some of the original citations. You can read the full literature review and details of this work in my dissertation.

Credibility. Credibility was defined in terms of expertise and trustworthiness (Judd, Farrow, & Tims, 2006) or the reliability of information (Kiili et al., 2008). A review of the literature suggested the following skills are necessary in making a credibility judgment: evaluations of author, purpose, source, content, argument, and accuracy of online information.

• Evaluation of author is defined as the creator or source of the information showing evidence of being knowledgeable, reliable, and truthful (Harris, 1997).
• Evaluation of purpose is defined as consideration by the reader to determine the desired intent of the information (Harris, 1997).
• Evaluation of source is defined as consideration of the provider of the information and whether they are able to answer the desired question (Rieh & Belkin, 1998; Strømsø & Bråten, 2010).
• Evaluation of content is defined as consideration of the style or manner in which information is presented (Harris, 1997; Kiili, Laurinen, & Marttunen, 2006).
• Evaluation of argument is defined as consideration of evidence provided by the author referencing claims made (Kiili, Laurinen, & Marttunen, 2006).
• Evaluation of Accuracy is defined as consideration of the vetted, reliable and error-free nature of the information (Meola, 2004).

Relevance. Relevance is defined in terms of the importance and currency (Judd, Farrow, & Tims, 2006), or judgments about the essential nature of information in relation to the task (Kiili, et al., 2008). A review of the literature suggested the following skills are necessary for making a relevancy judgment: relevance of a topic, relevance of a website, usability of a website, and currency of information (Tate & Alexander, 1998; Kiili et al.,; Rieh & Belkin, 1998).

• Relevance of topic is defined as consideration of the information as being essential to the student’s task (Kiili, Laurinen, & Marttunen, 2008).
• Relevance of website is defined as consideration of the essential nature of the website as a source of information in relation to other sources of online information (Kiili, Laurinen, & Marttunen, 2008).
• Usability is defined as consideration of the mode or medium of information presented as it relates to task (Kiili, Laurinen, & Marttunen, 2008).
• Currency of information is defined as consideration of the information as being circulating and valid at the present time (Meola, 2004; Kiili, Laurinen, & Marttunen, 2008).

How did we measure it?

To measure these constructs, I created an instrument with a colleague (Greg McVerry) called the Critical Online Information Literacies (COIL) instrument. The assessment was pilot tested in the pilot of my dissertation, my dissertation, and Greg’s dissertation.

The development of the COIL consisted of writing and revising several items for each of the constructs, and subconstructs listed above. These items were reviewed by an expert panel, and then tested and revised in each of our subsequent studies. This means that for each of the above bullet points, you need to write a question or two that focuses on the skills you’ve identified. As an example, I’ve included some of the better questions we wrote for identifying credibility and relevance as we critically evaluate online information.

Credibility items

• Which website uses strong words, phrases, and images to influence readers?
• Which website has the most reliable details to support the argument that a bird’s bones are hollow?
• Which website uses the best details to support the claim Pluto is not a planet?
• Which website uses information from the most reliable source?
• Where do you click to learn more about an author?
• What is the main argument?
• Who is the main audience of this website?
• What section of the website should you read to learn about parts of blood?

Relevance items

• Where do you look on a website to find out when it was written?
• Given this website’s “About” page, what is the expertise of the author of this site?
• Which website tries to influence the audience with strong words or images?
• Which website has information that tries to disprove the claim: Zoos are cruel?
• Which website uses information from the least reliable source?

To create the actual assessment, we took screencaptures (static images) of webpages and saved these images to pages on a Google Site. Some of the items included images from multiple websites. Some of the items included one image, with annotations added into the image to designate a specific part of the webpage. Students indicated their responses to the individual items on a separate piece of paper. You can review all of the items, and use the original version of the COIL in your classroom by clicking here.

How will you measure it?

After you have worked with learners to develop a criteria for critically evaluating online information, identify opportunities to assess their evaluation of this content. Take examples of text from a variety of sources and have students consider questions like the ones provided above. Your sources should include traditional, print sources such as magazines, newspapers, and textbooks, as well as online sources (e.g., websites, images, videos, audio podcasts). With each text, have students consider the credibility and or relevance of the materials presented by identifying specific content on the page.

To make the learning more authentic, I would use texts from your content or instructional area. Develop a series of questions surrounding these texts and either embed them into the texts using screencaptures as I’ve detailed up above, or simply have students review websites and leave feedback elsewhere. Google Forms now allows for integration of images and videos into quizzes and tests you can create for students. In recent iterations of this assessment of critical online information literacies, I’ve utilized tools like Hypothesis to scaffold discussion as I read online with students.

Get started by using different forms of text in your classroom. This is important in classrooms from Pre-K up through higher education. As students interact with these texts, ask them questions such as the ones suggested above. In this line of questioning, you’re also teaching them how to read from your perspective, and as a member of your content area or discipline. For example, how would a scientist read this source as opposed to a historian. By teaching and assessing these skills and practices, we help prepare students as they critically examine future online information sources.

 

Please be sure to subscribe to my weekly newsletter if you’re interested in more content just like this.

Image credit