FEATURE ARTICLE Investigating Criteria That Seventh Graders Use to Evaluate the Quality of Online Information Julie Coiro, Carla Coscarelli, Cheryl Maykel, & Elena Forzani This article provides a qualitative analysis of seventh graders’ critical evaluation of online information. Findings elaborate on the unfounded or otherwise superficial criteria that many students used to justify their responses. I know how to evaluate sources while constructing n today’s world, a well-prepared career- and meaning from online text. college-ready learner is often depicted as some- Previous research outlines the shortcomings of one who can actively and independently con- high school and college students when evaluating struct knowledge by collecting and judging the Internet sources (e.g., Goldman et al., 2012; Kiili, quality of information, integrate this information with Laurinen, & Marttunen, 2008; S. Zhang & Duke, prior knowledge, and act on this knowledge in cre- 2008), but only a few studies have focused on adoles- ative ways. These processes are also important dimen- cent learners’ evaluation practices. These studies sug- sions of online research. gest that adolescents rarely question the accuracy of Effective online research relies on a reader’s abil- information (e.g., Barzilai & Zohar, 2012; Walraven, ity to successfully evaluate the accuracy and reliabil- Brand-Gruwel, & Boshuizen, 2009; M. Zhang, 2013). ity of the information that he or she finds (Goldman, Younger adolescents, in particular, have a tendency Braasch, Wiley, Graesser, & Brodowinska, 2012). to equate information quantity with information Unfortunately, students rarely have the training to quality (Agosto, 2002; Wallace, Kupperman, Krajcik, & Soloway, 2000). These findings are especially troubling, given the results of a recent survey that found that 60% of 2,000 middle and high school teachers believe that today’s technologies make it harder for students to find cred- Authors (left to right) ible sources online (Purcell et al., 2012). Additionally, Julie Coiro is an associate professor at the University of Rhode Island, young teens are vulnerable to overestimating their Kingston, USA; e-mail jcoiro@uri.edu. online critical reading abilities (Flanagin & Metzger, Carla Coscarelli is an associate professor at Universidade Federal de Minas Gerais, Belo Horizonte, Brazil; e-mail cvcosc@yahoo.com.br. 2010; Miller & Bartlett, 2012), which is likely to exac- Cheryl Maykel is a doctoral candidate in school psychology at the erbate the problem. If we wish to prepare all students University of Connecticut, Storrs, USA; e-mail cherylmaykel@gmail.com. to deal with the challenges of online research, we Elena Forzani is a doctoral student in educational psychology at the need a richer set of insights into the specific qualities University of Connecticut, Storrs, USA; e-mail elenaforzani@gmail.com. of evidence that students use to judge online 1 Journal of Adolescent & Adult Literacy xx(x) xx 2015 doi:10.1002/jaal.448 © 2015 International Literacy Association (pp. 1–11)  FEATURE ARTICLE information, and common misconceptions that mid- over and above those required when reading printed dle school students have that might be easily eradi- books. Previous work suggests that traditional read- cated with explicit instruction about more appropriate ing skills are necessary but not sufficient to learn criteria. from information on the Internet (Afflerbach & This study sought to provide empirical data on Cho, 2009; Coiro, 2011; Coiro & Dobler, 2007). the specific types of evidence that seventh-grade stu- Consequently, we need to expand our understand- dents use to judge the quality of online information. ing of how reading comprehension is defined and Qualitative methods were employed to answer two enacted in ways that encompass new understand- research questions: ings about digital readers, online texts, new activi- ties, and new social contexts found in online spaces 1. What criteria do seventh graders use when (Coiro, 2003). asked to evaluate website sources and informa- In particular, the ability to read and evaluate the tion as part of an online reading assignment? level of accuracy, reliability, and bias of online infor- 2. What common patterns of evidence do seventh mation presents challenges that are different from graders use to justify their reasoning about a traditional print sources. The content of online in- website author’s level of expertise, his or her formation is even more diverse and commercially point of view, and the overall trustworthiness biased, and novel techniques are required to assess of information provided? information credibility (Flanagin & Metzger, 2010; Thoman & Jolls, 2005). For instance, promotional Answers to these questions can further enrich efforts and related advertising may be more difficult interpretations of previous research suggesting that to differentiate on the Internet than in print and adolescents asked to do online research struggle other mass media forms (Fabos, 2008). Others (Britt more with critical evaluation tasks compared with & Gabrys, 2001; Rouet, Ros, Goumi, Macedo- tasks requiring them to locate, synthesize, and com- Rouet, & Dinet, 2011) cite the lack of uniform stan- municate information about their research (see dards and cues regarding document type in online Forzani & Burlingame, 2012). This research also text environments as necessitating a renewed interest found significant differences in difficulty among in how students critically evaluate online informa- four types of critical evaluation tasks: identifying the tion. Moreover, students may know about strategies author was the easiest, followed by evaluating the for evaluating online sources but may not put this author’s point of view, evaluating the author’s exper- knowledge into action, even when asked to do so tise, and evaluating a website’s overall reliability. (Hogan & Vernhagen, 2012). The current study These findings offer initial insights into which as- more closely examines the nature of online evalua- pects might be more and less challenging for sev- tion processes among middle school students when XX (X) XX 2015 enth graders. The current study further enriches prompted as part of an online informational reading these interpretations with a better understanding of task. criteria that middle school students use when asked to critically evaluate online sources. This can help Methodology J OURN AL O F A DOL E SCE NT & A DULT L ITE RACY teachers design instructional practices that appropri- ately challenge and support adolescents engaged in This study took place within a larger, multiyear re- evaluating online information. search project designed to develop valid and reli- able assessments of online reading comprehension (see Leu, Kulikowich, Sedransk, & Coiro, 2009– Theoretical Framework 2014). Participants included 773 seventh-grade stu- This study was informed by a new literacies perspec- dents from classrooms in a stratified random tive of online reading comprehension (Leu, Kinzer, sample involving schools in 42 districts from two Coiro, Castek, & Henry, 2013). This perspective states in Northeastern United States. Students rep- frames online reading as a problem-based inquiry resented a diverse range of ethnic and socioeco- process involving new practices for locating, evalu- nomic backgrounds and achievement levels on ating, synthesizing, and communicating informa- state reading comprehension assessments. Seventh tion. Accordingly, each function requires new graders were selected because there is little infor- 2 online reading comprehension skills and strategies mation about middle school students’ critical evaluation practices, except that they are not espe- • Items 2A and 2B: Is [the author] an expert on cially concerned with the credibility of online in- topic A (e.g., energy drinks) or topic B (e.g., formation (Flanagin & Metzger, 2010). This study heart health), and how do you know? will help identify more specific trends in their un- • Items 3A and 3B: What is the author’s point of derstanding and misconceptions about online view? How does the author’s point of view af- sources. fect the words and images that are used at this Data sources included students’ open-constructed site? responses on two subsets of four evaluation items taken from a larger set of 16 items designed to mea- • Item 4: Is the information at this website reli- sure performance in online research and comprehen- able, and how do you know? sion. Eight science-based scenarios with established For our analysis, responses to items 2 and 3 were reliability and validity of scores (see Leu et al., 2012) broken into parts A and B, respectively, so all re- were designed to focus on different issues within the sponses could be scored separately. This allowed for a domain of health and human body systems (see more precise understanding of the areas in which stu- Table 1). Each student was randomly assigned to take dents were most challenged. two of the eight assessments. (See www.orca.uconn. edu/professional-development/show-me/show-me- Data Collection overview for complete descriptions and video exam- Students completed two 16-item assessments, each ples of two scenarios and all 16 items.) on one of two school days. Students were given as The four critical evaluation items analyzed for much time as they needed to complete each 16-item the present study were designed to measure students’ assessment, although some students finished in as lit- ability to determine the author of a given website, the tle as 35 minutes, and the average time was approxi- author’s level of expertise, the author’s point of view, mately 50 minutes. In the first sections of the and the overall reliability of the website. Specific assessment, students were asked to use a search en- wording for each was as follows: gine to locate two to four websites with information Investigating Criteria That Seventh Graders Use to Evaluate the Quality of Online Information related to the scenario and summarize relevant de- • Item 1: Who is the author or creator of this tails into a digital notepad embedded into the system. website? After they synthesized their ideas across these web- sites, students clicked on a link that sent them back to a particular website visited previously. TABLE 1 Topics and Research Questions for Eight At this point, a chat box appeared in the inter- Online Reading Comprehension Assessment face, and students were prompted with the first criti- Scenarios cal evaluation item. After students typed their Topic Research question response into the chat window, the next question ap- peared until four questions had been asked and an- Energy drinks How do energy drinks affect heart swered (as shown in Figure 1). Most students spent health? between four and six minutes reviewing the website Snacks How do snacks affect heart and responding to the four questions. After the criti- health? cal evaluation task, students were asked to organize Volume level Can listening to volume levels on and share their findings in either an e-mail message an MP3 player cause hearing loss? or a posted response on a class wiki (depending on the scenario). At no point during the assessment were Cell phone How well can adults hear ringtones mosquito ringtones? students given any help or support from a person out- side the assessment system (unless there was a techni- Third-hand smoke Is third-hand smoke dangerous to cal issue). lung health? To view annotated video clips of a higher per- Asthma Can Chihuahua dogs cure asthma? forming and a lower performing student completing Contact lenses Do decorative contact lenses harm the four critical evaluation items within the actual in- your eyes? terface, visit www.orca.uconn.edu/professional-devel opment/show-me/evaluate. For the purposes of the Video games Do video games harm your eyes? present study, students’ typed constructed responses to 3  FEATURE ARTICLE FIGURE 1 Screenshot of Chat Window With Four Analysis Critical Evaluation Items (prefaced with the beaker Two researchers applied content analysis (Krippendorff, icon) and Student Responses (prefaced with the 2004) using NVivo software. The lead author trained basketball icon) the second author to understand criteria used to score evaluation items with a binary scoring protocol (1 for correct, 0 for incorrect) established in previous work (Leu et al., 2012). Then, these two authors worked together to develop a more extensive qualitative cod- ing scheme that identified patterns in the types of evaluation criteria that students used to answer each question, while also focusing closely on the nature of error patterns students made. After reviewing all responses, descriptive content categories related to each evaluation item were devel- oped and applied to students’ constructed responses for each of the six evaluation items. Codes were re- fined when necessary by creating subcodes that more specifically described each response. Then, coded er- ror patterns were consolidated through discussion to establish the most common criteria and response pat- terns for each item across all eight tasks. Once code definitions were finalized, researchers coded data with an inter-rater reliability level of 90% accuracy. The final coding scheme is available as supporting information for the online version of this article. Findings What Is the Author’s Name? Across the eight tasks, 83% of students correctly an- swered “Who is the author of this website?” Of those responses that were incorrect, approximately 4% showed that students could identify the sponsor of the XX (X) XX 2015 entire website but not the specific name of the author who crafted the information on the page. Often, in- correct responses came from scenarios that required students to navigate to another page to locate the au- J OURN AL O F A DOL E SCE NT & A DULT L ITE RACY thor’s name (e.g., click on About Us), as opposed to scenarios that only required students to search the given page. For example, a student who only explored a given webpage and failed to click on the About Us link to notice that the author’s name appeared at the bottom of that hyperlinked page responded, “There is no official author but this article comes from the MLB which stands for Major League Baseball.” Approximately 13% of responses about the au- thor’s name were completely wrong. In these cases, students identified names of people other than the only these four items were exported into a spreadsheet author; typed filler text into the response box (e.g., and then imported into NVivo to analyze qualitative random letters or numbers); and wrote, “I can’t find 4 patterns within and across responses. it,” or answered a different question altogether. Is the Author an Expert? believe that readers have no way of inferring expertise Across the eight scenarios, when asked, “Is the author if they are not explicitly told so (e.g., “I don’t know an expert on [topic], and how do you know?” only because it doesn’t say”). That is, middle school stu- 31% of students’ responses included both a clear yes/ dents may assume that it is the author’s responsibility no decision about author expertise and an appropri- to explicitly identify himself or herself as an expert (or ate reason to support their claim. Acceptable criteria not). This also suggests that some students may ac- included accurate comments about an author’s oc- cept an author’s self-proclaimed expertise as truth cupation, level of education, affiliation, knowledge, without further investigation. and/or number of years working in that field. For ex- ample, one student determined, “Yes, [he is an ex- What Is the Author’s Point of View? pert], because he is the Head of Technology.” When asked, “What is the author’s point of view, and Another explained, “No, she is not, she is just a big how does it affect the words and images that are used dog lover and was curious if chihuahuas can cure at this site?” only 20% of responses included both a asthma.” Many responses included a clear claim correct explanation of the author’s point of view and about author expertise (40%), but 10% of those re- a clear description of how it affected the words and/or sponses claimed that an author was not an expert images used. For instance, one student explained, when he or she actually was, and 3% failed to include “The words on the site are all about how energy any evidence to support the students’ claims. drinks are bad for you and there’s little evidence they are good for you.” Another suggested, “He is against Insufficient Criteria for Evaluating Author Expertise. energy drinks. He communicates this through some Across incorrect responses on this item (69%), sev- negative word choices and a slightly scary picture of a eral error patterns emerged. More than half (51%) whistle.” Accurate responses about point of view for did not reflect any particular criteria for evaluating other scenarios are available as supporting informa- author expertise; instead, some responses were irrel- tion for the online version of this article. evant, and others were vague, such as “The author knows what he is talking about.” Some vague re- Insufficient Supporting Evidence for the Author’s Investigating Criteria That Seventh Graders Use to Evaluate the Quality of Online Information sponses included text from a passage that connected Point of View. Many students (43%) were able to to a possible source but was not in any way related identify the author’s point of view, but most (80%) to the author (e.g., “It says there was research in had difficulty using details about words or images Massachusetts”). Other vague responses reflected on the website to discuss the author’s point of view. students using strategies more useful for judging Most students (79%) did not make any specific con- the quality of information rather than the level of nections to images or words that would help infer an author’s expertise (e.g., “No, because it is just point of view. For example, one student suggested, her blog,” “She gives a lot of inside info and statis- “The authors point of view is that he is against video tics”), but more information was necessary to infer games because of eyestrain. It affects it by proving any possible criteria that might have informed the his point.” Many of these connections were vague students’ responses about author expertise. (23%), such as “It’s like he’s telling you himself,” and “She’s just stating information she found.” Insufficient Reasoning About Author Expertise. Other responses were irrelevant (5%), such as “He Approximately 11% of students’ incorrect responses has probably dealt with kids before,” or were state- failed to include any reasoning about author exper- ments about not knowing the answer (5%), such as tise, and 6% appeared to equate topical relevance or “I don’t know,” and “I’m not sure.” These types of completeness with author expertise. For example, responses suggest that many students had difficulty students reasoned that an author was an expert be- with either inferring or articulating details about an cause “He knows everything about asthma,” “He author’s position on a certain topic. backed up his statement with a lot of facts,” or sim- ply “She wrote the webpage.” Confusing the Author’s Point of View With the In some cases (2%), students found the presence Narrative Point of View. Approximately 9% of re- of certain visual or textual cues to be indicative of au- sponses included phrases such as “first person” or thor expertise: “Yes, she made a chart,” or “There is a even “fourth person,” indicating that some middle copyright date at the bottom.” Another small percent- school students confused an author’s point of view age of responses (2%) suggest that some students about an issue with the narrative point of view with 5  FEATURE ARTICLE which an author might write a story. This finding, in reliability for other scenarios are available as support- particular, has important implications for students ing information for the online version of this article. who will answer items on the upcoming Common Core State Assessments that focus on determining Difficulty With Articulating Judgments About and analyzing an author’s point of view within Reliability. Most students in our sample (75%) had informational texts. difficulty with articulating their judgments about the overall reliability of information on a website. Confusing Point of View With Purpose. A third set Almost half (48%) of the incorrect responses on of responses (4%) included comments that more ac- this item suggested that students’ judgments about curately reflected an author’s purpose as opposed to website reliability were informed by inaccurate or his or her point of view. Examples of these types of generalized assumptions of Internet sources, regard- responses included comments such as “She wants to less of the site’s content. For instance, the response teach people how to be heart healthy,” and “She has “You can’t tell because anyone can put anything on a question on if Chihuahua dogs can cure asthma.” the internet” suggests that it is never wise to trust Another small percentage of students (2%) discussed information found on the Internet. Another student the author’s level of experience (e.g., “She’s been wrote, “No, the website says .com which means any- studying this for a long time”) or copied facts from one can write it,” suggesting that the type of website the text (e.g., “Energy drinks contain caffeine”) to an- is sufficient for making judgments. A third student swer questions about the author’s point of view. explained, “No it’s not, because there is no orga- These types of responses suggest that students may nization stating that it is reliable.” This response not clearly understand the differences between details suggests that some students expected to be told used to determine the author’s main points and those explicitly about the reliability of information on a used to determine the author’s point of view. website; moreover, it appears that if told that it was Understandably, these are confusing issues, especially reliable, they would automatically believe any au- for students who have had little instruction about the thor’s claim to be true. similarities and differences among these concepts. The remaining 18% of responses about author’s point Insufficient Reasoning Regarding Overall Reliability. of view included comments such as “I don’t know,” A second pattern of inaccurate responses suggested irrelevant/filler text, or a blank answer box. that many students’ (20%) decisions about reliabil- ity were informed by an understanding of reliability Is the Website Reliable? criteria (e.g., author expertise, scientific evidence, Responses on this item suggest that students experi- source corroboration) but a limited ability to clearly ence considerable difficulty with generating argu- explain their reasoning or critically engage with in- XX (X) XX 2015 ments and evidence to explain a website’s reliability. formation to locate details that could support their Across the eight tasks, when asked, “Is the information decisions. In these cases, students indicated that the at this website reliable, and how do you know?” only author was an expert but were vague in their elabo- 25% of responses included both a clear decision about rations about an author’s area or level of expertise a website’s reliability and a correct and sufficient ex- or quality of the source. Some examples include J OURN AL O F A DOL E SCE NT & A DULT L ITE RACY planation of the students’ reasoning. One student ex- “Yeah, probably, because she is pretty well known,” plained, for example, “The information may be “It says he was on a commercial,” and “All the other accurate and reliable. The information was reviewed websites have the same information.” by the medical review board, and the author is an ex- Use of Naive or Surface-Level Criteria to Evaluate pert, but it was written in 2008 so the information Reliability. Another 10% of students’ responses used might be outdated.” Accurate responses about website website characteristics such as topical relevance or textual features to explain why they determined “Students experience considerable information to be reliable. For example, some stu- dents expressed that websites were reliable if they difficulty with generating learned something from them: “Yes because it pro- arguments and evidence to explain vides me with information I never knew.” Other responses, such as “The website has a lot of sta- 6 a website’s reliability.” tistics about third-hand smoke,” and “Yes because the table has numbers in it,” suggest that some stu- performance-based informational research task. dents consider websites to be more reliable when Content analysis revealed that across four items de- they contain features such as numbers and statis- signed to measure online critical evaluation skills, tics, rather than when they present only opinions. an average of only 25% of responses from students in In these cases, however, students did not typically a large, stratified random sample applied acceptable clarify their comments with interpretations about criteria (e.g., expertise, authority, scientific evi- the actual meaningfulness of those numbers or sta- dence) and clear reasoning to judge the information tistics. Still others in this category expressed that quality. Further, our analyses revealed that 69–79% a website containing a current date or a copyright of responses from students in our sample included a symbol is automatically reliable: “Yes, it is updated range of unacceptable, vague, or otherwise superfi- daily,” and “The information is reliable because cial criteria to determine a website’s author, evalu- it has a copyright and the information looks and ate author expertise, identify the author’s point of sounds legitimate.” view, and provide reasoned evidence about a web- Other responses (10%) suggested that students site’s overall reliability. See Table 2 for a concise based their judgments about information reliability summary of the most common error patterns for solely on their experience or the author’s personal ex- each of the four items. perience. For example, one student wrote, “I have Findings from the present study are consistent asthma and that never happened to me so it’s proba- with results from other work showing that when stu- bly wrong,” and another reasoned, “Yes, [it’s reliable] dents read for information on the Internet, they are because she smoked for 26 years.” Similarly, a small more concerned with content relevance than with percentage of responses (3%) indicated that some stu- credibility; they rarely attend to source features to dents consider information to be automatically reli- evaluate reliability and author perspective; and when able or unreliable based on the type of website rather they do refer to source features in their explanations, than the quality of the claims, as revealed in com- judgments about the quality of sources are often ments such as “This information might not be reli- vague, superficial, and characterized by a lack of rea- Investigating Criteria That Seventh Graders Use to Evaluate the Quality of Online Information able because it’s a blog, not a fact website,” and “Yes soned justification (see Barzilai & Zohar, 2012; Britt this website is reliable, because it is not on a wikipe- & Aglinskas, 2002; Kiili et al., 2008). It is important to dia and it is not a student’s essay.” note that few of these previous studies were con- Overall, with respect to source reliability, stu- ducted with middle school students. Understanding dents used a range of interesting but naive or surface- that these issues are prevalent with older adolescents level criteria. Some of these criteria included the suggests that the problem is not likely to be resolved number of times information had been accessed or without explicit intervention. read, when or how often it had been updated, who sponsored the website, the domain name (e.g., .com, .edu), whether you could find similar information on Implications other sites, or how the information compared with What implications do these findings have for class- one’s own experience or knowledge. From our per- room teachers in middle school and beyond? We be- spective, one strategy by itself is not usually enough to lieve that at least five sets of instructional practices sufficiently inform a well-reasoned explanation (as will help adolescent readers focus on appropriate and was typically the case in this study). Viewed as a set of multiple evaluation criteria during online inquiry. features, though, they certainly can help inform one’s Optimally, these practices should be contextualized judgment about information reliability. However, in within secondary content area instruction so students our sample, most students only considered one (if understand how their use of these criteria can also any) of these criteria to support reasoning about a foster deeper comprehension of disciplinary texts website’s reliability. written from multiple perspectives. 1. Encourage Students to Consider Information Summary About Authors and Their Affiliations When This study sought to describe seventh-grade stu- Determining Level of Expertise dents’ ability to critically evaluate information qual- When teaching how to determine a website’s author ity on a website they visited as part of an online and his or her level of expertise, it is important to first 7  FEATURE ARTICLE TABLE 2 Summary of Patterns and Percentages of Incorrect Responses and Unacceptable Criteria That Students Used to Judge Information Quality on the Internet Percentage of incorrect Evaluation item responses Patterns of unacceptable evaluation criteria or otherwise incorrect responses Identify the website 17% • Indicated “I can’t find it” author’s name. • Indicated “I don’t know” • Provided irrelevant or filler text in the answer box • Asked the avatar if one has to do this or if she can help Determine whether the 69% • Gave the author’s job title only, with no explanation author is an expert and • Indicated that the author made a website or wrote the information explain why. • Assumed that the author just knew what he or she was talking about • Indicated that the author provided a lot of details about the topic • Provided only a fact the author wrote on the website • Provided only a vague, unclear, or inaccurate explanation State the author’s 79% • Correctly identified the author’s point of view but gave a vague explanation point of view and how of how it affected images and words on the website it affects words and • Provided an inaccurate statement about the author’s point of view images on the website. • Confused with the narrative point of view (e.g., first person) • Provided a purpose statement rather than a viewpoint • Indicated the author’s job title, position, or type of work • Indicated “I don’t know” or gave an irrelevant or no response Determine the overall 75% • Indicated that the website is reliable because they learned something from reliability of the the information website. • Indicated that the website is reliable because the information made sense to them • Indicated that the website has a current date or copyright symbol at the bottom • Indicated that the website is or is not reliable because of the type of website (e.g., blog, Wikipedia) • Indicated that the website did not say so, so they do not know • Indicated acceptable criteria (i.e., author expertise, scientific evidence, source corroboration), but the supportive evidence is vague, irrelevant, or missing XX (X) XX 2015 point out that the location of author-related informa- Time also should be spent working with students tion varies from one source to another. The author’s to tease out which biographical details suggest that an name, for example, might appear at the top of the author is an expert (or not) in that area. For instance, J OURN AL O F A DOL E SCE NT & A DULT L ITE RACY homepage for one website, be embedded into the text some adolescents can benefit from explicit instruc- on another website, and appear on a separate About tion in reading the About Us webpage to determine Us webpage on a third website. Also, students should which details relate to author expertise (e.g., level of understand the difference between an author’s name education, years of background experience) and and the organization to which the author is con- which details are less likely to inform judgments nected. An author’s work may or may not represent about the quality of an author’s work (e.g., she lives in the opinions of the sponsoring organization, so it is Austin, Texas, with her husband and three children). important to differentiate between the author and his or her affiliation to understand the biases inherent in 2. Clarify How to Elaborate on Areas an author’s claims. Then, model for students how to of Author Expertise examine the relationship between an author’s agenda It is important that students understand (and practice) and the sponsoring organization(s) to better under- how to adequately describe author expertise with clar- stand why certain ideas are included (or excluded) in ity and confidence. For example, ask a student who 8 an author’s work (see #3 below for more details). justifies his or her reasoning with a vague response such as “She works with this company” to elaborate 4. Model Strategies for Dealing With with more details about who, what company, how long Conflicting Information that person has worked for that company, in what spe- Student responses in our study also suggest that many cific area his or her work is most focused, and how adolescents are not sure how to proceed when faced work in that area connects with claims that the author with two or more websites that convey conflicting in- makes about a certain topic or issue. Stopping to clarify formation about a topic or issue (see also Castek, thinking about these questions on a regular basis helps Coiro, Guzniczak, & Bradshaw, 2012). Consequently, students internalize a structure for articulating details adolescents can benefit from guided conversations about an author’s expertise (or lack thereof) in ways about how to recognize new ideas, weigh the useful- that help better judge the quality of author claims. ness (and reliability) of these ideas against previous be- liefs, and consider that new ideas may be more accurate 3. Scaffold Inferences About the Consequences of than original ones. Figure 2 includes a useful sequence an Author’s Point of View of thinking prompts to begin these conversations. Once an author appears to have an acceptable level of expertise in topics related to his or her claims, stu- 5. Demonstrate the Value of Using Multiple dents then need support in making inferences about Indicators of Reliability the author’s point of view and potentially about that Finally, teachers can be more explicit in helping stu- of the author’s organization as well. Our findings dents combine information about an author’s level of show that many students (43%) were able to correctly expertise, his or her inherent biases, the quality of his identify an author’s point of view. However, most stu- or her arguments, and other acceptable criteria to dents failed to realize and present the consequences provide multiple criteria in support of their reliability of this point of view on text construction (i.e., the judgments. Prejudgments and singular criteria will words and images that authors choose to include). not be helpful strategies in a world characterized by These findings suggest that students need help under- increasingly diverse perspectives and evolving forms standing how an author chooses his or her words and of digital texts. Instead, students can benefit from Investigating Criteria That Seventh Graders Use to Evaluate the Quality of Online Information images among many possibilities. Explicit modeling learning how to apply a combination of acceptable can foster students’ understanding of what each au- criteria that can help them evaluate a website’s over- thor’s choices tell us about his or her intentions and all reliability and information contained within. underlying beliefs. Some related lesson ideas include Students also need opportunities to read information the University of Michigan’s website News Bias at a range of different websites with different formats Explored (www.umich.edu/~newsbias/index.html), (e.g., blogs, wikis, online newspapers, discussion Thibault and Walbert’s (n.d.) “Reading Images: An boards) to learn how to flexibly apply evaluation crite- Introduction to Visual Literacy,” and Lopez’s (n.d.) ria and generate informed, logical, and well-supported “Pictures and Slogans Persuade an Audience!” conclusions about a website’s reliability. FIGURE 2 Suggested Sequence of Thinking Prompts for Evaluating Conflicting Information Across Multiple Online Sources 1. What are two claims the author is making that are relevant to my topic? 2. What evidence do I find on different websites to support these claims, and what expertise do the authors have to make these claims? 3. What evidence do I find on different websites to refute these claims, and what expertise do the authors have to make these claims? 4. Consider the range of claims being made and the expertise of each source. Discuss with classmates whether the set of claims represent equally valid but opposing perspectives or a collection of some claims that are valid and other claims that are completely bogus. 5. Come to consensus about the quality of claims being made and which information, if any, can be used to answer your original question. 9  FEATURE ARTICLE Educational Importance of the particular difficulties that readers face when asked to judge source and information quality while Having the skills, strategies, and dispositions to com- engaged in an academic task. Findings from this study prehend and think critically about information on the provide an initial window into the limited range of Internet will play a central role in students’ success in source and message credibility perceptions, as well as this information age. This study extends our knowledge the wide range of surface-level credibility perceptions that characterize many younger adolescents’ responses when asked to explain their evaluations. Moreover, in- Take Action sights from both skilled and less skilled online readers STEPS FOR IMMEDIATE IMPLEMENTATION can inform a preliminary understanding of the com- plex profiles that readers may bring to an online read- As you teach adolescents how to evaluate the ing task. Perhaps most importantly, findings from this quality of online information: study can help identify instructional considerations and priorities related to critical evaluation for those stu- • Talk with students about the multiple dimen- dents who struggle in new online reading contexts. sions of critical evaluation. Define terms such as relevance, accuracy, perspective, and reliability and discuss the differences among them. References Students should notice that evaluating relevance Afflerbach, P., & Cho, B. (2009). Identifying and describing con- and accuracy involves considering the quality of structively responsive comprehension strategies in new and traditional forms of reading. In S. Israel & G. Duffy (Eds.), the content itself. In contrast, judgments about Handbook of reading comprehension research (pp. 69–90). perspective and reliability require an examina- Mahwah, NJ: Erlbaum. tion of details about the author and his or her Agosto, D.E. (2002). A model of young people’s decision-making agenda in relation to a specific affiliation. in using the Web. Library & Information Science Research, • Make time to explicitly model how to evaluate 24(4), 311–341. doi:10.1016/S0740- 8188(02)00131-7 each dimension of quality, and provide repeated Barzilai, S., & Zohar, A. (2012). Epistemic thinking in action: opportunities for students to practice and apply Evaluating and integrating online sources. Cognition and Instruction, 30(1), 39–85. doi:10.1080/07370008.2011.636495 these strategies to information they encounter Britt, M.A., & Aglinskas, C. (2002). Improving students’ abilities during research. to identify and use source information. Cognition and • Guide students toward independence by pairing Instruction, 20(4), 485–522. doi:10.1207/S1532690XCI2004_2 strategy instruction with written prompts that Britt, M.A., & Gabrys, G.L. (2001). Teaching advanced literacy ask students to monitor and annotate their skills for the World Wide Web. In C.R. Wolfe (Ed.), Learning judgments as part of their research. and teaching on the World Wide Web (pp. 73–90). San Diego, CA: Academic. doi:10.1016/B978- 012761891-3/50007-2 XX (X) XX 2015 • Expect students to apply a combination of Castek, J., Coiro, J., Guzniczak, L., & Bradshaw, C. (2012). acceptable criteria to evaluate the overall Examining peer collaboration in online inquiry. The reliability of a website and information contained Educational Forum, 76(4), 479–496. doi:10.1080/00131725.20 within. Consider the following: 12.707756 • Who created the information on this website, Coiro, J. (2003). Reading comprehension on the Internet: J OURN AL O F A DOL E SCE NT & A DULT L ITE RACY and what is this person’s level of expertise? Expanding our understanding of reading comprehension to encompass new literacies. The Reading Teacher, 56(5), • Why did this person or group put this informa- 458–464. tion on the Internet? Coiro, J. (2011). Predicting reading comprehension on the • Where can I go to check the accuracy of this Internet: Contributions of offline reading skills, online read- information? ing skills, and prior knowledge. Journal of Literacy Research, 43(4), 352–392. doi:10.1177/1086296X11421979 • Does the website present only one side of the Coiro, J., & Dobler, E. (2007). Exploring the online comprehen- issue, or are multiple perspectives provided? sion strategies used by sixth-grade skilled readers to search for • How is the information and/or images on this and locate information on the Internet. Reading Research website shaped by the author’s stance? Quarterly, 42(2), 214–257. doi:10.1598/RRQ.42.2.2 Fabos, B. (2008). The price of information: Critical literacy, edu- • Is there anyone who might be offended or cation, and today’s Internet. In J. Coiro, M. Knobel, C. hurt by the information on this website? Lankshear, & D.J. Leu (Eds.), Handbook of research on new • How can I connect these ideas to my own literacies (pp. 839–870). Mahwah, NJ: Erlbaum. questions and interpretations? Flanagin, A.J., & Metzger, M.J. (with Hartsell, E., Markov, A., 10 Medders, R., Pure, R., & Choi, E.). (2010). Kids and  credibility: An empirical examination of youth, digital media Thibault, M., & Walbert, D. (n.d.). Reading images: An introduc- use, and information credibility. Cambridge, MA: MIT tion to visual literacy. Retrieved from www.learnnc.org/lp/ Press. pages/675 Forzani, E., & Burlingame, C. (2012, November). Evaluating Thoman, E., & Jolls, T. (2005). Literacy for the 21st century: An representative state samples of seventh-grade students’ ability to overview and orientation guide to media literacy education. critically evaluate online information. Paper presented at the Part I: Theory. Santa Monica, CA: Center for Media Literacy. 62nd annual meeting of the Literacy Research Association, Retrieved from www.medialit.org/cml-medialit-kit San Diego, CA. Wallace, R.M., Kupperman, J., Krajcik, J., & Soloway, E. (2000). Goldman, S., Braasch, J., Wiley, J., Graesser, A., & Brodowinska, Science on the Web: Students online in a sixth- grade class- K. (2012). Comprehending and learning from Internet room. Journal of the Learning Sciences, 9(1), 75–104. sources: Processing patterns of better and poorer learners. doi:10.1207/s15327809jls0901_5 Reading Research Quarterly, 47(4), 356–381. Walraven, A., Brand-Gruwel, S., & Boshuizen, H.P.A. (2009). Hogan, N., & Vernhagen, C. (2012). Critical appraisal of infor- How students evaluate information and sources when search- mation on the Web in practice: Undergraduate students’ ing the World Wide Web for information. Computers & knowledge, reported use, and behaviour. Canadian Journal of Education, 52(1), 234–246. doi:10.1016/j.compedu.2008. Learning and Technology, 38(1), 1–14. 08.003 Kiili, C., Laurinen, L., & Marttunen, M. (2008). Students evalu- Zhang, M. (2013). Supporting middle school students’ online ating Internet sources: From versatile evaluators to uncritical reading of scientific resources: Moving beyond cursory, readers. Journal of Educational Computing Research, 39(1), fragmented, and opportunistic reading. Journal of Computer 75–95. doi:10.2190/EC.39.1.e A s s i s t e d L e ar nin g, 29 (2) , 138 –152 . doi :10.1111/ Krippendorff, K. (2004). Content analysis: An introduction to its j.1365- 2729.2012.00478.x methodology (2nd ed.). Thousand Oaks, CA: Sage. Zhang, S., & Duke, N.K. (2008). Strategies for Internet reading Leu, D.J., Coiro, J., Kulikowich, J., & Cui, W. (2012, December). for different purposes: A descriptive study of twelve good Using the psychometric characteristics of multiple-choice, open Internet readers. Journal of Literacy Research, 40(1), 128–162. Internet, and closed (simulated) Internet formats to refine the doi:10.1080/10862960802070491 development of online research and comprehension assessments in science: Year three of the ORCA Project. Paper presented at the 62nd annual meeting of the Literacy Research Association, Supporting Information San Diego, CA. Additional supporting information may be found in Leu, D.J., Kinzer, C.K., Coiro, J., Castek, J., & Henry, L.A. the online version of this article at http://onlineli Investigating Criteria That Seventh Graders Use to Evaluate the Quality of Online Information (2013). New literacies: A dual-level theory of the changing na- brary.wiley.com/doi/10.1002/jaal.448/suppinfo: ture of literacy, instruction, and assessment. In D.E. Alvermann, N.J. Unrau, & R.B. Ruddell (Eds.), Theoretical • Coding Scheme for Coding Constructed Responses to models and processes of reading (6th ed., pp. 1150–1181). Critical Evaluation Items Newark, DE: International Reading Association. doi: • Examples of Correct Responses When Seventh 10.1598/0710.42 Graders Are Asked to Discuss an Author’s Point of Leu, D.J., Kulikowich, J., Sedransk, N., & Coiro, J. (2009–2014). View and How It Affects the Words and Images on Assessing online reading comprehension: The ORCA Project [Research grant funded by the U.S. Department of Educa- the Website tion, Institute of Education Sciences]. Retrieved from • Examples of Correct Responses When Seventh www.orca.uconn.edu/orca-project/project-overview Graders Are Asked to Decide Whether Information Lopez, E. (n.d.). Pictures and slogans persuade an audience! on a Website Is Reliable and Explain Why [Lesson plan]. Retrieved from www.scholastic.com/teachers/ lesson-plan/pictures-and-slogans-persuade-audience Miller, C., & Bartlett, J. (2012). ‘Digital fluency’: Towards young people’s critical use of the Internet. Journal of Information More to Explore Literacy, 6(2), 35–55. doi:10.11645/6.2.1714 CONNECTED CONTENT-BASED RESOURCES Purcell, K., Rainie, L., Heaps, A., Buchanan, J., Friedrich, L., Jacklin, A., … Zickuhr, K. (2012). How teens do research in the ✓ Coiro, J. (2014, April 7). Teaching adolescents to digital world. Washington, DC: Pew Research Center’s evaluate the quality of online information [Weblog Internet & American Life Project. Retrieved from post]. Retrieved from www.edutopia.org/blog/ w w w. p e w i n t e r n e t . o r g / 2 012 / 11 / 01 / h o w - t e e n s - d o - evaluating-quality-of-online-info-julie-coiro research-in-the-digital-world ✓ Information Fluency. (2015). Improve your search skills: Rouet, J.-F., Ros, C., Goumi, A., Macedo-Rouet, M., & Dinet, J. Four ways. Retrieved from 21cif.com/index.html (2011). The influence of surface and deep cues on primary ✓ Ostenson, J. (2014). Reconsidering the checklist in and secondary school students’ assessment of relevance in teaching Internet source evaluation. portal: Libraries Web menus. Learning and Instruction, 21(2), 205–219. and the Academy, 14 (1), 33–50. doi:10.1016/j.learninstruc.2010.02.007 11