Engagement with Online Media and Advertising Effectiveness

Engagement with Online Media and Advertising Effectiveness Bobby J. Calder1 Marketing Department Kellogg School of Management Northwestern University Edward C. Malthouse2* Department of Integrated Marketing Medill School of Journalism Northwestern University Ute Schaedel3 Department of Media Management Hamburg Media School University of Hamburg August 2007 1 Charles H. Kellstadt Distinguished Professor of Marketing and Psychology, Kellogg School of Management, Northwestern University, calder@northwestern.edu 2 Theordore and Annie Sills Associate Professor of Integrated Marketing Communications, Medill School of Journalism, Northwestern University, ecm@northwestern.edu 3 Doctoral student, Hamburg Media School and University of Hamburg, u.schaedel@hamburgmediaschool.com *Corresponding author  Engagement with Online Media and Advertising Effectiveness ABSTRACT This research examines the impact on advertising of an online media context. Measures of eight different experiences of online sites are developed. These are shown to form two kinds of overall engagement, Personal Engagement and Interactive Engagement, with online media. Both of these types of engagement are shown to affect advertising effectiveness. Moreover, Interactive Engagement, which is a more social experience and is more uniquely characteristic of the web as a medium, is shown to affect advertising independently of Personal Engagement. Key Words: Engagement, Advertising Effectiveness, Context Effects, Online Media Engagement with Online Media and Advertising Effectiveness 1 INTRODUCTION Media provide a context for advertising that may affect consumer responses to advertising. Many studies have investigated possible media context effects, clearly demonstrating the existence of such effects. The most general conclusion is that when consumers are highly involved with a medium they can be more responsive to advertising (e.g. Aaker & Brown, (1972); Bronner & Neijens, (2006); Coulter, (1998); Cunningham, Hall, & Young, (2006); DePelsmacker, Geuens, & Anckaert, (2002); Feltham & Arnold, (1994); Gallagher, Foster, & Parsons, (2001); Nicovich, (2005); Wang, (2006)). Recent research has begun to explore the notion of involvement in more detail and to recognize that there are many forms of involvement with media. Malthouse, Calder, and Tamhane (2007), in the case of magazines, have identified a number of specific dimensions of involvement and shown that these differentially impact advertising effectiveness. These dimensions are qualitatively different and combine to form an overall level of involvement that Calder and Malthouse (2007) term media engagement. This terminology is consistent with the recent definition proffered by the Advertising Research Foundation (ARF): “media engagement is turning on a prospect to a brand idea enhanced by the surrounding context” (ARF, 2006). This definition has gained much currency as advertisers search for ways to overcome the problems of ad clutter and avoidance. One way is to advertise in media with high engagement levels. We conceptualize media engagement as the user’s collective qualitative experiences with content. Content can engage users in different ways. Some content is engaging because it informs users about topics relevant to them. Other content could be engaging because it helps the user to relax and escape from the pressures of daily life. To be engaging, different media Engagement with Online Media and Advertising Effectiveness 2 products need not deliver the same experiences. Content could be engaging because it provides high levels of a relaxation experience while other content could be engaging because it is informative. Experiences are not necessarily mutually exclusive and some content could engender high levels of multiple experiences. It is necessary to realize that engagement can take many multifaceted forms based on the different experiences users of a medium have. A very similar view of experiences as the key to understanding engagement has been proposed by Bronner and Neijens (2006). Media engagement is particularly interesting in the case of online media. It is commonly thought that online media are experienced differently than more traditional media such as television and print. This difference is often described as “leaning forward” versus “leaning backward”. The online experience is thought to be more engaging, active, participatory, and personal. The internet is also thought to be more social in nature because it can be used for sharing and communicating and it therefore breeds social engagement (Mathwick, 2002; Rappaport, 2007). Previous studies have tended to focus on this high-level experience of the web. For example Thorbjørnsen, Supphellen, Nysveen, and Pedersen (2002) examined the overall amount of experience people have with the web but deal only with the level of experience and not the nature of that experience. McMillan and Jang-Sun (2002) developed a measure of the perceived interactivity of online sites. Beyond this sort of general characterization, however, the experience of online media sites needs to be examined in more detail. We will present an approach to measuring online experiences across different sites in a more complete and in-depth way. From our previous work we expect that the online media studied in this research will be characterized by a variety of experiences. We further anticipate that some of these Engagement with Online Media and Advertising Effectiveness 3 experiences will relate to the active and social nature of the web. The ultimate objective of this research is to examine the impact of online experiences on advertising. Whereas previous work has focused on the use of interactivity in online ads (e.g., Ariely, 2000; Chatterjee, Hoffman, & Novak, 2003; Pavlou & Stewart, 2000), the focus of this research is on how the experience of an interactive online medium affects an ad appearing on it. As shown by Bezjian-Avery, Calder, and Iacobucci (1998), interactivity with an online ad may or may not result in greater ad effectiveness. The question here, however, is whether engagement with the online medium, particularly in terms of the active, social experience so identified with the web, makes an ad more effective. The specific ad format examined in this study is the standard internet banner ad with a click-on interactive option. Measuring Online Media Experiences and Engagement Media experiences can be described at different levels. At the most basic level of course there is the concrete experience of the particular content of a given web site, newspaper, or other media product. While this level of description may well be of interest, it is too saturated with details specific to the content and other unique characteristics of the particular content to be useful for comparison purposes. If our goal is to compare across different sites, as in the present case, we need a more abstract or generalized description of experiences. We approached this in the following way. As a first step qualitative research, in the form of individual, in-depth interviews with users, was conducted. Each interview focused on a specific site. But we sought to describe the experiences talked about in the interviews at a level that is common across the sites. Specifically, we sought to paraphrase the specific things that people report experiencing with specific sites in a way that preserves the common Engagement with Online Media and Advertising Effectiveness 4 essence, or gist, across sites but does not include details peculiar to individual sites. Exactly what people say about CNN.com is different from exactly what they say about chicagotribune.com, but at a higher-level people may be describing the same experience. They may be describing, for example, the extent to which they would say, paraphrasing across users, that “Once you start surfing around this site, it's hard to leave.” We refer to this description as an experience item. The logic of our approach is thus as follows. From qualitative interviews we induce a large number of experience items. Then we employ quantitative methods to explore the relationships among the items. If some experience items are highly interrelated, this indicates that they are alternative measures of the same experience. No single item is a perfect measure in that no one item captures a single experience in total. Experiences are not single-attribute objects that could be measured with a single item (see Bergkvist and Rossiter, 2007), but are sets of related items, or factors, that can provide a reliable measure of an experience. We refer to these sets of items as experience scales and use them to measure online media experiences. In previous work we have applied this approach to traditional media such as magazines and newspapers. Malthouse, Calder, and Tamhane (2007), for instance, measured experiences across a sample of 100 magazines. A number of specific experiences were found. The majority of these were shown to relate to the effectiveness of a test magazine ad. The more readers of a magazine experience the magazine as “making them smarter,” for instance, the more effective the test ad is, relative to a context-free control group, and this holds across magazines. Calder and Malthouse (2004) applied this approach to newspapers. Prior to the present study, Calder and Malthouse (2005) measured online experiences, using experience items developed from qualitative research, with a sample of 39 web sites. A Engagement with Online Media and Advertising Effectiveness 5 sample of 2,127 people rated their experience on a set of 154 experience items for one of the sites that they used. Exploratory factor analysis revealed 22 experiences with online content. The predictive validity of these experiences was assessed by relating them to a measure of individual site usage based on a behavior score approach (Calder & Malthouse, 2003). Some of the experiences were similar to but others were different from the experiences previously found for magazines and newspapers, supporting the general view that online is experienced differently. Also supporting this view, it was found that the online experiences of heavy users of traditional print were even more strongly related to online usage than was the case for light print users. This tended to be especially true for online experiences that are more qualitatively different from print experiences. It indeed seems that some people are seeking and finding a different kind of media experience online. In this research we select eight experiences from this previous study for the purpose of testing the effect of these experiences and engagement on online advertising. The selection of these experiences was guided by a desire to include some experiences, such as intrinsic enjoyment, that characterize more traditional media, and others, such as participation in a community, that seem particularly relevant to online media. Constraints on survey length limited us to eight experiences. Although our selection was based on the previous study, we use this study to confirm our measures of the eight experiences. Using data collected for this study we first subject these experience measures to confirmatory factor analysis. The confirmatory model is presented next. Following this is the test of the effects of these experiences on advertising. THE ONLINE ENGAGEMENT MEASUREMENT MODEL Engagement with Online Media and Advertising Effectiveness 6 We develop measures of online engagement using a two-step process. We first estimate a confirmatory factor analysis measurement model to study the psychometric properties of our experience measures. We then develop second-order engagement factors by applying exploratory factor analysis to the eight experiences and then fitting a second-order confirmatory factor analysis model. Eleven online media web sites participated in the present confirmatory study. These sites were different from the 39 in the Calder and Malthouse (2005) study.2 In both studies, the target population was people who used the site at least once a month. The studies employed different methodologies for recruiting subjects. The ComScore panel was used in the first study while pop-ups appeared on the particular sites for purposes of recruiting participants for the second study. The sample sizes for the 11 sites ranged from n = 203 to n = 2,006, with a median sample size of n=1,141 and a total sample size of n=11,541. Respondents were asked about their usage and experiences with the particular site. The first step in developing the online engagement measures is to estimate a measurement model for the experiences, allowing each possible pair of experiences to be correlated. Fit statistics are provided in Table 1. Question wording, factor loadings, and the values of coefficient alpha are provided in Table 2. There were 37 items used to measure the 8 experiences. All eight scales are highly reliable, with coefficient alpha ranging from 0.87 to 0.91. In the measurement model, each item had a parameter for the loading and error variance ⎛8⎞ (37+37 = 74), and there were ⎜⎜ ⎟⎟ = 28 parameters for the covariances between the ⎝ 2⎠ experiences, giving a total of 102 parameters. GFI, CFI, and NNFI all exceed 0.90, indicating an acceptable fit. Put tables 1 and 2 about here Engagement with Online Media and Advertising Effectiveness 7 Convergent validity was assessed with the t-values of the factor loadings, computed as the ratio of the loading to the standard error of the item. Convergent validity is supported when t- values reach an absolute value greater than 2. The minimum t-value was 48.2, providing evidence in support of the convergent validity of the indicators (Anderson & Gerbing, 1988). We assess discriminant validity with the chi-square difference test. For each of the 28 pairs of experiences we estimated a separate measurement model identical to the one shown in Table 2, except that the covariance between the pair is fixed at 1. The chi-square statistics between the models were computed, and range from 4,132 to 12,073. The differences all have chi- square distributions with 1df, and all are very highly significant, supporting discriminant validity. Having checked the psychometric properties of our scales, we estimate the experience levels for each respondent with the simple average of the items. Pearson Correlations between the experiences are provided in Table 3. Note that the correlations follow a pattern that suggests the possibility of higher level second-order factors. The first six experiences are moderately correlated with each other, with values between .42 and .72. Participation and Socializing (7) is substantially less correlated with the first six, but moderately correlated with the Community experience (8). Community is somewhat less correlated with the first six experiences. This correlation structure suggests that there is a higher-order factor structure generating the data. Table 3 about here Therefore the second step in developing the measurement model is to identify the second- order engagement factors. To do this we did both an exploratory and confirmatory factor analysis. We performed an exploratory factor analysis with a varimax rotation on the first- Engagement with Online Media and Advertising Effectiveness 8 order experiences and found two eigenvalues greater than 1. The rotated factor loadings are provided in Table 4 and show two interpretable factors, hereafter called Personal Engagement and Interactive Engagement. The first six experiences from the correlation matrix have the largest loadings on Personal Engagement, although Community also has a cross-loading greater than .3. Participation and Socializing as well as Community have the largest loadings on Interactive Engagement, but several other experiences have sizable cross-loadings. The Utilitarian experience likely cross-loads on Interactive Engagement because much of the advice and tips could be coming from the community of users rather than from the site itself. Self-esteem likely cross-loads because contributing to an online conversation could contribute to one’s self-esteem. Table 4 about here We now attempt to estimate a second-order factor model, which is a more parsimonious model for the 37×37 covariance matrix than the CFA measurement model. The objective is to test whether it is plausible that the Personal and Interactive Engagement latent variables generate the observed correlation structure between the experiences and items. Personal and Interactive Engagement will be used in the subsequent analyses of advertising effectiveness. Instead of having 28 covariances between the experiences, we assume that correlations between the experiences are due to two second-order factors. This model can represent the correlations between the experiences with only 12 factor loadings shown in Table 5 above, and one additional term for the covariance between the second-order factors. Fit statistics are also shown in Table 1 above, with CFI, GFI, and NNFI all greater than .9 suggesting a good fit. Figure 1 shows the parameter estimates of the second-order factor structure. The loadings for the 37 items were very similar to those from the measurement model above and have been Engagement with Online Media and Advertising Effectiveness 9 omitted. Note that the second-order factor model finds a significant correlation between the two engagement latent variables. In the analyses that follow, we estimate the two engagement factors using a weighted average of the experiences, with the factor loadings as weights. Figure 1 about here Personal Engagement is manifested in experiences that are similar to those that people have with newspapers and magazines. For example, experience items such as “This site makes me think of things in new ways” or “This site often gives me something to talk about” could also apply to a newspaper or magazine. Interactive Engagement, however, is more specific to web sites. Items such as “I do quite a bit of socializing on this site” and “I contribute to the conversation on this site” would not characterize a newspaper or magazine, and we did not hear such statements in our qualitative interviews for these media. While Interactive Engagement is more closely associated with the web, aspects of it can be found for other media. For example, “A big reason I like this site is what I get from other users” could also apply to the letters-to-the-editor page of a daily newspaper. The Utilitarian experience is a manifestation of both forms of engagement. Service oriented magazines (e.g., cooking, gardening, health) and newspaper sections (e.g., food) will have a prominent utilitarian component as will user-contributed advice sites (e.g., Yahoo!Answers or chowhound.com). In sum, the measurement model and value of coefficient alpha have shown that the eight experiences have been measured reliably and that convergent and discriminant validity are supported. The second-order analysis shows two engagement factors, Personal Engagement and Interactive Engagement. Personal Engagement is manifested in experiences that have counterparts in newspapers and newspapers while Interactive Engagement is more specific to web sites. As reflected the loadings in Figure 1, with Personal Engagement users seek Engagement with Online Media and Advertising Effectiveness 10 stimulation and inspiration from the site, they want to use the site to facilitate their interactions with other people, they feel the site affirms their self-worth, they get a sense of intrinsic enjoyment in using the site itself, they feel it is useful for achieving goals, and they value input from other users. With Interactive Engagement, users experience some of the same things in terms of intrinsic enjoyment, utilitarian worth, and valuing the input form the larger community of users but in a way that links to a sense of participating with others and socializing on the site. Thus Interactive Engagement is motivated both intrinsically and extrinsically but in this case it is social relevance of these, rather than their personal or individual quality, that is associated with the larger engagement experience. And it is the valuing of input from the community and sense of participating with others and socializing that gives Interactive Engagement its dominant character. Comparing Media Experiences and Engagement Across Sites It is of interest to examine the variation of these variables across the 11 different web sites. It was our intention in this research to select web sites that would be comparable (all were news related sites) in order to facilitate testing for advertising effects across sites but also sites that would exhibit some variation on the experience measures. This analysis tells how much sites vary in experience levels with the following random- effects ANOVA model: xij = μ + mi + eij where xij is rating of some experience/engagement by reader j of site i, μ is the overall mean across sites, mi is the random effect on the mean for site i having mean 0 and standard deviation σm, and eij is the error term having mean 0 and standard deviatoin σe. Random variables mi and eij are assumed to be normally distributed and independent of one another. Engagement with Online Media and Advertising Effectiveness 11 This model is estimated separately for each experience/engagement factor. For comparison purposes, we also estimate the model for both the exploratory and confirmatory studies. Table 5 about here Estimates of the overall means (μ) and the variation across site (σm) are shown in Table 5. Recall that experiences are measured on scales ranging from 1 to 5, where 5 indicates a high level of the experience. Consider the Stimulation and Inspiration experience in the confirmatory study, with a grand mean across sites of μ = 3.30. The next column gives the between-site standard deviation σm = 0.1084, which is italicized indicated that we can reject the null hypothesis that there is no variation in the means across sites (i.e., H0: σm2 = 0), implying that readers of all sites have the same experience. For example, if σm were 0, we would conclude that all sites are perceived as equally stimulating. We conclude that sites differ in the level or degree for each of the different experiences. Under the assumption that the means across sites are normal, we can conclude that 68% of sites have Stimulation means between 3.30 ± 0.1084, 95% of magazines have means between 3.30 ± 0.1084×2, etc. Readers of some sites have higher means on the Stimulation experience, while readers of others have lower means on the Stimulation experience. The values of σe in the last column indicate the extent to which readers of a site agree on the experience. For the Simulation and Inspiration experience σe = 0.7162, so readers of sites do not agree about the level of this experience. We now use the variables discussed in the previous two sections to examine the impact of engagement in online media on advertising effectiveness. ENGAGEMENT AND ADVERTISING EFFECTIVENESS Engagement with Online Media and Advertising Effectiveness 12 As noted earlier there has been relatively little previous research on the impact of the online media context on advertising. Existing studies have approached this at either a very high level or very specifically. To wit Bronner and Neijens (2006) compare the experiences of different types of media (using an approach to media experiences similar to our own) with the experiences of advertising content. They find, for instance, that the experience of usefulness with a site is related to the ads on that site being experienced as useful. And Wang (2006) finds in the context of an online game that an online ad inviting users to play a game was more effective than an ad that did not, suggesting that the game-ad might have benefited from the game context. In this study we seek to test the relationship between the experience of the online site and advertising effectiveness with some generality and specificity. The measures developed in the previous section will be used to characterize user experiences with one of the 11 specific web sites. After answering the experience questions, users of these sites were shown an ad and told that it would appear on the site they used. Two dependent variables were measured for the ad, including a standard copy testing measure and a measure of the intention to click on the ad. The ad could not have been previously seen by the research participants. We discuss each of the individual experience measures but focus on the two overall second-order measures of engagement. The first specific hypothesis we wish to test is: Hypothesis 1: Both Personal and Interactive Engagement will be related to advertising effectiveness. This relationship will be stronger for those who are told the ad will appear on a specific site than for those in a context-free control group who are told only that it is a banner ad. Engagement with Online Media and Advertising Effectiveness 13 A second focus of this study is whether the effect of Interactive Engagement on the test ads is independent of Personal Engagement. Hypothesis 2: The relationship between Interactive Engagement and advertising effectiveness will only be partially mediated by Personal Engagement. The second hypothesis thus examines whether experiences that are more particular to the web context have an effect on advertising effectiveness separate from other experiences that are more typical of other media. METHODS AND RESULTS Users of the 11 media web sites were intercepted with a pop-up during their visit to the site and asked to complete a survey. Participants answered questions about their use of, and experiences with, this web site. They were then shown an ad for orbitz.com (an online travel agency) and asked to rate it using standard copy-testing measures and their intention to click on the ad. We shall relate experiences with the media to these ad ratings. Note that this manipulation of media context is a weak one compared to encountering the ad while actually on the site and actually provides a strong test of the hypothesis, however. If the site experiences affect reactions to the ad in this test, the effect would be expected to be, if anything, smaller than in the case of seeing the ad while being on the site. One threat to validity is that the mere measurement of the experiences of a given site might itself affect reactions to the ad. Whereas this would imply that all experiences would affect the ad equally, it is at least possible that some of the experiences could be differentially sensitive to measurement (measurement × scale interaction). In this way, merely thinking about how a site gives advice and tips could have produced a higher rating of the ad. Another Engagement with Online Media and Advertising Effectiveness 14 threat is that any effect on advertising is not due to experiences with a particular site context but to experiences with sites in general (which are correlated with the particular site participants are told the ad is on). Alternatively, the different experiences individuals had with their site and the responses to the ads in general could be construed as an individual difference not dependent per se on using any particular sites. To assess these threats we used a context- free control group design. The most important thing about the control group is that the ad was identified only as a banner ad and not linked to any particular site. Of the 11,536 intercepted on the 11 sites, 1,502 were randomly assigned to a context-free control group, which was asked about their experiences with reading news sites in general and told only that the ads were banner ads. If any effects of the experiences on the ad are due to simply rating the experiences and/or thinking about sites in general while taking a survey, then the control group should respond in a similar way to those asked about a specific site. The treatment group being different from the control group indicates that the results do not reflect mere measurement or experiences with sites in general but rather measure the effects of experiences with specific sites. Copy-testing measure. We developed a multi-item scale to measure attitude toward the ad. Respondents were asked “How well does each of the following words describe the ad in the [site name]?” The study included the items “interesting, lively, helpful, believable, attractive, imaginative, and soothing” (7-point scale from “Does not describe the ad at all” to “Describes the ad very well”). These items were selected to be typical of those that are commonly used to test reactions to advertising stimuli (see Bearden & Netemeyer, 1999, Chapter 5) and to fit the particular ads tested here. The values of coefficient alpha for both ads were .93, indicating Engagement with Online Media and Advertising Effectiveness 15 reliable scales. As a second dependent variable, respondents were asked: “How likely are you to click on this ad?’’ Correlations between the experience, the engagement factors, and the advertising variables are also provided in Table 3 above. All correlations are positive, indicating that higher experience and engagement levels are associated with more ad effectiveness, although these correlations do not account for the different sites nor control for confounding factors such as use of on-line travel sites. These limitations are addressed in the next analysis. We now conduct a more stringent test of the relationship between the ad ratings and experiences/engagement by comparing the slopes of the treatment group (those who were told that the ad appeared on a specific site) and the context-free control group (those told the ad was not linked to a site) using an ANCOVA model. The model includes a different, fixed- effect intercept for each site (αj for site j=1, …, 11), a dummy x1=1 if the respondent was in the control group, a measure for the use of on-line travel agents in general x2, the experience rating x3 (as a continuous variable on a 5-point scale), and an interaction term between experience rating and the control group dummy: y = αj + β1 x1 + β2x2 + β3x3 + γ x1x3. The parameter β3 thus gives the slope for an experience in the treatment group, γ indicates how much larger or smaller the experience slope is in the control group compared with the treatment group, and β3+γ gives the slope for the control group. We can test whether the slopes in the treatment and control groups are different with H0: γ = 0. The model is estimated separately for each of the 8 experience and 2 engagement measures, with the results summarized in Table 6 below. Parameter estimates for the intercept terms α1, …, α11, and β1, and the slope for product usage β2 are omitted in Table 6 for clarity. Engagement with Online Media and Advertising Effectiveness 16 All of the treatment-group experience slopes β3 are positive and highly significantly different, consistent with the conclusions from the correlation matrix above. The experience slopes for the control group (β3+γ) are also significantly different from 0, which could be due to any of the threats to internal validity mentioned above or to the method of recruiting subjects used in this study (members of the control group were also intercepted from the sites under study and some may not have completely understood that they were to answer questions about sites in general rather than the one from which they were recruited). For each of the dependent variables, the γ-values for both Personal and Interactive Engagement are highly significant, indicating that engagement has a stronger effect on ad ratings when the respondent associated an ad with a particular site, and supporting H1 from above. As we indicated above, the manipulation of media context is relatively weak, and the effect sizes γ might well be larger if respondents were actually experiencing the particular site when they were exposed to the ad. Table 6 about here The γ-values for most, but not all, of the individual experiences are also significant. It is to be expected that there are fewer effects for the behavioral intention to click variable but there is also variation for the attitude toward the ad variable. The Intrinsic Enjoyment experience is not significant for either. We return to these results below. We now examine H2, the hypothesis that the effect of Interactive Engagement is only partially mediated by Personal Engagement. All mediation analyses that follow use only the treatment group. The results are summarized in Table 7. Following Barron and Kenny (1986), the first step in the mediation analysis is to show that there is an effect to be mediated by showing that Interactive Engagement is related to the dependent variables. We regress the dependent variables on Interactive Engagement, allowing for each site to have a different Engagement with Online Media and Advertising Effectiveness 17 intercept and report the slopes in the second column. The slopes for Interactive Engagement predicting attitude and intention to click are both very highly significant. The second step is to show that the initial variable (Interactive Engagement) is related to the mediator (Personal Engagement). We regress Personal on Interactive, again allowing for different site intercepts, and find a highly significant slope (0.871 in third column). The third step is to regress the dependent variables on both the initial (Interactive) and mediator (Personal) variables. The results are shown in the fourth and fifth columns. Both predictor variables are highly significant and roughly of comparable size. Both forms of engagement are thus important in predicting advertising effectiveness. The last step in the mediation analysis is to compare the effects of Interactive Engagement in columns 2 and 5. If Personal Engagement completely mediated Interactive Engagement, the slopes in column 5 would not be different from 0, but this is not the case. The difference between the slopes quantifying the amount of mediation (indirect effect) is computed in the sixth column and is large indicating partial mediation. Table 7 about here DISCUSSION AND CONCLUSION It is commonly believed that the web is different from other media in terms of leaning forward instead of backward, being more interactive, more social, and so forth. This research attempted to understand and measure the actual experience of websites. As anticipated it was found that in addition to experiences related to Personal Engagement, such as the experience of being personally useful, the sites studied also involved experiences related to Interactive Engagement, such as the experience of participating and socializing and the experience of being part of a community. Engagement with Online Media and Advertising Effectiveness 18 Specifically, we have identified and measured eight experiences with online news web sites and shown that the measures are reliable with discriminant and convergent validity. By factoring the eight experiences, we found that these experiences take the form of two different kinds of overall engagement. Two second-order factors exist. Again, one factor, Personal Engagement, is manifested in experiences that are very similar to those that people have with newspapers and magazines. Just as people read articles in newspapers and magazines that they can bring up in everyday conversations, online content can play a similar role. Just as reading a newspaper at the breakfast can be habitual, so can reading a web site. The other second-order factor, Interactive Engagement, is weighted more to experiences that are more unique to the web, such as participating in discussions and socializing with others through a site. It is these experiences that give Interactive Engagement its dominant social character. We have also shown that web sites studied differ in terms of their experience and engagement levels. This work set the stage for examining the effect of online media engagement on advertising. We related experiences and engagement to the ratings of a banner ad using a quasi-experimental design. The results show that both Personal and Interactive Engagement affect reactions to the banner ad. So, in additional to the Personal Engagement context effects that have been demonstrated previously for traditional media, the interactive component of a user’s experience with a web site is also shown to affect advertising. A context-free control group was used to strengthen internal validity by ruling out measurement effects. As predicted the effect of Interactive Engagement on advertising effectiveness does not depend on the relationship between the two forms of engagement. The results of the Engagement with Online Media and Advertising Effectiveness 19 mediation analysis indicate that Personal Engagement only partially mediates the effect of Interactive Engagement on advertising effectiveness. We thus conclude that indeed online media does involve a distinctive form of engagement and that this engagement has its own impact on advertising effectiveness. At the level of the eight individual experiences examined in this research it is also interesting that these experiences had a differential effect on advertising. There are several interesting possible explanations for this. Dahlén (2005) does a literature review of media context effects and summarizes three possible theoretical rationales for why context should affect reactions to ads. The first is the mood congruency-accessibility hypothesis: “The ad context makes a certain mood or affect more accessible and relieves the processing of stimuli with similar moods or affects (p. 90).” The second is the congruity principle: “the medium and the advertised brand converge and become more similar in consumers’ minds (p. 90).” The third is that the context serves as a cognitive prime that “activates a semantic network of related material that guides attention and determines the interpretation of the ad (p. 90).” While this research was not intended to test such hypotheses, the varying effect sizes observed indicate that such processes are at work. Clearly further research is indicated. Our conclusions are of course subject at this point to the limitations of the methodology of this study. Three points should be kept in mind. First, no matter how “representative” this ad might be, further research is called for to examine different product categories and types of advertising execution. For example, ads that were more interactive may have even stronger relationships with Interactive Engagement. Second, the relationship between experiences and advertising creative requires further theorization. Experiences could facilitate testing of the mood congruency-accessibility hypothesis and the congruity principle. Third, it would also be Engagement with Online Media and Advertising Effectiveness 20 desirable to conduct future research with actual insertion of ads on web sites. This might have some value in being a more “realistic” methodology with potentially better external validity. We note, however, that at best achieving external validity through matching a research setting with some “real” context is always fraught with difficulty (Calder, Phillips, & Tybout, 1983; Sternthal, Tybout, & Calder, 1987). It is never possible to duplicate the exact context, or even to know what key variable might be missing. In our view additional work with ads varying along theoretically motivated dimensions would be most valuable. Taking into consideration the limitations of this study, we believe that the effects of online media experiences on advertising are potentially pervasive and in great need of further investigation. Engagement with Online Media and Advertising Effectiveness 21 REFERNCES Aaker, D. A., & Brown, P. K. (1972). Evaluating Vehicle Source Effects. Journal of Advertising Research, 12(4), 11. Anderson, J. C., & Gerbing, D. W. (1988). Structural Equation Modelling in Practice: A Review and Recommended Two-Step Approach. Psychological Bulletin, 103, 411. ARF. (2006). Engagement. Retrieved 23 July, 2007, from http://www.thearf.org/research/engagement.html Ariely, D. (2000). Controlling the Information Flow: Effects on Consumers' Decision Making and Preferences. Journal of Consumer Research, 27(2), 233. Baron, R. M., & Kenny, D. A. (1986). The Moderator-Mediator Variable Distinction in Social Psychology Research: Conceptual, Strategic, and Statistical Considerations. Journal of Personality and Social Psychology, 51(6), 1173. Bearden, W. O., & Netemeyer, R. G. (1999). Handbook of Marketing Scales: Multi-Item Measures for Marketing and Consumer Behaviour Research. Thousand Oaks, CA: Sage Publications. Bergkvist, L., & Rossiter, J. R. (2007). The Predictive Validity of Multiple-Item Versus Single-Item Measures of the Same Constructs. Journal of Marketing Research (JMR), 44(2), 175. Bezjian-Avery, A., Calder, B., & Iacobucci, D. (1998). New Media Interactive Advertising Vs. Traditional Advertising. Journal of Advertising Research, 38(4), 23. Bronner, F., & Neijens, P. (2006). Audience Experiences of Media Context and Embedded Advertising: A Comparison of Eight Media. International Journal of Market Research, 48(1), 81. Calder, B., & Malthouse, E. C. (2005). Managing Media and Advertising Change with Integrated Marketing. Journal of Advertising Research, 45(4), 356. Calder, B. J., & Malthouse, E. C. (2003). The Behavioral Score Approach to Dependent Variables. Journal of Consumer Psychology, 13(4), 387. Calder, B. J., & Malthouse, E. C. (2004). Qualitative Media Measures: Newspaper Experiences. JMM: The International Journal on Media Management, 6(1/2), 123. Calder, B. J., & Malthouse, E. C. (2007). Conceptualization and Measuring Media Engagement, Working Paper: Kellogg School of Management, Northwestern University. Calder, B. J., Phillips, L., & Tybout, A. (1983). Beyond External Validity. Journal of Consumer Research, 10(1), 112. Chatterjee, P., Hoffman, D. L., & Novak, T. P. (2003). Modeling the Clickstream: Implications for Web-Based Advertising Efforts. Marketing Science, 22(4), 520. Coulter, K. S. (1998). The Effects of Affective Responses to Media Context on Advertising Evaluations. Journal of Advertising, 27(4), 41. Cunningham, T., Hall, A. S., & Young, C. (2006). The Advertising Magnifier Effect: An Mtv Study. Journal of Advertising Research, 4, 46. Dahlén, M. (2005). The Medium as a Contextual Cue: Effects of Creative Media Choice. Journal of Advertising, 34(3), 89. DePelsmacker, P., Geuens, M., & Anckaert, P. (2002). Media Context and Advertising Effectiveness: The Role of Context Appreciation and Context/Ad Similarity. Journal of Advertising, 31(2), 49. Engagement with Online Media and Advertising Effectiveness 22 Feltham, T. S., & Arnold, S. J. (1994). Program Involvement and Ad/Program Consistency as Moderators of Program Context Effects. Journal of Consumer Psychology, 3(1), 51. Gallagher, K., Foster, K. D., & Parsons, J. (2001). The Medium Is Not the Message: Advertising Effectiveness and Content Evaluation in Print and on the Web. Journal of Advertising Research, 41(4), 57. Malthouse, E. C., Calder, B. J., & Tamhane, A. C. (2007). The Effects of Media Context Experience on Advertising Effectiveness. Journal of Advertising, 36(6), 7. Mathwick, C. (2002). Understanding the Online Consumer: A Typology of Online Relational Norms and Behavior. Journal of Interactive Marketing, 16(1), 40. McMillan, S. J., & Jang-Sun, H. (2002). Measures of Perceived Interactivity: An Exploration of the Role of Direction of Communication, User Control, and Time in Shaping Perceptions of Interactivity. Journal of Advertising, 31(3), 29. Nicovich, S. G. (2005). The Effect of Involvement on Ad Judgment in a Video Game Environment: The Mediating Role of Presence. Journal of Interactive Advertising. Pavlou, P. A., & Steward, D. W. (2000). Measuring the Effects and Effectiveness of Interactive Advertising: A Research Agenda. Journal of Interactive Advertising, 1(1), 5. Rappaport, S. D. (2007). Lessons from Online Practice: New Advertising Models. Journal of Advertising Research, 47(2), 135. Sternthal, B., Tybout, A., & Calder, B. J. (1987). Confirmatory Versus Comparative Approaches to Theory Testing. Journal of Consumer Research, 14(114). Thorbjørnsen, H., Supphellen, M., Nysveen, H., & Pederson, P. E. (2002). Building Brand Relationships Online: A Comparison of Two Interactive Applications. Journal of Interactive Marketing, 16(3), 17. Wang, A. (2006). Advertising Engagement: A Driver of Message Involvement on Message Effects. Journal of Advertising Research, 46(4), 355. Engagement with Online Media and Advertising Effectiveness 23 1 Sites included national and local news sites, aggregators (Yahoo.com and Google.com), business sites, and sites giving news and reviews of video games. 2 In all models, the product usage variables have very highly significant positive effects. Likewise, across models the extra sums of squares are large and highly significant for the site intercepts, allowing us to reject the null hypothesis that all 11 sites have the same intercept. The control group dummy shifting the intercept rather than the slope is occasionally significant, but the signs change across models suggesting that the significant results could be type I errors. 3 The sites are About.com, Washingtonpost.com, PalmBeachPost.com, Reuters.com, DallasNews.com, Projo.com, King5.com, AZFamily.com, WFAA.com, KHOU.com, and PE.com. Engagement with Online Media and Advertising Effectiveness 24 Table 1: Summary of Confirmatory Factor Analysis Model MEASUREMENT SECOND-ORDER MODEL CFA MODEL Parameters 102 87 GFI .9155 .9029 CFI .9482 .9392 NNFI .9426 .9343 RMSEA .0472 .0505 Note. n=5942 with 37 items. Engagement with Online Media and Advertising Effectiveness 25 Table 2: Question Wording and Parameter Estimates from Confirmatory Factor Analysis Measurement Model STAND. EXPERIENCE ITEM LOADING Stimulation & It inspires me in my own life. 0.85 Inspiration This site makes me think of things in new ways. 0.84 (α=0.88) This site stimulates my thinking about lots of different topics. 0.78 This site makes me a more interesting person. 0.79 Some stories on this site touch me deep down. 0.71 Social Facilitation I bring up things I have seen on this site in conversations with many other 0.85 (α=0.88) people. This site often gives me something to talk about. 0.85 I use things from this site in discussions or arguments with people I know. 0.81 Temporal It's part of my routine. 0.85 (α=0.90) This is one of the sites I always go to anytime I am surfing the web. 0.83 I use it as a big part of getting my news for the day. 0.84 It helps me to get my day started in the morning. 0.80 Self-Esteem & Using this site makes me feel like a better citizen. 0.86 Civic-Mindedness Using this site makes a difference in my life. 0.85 (α=0.91) This site reflects my values. 0.76 It makes me more a part of my community. 0.75 I am a better person for using this site. 0.88 Intrinsic It's a treat for me. 0.83 Enjoyment Going to this site improves my mood, makes me happier. 0.85 (α=0.87) I like to kick back and wind down with it. 0.82 I like to go to this site when I am eating or taking a break. 0.65 While I am on this site, I don't think about other sites I might go to. 0.71 Utilitarian This site helps me make good purchase decisions. 0.81 (α=0.88) You learn how to improve yourself from this site. 0.83 This site provides information that helps me make important decisions. 0.76 This site helps me better manage my money. 0.81 I give advice and tips to people I know based on things I've read on this 0.74 site. Participation & I do quite a bit of socializing on this site. 0.86 Socializing I contribute to the conversation on this site. 0.77 (α=0.88) I often feel guilty about the amount of time I spend on this site socializing. 0.82 I should probably cut back on the amount of time I spend on this site 0.78 socializing. Community I'm as interested in input from other users as I am in the regular content 0.84 (α=0.88) on this site. A big reason I like this site is what I get from other users. 0.85 This site does a good job of getting its visitors to contribute or provide 0.59 feedback. I'd like to meet other people who regularly visit this site. 0.80 I've gotten interested in things I otherwise wouldn't have because of others 0.73 on this site. Overall, the visitors to this site are pretty knowledgeable about the topics 0.66 it covers so you can learn from them. Engagement with Online Media and Advertising Effectiveness 26 Table 3: Correlation Matrix (Treatment Group Only) PEARSON CORRELATION EXPERIENCE 1 2 3 4 5 6 7 8 9 10 1 Stimulation & Inspiration 2 Social Facilitation .56 3 Temporal .51 .55 4 Self-Esteem & Civic-Mindedness .65 .57 .47 5 Intrinsic Enjoyment .65 .52 .62 .63 6 Utilitarian .62 .52 .42 .72 .58 7 Participation & Socializing .24 .19 .19 .29 .33 .35 8 Community .51 .41 .32 .53 .53 .59 .56 ENGAGEMENT 9 Personal Engagement .79 .75 .78 .82 .81 .71 .32 .51 10 Interactive Engagement .52 .43 .43 .69 .61 .67 .77 .77 .74 ADVERTISING 11 Click Intention .24 .19 .15 .25 .23 .27 .12 .23 .27 .26 12 Attitude Towards Ad .30 .23 .19 .31 .29 .31 .14 .27 .34 .32 Note. All correlations are significantly different from 0 at the .0001 level. Engagement with Online Media and Advertising Effectiveness 27 Table 4: Exploratory Factor Analysis Loadings of First-Order Experiences. FACTOR 1 FACTOR 2 PERSONAL INTERACTIVE EXPERIENCE ENGAGEMENT ENGAGEMENT Social Facilitation .768 Temporal .753 Stimulation & Inspiration .744 Self-Esteem & Civic Mindedness .710 .375 Intrinsic Enjoyment .701 .366 Utilitarian .612 .472 Participation & Socializing .881 Community .361 .755 Note. Loadings less than .3 were omitted. Engagement with Online Media and Advertising Effectiveness 28 Table 5: Estimates from Separate Models Including Context-Free Control Group EXPLORATORY STUDY CONFIRMATORY STUDY Between Site Within Site Between Site Within Site EXPERIENCE Mean Std Dev Std Dev Mean Std Dev Std Dev Stimulation & Inspiration 3.22 0.2452 0.6412 3.31 0.1084 0.7162 Social Facilitation 3.31 0.1797 0.7499 3.33 0.1330 0.8514 Temporal 3.05 0.1744 0.7745 3.39 0.2028 0.9315 Self-Esteem & Civic-Mindedness 3.01 0.1307 0.6904 3.03 0.1097 0.7525 Intrinsic Enjoyment 3.09 0.1599 0.6576 3.02 0.0990 0.7689 Utilitarian 3.11 0.1991 0.6800 2.99 0.1273 0.7447 Participation & Socializing 2.37 0.1743 0.7590 2.04 0.0728 0.7518 Community 2.97 0.1431 0.6602 2.74 0.1913 0.7071 ENGAGEMENT Personal Engagement 3.14 0.1020 0.5686 3.11 0.1311 0.6044 Interactive Engagement 2.83 0.1169 0.5629 2.50 0.1806 0.5168 Note. p < 0.05 marked in italic. p < 0.01 marked in bold. Engagement with Online Media and Advertising Effectiveness 29 Table 6: Estimates from Separate Models Including Context-Free Control Group EXPERIENCE ATTITUDE TOWARD AD INTENTION TO CLICK β3 γ β3 γ Stimulation & Inspiration 0.63 −0.16 0.67 −0.19 Social Facilitation 0.42 −0.11 0.44 −0.12 Temporal 0.32 −0.09 0.32 −0.08 Self-Esteem & Civic-Mindedness 0.61 −0.27 0.62 −0.27 Intrinsic Enjoyment 0.56 −0.06 0.60 −0.04 Utilitarian 0.62 −0.15 0.68 −0.16 Participation & Socializing 0.60 −0.12 0.67 −0.15 Community 0.29 −0.18 0.35 −0.21 Personal Engagement 0.81 −0.20 0.85 −0.20 Interactive Engagement 0.90 −0.23 0.99 −0.27 Note. p < 0.05 marked in italic. p < 0.01 marked in bold. Engagement with Online Media and Advertising Effectiveness 30 Table 7: Estimates for the Moderated Model Y ON BOTH DEPENDENT Y ON PERSONAL ON MEDIATION VARIABLE (Y) INTERACTIVE INTERACTIVE PERSONAL INTERACTIVE EFFECT Attitude Towards Ad 0.911 (.027) 0.871 (0.008) 0.536 (.033) 0.443 (.039) 0.478 Intention to Click 1.008 (.034) 0.871 (0.008) 0.479 (.043) 0.590 (.050) 0.418 Engagement with Online Media and Advertising Effectiveness 31 Figure 1: Second-Order Factor Structure Stimulation & Inspiration Social Facilitation .85 .73 Temporal .68 PERSONAL .79 ENGAGEMENT .78 Self-Esteem & Civic-Mindedness .69 .42 Intrinsic .29 Enjoyment .14 INTERACTIVE .25 Utilitarian ENGAGEMENT .76 .74 Participation & Socializing Community