Anatomy of an Academic Article, Round 3: Coming Out Swinging

July 10, 2009 by Robert Kozinets.

If you’ve been following these blog entries on the evolution of my article for the Journal of Marketing Research, you will likely have noticed that the last round of reviews was not very positive.

In fact, I knew after reading those reviews that this was do or die time.

First, I had a new editor. I didn’t know Wagner Kamakura. I had never met him (I don’t believe I have met him to this day). This editor had a new set of standards, and it looked like he had seriously raised the bar on me. Russ Winer had expressed, probably signaling to reviewers, his interest in the topic and in publishing something that covered this topic. I received no such signal from Prof. Kamakura. In fact, Prof. Kamakura’s cover letter let me know that the paper was deficient in a number of ways. He also signaled that he wanted me to take more of a content analytic, quantitative, numerical approach to accounting for my data. I didn’t like it, but there it was. I could either count the number of postings I was using, and the users, or pretty much put myself out of the game. In the process, though, I thought that maybe I could educate the Editor a little bit about the differences in approach between ethnography and other methods. For example, that comments about providing ‘objective, unbiased’ information. That indicated that ethnography might not be an area that the good Editor was very familiar with.

But the good thing, the surprising thing really, was that Prof., Kamakura could easily have rejected the paper, and he had not. Reviewer B had not moved in his/her opinion. In fact, that reviewer seemed to hate the paper more now that ever. She or he had given the Editor a good reason to bail on me if the Editor had wanted to. Rev B questioned everything about it-the ethical stance, the method, the focus, the topic, my skills. In their closing shot in the review, Rev B implied that the paper and method were useless. And I guess that meant that the last two years of my life had been wasted working on these topics.
Now, I have to tell you that, as a junior faculty member with nothing on my CV save a relatively obscure European journal article, this was more than disheartening. It was, at first, like someone had dropped a baby grand piano on my back. I felt flattened. I felt roughed up. Okay, you saw the pictures. That’s what I felt like. But then, as I read through the comments over and over, I realized something.

It was clear to me that Reviewer B didn’t understand what I was doing at all.

Reviewer C pretty much got it. She or he was still a tad snarky, but they could see what I was doing and they were offering comments that were raising the bar and generally helpful. They were seeing that there was improvement, progress here. They were working with me.

Of course, even Reviewer C gave me pause. What did they mean by the manuscript ‘improved again’?-this was the first revision of it they had seen.

At this point, I had a major decision to make. I needed to make a complete rewrite of this paper. This wasn’t a time for minor modifications-the paper would be shot down almost instantly if I was to try that.

But I didn’t know how to deal with Reviewer B’s negativity. It seemed like there was absolutely nothing I could do to please this reviewer.

So, after consulting with a few faculty members around Kellogg, I decided to ask the Editor to speak to me on the phone. I’m glad that I did. In that phone call, Prof. Kamakura and I discussed the paper and what I could do to improve it. We straightened out the requirements for a good revision of the paper.

And I came to that phone meeting armed to the teeth with references, citations, and all that stuff I had been ruminating about. Overwhelming force, or as much of it as I could muster. In the process, I sought to convince Prof. Kamakura that I knew a lot more about my topic matter than Reviewer B. Reviewer B was, I still believe, the wrong person to be reviewing this manuscript.  I politely mentioned this to Prof. Kamakura. I also politely suggested that he might replace this reviewer with someone more qualified to review my manuscript.

Although you are getting the scans of these reviews, I got paper copy of Reviewer B’s reviews, which were typed on an old typewriter and then photocopied on some ancient 1940s model typewriter. This was clearly someone who was stuck in another age and just didn’t get anything about technology. And they clearly didn’t want to get it.

So I argued about some of the main points that I later raised in my reviewer notes. Online ethics are obvious and a non-issue? I don’t think so! And here are three solid references and three excellent reasons why that is so….

I grew as an academic during this process. Although I was junior, I knew more about my topic than any of these reviewers. And I aimed to show it to them. I had to take a stand to do that. I had to take some risks.

For my example of netnography, I was also profoundly challenged. My last revision’s example was definitely lame, no doubt about it. But I was hamstrung by that 30-page limit. How could I express the richness of real ethnography in three journal pages? I felt like paraphrasing Scotty on Star Trek: “Jim, it’s against the quantum laws of ethnographic writing–it just can’t be done.” [Yeah, yeah, another Star Trek reference, I know…]

Add to that the fact that I had only written one ethnography before, in my thesis and in the paper coming out of it for JCR. I had started my Burning Man ethnography by that time as well, but in terms of being an expert in ethnographic writing, that I most definitely was not. I had been collecting data for a while for another project, which I thought would be a separate paper at some point, a study of coffee community and culture online. The only thing to do was to sacrifice it up for this JMR piece, to add the richness and detail I had to the table, see if I could condense it into something meaningful, and illustrate what this new approach could yield in terms of real marketing and business insight.

So, after all of that, here are my Reviewer Notes. Notice that they reflect my growth as a scholar. They are a bit more confident, a lot more assertive, and they reflect someone who realizes that, when you know more than your reviewers do, you have to educate them. And when they don’t want to be educated, you need to out their critiques as unreasonable, an alternative point of view that you don’t subscribe to or, sometimes, as just plain wrong.

Revision Notes RE: JMR MS# 1205-9-3: “The Field Behind the Screen: Using Netnography to Research Market-Oriented Virtual Communities”

First, I would like to thank Prof. Kamakura and the 2 Reviewers for their helpful comments. The enclosed manuscript has undergone a major revision (almost a complete rewrite) and a new positioning in an attempt to fully incorporate the feedback from the past round of review. The illustrative example is new (I had discussed this with Prof. Kamakura in a phone conversation and he supported my decision to change the example so that I could draw on a richer pool of data). The example is now much more fully realized, and takes up approximately half of the revised manuscript’s space (before references). To make room for it, the background information on cyberculture, as well as all tables and figures, have been jettisoned. These notes will overview the specific changes that have been made in response to the reviews. The notes will respond in considerable detail to each of the reviews in turn.

A. Editors Comments

Your comments were very clear, directive, constructive, and helpful. This revision reflects an attempt to implement all of your requirements. As you noted in your letter, this did “require a complete re-write of the manuscript.” It was very challenging to try to convey the richness of ethnography, along with the thoroughness of methodological description, and references in 30 pages total. The original coffee ethnography was over 25 pages, and could easily have been over 60.

In general, the paper was more clearly positioned as a methodological piece in this revision. It omits almost all of the background information and theorizing on cyberculture and virtual communities present in past versions of the manuscript. It also provides much more detail on how to apply the method to marketing research, its advantages over competing qualitative marketing research methods. Finally, the example is much more fully developed than in the prior manuscript. Specific comments follow:

1. The positioning of this manuscript has evolved through the review process. This version conceptualizes netnography as a technique useful to the study of market-oriented virtual communities. This is a much more specific, and for that reason, defensible, positioning for the method and the manuscript. In the manuscript, it is explained that many virtual communities exist that have market-oriented concerns. Many of these virtual communities are quite sizeable (many in the 100,000 range, quite a few in the 500,000-1 million range, or more), comparable in the number of readers they might have to many magazines, newspapers, or cable TV shows. In addition, they are more specifically consumer-to-consumer, and so they might carry considerable weight in consumer decision-making if, as the Almquist and Roberts (2000) article cited in the paper might suggest, consumer advocacy is an important facet influencing consumers’ choice of one brand over another. Netnography is positioned in a manner that is entirely consistent with the theory and practice of ethnography in anthropology, sociology, cultural studies and consumer research. That is, (as described further in the test on p. 4) it is positioned as a study of the local and the particularistic (cultures, communities, communications) that can be, but need not be, generalized to other groups. I have tried to convince readers of this manuscript that these groups are, in and of themselves, populous and important.

2. The general discussion of CMC and virtual communities has been almost completely removed. What remains is placed in the introduction section and is very brief. It has also been positioned so as to be of more interest and relevance to marketing research practitioners as well as academics.

3. The introduction clearly distinguishes netnography from ethnography. This is aided by the new positioning (i.e., it would be difficult, and probably mostly irrelevant, to study virtual communities in-person). Netnography is also detailed as being far less intrusive, less resource intensive and providing quicker research turnaround than ethnography. Netnography is also compared with focus groups and interviews, and detailed as more naturalistic, less susceptible to demand bias, less intrusive, and less resource intensive. Again, positioning the method as useful for the study of market-oriented virtual communities ameliorates many of these concerns about comparison. To use focus groups or personal interviews to study virtual communities seems less logical than to use netnography.

4. This draft provides a much more specific and detailed description of the steps and procedures involved in conducting netnography. It can now act, as you suggest, “as a blueprint to the reader interested in applying the method on [sic] her own research.” As explained in the text, the inherent flexibility of the method is consistent with the positioning of the method in anthropology and other fields. This manuscript’s description of netnography emphasizes methodological rigor. However, because of the importance of interpretive methods, and the researcher’s role “as instrument” (to use John Sherry’s turn of phrase), the method can not be said to lead to “unbiased, objective information” any more than can personal interviews, focus groups, or market-oriented in-person ethnography. This build’s upon Reviewer 3’s suggestion to more fully develop the “flexibility” of netnography, and is explicated further in the text.

5. The empirical illustration attempts to address your comments fully. While it is extremely condensed, and must leave out much analysis and description because of the limited space available, it tries to provide the reader with a lucid example of a netnographic application as well as a sense of its general flavor. On pp. 15-18, the various steps you mention in your letter are detailed (and in some cases expanded):

6. which newsgroups were chosen
7. what criteria for newsgroup selection was used
8. the number of messages downloaded for interpretation
9. the message threads downloaded for interpretation
10. what criteria for message thread selection was used
11. the #distinct informants or messages posters involved n those messages
12. some classification of the types of messages found (in addition, in several other places throughout the paper, e.g., p. 8, p. 16, different categorization schemes were provided that might be useful to other researchers)
13. amount of raw data utilized
14. methods of determining trustworthy messages/ exclusion of suspect messages
15. interpretive methods (this was a very brief overview, with references, many articles and books are available on the interpretive methodology that was used here)
16. some discussion of the more appropriate employment of the term and concept of research “trustworthiness” (rather than the inappropriate term and notion of “validity” applied to positivist or quantitative research)
17. methods of verifying information and conclusions with informant member checks
18. types of alterations to the text made in response to member checks

In addition, it is worth noting that I have been diligent in error-checking the ethnography with literary sources as well as with experts. In an extended communication with a coffee entrepreneur and author of 2 books about coffee production, this person confirmed the correctness of my netnographic analysis and the potential utility of some of the conclusions to those in the coffee business. This person (who requested anonymity) also provided numerous comments about the specifics of coffee production that helped to improve the manuscript.

B. Reviewer #1 Comments

Thank you for your helpful comments. I hope that this extensive revision meets some of the objections to the prior revision that you identify and describe.

19. The positioning has been changed, as detailed in 1, above. The attempt has been made to address your concerns about a lack of focus and overclaiming.

20. Theoretical descriptions of the differences between online and offline communication have been removed in this version in order to make room for the empirical example of netnography. That there are difference between online and offline communication seems an important concern, and one that has been treated in a range of articles and books. In this article, the focus is considerably more pragmatic than directly theorizing and specifying these differences (a matter I leave to other articles, and/or other scholars). Therefore, there are far fewer “constructs” in this paper, and hopefully less vagueness and less confusion. The article focuses on the method and the illustrative example of the method.

21. Building on the knowledge that (1) there are a multitude of virtual communities that deal with consumption and marketing related concerns (like coffee consumption), (2) these groups are growing in number and probably in influence, and (3) the numbers of people reading these groups is in the tens of thousands, oftentimes hundreds of thousands, and sometimes millions, it seems useful to marketing researchers to have a methodological tool (some guidelines) with which to study these social phenomena. While ethnography is venerable and effective, there are some specifics to virtual community that conventional ethnographic practice simply does not discuss. These are all in the realm of the specific configuration of virtual communities, which I describe in the revised manuscript. For example, how does one choose a site or newsgroup? Which site or newsgroup to choose? Where to find lists of these groups? What types of messages to read, save and download? What types of informants are present on these groups? What types of classifications may be useful for this type of data? What types of interpretations may be useful for this type of data? How do we handle the particular ethical concerns raised by the online medium? And so on….

22. Your insight that “online research is useful for studying net cultures,” was helpful to the new positioning of the paper. As you suggest, the prior version’s more general claims to represent other types of groups was a less defensible position than that of the current revised version.

23. Your critical comments on the “Trust and Rapport” section were useful, and this section has been eliminated because, as you point out, it only applied to a particular kind of netnographic research (participatory). In addition, the notion of informant identity is rendered far less problematic if we conceptualize it as an issue of trustworthiness that can be treated by altering our unit of analysis from the individual to the interaction (as with G.H. Mead’s social psychological methods). This also answers your question about “what is imaginary and what is real”? Every interaction in a market-oriented virtual community is real. These insights are in the revised text on p. 10.

24. Your critical comments on the “Interview” and “Digital Projectives” sections were useful, and these sections have been eliminated due to space constraints.

25. You contend that “the whole matter of public versus private is a non-issue” and that “electronic eavesdropping” is not likely to give rise to psychological harm. Both points are contestable. The latter point appears tangential to the topic at hand. Netnography is a method for researching virtual communities and as such it will often have impacts far beyond the mere act of observation. What is more salient than data collection (”eavesdroppping”) is the potential impact resulting from the publication of information that was disclosed in an online forum. If I overhear someone discussing his or her private matters online and then publish it, is there the potential for harm?

With all due respect, your position appears overly theoretical. My own thinking on the matter has been informed by over 13 years of experience as a virtual community participant and over 5 years as a virtual community researcher. In my research, I regularly come across people who do not want their information shared with others or published (even after promising them anonymity). I have published this finding elsewhere, but in deference to your comment I now include it in revised manuscript (on p. 12).

Here is an online example of the type of trouble that researchers can cause. The following quote comes from someone who posted to a “public” forum on suicide and felt compelled to write a web-site conveying her feelings about being researched:

I am NOT a guinea pig. I am not sitting here, writing everything so that I can become part of someone’s report. . . I am, however, pissed off.-Dana-Christene Umanetz, posted on http://ash.xanthia.com/researchers.html

While this is a strong case, these types of comments are common among newsgroup participants. In my coffee ethnography for this article, people told me they were uncomfortable with the “lab rat” aspect of being observed. This sort of comment is very familiar to ethnographers of all stripes, but in person one has the distinct advantages of usually being able to see one’s observer. Not so in cyberspace. For the alt.coffee ethnography, I had people who required considerable explanation (and insisted on seeing copies of the manuscript, in order to judge how their quotes were being represented) before they would give me permission to use their postings.

If this manuscript is published in JMR, it might be used by graduate students and professional researchers, as well as researchers in disciplines outside of marketing. The potential that they might study something sensitive, like drug use, pornography, dealing with handicaps and illness, addictions, stigmatic groups or beliefs, or other social concerns (that can and may well have marketing implications) is very real. In addition, as an anthropologist, I have a strong concern for the cultures and culture members that I study. I believe that they and their beliefs must be treated with dignity and respect. I feel strongly that a section on ethics that recommends the ethical “high ground” is imperative for this article. I would not want it published without it.

The differences between netnography and ethnography arise exactly because of misconceptions such as the one that you proffer. Because cyberspace is not clearly public or private, netnography must have methodological provisions that specify procedural solutions to ethical concerns that face-to-face ethnography does not face. This is the “new news” you solicit. That it is distinct from in-person ethnography is specified clearly in this version on p. 13. Because these matters are contestable, frequently being contested by academics, and of major practical concern, this element could be one of this manuscript’s most important contributions.

26. As noted in comments 1 and 2, above, notions of generalizability have been replaced by a more defensible positioning of netnography as a method for studying specific and particular virtual communities.

27. The manuscript has removed most of the “participative” elements of netnographic methodology (such as interviews), mainly to conserve space, and to help prevent the sort of confusion that this Reviewer identifies.

C. Reviewer #3 Comments

Your comments were very honest, insightful, directive, constructive, and helpful. I truly appreciate the positive tone and the constructiveness of your review and have tried to implement all of your suggestions.

26. As outlined in comments 1 and especially 2, above, the paper’s positioning as a methods piece has been substantially altered in this major re-write. This version realizes your suggestion by portraying netnography as much more attuned to and related to the openendedness and flexibility of ethnographic methodology. The sections on p. 3-4 in the introduction to the method try to convey this. Even more importantly, the illustrative example exemplifies this openendedness by providing rich verbatims and (abbreviated, but hopefully still containing some interpretive spirit) analysis of the rich data provided in the <alt.coffee> newsgroup. This reflects my attempt to, as you suggest, “push” the method and “try to explain to a relatively naïve audience the nuanced nature of the interpretive act.” This effort is present throughout the major revision, particularly in the “Ethnography and Netnography” section, and towards the end of the “Applying Netnographic Methodology” section. In addition, building on your suggestion, you will find some comments comparing netnography with conversation analysis on p. 10 of this manuscript.

27. As noted in 4 and 5, above, the illustrative example has been subject to radical revision. I have taken your suggestion to ‘really build it up.’ It is now the centerpiece of this article. It is also now much more “ethnographic” as you suggest it should be. I do not believe that this ethnography could have come as a result of any other method. I do not think the religious evocations of voices calling unto coffee preparers, and “god shots” would have been likely outcomes of focus groups and conjoint. Show you the money? That’s what I was aiming for.

28. The conclusion section has been reined in to curtail overclaiming, while still trying to convey that this might be a useful method, one that marketing researchers may wish to utilize.

29. USA Today’s source is contained in footnote 1. As well, the Arbitron data (Reid 1995) may be of interest. Some search engines (like google.com) have archived copies of these pages, whose links may be expired.

30. Laurel’s quote is an exaggeration, but I think there’s still some truth to it, in a contingent sense. The cybergeeks still remain constituents of almost every newsgroup (true, certainly of the alt.coffee group), and their early conventions still drive CMC online. Yes, things have progressed and the innovators no longer have the playing field to themselves, but the people who are now there are still very much playing by the rules that the geekforce originally set. I think this is especially obvious in the use of conventions like emoticons and abbreviations. You’re quite right that the quote as it stands is contestable, and I have removed it as I jettisoned the cyber-theory sections of the paper. But this is a very interesting point. Thanks for the insight.

31. Yes, culture’s problematized nature is illustrated by my diagram. When trying to draw diagrams from my text, I often end up drawing concentric circles. Perhaps I should just stick to text. That’s what I’ve done in this revised version.

32. “Social inhabiting of imaginative animal…or alien bodies”…quite right. My enthusiasm was showing there. That section, along with much else, was slashed.

33. Your insight that F2F triangulation need not occur at all is, I believe, right on target. However, I struggled with your position vis-à-vis Reviewer 1’s position and (I believe) Prof. Kamakura’s. My solution was conveyed on p. 10 and 11 and represents a bit of a compromise. Because netnography is interested in the particular and the specific (now that it has been repositioned as dealing only with the study of virtual communities), and because we can think of the unit of analysis as interactions, chunks of text that represent implied/intended/actual social dynamics (or Wittgensteinian “language games”), we need not triangulate outside the method itself. However, it might be important to recognize that marketing researchers are probably going to want to generalize outside of the virtual community (whether we advise them to or not). They are going to try to apply findings to wider audiences assuming (perhaps rightly) that the vast bulk of “lurkers” is more like a general population whose characteristics might be useful to have but are very difficult to determine. Therefore, I have included a section that states that if netnography is going to be more generally repurposed then it would benefit from triangulation. In addition, triangulation might be useful when the stakes of the research are high, and someone’s neck is on the block. I completely agree triangulation isn’t strictly necessary, but I want to make sure JMR readers are aware it might be prudent for some occasions, and necessary for some kinds of claims.

34. As you suggest, the various methodological discussions (e.g., ethics, member checks) are now much more focused on netnography.

Thank you again, Prof. Kamakura and Reviewers, for your ideas and assistance.

Here is the abstract of the revised paper, the third revision:

 

The Field Behind the Screen: Using the Method of Netnography
To Research Market-Oriented Virtual Communities

 

ABSTRACT

This article develops netnography as an online marketing research technique for providing consumer insight. Netnography is market-oriented ethnography conducted on virtual communities dedicated to marketing-relevant topics. As a method, netnography is faster, simpler, and less expensive than ethnography, and more naturalistic and unobtrusive than focus groups or interviews. It provides information on the symbolism, meanings, and consumption patterns of online consumer groups. Guidelines are provided that acknowledge the online environment, respect the inherent flexibility and openness of ethnography, and provide rigor and ethics in the conduct of marketing research. As an illustrative example, a netnography of an online coffee newsgroup is provided and its marketing implications discussed.

Here is the entire third round revision: field_behind_round3.pdf

Posted in Academic Life, Qualitative Research Methods, Netnography, Marketing Research, Marketing News & Insights, Marketing Science | Print | No Comments »

Anatomy of An Academic Article, Round 2: Pierced to the Hilt

July 7, 2009 by Robert Kozinets.

I had worked and poured my heart into my revision of the JMR manuscript and had sent it back to Russ Winer by the deadline, my mind filled with hope and optimism about the possibility of publishing in JMR.

It was the end of summer, and the weather in Chicago was still quite warm. But the response I got from the Editor and Reviewers at JMR stopped me cold and chilled me through. The manuscript was still alive, but it was being eaten alive. Enduring a slow death, its fate was hanging by an extremely thin thread. It was bad news, much worse than before.

Have a look.

Journal of Marketing Research

September 13, 2000

Professor Robert V. Kozinets
Kellogg graduate School of Management Northwestern University
Leverone Hall
Evanston IL 60208-2008

Dear Robert,

RE: MS 1205-9-2 “The field behind the screen: Using Netnography to research market-oriented virtual communities.”

The above-referenced manuscript was read by the two reviewers who had recommended rejection of the original version. Their current recommendations regarding the disposition of this manuscript are as follows: Reviewer #1: reject; Reviewer #3: invite revision.

I see this manuscript as a methodological piece. It introduces marketing researchers to a new approach for collecting data from Web communities and for analyzing these data. However, the current version of the manuscript is way too long on background and too short on details about how the method can be actually applied, about its advantages over traditional research methods and about the empirical illustration.

Typical JMR readers would want to know when “Netnography” would be appropriate and useful, and how exactly they would go about applying it. I believe your manuscript could be of value to JMR readers, but it must be positioned and written properly for this audience. First, I suggest that you replace the discussion about computer-mediated communications and virtual communities of consumption with the proper citation of seminal articles and let readers seek them if they need an elaborate discussion on these topics. Your focus on the introduction should be on distinguishing “netnography” from ethnography, and clearly demonstrating the potential value of the former in marketing research, relative to competing methods (focus groups, and other face-to-face methods).

Second, you should provide a more specific and detailed description of the steps and procedures involved in conducting “netnography.” This should be a blueprint to the reader interested in applying the method on her own research, explaining why the steps are necessary and how they lead to unbiased, objective information to the marketer.

Third, your empirical illustration should provide the reader with a clear idea of how the proposed approach is applied and of its methodological rigor. The two experts find that the description of your application does not make a strong case for ethnography, and in fact might give readers the impression that it lacks rigor. The current application gives the erroneous impression that netnography consists of a simple content analysis of news group discussions. I would like to know each one of the steps that you have followed in this application, from selecting the sample, collecting the data, interpreting the results, cross-validating the interpretation, and the managers’ use of these results. For example, I would like to know:

• which particular news groups were chosen

• what criteria were used for this selection

• how many messages were interpreted

• how many users were involved in these messages

• some classification of those messages into general categories (my own experience with newsgroups is that there is a lot of idle, useless talk in them.)

• how were the messages validated (product-related newsgroups are known to be visited by manufacturers, often anonymously).

• the method used in interpreting these messages

• how the interpretations were cross-validated (multiple interpreters? Crossvalidation?)

• does this provide better and richer information than a series of focus groups, or indepth interviews? Why?

I hope you will be able to revise your manuscript along the lines discussed above, and responding to the two reviewers. This revision will require a complete re-write of the manuscript. I also want you to reduce the total length of the manuscript to no more than 30 pages (for example, you can drop the two figures and glossary, and reduce the citations to those strictly necessary). The success of this revision will depend heavily on the detailed description of the methodology and on the quality and rigor of the empirical illustration.

Sincerely,

[Wagner]

Wagner A. Kamakura

A quarterly publication of the American Marketing Association / 311 S. Wacker Drive, Suite 5800/ Chicago, Illinois 60606

“JMR MS# 1205-9-X1

 

“The Field Behind the Screen ….. ” Reviewer # 1

The revision is a workmanlike job that shortens the paper as the editor and reviewers suggest. Nonetheless, cosmetic changes (e.g., deleting “scholars in diverse disciplines” in the first sentence and replacing it with” marketing researchers” or replacing “interest” with “important”) do not represent a “more focused positioning,” as the authors claim in Note 3. The issue of overclaiming identified by Reviewer 3 comes out even more clearly in the revised version.

The article claims to build on “humanistic” research (par 2, p 1) to theorize “fundamental differences” between ways that researchers can access online vs. offline behaviors. This seems a rather pedestrian undertaking, and it is up to the authors to convince the readers not only that there ARE differences, but that they are sufficiently noteworthy to provide added value to researchers. To this end, the authors have included an empirical application of the method. The reviewer’s task remains that of assessing the revised claims of “new and different” as well as that of evaluating the empirical evidence.

A major problem that surfaces at the outset is weakness of construct definition. To begin, the authors note (p.3) the “frequent incoherence of the term community.” Next, the issue of whether or not CMC is impoverished (p. 4) or “richly detailed” is confusing. Footnote 5 is not helpful, and the authors’ stance is itself equivocal. Finally, the laundry list of special characteristics of CMC is “consolidated” (p. 5) in a way that makes it impossible for readers to understand. How did the 12 characteristics morph into the 3 that the authors’ identify? The presence of so many vague and patchy constructs suggests a glossing-over of fundamentals essential to theorizing.

Lack of clarity about CMC vs. f2f in terms of similarities/differences makes the section on CMC mediation of social representation (p. 7) confusing: which is it that the authors claim -differences or similarities? The discussion is still off the mark, because cmc has changed the nature of “writing” far beyond what Derrida et al. considered. The relationship between the written/spoken word is so complex that merely mentioning putative differences and/or similarities is obfuscating.

That online research is useful for studying net cultures (p. 10, “entree”) is rather obvious. Likewise, so is its application for the study of virtual “communities,” however defined, with an eye towards “providing a contextualized understanding” of behavior. This squarely places it in the realm of interpretive research, which, admittedly, has been sparsely represented in JMR. However, the global claim of applicability to marketing research is not borne out. From the lists of group types on (pp. 12 ff.), the material covered is simplistic and obvious to those in market research. For example (p. 15), the presence of enormous amounts of data requires data management; anyone who has dealt with transcripts of taped interviews is well aware of this.

“Trust and Rapport” assumes participatory rather than observational research, and informant identity (pp 15-16) does likewise. This is essentially ethnographic research, and not newsworthy. The alternative of identity-construction, discussed in terms of Haraway’s quote (p. 16), is most problematical. How is the researcher to evaluate what is imaginary and what is real? And what is a “lived material-semiotic world?”

The section on “interviews” are admitted to be “like their face-to-face counterparts” (p. 17), and the promise that digital projective tasks offer more “profound” levels of cultural knowledge gives rise to two questions: first, “more profound” than what? And second, what supports the claim of “more X” than anything?

The whole matter of public versus private is a non-issue: electronic eavesdropping on public communication is no more likely to give rise to “psychological harm” than is eavesdropping on a conversation held in a restaurant. King (bottom, p. 19) points this out. Where, exactly, can one not distinguish private from public (p. 20)? Indeed, the conclusion (bottom p. 20) is a nonstarter, for the first situation is routine ethnographic procedure, and the second (observation) is non-invasive. What is new news here?

The empirical portion (pp. 21 ff.) is what this reviewer was awaiting. However, the first paragraph exemplifies destroys any pretense of generalizability, and cannot be explained away by mentioning it. As Wells reminds us, merely stating limitations does not make them go away. Further, mining the news group appears to be observational — thereby rendering most of the preceding sections (those dealing with participatory researchers) irrelevant. The dilemma here is that the technique is shown to be neither innovative nor of real value to researchers. If the “downloads and observations” are anonymously generated, how is the market researcher supposed to discriminate between the real and imaginary selves of the respondent? The research outcome is disappointing from the perspective of value added: f2f interviews yield the same information (useful in market segmentation), without the identity mystery. In market positioning, it is essential to know who the market segments are!

Thus, the implications (p. 25) are neither “major,” as claimed, nor– potentially — even useful.

Notes to Authors of “netnograhpy. [sic]” –1205-9-2 Rev. #3

General: First, I want to say that this manuscript has again improved. It has also moved in the direction requested by the reviewers and the editor. It is also clearly an important topic. We’ve come a long way from the first submission.

I really want to acknowledge that.

There are still, however, problems. They are:

Problems:

One: The paper, positioned as a methods piece, is still under-specified, or (alternatively) simply reflects the open and inherently under-specified nature of ethnography. I tend to believe the later. Ethnography, at least to me, represents the most open form of inquiry. This does not mean it is sloppy or purely a make things up as you go meta-method either. This point seems lost in our current discourse, and in this paper. The author gets caught in this dammed-if-you do, damned if you don’t dilemma. But, I don’t want to let him or her entirely off the hook. Right now the paper reads like a very-under-specified scientific positioning of ethnography. This isn’t working for me very well. I wish the author would either push it fully in that direction, or more explicitly speak of its flexibility as a virtue, and try to explain to a relatively naive audience the nuanced nature of the interpretive act. This is not easy, I know that.

Two: The example is a fairly weak. There’s really nothing ethnography (specifically) did here in order to recommend it above any other method. This is the weakest part of the manuscript. Either it should be cut out entirely, or really built up. JMR readers are going to rightfully say: I don’t see anything special about this: standard focus group/concept testing accompanied by conjoint would have done a much better job … and they are right. Either show us the money, or drop it. I prefer the former. Show us what this method can do above and beyond any other.

Three: conclusion is a bit of a job of over-claiming ..

Page by page:

p., par I: what was USA Today’s source?

P 1: I still think this is essentially the same (or very similar) to what the interpersonal communications people are doing with on-line communication: it’s conversation analysis. And even prior to the net, when conversations were tape recorded, transcribed and analyzed … isn’t this similar?

P 2: I don’t agree with Laurel (1990), and I assume you as well: “all virtual communities exist as ‘villages of activity within the larger cultures of computing.’” I suppose this is quite debatable, but there is good theoretical reason to believe that as we go further down the diffusion curve, the medium and it’s culture becomes more transparent. People who listen to the radio in their cars, are not like early Marconi (sp?) (users). My guess is that by this point many users are not really cyberculture members at all (Whatever that is), but simply users of a rapidly diffusing technology. But, this is just a matter of opinion. You are entitled to yours.

P2: I think what you discuss here and illustrate in Figure 1 beautifully illustrates the problematized nature of the term “culture.” As you no doubt know, the notion of a set thing called culture is not particularly viable these days. I see these concentric circles as much too tightly drawn and formal. Cultures are built, tom down, and re-invented constantly. But, if this is how you want to do it.

P4: extra word, first sent.

p. 5: I’m not convinced that the limited cue theories have been discredited. Maybe, but I’d like to see more evidence.

P.6; grammar: addition

p. 8: “for reversion of gender, or ungendered being, and even the social inhabiting of imaginative animal, mechanical or alien bodies.” … no doubt this is true, but I think we may be getting a bit far afield from the modal application here.

P 11: I’m not sure FtF triangulation has to occur at all. They are all different methods. For example, surveys don’t NEED experiments.

P12: again, I think unobtrusive data may stand alone.

P 16: extra “identity”

P 17- I we are now trading in very general terms about surveys and interviews, I thought this was about this unique form of ethnography. This doesn’t belong here.

P 19-20 this discussion of ethics should be more focused on your particular method. Very general discussion.

P 21: more on the idea of member checks, and its particular value here.

P22: total postings ==7

Pps 21-25: Illustrative Example: this really didn’t work for me, at all. Two problems: There is no methodological detail here, or really much of anything about METHOD, which is really the promise of the paper, and (2) I really don’t see what this yields beyond the standard focus group/concept tests by a commercial supplier, and maybe a conjoint. Given your focus on segments, why wouldn’t a conjoint do as well, and provided comparable attribute metrics, you could call them “netrics. “? Don’t get me wrong, I’m a fan of ethnography, but its value is not shining through here. Do it justice, or drop it.

P 26: missing word.

Posted in Academic Life, Qualitative Research Methods, Netnography, Marketing Research, Marketing News & Insights, Marketing Science | Print | No Comments »

Anatomy of an Academic Article: Back on the Horse

July 6, 2009 by Robert Kozinets.

So what did I make of these reviews?

Well, it really was a mixed bag, and as a junior scholar I wasn’t particularly well-equipped to interpret the results. I had the advantage of having a strong mentor in John Sherry, who gave me a lot of guidance. But there’s only so much you can do when the data point to paradox.

I think it’s pretty unusual that one reviewer on the first round says unconditionally accept. That certainly hasn’t happened to me very much (maybe one or two other times). Reviewer 2, whoever you are, I thank you. I never got to see the review. I don’t even know if they wrote one. But it certainly didn’t hurt innovation to only have to deal with two reviewers for the rest of the way through rather than three (or even, worse, four or five as I have sometimes seen & experienced).

Then, the other two reviews are both rejections. Now, that makes it interesting for the Editor. The Editor, at that point, has grounds to reject the paper. Forget the “Despite Merit” part, that’s just being polite. Those are two rejections, plain and simple. Those reviewers want this thing down and out–they don’t want to see it again.

Rejecting the paper would have been, I think, the most conventional, sensible, defensible, logical, easy decision here. Had the Editor been interested in keeping out qualitative research, or new methods, he easily could have simply said something like ‘We are open to new stuff, but the Reviewers have spoken, 2 out of 3 said reject, sorry, try another journal.’

But there seem to be at least two reasons why Russ Winer didn’t do that.

First, and I’m reading behind and between the lines to do this, I think this is a somwhat unusual set of reviews. He said that he sent the paper to one on my ‘don’t send’ list, one on my ‘do send’ list, and one other person he trusted. I think that the person from my send list gave the unconditional accept. However, I think the one on my ‘don’t send’ list was Reviewer #3, and I think Reviewer #3 was significantly more positive towards the paper than Reviewer #1 (but still rejected it). So Reviewer #1, who the Editor chose, would have been the most negative one towards the paper.

Secondly, Prof. Winer makes a very telling admission in his letter: “Let me state at the outset that I am interested in publishing a paper on this topic.” I read that statement then, and still do now, as very encouraging to me, and as a clear signal to reviewers. In fact, that whole paragraph is as much for the reviewers as it is for me. He continues: “JMR editors should publish and always have published papers demonstrating the value of new methodologies. Of course, what is particularly interesting and topical about netnography is that its domain is the Internet.”

Again, this says to me that this Editor is taking his job as gatekeeper in a top-tier publication very seriously, and weighing it in relation to the advancement of knowledge. He realizes that novel methods are going to be resisted by the old guard. And he sees his job as encouraging, fostering innovation, making sure that those flickering new flames aren’t prematurely snuffed out. And the new game in town for Marketing Research is the Internet, and new Internet-based methods should be of interest to JMR. This is the right place for this research–that’s my interpretation. And when he says he is interested in publishing a paper on this topic, I see that as very positive. Because I am very interested in writing a paper on this topic and, as far as I know, there was no one else doing so at that time.

So he asks me to take into consideration some of Reviewer #1 and #3’s comments, and then asks for a paper that is thoroughly edited, contained a good example of an application, and is shortened.

That sounded quite do-able to me. The example would take a bit of time, though.

Maybe the scariest part of this was that I was teaching a full load of demanding Kellogg courses until well into May. That included my Hollywood course on Entertainment Marketing, which finished with a weeklong trip to Hollywood with my students. Pretty intense.

It was very unlikely that I would finish the revisions by June 30th. So I may be looking at sending my revision to a new editor, an unknown editor, an editor who might not be as fair-minded or as open to new and Internet-based methods as the former one.

Timing is everything. As we will see.

* * *

I did make my revision, and my revision notes. Revision notes, if you aren’t familiar with them, are the notes that you are required to provide with your revision to point out exactly how you addressed the comments of the Reviewers and the Editor.

The example I chose to add into the shortened paper at this point, as you will see, is a kitchen appliance search that I performed. This was part of some innovation work I was doing with a company, and there was only so much of it that I could share, so the example is pretty sparse and spare.

Here are the revision notes, in their entirety. Reading through them now, I would say that they are not the best set of notes I’ve devised, but they certainly do indicate that I tried to make the major revisions asked of me, and many of the minor ones, too.

To be honest, I didn’t really know what to do with a lot of Reviewer #1’s comments. They seemed like nit-picks from someone who didn’t understand the method, the Internet, or what I was trying to do. Reviewer #1 seemed to be trying to put me into a familiar box. Deconstruction?

Reviewer #3 wrote the review in sort of an oddly confrontational style, but had some good points, and I thought about them a lot and tried to address them. That thinking was productive. But was the experience of reading that abrasive tone enjoyable in the least? NO WAY.

Here are my revision notes:

Revision Notes RE: JMR MS# 1205-9-1: “The Field Behind the Screen: Using Netnography to Research Market-Oriented Virtual Communities”

First, I wish to thank the Editor and reviewers for their insightful, helpful and encouraging comments. I have sought in these revisions to meet all of the reviewers and Editor’s comments. These notes will overview in detail the changes that have been made to address the comments provided.

1. As suggested by the editor, the paper has been shortened significantly. The section entitled “Ethnography and Netnography” has been overhauled and now focuses more precisely on the method of netnography. The paper has been carefully edited to remove redundancies. In addition, sub-sections and topics whose orientation appeared either redundant (e.g., cultural exit) or of mainly theoretical interest (e.g., representation) were either removed or trimmed back substantially. The result is a 30-page paper, excluding exhibits and references.

2. Within these 30 pages is included a 5-page illustrative example that demonstrates how netnography can be (actually, has been) applied in practice. Adding this “capstone” empirical application has been one important way in which this revised version of the earlier manuscript brings the method of netnography into a form that may be much more useful to marketing researchers than the previous version. In addition, the entire manuscript has been substantially revised to more overtly and clearly reflect the interests and objectives of marketing researchers (Editor’s comments).

3. To clarify the positioning of the paper, the opening paragraph now clearly describes and details the meaning of market-oriented interests, specifying them as social information that is not marketer-driven (Reviewer#1 comments). This more focused positioning is also reflected at other places within the revised manuscript.

4. This revision of the paper repositions netnography as an extension of current methods, in line with the comments of Reviewer #3. The current version presents netnography as more of an integrative method that extends and synthesizes current ethnographic methodology into a form that may be more suitable to some of the unique contingencies of the Internet environments (for example, there are new ethical issues, completely unobtrusive “lurking” is possible, informant accessibility is of an entirely new order). The revised version attempts to demonstrate how certain current ethnographic approaches must be modified for the online environment. These techniques are not in themselves a new methodology, but they do (arguably) represent a potentially useful extension of current techniques to a new and sometimes very different social environment. The claims for discontinuity no longer appear in this version. I agree with the reviewer that these claims were not convincing, and are not necessary (Reviewer #3 comments).

5. Footnote 4 has been added in order to raise the point that the discussion of the signs used in CMC in order to replicate face-to-face communication are similar to the differences between print media and f2f speech (Reviewer#1 comments).

6. The object of study of netnography has been overtly described in this revised version as dealing with the content of CMC as it may affect consumer choice (for example, on p. 4) (Reviewer#1 comments).

7. The use of “jargon” has been greatly reduced on p. 7, and throughout the paper. Potentially obfuscating phrases such as “obtrusive and opaque cultural experience” and “social specificity” have been cleaned up or removed outright (Reviewer#1 comments).

8. The differences between CMC and f2f time lags have been developed further, as presented on page 7 and Table 2 (Reviewer#1 comments).

9. The writing/speech relationship has been developed further, through citing relevant work in deconstructive criticism, on page 8 (Reviewer#1 comments).

10. The mutability of identity concept in CMC has been developed further as a more extreme manifestation of what often occurs during f2f communication, on p. 8 (Reviewer#1 comments).

11. The focus of the market-oriented behaviors considered in the paper has been clarified as those which are consumer- (rather than marketer-) driven (Reviewer#1 comments).

12. The focus of market-oriented netnography has been clarified as the topic of the discourse itself, as demonstrated in page 1 and 2’s revised definition and description of the method, and elsewhere in the revised manuscript. These revisions attempt to make clear that one of the important limitations of the method is that (as with any ethnography) it is not guaranteed to produce generalizable knowledge (Reviewer#1 comments).

13. Spurred by these insightful comments, the important issues regarding the accessibility of data have been reformulated. This has resulted in a rethinking and repositioning of the observation and participation comments throughout the manuscript on the basis of these helpful comments (Reviewer#1 comments).

14. The discussion of anonymous users has been revised, and has introduced an ontological discussion (Reviewer#1 comments).

15. More attention has been paid to the discussion of response bias, random selection and comparison of netnography and online surveys, as this version reflects my attempts to make these points clearer (see page 19) (Reviewer#1 comments).

16. As with the comment addressed in point 13, above, the Ethics section has been substantially revised to reflect the interesting comments regarding the role of strictly observational ethnography, and to directly address and further develop the ideas contained in Reviewer #1’s comments on the matter.

In summary, addressing these comments has resulted in a substantially revised paper that is more firmly focused on applications relevant to marketing researchers and practitioners, contains an empirical application, does not overstate the newness of the method, is considerably clearer, develops ideas more thoroughly and is 30 pages long excluding exhibits and references.

Thank you again, Reviewers and Editors, for your ideas and assistance.

* * *

This is the abstract for the Round 1 revision, entitled : “The Field Behind the Screen: Using the Method of Netnography
To Research Market-Oriented Virtual Communities”

ABSTRACT

Computer-mediated communication (CMC) has fostered cybercultures and virtual communities, many of which are market-oriented in their focus. This article develops netnography as an online ethnographic technique for marketing research. Netnographic techniques are adapted to the characteristics of CMC. Netnography presents flexible guidelines for conducting online fieldwork that adaptively address central ethnographic concerns, while providing practical marketing research. An illustrative example is provided to demonstrate the utility of using netnography for new product development in the small kitchen appliance market. The example demonstrates how netnographic observations inform product attribute and positioning decisions. Implications of the method are discussed in a concluding section.

Here is the complete Round 1 Revision: field_behind_round1_rev.pdf

And, in the next post in this series, you’ll see just how this revision was received by JMR.

Posted in Academic Life, Qualitative Research Methods, Netnography, Marketing Research, Marketing News & Insights, Marketing Science | Print | No Comments »