Tag Archives: practices

Guidelines for Reducing Implicit Bias in Your Grading

Leave a reply
Kyoto Garden

Garden in Kyoto

When asked what they most dislike about teaching, many instructors put grading at the top of the list. They find the process time consuming and stressful, topped off by demands from students that their assessments be logically justified. Who hasn’t found themselves in a situation of confronting a student who feels treated unfairly in the grading process and whose persistent questioning reveals that a grade does not stand up to scrutiny. Indeed, upon close inspection, the grading seems arbitrary and hard to defend. What to do?

Grading needs to be done well to give students the feeling that they are being treated fairly in the assessment process. Instructors need to use the same criteria of reliability and validity in designing assessments that they use in their empirical research. Just as they might be called upon by reviewers to defend the quality of data used in an article, so also must they have an answer to students’ queries about the rationale for their marks.

From my perspective, “implicit bias” constitutes a threat to the integrity of the assessment process. Instructors need to take every possible step to reduce the possibility that grades reflect less the merits of the answers than the personal characteristics of the student or the arbitrary whims and fancies of the grading process. Race, class, gender, sexual identity, social capital, and other student characteristics can affect grading if instructors haven’t created a process to limit their effects.

In addition, I believe instructors’ overconfidence in their grading abilities constitutes another form of implicit bias that hampers their ability to assign grades fairly. Keeping the process opaque and not sharing grading criteria with the students emphasizes the unequal power between students and instructors, and is another source of student cynicism about the educational process.

What steps might instructors take?

First, grade all assessments blindly. This means making sure there is no identifying information available while grading the assignment. You can do this by having students turn in blue books with the cover page turned back, by having identification numbers instead of names, by having students fold over the top of the page on which the name is written, and so forth.

Second, prepare an answer key beforehand for all answers. Some instructors call this a “rubric.” The answer key should not be simply bullet points, but rather a fully written out answer, of the kind you would expect to earn full credit on the question. You could also have a list of characteristics or features you’d expect in the question, but the sample answer – – which should be posted or handed out to the students – – should be complete and in prose form.

Preparing the answer ahead of time lets the instructor know that a question can, in fact, be graded. In addition, it prevents “bracket creep” in which an implicit and unwritten template for an answer subtly changes as the instructor reads through the answers and subtly changes the criteria for a grade. If you aren’t certain as to whether your template is too tough, you should read a sample of the answers ahead of time, before grading them, and revise your template if necessary. The rubric should not be changed, once grading is underway.

Third, write out the comments necessary to justify your mark. Don’t just write a simple one or two-word phrase, such as “good job,” or “not complete.” Write enough information next to the answers so that you can explain to students, when they come to you for advice, why you gave that particular mark.

Fourth, grade all of one question before beginning to grade the next question. For example, if your exam consists of four essay questions, you would grade all of question one first, going through all the exams, and then shuffle the exams and grade all of question two. And so on throughout the four questions, in order. This ensures that you are using the same standard throughout your grading and that your grading is not influenced by marks that you have given for previous answers.

You must ensure that a student’ s grade on a previous question is not visible to you. Otherwise, that grade is likely influence the grade you give the current question. Turn over the previous page so that you cannot see it.

Firth, take breaks while grading, and do not attempt to binge on finishing the grading in one sitting. Mistakes are much more likely if you continue grading to the point of exhaustion!

Working to ensure that your assessments are graded reliably and validly requires a bit more preparatory work on an instructor’s part, but the extra work returns huge dividends. When students realize that you are taking great care to grade their work fairly, they take a much more positive view of the assessment process. By the time they get to college, many have become quite cynical about the way instructors exercise their power in handling assessments, and they will appreciate the extra effort you take to make the process as transparent as possible.

Save

Save

The Sound of Silence Can Be Deafening & the Questions You Ask Your Students Can Provoke It

3 Replies

A colleague recently visited my office with a problem. He said the students in his undergraduate class “didn’t want to talk.” He and I had previously talked about how to get students more engaged, and I had suggested to him that he ask questions. I probed, “what kinds of questions have you asked your students?” He replied, “Well, the first question I asked this morning was ‘what is the main point of the article I assigned for the day?’” Nobody said anything.

Instructor pointing at student

You WILL answer my question!

I pointed out that even I might be afraid to answer such a question. Such questions pose a severe challenge to the confidence of undergraduate students, because the instructor clearly knows the answer and they don’t.  The answer is a “fact” which the instructor clearly thinks the students should have already known before they came to class. When it comes to answering questions about “facts,” there are many ways to be wrong, but only one way to be right. When faced with this dilemma, students are understandably silent.

I suggested that he come up with nonthreatening questions: questions that didn’t put a student’s self-confidence and reputation at risk. Trying to begin a discussion with questions for which there is a “correct” answer makes salient the asymmetrical relationship between instructors over students. No one wants to look bad in the eyes of peers and so it is safer to say nothing and wait till somebody else answers or instructors give up and answer the questions themselves.

Is there a better way? What types of questions could you ask to begin a discussion? First, you could ask about something that everyone has seen or experienced. For example, in a class on the sociology of work, I asked students what is the best job they ever had and what were its characteristics? Everybody can answer such a question. Second, you could write a concept or principle on the board and then ask students to suggest examples. For example, you can ask for personal examples of a principle identified in the readings, rather than asking students to define the principle itself. In a class session on organizations and bureaucracy, I asked the students which aspects of the ordering process best exemplified what George Ritzer identified as the rationalization of fast food restaurants. It is critical in this process that you do not comment on the examples that are offered. Instead, simply compile the list. Third, you could ask students which previous class sessions or readings the day’s assignment reminded them of and why. Again, simply compile the list without editing it. Later in the discussion, after students have gained some confidence in participating, you can ask more difficult questions.

In addition to choosing questions for which there is no clear right or wrong answer, I follow several guidelines in getting discussions started. First, if at all possible, I use the whiteboard to keep track of student responses. Otherwise, trying to control the discussion process while at the same time keeping track of what is been said creates a high stress situation. You could also use an interactive polling system, such as Poll Everywhere.  Writing responses on the board clears your short-term memory and also gives you a few seconds to collect your thoughts while you decide on follow-up questions.

Second, I take care to make sure that if I’m writing answers on the board, I write them in the students’ own words. I rarely edit student responses and when I do, I always ask their permission. Writing the responses verbatim sends a strong signal that you are going to privilege student voices in the discussion, rather than just looking for confirmation of what you are going to tell them anyway.

Third, you should follow-up short or incomplete responses. Probe for more information by telling students that you need to make sure that you understand their meaning and thus you need a sentence or two from them. Point out that the few words you’ve written on the board might make little sense in a few minutes, when the discussion turns to assessing what’s been written in response to your original question.

When you are satisfied that you have responses that adequately cover the readings assigned for the day, then turn to editing what you’ve written on the board. This is a dangerous stage in the process, as it is here that instructors often hijack the process and begin selectively drawing from the writing on the board to give the lecture they had planned in the first place. Although the impulse to “correct” the unedited responses may be strong, don’t do it. Instead, turn to the students and ask them for suggestions for improving or simply using the list. For example, after I’ve written down the responses to my question about the characteristics of the best job students have had, I asked students if they see any pattern to the responses. The discussion naturally leads into various ways in which the dimensions of jobs can be analyzed.  In another example, after listing all the comments about ordering systems in fast food restaurants, I asked if the list reminded them of any of Ritzer’s principles and why. Note that it is okay at this point to begin probing for relevance to the day’s instructional goals, because you’ll have shown the students that you are listening to them, that their words matter, and that you take their views seriously.

The next time you find yourself stumped by why students are sitting passively in the classroom, seemingly unable or unwilling to answer your questions, take a hard look at your questioning strategy. Rather than sparking a discussion, the very questions you’re asking may be shutting down the process.

Save

Save

Save

Why Students Need Milestones & Small Wins

2 Replies

In my first year honors seminar, 5% of the grade is earned by making five posts on a webpage Forum. I added this to the course because I was searching for a way to keep the students engaged between class meetings. I invited students to comment on the readings, posts from other students, and anything else relevant to the course theme. I tried to reinforce their postings by commenting on those that I thought were particularly insightful. By mid-semester, I noticed that only about half the students had posted anything. I sent out an email to the entire class, reminding them of the requirement. Nothing happened.

As we approached the three-quarter mark, with about a month ago, I noticed that a few more students had posted. However, only a handful of the 24 students had come close to meeting the five post-requirement. To my amazement, many students had still not posted anything. I sent out another mass email, with an additional targeted group email to those students who hadn’t posted anything. The response was desultory.

With about two weeks to go in the semester, my disappointment deepened: three or four students had completed the requirement, about half the class had posted two or three items, and four or five students had still not posted anything. I made an announcement in class, reminding students that this requirement was sort of like “free money” in the sense that they got credit for simply making a posting, without any evaluation of its content. I then sent out another email, and this time I noticed that the number of postings began to increase.

At our penultimate class meeting, I made one last reminder of the requirement and encouraged students to set aside a few minutes to complete the requirement. I also sent personal emails to all the students who had posted nothing to that point. I was actually beginning to worry that so few students were going to complete the requirement that the grade distribution would be materially affected, as three or four points can make the difference between a B+ and an A-. (My friend Joe told me he would have stopped before going this far, as he saw my tactics as “coddling” the students.)

I watched the webpage intently over the weekend, and I noticed a few more people posting things. About one quarter of the class hit the five posting goal. But there were still many laggards.

On the evening before the last day, the posts finally began pouring in. Indeed, it was almost like watching a video game – – announcements were rolling into my email account, showing me an hourly tally. By the 5 PM deadline, most students had met the requirement. However, a few still had only one or two postings. One very surprised student discovered, after 5 PM, that he could no longer post to the webpage and emailed me. He ended up with only two postings to his credit.

What had gone wrong with my simple plan to increase out of class engagement with the course? When Melissa came to my office hours, I asked her why she hadn’t done the Forum postings until the end of the term and she nonchalantly replied, “It wasn’t high on my priority list.” She explained that with everything else she had to do, posting to the class Forum fell far down the list. Not only did she have work to do for her other classes but there are also lots of extracurricular activities to contend with, such as athletic events, concerts, and clubs. Other things were simply more important at the moment.

She could see that I was puzzled and volunteered an obvious – – to her – – solution: prevent students from waiting until the end of the term to fulfill the requirement by setting a target of one posting per month, which would result in five total postings. She told me that she was personally disappointed that students had waited, as she found the flurry of postings over the last couple of days of the class really provocative and wished that she had an opportunity to talk with the students in the class about their ideas in a more timely fashion.

As she was one of the five or six students who had also waited to turn in their term paper during the eight hour grace period on the last day, rather than the morning it was due, I asked her why she’d waited. She offered much the same explanation: she had lots of other stuff to do and had counted on the afternoon of that final day to allow her to finish up the proofreading of her paper.

Looking back, I realized that the milestones I had built into the course for completing various parts of the term paper assignment were simply not strong enough. Once again, Melissa volunteered a solution: set up stronger milestones and more closely assess compliance with the course requirements, rather than settling for “check plus” or simple peer review of outlines and drafts.

I’ll admit to being profoundly embarrassed by what I now realize was my failure to take account of the larger context in which my course was embedded. I had committed an elementary mistake inexperienced instructors often make: I thought that if something were important to me, it would also be important to the students. I had assumed I could motivate students by setting up incentives and creating a few simple milestones that allowed me to track students’ progress in meeting course goals. I had failed to account for the complex and overloaded life – as they perceive it — of today’s college students.

Students are confronted with an enormous variety of activities from which they must choose, and the priorities they follow don’t always accord with what we as instructors would prefer. The tasks we set for them are often overshadowed by much more immediate and pressing demands, including not only work for other courses but also their desire to live a richer social life now that they are on their own. Often, we are just not salient in the midst of more attractive options.

What to do? I suggest being much more mindful of the need for building frequent and graded milestones into your course that give you the opportunity to provide feedback on how well students are meeting course requirements. Management theorists talk about the power of “small wins” that give people a sense of making progress toward a goal. When people feel that they are making progress toward a goal, they feel more positive about the process and the positive emotions feed back into the amount of investment they make in the activity. Motivation increases and people began looking for the next small win in the process.

For example, when I enact milestones requiring monthly postings, I give students a periodic and highly visible reminder of the course themes. Because this particular requirement just assesses whether students have posted, rather than the content of their posting, it is also an easy win. An added advantage of requirements that involve highly visible activities is that students also gain public confirmation of their progress.

Small-stakes assignments also mean that students don’t put a lot at risk with any particular submission. Sim Sitkin described this strategy as one of “small losses.” Either way, it can be effective in motivating students to focus on completing assignments.

Term project milestones are bit more complicated, but they too provide opportunities for small wins. Students might simply get a check for turning in a proposed theme or a plan for researching the paper. Turning in outlines and drafts involves higher stakes’ assessments, and I believe instructors should provide fairly detailed written feedback for such assignments. Again, if it is to be a “milestone,” then students must not be given the impression they have passed the milestone until you, as the instructor, give them the go-ahead.

When students fail to keep up in our courses or turn assignments in late, we often accuse them of procrastination. But I’ve argued in this note that part of the problem is our failure to provide incentives powerful enough to motivate students to keep up. By building “small wins” into the milestones we set, we can rely on positive motivation, rather than draconian punitive measures, such as late penalties. If the milestones are simple and clear enough, students will do their assignments on time.

Save

Save

Save

Teaching a class? Be the 1st to arrive

1 Reply

When I leave my office and head for class, I’ve noticed that my step quickens noticeably. I feel excited about the class I’m about to teach and eager to share with the students what I’ve learned about the day’s topic. However, I also walk briskly because I want to arrive in the classroom before my students do. My goal is always to be the first to arrive so that I’m there to greet the students as they enter. This may seem like a minor detail in the greater scheme of all things pedagogic, but I actually have a well thought out plan for this tactic.

Workers in the Great Depression

Men arriving at work

First, arriving early gives me the opportunity to engage in small talk with individual students about the course and how things are going for them. Information gleaned from these discussions may generate a question that I bring to the general class discussion or may lead to modifications in an assignment I had planned. For example, students might point out some better contemporary examples of the principles discussed in a reading or they might have seen something in social media that’s worth mentioning to the whole class. I also find out about what’s happening in their other classes, which is information I can often work into our class discussion. Some students will be motivated to show up early because they want to hear what I’m telling other students – – they don’t want to miss anything.

Second, chatting informally with the students who have arrived early gives me a window into college life in general. Has sorority rush started yet? Is anybody in the class a candidate for election to student government? Are students excited about an upcoming appearance by our women’s soccer team in the conference tournament? Some of what I learn can be used subsequently as examples in the class discussion. For example, issue-oriented student groups are very active on my campus and their organizing tactics make for great examples in a social movement course. Generally, I get a good sense of the pulse of the campus and the rhythm of students’ daily lives.

Third, students are often willing to share things with me before class starts that they would not mention when their peers are seated around them. For example, did they find the readings for the day difficult? Why? Are they worried about meeting deadlines for the term paper? Revealing such anxieties to everybody, once the class is underway, can be a daunting experience for shy students. By contrast, as I walk around the room before class starts, engaging in small talk, I find they are more likely to open up and reveal such concerns. Speaking to students individually, or in small groups prior to the start of class, is particularly helpful for students who are nervous about speaking during a large class session.

Fourth, my students often use this time to ask me questions about how I spent the weekend, how I feel about recent political events, and so forth. Students often seem surprised that I have a life outside of the classroom, one that includes children and grandchildren. Sharing – – but not over-sharing – – some recent events in my life helps to humanize me as more than just their instructor. In my experience, this also helps in making students feel comfortable about coming to visit me during office hours.

Fifth, showing up early carries significant symbolic value. It signals to the class that you take teaching seriously and are prepared to put in whatever time it takes. When students realize that I will be in the classroom 15 minutes before the official start of class, they start showing up early as well. Often, almost all the students are seated several minutes before the “beginning” of class. Few people walk in late, where “late” is defined as not being seated at the official starting time. In contrast, were I to set a bad example by coming in just before class started, I would be encouraging my students to do the same.

Sixth, setting aside time in your daily schedule to leave your office early to head off to class will give you a bit of extra time for those unusual circumstances that sometimes disrupt your schedule. If a student visiting your office hours has an issue that needs more time or if an emergency phone call keeps you in your office for a few extra minutes, you will still arrive well in advance of the starting time. Do not treat arriving early as an option or you will end up finding the time occupied and you won’t ever get to class early. However, on rare occasions when “stuff happens,” you will still arrive on time for the start of class, given your new routine.

I strongly suggest you try out this practice at the beginning of the next term. Make it a routine practice for all your courses. Arriving 10 to 15 minutes before the published start time will make a dramatic difference in how much you know about your students, how much they know about you, and in creating a more comfortable and positive classroom atmosphere.

Professional meetings: act like you’ve been there before

1 Reply

Many professional associations have their annual meetings this time of year, and thousands of attendees will mingle in the corridors of super cooled hotels, many for the first time. When newbies ask me for my advice on what to do at these meetings, I’m reminded of a quote variously attributed to John Wooden, Bear Bryant, or Vince Lombardi: when you score, “act like you’ve been there before.” In other words, don’t make a fool of yourself. As a professional, you were supposed to score in that situation and so don’t make a big deal of it by excessive celebration.

Robert Davidson sculpture

Meeting at the Centre, by Robert Davidson

Now, very few interactions at professional meetings carry the same high-stakes outcomes as a sports competition, but Wooden’s advice is still relevant. Understanding what kinds of questions are appropriate and potential openings for productive conversations with more experienced participants is a key to having a good time. In this post, I review the kinds of questions that I think are conversation stoppers and clearly mark someone as either a first-time attendee or possibly socially inept. I then offer suggestions about more appropriate conversational openers.

At a national professional meeting, you are quite likely to meet a senior scholar whose work you have read and admired. You will be anxious to make a good impression on the person. You probably know that Professor Proteus works at Ivy League University, and so two questions immediately pop into your mind: “are you still at Ivy League University?” And “do you like it there?” The first question reveals that you appear out of touch with the current literature, as Professor Proteus’ affiliation is always prominently displayed right below her name in publications. The second question is fundamentally flawed in two respects. First, it can be answered with a “yes/no” answer, which gives Professor Proteus a chance to terminate the conversation and walk away. Second, if Professor Proteus has been at Ivy League University for several decades, what is she to say? “No, I hate it there, but I just can’t get a job anywhere else.” Or, “now that you mention it, I guess I should begin to look for better jobs elsewhere.”

Professor Proteus has probably answered this question dozens of times before, and thus she will resist the temptation to give a snarky answer. However, she will begin casting her eyes around the room, looking for more interesting conversational partners.

Another apparently obvious opening gambit has similar hidden dangers: “so, what are you working on now?” First, like the earlier question, it reveals that you evidently are not paying attention to recent literature, or else you would know what Professor Proteus is working on. Second, it gives Professor Proteus another chance to terminate the conversation, as she can say “oh, I have organized a session on the apocalyptic consequences of Brexit at this meeting. You should attend.” Third, the question is likely to remind Professor Proteus of those early days in her career when she was explaining to potential employers, over and over again, the contents of her research portfolio. There is no satisfactory short answer to such a question, especially if Professor Proteus has no idea how much you already know about her work.

Finally, a third problematic opening remark is more nuanced: “in my graduate seminar, we read your paper on the semiotics of transcendental strategic growth initiatives, and I didn’t really understand it. Can you explain it to me?” At this point, you will notice Professor Proteus glancing at her watch and making her excuses for why she has to be somewhere else as soon as possible.

What are some better ways of beginning a conversation with senior scholars whose work you have admired? First, after telling them your name and where you’re from, resist the temptation to immediately begin explaining in great detail what you’re working on, unless they ask you. Even if they ask you, keep your remarks short. Turn the conversation back to their work. Second, show them you are familiar with their work. A good opening question would be something along the following lines: in the last issue of JoIR (The Journal of Irreproducible Results), I read your paper on the semiotics of transcendental strategic growth initiatives and I found it really interesting. Are you working on a follow-up to that paper? I’d love to read a draft, if you are interested in circulating it, and send you comments if you wish.”

Professor Proteus will be pleased that you are aware of her recent work and that you didn’t ask her to explain it to you. More importantly, because she gets few invitations from others to make comments on her draft papers, your voluntary expression of availability might pique her curiosity. If she indicates even a tentative willingness to send you a draft, do not – – I repeat, do not – – ask her for her email address. Any competent user of Google these days can find her email address on the web and conversational time is better spent on more substantive matters. (This is another way you show that you’ve done this before.)

Third, if you attend the meetings pre-armed with a short list of people whose work you admire and whom you would like to meet, then go ahead and prepare a question or two for them. The question should not be answerable with a simple “yes/no” answer, and indeed, it should not have an obvious answer. Senior people hate questions with obvious answers, as it appears that you are just trying to flatter them. You will notice that their answers to such questions are quite short.

Here is an example of a possible challenging question: “I have always thought of your work as questioning our assumptions about the power of human agency, but I noticed that your recent work seems to back away from that view. In your recent paper in JoIR, your explanation heavily privileges not just collective but individual action. Am I reading you correctly?” Okay, I know this seems like it permits a yes/no answer, but it is not a simple answer – – the way you frame the question shows that you are familiar with her work and are looking for deeper explanations. Even if Professor Proteus has no time to engage you in a full exploration of this question, she may actually set up an appointment to meet with you later.

So, what are the principles underlying my advice? First, show familiarity with a scholar’s work. Second, ask more than simple yes/no questions. Instead, make the question challenging, perhaps by pointing out what you perceive as contradictions in recent work. Third, don’t act like a tourist. Questions about the weather and complaints about the hotel are best discussed with the concierge or doorman. Fourth, spend some time preparing a short list before you go, listing not only whom you’d like to meet but also the kinds of questions you’d like to ask. At the meeting, ask senior scholars whom you know for introductions, if you are too shy to introduce yourself.

However, if you follow the advice given here and avoid the nonstarter questions, you need not worry about making your own introductions.

Save

Save

What To Do After the Reviews Arrive

Leave a reply

Over the past decades, I have responded to more than 100 revise and resubmit requests from editors, served about 10 years as Associate Editor of the Administrative Science Quarterly, and reviewed hundreds of papers for dozens of journals. Closer to home, I’ve had the experience in the past year of responding to several tough R&R requests, and thus I decided to see whether I had learned enough to share some general tips with other authors. So, here are a few, with no claim to originality on any of them.

Ryōan-ji Temple

Ryōan-ji Temple, Kyoto, Japan. Only 14 of the 15 rocks are visible from all vantage points

First, just wait. After you finish reading the editor’s letter and the reviews, you might feel the need to “do something.” My advice? Don’t! Read the letters a couple of times to make sure you’ve noticed everything and haven’t spent more time on the negative then the positive reviews. Then, put them aside and just think about them for a day or two. Or longer.

Second, moderate your emotions. It is natural to feel annoyed, irritated, and even angry about a letter that points out the flaws in your work. However, letting your anger get the best of you will lead to unprofessional behavior which you will later regret. Given that the letter is a revise and resubmit request, there will be sentences in the reviews that say positive things about your work. Relish them and then move on. Similarly, there will be sentences in the reviews that lead you to question the intelligence and motives of the reviewers. Such thoughts are not helpful. They block rational thinking about the strategic course of action you need to take.

Third, thank the editor. The letter probably asks you to indicate to the Journal whether you plan to resubmit and if so, by what date. I never turn down such invitations. Even if the letter says something like “meeting the reviewers’ comments will require major changes in the manuscript,” the fact that the editor thinks you have a chance is reason for celebration. So, don’t complain in the message you send back to the Journal; just tell them that you’re grateful for the opportunity to revise the manuscript and you will be returning it within 30, 60, or 90 days, or whatever time frame it is that you’ve been given. Editing is mostly a thankless job — give the editor a break.

Fourth, do not send the manuscript unchanged to another journal! Occasionally I hear friends and colleagues tell me that the effort to meet the revision requests is not worth it and they’re just giving up and sending the paper to another journal. That’s a big mistake. Why? First, for most journals, far less than half of the authors submitting papers are given opportunities for revision. Somebody likes your work. Second, the odds of an acceptance skyrocket for most journals, once the paper has been given an R&R. For some journals, the odds approach 50% that a revision will be accepted. Whatever the number, if you cared enough about the Journal’s reputation to submit to it in the first place, you’ve now got a much better opportunity to publish in it and so you shouldn’t turn it down. Third, and perhaps most important, it is quite likely that at least one of the reviewers who told the editor that the paper needed work will be a reviewer for a subsequent journal to which you send the paper. Nothing angers a reviewer more than to learn that all the work put into a review has been ignored by an author who has chosen to send the unchanged paper elsewhere. Sending the paper elsewhere without revising it is likely to elicit a strong rejection or at least one really strong negative review!

Fifth, consider the possibility that the reviewers were right. Lots of research in cognitive neuroscience tells us that people consistently overestimate the value of something they have produced, as well as being wildly overconfident that their work is above average. Don’t be one of those people. Take the reviews to a colleague and, without biasing their response with a negative cue, ask them to tell you what they think of the reviews. You will be surprised at the response. Although your colleagues can put themselves in your shoes as a spurned author, they probably have had much more experience as a reviewer than as an author (judging by the average colleague’s CV), and so their sympathies are quite likely to lie with the reviewers. Listen to what they have to tell you.

Sixth, make a plan. Every revise and resubmit editor’s letter asks an author to not only revise the manuscript but to include a document that indicates how each of the comments made by the reviewers was dealt with. You can get a head start on that document by using the reviews to create a plan for revision. First, fit the plan to the comments. If the points in the reviews aren’t already numbered, number them. Keep track of which reviewer said what by giving each reviewer a letter, e.g. A, B, and C. Second, indicate whether you accept the criticism and can do something about it (adding literature, clarifying language, conducting new analyses, and so forth) or you can’t. If you can’t do something about it, indicate why. Third, clearly indicate which of the comments will require moving words around (theories, concepts, models, etc.) versus which will require new analyses. Externalizing your thoughts in this way will show you whether you truly understand what the editor is asking you to do. (You can write back and asked for clarification of murky points.) It also gives you a sense of the magnitude of the effort required to meet the comments. This document will be both a plan for revising the manuscript and the template for the letter you will write to the editor.

Seventh, don’t lengthen the text. Almost all papers that are submitted are close to or over the ideal word limit set by a journal. Although you may claim that you can’t deal with the reviewers’ comments without adding more words to the text, that is surely not true. Unless you had hired a professional editor before submitting the paper, there are many places in the text where you will find redundancies, unnecessary digressions, and so forth. Use the revision opportunity to shorten the text, or at least to ensure that it is no longer than the original submitted.

Eighth, be generous. Avoid gratuitous insults to the editor or reviewers in the letter you of explanation write. Although it is tempting to claim to an editor that the reviewers misunderstood your manuscript because they’re out of touch with the literature or have inferior reading skills, you’ll gain no profit by doing so. Avoid effusive praise, but do thank the reviewers when they genuinely pointed out something that you had missed. In simple declarative prose, explain to the editor and the reviewers how you understood each of their points, how you responded to it, or why you were not able to.

If you follow these tips, I cannot guarantee you that your resubmitted manuscript will be accepted. However, I can guarantee that your professional reputation will emerge intact and you’ll live to fight another day. You will also feel better about the process.

Save

Organize literature reviews by ideas, not authors

Leave a reply

A few days ago I received a draft manuscript from some friends who asked for comments. The manuscript was prepared for a handbook meant to summarize the state-of-the-art in an emerging field and thus was intentionally focused on reviewing the literature and identifying trends. I first checked the references and saw that they had included what I expected. Therefore, the review was certainly up to date. I settled in for a good read.

Wilhelm Hofer

Wilhelm Hofer “Medir a Coherencia da Vitalidade.” Santiago de Campastella

The first paragraph announced the paper’s purpose and then laid out the plan of the paper. It was organized into thematic sections, with short headlines identifying the themes. No surprises so far.

However, as I read through the paragraphs, I noticed a pattern that has become all too familiar to me over the years, in my roles as reviewer, editor, and commentator on other people’s work. The first sentence of nearly every paragraph began by mentioning a particular author’s work, often with the author’s name the first words in the first sentence. For example, “Smith (2014) studied 94 entrepreneurs in the process of beginning their first ventures, focusing on how they picked the industries they wanted to enter.” Subsequent sentences provided more detail on what Smith had done. Occasionally the paragraph included mentions of other authors, but often the entire paragraph was devoted to a single author.

In the paragraphs that followed, the pattern recurred. Each told us about a particular author or authors and their work. Sometimes the authors were strung out in a narrative that presented the work chronologically, but often the organizing principle wasn’t clear, beyond the fact that all the authors in this section could be grouped under a common theme.

By the third page, I was ready to throw in the towel. The authors were asking me to do the work that they should have done, before beginning to write the paper: group the papers reviewed by concepts and principles and then use topic and concluding sentences to tell us why the work reviewed is important. Rather than organizing the review by authors’ names, they should’ve organized it by ideas and concepts.

Thus, the paragraph that now began with “Smith (2014),” should have instead begun with “Most studies of how entrepreneurs choose the industries they want to enter have found that entrepreneurs enter industries in which they have worked, rather than taking a chance on coping with unfamiliar environments.” Smith’s contribution could then have been placed into the context set by a strong topic sentence. For example, Smith might have conducted a particularly valuable study because it included a large nationally representative sample and well – documented indicators of the concepts under investigation. By contrast, Jones could’ve been mentioned in the same paragraph as conducting a study that seemed not to support the topic sentence’s empirical generalization but, in fact, the study should be discounted because of fatal flaws in methodology.

Literature reviews that follow a simple narrative structure of serially presenting author after author are easy to write but do little to advance our understanding of how to interpret previous research and how much confidence we should place in it. As I’ve mentioned in other blog posts, a better strategy is to read all the relevant works and write interpretive notes on them. Then, the interpretive notes should be sorted into conceptual categories and further ordered along the dimensions that emerge from a close reading of the notes. For example, one dimension might be “research designs used,” and another might be “theoretical perspectives framing the research.”

The resulting emergent conceptual categories and principles should then be used to organize the literature review, with authors’ names subordinate to the higher-order principle of capturing central tendencies and the extent of diversity in the literature.

So, the next time you read a literature review and notice that nearly every paragraph starts with an author’s name, ask yourself, did the authors really do the tough analytic work required to add value to the literature? If not, perhaps here is an opportunity for you to step in and do more! Organize your own review by ideas, not authors.