Download Understanding Research Methodology: The Key to Quality Evidence and more Lecture notes Public Policy in PDF only on Docsity!
3
Research in the
Real World
do Methods Matter?
We want to do things in our lives and in our work to make a difference in the world—to educate children, treat or prevent sickness, reduce crime and violence, promote the arts, feed the hungry, house the homeless, or improve public services to communities. We share a desire to do something meaningful, to leave our mark in the world. But doing so requires a base of evidence beyond our own personal knowledge and experience—evidence about how things really are, and evidence about how to make things better. We need such evidence not only to enhance our own understanding and decision making but also to convince others—those with the authority and resources that we need to accomplish our aims, or those with opposing points of view who stand in our way.
Good Evidence Comes From Well-Made Research
The best evidence comes from good research. Good research can appear in the form of a study pub- lished in a journal, but it can also be an internal analysis of administrative data, a government or foundation report, a performance measurement brief, a program evaluation, a needs assessment, or a client or employee survey. Government agencies collect and disseminate a great variety of empirical evidence on many important topics, such as health services and outcomes, educational attainment, labor market characteristics, crime victimization and punishment, housing conditions, environmental air and water quality, and so on. (See www.fedstats.gov to get a flavor of all that is available from the U.S. federal government alone.) Because of the Internet and modern communications technology, we now live and work in a world in which an abundance of studies and statistics swirl all about us and hover within easy grasp— provided we know what to choose, how to make sense of it, and where to apply it.
44 PART I :: FOUNDATIONSPART I :: FOUNDATIONS Good research—just like a good car or a good pair of shoes—must be well designed and well made. But we cannot simply rely on brand names (although knowing that research comes from a respected scientific journal or reputable research institution does provide some assurance). Still, each study is unique, and each has unique strengths and weaknesses. So we need to understand how research is made—that is, research methods or methodology. Methodology refers to the sampling strategies, measurement instruments, comparisons, statistical techniques, and other procedures that produce research evidence. So we need to understand meth- odology to judge the quality of research. Research methodology is what this book is all about.
May the Best Methodology Win
We also need an understanding of research methodology to attack evidence that hurts our cause or defend evidence that helps it. Consider the controversy over abstinence-only sex education for teenagers. Some communities feel strongly that teens should be discouraged as much as possible from engaging in sexual activity and that comprehensive sex education (which can involve distributing condoms and instructing teens in their use) sends the wrong signal. Others warn that the abstinence-only approach does little to change the reality of teenagers’ lives, leaving them vulnerable to unwanted pregnancy and sexually transmitted diseases (including AIDS). Sex education in schools has spurred controversy—and research. Source: iStockphoto.com/IsaacLKoval.
66 PART I :: FOUNDATIONSPART I :: FOUNDATIONS it. New York City’s COMPSTAT program—a data-driven effort to closely track crime and hold manag- ers accountable for controlling it—is an often-cited example. The mass of data available today, thanks to the information revolution, fuels this trend. Performance measurement has now become a pillar of contemporary policy and practice in the public and nonprofit sectors (Hatry, 2007; de Lancer Julnes & Holzer, 2008; Poister, 2003). In the following chapters, you will see how logic models can help you figure out what to mea- sure. You will learn what makes for valid and reliable measurements. And you will be introduced to various sources of data to measure outputs and outcomes, including both existing data and original surveys. All this material is critical to understanding and implementing performance mea- surement and management.
Evaluation Research
Many program evaluations aim to answer the question: Did the program or intervention have an impact? Did it improve or change things? Other program evaluations seek to describe the process of implementing the program. Evaluation research is now a standard requirement of most government or foundation grants and contracts. Most new policy or management initiatives demand some form of evaluation as well. So evaluation research, too, has become a pillar of contemporary public policy and management in the public and nonprofit sectors (Rossi, Lipsey, & Freeman, 2003; Weiss, 1997). But how can we know if a program or initiative is having its intended effect? Later chapters will introduce you to the basic ideas involved in thinking about cause and effect. They will cover strategies for estimating causal effects, including the use of control variables, randomized field experiments, and various forms of what are called natural and quasi experiments. These are the major strategies for conducting evaluations of program impact.
Evidence-Based Policy and Programs
As suggested earlier, governments and foundations increasingly favor evidence-based policies and programs—strategies that have proven their effectiveness with research. It’s not enough anymore to have a few heartwarming testimonials or a plan that just looks good on paper. The trend toward evidence-based policy and practice now permeates many fields (Davies, Nutley, & Smith, 2000). Due to limited resources, policymakers and practitioners must often choose between effective programs. Therefore, comparing the effectiveness of different programs is crucial, as is comparing cost-effectiveness —the outcome obtained relative to the cost of the program. Such comparisons require evidence about the magnitude of a program’s effect—how large an influence a program has on the outcome. The chapters that follow will give you tools to identify and assess evidence that backs up your program or initiative. And it will help you understand how to produce good research evidence to sup- port your aims.
Chapter 1 ::Chapter 1 :: Research in the Real WorldResearch in the Real World 77
Evidence Can Mislead
On top of all that we’ve mentioned so far about the importance of research methods, it can be embar- rassing to be wrong—and sometimes, if you’re not careful, evidence can mislead.
Misleading Measurements
“No Child Left Behind” (NCLB) was signed into law in 2002, setting in motion a wave of reform in schools all across the United States that became suddenly preoccupied with high-stakes testing, wor- ried about closing the race gap, and apprehensive about the need to demonstrate extraordinary gains in test scores. NCLB won support in part because of the “Houston Miracle,” the fact that this large, diverse city had itself demonstrated remarkable gains in reading and math scores, especially for Black and Hispanic students—at least according to scores on the Texas Assessment of Academic Skills (TAAS). If Houston could do it, so could the rest of the nation. But scores on another test—the Stanford Achievement Test—taken by the same Houston students during the same school years showed a much different picture, according to an analysis by the New York Times (Schemo & Fessendent, 2003). Scores on the Stanford test, which is used nationwide, showed little or no gain overall in Houston and little or no narrowing of the race gap. Several well- known experts in education statistics, asked by the New York Times to review the discrepancy, con- cluded that the TAAS had considerably overstated the progress made by Houston students. Standardized tests do not necessarily provide a consistent measure.
Misleading samples
In response to a recent Supreme Court decision on gun control, A USA Today (2008) quick poll asked visitors to its Web site, “How many guns do you own?” About 30,000 people die in the United States each year from gun-related accidents or violence (Centers for Disease Control and Prevention, 2008), so the high rate of gun ownership in the United States is an important public health as well as crimi- nal justice issue. But how high is the rate of gun ownership? A total of 1,987 people responded to the USA Today quick poll, and the results showed that fully 89% owned a gun—it seems the United States is awash in guns! But the quick poll relied on a voluntary sample—Web site visitors who found the article online and decided to participate in the poll. When the General Social Survey (GSS) asked a sample of 1,996 adults if they owned a gun, only 35% reported that they did. Which survey do we believe? It turns out that the GSS uses much better methods—including careful random sampling—to produce its results. The true rate of gun ownership in the United States is certainly much closer to 35% than it is to the strikingly high figure of 89%. The flawed methods of the USA Today (2008) quick poll grossly overstated gun ownership in the United States. The solid methods of the GSS get us closer to the truth.
Chapter 1 ::Chapter 1 :: Research in the Real WorldResearch in the Real World 99 collected by others, such as existing government surveys, administrative records, or transcripts. Indeed, much primary research gets done using secondary data. Unfortunately, the term data also can be a bit confusing. If we looked up a few published facts or even a table of statistics online or in the library, we sometimes refer to this as finding “data” on a topic. But in this book, we use the term data to refer to largely unprocessed observations— raw data , it’s sometimes called. We now turn to some of the key features of research, particularly as it applies to policy and practice.
It Comes in Various shapes and sizes
As you will see from the many examples throughout the chapters in this book, research comes in a surprisingly wide variety of shapes and sizes:
- Large-scale studies of broad populations
- Small-scale studies of one locally situated group
- Snapshots in time
- Studies of outcomes or events that occur over many periods of time
- Laboratory simulations
- Naturalistic observations of real-world settings
- Carefully planned manipulations
- Theoretical analyses
- Opportunistic discoveries of unplanned events
- Informal research conducted for the purpose of organizational strategy or management One of the important points to realize about research, and about researchers, is that inventiveness and creativity are an important part of the process. Good research often involves the imaginative application of new methods, innovative techniques, or clever strategies to learn about the world. It is this creative aspect of research methods that makes the topic so interesting to those familiar with it. We will try to give you a flavor of this variety and creativity in the chapters to come.
It’s never Perfect
Research, like everything else that is human, is not perfect—far from it. Every study has weak- nesses, as you will learn in this book. It is important to spot these shortcomings and understand their implications. But it is also important not to entirely discard a study because it has some methodological or other shortcomings. We don’t want to throw the baby out with the bathwater. Every study also has strengths, too—or at least, most studies do. There is often something to be learned from almost any study, and the perfect study is just not possible—especially in social and policy research. A good consumer of research can both spot the weaknesses and recognize the strengths.
1010 PART I :: FOUNDATIONSPART I :: FOUNDATIONS
It’s uncertain and Contingent
We tend to think of research as providing certain and universal conclusions. But experienced research- ers know otherwise—especially those involved in social and policy research. Research evidence includes a large dose of uncertainty , often expressed in the form of probability statements or qualified conclusions. Thus, researchers talk about the results “indicating” this or that, “suggesting” that some- thing is true, or showing that an outcome was “likely” due to a presumed cause. In part, this comes from the language of modern statistics, which uses the laws of probability to make inferences about unknown populations or causes. But this way of speaking and writing also reflects the inherent uncer- tainties involved in making firm statements or conclusions about complex human and public affairs. Social and policy research is also contingent —bounded in space, time, and context. A study that finds evidence for the effectiveness of an education reform in one school district, for example, may not hold true in other districts with different children, teachers, budgets, and administrative struc- tures. A mental health intervention that is shown to be effective with affluent suburban adults may not have the same effect on poor, inner-city adults living much different lives. The motivations found to encourage productivity in one organization may not be the same as the motivations that matter in another organization.
It Aims to Generalize
Generalizability is the ability to take the results of research and apply them in situations other than the exact one in which the research was carried out. Although we just noted that research is often contin- gent, researchers nevertheless strive at the same time to make their work generalizable. This is quite important: If the research results only apply in the exact setting (time, place, circumstances) in which the study was conducted, then they cannot be used to inform policies or practices in other situations. For example, suppose a study looks at a policy of requiring copayments for emergency room visits and finds no impact on health outcomes for patients. But say the study is done using data from one insurance plan that covers mostly younger, healthy workers with good incomes. Do the results apply to insurance plans that cover older, less healthy individuals with low incomes? Probably not: Such individuals might well behave differently if required to make copayments for their visits to the emergency room. So the study has limited generalizability. We might even worry that the study is only relevant for that one particular insurance plan and the population it serves, making it of little use to anyone else. While generalizability is always a goal, real-world research is often less generalizable than we would like. This is not to say that social and policy research has little to offer—on the contrary. But you do need to be realistic and appreciate the limits, as well as the rewards, of research.
Bits and Pieces of a Puzzle
It’s also true that a single study is almost never definitive. Rather, empirical evidence on a topic is cumulative. Research produces a body of evidence, and researchers talk about arriving at a scientific consensus within the bounds of what is likely to be true (or not).
1212 PART I :: FOUNDATIONSPART I :: FOUNDATIONS the interpretation of language can be every bit as rigorous and scientific as quantitative studies— despite the lack of scientific-looking tables and formulas. Numbers do not make a study good or scientific. These days, social and policy research often uses mixed methods that combine the advan- tages of both quantitative and qualitative techniques. Because social phenomena are so difficult to pin down, researchers often use multiple methods to confirm a finding, a process referred to as triangulation. Although more of the chapters in this book are devoted to topics typically thought of as part of quantitative research, we discuss the role and contribution of qualitative methods in all the chapters. And we devote an early chapter to qualitative research because we consider it to be foundational. In an important sense, good quantitative research is based on good qualitative research. The two per- spectives enhance one another.
Formulating Research Questions
Research answers questions—but what kinds of questions, and where do they come from?
How the World Is—not How It should Be
To begin with, it is important to understand that research questions are positive —about how the world really is; they are not normative —about how we want the world to be. Consider again the debate between those who favor an abstinence-only approach to sex education for teenagers and those wanting comprehensive sex education (including the distribution of condoms). Research is well suited to answering the positive question of whether these approaches do, or do not, do anything to reduce the rate of sexual activity and sexually transmitted disease among teenagers. Research does not help much with answering the normative question of which approach is the right thing to do, given our moral beliefs or community values. However, doing research does not mean giving up your values. In fact, you can use research to support or create policies that further your goals and reflect your values. For example, if you care about the homeless, you can choose to study the homeless—who they are, how they became home- less, and what programs are most effective at improving their well-being. Values can legitimately shape what one chooses to study. But researchers try hard not to let their values lead them to sup- press or distort their findings to conform to their normative beliefs. Good research gives us a window on reality—even if we do not always like what we see.
Applied and Basic Research
Research questions—particularly those in applied research —often come from a practical need to know. For example: How many people are currently unemployed? Would smaller classes improve learning? Does adding police officers reduce crime? The answers to questions in applied research
Chapter 1 ::Chapter 1 :: Research in the Real WorldResearch in the Real World 1313 typically have direct implications for policy and practice. Most of the examples in this book focus on applied research. In contrast, basic research is often thought of as the pursuit of knowledge for its own sake, rather than being based on an immediate practical need. Basic research in a given field also tends to focus on more abstract or fundamental processes of nature or society. For example, we might be interested in studying how people make decisions involving uncertainty, how the human body responds to long- term exposure to stress, or how children acquire a language. Basic research also advances policy and practice by providing a solid foundation of knowledge. But the link is less direct.
Questions We Ideally Would Like
to Answer, and Those We Really Can
Although practical needs generate pressing questions, research frequently cannot answer exactly the question we might ideally want answered. Often, we must settle for an answer to a related question—an approximation of sorts. But it is better to have an approximate answer than none at all. For example, suppose we want to describe lab tests among those with diabetes in the United States. Due to the fragmented nature of the U.S. health care system, there is no one source of medi- cal records representative of all people with diabetes in the United States. Instead, a study might be done on people over 65 with diabetes using Medicare data, while another study might get records on younger adults from a private insurer operating in one region of the country. Neither study would generalize to the entire U.S. population of diabetics—far from it. So the best that can be done is to conduct several studies that collectively might help us approximate an answer. Consider another example: Does divorce have long-term detrimental effects on children, psycho- logically or educationally? Because we cannot rewind the clock and see what would have happened to these children, had their parents not divorced, this question is hard to answer directly. We could compare differences between those whose parents divorced and those whose parents did not, as in a study by Huurre, Junkkari, and Aro (2005). But because these groups may be different in important ways, this comparison answers a different, but still valuable, question. A study by Gruber (2004) used changes over time in state divorce laws to determine the long-term effects on children of making divorce easier. But this, too, is a slightly different question from our desired question, because we still aren’t sure if the detrimental effects on children can be attributed to divorce itself or perhaps some other consequence of the changes in law. And Gruber’s study was unable to consider some of the psychological outcomes observed by Huurre and colleagues (2005). Flexibility and compromise are required when translating a policy question into a doable research question. To make good use of research, it is important to understand what question a study addresses, and how it might differ from the one you ideally want answered. In Chapter 15, we return to the differ- ence between applied policy and practice questions and the often more limited, approximate answers that research can provide. We will look at how those differences shape both research and its effect on policy. Also in Chapter 15, we offer some suggestions on developing your own research question, which similarly involves a trade-off between what you ideally want to answer and what you can answer, given available data and resources.
Chapter 1 ::Chapter 1 :: Research in the Real WorldResearch in the Real World 1515 evaluating how to react to autism, policymakers and practitioners need to know how many autistic people (and particularly autistic children) there are in the population. They need to know if the rate of autism is growing—and if so, how quickly. They need to know whether autism is more concentrated in certain places or groups in the population. They need to know the severity and forms of autism. In other words, policymakers and practitioners need a good description of autism to address the problem. The goal of descriptive research is to paint an accurate picture of the way the world is. Descriptive research includes describing just one variable—such as the rate of autism in the population. It also includes describing relationships —how two different variables are related. Relationships are often referred to as associations or correlations. For example, autism rates have been growing—so time and autism are related. Or at least it seems so—researchers worry that perhaps we have simply gotten better over time at identifying those with autism and that this enhanced ability to identify the disease accounts for the upward trend. Autism and geographical region are also related—the disease is more common in California, for example, than in other parts of the United States. But it turns out that this description is also not so certain—perhaps autism is not consistently identified everywhere. Descriptive research can be harder than you might expect. Before figuring out what to do about a problem like autism, the problem must be described. Knowing the lay of the land is important before deciding where to go. But once practitioners have described the problem, the task of tackling and solving it has just begun. After all, we want to figure out how to make things better—not just sit and watch things happen. In the case of autism, policymakers and practitioners want to figure out how to prevent and hopefully cure, or at least ameliorate, the disease.
Causation: How Would the World
Be different If something Changed?
The goal of causal research is to answer “what if?” questions to find out how to make things happen. Specifically, if we change one thing, will other things (outcomes we care about, such as autism) change? And if they do change, by how much? For example, what would happen to the severity of the disease if autistic children stopped eating gluten? Would it change at all? If so, by how much? Or what would happen to autism rates if children stopped being vaccinated?^1 More generally, we want to know what factors have caused the growth of autism over time (if indeed the trend is real and not just an artifact of better identification techniques). Descriptive and causal research are both important in practice, but answering causal questions is especially central to the work of practitioners. Public policies, social programs, and management initiatives aim to do things—to make something happen. So answering questions such as “What will happen if we do X?” is essential.
Causal Research needs Qualitative Research
Qualitative research is also very important to causal research. We often get ideas about possible causes from case studies or qualitative observations. And qualitative methods do an especially good (^1) See the chapter “Vaccines and Autism?” in the book by Arthur Allen (2007), which addresses this controversy.
1616 PART I :: FOUNDATIONSPART I :: FOUNDATIONS job of uncovering and documenting causal processes or mechanisms—for example, the precise ways in which a program influences participants. Quantitative, statistical studies do a good job of demonstrating a connection between two vari- ables, such as a program and an outcome, but have less to contribute to our understanding of what process or mechanism was responsible for how the program influenced the outcome. Good causal research often needs the insight from qualitative research.
don’t Confuse Correlation With Causation
It is easy to confuse correlation, the existence of a relationship, with causation. If more educated mothers are more likely to have autistic children—a correlation—then it is easy to conclude that something about educated mothers causes autism. However, that may not be so. Think about the earlier example of fluoridated water and fractured bones. When researchers, policymakers, or practi- tioners naively assume that a correlation implies causation, grave errors can be made. For example, because autism rose over the same period that vaccine use rose, and because autism symptoms start at about the same time that toddlers receive many vaccines, many concluded that vaccines cause autism. Many parents started to reject vaccines, causing some outbreaks of previously suppressed illnesses. One of the most important skills you will gain from this book is how to distinguish a correlation, the description of a relationship, from evidence of a causal effect. We address this in Chapter 10, our chapter on causation. Another, perhaps even more important, skill is how to judge the quality of evidence of causation and how to do research that provides evidence of whether a causal effect exists and how big it is. Because distinguishing description from causation is so important, we have organized this book around that distinction. Part II of the book covers strategies for description, while Part III covers strat- egies for causation. We will stress again and again the distinctions between description and causation and between correlation and causation. Let’s look now more closely at what we call the scientific method as one of several ways that we have of knowing about the world.
Epistemology: Ways of Knowing
How much do you weigh? How do you know that’s how much you weigh? Probably you used a scale and remember the result. You measured your weight—an elementary act of research. How high is Mount Everest? If you know, how do you know? Did you measure it? If you don’t know, how might you try to learn how high Mount Everest is? You will probably turn to other sources, perhaps search- ing the Internet and examining a Web site that you trust (secondary research). But of course, you should consider how the Web site got its information. We have many ways of knowing—what philosophers of science call epistemologies. Sometimes we directly learn something ourselves. But we can’t do that about most things in the world. Often, we just accept what some trusted authority says is true. Sometimes, we rely on
1818 PART I :: FOUNDATIONSPART I :: FOUNDATIONS researchers might develop a theory that predicts when a planet will appear where in the night sky. They then use telescopes to make observations of the planet and to see if its appearances agree with their prediction. In social research, some researchers called structuralists insist that social research must always start with theories and test these with empirical predictions. Most researchers, however, practice a combination of induction and deduction. They have a the- ory, gather data to test it, but then also examine the data in other ways to develop new theories.
Proof Requires Fresh data
While most researchers practice a combination of induction and deduction, even in a single study, the structuralists have a point that applies broadly in research: Data cannot be used to both develop a theory and definitively confirm it.^2 Fresh data are required to truly test a theory or support a hypoth- esis. Think of your favorite detective novel. The detective struggles to come up with a theory of the crime or mystery that fits all the existing clues. Then the detective uses the theory to predict some previously unknown and unsuspected fact—mud on the perpetrator’s evening dress, for example. If the prediction indeed matches the fact, the perpetrator’s guilt appears much more likely. Prediction provides powerful proof. (^2) We cover this issue in more detail in Chapter 8. It is related to issues of statistical power in hypothesis testing. Logical explanation (theory, models, hypotheses) Systematic observation (measurement, sampling, experiments) Problem or curiosity Induction Induction Deduction Deduction Figure 1. 1 Induction and Deduction
Chapter 1 ::Chapter 1 :: Research in the Real WorldResearch in the Real World 1919 In social and policy research, some studies strive to generate theory, while others aim to test theo- ries. Research is often an iterative process in which deduction and induction alternate (as Figure 1. illustrates).
Truth in social science: Controversy and Consensus
The scientific method originated with physical sciences such as astronomy and physics. The discover- ies of Newton and Galileo are early examples. Natural and biological scientists were also early users of the method. The more isolated the study is from real-world complexities, the better the scientific method works. For example, the scientific method works better for physics and biology than for engineering and medicine. Nevertheless, the scientific method is responsible for enormous strides in both engi- neering and medicine. Social phenomena are more complex and varied than physical or even biological phenomena due to factors such as human self-awareness, personality, and culture. Social phenomena vary by place and time much more than physical or biological phenomena. So truths in the social and policy sciences, such as how markets work or how children learn, are much grosser approximations of truth than in engineering or medicine—and far from the more universal truths of physics. Moreover, how we interpret social phenomena is shaped by language and culturally constructed categories. These categories shape our interpretation and even what we observe, and our construc- tions vary from time to time, from culture to culture, and from political perspective to political per- spective. So even when we try to be objective, our interpretations will be at least somewhat conditioned by our categories of subjective experience and judgment. Because social ideas and facts are constructed, some reject the relevance of the scientific method to the study of society and policy. Indeed, some even reject the idea that an objective truth, even a contingent truth, exists for social phenomena. We acknowledge that all real-world social and policy research will have limitations, based on the researchers’ interests and perspectives, and will only get at part of the truth. But we prefer to take what is probably best described as a critical pragmatic point of view in this book. That is, we seek feasible methods and practical knowledge that will help us improve some problem or condition in the world. In doing so, we believe that it is important to try to be objective (as best as we can agree), because that is how we will find the most useful information. When study- ing social phenomena, the scientific method can serve as an ideal—even if social and policy research does not always live up to this ideal.
Approaching Research From different Angles
You may encounter research from various angles: as a consumer of research findings in news articles, government reports, or journal articles, as someone who commissions research to satisfy a pressing information need, or as someone who conducts research on your own or as part of a team. This book addresses all these perspectives.
Chapter 1 ::Chapter 1 :: Research in the Real WorldResearch in the Real World 2121
Conducting Research
When we think of a researcher, we tend to envision a tweedy professor in a university or a white- coated scientist in a laboratory. If this is your calling, then of course you must know research methods especially well. Indeed, a solid grasp of research methodology can help a new scholar become more productive, succeed at publishing, and participate more fully in his or her field of study. But not everyone who conducts research these days fits this traditional mold. On the contrary, increasingly, applied research of various kinds is being conducted in large and small government agencies, nonprofit organizations, foundations, advocacy groups, the news media, and a growing industry of consulting firms that support the research needs of public, nonprofit, and business orga- nizations. Some of these applied researchers have PhDs or other doctoral degrees, but many do not. Indeed, there are quite a few master’s and even undergraduate programs that provide research and analysis skills sufficient to begin a career as an applied researcher or policy analyst. Moreover, there are many situations in which practitioners engage in informal research. Examples include doing your own survey of employees or clients; making comparisons using per- formance measures; examining the effectiveness of a new program or management initiative using administrative data; and doing qualitative interviews or focus groups for purposes of internal strat- egy, marketing, or decision making. These activities are research, too, even though the people doing them do not aim to publish in peer-reviewed journals or release results in fancy research reports. Knowledge of research methods can dramatically improve one’s ability to do informal research well.
Ethics of Research
Social and policy research raises important ethical issues because it deals with human beings—their health, living conditions, rights, and well-being. In the chapters to come, we deal with these ethical concerns in the context of particular methods and study types covered in this book. For example, qualitative research presents unique ethical issues that we address in Chapter 3; the use of existing administrative data raises confidentiality concerns that we cover in Chapter 6; and experiments involving human beings require many ethical considerations that we discuss at length in Chapters 10 and 12. However, there are certain core principles that are worth reviewing here as a preview of what lies ahead. These principles come from an important document known as the Belmont Report (available from the U.S. Department of Health and Human Services at www.hhs.gov/ohrp/humansubjects/guidance/ belmont.htm).
- Respect for persons dictates that people used as the subjects of research provide informed consent and are not coerced into participating in research.
- Beneficence dictates that people who participate in research are not harmed and, indeed, that they should realize some benefit from the research.
- Justice requires consideration of equity among subjects and fairness in regard to who becomes a research subject.
22 PART I :: FOUNDATIONS The history of applied policy research has some ugly chapters. In the infamous Tuskegee syphilis study, for example, the U.S. Public Health Service recruited poor African American sharecroppers with the disease into a long-term study and then left them untreated for many years—even after penicillin was found to be an effective cure—to study the progression of the disease. The Belmont Report came out of this experience. In Chapter 14, we cover the history and practice of ethics in research in more detail.
Conclusion: The Road Ahead
The ideas and concepts of research methodology come from many different disciplines—sociology, economics, the health sciences, and education, to name a few. As a result, it is less a neatly ordered landscape than a somewhat tangled and overgrown woods. So as you travel the road ahead, we will try to clear away the brush along the way—yet still preserve the variety of ideas and concepts that you will find in the many disciplinary journals and reports of research. It might comfort you to know that even experienced research methodologists, talking across disciplinary boundaries, often do not understand one another because of the many dialects of research methodology. This communication gap is unfortunate, of course, but it is part of the real world of research. Thus, an important skill to have, both as a researcher and a consumer of research, is to be able to see through the tangle of terms and to get a good view of what issue or idea is really at stake. We begin our journey in the next chapter with an introduction to theory and models —the concep- tual tools researchers use to think about the world and to begin to figure out how to study it. ExErcisEs
Battleship Research
1.1. You saw in this chapter how research gets used in battles over controversial public poli- cies, such as sex education or global warming. Can you think of other, important policy debates in which opponents use research to support their arguments? How has research methodology played a role in these debates?
Research in the Corner office
1.2. We made the case in this chapter that the ability to judge and apply research evidence is an important qualification for top management and leadership positions. Identify this kind of position in an organization or agency in your area of policy or practice. In what ways does the person in this job use or commission research? If possible, inter- view the person.
