10 Ethics

Data science is both an academic and an applied discipline. It is taught in universities, and its techniques are widely practiced in industry, government and other walks of life. In its academic mode, issues of ethics and ethical conduct arise primarily in terms of research. In the United States at least, the basic ethical framework for research in data science is a product of two government reports—the Belmont Report and Menlo Report. Those reports, and thus the framework, were a response to an egregious and tragic ethical failure known as the Tuskegee Syphilis Study or Tuskegee Experiment. Reviewing the history of that experiment is important instrumentally, because it helps us understand where our current pratice of research ethics comes from. But it is also important more broadly, as an example of an injustice that both reflected and perpetuated class and racial discrimination. That is, the Tuskegee case reminds us that everything we do in data science has potential consequences beyond ourselves.

Some of the material in this chapter is adapted from Matthew J. Salganik’s book Bit by Bit: Social Research in the Digital Age, which provides a much more detailed discussion of the research principles and frameworks, as well as several special problems in the digital age outlined here. We recommend his book to all who are interested in reading further.

10.1 Tuskegee Syphilis Study, 1932-1972

The study began in 1932 with the recruitment of around 600 men, of whom around 400 had syphillis. This disease is sexually transmitted, and can have devastating consequences for those affected. The study was run by the US Public Health Service and participants were informed of this fact.

Several other features of the experiments are worth noting:

it was non-therapeutic. This means that there was attempt to treat the men; the object was, instead, to observe the effects of the disease in the subjects.
the men, who were primarily poor and Black, were told they would receive (free) health care. In practice though, they were offered false and ineffective treatments.
the men were also initially told that the length of the study would be 6 months. In practice, it lasted for some 40 years.
at the beginning of the study, there were no effective treatments available for syphilis. But this changed with the advent of antibiotics. Yet the men were actively prevented from obtaining that treatment by, for example, preventing them being drafted to the military where they would have received appropriate medical care.

By the late 1960s, given its deep moral failings, pressure built to end the study. But these attempts were rebuffed by the Public Health Service itself. Members of the relevant panel convened to discuss the experiment. They recognized that the study was unusually ethically dubious, but argued that this made it rare and thus worth continuing. In addition, while panelists agreed that informed consent was important for medical research, they claimed that the relevant subjects in this experiment were too uneducated to give it.

Ultimately, the press discovered the details of the experiment and it ceased in the early 1970s. President Clinton publicly apologized for the treatment of the subjects in 1997 saying

What was done cannot be undone, but we can end the silence. We can stop turning our heads away. We can look at you in the eye, and finally say, on behalf of the American people, what the United States government did was shameful and I am sorry.

10.1.1 Consequences of Tuskegee

Of course, the Tuskegee experiment affected its subjects negatively. But it also affected the subjects’ families, who were exposed to the disease. In the longer term, it decreased trust in the medical community, especially from Black citizens. Experience with the experiment affected subsequent public health efforts to, for example, treat HIV/AIDs in the 1980s, and Covid-19 in more recent times.

The ethical failings also lead to bureaucratic and legal responses: specifically, the Belmont Report (1974) and Menlo Report (2012) that provide the basic framework for university human subject research today.

10.1.2 Belmont Report (1974)

The Belmont Report was a response to the Tuskegee Study, and with the Menlo Report…

it defined principles for research, as opposed to practice. In the Belmont Report, research is concerned with creating generalizable knowledge as opposed to everyday treatments.
it laid out three ethical principles that must be considered when doing research. These are Respect for Persons, Beneficence and Justice. The Menlo Report added Respect for law and public interest.
provided the framework and guidance for all academic research in the US in terms of an Institutional Review Board (IRB) that is charged with reviewing research proposals in terms of the potential ethical considerations.

10.1.2.1 Challenges of the Modern Data Science Era

As we have argued elsewhere in this course, data science is a fast moving area. This poses problems in the ethical domain. In particular, it is often unclear how our ethical principles should be applied. This all the more difficult as we scale up our ability to gather data on humans, often without them knowing. It is not simply that this causes problems for privacy and informed consent, it also that we run the risk of unanticipated secondary use. That is, researchers can take (huge) human subject datasets and in way not expected by the original researchers, use them for malign purposes. We will return to such issues below. First though, we lay out the principles of research ethics.

10.2 Principles and Philosophy: Four Principles of Research Ethics

The four principles of research ethics are:

Respect for Persons: this is the recognition that people (subjects) are autonomous and their wishes should be respected
Beneficence: research must strike the right balance (particularly for the subjects) between risks and benefits
Justice: the risks and benefits of research more generally must be distributed fairly across society
Respect for law and public interest: beneficence for all stakeholders affected by research, beyond the subjects themselves

We cover each in turn.

10.2.1 1. Respect for Persons

The basic idea is that subjects should be treated as autonomous. By this we mean that it is the subject themselves, not the researchers, who decide what happens to them and their lives. This is important even if the study is harmless or beneficial to the subjects. Though not identical, the central idea here is informed consent. That is the notion that

potential subjects receive relevant information in a comprehensible format followed by a voluntary agreement to participate.

In typical case, subjects should expect to get information on

purpose of the study
what the study does/its procedures
how long the study lasts
the risks of the study
the potential benefits of the study

Sometimes it may be permissible to deceive subjects, about the purpose of the study or what the treatment is. Typically this is done when revealing this information would be incompatible with the experiment per se. For instance, we may want to see how subjects respond to some surprise stimulus and its important they don’t anticipate and prepare before “genuinely” reacting to it. In these situations we do not hold the researchers to the same informed consent standard as usual. However, after the experiment we would expect to debrief our subjects: to tell them what the experiment was about, and answer their questions. This also provides an opportunity to guide them to resources should they feel negatively affected by the procedures.

To the extent a study deals with subjects who have diminished autonomy, it should expect to use greater protections for them. That is, for “vulnerable populations”—who are inherently less able to make informed and free decisions about costs and benefits to themselves—we should expect to take more steps in an attempt to follow the Respect for Persons principles.

Examples of such groups include:

Minors
Prisoners
Less educated people
Non-native speakers of whatever language the informed consent information is written and in which the experiment is being conducted
HIV/AIDS patients

Perhaps unsurprisingly, there is a troubling history of experimenting on such people without proper thought about whether they can truly consent to being subjects. See for example, the Statesville Malaria Study.

10.2.2 2. Beneficence

Beneficence is the principle that, as regards a study, one should

maximize the possible benefits and minimize the possible harms

Notice that there is no requirement to do no harm: indeed, if one could not impose any risks at all, many important experiments (including vaccine trials) could not happen.

Generally, the Beneficence is assessed in two linked stages: first, a technical assessment is made as regards the potential risks and benefits to subjects. Second, an ethical assessment is made as to whether the technical risk/benefit ratio is normatively reasonable.

10.2.2.1 Assessing Risk (Technical)

Risk is a product of

probability of adverse event \(\times\) severity of adverse event

The two parts of this equation can be affected by changes to different parts of the experiment design, and the trade-offs may not be trivial.

The other important issue here is that risk is not simply to subjects: as in the Tuskegee case, their are potential affects on non-participants and social systems around the experiment.

10.2.2.2 Assessing Ethics

Once the technical assessment is done, the ethical assessment can take place. The framework we choose for this assessment is important and some simple approaches are obviously problematic. For example, it may be tempting to use a Utilitarian framework that allows the research design to go ahead if benefits are greater than costs. But this is dangerous: we generally think certain designs—even if they have large potential benefits on average—are just morally wrong (perhaps because they impose costs that are too large on some participants). To provide checks on this, IRBs bring in researchers from outside the field of the experiment to avoid “group think” on what is or is not appropriate.

Given the above, it is somewhat ambiguous as to how the modern era of data science studies is affecting levels of Beneficence. On the one hand, we can imagine that larger scale data gathering and expectations of sharing increases benefits because the same data can be used for multiple projects across institutions. On the other hand, this also adds to the risk of subsequent exploitation by researcher with malign intentions.

10.2.3 3. Justice

The idea of Justice is that the broader benefits and burdens of research should fall in a fair way on society. It is different to Beneficence in that Beneficence is focused more narrowly on the cost/benefit analysis of a particular experiment for those who participate in it (or are close to those that do). Justice about is making sure that we don’t exploit historically disadvantaged groups for the benefit of others.

It takes two main forms. Historically it meant no exclusively experimenting on marginalized groups (like poor people, minority populations, or children) for the benefit of the most powerful. A representative case might be a poor African American woman, Henrietta Lacks, whose visited Johns Hopkins hospital in 1951 and ultimately underwent a biopsy. Her cells were collected, preserved and used for research. Lacks never gave informed consent for this, nor was she compensated.

Today, Justice is often more concerned with not excluding groups from studies such that the lessons of those studies can help them too. An example would be the historical exclusion of women from experiments on cardiovascular health.

In any case, Justice implies that paying subjects appropriately is a reasonable requirement.

10.2.4 4. Respect for Law and Public Interest

Beneficence concerns the risks and benefits to participants: Respect for Law and Public Interest makes this idea broader, and extends it to other stakeholders. It has two main parts: Compliance and Transparency-Based Accountability.

10.2.4.1 1. Compliance

Compliance says that, generally speaking, researchers should try to identify and follow the relevant laws, contracts and terms of service. That is, doing research does not give one the automatic right to break legal rules. Followed to the letter though, this may be quite restrictive. For example, in a 2017 study, Kevin Munger created Twitter bots to automatically ask tweeters using racially offensive terms to stop doing so, with various justifications for this request. The use of automated bots is, in fact, not allowed by Twitter’s terms of service. Nonetheless, the study was approved by the relevant IRB, presumably because the scientific merits outweighed the risks.

A more controversial example is provided by a set of researchers from Stanford and Dartmouth who sent out mailers to Montana voters with information on an upcoming election. The aim was to see if receiving such a mailer made one more likely to vote. There were various concerns raised with the study, but one particular compliance issue was in terms of the State of Montana’s Seal. This may technically have required explicit permission to appear on the mailer, which the researchers did not ask for.

10.2.4.2 2. Transparency-Based Accountability

As its name suggests, the central idea here is to open about the goals, methods and results of our studies. That is, to take responsibility for our experiments, including mistakes that are made. Recent discussions of two older (but well known) studies make this point:

In the Stanford Prison Experiment, students were recruited to act as either guards or prisoners, with a view to understanding how behaviors change after simply being ascribed as a given role in a social situation. Separate to the fact that the experiment was stopped early for ethical reasons, there have been allegations that the study authors in fact coached the “guards” to produce particular results. The lead investigator, Philip Zimbardo, denies these claims.
In the Rosenhan Experiment, a team of researchers acted as if they had hallucinations such that they were admitted to mental hospitals. They then acted normally, but were diagnosed with psychiatric illnesses, and given medication for treating the same. The results of the experiment lead to skepticism about the reliability of psychiatric diagnosis in general. In recent times, however, allegations have been made that, in fact, Rosenhan fabricated data (including about himself). For example, Rosenhan used himself was a psuedo-patient (itself a questionable decision for a researcher) and appears to have presented to the relevant doctor as having much more severe symptoms than his subsequent write-up implies.

10.2.5 Ethical Frameworks

Ultimately, observing our ethical principles depend on particular frameworks: that is, particular ways to thinking about how to trade-off costs and benefits. There are two broad schools of thought:

Consequentialism: in which we ask about the ends of the study
Deontology: in which we focus on the means to the ends of the study

10.2.5.1 1. Consequentialism

In Consequentialism whether a study is ethical or not depends on its ends: its consequences. The classic example is Utilitarianism, associated with philosophers like Bentham and Mill. Here, the idea that

an action (a study) is ethically permissible if it improves the world (net of any costs)

Beneficence, which requires us to explicitly consider costs and benefits, has a utilitarian feel. Obviously though, Consequentialism taken to extremes might lead to undesirable outcomes. For example, it might mean that subjects should expect to die with high probability, because the study will lead to very large benefits for (at least) some members of society.

10.2.5.2 2. Deontology

Deontology says that a study is ethical if

we act in an ethical way while performing the study.

In that sense, it is not depend on the consequences of the experiment at all: all that matters is the morality of how it is carried out. This idea is associated with Kantianism (the efforts of Emmanuel Kant). Respect for Persons is a deontological principle: we must respect autonomy because it the right and moral way to treat subjects, whatever our end goals. Like Consequentialism, Deontology can be taken to extremes: it might mean, for example, that we could never use deception in a study because we deem lying to subjects to be morally impermissible.

Obviously, a given experimental practice can be justified by either/both schools of thought. For example, we might think that informed consent is good in the Consequentialist tradition because it leads to subjects (and thus experimenters) thinking carefully about costs and benefits. Meanwhile, we can justify the same practice under deontology because giving people autonomy is axiomatically the right thing to do. That said, there are many situations where the schools clash on what is permissible, and this clashing is not easily resolved.