Immortal Time Bias in Medical Studies, Study Guides, Projects, Research of Nursing

Download Immortal Time Bias in Medical Studies and more Study Guides, Projects, Research Nursing in PDF only on Docsity!

Problems and

Pitfalls in Medical

Literature

A Practical Guide for Clinicians

Adam L. Cohen

Problems and Pitfalls in Medical Literature

Adam L. Cohen Medical Oncology, Internal Medicine Inova Health System Fairfax, VA, USA ISBN 978-3-031-40294-4 ISBN 978-3-031-40295-1 (eBook) https://doi.org/10.1007/978-3-031-40295- © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and trans- mission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publica- tion does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Paper in this product is recyclable.

v

Preface

Although I have spent many years teaching medical residents and fellows about evidence-based medicine, the need for a resource on identifying problems in articles was crystallized for me at a national clinical trial meeting some years ago. A clinical trial was being proposed based on a recent article that had made a big splash in popular media after being published in a respectable medical journal. I stood up to note that the article in question had a major methodologic flaw and when that flaw was corrected, the conclusion was exactly the opposite of what the paper claimed. The speaker replied that different people had different opinions about the article. Now, there are many things one can have opinions about but statistics and method- ology, in general, are not two of them. When a study is done badly, it is done badly, and no amount of hope or opinion can fix that. This book is for medical professionals who have had some exposure to evidence- based medicine and who wish to improve their skills at reading medical literature. During medical school and residency, we pay attention to certain aspects of evidence-based medicine. We teach students to frame a question, search pubmed for articles, identify the population of an article, identify the type of article, and use statistics to calculate effects. These are important, but less attention tends to be paid to assessing the quality of an article beyond the advantages of randomization and blinding. Assessing quality is not just an essential skill for reading articles for clini- cal purposes. It is also an essential skill for being a peer reviewer. Lastly, in statistics classes, much attention is paid to how to calculate various tests, like t-tests or chi-square, which many people find frightening. You will find no formulas in this book. I hope you find that fact comforting and the topic therefore less scary. When tests are discussed, it will be to note whether or not they are used properly, not to calculate them. There are statisticians with computers for a reason. This book is not for them. This book is for people who need to read medical scien- tific articles and decide whether what they are reading is worth remembering or not. To approach assessing the quality of articles, some advocate a checklist approach. Checklists are useful for identifying if an article answers the question one is asking. They are less helpful for looking for fatal problems. To bowdlerize Tolstoy, good articles are all alike, but every bad article is bad in its own way. Checklists can lead to the view that one just needs enough checks to be good or that all checks are equally important, neither of which is true.

vii

viii

• 1 Lead Time Bias Part I Biases
  • References
• 2 Length Time Bias
  • Reference
• 3 Immortal Time Bias
  • Reference
• 4 Selection Bias
  • References
• 5 Ascertainment Bias
  • References
• 6 Publication Bias
  • References
• 7 Berkson’s Bias
  • Reference
• 8 Hawthorne Effect Part II Time Effects
  • Reference
• 9 Will Rogers Phenomenon
  • References
• 10 Confounding by Severity Part III Confounding
  • References
• 11 Simpson’s Paradox
  • Reference
• 12 Multiple Comparisons Part IV Misuse of Tests
  • Reference
• 13 Validation
  • Reference
• 14 Proportional Hazard Violation
  • Reference
• 15 Changing Endpoints
  • Reference
• 16 Underpowering Part V Power
  • References
• 17 Overpowering
  • Reference
• Conclusion
• Index

3 1

Lead Time Bias

We have all been taught the dogma that for diseases like cancer, early detection coupled with early treatment saves lives and makes people live longer. Even for diseases other than cancer, we now recognize presymptomatic states, such as pre- diabetes or borderline hypertension, that can lead to early intervention to prevent complications. For people who have had cancer, there is high allure for tests that will both reassure that the cancer is not back but also pick it up early when it comes back, with the idea that treating it earlier will be better. One test that at first seems to fit the bill is a blood test called CA-125 for detect- ing ovarian cancer. When ovarian cancer is detected at an advanced stage, it is treat- able but most of the time is not cured and repeatedly recurs until the person dies from the cancer. Thus, if there were a test that could detect the cancer early, before it spreads, it could be treated with a potentially higher cure rate. Moreover, it makes sense that treating recurrences before they get too big would be more successful than treating them when they are large and causing symptoms and may have devel- oped new mechanisms of resistance. CA-125 is a protein released from ovarian cancer cells that correlates very highly with the overall tumor mass in the body. Therefore, people have looked many times at using CA-125 to both screen for ovar- ian cancer and to monitor for recurrence in people who have been treated for ovarian cancer. The CA-125 is such an ingrained part of ovarian cancer monitoring that people with other cancers, like breast cancer, often ask their doctor why they aren’t being monitored with such a blood test. Lead time bias refers to the artifact of apparent improved survival when measur- ing survival from the time a disease is detected. Even when there is no advantage to diagnosing or detecting a disease early, people will survive longer from the time of diagnosis when a disease is detected in an earlier, asymptomatic state. The time between asymptomatic detection and symptomatic detection is called the lead time. As an example of lead time, consider two otherwise identical women, whom we will call Alice and Betty, who both develop identical ovarian cancers at the same time in January (Fig. 1.1). For both of them, the cancer doubles in size every month, and when the cancer reaches a size that the CA-125 is above 1000, they will die of © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 A. L. Cohen, Problems and Pitfalls in Medical Literature , https://doi.org/10.1007/978-3-031-40295-1_

4 1200 1000 800 600 400 200 0 JanuaryFebruary March April May June July August SeptemberOctoberNovember Detectable by CA- Detectable by symptoms Death CA- Fig. 1.1 Disease course for Alice and Betty. The thick lower arrow shows Betty’s survival from the time her recurrence is found. The thinner upper arrow shows Alice’s survival from the time her recurrence is found the effects of the cancer. Alice lives a happy life for 8 months and then in October develops symptoms and is diagnosed with ovarian cancer. At that time her CA- is about 250. Alice has a negative attitude and does no treatment, so her cancer con- tinues to grow, and she dies 2 months later of ovarian cancer, in December. Betty has her CA-125 drawn monthly for unknown reasons. In May, the CA-125 reaches a level where it is detectable, and Betty is diagnosed with ovarian cancer. Betty has a positive attitude and believes she can will the cancer away, so she has no other treatment. Like Alice, her cancer continues to grow at the same rate, and she also dies in December of ovarian cancer. If we were to compare how long they lived from the time of diagnosis of the cancer, it looks like Betty must have done some- thing right, because she lived 7 months from diagnosis, while Alice only lived 2 months. That difference is misleading because in reality the difference is wehn we detected the cancer, not anything Betty did to impact the growth rate of the cancer. The 5 months that Alice knew about her cancer before Betty was the lead time. To avoid erroneously concluding that Alice’s good attitude and willpower extended her life by 5 months, we need to make sure we are measuring survival from a time point that does not depend on when the cancer was detected. Two examples of publications on ovarian cancer illustrate the dangers of lead time bias and how to avoid them. The Prostate, Lung, Colorectal, and Ovarian (PLCO) Cancer Screening Trial was one of the largest trials of cancer screening ever conducted in the United States [ 1 ]. 78,000 people, give or take, were

1 Lead Time Bias Part I Biases

7 2

Length Time Bias

The goal of screening for any disease is to identify the disease in an early presymp- tomatic state when it can be treated or cured more easily than if it becomes symp- tomatic. Thus, in an ideal screening program, a simple test is done that detects an otherwise fatal or morbid disease at early stage. The medical profession then swoops in with a simple and nontoxic treatment for that early disease, curing the person who then goes on to live a long, happy life full of gratitude toward the doctors who saved their life. Everyone gets to feel good. Lung cancer is an attractive target for screening. As the most common cancer and most common cause of cancer deaths in the United States, lung cancer is a major cause of morbidity and death. We know the major causes of lung cancer, making identification of people at higher risk easier. Lung cancers start as single tumors in the chest that grow slowly and asymptomatically for a while and can be cured with surgery if caught early enough. (Here “early enough” refers to stage 1 cancers. Lung cancers are given a stage based on how far they have spread. Stage 1 lung cancer is a single small tumor. Stage 2 is a larger tumor. Stage 3 is a tumor with cancer in regional lymph nodes. Stage 4 is cancer spread to other parts of the body that is now incurable.) Thus, the goal of lung cancer screening is to do an imaging study on asymptomatic people to find stage 1 lung cancer and perform surgery to remove the cancer. It would seem, therefore, that showing that people who have such surgery don’t go on to die of lung cancer would be evidence that the screening is effective. Length time bias refers to the tendency of screening to detect indolent disease, meaning disease that is so slow that it would not cause a problem in a person’s life- time if left untreated. It is called length time because the diseases that are around for a long time (have high length time) are more likely to be detected by screening. Imagine you have ten people with lung tumors (Fig. 2.1). Some are super-fast grow- ing (thick arrows). These are very aggressive tumors that spend very little time as stage 1 and will become symptomatic quickly, so they will never be detected by screening. Some are medium growing (thin arrows that then become thick). These are tumors that start out slowly and spend time as stage 1 before speeding up and spreading. When these medium growing tumors are detected as stage 1 by © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 A. L. Cohen, Problems and Pitfalls in Medical Literature , https://doi.org/10.1007/978-3-031-40295-1_

8 Fig. 2.1 Illustration of length time bias. More people with slower disease have disease to be found at the time of the screenings. (Figure adapted, with permission from one created by Dr. Saundra Buys) screening, the situation is like our ideal program discussed above, where screening leads to an intervention that saves a life. These medium growing tumors are in people whose lives are saved by screening. The rest of the tumors are very slow growing (thin arrows). They stay small and stage 1 for a long time, maybe even forever. These tumors may grow so slowly that people may die of other causes with- out ever even knowing they had them. In this hypothetical group of ten people, three had super-fast tumors, three had medium tumors, and four had slow tumors. However, at any given time, most of the people have slow growing tumors, because people with fast growing tumors die quickly. The vertical lines in the figure indicate two different time points when screening can occur. At the first screening time point, there are four people with tumors, of which three are slow growing (people 6, 8, and 9) and one is medium growing (person 7). At the second screening time point, there are five people with tumors, three of which are slow growing (people 6, 9, and 10) and two are medium growing (people 1 and 5). This difference in frequency of slow growing tumors, where the proportion of the screen-detected tumors that are slow growing is higher than the proportion in the general population, is length time bias. As an example of length time bias, consider the I-ELCAP (International Early Lung Cancer Action Program) study, published in the New England Journal of Medicine [ 1 ]. From the title of the paper, “Survival of Patients with Stage I Lung Cancer Detected on CT Screening,” we know that we are looking at how long peo- ple with early disease detected by screening are living, which is a trigger to look for length time bias and how it is handled. The paper reports the results of two rounds of screening with low-dose CT scans. The screening program detected 484 lung 2 Length Time Bias

11 3

Immortal Time Bias

As I was standing in my kitchen one morning making my breakfast of bran flakes and banana, my child came into the room and asked me, “Hey dad, do you know the best way to live to be one hundred?” As thoughts about non-smoking, exercise, and good genes flashed through my mind, they dead panned, “Eat an apple a day for one hundred years.” This rim-shot deserving “dad joke” illustrates immortal time bias, which occurs when the exposure being measured, in this case eating apples for one hundred years, occurs over time. Immortal time bias is also known as the Oscar bias, after the annual awards given by the Academy of Motion Picture Arts and Sciences. It is well documented that actors who win the award for Best Actor live longer, on average, than those who don’t. One could imagine several reasons for this observation. For example, one could hypothesize that the gold in the Oscar statuette has properties that make the air in an actor’s home healthier. Alternatively, perhaps winning an Oscar inspires actors to stop drinking, smoking, and partying and focus on their health. Perhaps one might think that some people are genetically superior with genes that both pro- mote long life and a talent in acting. The truth is much more mundane, alas. People who die young have less chance to win an Oscar. Consider, for example, James Dean, who had acclaimed roles in East of Eden , released 1955, Rebel Without a Cause , released 1955, and Giant , released 1956. With the promise shown in these roles, he had the potential to win many Oscars. Unfortunately, he was killed in an automobile accident in late 1955. It would certainly be impossible for him to have an Oscar winning role after that. Contrast Dean with Henry Fonda, another iconic actor of the twentieth century. Fonda appeared in 86 movies over the course of his career, including classic performances in Twelve Angry Men and Mister Roberts , to name just two. He won the Best Actor Oscar for his performance in On Golden Pond , at the age of 76 years and 317 days. Less than 6 months later he died. Clearly, the Oscar statuette did not prolong his life. Rather, he happened to live long enough to be able to win an Oscar. Immortal time bias is this reversal of causation, where we think that the exposure (winning an Oscar) makes people live longer, when it is in fact the living longer that allows people to win the Oscar. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 A. L. Cohen, Problems and Pitfalls in Medical Literature , https://doi.org/10.1007/978-3-031-40295-1_

12 Imagine ten actors, all of whom would have the role of a lifetime that wins them an Oscar at the age of 50 (Fig. 3.1). Some, like James Dean, die young and never reach 50. Others, like Henry Fonda, may die immediately after winning their Oscar. As you can see from the figure, even with no causative relationship between win- ning an Oscar and lifespan, people who won the Oscar will on average live longer, because they can’t win unless they have lived to be 50. The time they have to live to have the exposure is called immortal time, because the person could not have died during this time. In medical studies, immortal time bias often shows up when looking at the rela- tionship between side effects and efficacy. From a physiologic point of view, because both side effects and efficacy of a drug are likely related to achieving sufficient level for needed target engagement, it makes sense that people who have side effects may have greater efficacy from a drug. However, since it takes time for side effects to show up, people who have side effects will also be those who necessarily take the drug for longer. Thus, the time between when someone starts a drug and when the side effects show up is an “immortal time” that can introduce bias. Immune therapies have revolutionized the treatment of deadly cancers like meta- static melanoma, lung cancer, or bladder cancer. Antibodies that block PD1, the programmed death protein 1, or its ligand, PDL1, activate T cells that then can attack and kill cancer cells, causing tumors to shrink and go away making people live longer. This is a good thing. Unfortunately, these T cells can sometimes get confused and attack normal tissue, causing autoimmune or immune-related side effects, such as colitis, hypothyroidism, etc. For several years, researchers have wondered if the presence of autoimmune side effects predicts benefit from immune therapy. Most studies have compared the survival of people who had autoimmune side effects with those people who don’t. This situation has a great risk of immortal time bias. As my daughter would say, the best way to live a long time after being Born Age 50 Immortal time Fig. 3.1 Illustration of immortal time among actors who would win an Oscar at age 50 3 Immortal Time Bias

14 12 months as the landmark time. Sometimes it is more arbitrary. The landmark time needs to be long enough to allow sufficient time for the exposure but not so long that too many people have died before the landmark is reached. The second downside is that people can continue to have events after the landmark. Thus, someone who has an autoimmune event 4 months after the start of treatment is included with people who never have an event. This misclassification may bias us toward not seeing a difference that we may have seen if we had picked a longer landmark. The second approach for immortal time bias avoids the problem of an arbitrary landmark by allowing people to move from one group to another over time. Thus, someone who has an autoimmune event 4 months after the start of treatment would be considered to be in the no autoimmune event group for the first 4 months and in the autoimmune group after that. Such an analysis is called a time-dependent expo- sure or a person-year analysis. Time-dependent analyses work well when the timing of all events and potential confounders is known. In some retrospective studies, unfortunately, there is not enough data to complete a time-dependent analysis. Once you are aware of immortal time bias, you will start to see it all over the place. Does removal of the primary tumor make women with metastatic breast can- cer live longer? Does removing the ovaries of women predisposed to breast cancer reduce the risk of breast cancer? Does prescription of a statin reduce the risk of lung cancer recurrence? All of these questions are affected by immortal time bias. Anytime you can rephrase the question, “Does the occurrence of X delay the occur- rence of bad thing Y?” as “The best way to avoid bad thing Y is to not have it happen long enough for X to happen”; you have the potential for immortal time bias.

Reference

1. Maher VE, et al. Analysis of the association between adverse events and outcome in patients receiving a programmed death protein 1 or programmed death ligand 1 antibody. J Clin Oncol. 2019;37(30):2730–7.

3 Immortal Time Bias

15 4

Selection Bias

They say the grass is always greener on the other side of the fence. In medicine, this feeling often manifests as, “Why do my patients not do as well as the patients in clinical trials?” or “Why do my patients not do as well as those reported by the big, famous academic center?” As an academic who runs clinical trials, I like to think the answer lies in the quality and structure of the care I provide. That’s likely true, but the answer also lies in something just as significant. Selection bias. Selection bias refers to the differences in outcome that result when the patients in a study are not representative of the general population. Multiple attributes can feed into selection bias, whether we are talking about a prospective study or retro- spective studies. People who have good mobility will live longer than people who don’t. People with normal organ function (liver, kidney, heart) will live longer than those with low organ function. People who have more indolent cancer that doesn’t grow out of control while they are being screened for studies will live longer than those who don’t. People who have the social and economic support to be able to travel for treatment will live longer than those who don’t. People who have had fewer side effects with prior treatments and therefore feel able to do more treatment will live longer than those who don’t. People who have good nutrition will live lon- ger than those who don’t. Et cetera et cetera. In short, sometimes consciously and sometimes subconsciously, researchers and clinicians tend to select for studies peo- ple who have a better prognosis, regardless of disease status, particularly when those studies involve more aggressive treatment. Selection bias is particularly a problem for single-arm studies where a group of people are compared to “historical controls” or even contemporary controls. Indeed, when controls are contemporary, the role of selection may be even more powerful because if two people were treated at the same time but received different treat- ments, why was that? Even when we try to match for disease characteristics or a few other variables, it is hard if not impossible to account for everything, whether appro- priate or not, that goes into clinical discussions and decisions. One of the longest and saddest examples of the impact of selection bias comes in the treatment of breast cancer with high-dose chemotherapy with autologous stem © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 A. L. Cohen, Problems and Pitfalls in Medical Literature , https://doi.org/10.1007/978-3-031-40295-1_