Playing sports sparked my interest in science. In fact, one of my first science fair projects was testing the effectiveness of caffeine on athletic performance.

It was in 1983, and I had some of my friends run an obstacle course three times. I timed them, then a few days later made them drink a Coke and run it again. The data I recorded demonstrated an increased performance on the task.

I was certain of the data. To me as a young scientist, data was black and white. Your hypothesis was either correct or wrong.

But the truth is science is not absolute, and data, although measurable, is open to interpretation. How we interpret data and when we find consensus are what gives it meaning. There is an art to science, and often the differences in how we interpret data causes controversy.

I was sad to learn of Kevin Turner’s death in March. I remember watching him play for the New England Patriots. Turner was a great player and an even greater person. A recent article on CNN revealed that Turner suffered from CTE in addition to ALS.

The CNN article quoted some interesting statistics, stating that professional football players are four times more likely to have ALS and three times more likely to die from neurodegenerative diseases like ALS or Alzheimer's.

The numbers CNN uses are derived from a 2012 paper published in the medical journal Neurology. The findings sound alarming, but it’s only one interpretation of the data and maybe not the correct interpretation.

Let's do the math

To understand the reported increased deaths from ALS and Alzheimer’s of NFL players, we have to talk about a little bit of math. Sorry.

When a scientist is trying to determine if a study’s population is dying at a different rate of a disease than everybody else, the scientist will compare the number of deaths from the disease of the study group to the number of deaths from the disease in the general population.

To see if the two groups are dying at different rates, something called a standardized mortality ratio (SMR) is calculated. This is essentially a ratio of the number of deaths in the study group to the numbers of deaths in general population. A SMR is a way to express how many deaths would be expected in the study group based on the number of deaths in the general population.

If the calculated SMR is greater than 1, then the deaths from the disease in the study group occur at a higher rate than the general population. If the SMR is less than 1, then the deaths in the study group happen at a lower rate than the general population.

It seems pretty straightforward, but we’re not done. A calculated SMR might be greater or lesser than 1, but is it a significant difference?

Enter the wild world of statistics. For data to have meaning, there must be a mathematical difference between the groups being studied. This difference is called “significance.”

A significant result means that the data is really different and could not have occurred by random chance or error. A non-significant difference means that although the findings may show difference between the groups, it’s not enough to say with any certainty that the difference occurred by any anything other than chance or error.

The statistical test for the significance of a SMR is whether or not it is different from 1. To determine the statistical significance of the SMR, the researcher will calculate a 95 percent confidence interval, which describes the uncertainty associated with the collected data. It means that if the same population is sampled on numerous occasions, you’d get the same result 95 percent of the time. It tells us we are 95 percent sure that our finding is correct.

With SMRs, if the 95 percent confidence interval includes the value 1, it is not statistically significant, meaning there is no statistical difference in the numbers of deaths from a disease between the study population and the general population.

Now, back to the numbers

OK, with all that out of the way, let’s take a look at the paper published in Neurology that investigated the causes of death of 334 former NFL players.

CNN reported that NFL players are three times more likely to die from Alzheimer’s or other neurodegenerative diseases. The authors of the original study quoted by CNN reported that there were two deaths directly related to dementia/Alzheimer’s disease in the NFL population. The calculated SMR for this group was 1.80. But the 95 percent confidence interval was 0.22 to 6.50, which includes the value 1, so there is no statistically significant difference between the number of deaths among the NFL population and the general public in regards to dementia/Alzheimer’s disease.

The authors also looked at the number of deaths from Parkinson disease in the NFL population, of which there were two. The SMR was 2.14, but the 95 percent confidence interval was 0.26 to 7.75 – meaning once again there is no statistically significant difference in the number of deaths from Parkinson disease in the NFL population and the general public.

Things get a bit more complicated when the authors looked at the number of deaths related to ALS. The authors report a SMR of 4.04, which CNN tells us is four times more common in the NFL players than the general population. The 95 percent confidence interval was 1.48-8.79, which does not include the value 1, so it is statistically significant. However, there is a caveat here: three of the ALS deaths included in the study were from a strange cluster of ALS deaths which occurred within players on the 1964 San Francisco 49ers.

Typically, ALS affects two out of every 100,000 people. For three cases to occur in one year on a single team roster is statistically improbable if not impossible. So why did these cases occur? No one knows, but there are many theories – from toxins to viruses. Regardless, because this cluster is so unusual, it is considered an outlier and, in this case, should have been excluded from the study conducted by the authors of the Neurology paper.

There are times when outliers need to be excluded from data sets because they can skew the data and lead to an overestimate or an underestimate of the condition being studied. Statistics are complicated, and sometimes outliers are included in data calculations. However, when an outlier creates a statistically significant association, it should be dropped, and the researcher should not report any statistical significance from the analysis. This is accepted and expected practice within the scientific community.

The three San Francisco 49ers ALS deaths are so unusual they shouldn’t consider in the SMR calculation, especially because the use of these three deaths in this study creates a statically significant result that may or may not be real.

This changes the numbers

If those three ALS cases are removed from the study population, the data becomes statistically insignificant, and there is no difference in the number of deaths from ALS between the NFL population and the general public.

The ALS incidence data used in the Neurology paper came from the 1994 National Institute for Occupational Safety and Health (NIOSH) mortality study of NFL players. The authors of the NIOSH study caution on the interpretation of the ALS findings. They point out how the 49ers cases cluster and state: “The causes of ALS are not clearly understood thus making it difficult to assess a potential association with playing professional football. … Further follow-up of this cohort over time should clarify whether this increase is due to chance.”

There are a number of other major weaknesses of the Neurology paper that go beyond the scope of this blog, but using death certificate data is notoriously problematic as death certificates are often vague or event incorrect. Because of a decrease in numbers people seeking autopsies, death certificates then become a “best guess” based on a medical history.

Alzheimer’s disease can only be diagnosed by an autopsy. Clinically, many different diseases can mimic Alzheimer’s disease, and it is estimated that up to 20 percent of clinically diagnosed cases of Alzheimer’s disease are something else, such as a drug interactions or metabolic conditions. Without autopsy confirmation of the existence of Alzheimer’s disease, it’s really only a guess on the death certificate.

Furthermore, CTE and Alzheimer’s disease are distinct pathologic entities, not one and the same, and an association between the two has yet to be convincingly demonstrated.

Another issue with the Neurology paper is there is zero concussion history for any members of the NFL population. It is just assumed that because they played football, it was concussions that resulted in neurodegenerative diseases.

When we hear things like NFL players have a three times greater risk of dying of Alzheimer’s disease, it scares us. When I read the Neurology paper the first time in 2012, it cemented my belief that my son was never going to play football. It wasn’t until I sat down and really read the paper that I realized the weaknesses of the study and discovered the truth was NFL players do not have a greater risk of neurodegenerative diseases than the general population.

This was confirmed by a study published in the medical journal Mayo Clinic Proceedings’. The authors compared the number of cases of neurodegenerative diseases among former participants of high school football, glee club and school band active between the years of 1946 and 1956. There were 438 football players and 140 non-contact subjects in the study. That’s more participants than the Neurology study. Despite less rigorous equipment regulations and less regard for concussions compared with today, they found no increased risk of developing dementia, ALS or Parkinson disease associated with playing football.

CTE exists, and there are athletes out there suffering, but our knowledge of what is happening is in its infancy. We are just beginning to understand what we are seeing under the microscope, and we have a long way to go.

But in order to get to a point where we as scientists can make a difference in the lives of those suffering from CTE, we have to be honest about what the science is and isn’t telling us. We must be honest about the weaknesses of our studies so that future research can fill in the gaps.

It has taken us decades to develop the understanding of Alzheimer’s disease that we have today. It’s not reasonable to think that the secrets of CTE can be deduced in a six-year time span. In the meantime, we take a cautious approach and do the best that we can with the knowledge we have.

We owe this to our young athletes.

Dr. Peter Cummings is board certified in anatomic pathology, forensic pathology and neuropathology. Neuropathology is a subspecialty of pathology focusing on the study diseases and injuries of the brain, muscles, nerves and eyes. Forensic pathology is another subspecialty of pathology focusing on how to interpret injuries and injury patterns. Cummings practices both areas of pathology and is an expert in head trauma.