T-Test: Dr. Semmelweis and the discovery of handwashing

This article only illustrates the use of t-test in a real life problem but does not provide any technical information on what is T-Test or how T-Test works. I will go through the T-test in details in another post and will link it into this post.

Intro

I was looking for a cool dataset to illustrate the use of T.test and I found this DataCamp project “Dr. Semmelweis and the discovery of handwashing”. This a straightforward project but I really like the way they introduce it and specifically how they show beyond doubt that statistic plays a vital role in the medical field.

Here is the discovery of the Dr.Ignaz Semmelweis:
“In 1847 the Hungarian physician Ignaz Semmelweis makes a breakthough discovery: He discovers handwashing. Contaminated hands was a major cause of childbed fever and by enforcing handwashing at his hospital he saved hundreds of lives.”

1. Meet Dr. Ignaz Semmelweis

ignaz_semmelweis_1860This is Dr. Ignaz Semmelweis, a Hungarian physician born in 1818 and active at the Vienna General Hospital. If Dr. Semmelweis looks troubled it’s probably because he’s thinking about childbed fever: A deadly disease affecting women that just have given birth. He is thinking about it because in the early 1840s at the Vienna General Hospital as many as 10% of the women giving birth die from it. He is thinking about it because he knows the cause of childbed fever: It’s the contaminated hands of the doctors delivering the babies. And they won’t listen to him and wash their hands!

In this notebook, we’re going to reanalyze the data that made Semmelweis discover the importance of handwashing. Let’s start by looking at the data that made Semmelweis realize that something was wrong with the procedures at Vienna General Hospital.

# Load in the tidyverse package
library(tidyverse)
library(ggplot2)
# Read datasets/yearly_deaths_by_clinic.csv into yearly
yearly <- read_csv("datasets/yearly_deaths_by_clinic.csv")
# Print out yearly
yearly
year births deaths clinic
1841 3036 237 clinic 1
1842 3287 518 clinic 1
1843 3060 274 clinic 1
1844 3157 260 clinic 1
1845 3492 241 clinic 1
1846 4010 459 clinic 1
1841 2442 86 clinic 2
1842 2659 202 clinic 2
1843 2739 164 clinic 2
1844 2956 68 clinic 2
1845 3241 66 clinic 2
1846 3754 105 clinic 2

2. The alarming number of deaths

The table above shows the number of women giving birth at the two clinics at the Vienna General Hospital for the years 1841 to 1846. You’ll notice that giving birth was very dangerous; an alarming number of women died as the result of childbirth, most of them from childbed fever.

We see this more clearly if we look at the proportion of deaths out of the number of women giving birth.

# Adding a new column to yearly with proportion of deaths per no. births
yearly$proportion_deaths<-yearly$deaths/yearly$births
# Print out yearly
yearly
year births deaths clinic proportion_deaths
1841 3036 237 clinic 1 0.07806324
1842 3287 518 clinic 1 0.15759051
1843 3060 274 clinic 1 0.08954248
1844 3157 260 clinic 1 0.08235667
1845 3492 241 clinic 1 0.06901489
1846 4010 459 clinic 1 0.11446384
1841 2442 86 clinic 2 0.03521704
1842 2659 202 clinic 2 0.07596841
1843 2739 164 clinic 2 0.05987587
1844 2956 68 clinic 2 0.02300406
1845 3241 66 clinic 2 0.02036409
1846 3754 105 clinic 2 0.02797017

3. Death at the clinics

If we now plot the proportion of deaths at both clinic 1 and clinic 2 we’ll see a curious pattern…

# Setting the size of plots in this notebook
options(repr.plot.width=7, repr.plot.height=4)
# Plot yearly proportion of deaths at the two clinics
ggplot(data=yearly, aes(x=year, y=proportion_deaths, group=clinic, color=clinic)) +
 geom_line() + geom_point()+
 scale_color_brewer(palette="Paired")+
 theme_minimal()

1

4. The handwashing begins

Why is the proportion of deaths constantly so much higher in Clinic 1? Semmelweis saw the same pattern and was puzzled and distressed. The only difference between the clinics was that many medical students served at Clinic 1, while mostly midwife students served at Clinic 2. While the midwives only tended to the women giving birth, the medical students also spent time in the autopsy rooms examining corpses.

Semmelweis started to suspect that something on the corpses, spread from the hands of the medical students, caused childbed fever. So in a desperate attempt to stop the high mortality rates, he decreed: Wash your hands! This was an unorthodox and controversial request, nobody in Vienna knew about bacteria at this point in time.

Let’s load in monthly data from Clinic 1 to see if the handwashing had any effect.

# Read datasets/monthly_deaths.csv into monthly
monthly <- read_csv("datasets/monthly_deaths.csv")

# Adding a new column with proportion of deaths per no. births
monthly$proportion_deaths<-monthly$deaths/monthly$births

# Print out the first rows in monthly
head(monthly)
date births deaths proportion_deaths
1841-01-01 254 37 0.145669291
1841-02-01 239 18 0.075313808
1841-03-01 277 12 0.043321300
1841-04-01 255 4 0.015686275
1841-05-01 255 2 0.007843137
1841-06-01 200 10 0.050000000

5. The effect of handwashing

With the data loaded we can now look at the proportion of deaths over time. In the plot below we haven’t marked where obligatory handwashing started, but it reduced the proportion of deaths to such a degree that you should be able to spot it!

ggplot(data=monthly, aes(x=date, y=proportion_deaths)) +
geom_line() + geom_point()+
scale_color_brewer(palette="Paired")+
theme_minimal()

2

6. The effect of handwashing highlighted

Starting from the summer of 1847 the proportion of deaths is drastically reduced and, yes, this was when Semmelweis made handwashing obligatory.

The effect of handwashing is made even more clear if we highlight this in the graph.

# From this date handwashing was made mandatory

handwashing_start = as.Date('1847-06-01')

# Add a TRUE/FALSE column to monthly called handwashing_started
monthly$handwashing_started=handwashing_start,TRUE,FALSE)

# Plot monthly proportion of deaths before and after handwashing
ggplot(data=monthly, aes(x=date, y=proportion_deaths, group=handwashing_started, color=handwashing_started)) +
geom_line() + geom_point()+
scale_color_brewer(palette="Paired")+
theme_minimal()

3

7. More handwashing, fewer deaths?

Again, the graph shows that handwashing had a huge effect. How much did it reduce the monthly proportion of deaths on average?

# Calculating the mean proportion of deaths
# before and after handwashing.

monthly_summary % group_by(handwashing_started) %>% summarise(mean_proportion_detahs=mean(proportion_deaths))

# Printing out the summary.
monthly_summary
handwashing_started mean_proportion_detahs
FALSE 0.10504998
TRUE 0.02109338

8. A statistical analysis of Semmelweis handwashing data

It reduced the proportion of deaths by around 8 percentage points! From 10% on average before handwashing to just 2% when handwashing was enforced (which is still a high number by modern standards).
To get a feeling for the uncertainty around how much handwashing reduces mortalities we could look at a confidence interval (here calculated using a t-test).

# Calculating a 95% Confidence intrerval using t.test
test_result <- t.test( proportion_deaths ~ handwashing_started, data = monthly)
test_result

9. The fate of Dr. Semmelweis

That the doctors didn’t wash their hands increased the proportion of deaths by between 6.7 and 10 percentage points, according to a 95% confidence interval. All in all, it would seem that Semmelweis had solid evidence that handwashing was a simple but highly effective procedure that could save many lives.

The tragedy is that, despite the evidence, Semmelweis’ theory — that childbed fever was caused by some “substance” (what we today know as bacteria) from autopsy room corpses — was ridiculed by contemporary scientists. The medical community largely rejected his discovery and in 1849 he was forced to leave the Vienna General Hospital for good.

One reason for this was that statistics and statistical arguments were uncommon in medical science in the 1800s. Semmelweis only published his data as long tables of raw data, but he didn’t show any graphs nor confidence intervals. If he would have had access to the analysis we’ve just put together he might have been more successful in getting the Viennese doctors to wash their hands.

 

Export Data from Power BI into a file using R

We usually import Data from file into Power BI, but exporting data from Power BI can be very handy when you want to create a custom visual using R.

In fact it can be very cumbersome to code your visual directly into the Power BI script editor.
Here are few reasons why you should opt for exporting your Power Bi dataset first and re-import it in R to create your visual.

  • Intellisense is not available in Power BI R script embedded
  • Does not highlight keywords in colour
  • Hard to debug & hard to code (you can’t print intermediate calculation)
  • Slower than Rstudio

So unlike you’re a R master or you want to create a very simple visual it is definitely worth exporting your data into a file and then re-import it into R.

You can then create your visual in Rstudio first and once you’re happy with it just copy and paste your code into the Power BI visual script.

Export you data

If you haven’t already installed the package (gdata) you’ll need to install it:

#open an instance of R and type the command below
install.packages("gdata");

 

Once the “gdata” package is installed, select the R visual script and drag into values the measures and columns you need.

Capture

In the R script editor type the following R code:

require(gdata)
write.table(trim(dataset), file="your filepath.txt", sep = "\t", row.names = FALSE)

Capture

You can add plot(dataset) like I did int the above screenshot to make sure there isn’t any errors in your script hence as long as you can see a plot whatever it is(line-plot, box-plot, correlation-plot) it means your export was successful or obviously you can just check if your file is present in your directory.

Here is my output file:
Capture

Re-import you Power BI dataset into R

Now we can import our Power BI dataset into R as follows:

dataset = read.table(file="myfile2.txt", sep = "\t",header = TRUE)

See the R output below:
Capture

You can now work with your dataset in Rstudio until you get your visual right and then you’ll just need to copy & paste your code into the Power Bi script..

 

R – Import multiple CSV files and load them all together in a single dataframe

 

List of all the filenames

One approach I found really straight forward is just to create a list of all your filenames.
You can also create a pattern to fetch your directory and returns all the matching files.
In my example I need to read all the files starting with “FR”.

setwd("H:/R Projetcs/Accidents")
fileNames<-Sys.glob("csv/FR*.csv")
zonnesFiles<- lapply(fileNames, read.csv)

The function lapply (equivalent of a loop) reads every single file presents in my list fileNames and store them into my variable zonnesFiles.
The variable zonnesFiles is a list of data frames, I have to read 15 files so there's 15 different dataframes in the list.

Merge all the files into a single data frame

Once we have our list of dataframe we want to merge them in one single dataframe.
As my files don’t have any headers I first need to make sure they all have the same column names, to do so I loop through my list of zonnesFiles and rename them.

I then create a function “merge.all”, my function just call the base r “merge” function but I like to create my own so I don’t have to bother with parameter every time I need to call the function.
Finally we just need to call our function for every single df in the zonnesFIles list.
I use the Reduce function to successively merge each dataframe of my list. The Reduce function takes a binary function and a vector/list and successively applies the function to the list elements.

And here is the code:

#Rename column names of each df
for(i in 1:length(zonnesFiles)){
 colnames(zonnesFiles[[i]])<-c("Longitude","Latitude","Type")
}

#Create a function to merge my df
merge.all<- function(x, y) {
 merge(x, y, all=TRUE, by=listCols)
}

#Lits of columns to merge on
listCols<-c("Longitude","Latitude","Type")
#call the merge function
zonnes<- Reduce(merge.all, zonnesFiles)

PowerBI – Dynamic Chart Title

Unlike Qlikview, the chart titles in PowerBI can only be static. as you can only pass a static text in the title parameter.

Capture

However, there’s a way around it!
The workaround I found is pretty simple you just need to fake a title by creating a measure which contains your title expression and drop this measure into a Card visual .

Then by applying the same transparency and colours of your chart you just need to turn off the chart tile and put the Card visual on top of your chart.

Here is the code for my  title measure:

 MyMeasureTitle = ("Total Cost of the Top " & [TopN Value] & " Depts VS all other Depts") 

Capture

So my title will interact with the above slicer dynamically however if no values are ticked off I still want a default value to be returned so here is the code for this (you might not need to implement it)

TopN Value =
IF (
        HASONEVALUE ('TopN Filter'[TopN])
         , VALUES ('TopN Filter'[TopN])
         , 10
  )

So after dropping your measure into a Card visual you’ve got your title ready!

Capture

And this how it looks when you place it right above your chart:

Capture10

Make sure your chart and the card have the same size and colour and by setting the right location x,y it will look like the embedded chart title.