Mike Denly

Making Merges Go Through Using tidylog and anti_join

2021-07-01T00:00:00-07:00

Packages in the tidyverse suite, including dplyr, represent amazing contributions data science. One thing that constantly vexed me, though, was the inability of dplyr’s merge functions–left_join, full_join, etc.– to actually let you know if the merges went through without having to do extra work. It just seemed so cumbersome to have to individually inspect each data frame after each merge. Well, thankfully, that is no longer necessary due to tidylog.

To explain how tidylog helps with ensuring merges go through as desired, let’s start by loading two data frames, df1 and df2. First, let’s load and inspect df1:

# create df1
df1 = data.frame(country = c("Brazil", "USA"),
                 year = c(2020,2020),
                 population_millions = c(213,331))

# inspect df1
head(df1)

##   country year population_millions
## 1  Brazil 2020                 213
## 2     USA 2020                 331

Then, let’s load and inspect df2:

# create df2
df2 = data.frame(country = c("Brasil", "USA"),
                 year = c(2020,2020),
                 poverty_rate = c(21,9))

# inspect df2
head(df2)

##   country year poverty_rate
## 1  Brasil 2020           21
## 2     USA 2020            9

As should be apparent, they are both data frames with demographic information on the poverty rates and population of the US and Brazil in the year 2020. Using the dplyr package, we could merge df1 and df2 together using left_join:

# suppress warnings (unnecessary, but makes the script cleaner)
options(warn=-1)
options(tidyverse.quiet = TRUE)

# load library
library(tidyverse)

# merge
initial_merged_df = left_join(df1, df2, by=c("country", "year"))

# inspect the data frame
head(initial_merged_df)

##   country year population_millions poverty_rate
## 1  Brazil 2020                 213           NA
## 2     USA 2020                 331            9

As is clear from above, the poverty rate data is NA for Brazil. Why did this happen? The tidylog package helps us answer that without even having to inspect the data frame, which is really useful when working with larger data frames. To better understand the benefits of tidylog, let’s re-run the merge, but this time let’s load the tidylog package first:

# load tidylog
library(tidylog, warn.conflicts = FALSE)

# re-run the merge
initial_merged_df = left_join(df1, df2, by=c("country", "year"))

## left_join: added one column (poverty_rate)
##            > rows only in x   1
##            > rows only in y  (1)
##            > matched rows    (1)
##            >                 ===
##            > rows total       1

Loading the tidylog package lets us know that our merge did not fully go through. Which observations? To figure that out, let’s use anti_join, which tells us which observations from df2 did not merge over to df1. To do that, just flip the order of df1 and df2 as follows:

# checking for problematic observations with anti_join
checking = anti_join(df2, df1, by=c("country", "year"))

## anti_join: added no columns
##            > rows only in x   1
##            > rows only in y  (1)
##            > matched rows    (1)
##            >                 ===
##            > rows total       1

Now, let’s take a peak at the data frame:

head(checking)

##   country year poverty_rate
## 1  Brasil 2020           21

As we can clearly see, the Brazil observation has been spelled with an “s” instead of an “z”. Because we are working in English, let’s correct the spelling in English:

# correct the spelling
df2$country[df2$country=="Brasil"] = "Brazil"

Now, we can re-run the merge for a final time, and everything will go through:

# final time
final_merged_df = left_join(df1, df2, by=c("country", "year"))

## left_join: added one column (poverty_rate)
##            > rows only in x   0
##            > rows only in y  (0)
##            > matched rows     2
##            >                 ===
##            > rows total       2

Now, let’s inspect the data frame:

head(final_merged_df)

##   country year population_millions poverty_rate
## 1  Brazil 2020                 213           21
## 2     USA 2020                 331            9

Clearly, tidylog was not 100% necessary for merging the toy data frames above, df1 and df2. With larger data frames, though, combining tidylog and anti_join will be crucial to ensuring that all of your data actually merge.

Removing Accents in R

2021-05-25T00:00:00-07:00

R is a fantastic open-source program that allows users to do just about anything, but sometimes the program requires some tricks to accomplish seemingly simple tasks. Removing accents is a case in point, so I’d like to provide everyone with some guidance to overcome some of the thornier accent removal issues.

Typical One- and Two-Character Substitutions

Before even getting into the accent removal, the first order of business is to ensure that your R Studio is using UTF-8 file encoding:

In my case, everything was already in UTF-8, but it was good to check just in case:

Now, let’s create a data frame that will allow us to remove different types of accents that we will encounter:

# clear environment
rm(list=ls(all=TRUE)) 

# create data frame
df <- data.frame(
  country = c("Argentina", "Honduras","Germany"),
  city = c("San Martín", "San José","Aßlar"))

head(df)

##     country       city
## 1 Argentina San Martín
## 2  Honduras   San José
## 3   Germany      Aßlar

I specifically chose those cities above because they embody different types of accent fixes that you may need to perform. Removing the “é” from San José and “í” from “San Martín” are typical cases that won’t cause many problems–regardless of how you chose to remove them. The “ß” in “Aßlar”, however, requires a two-character substitution: the “ß” needs to be replaced with “ss” when it is transcribed into English.

So that we can take care of the one- and two-character substitutions all at once, let’s write a function called remove.accents. The old1 vector contains the letter with the accent in the original language for the one-character substitutions, and the new1 vector contains the respective replacements without the accents. Note how old1 and new1 are kept in order of the replacements. The logic is similar for old2 and new2.

# define the function
remove.accents <- function(s) {
  
  # 1 character substitutions
  old1 <- "éí"
  new1 <- "ei"
  s1 <- chartr(old1, new1, s)
  
  # 2 character substitutions 
  old2 <- c("ß")
  new2 <- c("ss")
  s2 <- s1
  
  # finalize the function
  for(i in seq_along(old2)) s2 <- gsub(old2[i], new2[i], s2, fixed = TRUE)
  
  s2
}

Given that the function is now defined, let’s execute it and see whether it works:

# finish the accent fix
df$city = remove.accents(df$city)

# examine the data frame
head(df)

##     country       city
## 1 Argentina San Martin
## 2  Honduras   San Jose
## 3   Germany     Asslar

When the Above Doesn’t Work

Sometimes, the above tricks won’t work. I recently ran into such an instance while cleaning Moldova data in Romanian for my project on natural resources and subnational public goods provision. To show how I overcame this challenge, let me load the shapefile with the sf package and only keep the admin1 column with the accented values:

# load libraries
library(sf)
library(dplyr)

# load shapefile
moldova = st_read("Moldova_SHP/MDA_adm1.shp")

# keep only the admin1 column with the accented characters
moldova <-  
  moldova %>% 
  dplyr::select(NAME_1) %>% 
  st_set_geometry(NULL) %>% # drop latitude & longitude pairs separately
  dplyr::rename(admin1 = NAME_1)

Let’s see what these accented characters look like. Because R Markdown has a tough time reading them, I’ll use a screenshot to show the results here:

In such instances, simply use the make_clean_names function from the janitor package to remove the accents:

# load library
library(janitor)

# perform fixes
moldova$admin1 = make_clean_names(moldova$admin1)

# make sure it goes through
table(moldova$admin1)

##   anenii_noi        balti basarabeasca       bender      briceni        cahul 
##            1            1            1            1            1            1 
##     calarasi     cantemir      causeni     chisinau     cimislia     criuleni 
##            1            1            1            1            1            1 
##    donduseni      drochia     dubasari       edinet      falesti     floresti 
##            1            1            1            1            1            1 
##     gagauzia      glodeni     hincesti     ialoveni        leova    nisporeni 
##            1            1            1            1            1            1 
##       ocnita        orhei       rezina      riscani     singerei   soldanesti 
##            1            1            1            1            1            1 
##       soroca  stefan_voda     straseni     taraclia    telenesti transnistria 
##            1            1            1            1            1            1 
##      ungheni 
##            1

To conclude, let’s just change the underscore back to a space:

# perform replacement
moldova$admin1 = gsub("_", " ", moldova$admin1)

# make sure it goes through
table(moldova$admin1)

##   anenii noi        balti basarabeasca       bender      briceni        cahul 
##            1            1            1            1            1            1 
##     calarasi     cantemir      causeni     chisinau     cimislia     criuleni 
##            1            1            1            1            1            1 
##    donduseni      drochia     dubasari       edinet      falesti     floresti 
##            1            1            1            1            1            1 
##     gagauzia      glodeni     hincesti     ialoveni        leova    nisporeni 
##            1            1            1            1            1            1 
##       ocnita        orhei       rezina      riscani     singerei   soldanesti 
##            1            1            1            1            1            1 
##       soroca  stefan voda     straseni     taraclia    telenesti transnistria 
##            1            1            1            1            1            1 
##      ungheni 
##            1

Balancing Panel Data and Removing Duplicate Unique IDs

2020-03-25T00:00:00-07:00

Properly balancing panel data and removing duplicate unique identifiers is a recurring challenge for students in my Research Practicum course. Accordingly, I thought it would be helpful to provide an example for everyone.

Some may ask: What are properly balanced panel data? Generally, a properly balanced panel means that all possible ID variables and have data for the same (or appropriate) time period. In the example that follows, we will be working with a dataset of United States (US) foreign aid disbursements to all African countries for the 2001-2017 period. The data come from the US Foreign Aid Explorer (FAE), and we have deflated them to constant 2010 dollars to account for potential inflationary pressures. For these data, we want to ensure that all existent African countries have a foreign aid disbursement value for each year between 2001 and 2017.

To start, let’s import our data frame and take a quick look at it:

rm(list=ls(all=TRUE)) # clear the environment
library(rio) # load package
fae_all <- import("african_aid.csv") # import the data
summary(fae_all) # summarize the data

##  country_name            year       aid_constant       
##  Length:911         Min.   :2001   Min.   :    -38845  
##  Class :character   1st Qu.:2005   1st Qu.:   9568481  
##  Mode  :character   Median :2009   Median :  58087880  
##                     Mean   :2009   Mean   : 165454902  
##                     3rd Qu.:2013   3rd Qu.: 167795723  
##                     Max.   :2017   Max.   :3173867164  
##                                    NA's   :         1

head(fae_all) # take a quick peak at the data

##   country_name year aid_constant
## 1      Algeria 2001       563682
## 2       Angola 2001     88977561
## 3        Benin 2001     41040486
## 4     Botswana 2001      3786002
## 5 Burkina Faso 2001     19766559
## 6      Burundi 2001     28287903

Let’s check how many countries are in the dataset:

length(unique(fae_all$country_name))

## [1] 54

Let’s also check how many years are in the dataset:

length(unique(fae_all$year))

## [1] 17

Now, let’s check whether the data are balanced:

suppressMessages(library(plm)) # to remove note about dependencies
is.pbalanced(fae_all) # are the data balanced?

## Warning in pdata.frame(x, index): duplicate couples (id-time) in resulting pdata.frame
##  to find out which, use e.g. table(index(your_pdataframe), useNA = "ifany")

## Warning in is.pbalanced.default(id, time): duplicate couples (id-time)

## [1] FALSE

Let’s first find the duplicate couple(s):

fae_all$unique_id <- paste(fae_all$country_name,fae_all$year) # concatenate to make unique ID
fae_all$duplicate = duplicated(fae_all$unique_id) # generate the duplicate variable
subset(fae_all, duplicate=="TRUE") # find the duplicate

##    country_name year aid_constant  unique_id duplicate
## 17        Egypt 2001            0 Egypt 2001      TRUE

The only duplicate value is for Egypt in 2001. It has an aid disbursement value of 0, so we can simply summarize the data. Stata users would say “collapse” the data. In this case, summarizing or collapsing the data means that we add together the two values for Egypt in 2001 and delete the duplicate row in the process. The process is generally the same if there were more duplicates in our data frame:

suppressMessages(library(tidyverse))
fae_all <-
  fae_all %>% # refer to the fae_all data frame
  group_by(country_name,year) %>% # tell R the unique ID
  select(-c(unique_id,duplicate)) %>% # drop (now) useless variables
  summarize(aid_constant = sum(aid_constant, na.rm=TRUE)) # summarize

Did that get rid of our duplicates couples problem?

library(plm)
is.pbalanced(fae_all) # duplicate couple issue now gone

## [1] FALSE

We are no longer getting the duplicates couples warning, so we can now concentrate on balancing the panel. Before proceeding, let’s first create a numeric country ID variable:

fae_all$country_id <- as.numeric(factor(fae_all$country_name)) # numeric country ID

We need to create a numeric country ID variable because the plm package needs two numeric variables to constitute its ID-time index. With that step out of the way, let’s now balance the panel:

fae_balanced <- make.pbalanced(fae_all, 
                               balance.type = c("fill"), 
                               index = c("country_id","year")) 

Since we used the numeric country_id variable to balance the panel, it means that we are going to have some missing country_name observations. Let’s first figure out which countries do not have a country name attached to them:

subset(fae_balanced, is.na(country_name))

##     country_name year aid_constant country_id
## 273         <NA> 2001         <NA>         17
## 477         <NA> 2001         <NA>         29
## 478         <NA> 2002         <NA>         29
## 479         <NA> 2003         <NA>         29
## 783         <NA> 2001         <NA>         47
## 784         <NA> 2002         <NA>         47
## 786         <NA> 2004         <NA>         47
## 787         <NA> 2005         <NA>         47

Note that we have country_id observations with missing country names, so we can use those IDs to identify the corresponding missing countries:

subset(fae_balanced, country_id==17|country_id==29|country_id==47) 

##          country_name year aid_constant country_id
## 273              <NA> 2001         <NA>         17
## 274 Equatorial Guinea 2002        81590         17
## 275 Equatorial Guinea 2003       300441         17
## 276 Equatorial Guinea 2004        42583         17
## 277 Equatorial Guinea 2005       247406         17
## 278 Equatorial Guinea 2006        76640         17
## 279 Equatorial Guinea 2007        29844         17
## 280 Equatorial Guinea 2008        15953         17
## 281 Equatorial Guinea 2009       191463         17
## 282 Equatorial Guinea 2010       143462         17
## 283 Equatorial Guinea 2011        55248         17
## 284 Equatorial Guinea 2012       414390         17
## 285 Equatorial Guinea 2013       207469         17
## 286 Equatorial Guinea 2014       204699         17
## 287 Equatorial Guinea 2015       244107         17
## 288 Equatorial Guinea 2016       948035         17
## 289 Equatorial Guinea 2017      3321453         17
## 477              <NA> 2001         <NA>         29
## 478              <NA> 2002         <NA>         29
## 479              <NA> 2003         <NA>         29
## 480             Libya 2004        52165         29
## 481             Libya 2005       161604         29
## 482             Libya 2006     30792152         29
## 483             Libya 2007      4806971         29
## 484             Libya 2008     16698323         29
## 485             Libya 2009      6528419         29
## 486             Libya 2010     10736737         29
## 487             Libya 2011    236478945         29
## 488             Libya 2012    162328736         29
## 489             Libya 2013     54600235         29
## 490             Libya 2014     21013668         29
## 491             Libya 2015     25773448         29
## 492             Libya 2016     17908764         29
## 493             Libya 2017     40775906         29
## 783              <NA> 2001         <NA>         47
## 784              <NA> 2002         <NA>         47
## 785       South Sudan 2003            0         47
## 786              <NA> 2004         <NA>         47
## 787              <NA> 2005         <NA>         47
## 788       South Sudan 2006       262143         47
## 789       South Sudan 2007      4463780         47
## 790       South Sudan 2008      4803235         47
## 791       South Sudan 2009      3111907         47
## 792       South Sudan 2010     36194887         47
## 793       South Sudan 2011    107385654         47
## 794       South Sudan 2012    406072707         47
## 795       South Sudan 2013    511596816         47
## 796       South Sudan 2014    650811758         47
## 797       South Sudan 2015    750664797         47
## 798       South Sudan 2016    638801967         47
## 799       South Sudan 2017    942560453         47

Now that we know which country IDs correspond to the missing country names, let’s make the appropriate fixes and check to ensure that there are no longer any missing country names:

fae_balanced$country_name[fae_balanced$country_id==17] = "Equatorial Guinea"
fae_balanced$country_name[fae_balanced$country_id==29] = "Libya"
fae_balanced$country_name[fae_balanced$country_id==47] = "South Sudan"

subset(fae_balanced, is.na(country_name)) # nothing now: all good

## [1] country_name year         aid_constant country_id  
## <0 rows> (or 0-length row.names)

Returning to the actual data balancing, let’s now ensure that everything went through as planned:

is.pbalanced(fae_balanced) # are data balanced?

## [1] TRUE

Although the above command indicates that the data are now balanced, let’s table the country and year variables to verify. Recall from above that the original (unbalanced) dataset contains 54 distinct countries and 17 distinct years. Accordingly, the new (balanced) data frame should contain 54 countries for our 17 years of data (2001-2017). To verify, let’s proceed one step at a time, starting with the years:

table(fae_balanced$year) # years OK?

## 
## 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 
##   54   54   54   54   54   54   54   54   54   54   54   54   54   54   54 
## 2016 2017 
##   54   54

The years look right, and we can say the same for the countries:

table(fae_balanced$country_name) # countries OK?

## 
##                  Algeria                   Angola                    Benin 
##                       17                       17                       17 
##                 Botswana             Burkina Faso                  Burundi 
##                       17                       17                       17 
##               Cabo Verde                 Cameroon Central African Republic 
##                       17                       17                       17 
##                     Chad                  Comoros      Congo (Brazzaville) 
##                       17                       17                       17 
##         Congo (Kinshasa)            Cote d'Ivoire                 Djibouti 
##                       17                       17                       17 
##                    Egypt        Equatorial Guinea                  Eritrea 
##                       17                       17                       17 
##                 Eswatini                 Ethiopia                    Gabon 
##                       17                       17                       17 
##                   Gambia                    Ghana                   Guinea 
##                       17                       17                       17 
##            Guinea-Bissau                    Kenya                  Lesotho 
##                       17                       17                       17 
##                  Liberia                    Libya               Madagascar 
##                       17                       17                       17 
##                   Malawi                     Mali               Mauritania 
##                       17                       17                       17 
##                Mauritius                  Morocco               Mozambique 
##                       17                       17                       17 
##                  Namibia                    Niger                  Nigeria 
##                       17                       17                       17 
##                   Rwanda    Sao Tome and Principe                  Senegal 
##                       17                       17                       17 
##               Seychelles             Sierra Leone                  Somalia 
##                       17                       17                       17 
##             South Africa              South Sudan           Sudan (former) 
##                       17                       17                       17 
##                 Tanzania                     Togo                  Tunisia 
##                       17                       17                       17 
##                   Uganda                   Zambia                 Zimbabwe 
##                       17                       17                       17

Lastly, let’s remedy our missing data problem for our foreign aid disbursement variable:

subset(fae_balanced, is.na(aid_constant))

##          country_name year aid_constant country_id
## 273 Equatorial Guinea 2001         <NA>         17
## 477             Libya 2001         <NA>         29
## 478             Libya 2002         <NA>         29
## 479             Libya 2003         <NA>         29
## 783       South Sudan 2001         <NA>         47
## 784       South Sudan 2002         <NA>         47
## 786       South Sudan 2004         <NA>         47
## 787       South Sudan 2005         <NA>         47

As shown above, there are missing (NA) values. The Foreign Aid Explorer (FAE) is a very comprehensive data source, and the US government provides very transparent data about how its spends taxpayer money. It is thus impossible that there are missing data. For this reason, let’s initially replace the missing values with 0, indicating that the country did not receive any foreign aid for the given years.

fae_balanced$aid_constant[is.na(fae_balanced$aid_constant)] <- 0

Was this the right decision? Unfortunately, it was a mistake: South Sudan only became an internationally-recognized country in 2011, and we don’t want to have data for the country when it did not exist. Accordingly, let’s check the foreign aid disbursement values for South Sudan:

subset(fae_balanced, country_name=="South Sudan")

##     country_name year aid_constant country_id
## 783  South Sudan 2001            0         47
## 784  South Sudan 2002            0         47
## 785  South Sudan 2003            0         47
## 786  South Sudan 2004            0         47
## 787  South Sudan 2005            0         47
## 788  South Sudan 2006       262143         47
## 789  South Sudan 2007      4463780         47
## 790  South Sudan 2008      4803235         47
## 791  South Sudan 2009      3111907         47
## 792  South Sudan 2010     36194887         47
## 793  South Sudan 2011    107385654         47
## 794  South Sudan 2012    406072707         47
## 795  South Sudan 2013    511596816         47
## 796  South Sudan 2014    650811758         47
## 797  South Sudan 2015    750664797         47
## 798  South Sudan 2016    638801967         47
## 799  South Sudan 2017    942560453         47

Interestingly, South Sudan has aid values from 2006-2017, indicating that the US government figured out which funds supported South Sudan even before it became a recognized country in 2011. We should thus not automatically delete anything prior to 2006. By the same token, we need to delete the zero values from 2001-2005, since South Sudan could not receive any foreign aid for that period:

library(tidyverse)
fae_final <- 
  fae_balanced %>% 
  filter(!(country_name=="South Sudan" & year<=2005))

We now have properly balanced panel data. In other words, (i) we have a panel dataset that has the appropriate number of countries and years; and (ii) we have carefully examined the extent to which missing/NA values should be part of our dataset. I hope this helps!

Hello, World/Importing Data in R

2019-04-21T00:00:00-07:00

Hello, world! I have finally figured out GitHub and how to make a free website with Jekyll. It’s a great day!

Today, I would like to point everyone to my favorite R package for importing data: the rio package. It is my favorite because the syntax is basically the same, no matter what type of data file you are trying to load.

To import a .csv file:

library(rio)
data1 <- import("data1.csv")

To import a Stata dataset, all you need to do is change the extension to .dta.

library(rio)
data2 <- import("data2.dta")

For Excel files, you just need to change the extension to .xlsx or .xls (depending on the file), as well as specify which tab you are attempting to reference with “which”. This is necessary because Excel files can have multiple tabs, whereas that is not possible for a .csv file. Here is some code for referencing the second tab in an Excel workbook:

library(rio)
data3 <- import("data3.xlsx", which=2)

I hope this information helps someone. For more information, kindly refer to the rio package help file.