Part 3 I need to make sure I’m not going to be in an emergency coding position in future

Congratulations, you’ve arrived at the future-proofing R code section of this book, where we think about what small actions you can take to your existing and future code bases to make sure you never need to read the first two chapters!

This chapter aims to talk you through putting best practice in place in your code, all with the aim of making it as easy as possible to read, run and edit in future. This is important even if you don’t plan to ever change this code, as changes to the infrastructure of R itself means that code that remains static can break over time.

It assumes that you have some familiarity with R code and have a code base that you and your team use, but doesn’t expect you to be an expert coder, or even to be coding regularly yourself.

3.1 Writing quality documentation

As you’ve probably seen throughout this book, documentation is key to code. The aim is that code should be human-readable first, and then machine-readable second, that is the documentation accompanying the code and written as comments throughout the code should make it easy for a human to read and understand even without being an expert coder.

3.1.1 Code comments

Documentation throughout code should be written as comments (these are prefaced with one or more # symbols so R knows to ignore them); most people do this, but often in a way that isn’t best practice or doesn’t make it easy for a beginner to understand.

  • Code comments should explain what is happening in a human readable way, not just repeating what the code is saying.
#Not a helpful code comment, doesn't add anything human readable:
## Filtering mtcars so cyl == 6
mtcars %>%
  dplyr::filter(cyl == 6)

#Helpful code comment, explains in a human way what is going on:
## Keeping only cars with 6 cylinder engines
mtcars %>%
  dplyr::filter(cyl == 6)

  • Code comments shouldn’t contain step-by-step running instructions, these should be in an accompanying README file. This is because it’s easy to miss a simple code comment in thousands of lines of code!

  • Code comments shouldn’t be used to run code in a specific way (e.g. “uncomment this code if it’s June”) or to stop redundant code running. Instead, make use of if statements and version control to achieve these, which allows you to know whether the commented out-code was run or not on a given occasion.

3.1.2 Readme documentation

A README document is a standard piece of documentation that you can expect to accompany code; an experienced coder will automatically look for one within a new project.

While you may have accompanying desk notes for your code, it’s always a good idea for the step-by-step running process to be stored in a README document alongside the code itself. This doesn’t have to be full instructions for the whole project, but should include:

  • How the user gets access to the code; do they clone from Github, do they run from a shared drive?
  • Anything else they need to do to make the code run; get shared drive access or GCP permissions?
  • The steps the user needs to carry out the first time they run the project; do they set up something like renv, or install specific packages?
  • The steps the user needs to carry out every time they run your code; do they run the files in a specific order, or change certain parameters?

You can see an example of a full README document here, as you can see it covers all of the points above.

If you would like to keep full desk instructions alongside your code, you can also do this, particularly in Github where there is space to add a wiki (see an example of how this works here).

What you can do to achieve this:

  • Make sure all of your code has documentation that you can understand (especially if you’re the least technical person in your team!)
  • Get your desk instructions stored in a README or wiki alongside your code
  • Get someone unfamiliar with the project to check if they could run your code using only that documentation to guide them

3.2 Using Github

3.3 Adding renv to a project

Renv is often thought of as “advanced” level coding, but actually it’s something that is easy to understand and implement, and makes it much easier for beginners to pick up and use your code.

Renv is an example of a piece of package management software. Normally, when you set up a project in R, it makes use of the existing libraries you’ve got installed in R. When someone else opens and uses the same project, it uses the existing libraries that they have installed. This is a problem as packages change regularly, and you will likely have different versions of the same packages; this can introduce breaking changes in the following ways:

  • You and someone else have different versions of the same packages, so you can’t run their code without it breaking
  • You have two different piece of code which need two different versions of the same package, so running one breaks the other
  • You return to your old code after 6 months and updates you’ve made to your packages in that time mean that it won’t run any more
  • You move your code from one R platform to another, and the change in packages stops it from working on the new platform
  • Someone else in your team doesn’t know how to download non-CRAN packages (from github or other sources)

Renv solves all of these problems by creating a separate package library for each coding project you create. The packages are associated with the project not the user, so every package can have different package versions without problems or conflict, and all the correct package versions can be made available to any user by pressing one button.

Renv is simple to put in place and keep updated as part of normal code, can be version controlled on Github and doesn’t require any additional software to use.

What you can do to achieve this:

  • Encourage your team to put renv in place on coding projects moving forward, and to add instructions for how to use it in associated documentation
  • Make a note of projects which don’t currently use renv so you’re aware of where package management is a risk for your analysis
  • Know where to find support for using renv in your team (learning resources are available on the C&C Teams channel!)

3.4 Making your code reusable

A major hurdle for producing code which is both reusable and stable is the use of hard-coded values; these are values present in inputs, used during data processing, or used to create outputs which need to be manually updated whenever the code is run.

Virtually anything can be a hard-coded value, but some of the most common and problematic are:

  • Dates to filter data on, or in commentary
  • Names of columns which include things like month or year values
  • Static cell ranges in input data sheets which need to have an additional row added each time
  • Ranges, labels and values on axes and labels in data visualisation

All hard coded values like this cause points of inefficiency, bugs and risks to data quality in your analysis:

  • Inefficiency: As your code will require manual correction every time it is run, and any points missed will require repeated runs.
  • Bugs: Particularly with input data, hard-coded values can lead to code failing completely, or producing the wrong outputs.
  • Data quality: Both failing to update values and inflexible code which can’t accommodate changing values can lead to code “silently” failing, and producing nonsense values which are hard to spot during QA.

While it’s not possible to produce an exhaustive list of ways to remove hard-coded values in code, it is possible to entirely remove or reduce hard coded values to 1-2 key objects for almost all analytical projects. A key aspect to this is working in a “tidy data” fashion and making use of the tidyverse packages, which build flexible code by default.

What you can do to achieve this:

  • Ensure that your desk instructions list every point in your projects where values need to be updated for a new quarterly/annual run.
  • Make sure your team understand the use of tidy data structures for imported, working and clean data to minimise the impact of things like hard-coded values in column names. A tidy data in R course is available through C&C if you need L&D on this.
  • Encourage your team to eliminate or minimise these points wherever possible; support for doing this can be found on the C&C channel, but there’s also a growing number of DfT R packages such as the {dftutils} package which offer functions to help with this.

3.5 Thinking about reproducibility in analysis

A closely related idea to the concept of resuable code is best practice around reproducibility of the code-running process. Rather than how values within the code are updated, this focusses on making the physical act of running your code as reproducible as possible, and eliminating points where projects move away from best practice.

Key ideas in this space are:

3.5.1 Don’t rely on people running code files in order

It’s great practice to divide code up across multiple code files; it makes it clean and easy to read and update, and is a good way to structure your project.

However, many projects rely on complicated running orders of these separate code files, and desk instructions to encourage this to happen. This is tricky because running files out of order or missing one can result in strange results, errors and concerns that code is unreliable or temperamental.

A simple solution to this is use of the R source() function. Discussed in more detail in the last chapter, this function allows you to write one line of code that runs an entire file worth of code, and quickly and easily build a “run file” to run multiple files of code in the correct order every time.

3.5.2 Not changing other people’s R setup

There are a couple of very commonly used functions in R scripts which are very bad practice and very widely used! Both of these are bad practice because they fundamentally alter the user’s R environment in a way that is unexpected, easy to break, and may cause temporary and permanent changes to the way all of their other code runs.

  • install.packages(): Your code should always include library calls, but never package installation calls. Forcing package installation (or update if the user already has the package) is the easiest way to introduce breaking changes to both the current and other projects. Use of renv (as described above) is the proper alternative to this.
  • setwd(): Setting a working directory can break other code that the person is trying to run, and can also fail in strange and unexpected ways on different laptops, or if folder names change. The proper alternative to this is use of R project files and ideally Github!

3.5.3 Not overusing your R workspace

The default of R is to create you a “workspace” for every project; this includes every object you’ve created during the current session of running code, and you can save that workspace so that you can return to it in the same condition every time.

This can cause major problems for running the same code on multiple platforms, for multiple users, or even for the same user after a workspace crash in R, and encourages bad habits with coding. Good practice is that code should never be reliant on the R workspace to do anything, and clearing the R workspace should not cause any changes to how the code runs.

What you can do to achieve this:

  • Understand why the problems listed above can cause problems with even robust and well-written code.
  • Support your team to understand why these are problems, what they look like and how to avoid them; the what they forgot to teach you about R book goes into detail on this in the first few chapters
  • Request an expert code review to help you spot where problems like this might have occurred in your code

3.6 Simplifying library use

The availability of a wide range of packages in R is one of its best features, but it can also be one of its biggest weaknesses, as it encourages people to use weird and wonderful packages where standard ones are available!

Just as an example, if you wanted to read in an xlsx file in R, you could use:

readxl::read_excel()
readxl::read_xlsx()
openxlsx::read.xlsx()
openxlsx2::read_xlsx()
xlsx::read.xlsx()
xlsx::read.xlsx2()
XLConnect::readWorksheetFromFile()

While these packages all have different advantages in terms of speed and dependencies, they all also have different usage, arguments and outcomes, so an inconsistent approach across your team could mean that people have to learn 7 times more code than they need to!

The best solution to this is to make use of a single core group of packages wherever possible, and only to add or substitute packages where there is a clear need to do something different (rather than just a preference).

3.6.1 Tidyverse

The tidyverse is a core group of packages recommended for use within DfT. There is a tidyverse option for all standard data operations included reading in data, tidying it, analysis and visualisation.

Your code should always use tidyverse packages where possible, and use of any other package should be documented with a code comment of a few words to justify its addition.

What you can do to achieve this:

  • Make sure you’re familiar with the names and the purposes of the main tidyverse packages, and that your team understands why its preferable to use them
  • Get into the habit of being a curious reader of library calls; if you see code which uses a large number of packages or ones you don’t recognise, ask why!
  • Add a few words as a code comment to the library call of any non-tidyverse package so future users understand why they were chosen e.g.:
library(rmarkdown) #Producing HTML reports
library(openxlsx) #Writing formatted xlsx
library(seasonal) #For seasonal adjustment

3.7 QA and code reviews

Code as part of an analytical process needs to be checked to ensure it meets the same level of quality as other analytical steps, but sometimes it’s tricky to know how to start with that! There are two related but fundamentally different parts to this; quality assurance and code review.

3.7.1 Quality assurance of code

Quality assurance of code and its output is similar to QA on an Excel spreadsheet or any other manual process; you’re checking to ensure that the process is analysing your data correctly and giving you a sensible output.

To do QA, you’ll want someone who has access to the code and all of the data, and ideally has good familiarity with the analytical process and the data itself. They will check things like:

  • Does the process give the same results as the process it’s replacing, or another equivalent comparison point?
  • Are the figures produced consistent with what they know about the data?
  • Are there NA values produced in unexpected places?
  • Are processes within the code applied in a logical way (are all ports in England included in an English ports figure), and are the calculations applied to give appropriate results (is a seasonal adjustment giving the expected output)?

Someone carrying out QA in this way doesn’t need to be a coding expert, and may need a more experienced coder to explain what complex pieces of code are actually doing to allow them check it thoroughly.

It should be relatively easy to build code QA into existing QA checks/documentation, and in many cases QA of the data and the code that produced it will be done at the same time.

3.7.2 Code review

In contrast to quality assurance, code reviewing is about evaluating the code itself and spotting the potential for issues further down the line.

As a result, it’s not always necessary to have access to the data to do a code review (although it often helps), and the person doing the review doesn’t need to be familiar with the dataset at all, but should be a competent coder. They will check things like:

  • Has your code got syntax problems that mean it will stop working in 6-9 months?
  • Is it written so that it will be easy to use, understand and modify in future?
  • Does the code do what the documentation says it does, and does this happen in a reproducible and stable way?
  • Is your code written and running in an efficient way?

Ideally, you should do a code review at least the first time new code is written, and every time major changes are made to the code after that.

Use of Github in combination with code reviewing is a really powerful tool. It offers a flexible and clear interface to actually carry out the code review, and stores that review alongside the code forever. This makes it easy for you to audit code reviewing on existing code bases; understand the risks within your code and how and when they were fixed, even if you weren’t in the team when that happened.

What you can do to achieve this:

  • Build code QA and review into your existing QA processes the first time code is written or when major changes are made
  • Learn how to read through current and old code reviews on Github by attending the Github for Analytical Managers training (available through C&C)
  • Encourage your team to get code onto Github to make code reviews easy to do and find again in a few years time!
  • You can see how code reviews work by attending the Code Reviewing Workshop offered through C&C

3.8 Testing your code

While in a crisis situation you can often find yourself debugging code, having code testing in place is an easy way to prevent the need for debugging in the first place.

Code testing makes use of packages like {testthat} to check that small units of code are functioning in the way they were intended, forever. This means that if code is edited or added to, you can be confident that this hasn’t broken something else further down the line. Usually testing like this compares the output of functions you’ve written using an example data set, so you can remove the impact of changing datasets and check if it’s the code or the data that has changed.

The kind of things unit testing checks includes:

  • The range of unique values produced
  • The size and shape of the output data
  • Whether processes happen without errors or warnings
  • That your code will warn you sensibly if something has gone wrong

Unit testing like this isn’t the easiest thing to do, and it requires your code to be written in a specific way (making use of functions). It’s therefore something to be aware of early on, but you may not be able to implement it immediately in your code.

What you can do to achieve this:

  • Understand where your team is making use of code which does not have unit testing associated with it, and record this as a risk in your analysis
  • Encourage your team to store the result of tools such as devtools::test_coverage() alongside their code so you can see at a glance what % of your code is untested
  • Understand what unit testing L&D resources are available, such as this unit testing book chapter to help you add testing to code as you go along
  • Get support to understand where unit testing could be easily added to your existing code, and where it would be more difficult