VCR, the Must-Have Companion for API Testing

October 04, 2022

by Junjie Chen

Unit testing external APIs can be time-consuming and non-deterministic because the unit test has to rely on the external service, and assertions would fail if the API response is inconsistent. In this post, I would like to share with you why VCRs are a handy solution for handling these API calls in unit testing, along with how we open sourced our own VCR solutions for C# and Java. Unit testing ensures that all code meets quality standards before it is deployed. Higher code coverage reduces the number of bugs in software; therefore, writing a unit test is a vital skill that all developers should learn.

What is VCR testing?

When writing a unit test that has API calls that rely on an HTTP request being sent over the network and being processed by an external dependency, we cannot be certain that a valid response will be received, or even any response at all. There could be network issues, temporary outages, or other external factors beyond our control, meaning we may not receive the expected response every time - resulting in unit test failures.

VCR, like the now-obsolete Video Cassette Recorder technology from years ago, is a tool that records HTTP requests and responses to a "cassette" file which can be replayed on later runs of the test suite as if it was the real response, effectively enabling offline testing. The first prominent VCR solution was introduced in 2010 for Ruby and was then ported into other languages. When first running a unit test with VCR, HTTP interactions such as the request body, request headers, request URL, status code, and other details about the request, will be saved into a cassette file, typically as a .yaml or .json file. After the cassette file is generated, re-running the unit test will not make the live API call; instead, it will use the previously-saved HTTP interactions from the corresponding cassette file.

Pros of VCR Testing:

Unit tests behave consistently by using the previously stored HTTP interactions, avoiding potentially different data from the live API call
External service downtime does not affect unit tests since the request and response are already saved locally and no live API calls need to be made
Unit tests with API calls run faster since the response is available immediately
Unit tests can run offline once recorded

Cons of VCR Testing:

Auto-generated cassette files can introduce a significant number of line changes that could be distracting during code review (see last section for how to hide them in a pull request)
HTTP interactions are stored in plaintext in a file, which can pose security concerns (see last section for more details on filtering sensitive data)
Cassettes need to be re-recorded if the API response changes (to ensure unit tests reflect the latest data)

The result of using VCR in all seven client libraries

At EasyPost, we have seven client libraries: Python, Ruby, PHP, Node, C#, Java, and Golang. Nearly every function in the client libraries involves at least one API call.

After implementing the VCR solution in our seven client libraries, we no longer had to worry about inconsistent API responses. For example, in our client library, we have a regenerate rate function that makes a live API call; we might receive different rates depending on what day of the week or time of day we ran the unit test and the unit test assertions would fail, but we know that the code itself never changed. By using VCR in unit testing, it also makes debugging easier during the implementation of new functions in client libraries, because we can see the entire HTTP interactions instead of a single error message from the API response. For instance, you can see the request body that you pass in and the API endpoint you hit - these are the two common mistakes you can make when making an API call.

Finally, the VCR solution saves us so much time during unit testing. We do not need to wait for the live API call after the HTTP interactions are recorded, so we saw unit tests speed up significantly from 108 seconds to just 1.8 seconds. This is an approximate 50x jump and also made our GitHub Action CI test much faster.

Without the VCR solution, live API call unit testing take 108.6 seconds

With VCR solution unit testing only takes 1.8 seconds

Our own VCR solution for Java and C#

When I joined the team, our Java and C# client libraries were using live API calls in unit testing because there was not a good VCR solution for these two languages. This caused several problems:

Unit tests on CI in GitHub action were skipped because of inconsistent responses that could make test assertions fail
Several unit tests would fail due to inconsistent responses from our API calls
Unit testing speed was slow due to live API calls in testing

We thought about using mock testing for Java and C# libraries, but it would take a lot of time to create objects. For example, creating a shipment object is time-consuming to manage because it has many properties, and lastly, we also want to use real requests from the API call for unit testing.

Most languages have their VCR solution built by open-source contributors, but when it comes to VCR integrations for our Java and C# test suites, we realized there were some problems with other open-source alternatives on the market. The C# VCR only targets .NET Standard 1.6, which potentially causes compatibility issues with our target framework, and Java VCR integration is not possible with our API endpoint because our servers do not accept self-signed certificates, which it requires. Therefore, our Developer Experience team ended up creating EasyVCR, an open-source tool for Java and C# that allows anyone to record HTTP requests and responses. Shout out to Nate Harris who spent countless hours building the EasyVCR tools. EasyVCR was started in early 2022 and was officially released in May 2022. They are open-source so anyone can take advantage of the same great test tooling we use every day. I highly encourage you to check them out.

As part of this release and ongoing enhancements, we have made several improvements in our own VCR solutions for Java and C#:

Added the ability to log warnings and errors
Added support for multiple modes
Implemented censoring functionality that allows users to filter out sensitive information before storing it in a cassette
Supported .NET Standard and NET/.NET Core in C#
Implemented optional expiration dates for cassettes
Customized how a recorded request is determined to be a match
Improved documentation

VCR tips and tricks

Each language's VCR provides flexibility, allowing users to customize the VCR configuration, such as filtering any sensitive data in a cassette file or changing the request matching logic.

Request match

To replay previously-recorded cassettes, the VCR needs to match new HTTP requests against the details of the recorded one. Often, by default, a recording is matched based on the HTTP method and URI. You can customize this matching in one or more of the following options:

Host: The domain name of the server
Body: The body of the API request
Header: The request headers
Path: The path of the URI
Query: The query string of the request URI

Our seven client libraries have strict matching rules to make sure every test replays its corresponding interactions; in addition to the defaults, we include body, query, and URI in the request matching rules. If you have many unit tests that make API calls to the same endpoint, I would highly recommend adding additional options to prevent a unit test from replaying the wrong cassette file.

Filter sensitive data

When interacting with an API service, you will often need to provide authentication such as an API key as part of the request. This data will be present in the cassette file which can pose a serious problem in a public repository. Before storing HTTP interactions into a cassette file, there is an option to censor data to avoid exposing sensitive information. Our seven client libraries redact sensitive data before storing it into a cassette file; you can find the full code configuration in our Ruby library which uses built-in filter_sensitive_data.

Below is a simplified example of redacted cassette file looks like

interactions: 
- request:     
       body: '{"address": {"name": "Jack Sparrow", "street1": "388 Townsend 
St", "street2": "Apt 20", "city": "San Francisco", "state": "CA", "zip": 
"94107", "country": "US", "email": "test@example.com", "phone": 
"5555555555"}}'     
        headers:       
            Content-Length:       
             - '213'       
             Content-Type:       
             - application/json       
             authorization:       
             - <REDACTED>       
             user-agent:       
             - <REDACTED>     
        method: POST     
        uri: https://api.easypost.com/v2/addresses     
        response:     
        body:       
             string: '{"id": "adr_…", "object": "Address", "created_at": "2022-
08-15T18:51:18+00:00", "updated_at": "2022-08-15T18:51:18+00:00", "name": 
"Jack Sparrow", "company": null, "street1": "388 Townsend St", "street2": 
"Apt 20", "city": "San Francisco", "state": "CA", "zip": "94107", 
"country": "US", "phone": "<REDACTED>", "email": "<REDACTED>", "mode": 
"test", "carrier_facility": null, "residential": null, "federal_tax_id": null, "state_tax_id": 
null, "verifications": {}}'     
      status:       
         code: 201

Re-record the cassette

An external API could change over time, which makes previously recorded cassettes out of date. There are multiple ways to re-record the cassette to reflect the latest change on the external API:

Manually delete the cassette file and re-run the corresponding test. If no cassette exists for the corresponding test, the VCR by default will automatically record a new one.
Configure an expiration date on a specific cassette. The VCR will check the recorded_at timestamp to determine if it has expired and you can choose to re-record the cassette.
Set recording modes to determine when new recordings are made. There are several recording modes available; each language's VCR tool may have a different name. For example, in Python's VCR, there are four modes: once, new_episodes, none, and all. If the external API response changes periodically and you want the test data to be up-to-date, new_episodes is a mode that can be used, because it will re-record the cassette if the HTTP request details do not match the previously-recorded one.

Ignore the cassette diffs in pull requests

As previously mentioned, auto-generated cassette files could be distracting during code review; even a little change in the source code can bring a lot of changes in the cassettes if you re-record the file. However, it is possible to hide the cassette files in pull requests by adding two lines of code in the .gitattributes file in your repository.

* text=auto 
spec/cassettes/**/* -diff

I hope you found this blog useful when comes to API testing. We cannot wait to see you start using EasyVCR Java and EasyVCR C# in your project for unit testing. We also welcome any suggestions or questions, our Developer Experience team actively monitoring issues and pull requests, feel free to submit one for the EasyVCR C# repo, EasyVCR Java repo, or one of our seven client libraries!