Don’t Do Small Pull Requests

How asynchronous reviews and wait times harm throughput and code quality

Smaller the pull request, the better? (illustration by Jane Kim)

Introduction

In this blog post, I share my learnings in the last couple of months regarding the delivery process of our engineering team at Elli. It was during this time that I came across Dragan Stepanovic’s presentation at the Seattle Software Crafters Meetup. This post is in large part inspired by the work Dragan Stepanovic presented in his talk.

My Hypothesis

Does smaller PR size always mean higher velocity and code quality? (illustration by Jane Kim)

During our sprint retrospective, my team and I identified a core bottleneck in our delivery process. During our sprint, we worked on pull requests (PRs) that stayed open for an unusually long time, waiting for reviews, comments to be addressed, merge conflicts to be resolved, you know the deal. As a way to eliminate this bottleneck, I suggested: let’s simply slice our tasks and PRs to smaller parts. My hypothesis was that smaller PRs are easier to comprehend, thereby allowing more thorough and time-efficient reviews and, as a result, increase our team’s velocity and code quality. I was convinced this would be the solution to our bottleneck.

The Method

In order to test my hypothesis I did two things:

  1. First, I started to slice my PRs to really small parts.
  2. Second, I decided to analyse some data.

I analysed the following variables over the course of 9 months:

  • Size of the PR: the unit of measurement for PR size in this case would be the number of changes that this PR includes. Azure DevOps reports a “change count” metric for pull requests, which we will conveniently use for our data analysis.
  • PR’s Time to Completion: The total time (in business hours) taken from the point at which the PR was opened to when it was merged.
  • Reaction time (or wait time): The time taken for a reviewer to “react” with a comment or an approval to the PR.

Disclaimer: This analysis does not consider the corresponding tasks and therefore ignores the time spent before the PR was created.

The Analysis

Do we increase our velocity when we slice asynchronous pull requests to smaller parts?

To get a better understanding of the relation between PR size and time to completion, I plotted a line chart that shows the number of business hours spent until the PR merges. The graph clearly shows that smaller PRs complete faster, which we all probably assumed from personal experience. The drop-off to the far right in the graph is an interesting exception and we will get back to this in a later part of this post.

The relation between PR size and time to completion of a PR in business hours, proving the initial hypothesis that smaller PRs complete faster. (graph by Matthias Förth)

However, when I sliced my PRs really small I did not get the impression that I completed my work faster, rather it felt like the opposite. The next graph reveals that the time per change actually reaches its maximum when change count is really small (to the left) and reaches its minimum when change count gets exceptionally large (to the right).

The time required to merge a change in business hours (purple) and the ratio of PR with comments (green) depending on the PR size to illustrate the effect of the asynchronous discussion and approval process. (graph by Matthias Förth)

Ultimately our goal is not to get as many PRs closed as possible but to merge meaningful code. If we want to achieve that in the fastest way possible we should, in fact, make larger PRs, according to these numbers.

Yet, in the middle, there is an interesting minimum on the completion time per change at 5–10 changes, which suggests that slicing a 22 change-PR in half actually reduces its time for completion. Let’s look at this dip more closely, since this area is quite important (we merge 50% of our changes as PRs with a change count between 5 and 23).

What is the effect of asynchronous comments and responses?

The green line shows the percentage of PRs with one or more comments. Below 13 changes there are more PRs without comments. Above 13 changes there are more PRs with comments. This asynchronous discussion that arises as a result of comment submission leads to the increase in the time to completion. Below 13 changes, the wait time for a reviewer to react to a pull request makes up a large part of the time for a PR to complete. Above 13 changes, the initial reaction time (or the wait time taken for a reviewer to react) makes up less of the total completion time since the asynchronous discussion includes not just the initial wait for comments but also addressing them, and potentially further review loops.

How big is the impact of these reaction times actually? To gauge the size of this impact, I plotted the wait time until the first reaction (approval or comment) as a percentage of the total time taken for the PR to complete.

Comments per change as an indicator for code quality (purple) and the ratio of the time to a first reaction compared to the total completion time of the PR (pink) to show the tradeoff between the impact of wait times in the asynchronous review process and code quality.

We see that especially for smaller PRs this initial reaction time makes up the majority of the time to actually complete the PR. Yet, even though the majority of small PRs get approved without any comment from the reviewer, the ratio of comments per change is still the highest in this area. We would conclude from this that we get the most in-depth reviews on these small PRs, but for many, the code can be merged right away. So what do we make out of this now?

My Findings

In general, aim for medium-sized PRs! (illustration by Jane Kim)
  • To get the best combination of merging changes quickly but still receiving an in-depth review we should aim for medium-sized PRs.
Pass the PR-baton as soon as possible! (illustration by Jane Kim)
  • On these medium-sized PRs, the reaction time makes up a considerable portion of the total time to complete the PR. Therefore, there is quite some potential to speed up delivery by keeping any reaction time low (i.e. complete reviews, addressing comments or pipeline failures as efficiently as possible)
Do the heavy-lifting of PRs together in pairs! (illustration by Jane Kim)
  • The first chart showed a drop-off in the “time to completion” for very large PRs. This is due to the fact that many of these PRs have been completed in pairing mode. At Elli, we tend to do large refactoring’s or complex business logic implementations via pair programming. Instead of doing the review process asynchronously, where one person creates the PR and another reviews, we synchronously code and review in the pairing session. Therefore, there are no comments on the PRs and approval may be granted right away. This leads to the drop in the completion time for very large PRs, but this reduction should very well apply to medium-sized PRs as well.

Conclusion

Going back to the initial problem of long-running PRs in our retro, I would now no longer simply recommend slicing the PRs to really small pieces as a blanket solution to faster delivery times. Instead, I would recommend slicing PRs into parts such that the “heavy lifting” is done via pair-programming. The more independently manageable PRs should have a medium size to justify the wait times of asynchronous reviews.

But ultimately, there is no magic number when it comes to pull request size. However, what I did discover from my data analysis were several important guidelines to keep in mind in order to optimise the pull request workflow of our product teams. I hope these findings are useful and extend to anyone who is interested in improving delivery times within their team or organization.

Learn More

At Elli, we are always in search of data-driven ways to improve the way we work together as a team. If you are interested in finding out more about how we work, please subscribe to the Elli Medium blog and visit our company’s website at elli.eco! See you next time!

About the author

Matthias Förth is a Product Owner and former Software Engineer focused on backend development. His current interests are the streamlining of development and pushing data-driven decision making at Elli.


Don’t Do Small Pull Requests was originally published in Elli Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

More
Back

More News

  • Press Release
    Oct 09th, 2024

    Volkswagen Group brand Elli launches smart charging product offensive for Europe

    More
  • Press Release
    Sep 17th, 2024

    Elli presents new charging products for e-fleets and companies at the IAA Transportation 2024

    More
  • Press Release
    Jul 01st, 2024

    "Elli Drive Plus" – New charging tariff to kick off holiday season in Europe

    More
  • Press Release
    Jun 07th, 2024

    Introducing the all-new Elli Charger 2: cost-effective charging to drive the energy transition at home

    More
  • Press Release
    Jun 07th, 2024

    Elli enters the industrial energy storage business

    More
  • Image
    Jun 07th, 2024

    Large Storage

    Download
  • Image
    Jun 06th, 2024

    Charger 2 on garage wall

    Download
  • Image
    Jun 06th, 2024

    Charger 2 on Wall

    Download
  • Image
    Jun 06th, 2024

    Charger 2 on wall 2

    Download
  • Image
    Jun 06th, 2024

    Charger 2 Isolated

    Download
  • Press Release
    May 27th, 2024

    Elli presents its first smart energy tariff: Volkswagen Naturstrom Flex empowers customers to save on electricity costs

    More
  • Press Release
    Apr 22nd, 2024

    Hannover Messe 2024: Pioneering Research Project – Charging Electric Vehicles with Wind Energy

    More
  • Press Release
    Apr 09th, 2024

    "Charging Site Management" – Elli launches new commercial product

    More
  • Press Review
    Mar 28th, 2024

    VW macht Lade- und Energiegeschäft schick für Investoren

    More
  • Press Release
    Mar 05th, 2024

    New partnership with SIXT – Elli opens its charging network to external partners

    More
  • Press Review
    Feb 22nd, 2024

    Elli: Ladenetz wächst auf 600.000 Ladepunkte

    More
  • Download (.zip)
    Feb 18th, 2024

    Elli Logo

    Download
  • Image
    Feb 17th, 2024

    Giovanni Palazzo, CEO Elli & SVP Volkswagen Group Charging and Energy

    Download
  • Image
    Feb 16th, 2024

    Flexpole_1

    Download
  • Image
    Feb 16th, 2024

    Flexpole_2

    Download
  • Image
    Feb 16th, 2024

    Flexpole white background

    Download
  • Image
    Feb 16th, 2024

    Wallbox_1

    Download
  • Image
    Feb 16th, 2024

    Wallbox_2

    Download
  • Image
    Feb 16th, 2024

    Wallbox_3

    Download
  • Image
    Feb 16th, 2024

    Wallbox_white_background

    Download
  • Press Release
    Dec 28th, 2023

    Elli expands to 600,000 charging points across Europe

    More
  • Press Release
    Nov 02nd, 2023

    Elli launches pan-European charging solution for electric fleets

    More
  • Video
    Sep 13th, 2023

    ICNC23 Keynote by Giovanni Palazzo

    Beyond the Plug- Elli’s Evolution into a Holistic Energy Ecosystem

    More
  • Video
    Sep 08th, 2023

    IAA MOBILITY 2023- Giovanni Palazzo speaks about Elli´s position and potential

    More
  • Press Release
    Sep 05th, 2023

    Elli at IAA 2023: Showcasing Elli’s new e-mobility products, and future projects

    More
  • Elli Insights
    Aug 03rd, 2023

    Navigating the Evolution of Stream-Aligned Teams: Lessons from Our CI/CD Journey

    When they fly like a bird and when they hit the unexpected window hard (illustration made by Lukas Hanke) Introduction and Scope At Elli we aim for “stream-aligned teams”. According to the book “Team Topologies”  — by Manuel Pais and

    More
  • Elli Insights
    Aug 03rd, 2023

    On Communities of Practice

    When they fly like a bird and when they hit the unexpected window hard (illustration made by Lukas Hanke) Introduction and Scope At Elli we aim for “stream-aligned teams”. According to the book “Team Topologies”  — by Manuel Pais and

    More
  • Elli Insights
    Aug 03rd, 2023

    To TOX or not to TOX

    How technical and operational excellence is achieved (illustration made by Lukas Hanke) In our very first blog post, we introduced you to Elli’s guiding principles for achieving engineering excellence. In this article, we delve deeper into one of the

    More
  • Press Review
    Jul 26th, 2023

    Rethinking mobility

    More
  • Press Release
    Jul 13th, 2023

    Volkswagen Group and Elli launch electricity trading on the European energy exchange

    More
  • Press Release
    Jul 10th, 2023

    Volkswagen Group pushes ahead with strategic realignment of charging and energy business

    More
  • Press Release
    May 04th, 2023

    Shell and Volkswagen push ahead the expansion of charging infrastructure: Opening of the first innovative Flexpole charging station

    More
  • Social Media
    Apr 18th, 2023

    Elli’s charging network has now exceeded a remarkable 500,000 charge points

    We’re incredibly proud to be at the forefront of Europe’s e-mobility service providers. And it’s especially gratifying that our customers enjoy easy access to charging stations across 28 European countries.

    More
  • Video
    Mar 31st, 2023

    Wind and sun do not depend on charging times

    How can the batteries of electric vehicles contribute to make better use of renewable energies.

    More
  • Video
    Mar 30th, 2023

    Elli Fleet Charging

    The simplest way to manage your company e-cars

    More
  • Social Media
    Mar 29th, 2023

    „Flotte! The leading fleet trade fair“ in Düsseldorf

    Visit Elli at booth K23-27

    More
  • Elli Insights
    Feb 01st, 2023

    Parlez-vous OCPP?

    How charging stations communicate with their central backend system OCPP, c’est quoi? If we observed a group of people with different native tongues trying to engage in discussion, we would likely observe them consolidating to the language that is

    More
  • Social Media
    Jan 12th, 2023

    Elli Expands Charging Network in Europe with 400,000 Points

    More
  • Social Media
    Jan 12th, 2023

    New Ellians Start Their Journey in January 2023

    More
  • Social Media
    Dec 01st, 2022

    What are ELLIans doing after the first company meeting?

    More
  • Social Media
    Dec 01st, 2022

    New benefits for our ELLIans!

    More
  • Social Media
    Dec 01st, 2022

    All the MSP Business Unit on site in Munich

    More
  • Social Media
    Dec 01st, 2022

    We welcomed the new Ellians joining in December

    More
  • Elli Insights
    Dec 01st, 2022

    Catch Me If You Can — Memory Leaks

    Catch Me If You Can — Memory Leaks A retrospective on a memory leak Elli engineers vs. memory leak (illus. by Jane Kim) Introduction Memory leaks are one of those things that, when they happen, can really throw you in at the deep end. Diagnosing

    More
  • Elli Insights
    Aug 01st, 2022

    Electric Vehicle Charging for Newbies

    A quick read for all newbies to EV charging! Creating a sustainable future means changing the way we get around. Perhaps this means switching to “greener” modes of transportation like commuting by bike or public transit. It could also mean reducing

    More
  • Video
    Nov 07th, 2022

    Discover Cupra Plug & charge.

    Making the street charging easier for you.

    More
  • Video
    Nov 07th, 2022

    How to use Cupra Plug & Charge.

    The fast, easy and secure way to charge your Cupra born.

    More
  • Press Review
    Oct 25th, 2022

    Dank einer Vereinbarung zwischen Elli und Vattenfall gibt es jetzt über 24.000 neue Stationen

    More
  • Video
    Oct 16th, 2022

    Collect points with this app and charge your vehicle for free.

    Thanks to the partnership between Elli and &Charge, users can collect "&Charge kilometers" via the &Charge app and redeem them as € vouchers to use them in the Elli app for free charging kilometers. There is more information in this video by Un Gallego en Munich.

    More
  • Video
    Oct 10th, 2022

    The Elli app: the most convenient workflow ever to charge your car.

    Unlock over 340 000 charging stations across Europe with the Elli app. Watch this video by Jonah Plank to discover more.

    More
  • Press Review
    Oct 06th, 2022

    The Wallbox is now available at any Volkswagen dealer

    Now you can get your Wallbox directly from Audi, Seat, Cupra, Škoda and Volkswagen.

    More
  • Video
    Oct 01st, 2022

    Flexpole: Yes! The most flexible charging station for you.

    With the Flexpole there are no complicated grid connections, just choose a location and start charging your cars. Watch a full review of the Flexpole in this video by Stefan S.

    More
  • Press Review
    Sep 27th, 2022

    Das Elektroauto als mobile Powerbank

    More
  • Elli Insights
    Jun 01st, 2022

    Introduction to Elli Engineering: Our Guiding Principles

    The six guiding principles for technical excellence Elli is a brand of Volkswagen Group providing energy and electric charging solutions. Software and hardware engineering are key to the business. We as engineers focus on creating and maintaining awesome

    More