On Communities of Practice

When they fly like a bird and when they hit the unexpected window hard

(illustration made by Lukas Hanke)

Introduction and Scope

At Elli we aim for “stream-aligned teams”. According to the book “Team Topologies” — by Manuel Pais and Matthew Skelton, we focus on optimising our workflow when it comes to feature delivery and minimising the number of handovers between teams. To reduce handovers, these teams have end-to-end ownership of their services and infrastructure. Ownership means freedom and independence but also a high cognitive load. To reduce this load and upskill teams in cross-team topics like infrastructure or continuous integration and delivery (CI/CD), we have various Communities of Practice (CoPs). We are proud of our CoPs and our culture of continuous improvement. It has served us well but recently we hit some barriers we cannot push through with our current setup. Facing these difficulties we see the need for having dedicated teams to also drive such improvements across the organisation.

In this blog post, I want to share with you the story of how we got here, what our CoPs have done well and where they ended up coming short. To illustrate my points I will use one CoP as an example — the CI/CD CoP. But we see similar issues also in other CoPs.

Status Quo of CI/CD

Before we start, CI/CD at Elli is in a good state and it is consistently improving. Our services deploy multiple times per week or even per day as we deploy every merged change. Deployments usually take less than 30 minutes (median is around 25 minutes). The pie chart shows the 2023 survey results for the statement “I am happy with the state of CI/CD in my team”. In 2023, over 50% strongly agree, which is an over 20% improvement to 2022. But we noticed a couple of difficulties with our approach of solely relying on CoPs for topics like CI/CD. This blog post is about these difficulties and barriers, which we face on our way to turn the pie chart fully green.

The early achievements and the following decline

We have around 30% of our working time dedicated to technical and operational excellence (TOX) in software development. Most of this budget is spent on our TOX work in each of our feature teams. But it also includes the time we spend in our CoP’s one-hour weekly meetings and bi-weekly four-hour workshops. Reserving a time budget aside from feature delivery work has reaped many benefits across the entire organisation: We went from triggering releases with tags once or twice per story to releasing every merged change. The median deployment frequency is about 3–4 times per week now with a maximum of 20 for our main frontend. The median run time for our releases is around 25 minutes with deployments on several environments, end-2-end tests and so on.

Such improvements were either driven by some team with a need for it and then spread through the CoP or driven by the CoP itself. Workshops are especially effective for propagating best practices such as these.

However, the participation in these workshops, relative to the number of engineers working at Elli, has steadily decreased over the years.

To get a better understanding of why participation levels are declining, we administered a survey to identify the core causes of low participation rates. The main reasons for people not to join were:

  • Feature pressure in the team
  • Sprint rituals and many other meetings collide with CoP meetings
  • Engineers like working on features better
  • Topics are not relevant or exciting

The first two points are issues that came through our scale up. Engineering management needs to emphasize the priority and importance of the TOX budget repeatedly to our engineers to prevent it from falling by the wayside. Regarding scheduling, CoP meetings are visible in our company’s engineering department-wide calendar for everyone to see. Thus Product Owners and engineers should be wary of this and not schedule any conflicting meetings. Lack of prioritisation and commitment ultimately caused our CoP rituals to fall through.

But now to the really shocking part. Topics are not relevant and engineers rather choose more feature work. This is clearly on the CoPs and must be improved.

Out of the rabbit hole and addressing the needs of engineers

Every year, the CI/CD CoP, led by our senior infrastructure engineer –Madhusudan, conducts a survey on the state of DevOps here at Elli. After the 2022 survey, they decided to address these shortcomings with a couple of initiatives:

  • Pipeline template repository: A dedicated repository with templates for the initial pipeline setup and continuous improvements.
  • Upskilling workshop: An Elli Engineering Academy workshop on our general pipeline setup and fundamentals of Azure Pipelines.
  • “Bring your own pipeline” (BYOP) workshops: Teams can look into and improve their pipelines together with experts from the CI/CD CoP.
  • Data analysis: Collect data from Elli’s Azure Pipelines in Big Query for more in-depth analysis of our software delivery processes.

Let’s examine some key insights from the 2022 survey and compare them to the new 2023 results:

  • Complexity of pipelines: In 2022, 35% of participants thought our pipelines are complex. This went down to 20% in 2023. The usage of the pipeline templates definitely helps here as well as the BYOP workshops.
  • Lack of understanding of pipelines: In 2022, 45% of participants were not confident in modifying the pipelines. This decreased to 15% in 2023. The upskilling workshops surely did not hurt.
  • Flaky pipelines: PR pipelines fail for the wrong reasons 60% of the time in 2022 and 2023.
  • Slow pipelines: In 2022, 70% thought our pipelines are slow, in 2023 50% still have the need for speed. 60% of participants in 2022 actively waited for the pull request (PR) pipeline to complete. This dropped to 23% in 2023.

The first two issues we address rather well with the measures taken. Regarding the workshop participation, we clearly see that these upskilling and BYOP workshops have way higher participation than other workshops (roughly double). However, especially Elli Engineering Academy workshops require a lot of time to prepare and facilitate. It is unrealistic to hold these lengthy workshops every week for each of the different engineering topics. So far this year we do not see an improvement in the overall participation.

(illustration made by Lukas Hanke)

Flaky tests and slow pipelines however are more difficult. Let’s look into that in more detail and start with the drop of engineers waiting for the PR pipeline. We started collecting pipeline data only 3 months after the 2022 survey. For the time that we have data our median runtime of our PR pipelines is basically constant at about 15 minutes. So either, the drop of 60% to 23% is because of improvements before we collected data (I doubt it) or people adapted to the wait time and do other things in parallel, which is what I often hear when talking to other engineers.

This is something which we actually want to avoid. Starting a new task, which will occupy you for the next 4 hours and increase the amount of work in the delivery system, all because of a 15 minute wait time, is not desirable. So this decrease in people waiting may look better than it actually is. However, sitting around for 15 minutes is not great as well. But getting the runtime down to 10–12 minutes does not help. You still wait. You still jump to another task. We need less than 5 minutes, a small bathroom break, and getting the pipeline this fast is hard.

Also, it is difficult to make the pipelines and tests more robust. We have a failure rate of roughly 20% on the PR pipeline, also quite constant over time. The survey hints that these 20% are made up of 12% unnecessary failures and only 8% meaningful failures. This shows that teams see this pain point but struggle to significantly improve on it with their regular TOX work. For these hard topics, the small CoP time is not enough. What we really need is a dedicated team of experts with full-time capacity to drive such improvements.

Conclusion

We have seen early on at Elli that CoPs can drive great improvements across the engineering teams. Especially in our company’s earlier stages, our CoPs tackled realistically achievable and time-bound topics with high impact. But at some point you need to shift gears when an existing way of working does not achieve desired results. Our CoP approach struggled to generate the desired impact and lost support among engineers. And hence the need to challenge the status quo, be agile and look for better solutions.

(illustration made by Lukas Hanke)

Setting goals that are achievable with the CoP capacity, as well as focusing on upskilling and knowledge sharing across teams, shows promising participation levels and improvements. At the same time we should not buckle when continuous improvement gets tough. Instead, we can seek alternative data-driven solutions. Surveys and having honest conversations with our fellow engineers helped us see the need for a proper full-time team, small in size but highly skilled, that can dig deep down their specialty rabbit holes. They can work closely with the CoPs, share their insights and help teams to incorporate the tough improvements.

Learn More

At Elli, we are always in search of improving the way we work together as a team. If you are interested in finding out more about how we work, please subscribe to the Elli Medium blog and visit our company’s website at elli.eco! See you next time!

About the author

Matthias Förth is a Product Owner and former Software Engineer focused on backend development. His current interests are the streamlining of development and pushing data-driven decision making at Elli.


On Communities of Practice was originally published in Elli Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

More
Back

More News

  • Press Release
    Oct 09th, 2024

    Volkswagen Group brand Elli launches smart charging product offensive for Europe

    More
  • Press Release
    Sep 17th, 2024

    Elli presents new charging products for e-fleets and companies at the IAA Transportation 2024

    More
  • Press Release
    Jul 01st, 2024

    "Elli Drive Plus" – New charging tariff to kick off holiday season in Europe

    More
  • Press Release
    Jun 07th, 2024

    Introducing the all-new Elli Charger 2: cost-effective charging to drive the energy transition at home

    More
  • Press Release
    Jun 07th, 2024

    Elli enters the industrial energy storage business

    More
  • Image
    Jun 07th, 2024

    Large Storage

    Download
  • Image
    Jun 06th, 2024

    Charger 2 on garage wall

    Download
  • Image
    Jun 06th, 2024

    Charger 2 on Wall

    Download
  • Image
    Jun 06th, 2024

    Charger 2 on wall 2

    Download
  • Image
    Jun 06th, 2024

    Charger 2 Isolated

    Download
  • Press Release
    May 27th, 2024

    Elli presents its first smart energy tariff: Volkswagen Naturstrom Flex empowers customers to save on electricity costs

    More
  • Press Release
    Apr 22nd, 2024

    Hannover Messe 2024: Pioneering Research Project – Charging Electric Vehicles with Wind Energy

    More
  • Press Release
    Apr 09th, 2024

    "Charging Site Management" – Elli launches new commercial product

    More
  • Press Review
    Mar 28th, 2024

    VW macht Lade- und Energiegeschäft schick für Investoren

    More
  • Press Release
    Mar 05th, 2024

    New partnership with SIXT – Elli opens its charging network to external partners

    More
  • Press Review
    Feb 22nd, 2024

    Elli: Ladenetz wächst auf 600.000 Ladepunkte

    More
  • Download (.zip)
    Feb 18th, 2024

    Elli Logo

    Download
  • Image
    Feb 17th, 2024

    Giovanni Palazzo, CEO Elli & SVP Volkswagen Group Charging and Energy

    Download
  • Image
    Feb 16th, 2024

    Flexpole_1

    Download
  • Image
    Feb 16th, 2024

    Flexpole_2

    Download
  • Image
    Feb 16th, 2024

    Flexpole white background

    Download
  • Image
    Feb 16th, 2024

    Wallbox_1

    Download
  • Image
    Feb 16th, 2024

    Wallbox_2

    Download
  • Image
    Feb 16th, 2024

    Wallbox_3

    Download
  • Image
    Feb 16th, 2024

    Wallbox_white_background

    Download
  • Press Release
    Dec 28th, 2023

    Elli expands to 600,000 charging points across Europe

    More
  • Press Release
    Nov 02nd, 2023

    Elli launches pan-European charging solution for electric fleets

    More
  • Video
    Sep 13th, 2023

    ICNC23 Keynote by Giovanni Palazzo

    Beyond the Plug- Elli’s Evolution into a Holistic Energy Ecosystem

    More
  • Video
    Sep 08th, 2023

    IAA MOBILITY 2023- Giovanni Palazzo speaks about Elli´s position and potential

    More
  • Press Release
    Sep 05th, 2023

    Elli at IAA 2023: Showcasing Elli’s new e-mobility products, and future projects

    More
  • Elli Insights
    Aug 03rd, 2023

    Navigating the Evolution of Stream-Aligned Teams: Lessons from Our CI/CD Journey

    When they fly like a bird and when they hit the unexpected window hard (illustration made by Lukas Hanke) Introduction and Scope At Elli we aim for “stream-aligned teams”. According to the book “Team Topologies”  — by Manuel Pais and

    More
  • Elli Insights
    Aug 03rd, 2023

    To TOX or not to TOX

    How technical and operational excellence is achieved (illustration made by Lukas Hanke) In our very first blog post, we introduced you to Elli’s guiding principles for achieving engineering excellence. In this article, we delve deeper into one of the

    More
  • Press Review
    Jul 26th, 2023

    Rethinking mobility

    More
  • Press Release
    Jul 13th, 2023

    Volkswagen Group and Elli launch electricity trading on the European energy exchange

    More
  • Press Release
    Jul 10th, 2023

    Volkswagen Group pushes ahead with strategic realignment of charging and energy business

    More
  • Press Release
    May 04th, 2023

    Shell and Volkswagen push ahead the expansion of charging infrastructure: Opening of the first innovative Flexpole charging station

    More
  • Social Media
    Apr 18th, 2023

    Elli’s charging network has now exceeded a remarkable 500,000 charge points

    We’re incredibly proud to be at the forefront of Europe’s e-mobility service providers. And it’s especially gratifying that our customers enjoy easy access to charging stations across 28 European countries.

    More
  • Video
    Mar 31st, 2023

    Wind and sun do not depend on charging times

    How can the batteries of electric vehicles contribute to make better use of renewable energies.

    More
  • Video
    Mar 30th, 2023

    Elli Fleet Charging

    The simplest way to manage your company e-cars

    More
  • Social Media
    Mar 29th, 2023

    „Flotte! The leading fleet trade fair“ in Düsseldorf

    Visit Elli at booth K23-27

    More
  • Elli Insights
    Feb 01st, 2023

    Parlez-vous OCPP?

    How charging stations communicate with their central backend system OCPP, c’est quoi? If we observed a group of people with different native tongues trying to engage in discussion, we would likely observe them consolidating to the language that is

    More
  • Social Media
    Jan 12th, 2023

    Elli Expands Charging Network in Europe with 400,000 Points

    More
  • Social Media
    Jan 12th, 2023

    New Ellians Start Their Journey in January 2023

    More
  • Social Media
    Dec 01st, 2022

    What are ELLIans doing after the first company meeting?

    More
  • Social Media
    Dec 01st, 2022

    New benefits for our ELLIans!

    More
  • Social Media
    Dec 01st, 2022

    All the MSP Business Unit on site in Munich

    More
  • Social Media
    Dec 01st, 2022

    We welcomed the new Ellians joining in December

    More
  • Elli Insights
    Dec 01st, 2022

    Catch Me If You Can — Memory Leaks

    Catch Me If You Can — Memory Leaks A retrospective on a memory leak Elli engineers vs. memory leak (illus. by Jane Kim) Introduction Memory leaks are one of those things that, when they happen, can really throw you in at the deep end. Diagnosing

    More
  • Elli Insights
    Oct 04th, 2022

    Don’t Do Small Pull Requests

    How asynchronous reviews and wait times harm throughput and code quality Smaller the pull request, the better? (illustration by Jane Kim) Introduction In this blog post, I share my learnings in the last couple of months regarding the delivery process of

    More
  • Elli Insights
    Aug 01st, 2022

    Electric Vehicle Charging for Newbies

    A quick read for all newbies to EV charging! Creating a sustainable future means changing the way we get around. Perhaps this means switching to “greener” modes of transportation like commuting by bike or public transit. It could also mean reducing

    More
  • Video
    Nov 07th, 2022

    Discover Cupra Plug & charge.

    Making the street charging easier for you.

    More
  • Video
    Nov 07th, 2022

    How to use Cupra Plug & Charge.

    The fast, easy and secure way to charge your Cupra born.

    More
  • Press Review
    Oct 25th, 2022

    Dank einer Vereinbarung zwischen Elli und Vattenfall gibt es jetzt über 24.000 neue Stationen

    More
  • Video
    Oct 16th, 2022

    Collect points with this app and charge your vehicle for free.

    Thanks to the partnership between Elli and &Charge, users can collect "&Charge kilometers" via the &Charge app and redeem them as € vouchers to use them in the Elli app for free charging kilometers. There is more information in this video by Un Gallego en Munich.

    More
  • Video
    Oct 10th, 2022

    The Elli app: the most convenient workflow ever to charge your car.

    Unlock over 340 000 charging stations across Europe with the Elli app. Watch this video by Jonah Plank to discover more.

    More
  • Press Review
    Oct 06th, 2022

    The Wallbox is now available at any Volkswagen dealer

    Now you can get your Wallbox directly from Audi, Seat, Cupra, Škoda and Volkswagen.

    More
  • Video
    Oct 01st, 2022

    Flexpole: Yes! The most flexible charging station for you.

    With the Flexpole there are no complicated grid connections, just choose a location and start charging your cars. Watch a full review of the Flexpole in this video by Stefan S.

    More
  • Press Review
    Sep 27th, 2022

    Das Elektroauto als mobile Powerbank

    More
  • Elli Insights
    Jun 01st, 2022

    Introduction to Elli Engineering: Our Guiding Principles

    The six guiding principles for technical excellence Elli is a brand of Volkswagen Group providing energy and electric charging solutions. Software and hardware engineering are key to the business. We as engineers focus on creating and maintaining awesome

    More