Introducing a product delivery culture at Etsy

Selecting a partner

Etsy had never worked with a software development consultancy. To be able to scale product delivery and bring in some outside expertise, they started looking for a vendor. After an extensive search, they selected Thoughtworks, primarily because they perceived a close cultural fit and deep alignment regarding modern software development.

Thoughtworks has a similar approach to Agile as Etsy – being Agile rather than doing Agile. Etsy's team was not looking to adopt a methodology that blindly follows strict rituals and practices, and wanted a partner that bases its approach on agile principles and culture. Thoughtworks is known for technical excellence and leadership in DevOps, Continuous Integration, and Continuous Delivery, all of which Etsy relies on heavily. In addition, the two companies shared similar principles regarding employee care, diversity, open-source, and technology sustainability.

The Thoughtworks team started by embedding into product teams, working with both buyer and seller teams, on various critical initiatives such as payment model changes, Search Engine Optimization (SEO), and a notification platform. This enabled Thoughtworks to understand how Etsy worked, created an immediate impact on delivery, and improved development practices such as automated testing.

For Thoughtworks, Etsy was quite different from the typical client. They are a fully digital company, and their technical process and product practises are very mature. Unlike most clients who engage Thoughtworks, Etsy had no large transformation to undertake, and no ingrained habits that needed a major shift. What they were seeking was outside perspective and supplementive skills, to help them incrementally improve and tackle their pressing initiatives. The culture and principles were already very aligned with Thoughtworks’ own, so the partnership became more of an exchange of ideas drawn from different experiences.

Scaling Product Delivery and Discovery

Thoughtworks and Etsy formed a cross-functional Product Delivery Culture (PDC) team to analyze the current product delivery and discovery process. They analyzed the entire software development value stream, looking closely at the complaints of managers and team members. Selected events were replayed to gain a deep understanding of “how” it happened rather than who?. The Thoughtworks team, because they hadn’t lived through the problems, were able to bring an objective outsider perspective.

There were a number of things Etsy was doing really well:

• Cross-functional teams: They structured their teams around “4 table legs”: Product, Design, Engineering, and Analytics. All planning and delivery practices happen with collaboration among the groups.
• Incremental delivery of value to users: Continuous delivery is the core practice of their approach. A code change sitting in a repository and not deployed to users is not valuable, and effectively costs the company money. Etsy deploys hundreds of times a day, with a 30-minute commit to production lead time.
• Features are prioritized using data: Product managers and data analysts collaborate to determine an expected value, measured by KPI changes. If the feature does not deliver enough value versus its cost, as evidenced by an A/B test, it is not rolled out, and the code is removed.

The team also uncovered opportunities for improvement:

• Each team was doing delivery in different ways, having disparate processes and rituals, making onboarding new employees and changing teams difficult. Some teams were using a process like Scrum, while others did not use any formal process.
• There were a number of initiatives that had not created value as they expected. For example, Etsy tried a gift wrap feature that had lower than expected adoption rates by sellers and buyers.
• Prior to 2018, Etsy had been through a transitional period, with pivots in product strategy and leadership changes leading some product managers to expres a desire for greater ownership over their product decisions.

Time to learning

The team created an improvement program, based on lean thinking and the ideas of Marty Cagan, who had recently presented at Etsy. The leadership team was reading his book Inspired.

I promise you that at least half the ideas on your roadmap are not going to deliver what you hope. (By the way, the really good teams assume that at least three quarters of the ideas won’t perform like they hope.)

-- Marty Cagan, Inspired (p19)

After some initial research the team came up with a metric they called Time to Learning – the time it took for a product team to validate an idea with a customer and gain a better understanding of its value. They had a baseline of 50 days that they wanted to reduce.

They were also looking at other metrics.

• Improving the time to market
• Improving the effectiveness of products (e.g. revenue, customer engagement)
• Employee happiness

To impact the KPIs, they came up with a number of solution hypotheses:

Light-weight prototypes that incorporate more direct user research

Etsy has strong experimentation infrastructure and analytics capabilities. They run many concurrent A/B tests on the marketplace. The problem the PDC team observed is that the feedback cycle can be quite slow. For an idea to be ready for an A/B test it had to be near production-level quality. It also had to have enough data to show statistical relevance. A/B tests would run for months in lesser-used parts of the marketplace.

To set your expectations, strong teams normally test many product ideas each week—on the order of 10 to 20 or more per week. I want to emphasize that these are experiments, typically run using prototypes. A prototype is not something that’s ready for prime time and certainly not something your company would try to sell and stand behind. But they’re immensely useful, as they’re all about learning fast and cheap.

-- Marty Cagan, Inspired (p27)

To reduce the Time to Learning metric, the PDC team applied a lean UX approach. First, they started by quickly creating lo-fi prototypes and immediately showing them to users. Second, the team increased user feedback sessions to every two weeks and reduced the level of ceremony. Third, they created a dual-track system, with a design and research team continually working ahead of a delivery team. The idea was to efficiently gain confidence in an idea before it qualifies for an A/B test. By reducing the cost of experimentation, they could increase the variety and quantity of experiments, finding ideas that showed more value.

Figure 1: Experiment validation process

Blueprint for product delivery

The PDC, in conjunction with product team leaders, came up with a set of product delivery principles and related delivery practice changes. The practices included: smaller more focused stories, redesigning story walls to better fit the team process, using spikes to reduce technical uncertainty, and better ways of managing tech debt. They did not want to remove the autonomy of teams. Continuous improvement is a big part of Etsy culture. These practices are meant to be treated as sensible defaults that can be adapted and improved for each team’s specific context.

The PDC team first tested the practice changes on pilot product teams to establish their effectiveness and then iterated. Once the practice demonstrated an improvement, the team added to a knowledge base and provided light coaching to introduce the new practices to all product teams across Etsy.

Lean Portfolio management

Some large initiatives were still being carried out, even though there was not clear evidence of the initiative creating value. This is the typical sunk cost problem. To solve this, the PDC team suggested taking a lean approach to portfolio management.

Our runway should be a list of hypotheses to test, not a list of requirements to build. When we reward our teams for their ability to deliver requirements, it’s easy to rapidly bloat our products with unnecessary features—leading to increased complexity, higher maintenance costs, and limited ability to change. Features delivered are not a measure of success, business outcomes are. Our runway is a series of questions that we need to test to reduce uncertainty and improve our understanding of growth opportunities.

-- Humble et al, Lean Enterprise (p96)

Instead of creating lengthy project specifications and plans, the PDC team coached product teams to develop outcome statements for each initiative. With outcome statements, starting with a customer or business outcome allows the team to experiment with many different ideas to achieve the outcome by tying them to measurable KPIs. The less precise an outcome is, including its KPIs, the harder it is to specify. When preparing for possible initiatives, each product team created a two-page brief summarizing the outcome, the measurable impact, and the initial investment required.

The executive team prioritized the outcome statements, and each statement was viewed as a “bet” amongst a portfolio. Inspired by the Lean Value Tree concept, the executive team would practice a regular cadence to determine if a bet was paying off, giving them the opportunity to pivot early.

Stronger product & engineering collaboration

Another improvement was to include engineers early in the product ideation process and to have them join user feedback sessions. The team tested engineers joining the hypothesis generation and validation sessions and working closely with designers during design sprints. The results of the test showed a number of benefits. Engineers improved their understanding of the customer, building empathy for their needs. They could provide on-the-spot feedback on the feasibility of an approach and quickly create prototypes to demonstrate ideas to users. Cross-functional engagement allowed engineers to contribute a number of great ideas that made it to the product.

After the initial PDC initiative Etsy continued to strengthen the collaboration between groups. At every level of the organization, product and engineering are partners, which means they’re able to have healthy trade-off discussions. The platform group expanded the use of product managers for infrastructure and developer tooling teams – encouraging teams to think about technical capabilities as products and end-users being Etsy engineers. The technical product managers do research and prioritization activities, just as they would for a customer product.

What was the impact of product delivery culture?

Testing with pilot teams indicated that the Time to Learning went from 50 days to 5 days. During the pilot, one product manager said, “What I love about the [direct user research sessions] pilot is that it lets us test our ideas faster, it was really nice to not spend as much time diving deep into something that we didn’t even know we were going to use.” An initiative on inventory quality that was predicted to deliver a seven million dollar revenue increase instead delivered eleven million dollars in incremental value using the PDC suggested improvements. Following this pilot success, the team coached the entire product development organization on the product delivery approach.

According to Mike Fisher, “It is very hard to isolate the benefits of a process change on an organization of several hundred engineers since we don’t run A/B experiments of this type. However, we did look at high level productivity metrics after having PDC implemented for over 2 years, and all of our metrics were very positive. We had improved our velocity by doing this work, and we were able to produce ultimately more benefits for the buyers and sellers – more features, better search, better processes; the whole marketplace became better for buyers and sellers and that was the goal.”