Catalog Crunch

The TL/DR:

I was tasked with creating an initiative to attempt to close a data gap in our product. I did this by collaborating with finance and leadership across departments to create an internal, cross-functional incentive contest. I led a 3-month, company-wide initiative that took into account every department, their different technical competencies, bandwidth, and general constraints. The result of this work provided a new SLA to take to the market, increased domain knowledge org-wide, and added over 13,000 data points to our product, impacting over 11 different customers! This is the story of how I designed and led the Catalog Crunch Contest.

Introduction

An internal data-audit was conducted by our data analyst, which determined that we had thousands of missing crucial data points that needed to be filled. These missing data points were visible to the user and as a result, made it very easy for users to question the accuracy and quality of our data, as this experience was less-than-optimal.

Open source software updates hourly, as there are many changes that occur on a daily basis, both to existing software packages and new ones ingested daily. With this being the case, there can be hundreds, sometimes even thousands, of changes to data points on a weekly bases. At the time of this initiative, we only had one data analyst, who had other responsibilities, outside of data maintenance. As a result, this was going to require a coordinated effort org-wide. I was tasked with designing a solution that could allow us to, more rapidly, close the missing data gaps, and incentivize internal stakeholders to utilize their idle time to work towards this effort, an initiative now known as the Catalog Crunch Contest.

Discovery

While internal stakeholders served as the users in this project, there was still knowledge about team structures and individual users experience, that I needed to understand. I held 22 sessions over slack video and zoom, meeting with users in every department, to get a better understanding of the breakdown of technical knowledge, bandwidth, and prior data research experience. Through meeting with individuals of all departments, I was able to breakdown the variations between the individual team structures, which would impact the experience I was designing a solution for.

Visual breakdown of the different teams/stakeholders at the organization. (%)

It was important to the design solution that the incentive structure take into account the variations between the teams. If users feel that they have no ability to obtain any incentive, they are not going to feel inclined to participate. In addition, I had to ensure that I met with the leads of every department, so all of the executives and management also understood the importance of this initiative and provided buy-in, as well as informing them of how each specific department will benefit from participation.

Having met with various stakeholders of all departments, I needed to be sure that the solution accounted for all levels of experience. We had users within our organization who had never touched open source, worked within the domain, nor had any experience conducting research on open source software. This meant that many of those same users could feel disincentivized to participate, as they may assume that engineers or product users would have much more rapid success with conducting this work.

Prototyping/Solutioning

With all of this new data collected, I began to design the experience. I knew that I needed to design the act of participation, as the driving force for incentives. With this in mind, I met with our CFO and coordinated the incentive structure and allowance I could use for this project. I requested $2,500, which later increased slightly to $2,550, to contribute to the contest. This included a breakdown of how the money would be spent, how each individual member could win incentives, and discussed all, if any, constraints to keep in mind, which our CFO agreed to. The only constraint noted, was that we specifically did not want to just throw blanket money at our stakeholders, rather we decided to use Bonusly, a software platform that is directly used to help teams improve engagement, to incentivize our users.

Catalog Crunch Contest Incentive layout, 2023

As I was iterating on the design of the contest, I requested the assistance of one of our Product managers to facilitate domain knowledge exercises and to ensure that we had a technical lead to assist, at any point, with open source data inquiries. With her input, we were able to finalize the contest structure and planned to hold the contest, over the duration of Summer 2023. The reason for this time frame, was to allow for the incentive dispersement to be calculated and distributed on a weekly cadence, while taking into account that this is a project, not a permanent new structure within our organization. In addition, this provided a window of time that our metrics could be measured against, and naturally proved to be a more idle time period, as summer often contains vacations, time off, and noticeable reduction is turnaround time with other organizations.

I announced the contest at a company-wide all hands meeting, and provided specific clear instructions on how the contest would be conducted. I took the time to answer any questions, ensure folks that all participants had equal opportunity to win incentives, and allow for anyone to request specific accommodations prior to the contest, if needed.

Users were asked to record their sessions as often as they were conducting them, via google forms, so that I could track the progress and ensure that I was accurately tallying results for the weekly prizes being awarded to users. Even if a user were only doing data research for 15 minutes, that could still be recorded, and for every 15 minutes, a point would be awarded to the individual, which would be tallied, and at the end of every week the highest number of points would be the winner of the weekly prize. Every week, the score would reset for the weekly prizes, however, because the end of the month was always the largest amount of incentive, that incentive was awarded to the user with the most recorded sessions for the entire month! With this structure, users had equal opportunity to win incentives weekly, and also work towards sessions for a monthly prize, if the weekly incentives were not attainable.

This Google form was designed to ensure we were tracking:

The number of participants across departments

Number of working sessions by each individual for weekly and end-of-month incentives

How many data points were being filled and which specific data points they were

Total time spent conducting data research and data points completed within those time frames.

Some of the Catalog Crunch tracker form questions, 2023

During the two-week window between the announcement of the contest, and the kickoff, I ensured all preparation was complete and all questions were answered. Following that, the contest kicked off on June 5th and the contest concluded on August 25th!

Official contest poster, 2025

Last working session of the Catalog Crunch Contest, August 15th, 2023

Takeaways

The project absolutely produced results!! We were able to get over 35% of missing data points affecting our customers added to the product, and we were able to increase internal literacy on evaluating open source projects. We were able to maintain participation and continued the project from start to finish. I achieved the business goals, by confirming that we could create an incentive program to get folks to complete tasks and reduce our overall task count.

Over 300+ hours spent working on data research!

Between a mix of group working sessions, as well as individual reported sessions.

Over 13,000+ data gaps filled!

With two entire categories of data points being filled in! Reducing our categories of incomplete data from nine to seven!

Almost 200 reported working sessions over 3 months!

Between 21 participants company-wide, with members of almost every team, and managers from almost every department!

As successful as this project was, this project also confirmed that we need a much more robust plan to continue updating our data, and keeping it up to par with our users expectations. There are a large number of data points that still need to be completed, and using internal stakeholders to fill in missing data points, during their spare time, definitely reduces the number of data points, but not to the degree that is needed, nor does it compare to the number of missing data points that remain, as tens, sometimes even hundreds are added every day, due to the realities of open source software.

As of August 2023, Product leadership is now working through full time staffing plans to ensure a premium customer experience with our data.