Disclaimer: I am a community practictioner. I follow the practice of community management closely and apply what I learn to the work I do and the communities I build. I do not take credit for the creation of these ideas. I have tried to give the credit to the original sources as much as possible.
As luck would have it, I had the chance to follow last week’s FlylessWeekly with DevRel legend Phil Legetter who covered one of the most popular topics in DevRel,
Defining DevRel roles, by tackling another widely-discussed topic in DevRel,
How to measure the impact of community. Here are my notes from that conversation.
Community metrics are a popular topic in DevRel because they’re hard. They’re hard to define, hard to measure, and hard to get right. A dollar of revenue is roughly equal in value across companies, the value of a user (LTV) to a company is fairly easy to quantify for most businesses, but the value of an interaction with a community member can vary pretty widely across communities and even within a community. How does the impact of a person attending a meetup to hear you speak compare to the value of a merge request that quickly fixes a bug impacting your users?
Community metrics 101
The difficulty with defining community metrics sometimes, many times?, leads to a set of metrics that fail to capture the true impact of a community. For example, maybe you have evaluated the value of or event tracked GitHub stars for your project, Twitter followers, Meetup attendees, or blog posts published. All of those metrics are directionally helpful. If those numbers are growing, you are probably doing something right.
But these metrics don’t tell the whole story and as your community grows so does the need to invest in it.
Tell me your favorite community metric. Wrong answers only.— John Coghlan (@john_cogs) March 2, 2021
If I missed any of your (least) favorite metrics, make sure to reply to the tweet above and let me know.
Emerging community metrics for tech communities
In 2016 as a newcomer to community management and developer relations, the so-called “vanity metrics” (note: I don’t love this term as these metrics are often quite important for new communities) seemed to be the best we could do. In the last year, however, I’ve seen the tooling and availability of community metrics shift in an exciting new direction.
One cool new trend is the emergence of new ways to track community impact through a more wholistic lense. Tools like Orbit and Bitergia allow businesses and project maintainers to view “atomic” level activity (Kudos to Sean Goggin from CHAOSS who introduced me to this term) to build out a more complete view of the contributions from a developer or a community at large. Mary Thengvall’s DevRel Qualified leads is a new metric that looks to take an established business metric (XQLs) and adapt it to measure the impact of community and DevRel activities. I would be remiss not to mention CHAOSS, a Linux Foundation project that focuses on metrics to track community health.
While presenting at Flyless, I also learned about how some other teams are tracking their metrics. My notes include:
- Ryan MacLean tracks engagement with the content (how many people are raising their hand to ask me a question?) as a KPI
- Lorna Mitchell pointed to Phil Legetter’s AAARRRP framework as model of more advanced metrics
- Caroline Lewko and others pointed to Airtable, Marketo, Mixpanel, and Hubspot as tools to help track community impact
Each of these tools and metrics aim to quantify the impact of interactions with a community member, rather than count the number of interactions. We have taken similar steps at GitLab, including the creation of a new metric that we are using to track the impact of our community.
Community at GitLab
Before jumping into the metrics, I want to offer some background on GitLab and why we take community so seriously. Gitlab started in 2011 as an open source project and the company launched as a Show HN post from our CEO the following year. Our CEO remains an active member of the community on Hacker News. Our well-known transparency, “everyone can contribute” mission, and open source stewardship create many opportunities for our community to contribute to GitLab.
Our long standing community KPI: Wider community MRs per release
Our first and longest standing company-level community metric is Wider Community merged MRs per release. Essentially, this tracks the number of new contributions to our product from the community on a monthly cadence. These contributions are essential to GitLab’s dual flywheels strategy. This strategy defines the virtuous cycle of community contributions creating more features leading to more users and revenue which enbables us to invest in new features which attract more users and contributors extending the cycle.
The new kid on the block: MRARR
Our latest community-focused metric, though it is owned by our Engineering org, is MRARR. At GitLab, we affectionately call this the pirate metric. If that doesn’t make sense to you, try pronouncing it as a single word. :)
When thinking about next level community metrics, this is a favorite for me. MRARR tracks how frequently customers are contributing to GitLab by measuring the number of merge requests from customers multiplied by their Annual Run Rate. By contributing to GitLab, these customers are making the product better suited to their needs (increasing stickiness/retention) and likely better for other enterprises (improving product market fit). I love it and would love for other OS companies to consider adopting this metric.
As DevRel evolves as a practice and new tools and metrics emerge, I hope that we’ll see increasing standardization of metrics across communities. When I posed this idea to today’s FlylessWeekly attendees, the response was mixed. Some felt that the variability of communities necessitates a wide variety of metrics. While I trust their judgement, I remain optimistic that better metrics will lead to increased invest in community. If you’ve got ideas, please drop me a line on Twitter.
The other “top 3” topic is
Where does DevRel belong in an organization? and I’m excited to see who will volunteer to tackle that one.