Metrics for Innovation Fund's cohort companies
Photo by Edward Howell on Unsplash
As a global organization dedicated to improving the lives of children and women, UNICEF Innovation Fund has been investing in companies that are working towards innovative solutions to tackle some of the world's most pressing problems. To ensure that these investments are being utilized effectively, we have been using CHAOSS Project's tools to keep track of open source metrics of the companies we invest in.
Metrics are important because they allow us to measure the impact of our investments and make informed decisions based on data. Without metrics, it would be difficult to know whether a particular company or project is on track to meet its goals, or whether changes need to be made to improve its performance. By tracking metrics, we can
- identify trends,
- spot areas that need improvement, and
- make data-driven decisions
These can lead to better outcomes, measuring the health and sustainability of these projects and the level of engagement from the community.
Open source metrics are particularly important for UNICEF Innovation. That's the central component of what makes Innovation Fund unique:
- Equity Free funding
- Mentorship in different areas (Dev, Business strategy, Open Source, Impact evidence etc)
- Must be Open Source.
What do we track: Right now, we track
- Number of Commits
- Number of PRs
- Number of collaborators
While this is not-a-bad thing to focus on, I believe there are more ways we can Improve. While tracking simple numerical metrics such as number of commits, PRs, or stars can provide a snapshot of project activity, it doesn't paint the full picture of an open source project's health or potential. Such quantitative metrics, while easy to measure, may not capture the subtleties that determine the project's long-term viability and success. To enhance our understanding, I recommend including a few qualitative measures - CHAOSS Project's Starter Project Health Metrics Model. Idea is to explain companies the importance of these metrics, and take charge of improving these.
Time to First Response:Determine the amount of time between when an activity was opened (e.g. Issue or Change Request) and when it received the first response from a human. This would give us valuable insight into the responsiveness and engagement of project maintainers. This is not merely a measure of speed, but an indicator of an active and collaborative community. A short response time can encourage more interaction and foster a more vibrant and engaged contributor base.
Change Request Closure Ratio: Measure the ratio between the total number of open change requests during a time period versus the total number of change requests closed in that same period. This would help us assess the project's efficiency in handling new inputs. This ratio reflects how well the project processes and incorporates changes and whether they are able to keep up with incoming contributions. It isn't about sheer quantity of completed requests, but the ability to effectively handle and integrate these changes into the project.
Bus Factor: Determine the smallest number of people that make 50% of contributions. Understanding the smallest number of people that make 50% of contributions, offers a measure of the project's resilience. It goes beyond the number of contributors to evaluate the distribution of responsibilities and knowledge among them. A higher Bus Factor shows that the project is less dependent on a few individuals and is therefore more sustainable in the long run.
Release Frequency: Determine the frequency of project releases (including point releases with bug fixed. This provides a perspective on the project's vitality. Regular releases indicate a well-maintained project, suggesting an active development team that's responsive to the needs of its users and contributors. It's not just about the number of releases, but the consistent effort to improve and evolve the project.
I don't believe understanding a long term sustainability is an easy task, and it's definitely not easy without context. They become numbers to be gamed. By integrating these qualitative metrics into our impact evaluation process, we can better understand the open source projects we support, allowing us to identify the effectiveness of our investments, and in turn, the impact of the solutions we help bring to life.