- …
Excerpts from How to evaluate dependencies by :
25th February 2023
One of my stock interview questions goes: “When picking between dependencies to use in production, what factors contribute to your decision?” I’m surprised by how often I receive an answer along the lines of “Github stars” and not much else. I happen to think Github stars is a terrible metric for selecting production code, so this post sets out my idea of a healthier framework to evaluate dependencies.
0. Read the documentation
The starting point should always be RTFM. The docs will be your guide to using this thing, so make sure you understand them.
- Does it solve your problem?
- Do you know how to use it?
- Is there anything confusing about the instructions?
- Is there a changelog or release notes?
1. Read the code
After you’ve read the docs, the absolute most important thing you can do is read the code. This can be an intimidating prospect, especially if the functionality seems complicated or is outside your normal area of expertise. But even if you’re not able to fully grok the entirety of a codebase, there are still many things it’s possible to glean by looking at it.
- How clean is it? Is it broken down into functions and/or files in a way that makes sense to you?
- Does what you see in the code match what you understood from the docs? Does it adhere to the principle of least astonishment?
- Does it contain other dependencies? Are they up to date? What do those other dependencies look like? How deep does the rabbit hole go?
- Does it do just the thing that you need, or does it also cater for a bunch of other concerns that don’t match your usage? Do those other concerns impact performance or usability?
- Does it receive untrusted input, if so does it prevent that input from being abused?
- Can you estimate a ballpark big-O for the implementation? Look for nested loops or inappropriate datatypes that could cause quadratic performance.
- Are there tests? How many? Are the tests clean? Can you run them? Do they all pass? How long do they take?
- Try to imagine a situation where you have to fork the code and implement something yourself. Does that feel comfortable or scary?
2. Look at project activity
After looking at the docs and code, try to get a sense of activity on the project.
- How many unresolved issues are there? How long have they been open? Are any of the unresolved issues concerning?
- How many resolved issues are there? How quickly were they closed? How helpful were the maintainers in addressing any issues? Were any issues closed in disagreement with the reporters?
- Look at the commit history. Are the original maintainers still involved? If the current maintainers aren’t the original ones, how long have they worked on it? Are they working on the guts of the code, or just touching the edges?
- Look at the release history. How often do new releases come out? How often are there breaking API changes? Is the cadence something you’d be comfortable keeping up with?
3. Look at project stats
Next try to get a feel for the community around the project.
- How many downstream dependents are there?
- How many downloads per day/week?
- Are other people successfully using it in prod?
If you really want to consider Github stars at this point, do so but understand that starring a project on Github does not signify any specific intention. Some people may use it like an upvote but for others it’s just a reminder system. Mostly Github stars seem to correlate with how aggressively a project has been promoted on social media.
4. Make a decision
Fundamentally, using a dependency is not a one-off decision that you can subsequently forget about. Dependencies are more like a liability or a tax, and a prerequisite to your decision is working out the long-term cost. You must do the research so that cost can be calculated, then weigh it up against the alternatives (and one of those alternatives might be implementing it yourself).