Extra metrics for assessing package maturity

[Bouke]

Extra metrics for assessing package maturity

There is this thing that keeps bothering me when I'm evaluating packages that I want to use for my next superduper application. Assessing project maturity. There are a lot of packages that once were supercool (e.g. PIL and django-page-cms), but have fallen into decay. Sometimes due to lack of interest by the community, or by the maintainer. Sometimes there is a popular fork which takes over (e.g. Pillow and django-cms), and at other times, users switch to an alternative.

Choosing a package is a risk to me. Once I choose one package over another, I have to live with that decision. It would be short-sighted to say that I could easily switch to another package. In order to get started with a package, I must invest a substantial amount of time into understanding how the package and how I should integrate it. When running into problems, I have to put in substantial amounts of time into fixing the problem: reading hastly-written documentation, tweak configuration, re-read, tweak again, hack package's code, submit pull request, additional work. So my first pick better be a good one!

What I'm proposing is to add additional metrics that help me assessing package maturity. Based on those metrics I can compare various packages and make an informed decision. From the top of my head, some of those metrics could include (but certainly not limited to):

• Repository Activity: How many commits have been made recently? If a package is stalled for a long period, it might indicate lack of interest from the community/maintainer.
  • Repository Contributor Diversity: Does the community show interest in this package? Large diversity might result in wider support and faster finding and resolving of issues.
  • Pull Requests: Does the package receive pull requests? From a large variety of contributors? Does the maintainer participate in the pull requests, do they get merged? In what time frame?
• Package Age: How long is this package been around? New packages might not have enough traction in the community, but old packages might suffer from code rot.
• Unit Test / Continuous Integration: Does the package contain unit tests? And if so, is some form of continuous integration used (like travis-ci)? If this is true, the package is more robust and contributor-proof.
  • Unit Test Code Coverage: If the package contains unit tests, how much code is covered by those tests? High code coverage shows the maintainer cares about the validity of the unit tests.
• Maintainer Track Record: Does the maintainer also known for any other (good) packages? Does the maintainer contribute in other packages?
• Maintainer Assurance: By how many people is the package being maintained? How many are allowed to publish new versions (can this be retrieved somehow?)? How many people can push to the package's repository?

[rubik]

Great analysis of the problem, although this seems to me not so simple to implement. Especially since the packages repository can be on Github, Bitbucket, &c., all of which have a different API. Integrate with so many providers might be tricky.

[Bouke]

Tricky but not impossible. First GitHub and Bitbucket, that will probably cover the majority of the packages released in the past few years. I'm willing to put some time into this, if it is something @dstufft would accept to crate.io.