Include licensing information for transitive dependencies

[sersut]

This PR replaces #703 without including the changes from ksubrama/gcc_investigate branch.

Description

This commit adds the ability to collect licensing information from the dependencies of software definitions that are not defined in omnibus. We use license_scout gem in order to collect licensing information from the dependency managers used by the software. Since many softwares are not using dependency managers we are also including some options to enable/disable collection of licensing information:

1. New software dsl method 'skip_transitive_dependency_licensing'
   When set omnibus will skip collecting licensing information for the dependencies. By default this is set to false.
2. Temporary configuration value 'fatal_transitive_dependency_licensing_warnings'
   When set omnibus will fail the build when there is a warning from transitive dependency license collection. We are introducing this as a feature flag while we are implementing support for new types of dependency managers in license scout. Once we implement license detection for most commonly used set of projects, we will remove this configuration option and fall back to 'fatal_licensing_warnings'.

This feature is compatible with git_cache. We drop in a manifest and a set of license files after the software is built and before it is cached. Therefore any software restored from git_cache will always have the correct licensing information.

Just like the other licensing information, the license files of the transitive dependencies will be placed under LICENSES directory and information about them will be included in the defined license file for the project (by default LICENSE).

Important Note

Note that we are forcing software definitions to declare skip_transitive_dependency_licensing true if they do not have any dependency managers. Even though this will create a lot of updates to a lot of the code, we believe this is the best approach to ensure we leak minimal amount of software without license information. See this for more information:

https://chefio.slack.com/files/dan/F1WK7KJ06/Omnibus___License_Scout_Integration_Options

Notes

Note that this PR is not yet ready to merge. Things to do before merge:

• Release license_scout gem.
• Make omnibus-software depend on the latest version of omnibus (because of the new DSL method).

---

/cc @chef/omnibus-maintainers

[danielsdeleo]

Might want to put the word "license" or "licensing" in here and the related methods. I thought this was a different cache when I was reading the code.

[danielsdeleo]

On second thought, never mind, it should be clear from the context. I just thought I was in a different file.

[sersut]

👍

[danielsdeleo]

👍

[sersut]

The CI issues seem to be due to sourceforge connectivity.

[yzl]

👍

[schisamo]

Overall I find negative DSL methods like skip_transitive_dependency_licensing confusing. Have y'all thought about making using transitive_dependency_licensing which has a default value of true? Then you would just update software definitions where you want this disabled with transitive_dependency_licensing false. It might make sense to also just make the default value in this case false and force software definition authors to opt-in for the transitive behavior.

[danielsdeleo]

We definitely want to have the behavior be to opt-out so that we minimize the chance that we accidentally miss something.

[danielsdeleo]

I'm 👍 on the commits since my last 👍 Are the test failures all issues with that source forge URL? Might want to fix that soon....