-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Extend Azure DevOps project creation
Today, DevOps projects experience in Ibiza portal allows an user to create CI/CD pipeline for a variety of programming languages, frameworks and azure services. It uses a sample code based on user's selection and then creates a new Azure DevOps project and bootstraps that project with a CI/CD pipeline. This CI/CD pipeline is capable of building the sample code and deploying it to an Azure service selected by the user. Users can even choose to use their custom application code. We have also built extensibility model for DevOps project which allows contributions to enhance the create experience.
This extensibility can also be useful for a admin/IT/consultant persona. Typically, such a persona would want to use a boilerplate application code and CD/CD piepline template for on-boarding new teams or clients. Additionally, they would most probably be using ARM template to create DevOps project instead of going via the Azure portal. This documents shows how this persona can extend the Azure DevOps create experience and use it as per their custom requirement.
There are two broad steps in order to achieve this extensibility:
- Author a
Pipeline Templateextension - Install this extension in team/client's Azure DevOps organization and create a new project using ARM template
Pipeline template extension can be authored as a Azure DevOps extension.
The extension should specify following:
- How to create a CI/CD pipeline for this scenario? This information will be used to import source repository, create build definition and release definition
- What are the parameters required to create this pipeline? These can be used to parameterize the pipeline creation
More details about it is in next section.
One Azure DevOps extension can provide multiple contributions and each contribution maps to one DevOps project scenario.
Note: It is recommended that an author(or a team) creates one extension and add all related contributions to that extension.
A pipeline template has following properties:
- id: a unique identifier for this contribution. Should depict the scenario this contribution is lighting up.
- type: should be set to
ms.vss-continuous-delivery.pipeline-template-type - targets: should contain
ms.vss-continuous-delivery.pipeline-templates - properties: this describes various properties of contribution. Should have following fields:
- description: Should describe the scenario which this contribution supports in a layman's terms
- attributes: A dictionarty which can be used to specify any attribute/tag for a pipeline template. One such attribute is
codeRepositoryTypewhich is used to specify whether this pipeline template is for Sampe code scenario for Bring-your-own-code (BYOC) scenario. It needs to be set to eitherSampleorBYOC. - parameters: This section contains inputs which are required to configure a CI/CD pipeline. More details in in parameters section.
- configuration: This section contains all details required to create a fully functional CI/CD pipeline which includes:
- sample source code repository which needs to be imported
- any
Assetssuch as service endpoints, personal access tokens, deployment group etc required to create a BD/RD. Although most of the assets will be required prior to creation of BD and RD, in some circumstances they can be needed after BD and RD is created. There is an option to do so. - a build definition template and required task parameter overrides
- a release definition template and required task parameter overrides More details in Configuration section.
Parameters section in a pipeline template has three types of details:
-
Inputs: this is an array of inputs required to configure CI/CD pipeline. .
- id: a unique identifier in the template scope
- type: data type of this input. Currently supported values are:
- string
- securestring
- int
- boolean
- authorization: this data type can be used for an input which represents an authorization information. One possible use of this type is to pass AAD Service Principal information. The value is serialized form of an object having two properties:
- scheme: authorization scheme to be used. For specifying SPN detail, this should be set to
ServicePrincipal. Please note that the specified service principal should have contributor permission on the Azure subscription and should have role based access on the resources/resource groups which it would deploy to. - parameters: A dictionary containing parameters specific to authorization scheme. For example, SPN would have following details:
- tenantid: tenant Id in AAD
- serviceprincipalid: id of the service principal
- serviceprincipalkey: secret for the service principal
- scheme: authorization scheme to be used. For specifying SPN detail, this should be set to
- description: A clear and concise description of what this input is for
Parameters can be specified inline or in a separate file. This file must be part of the extension VSIX. They can be referenced using imports keyword.
"parameters": {
"imports": [
"Files/Parameters/WindowsAppService.json"
]
}
Example of a complete parameter section:
{
"inputs": [
{
"id": "azureAuth",
"description": "Authorization for Azure ARM endpoints.",
"type": "string"
},
{
"id": "subscriptionId",
"description": "Id of subscription where azure resources will be created.",
"type": "string"
},
{
"id": "resourceGroup",
"description": "Name of resource group which should contain web app.",
"type": "string"
},
{
"id": "webAppName",
"description": "Name of web app to be created",
"type": "string"
},
{
"id": "location",
"description": "Location of the app service.",
"type": "string"
},
{
"id": "appServicePlan",
"description": "Details of cost and compute resource associated with the web app",
"type": "string"
},
{
"id": "appInsightLocation",
"description": "Application insights location.",
"type": "string"
}
]
}
Note: More such examples can be found by hitting the URL: https://peprodscussu2.portalext.visualstudio.com/_apis/arm/pipelinetemplates
Configuration section in a pipeline template has following details:
-
Assets: An array of assets. An asset is a pre-requisite which needs to be created before creating CI/CD pipeline. For example, a service endpoint might be needed to create release definition, or a self-signed certificate needs to be created for a service fabric cluster.
If you need a new asset type, please connect with product group team. These assets are created by VSTS before importing source repo and creating BD/RD. An asset has following properties:
- id: unique Id for the asset
- type: Type of the asset. Currently supported values are:
- endpoint:AzureRM [Creates Azure ARM service endpoint]
- PersonalAccessToken [Created Azure DevOps personal access token]
- TeamServicesAgentVMExtension [Installs Azure VM extension]
- KeyVault [Creates Azure Key Vault]
- KeyVaultCertificate [Creates a certificate in a Key Vault]
- DeploymentGroup [Creates a Deployment group in Azure DevOps]
- endpoint:ServiceFabric [Creates Service fabric service endpoint]
- REST:ARM [Makes an ARM call before/after creating pipeline]
- inputs: A dictionary of key-values which are required by the particular asset type. To see what to specify for each type, see here.
- stage: Specifies whether this asset will be created before(use
pre) or after pipeline is created (usepost) Exmaple:"assets": [ { "id": "ARMEndpoint" "type": "endpoint:AzureRM", "inputs": { "subscriptionId": "{{inputs.subscriptionId}}", "authorization": "{{inputs.azureAuth}}" } } ]
-
Source: Url of the sample code repository. Currently we only support GIT and this url can point to any public git repository. This repository will be imported to created Azure DevOps project. It has following properties:
- repository: details of source repo
- id: url of repo
- defaultBranch: branch for configuring the continuous integration. Note: Source should only be specified if source needs to be imported. It should not be specified for BYOC scenario.
- repository: details of source repo
-
BuildDefinition: Specifies how to create a Build Definition and what agent queue to use for running builds. It has following properties:
- template: A JSON string containing build definition template. You can get built-in template using REST API or can create a template yourself. See more here on how to get built-in templates.
- templateFile: The template is a very re-usable entity and multiple contributions in a VSTS extension can use same template. Hence, template JSON can be saved in a separate file in extension and referred here using this property. Specify only one of template or templateFile and prefer templateFile
- queue: This will override the queue to be used for executing build. This should be same as one of the display values in the queue drop-down when creating build definition in VSTS UX.
- taskInputs: A array of overrides which will override inputs for tasks in the definition. Each item is a dictionary of key-value pairs and will map to one task in that order. Each item must contains a key called
__TaskId__which should match exactly with task's id. Other keys should be same as id of the task input which needs to be overridden. For example, below task inputs will override tasks with ids 94a74903-f93f-4075-884f-dc11f34058b4 and E28912F1-0114-4464-802A-A3A35437FD16. And for task 94a74903-f93f-4075-884f-dc11f34058b4, its inputConnectedServiceNamewill be updated with value{{{assets.ARMResourceGroup}}}"taskInputs": [ { "__TaskId__": "94a74903-f93f-4075-884f-dc11f34058b4", "ConnectedServiceName": "{{{assets.ARMResourceGroup}}}", "resourceGroupName": "{{{inputs.resourceGroup}}}", "location": "{{{inputs.location}}}", "csmFile": "$(System.DefaultWorkingDirectory)/**/containerRegistry-template.json", "overrideParameters": "-registryName \"{{{inputs.containerRegistryName}}}\" -registryLocation \"{{{inputs.containerRegistryLocation}}}\" -registrySku \"{{{inputs.containerRegistrySKU}}}\"" }, { "__TaskId__": "E28912F1-0114-4464-802A-A3A35437FD16", "containerregistrytype": "Azure Container Registry", "azureSubscriptionEndpoint": "{{{assets.ARMResourceGroup}}}", "azureContainerRegistry": "{{{inputs.containerRegistryName}}}.azurecr.io", "command": "Build an image", "dockerFile": "**/Dockerfile" }
-
ReleaseDefinition: Specifies how to create a multi-environment, multi-phase releases definition. Also specifies agent queue/deployment group for each phase. It has following properties:
- environments: An array of configurations required to create an environment inside RD. In each environment you can specify following:
-
name: Display name of the environment
-
deployedResourceIds: An array of resource URIs which this pipeline is expected to create. This information is used to show a section of azure resources in DevOps project dashboard. Note, that these resources might not be existing before release runs, but you are still expected to list down the expected resource URIs. The array should contains all the resources which will need to be shown in the Dashboard. First item in array should always be resource group where resources will be created. Second items should be the primary resource which is being createdd. All other items should be listed in the order in which they need to be shown in Dashboard UI.
This information is also used to decide which resources need to be deleted when DevOps project is deleted.
-
template: A JSON string containing release definition template. You can get built-in templates using REST API or can create a template yourself.
-
templateFile: Similar to build definition, you should prefer using a templateFile for specifying RD template.
-
phaseInputs: Overrides the values specified in release definition templates. This is an array and one item maps to one phase in that order. Each phase can override following properties:
- queue: This will override the queue to be used for executing that phase in release. This should be same as one of the display values in the queue drop-down when creating release definition in VSTS UX.
- taskInputs: A array of overrides which will override inputs for tasks in the phase. Each item is a dictionary of key-value pairs and will map to one task in that order. Each item must contains a key called
__TaskId__which should match exactly with task's id. Other keys should be same as id of the task input which needs to be overridden. For example, below task inputs will override tasks with ids 94a74903-f93f-4075-884f-dc11f34058b4 and 497d490f-eea7-4f2b-ab94-48d9c1acdcb1. And for task 94a74903-f93f-4075-884f-dc11f34058b4, its inputConnectedServiceNamewill be updated with value{{{assets.ARMResourceGroup}}}"taskInputs": [ { "__TaskId__": "94a74903-f93f-4075-884f-dc11f34058b4", "ConnectedServiceName": "{{{assets.ARMResourceGroup}}}", "ResourceGroupName": "{{{inputs.resourceGroup}}}", "location": "{{{inputs.location}}}", "csmFile": "$(System.DefaultWorkingDirectory)/**/windows-webapp-template.json", "overrideParameters": "-webAppName {{{inputs.webAppName}}} -hostingPlanName {{{inputs.webAppName}}}-plan -appInsightsLocation \"{{{inputs.appInsightLocation}}}\" -sku \"{{{inputs.appServicePlan}}}\"" }, { "__TaskId__": "497d490f-eea7-4f2b-ab94-48d9c1acdcb1", "ConnectedServiceName": "{{{assets.ARMResourceGroup}}}", "WebAppName": "{{{inputs.webAppName}}}", "ResourceGroupName": "{{{inputs.resourceGroup}}}", "Package": "$(System.DefaultWorkingDirectory)\\**\\*.zip" } ]
-
- environments: An array of configurations required to create an environment inside RD. In each environment you can specify following:
The configuration section can be parameterized with the parameters defined in parameters section in pipeline template. To do so, a mustache expression containing reference to input parameter should be used. The format is {{{inputs.<input-id>}}}. This can be used for any of the fields in configuration section i.e. asset, or build definition or release definition. For example, in the below example, ResourceGroupName task input is set to input resourceGroup's value.
Just as was the case with parameters, same configuration can be used across multiple pipeline template. Hence, to introduce re-usability configurations can also be imported in a pipeline template.
"configuration": {
"imports": [
"Files/Configuration/ArmEndpointConfiguration.json",
"Files/Configuration/WindowsWebAppReleaseConfiguration.json"
]
}
Any asset specified in imported configuration is merged with assets directly specified in the contribution. On the other hand, Source, Build definition and Release definition properties will be over-written.
An example of a shared configuration could be:
{
"releaseDefinition": {
"environments": [
{
"name": "dev",
"deployedResourceIds": [
"/subscriptions/{{{inputs.subscriptionId}}}/resourceGroups/{{{inputs.resourceGroup}}}/providers/Microsoft.Web/sites/{{{inputs.webAppName}}}"
],
"templateFile": "Files/ReleaseDefinitionTemplates/deployAzureAppServiceWithVsTest.json",
"phaseInputs": [
{
"queue": "HostedVS2017",
"taskInputs": [
{
"__TaskId__": "94a74903-f93f-4075-884f-dc11f34058b4",
"ConnectedServiceName": "{{{assets.ARMEndpoint}}}",
"ResourceGroupName": "{{{inputs.resourceGroup}}}",
"location": "{{{inputs.location}}}",
"csmFile": "$(System.DefaultWorkingDirectory)/**/windows-webapp-template.json",
"overrideParameters": "-webAppName {{{inputs.webAppName}}} -hostingPlanName {{{inputs.webAppName}}}-plan -appInsightsLocation \"{{{inputs.appInsightLocation}}}\" -sku \"{{{inputs.appServicePlan}}}\""
},
{
"__TaskId__": "497d490f-eea7-4f2b-ab94-48d9c1acdcb1",
"ConnectedServiceName": "{{{assets.ARMEndpoint}}}",
"WebAppName": "{{{inputs.webAppName}}}",
"ResourceGroupName": "{{{inputs.resourceGroup}}}",
"Package": "$(System.DefaultWorkingDirectory)\\**\\*.zip"
},
{
"__TaskId__": "2c65196a-54fd-4a02-9be8-d9d1837b7111"
},
{
"__TaskId__": "ef087383-ee5e-42c7-9a53-ab56c98420f9"
}
]
}
]
}
]
}
}
Once the extension is authored, it can be published via Azure DevOps marketplace. Create a publisher here and publish the extension. Initially, during the testing phase, the extension can be published as a private extension. Make sure that you haven't put public flag in your extension json.
- To create an extension package, use tfx cli.
Once published, share the extension with a test Azure DevOps Organization by right clcking on extension and choosing "Share"
Navigate to the test Azure DevOps organization to which the extension was shared. Install this extension by going to a url similar to https://<account-name>.visualstudio.com/_admin/_extensions?tab=manage&status=active
Now, we can use this extension to create a new project in the above Azure DevOps organization. For this, we will do an ARM template deployment. Create a ARM template by copying this content:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/Microsoft.Resources.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"projectSettings": {
"type": "Object"
},
"azureAuth": {
"type": "SecureString"
}
},
"variables": {},
"resources": [
{
"type": "microsoft.visualstudio/account",
"name": "[parameters('projectSettings').accountName]",
"apiVersion": "2014-02-26",
"location": "[parameters('projectSettings').accountLocation]",
"properties": {
"operationType": "[parameters('projectSettings').accountOperationType]",
"accountName": "[parameters('projectSettings').accountName]",
"ownerUpn": "[parameters('projectSettings').ownerUpn]"
},
"resources": [
{
"type": "project",
"name": "[parameters('projectSettings').projectName]",
"apiVersion": "2014-02-26",
"location": "[parameters('projectSettings').accountLocation]",
"properties": {
"ProcessTemplateId": "[parameters('projectSettings').processTemplateId]",
"VersionControlOption": "[parameters('projectSettings').versionControlOption]",
"ownerUpn": "[parameters('projectSettings').ownerUpn]",
"PipelineBootstrapConfigurations": [
{
"name": "[parameters('projectSettings').projectName]",
"templateId": "<Replace-with-id-of-the-installed-contribution>",
"templateParameters": {
"parameter1": "parameter1 value",
"azureAuthParameter": "[parameters('azureAuth')]"
},
"repository": null
}
]
},
"dependsOn": [
"[concat('microsoft.visualstudio/account/', parameters('projectSettings').accountName)]"
]
}
]
}
]
}
Similarly, create a template parameters file by copying following content:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"projectSettings": {
"value": {
"accountName": "<organization-name-on-which-extension-was-installed>",
"accountLocation": "<location-for-org>",
"accountOperationType": "Link",
"ownerUpn": "",
"projectName": "new project name",
"processTemplateId": "adcc42ab-9882-485e-a3ed-7678f01f66bc",
"versionControlOption": "Git",
"requestSource": "DevOpsExtensionThirdParty"
}
},
"azureAuth": {
"value": "{\"scheme\":\"ServicePrincipal\",\"parameters\":{\"tenantid\":\"XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX\",\"objectid\":\"XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX\",\"serviceprincipalid\":\"XXXXXX-XXXX-XXXX-XXXX-XXXXXXXXX\",\"serviceprincipalkey\":\"password\"}}"
}
}
}
Replace following fields in the ARM template:
- Replace
templateIdvalue specified in project resource properties with the value of the pipeline template contribution's id - Replace
templateParametersvalue with the dictionary of the parameters and its values. These parameters should be same as defined in the contribution
Trigger a ARM deployment on the above template and parameter, and it would create a new project in the Azure DevOps organization with a CI/CD pipeline set as per the configuratin defined in the pipeline template.
To create a DevOps project when instead of a Sample application code end user will specify her custom code, source repo information should also be part of ARM template. Add a property called repository should be added to PipelineBootstrapConfigurations.
The payload will something similar to:
"repository": {
"repositoryType": "Git",
"id": "<link-to-a-git-repo>",
"defaultBranch": "master",
"authorization": <authorization-for-a-private-repo>
}
Authorization data has:
- scheme: authorization scheme to be used
- parameters: A dictionary containing parameters specific to authorization scheme
A sample pipeline template extension can found here. This extension will create a CI/CD pipeline which will create a Windows WebApp and deploy an application on this web app.
Once this extension has been published and shared with a Azure DevOps organization, an ARM template deployment can be created to use this pipeline template to create a project in this organization. A sample ARM template can be found here.
- endpoint:AzureRM - This will create Azure RM service endpoint type
"type": "endpoint:AzureRM",
"inputs": {
"subscriptionId": "<subscription-id>",
"authorization": "<Azure-auth-data>"
}
Note: if there is a SPN authoriaztion parameter name azureAuth, it can be specified here as {{inputs.azureAuth}}
- KeyVault - this will create Azure Key vault
"type": "KeyVault",
"inputs": {
"keyVaultName": "<name of KV>",
"location": "<lcoation of KV>",
"subscriptionId": "<subscription id>",
"resourceGroupName": "<resource group name>",
"serviceEndpoint": "<ARM service endpoint>"
}
Note: It is possible to refer another asset by using {{assets.<asset-id>}} mustache expression. For example, {{assets.ARMEndpoint_object}} can be specified to refer to ARM service endpoint created using asset with id ARMEndpoint. Adding suffix _object means full object, otherwise only id is used
- KeyVaultCertificate - this will create a certificate in a key vault
"type": "KeyVaultCertificate",
"inputs": {
"keyVaultCertName": "<cert name>",
"keyVaultName": "<name of KV in which to create cert>",
"certSubject": "<subject for cert>",
"serviceEndpoint": "<ARM service endpoint>"
}
- DeploymentGroup - this will create a Deployment group in Azure DevOps project
"type": "DeploymentGroup",
"inputs": {
"deploymentGroupName": "<name>"
}
- PersonalAccessToken - this will create a Personal Access token for user in Azure DevOps
"type": "PersonalAccessToken",
"inputs": {
"scope": "<scopes seprated by space>",
"validityInMinutes": "<validity in minutes>",
"tokenName": "<name of the token>"
}
- TeamServicesAgentVMExtension - this will install a
Team services agentVM extension on a VM
"type": "TeamServicesAgentVMExtension",
"inputs": {
"vmUri": "<uri of the virtual machine on which to install the VM extension",
"deploymentGroupName": "<name of DG against which the Azure DevOps agent should be configured>",
"location": "<VM location>",
"tags": "<Tags to add on DG>",
"token": "<Personal acccess token>",
"osType": "Windows",
"vmName": "<VM name>",
"tenantId": "<AAD tenant id>",
"accessToken": "<Azure access token>"
}
Note: PAT created in another asset can be referenced as {{assets.PatToken}} where PatToken is id of that asset.
- endpoint:ExternalTfs - this will create a
external tfstype service endpoint
"type": "endpoint:ExternalTfs",
"inputs": {
"token": "<Personal acccess token>"
}
- endpoint:ServiceFabric - this will create a
service fabrictype service endpoint
"type": "endpoint:ServiceFabric",
"inputs": {
"name": "<name of endpoint>",
"location": "<location of service fabric cluster>",
"serverCertificateThumbprint": "<thumbprint of service certificate>",
"clientCertificate": "<client certificate>"
}
Note: For thumbprint, Certificate created in another asset can be referenced as {{assets.KeyVaultServerCertificate_thumbprint}} where KeyVaultServerCertificate is id of the certificate asset.
For client certificate, Certificate created in another asset can be referenced as {{assets.KeyVaultClientCertificate_certificate}} where KeyVaultServerCertificate is id of the certificate asset.
- REST:ARM - this will make a REST call to ARM
"type": "REST:ARM",
"inputs": {
"url": "<ARM uri(without host) with API version>",
"body": "<stringified body payload>",
"method": "<REST method to use>",
"endpoint": "<ARM service endpoint>",
"authentication": "<Azure access token>" // use either endpoint or authentication field
}