Merge branch 'main' into feat-bitbucket-server-support

docs/Configuration/AdvancedMode.md

@@ -383,6 +383,18 @@ To edit a Blueprint created in the Advanced mode, you can simply go the Configur
 
 ![img](/img/ConfigUI/BlueprintEditing/blueprint-edit2.png)
 
+## How to skip collectors in a Blueprint (Advanced Mode)?
+- 1. Create a Blueprint in the Advanced Mode.
+- 2. You can skip collectors in a Blueprint by setting `skipCollectors` to `true` in the request body of the trigger API. For example:
+```
+  curl --request POST \
+    --url http://localhost:8080/blueprints/:blueprintId/trigger \
+    --header 'content-type: application/json' \
+    --data '{
+      "skipCollectors": true
+  }'
+```
+
 ## Troubleshooting
 
 If you run into any problem, please check the [Troubleshooting](/Troubleshooting/Configuration.md) or [create an issue](https://github.com/apache/incubator-devlake/issues)

docs/Configuration/AzureDevOps.md

@@ -46,6 +46,8 @@ Azure DevOps supports the following data entities.
 
 ### Step 1.3 - Add Scope Config (Optional)
 
+![azuredevops-add-scope config](images/azuredevops-add-scope-config.png)
+
 Scope config contains two parts: 
 - The entities of which domain you wish to collect: Usually, you don't have to modify this part. However, if you don't want to collect certain Azure DevOps entities, you can unselect some entities to accelerate the collection speed.
   - Source Code Management: Azure repos, refs, commits, etc.
@@ -54,12 +56,20 @@ Scope config contains two parts:
   - Cross Domain: Azure accounts, etc.
 - The transformations on the Azure DevOps data you are going to collect. 
 
+![azuredevops-set-transformation](images/azuredevops-set-transformation.png)
+
+#### CI/CD
 
-The transformations are mainly used for calculating [DORA metrics](../DORA.md), so DevLake needs to know what are `deployments` in your Azure Pipelines. You can configure:
-    - Regex for `Deployments`: Azure DevOps pipeline or one of its jobs whose names match this regex will be registered as deployments in DevLake
-    - Regex for `Production` environment: Azure DevOps pipeline or one of its jobs whose names match this regex will be considered as a PRODUCTION deployment.
+To effectively measure [DORA metrics](../DORA.md) through Azure DevOps, it is necessary to define the concept of a 'deployment'. DevLake considers an Azure Pipeline Run (see the blue rectangle) as a DevLake deployment using specific conditions expressed through regular expressions (regex):
 
-![azuredevops-set-transformation](images/azuredevops-set-transformation.png)
+- Deployment: The provided regex should match either the name of the Azure pipeline (see the red rectangle) that the pipeline run belongs to or any of the job display names (see the yellow rectangle) associated with the pipeline run. This will designate it as a deployment. For example, if the deployment pipeline is named 'build-and-push-image', you can input (push-image) as the regex. To ensure case insensitivity, include (?i) before the regex.
+- Production: The given regex should match either the pipeline run's name or any of its job display names to classify it as a deployment within the production environment. For instance, if the deployment pipeline is named 'build-to-prod', you can input (prod) as the regex. To ensure case insensitivity, include (?i) before the regex.
+
+![azure-pipeline](images/azuredevops-ui-pipeline.png)
+![azure-job](images/azuredevops-ui-job.png)
+
+
+#### Additional Settings
 
 The additional settings for transformations are RefDiff options:
 - Tags Limit: the number of tags to compare.

docs/Configuration/CircleCI.md

@@ -0,0 +1,86 @@
+---
+title: "CircleCI"
+sidebar_position: 8
+description: Config UI instruction for CircleCI
+---
+
+Visit Config UI at : `http://localhost:4000` and go to `Connections` page.
+
+## Step 1 - Add Data Connections
+
+### Step 1.1 - Authentication
+
+![image](/img/ConfigUI/circleci-add-data-connections.png)
+
+#### Connection Name
+
+Give your connection a unique name to help you identify it in the future.
+
+#### Endpoint URL
+
+For CircleCI, you do not need to enter the REST API endpoint URL, which is always `https://circleci.com/api/`.
+
+
+#### Token
+
+Learn about [Managing API Tokens](https://circleci.com/docs/managing-api-tokens/).
+
+Tokens you have generated that can be used to access the CircleCI API. Apps using these tokens can act as you and have full read- and write-permissions!
+There are two types of API token(Personal and Project) you can create within CircleCI.
+
+
+#### Proxy URL (Optional)
+
+If you are behind a corporate firewall or VPN you may need to utilize a proxy server. Enter a valid proxy server address on your network, e.g. `http://your-proxy-server.com:1080`
+
+
+#### Fixed Rate Limit (Optional)
+
+DevLake uses a dynamic rate limit to collect CircleCI data. You can adjust the rate limit if you want to increase or lower the speed.
+Learn more about [CircleCI API rate limit](https://circleci.com/docs/api-developers-guide/#rate-limits).
+
+
+#### Test and Save Connection
+
+Click `Test Connection`, if the connection is successful, click `Save Connection` to add the connection.
+
+### Step 1.2 - Configure Data Scope
+
+![image](/img/ConfigUI/circleci-choose-data-scope.png)
+
+#### Projects
+
+Select the CircleCI projects to collect.
+
+### Step 1.3 - Adding Scope Config (Optional)
+![image](/img/ConfigUI/circleci-scope-config.png)
+You can add a `transformation` to standardize the data. A `transformation` acts on the CircleCI data in the [tool layer](/docs/DataModels/ToolLayerSchema.md), transforming it to the [domain layer](/docs/DataModels/DevLakeDomainLayerSchema.md).
+
+## Step 2 - Collect Data in a Project
+### Step 2.1 - Create a Project
+Collecting CircleCI data requires creating a project first. You can visit the Project page from the side menu and create a new project by following the instructions on the user interface.
+
+![create-a-project](images/create-a-project.png)
+
+### Step 2.2 - Add a CircleCI Connection
+You can add a previously configured CircleCI connection to the project and select the boards for which you wish to collect the data for.
+Please note: if you don't see the repositories you are looking for, please check if you have added them to the connection first.
+
+![add-a-connection](images/add-a-connection-project.png)
+
+### Step 2.3 - Set the Sync Policy
+There are three settings for Sync Policy:
+- Data Time Range: You can select the time range of the data you wish to collect. The default is set to the past six months.
+- Sync Frequency: You can choose how often you would like to sync your data in this step by selecting a sync frequency option or enter a cron code to specify your preferred schedule.
+- Skip Failed Tasks: sometime a few tasks may fail in a long pipeline; you can choose to skip them to avoid spending more time in running the pipeline all over again.
+
+![sync-policy](images/sync-policy.png)
+
+### Step 2.4 - Start Data Collection
+Click on "Collect Data" to start collecting data for the whole project. You can check the status in the Status tab on the same page.
+![collect-data](images/collect-data.png)
+
+
+## Troubleshooting
+
+If you run into any problem, please check the [Troubleshooting](/Troubleshooting/Configuration.md) or [create an issue](https://github.com/apache/incubator-devlake/issues)

docs/Configuration/GitHub.md

@@ -117,7 +117,7 @@ Scope config contains two parts:
   - Cross Domain: GitHub accounts, etc.
 - The transformations on the GitHub data you are going to collect.
   - The details of the transformations will be explained below.
-  - Without adding transformation rules, you can still view the "[GitHub Metrics](/livedemo/DataSources/GitHub)" dashboard. However, if you want to view "[Weekly Bug Retro](/livedemo/QAEngineers/WeeklyBugRetro)", "[Weekly Community Retro](/livedemo/OSSMaintainers/WeeklyCommunityRetro)" or other pre-built dashboards, the following transformation rules, especially "Type/Bug", should be added.
+  - Without adding transformation rules, you can still view the "[GitHub Metrics](/livedemo/DataSources/GitHub)" dashboard. However, if you want to view "[Weekly Bug Retro](/livedemo/EngineeringLeads/WeeklyBugRetro)", "[Weekly Community Retro](/livedemo/OSSMaintainers/WeeklyCommunityRetro)" or other pre-built dashboards, the following transformation rules, especially "Type/Bug", should be added.
   - Each GitHub repo has at most ONE set of transformations.
 
 ![github-add-transformation-rules-list](images/github-scope-config.png)

docs/Configuration/HowToOrganizeDevlakeProjects.md

@@ -55,9 +55,9 @@ In situations where a repo or board is shared by multiple projects in real life,
 
 To clarify the concepts, let's define three terms:
 
-- [Project](/docs/Overview/KeyConcepts.md#project): Refers to a real-world project or product line, such as Apache DevLake or Apache Spark. It focuses on the work to be done.
-- Team: Represents a department, such as the 'product team' or 'engineering team'. It focuses on the people and their roles. Note that people within the same team may not always work on the same projects.
-- Project Team: Comprises individuals working on a specific project.
+- [`Project`](/docs/Overview/KeyConcepts.md#project): Refers to a real-world project or product line, such as Apache DevLake or Apache Spark. It focuses on the work to be done.
+- `Team`: Represents a department, such as the 'product team' or 'engineering team'. It focuses on the people and their roles. Note that people within the same team may not always work on the same projects.
+- `Project Team`: Comprises individuals working on a specific project.
 
 DevLake does support measuring DORA metrics at the project-team level, which is essentially the same as measuring at the project level. However, it is important to note that DevLake does not recommend measuring DORA metrics at the team level. Despite the existence of the 'DORA by team' dashboard contributed by the community. Doing so may introduce inaccuracies and dilute the significance of measuring DORA metrics from the outset.
 

docs/Configuration/Opsgenie.md

@@ -0,0 +1,83 @@
+---
+title: "Opsgenie"
+sidebar_position: 11
+description: Config UI instruction for Opsgenie
+---
+
+Visit Config UI at: `http://localhost:4000`.
+
+## Step 1 - Add Data Connections
+
+On the Connections page, you can select Opsgenie and create a new connection on it.
+
+### Step 1.1 - Authentication
+![opsgenie-add-data-connections](images/opsgenie-create-a-connection.png)
+
+#### Connection Name
+
+Give your connection a unique name to help you identify it in the future.
+
+#### Endpoint URL
+
+Opsgenie makes available two types of REST API endpoints, US and EU, eg. `https://api.opsgenie.com/` or `https://api.eu.opsgenie.com/`. You can choose wich instance by selecting the due radio input.
+
+#### API Access Key
+
+In the `API key management` section of your Atlassian Opsgenie account settings, you can create a API key to manage your requests.
+
+Learn about [how to create a Opsgenie API key](https://support.atlassian.com/opsgenie/docs/api-key-management/). The following permissions are required to collect data from repositories:
+
+- `Read`
+- `Create and Update`
+- `Configuration Access`
+
+#### Proxy URL (Optional)
+
+If you are behind a corporate firewall or VPN you may need to utilize a proxy server. Enter a valid proxy server address on your network, e.g. `http://your-proxy-server.com:1080`
+
+#### Fixed Rate Limit (Optional)
+
+DevLake uses a dynamic rate limit to collect Opsgenie data. You can adjust the rate limit if you want to increase or lower the speed.
+
+Opsgenie doesn't establishes a maximum rate limit for its API request, thus you should check this **[detailed doc](https://docs.opsgenie.com/docs/api-rate-limiting)** on how to calculate your rate limit, based on number of user and account plan that you hired. For now, the default rate limit is in 6,000 request/hour (100 request/minute).
+
+#### Test and Save Connection
+
+Click `Test Connection`, if the connection is successful, click `Save Connection` to add the connection.
+
+### Step 1.2 - Add Data Scopes
+
+#### Services
+
+Choose the Opsgenie services you wish to collect either by finding them in the miller column, or searching.
+
+![opsgenie-set-data-scope](images/opsgenie-add-data-scopes.png)
+
+## Step 2 - Collect Data in a Project
+### Step 2.1 - Create a Project
+Collecing Opsgenie data requires creating a project first. You can visit the Project page from the side menu and create a new project by following the instructions on the user interface.
+
+![create-a-project](images/create-a-project.png)
+
+### Step 2.2 - Add a Opsgenie Connection
+You can add a previously configured Opsgenie connection to the project and select the boards for which you wish to collect the data for.
+Please note: if you don't see the services you are looking for, please check if you have added them to the connection first.
+
+![add-a-connection](images/add-a-connection-project.png)
+
+### Step 2.3 - Set the Sync Policy
+There are three settings for Sync Policy:
+- Data Time Range: You can select the time range of the data you wish to collect. The default is set to the past six months.
+- Sync Frequency: You can choose how often you would like to sync your data in this step by selecting a sync frequency option or enter a cron code to specify your prefered schedule.
+- Skip Failed Tasks: sometime a few tasks may fail in a long pipeline; you can choose to skip them to avoid spending more time in running the pipeline all over again.
+
+![sync-policy](images/sync-policy.png)
+
+### Step 2.4 - Start Data Collection
+Click on "Collect Data" to start collecting data for the whole project. You can check the status in the Status tab on the same page.
+![collect-data](images/collect-data.png)
+
+
+## Troubleshooting
+
+If you run into any problem, please check the [Troubleshooting](/Troubleshooting/Configuration.md) or [create an issue](https://github.com/apache/incubator-devlake/issues)