-
Notifications
You must be signed in to change notification settings - Fork 140
Cleanup jenkins scripts #7648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Cleanup jenkins scripts #7648
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Copyright 2025-Present Couchbase, Inc. | ||
| # | ||
| # Use of this software is governed by the Business Source License included | ||
| # in the file licenses/BSL-Couchbase.txt. As of the Change Date specified | ||
| # in that file, in accordance with the Business Source License, use of this | ||
| # software will be governed by the Apache License, Version 2.0, included in | ||
| # the file licenses/APL2.txt. | ||
|
|
||
| name: shell | ||
| permissions: | ||
| contents: read | ||
|
|
||
| on: | ||
| push: | ||
| # Only run when we change a shell script | ||
| paths: | ||
| - "*.sh" | ||
| branches: | ||
| - "main" | ||
| - "release/*" | ||
| - "feature/*" | ||
| - "CBG*" | ||
| - "ci-*" | ||
| pull_request: | ||
| # Only run when we change a shell script | ||
| paths: | ||
| - "*.sh" | ||
| branches: | ||
| - "main" | ||
| - "release/*" | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: ${{ !contains(github.ref, 'release/')}} | ||
|
|
||
| jobs: | ||
| shellcheck: | ||
| name: Shellcheck | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v5 | ||
| - name: Run ShellCheck | ||
| uses: ludeeus/action-shellcheck@master | ||
| with: | ||
| ignore_paths: | | ||
| ./build.sh | ||
| ./bootstrap.sh | ||
| ./bench.sh | ||
| ./integration-test/service-test.sh | ||
| ./integration-test/service-install-tests.sh | ||
| ./rewrite-manifest.sh | ||
| ./snap-manifest.sh | ||
| ./set-version-stamp.sh | ||
| ./service/sync_gateway_service_install.sh | ||
| ./service/sync_gateway_service_uninstall.sh | ||
| ./service/sync_gateway_service_upgrade.sh | ||
| ./test-integration-init.sh | ||
| ./test.sh |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| #/bin/sh | ||
| #!/bin/bash | ||
|
|
||
| # Copyright 2024-Present Couchbase, Inc. | ||
| # | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,32 +10,42 @@ | |
| DEFAULT_PACKAGE_TIMEOUT="45m" | ||
|
|
||
| set -u | ||
| set -e # Abort on errors | ||
|
|
||
| if [ "${1:-}" == "-m" ]; then | ||
| echo "Running in automated master integration mode" | ||
| # Set automated setting parameters | ||
| SG_COMMIT="master" | ||
| TARGET_PACKAGE="..." | ||
| TARGET_TEST="ALL" | ||
| RUN_WALRUS="true" | ||
| USE_GO_MODULES="true" | ||
| DETECT_RACES="false" | ||
| SG_EDITION="EE" | ||
| XATTRS="true" | ||
| RUN_COUNT="1" | ||
| # CBS server settings | ||
| COUCHBASE_SERVER_PROTOCOL="couchbase" | ||
| COUCHBASE_SERVER_VERSION="enterprise-7.6.6" | ||
| SG_TEST_BUCKET_POOL_SIZE="3" | ||
| SG_TEST_BUCKET_POOL_DEBUG="true" | ||
| export SG_TEST_BUCKET_POOL_DEBUG="true" | ||
| GSI="true" | ||
| TLS_SKIP_VERIFY="false" | ||
| SG_CBCOLLECT_ALWAYS="false" | ||
| fi | ||
|
|
||
| set -e # Abort on errors | ||
| set -x # Output all executed shell commands | ||
| REQUIRED_VARS=( | ||
| COUCHBASE_SERVER_PROTOCOL | ||
| COUCHBASE_SERVER_VERSION | ||
| GSI | ||
| TARGET_TEST | ||
| TARGET_PACKAGE | ||
| TLS_SKIP_VERIFY | ||
| SG_EDITION | ||
| ) | ||
|
|
||
| for var in "${REQUIRED_VARS[@]}"; do | ||
| if [ -z "${!var:-}" ]; then | ||
| echo "${var} environment variable is required to be set." | ||
| exit 1 | ||
| fi | ||
| done | ||
| # Use Git SSH and define private repos | ||
| git config --global --replace-all url."[email protected]:".insteadOf "https://github.com/" | ||
| export GOPRIVATE=github.com/couchbaselabs/go-fleecedelta | ||
|
|
@@ -44,37 +54,21 @@ export GOPRIVATE=github.com/couchbaselabs/go-fleecedelta | |
| SG_COMMIT_HASH=$(git rev-parse HEAD) | ||
| echo "Sync Gateway git commit hash: $SG_COMMIT_HASH" | ||
|
|
||
| # Use Go modules (3.1 and above) or bootstrap for legacy Sync Gateway versions (3.0 and below) | ||
| if [ "${USE_GO_MODULES:-}" == "false" ]; then | ||
| mkdir -p sgw_int_testing # Make the directory if it does not exist | ||
| cp bootstrap.sh sgw_int_testing/bootstrap.sh | ||
| cd sgw_int_testing | ||
| chmod +x bootstrap.sh | ||
| ./bootstrap.sh -c ${SG_COMMIT} -e ee | ||
| export GO111MODULE=off | ||
| go get -u -v github.com/tebeka/go2xunit | ||
| go get -u -v github.com/axw/gocov/gocov | ||
| go get -u -v github.com/AlekSi/gocov-xml | ||
| else | ||
| # Install tools to use | ||
| go install github.com/axw/gocov/gocov@latest | ||
| go install github.com/AlekSi/gocov-xml@latest | ||
| go install gotest.tools/gotestsum@latest | ||
| fi | ||
| echo "Downloading tool dependencies..." | ||
| go install github.com/axw/gocov/gocov@latest | ||
| go install github.com/AlekSi/gocov-xml@latest | ||
| go install gotest.tools/gotestsum@latest | ||
|
|
||
| if [ "${SG_TEST_X509:-}" == "true" -a "${COUCHBASE_SERVER_PROTOCOL}" != "couchbases" ]; then | ||
| if [ "${SG_TEST_X509:-}" == "true" ] && [ "${COUCHBASE_SERVER_PROTOCOL}" != "couchbases" ]; then | ||
| echo "Setting SG_TEST_X509 requires using couchbases:// protocol, aborting integration tests" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Set environment vars | ||
| GO_TEST_FLAGS="-v -p 1 -count=${RUN_COUNT:-1}" | ||
| GO_TEST_FLAGS=(-v -p 1 "-count=${RUN_COUNT:-1}") | ||
| INT_LOG_FILE_NAME="verbose_int" | ||
|
|
||
| if [ -d "godeps" ]; then | ||
| export GOPATH=$(pwd)/godeps | ||
| fi | ||
| export PATH=$PATH:$(go env GOPATH)/bin:~/go/bin | ||
| export PATH=$PATH:~/go/bin | ||
| echo "PATH: $PATH" | ||
|
|
||
| if [ "${TEST_DEBUG:-}" == "true" ]; then | ||
|
|
@@ -83,38 +77,37 @@ if [ "${TEST_DEBUG:-}" == "true" ]; then | |
| fi | ||
|
|
||
| if [ "${TARGET_TEST}" != "ALL" ]; then | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -run ${TARGET_TEST}" | ||
| GO_TEST_FLAGS+=(-run "${TARGET_TEST}") | ||
| fi | ||
|
|
||
| if [ "${PACKAGE_TIMEOUT:-}" != "" ]; then | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -test.timeout=${PACKAGE_TIMEOUT}" | ||
| GO_TEST_FLAGS+=(-test.timeout="${PACKAGE_TIMEOUT}") | ||
| else | ||
| echo "Defaulting package timeout to ${DEFAULT_PACKAGE_TIMEOUT}" | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -test.timeout=${DEFAULT_PACKAGE_TIMEOUT}" | ||
| GO_TEST_FLAGS+=(-test.timeout="${DEFAULT_PACKAGE_TIMEOUT}") | ||
| fi | ||
|
|
||
| if [ "${DETECT_RACES:-}" == "true" ]; then | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -race" | ||
| GO_TEST_FLAGS+=(-race) | ||
| fi | ||
|
|
||
| if [ "${FAIL_FAST:-}" == "true" ]; then | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -failfast" | ||
| GO_TEST_FLAGS+=(-failfast) | ||
| fi | ||
|
|
||
| if [ "${SG_TEST_PROFILE_FREQUENCY:-}" == "true" ]; then | ||
| export SG_TEST_PROFILE_FREQUENCY=${SG_TEST_PROFILE_FREQUENCY} | ||
| fi | ||
|
|
||
| if [ "${RUN_WALRUS}" == "true" ]; then | ||
| set +e | ||
| set +e -x | ||
| # EE | ||
| gotestsum --junitfile=rosmar-ee.xml --junitfile-project-name rosmar-EE --junitfile-testcase-classname relative --format standard-verbose -- -coverprofile=coverage_walrus_ee.out -coverpkg=github.com/couchbase/sync_gateway/... -tags cb_sg_devmode,cb_sg_enterprise $GO_TEST_FLAGS github.com/couchbase/sync_gateway/${TARGET_PACKAGE} > verbose_unit_ee.out 2>&1 | ||
| gotestsum --junitfile=rosmar-ee.xml --junitfile-project-name rosmar-EE --junitfile-testcase-classname relative --format standard-verbose -- -coverprofile=coverage_walrus_ee.out -coverpkg=github.com/couchbase/sync_gateway/... -tags cb_sg_devmode,cb_sg_enterprise "${GO_TEST_FLAGS[@]}" "github.com/couchbase/sync_gateway/${TARGET_PACKAGE}" > verbose_unit_ee.out 2>&1 | ||
| xmlstarlet ed -u '//testcase/@classname' -x 'concat("rosmar-EE-", .)' rosmar-ee.xml > verbose_unit_ee.xml | ||
| # CE | ||
| gotestsum --junitfile=rosmar-ce.xml --junitfile-project-name rosmar-CE --junitfile-testcase-classname relative --format standard-verbose -- -coverprofile=coverage_walrus_ce.out -coverpkg=github.com/couchbase/sync_gateway/... -tags cb_sg_devmode $GO_TEST_FLAGS github.com/couchbase/sync_gateway/${TARGET_PACKAGE} > verbose_unit_ce.out 2>&1 | ||
| gotestsum --junitfile=rosmar-ce.xml --junitfile-project-name rosmar-CE --junitfile-testcase-classname relative --format standard-verbose -- -coverprofile=coverage_walrus_ce.out -coverpkg=github.com/couchbase/sync_gateway/... -tags cb_sg_devmode "${GO_TEST_FLAGS[@]}" "github.com/couchbase/sync_gateway/${TARGET_PACKAGE}" > verbose_unit_ce.out 2>&1 | ||
| xmlstarlet ed -u '//testcase/@classname' -x 'concat("rosmar-CE-", .)' rosmar-ce.xml > verbose_unit_ce.xml | ||
| set -e | ||
|
|
||
| set -e +x | ||
| fi | ||
|
|
||
| # Run CBS | ||
|
|
@@ -129,26 +122,26 @@ else | |
| fi | ||
|
|
||
| # Set up test environment variables for CBS runs | ||
| export SG_TEST_USE_XATTRS=${XATTRS} | ||
| export SG_TEST_USE_GSI=${GSI} | ||
| export SG_TEST_COUCHBASE_SERVER_URL="${COUCHBASE_SERVER_PROTOCOL}://127.0.0.1" | ||
| export SG_TEST_BACKING_STORE="Couchbase" | ||
| export SG_TEST_BUCKET_POOL_SIZE=${SG_TEST_BUCKET_POOL_SIZE} | ||
| export SG_TEST_BUCKET_POOL_DEBUG=${SG_TEST_BUCKET_POOL_DEBUG} | ||
| export SG_TEST_TLS_SKIP_VERIFY=${TLS_SKIP_VERIFY} | ||
|
|
||
| if [ "${SG_EDITION}" == "EE" ]; then | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -tags cb_sg_devmode,cb_sg_enterprise" | ||
| GO_TEST_FLAGS+=(-tags "cb_sg_devmode,cb_sg_enterprise") | ||
| else | ||
| GO_TEST_FLAGS="${GO_TEST_FLAGS} -tags cb_sg_devmode" | ||
| GO_TEST_FLAGS+=(-tags cb_sg_devmode) | ||
| fi | ||
|
|
||
| gotestsum --junitfile=integration.xml --junitfile-project-name integration --junitfile-testcase-classname relative --format standard-verbose -- ${GO_TEST_FLAGS} -coverprofile=coverage_int.out -coverpkg=github.com/couchbase/sync_gateway/... github.com/couchbase/sync_gateway/${TARGET_PACKAGE} 2>&1 | stdbuf -oL tee "${INT_LOG_FILE_NAME}.out" | stdbuf -oL grep -a -E '(--- (FAIL|PASS|SKIP):|github.com/couchbase/sync_gateway(/.+)?\t|TEST: |panic: )' | ||
| set +e -x # Output all executed shell commands | ||
| gotestsum --junitfile=integration.xml --junitfile-project-name integration --junitfile-testcase-classname relative --format standard-verbose -- "${GO_TEST_FLAGS[@]}" -coverprofile=coverage_int.out -coverpkg=github.com/couchbase/sync_gateway/... "github.com/couchbase/sync_gateway/${TARGET_PACKAGE}" 2>&1 | stdbuf -oL tee "${INT_LOG_FILE_NAME}.out" | stdbuf -oL grep -a -E '(--- (FAIL|PASS|SKIP):|github.com/couchbase/sync_gateway(/.+)?\t|TEST: |panic: )' | ||
| if [ "${PIPESTATUS[0]}" -ne "0" ]; then # If test exit code is not 0 (failed) | ||
| echo "Go test failed! Parsing logs to find cause..." | ||
| TEST_FAILED=true | ||
| fi | ||
|
|
||
| set +x # Stop outputting all executed shell commands | ||
|
|
||
| # Collect CBS logs if server error occurred | ||
| if [ "${SG_CBCOLLECT_ALWAYS:-}" == "true" ] || grep -a -q "server logs for details\|Timed out after 1m0s waiting for a bucket to become available" "${INT_LOG_FILE_NAME}.out"; then | ||
| docker exec -t couchbase /opt/couchbase/bin/cbcollect_info /workspace/cbcollect.zip | ||
|
|
@@ -163,7 +156,7 @@ fi | |
|
|
||
| # Get coverage | ||
| ~/go/bin/gocov convert "coverage_int.out" | ~/go/bin/gocov-xml > coverage_int.xml | ||
| if [ "${RUN_WALRUS}" == "true" ]; then | ||
| if [ "${RUN_WALRUS:-}" == "true" ]; then | ||
| ~/go/bin/gocov convert "coverage_walrus_ee.out" | ~/go/bin/gocov-xml > "coverage_walrus_ee.xml" | ||
| ~/go/bin/gocov convert "coverage_walrus_ce.out" | ~/go/bin/gocov-xml > "coverage_walrus_ce.xml" | ||
| fi | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why
docker exec curlhere and not on line 88?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I probably hit this with accidental debugging, I'd rather have it call curl from inside the container than outside given we don't know the network constraints.