Skip to content

Commit 5e68444

Browse files
tiborsimkotiborsimko
committed
ci(check-fixtures): parallelise fixture checking commands
Introduces several independent `run-tests.sh` fixture-checking commands in order to speed up fixture checking by parallelisation. Renames `run-tests.sh` script options and CI rules to better separate data checks, formatting checks and linting checks. Adds data formatting checks and fixes several JSON data files. Adds `shfmt` formatting checks, `commitlint`, flake8` and `yamllint` linting checks. Removes `pydocstyle` formating checks since we moved to `black` code formatter. Introduces `/run-tests.sh --help` explaining all the checking options. Updates CI environment to Ubuntu 24.04 and latest actions (`actions/checkout@v4`, `actions/setup-node@v4`, `actions/setup-python@v5`). Amends `.editorconfig` to add rules for shell scripts and remove rules for ReST files that are no longer needed after switch to Markdown. BREAKING CHANGE: Refactors `run-tests.sh` script options.
1 parent 3bb078c commit 5e68444

19 files changed

+433
-149
lines changed

‎.commitlintrc.yaml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
rules:
2+
body-case: [2, always, sentence-case]
3+
body-full-stop: [1, always]
4+
body-leading-blank: [2, always]
5+
body-max-line-length: [2, always, 72]
6+
footer-leading-blank: [2, always]
7+
footer-max-line-length: [2, always, 72]
8+
header-max-length: [2, always, 72]
9+
scope-case: [2, always, lower-case]
10+
subject-case:
11+
- 2
12+
- never
13+
- [pascal-case, sentence-case, start-case, upper-case]
14+
subject-empty: [2, never]
15+
subject-full-stop: [2, never, "."]
16+
type-case: [2, always, lower-case]
17+
type-empty: [2, never]
18+
type-enum:
19+
- 2
20+
- always
21+
- [build, chore, ci, docs, feat, fix, perf, refactor, style, test]

‎.editorconfig

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# -*- coding: utf-8 -*-
22
#
33
# This file is part of CERN Open Data Portal.
4-
# Copyright (C) 2016 CERN.
4+
# Copyright (C) 2016, 2025 CERN.
55
#
66
# CERN Open Data Portal is free software; you can redistribute it
77
# and/or modify it under the terms of the GNU General Public License as
@@ -40,8 +40,8 @@ known_first_party = cernopendata
4040
multi_line_output = 2
4141
default_section = THIRDPARTY
4242

43-
# RST files (used by sphinx)
44-
[*.rst]
43+
# Shell script files
44+
[*.sh]
4545
indent_size = 4
4646

4747
# CSS, HTML, JS, JSON, YML

‎.flake8

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[flake8]
2+
max-line-length = 89
3+
4+
exclude =
5+
build
6+
dist
7+
docs
8+
coverage.xml
9+
cernopendata_client.egg-info
10+
.*/
11+
env/
12+
.git
13+
__pycache__
14+
15+
ignore = E203, E231, E266, E501, W503, F403, F401, W605
16+
17+
max-complexity = 18
18+
19+
select = B,C,E,F,W,T4,B9

‎.github/workflows/ci.yml

Lines changed: 180 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file is part of CERN Open Data Portal.
2-
# Copyright (C) 2020, 2023, 2024 CERN.
2+
# Copyright (C) 2020, 2023, 2024, 2025 CERN.
33
#
44
# CERN Open Data Portal is free software; you can redistribute it
55
# and/or modify it under the terms of the GNU General Public License as
@@ -20,117 +20,241 @@
2020
# waive the privileges and immunities granted to it by virtue of its status
2121
# as an Intergovernmental Organization or submit itself to any jurisdiction.
2222

23-
name: CI
23+
name: ci
2424

2525
on: [push, pull_request]
2626

2727
jobs:
28-
lint-shellcheck:
29-
runs-on: ubuntu-20.04
28+
data-dois:
29+
runs-on: ubuntu-24.04
30+
steps:
31+
- name: Checkout
32+
uses: actions/checkout@v4
33+
34+
- name: Setup node
35+
uses: actions/setup-node@v4
36+
37+
- name: Check data DOIs
38+
run: ./run-tests.sh --data-dois
39+
40+
data-json:
41+
runs-on: ubuntu-24.04
3042
steps:
3143
- name: Checkout
32-
uses: actions/checkout@v2
44+
uses: actions/checkout@v4
3345

34-
- name: Runs shell script static analysis
46+
- name: Setup Python
47+
uses: actions/setup-python@v5
48+
with:
49+
python-version: 3.9
50+
51+
- name: Check data JSON
3552
run: |
36-
sudo apt-get install shellcheck
37-
./run-tests.sh --check-shellscript
53+
pip install click
54+
./run-tests.sh --data-json
3855
39-
lint-black:
40-
runs-on: ubuntu-20.04
56+
data-licenses:
57+
runs-on: ubuntu-24.04
4158
steps:
42-
- uses: actions/checkout@v2
59+
- name: Checkout
60+
uses: actions/checkout@v4
4361

4462
- name: Setup Python
45-
uses: actions/setup-python@v2
63+
uses: actions/setup-python@v5
4664
with:
4765
python-version: 3.9
4866

49-
- name: Check Python code formatting
67+
- name: Check data licenses
5068
run: |
51-
pip install black
52-
./run-tests.sh --check-black
69+
./run-tests.sh --data-licenses
70+
71+
data-recids:
72+
runs-on: ubuntu-24.04
73+
steps:
74+
- name: Checkout
75+
uses: actions/checkout@v4
76+
77+
- name: Setup node
78+
uses: actions/setup-node@v4
5379

54-
lint-pycodestyle:
55-
runs-on: ubuntu-20.04
80+
- name: Check data record IDs
81+
run: ./run-tests.sh --data-recids
82+
83+
data-slugs:
84+
runs-on: ubuntu-24.04
85+
steps:
86+
- name: Checkout
87+
uses: actions/checkout@v4
88+
89+
- name: Setup node
90+
uses: actions/setup-node@v4
91+
92+
- name: Check data slugs
93+
run: ./run-tests.sh --data-slugs
94+
95+
data-types:
96+
runs-on: ubuntu-24.04
5697
steps:
5798
- name: Checkout
58-
uses: actions/checkout@v2
99+
uses: actions/checkout@v4
100+
101+
- name: Setup node
102+
uses: actions/setup-node@v4
103+
104+
- name: Check data types
105+
run: ./run-tests.sh --data-types
106+
107+
format-black:
108+
runs-on: ubuntu-24.04
109+
steps:
110+
- name: Checkout
111+
uses: actions/checkout@v4
59112

60113
- name: Setup Python
61-
uses: actions/setup-python@v2
114+
uses: actions/setup-python@v5
62115
with:
63116
python-version: 3.9
64117

65-
- name: Check compliance with Python coding style conventions
118+
- name: Check formatting of Python code
66119
run: |
67-
pip install --upgrade pip
68-
pip install pycodestyle
69-
./run-tests.sh --check-pycodestyle
120+
pip install black
121+
./run-tests.sh --format-black
70122
71-
lint-pydocstyle:
72-
runs-on: ubuntu-20.04
123+
format-isort:
124+
runs-on: ubuntu-24.04
73125
steps:
74126
- name: Checkout
75-
uses: actions/checkout@v2
127+
uses: actions/checkout@v4
76128

77129
- name: Setup Python
78-
uses: actions/setup-python@v2
130+
uses: actions/setup-python@v5
79131
with:
80132
python-version: 3.9
81133

82-
- name: Check compliance with Python docstring conventions
134+
- name: Check formatting of Python imports
83135
run: |
84-
pip install --upgrade pip
85-
pip install pydocstyle
86-
./run-tests.sh --check-pydocstyle
136+
pip install isort
137+
./run-tests.sh --format-isort
138+
139+
format-shfmt:
140+
runs-on: ubuntu-24.04
141+
steps:
142+
- name: Checkout
143+
uses: actions/checkout@v4
144+
145+
- name: Check formatting of shell scripts
146+
run: |
147+
sudo apt-get install shfmt
148+
./run-tests.sh --format-shfmt
87149
88-
check-fixtures:
89-
runs-on: ubuntu-20.04
150+
format-whitespace:
151+
runs-on: ubuntu-24.04
90152
steps:
91153
- name: Checkout
92-
uses: actions/checkout@v2
154+
uses: actions/checkout@v4
93155

94156
- name: Setup node
95-
uses: actions/setup-node@v1
157+
uses: actions/setup-node@v4
158+
159+
- name: Check formatting of whitespace
160+
run: ./run-tests.sh --format-whitespace
161+
162+
lint-commitlint:
163+
runs-on: ubuntu-24.04
164+
steps:
165+
- name: Checkout
166+
uses: actions/checkout@v4
96167
with:
97-
node-version: "14"
168+
fetch-depth: 0
98169

99-
- name: Install Node dependencies
100-
run: npm install -g jsonlint
170+
- name: Setup Node
171+
uses: actions/setup-node@v4
101172

102-
- name: Check fixtures
103-
run: ./run-tests.sh --check-fixtures
173+
- name: Install commitlint
174+
run: |
175+
npm install conventional-changelog-conventionalcommits
176+
npm install commitlint@latest
177+
178+
- name: Check commit message compliance of the recently pushed commit
179+
if: github.event_name == 'push'
180+
run: |
181+
./run-tests.sh --lint-commitlint HEAD~1 HEAD
104182
105-
check-isort:
106-
runs-on: ubuntu-20.04
183+
- name: Check commit message compliance of the pull request
184+
if: github.event_name == 'pull_request'
185+
run: |
186+
./run-tests.sh --lint-commitlint ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} ${{ github.event.pull_request.number }}
187+
188+
lint-flake8:
189+
runs-on: ubuntu-24.04
107190
steps:
108-
- name: Checkout
109-
uses: actions/checkout@v2
191+
- uses: actions/checkout@v4
110192

111193
- name: Setup Python
112-
uses: actions/setup-python@v2
194+
uses: actions/setup-python@v5
113195
with:
114196
python-version: 3.9
115197

116-
- name: Check isort
198+
- name: Check linting of Python code
117199
run: |
118-
pip install --upgrade pip
119-
pip install isort
120-
./run-tests.sh --check-isort
200+
pip install flake8
201+
./run-tests.sh --lint-flake8
202+
203+
lint-jsonlint:
204+
runs-on: ubuntu-24.04
205+
steps:
206+
- name: Checkout
207+
uses: actions/checkout@v4
208+
209+
- name: Setup Node
210+
uses: actions/setup-node@v4
211+
212+
- name: Check linting of JSON files
213+
run: |
214+
npm install jsonlint --global
215+
./run-tests.sh --lint-jsonlint
121216
122-
check-licenses:
123-
runs-on: ubuntu-20.04
217+
lint-pydocstyle:
218+
runs-on: ubuntu-24.04
124219
steps:
125220
- name: Checkout
126-
uses: actions/checkout@v2
221+
uses: actions/checkout@v4
127222

128223
- name: Setup Python
129-
uses: actions/setup-python@v2
224+
uses: actions/setup-python@v5
130225
with:
131226
python-version: 3.9
132227

133-
- name: Check licenses
228+
- name: Check linting of Python docstrings
229+
run: |
230+
pip install pydocstyle
231+
./run-tests.sh --lint-pydocstyle
232+
233+
lint-shellcheck:
234+
runs-on: ubuntu-24.04
235+
steps:
236+
- name: Checkout
237+
uses: actions/checkout@v4
238+
239+
- name: Check linting of shell scripts
240+
run: |
241+
sudo apt-get install shellcheck
242+
./run-tests.sh --lint-shellcheck
243+
244+
lint-yamllint:
245+
runs-on: ubuntu-24.04
246+
steps:
247+
- name: Checkout
248+
uses: actions/checkout@v4
249+
with:
250+
fetch-depth: 0
251+
252+
- name: Setup Python
253+
uses: actions/setup-python@v5
254+
with:
255+
python-version: "3.12"
256+
257+
- name: Lint YAML files
134258
run: |
135-
pip install --upgrade pip
136-
./run-tests.sh --check-licenses
259+
pip install yamllint
260+
./run-tests.sh --lint-yamllint

‎.yamllint.yaml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
extends: default
2+
3+
rules:
4+
comments:
5+
min-spaces-from-content: 1
6+
document-start: disable
7+
line-length: disable
8+
truthy: disable

‎data/docs/atlas-heavy-ion-open-data-for-research-release-2024/atlas-heavy-ion-open-data-for-research-release-2024.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@
66
"format": "md"
77
},
88
"date_published": "2024-12-13",
9-
"experiment": ["ATLAS"],
9+
"experiment": [
10+
"ATLAS"
11+
],
1012
"featured": 1,
1113
"short_description": {
1214
"content": "The ATLAS Collaboration has released its first open data of heavy-ion collisions for research purposes. This dataset features lead-ion (Pb-Pb) collisions at an energy of 5 TeV per nucleon pair, recorded in 2015 as part of the Large Hadron Collider’s second operation period (LHC Run 2)."

‎data/docs/lhcb-about/lhcb-about.json

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,6 @@
44
"content": "lhcb-about.md",
55
"format": "md"
66
},
7-
87
"experiment": [
98
"LHCb"
109
],

0 commit comments

Comments
 (0)