Skip to content

Commit 726f17c

Browse files
macpijanmacpijan
committed
import v0.2.0
Signed-off-by: Maciej Pijanowski <[email protected]>
1 parent 526280d commit 726f17c

File tree

123 files changed

+18001
-643
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

123 files changed

+18001
-643
lines changed

‎.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,6 @@
1010
[submodule "lib"]
1111
path = lib
1212
url = [email protected]:3mdeb/testing-stands.git
13+
[submodule "osfv-scripts"]
14+
path = osfv-scripts
15+
url = https://github.com/Dasharo/osfv-scripts.git

‎README.md

Lines changed: 129 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,98 +1,170 @@
1-
# open-source firmware validation
1+
# Open Source Firmware Remote Test Environment
22

3-
The following repository contains set of tests and other features to conduct
4-
Dasharo firmware validation procedures.
3+
## Warning
54

6-
## Test stands information and architecture
5+
**!!! WARNING !!!**
6+
This repository is in the process of migration and multiple major reworks. If
7+
you do not know what you are doing, consider not using it until at least
8+
`v0.5.0` is relased. When this is scheduled, link to such a milestone will
9+
apear here.
10+
**!!! WARNING !!!**
711

8-
Every validation procedure is carried out on the testing stand, that is
9-
specially prepared for each platform including:
12+
![regression-architecture](https://cloud.3mdeb.com/index.php/s/KkERgGoniBtjfC4/preview)
1013

11-
* method of connection with the platform,
12-
* method of controlling platform power supply,
13-
* method of controlling platform power on, power off and reset,
14-
* method of flashing the platform.
14+
The following repository contains set of tests and other features to conduct
15+
Dasharo firmware validation procedures.
1516

16-
![regression-architecture](https://cloud.3mdeb.com/index.php/s/KkERgGoniBtjfC4/preview)
17+
## Test environment overview
1718

18-
## Supported platforms
19+
In fact, OSFV currently consist of two separated testing environments:
20+
21+
1. Dasharo OSFV (dedicated for all dasharo platforms; consists of modules:
22+
`dasharo-compatibility`, `dasharo-security`, `dasharo-performance` and
23+
`dasharo-stability`).
1924

20-
| Manufacturer | Platform | Support | $CONFIG | Stand documentation |
21-
|--------------|-----------------------|---------|--------------------------|------------------------|
22-
| MSI | PRO Z690-A DDR5 | Limited | `msi-pro-z690-ddr5` | [Documentation][doc_1] |
23-
| MSI | PRO Z690-A WIFI DDR4 | Limited | `msi-pro-z690-wifi-ddr4` | [Documentation][doc_1] |
25+
Each of these groups differs in the mechanisms implemented and the extent of
26+
support for different payloads.
2427

25-
[doc_1]: https://docs.dasharo.com/transparent-validation/msi-z690/laboratory-assembly-guide/
28+
In addition, keep in mind that due to the approach to generating release files,
29+
for the `raptor-CS talos2` platform dedicated mechanism for testing environment
30+
and running tests have been implemented.
2631

27-
## Startup and configuration of the environment
32+
## Supported platforms
2833

29-
To set the environment locally, use the following commands:
34+
| Manufacturer | Platform | Firmware | $CONFIG |
35+
|--------------|----------------------|--------------------------|----------------------------------------|
36+
| NovaCustom | NV41MZ | Dasharo | `novacustom-nv41mz` |
37+
| NovaCustom | NV41MB | Dasharo | `novacustom-nv41mb` |
38+
| NovaCustom | NS50MU | Dasharo | `novacustom-ns50mu` |
39+
| NovaCustom | NS70MU | Dasharo | `movacustom-ns70mu` |
40+
| NovaCustom | NV41PZ | Dasharo | `novacustom-nv41pz` |
41+
| NovaCustom | NS50PU | Dasharo | `novacustom-ns50pu` |
42+
| NovaCustom | NS70PU | Dasharo | `novacustom-ns70pu` |
43+
| MSI | PRO Z690 A WIFI DDR4 | Dasharo | `msi-pro-z690-a-wifi-ddr4` |
44+
| MSI | PRO Z690 A DDR5 | Dasharo | `msi-pro-z690-a-ddr5` |
45+
| Protectli | VP2410 | Dasharo | `protectli-vp2410` |
46+
| Protectli | VP2420 | Dasharo | `protectli-vp2420` |
47+
| Protectli | VP4630 | Dasharo | `protectli-vp4630` |
48+
| Protectli | VP4650 | Dasharo | `protectli-vp4650` |
49+
| Protectli | VP4670 | Dasharo | `protectli-vp4670` |
50+
| Raptor-CS | TalosII | Dasharo | `raptor-cs_talos2` |
51+
52+
## Getting started
53+
54+
### Initalizing environment
55+
56+
* Clone repository and setup virtualenv:
3057

3158
```bash
59+
git clone https://github.com/Dasharo/open-source-firmware-validation
60+
cd open-source-firmware-validation
3261
git submodule update --init --checkout
33-
virtualenv -p $(which python3) robot-venv
34-
source robot-venv/bin/activate
35-
pip install -U -r requirements.txt
62+
python3 -m virtualenv venv
63+
source venv/bin/activate
64+
```
65+
66+
* Install modules (in case of Raptor Talos II platform):
67+
68+
```
69+
pip install -U -r requirements-openbmc.txt
70+
```
71+
72+
* Install modules (in case of other platforms):
73+
74+
```
75+
pip install -r requirements.txt
3676
```
3777

38-
If the environment has been initialized already, use only the following command:
78+
* If you initialize the environment and try to run the environment again you
79+
just need to use only this command:
3980

4081
```bash
41-
source robot-venv/bin/activate
82+
source venv/bin/activate
4283
```
4384

44-
## Runnig test modules, test suites and test cases
85+
### Running tests
86+
87+
When running tests on Dasharo platforms use the following commands:
4588

46-
To run all the test cases dedicated for tested platform, execute the following
47-
command after initialize the environment:
89+
* For running single test case:
4890

4991
```bash
50-
./regression.sh
92+
robot -L TRACE -v device_ip:$DEVICE_IP -v config:$CONFIG -v fw_file:$FW_FILE \
93+
-t $TEST_CASE_ID $TEST_MODULE/$TEST_SUITE
5194
```
5295

53-
To run only one test module, execute the following command after initialize the
54-
environment:
96+
* For running single test suite:
5597

5698
```bash
57-
robot -L TRACE -v stand_ip:$STAND_IP -v config:$CONFIG -v fw_file:$FW_FILE ./<module_name>
99+
robot -L TRACE -v device_ip:$DEVICE_IP -v config:$CONFIG -v fw_file:$FW_FILE \
100+
$TEST_MODULE/$TEST_SUITE
58101
```
59102

60-
Where:
103+
* For running single test module:
104+
105+
```bash
106+
robot -L TRACE -v device_ip:$DEVICE_IP -v config:$CONFIG -v fw_file:$FW_FILE \
107+
./$TEST_MODULE
108+
```
109+
110+
Parameters should be defined as follows:
111+
112+
* $DEVICE_IP - testing manager IP address; for platforms mounted on the stands
113+
in the lab it will be RTE address; for DUTs, on which we perform tests by
114+
using SSH connection it will be their own IP address.
115+
* $FW_FILE - path to and name of the coreboot firmware file,
116+
* $CONFIG - tested platform config; the value given for this parameter should be
117+
derived from the configuration name of the corresponding platform (folder
118+
platform_configs),
119+
* $TEST_MODULE - name of the test module (i.e. `dasharo-compatibility`),
120+
* $TEST_SUITE - name of the test suite (i.e. `coreboot-base-port`),
121+
* $TEST_CASE_ID - ID of the requested to run test case (i.e. `CBP001.001`).
122+
Note that after test case ID asterisk should be added. This is necessary due
123+
to the construction of the flag `-t` (or `--test`)
61124

62-
* `$STAND_IP` - testing stand IP address.
63-
* `$CONFIG` - tested platform config. All platform configs are available in
64-
`platform-configs` folder.
65-
* `$FW_FILE` - name and path to the firmware file, on which testing should be
66-
performed.
125+
When running tests on Talos2 platform use the following commands:
67126

68-
To run only one test suite, execute the following command after initialize the
69-
environment:
127+
* For running single test case:
70128

71129
```bash
72-
robot -L TRACE -v stand_ip:$STAND_IP -v config:$CONFIG -v fw_file:$FW_FILE ./<module_name>/<suite_name>.robot
130+
robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
131+
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
132+
-t $TEST_CASE_ID $TEST_MODULE/$TEST_SUITE
73133
```
74134

75-
* `$STAND_IP` - testing stand IP address.
76-
* `$CONFIG` - tested platform config. All platform configs are available in
77-
`platform-configs` folder.
78-
* `$FW_FILE` - name and path to the firmware file, on which testing should be
79-
performed.
135+
* For running single test suite:
80136

81-
> Note: If test suite does not contain any test case involving flashing the
82-
device, `$FW_FILE` parameter might be skipped.
137+
```bash
138+
robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
139+
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
140+
$TEST_MODULE/$TEST_SUITE
141+
```
83142

84-
To run only one test case, execute the following command after initialize the
85-
environment:
143+
* For running single test module:
86144

87145
```bash
88-
robot -L TRACE -t "<case_name>*" -v stand_ip:$STAND_IP -v config:$CONFIG -v fw_file:$FW_FILE ./<module_name>/<suite_name>.robot
146+
robot -L TRACE -v device_ip:$DEVICE_IP -v config:raptor-cs_talos2 -v fw_file:$FW_FILE \
147+
-v bootblock_file:$BOOTBLOCK_FILE -v zImage_file:$ZIMAGE_FILE -v pnor_file:$PNOR_FILE \
148+
./$TEST_MODULE
89149
```
90150

91-
* `$STAND_IP` - testing stand IP address.
92-
* `$CONFIG` - tested platform config. All platform configs are available in
93-
`platform-configs` folder.
94-
* `$FW_FILE` - name and path to the firmware file, on which testing should be
95-
performed.
151+
Parameters should be defined as follows:
152+
153+
* $DEVICE_IP - OBMC IP address (currently `192.168.20.9`),
154+
* $FW_FILE - path to and name of the coreboot firmware file,
155+
* $BOOTBLOCK_FILE - path to and name of the bootblock file,
156+
* $ZIMAGE_FILE - path to and name of the zImage file,
157+
* $PNOR_FILE - path to and name of the pnor file,
158+
* $TEST_MODULE - name of the test module (i.e. `dasharo-compatibility`),
159+
* $TEST_SUITE - name of the test suite (i.e. `coreboot-base-port`),
160+
* $TEST_CASE_ID - ID of the requested to run test case (i.e. `CBP001.001`).
161+
Note that after test case ID asterisk should be added. This is necessary due
162+
to the construction of the flag `-t` (or `--test`)
163+
164+
## Contributing
96165

97-
> Note: If test suite does not contain any test case involving flashing the
98-
device, `$FW_FILE` parameter might be skipped.
166+
* Install pre-commit hooks after cloning repository:
167+
168+
```bash
169+
pre-commit install
170+
```

‎dasharo-compatibility/__init__.robot

Lines changed: 36 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -6,26 +6,45 @@ Library OperatingSystem
66
Library String
77
Library RequestsLibrary
88
Library Collections
9-
Library ../../lib/TestingStands.py
10-
Resource ../keys-and-keywords/setup-keywords.robot
11-
Resource ../keys-and-keywords/keys.robot
12-
Resource ../rtectrl-rest-api/rtectrl.robot
13-
Resource ../pikvm-rest-api/pikvm_comm.robot
149
Resource ../sonoff-rest-api/sonoff-api.robot
10+
Resource ../rtectrl-rest-api/rtectrl.robot
11+
Resource ../variables.robot
12+
Resource ../keywords.robot
13+
Resource ../keys.robot
1514

16-
Suite Setup Prepare platform ${fw_file}
15+
Suite Setup Prepare platform
1716

1817

1918
*** Keywords ***
2019
Prepare platform
21-
[Documentation] Keyword allows to initialize connections and flash
22-
... firmware by using default method. Takes firmware file (${fw_file})
23-
... as an argument.
24-
[Arguments] ${fw_file}
25-
Variable Should Exist ${fw_file}
26-
Prepare Test Suite
27-
Flash firmware ${fw_file}
28-
${coreboot_version}= Get firmware version from binary /tmp/coreboot.rom
29-
${version}= Get firmware version
30-
Log Out And Close Connection
31-
Should Contain ${coreboot_version} ${version}
20+
[Documentation] Keyword allows to flashing the device with the required
21+
... firmware version. Number of files used in the flashing procedure and
22+
... method of the flashing depends on the platform specify:
23+
... -> for the platforms that have external flashing enabled,
24+
... flashrom utility installed on the RTE is used.
25+
... -> for the platforms that have BMC, pflash utility built in OBMC is
26+
... used.
27+
... -> for the platforms that don't both don't have external flashing
28+
... enabled and OBMC, internal flashing mechanism is used.
29+
IF '${config}'=='raptor-cs_talos2'
30+
Variable Should Exist ${fw_file}
31+
Variable Should Exist ${bootblock_file}
32+
Variable Should Exist ${zImage_file}
33+
Prepare Test Suite
34+
Flash Heads From OpenBMC ${bootblock_file} ${fw_file} ${zImage_file}
35+
${fw_ver_file}= Get firmware version from coreboot file ${fw_file}
36+
Power On
37+
${fw_ver_bootblock} ${fw_ver_romstage}= Get firmware version from bootlogs
38+
Log Out And Close Connection
39+
Should Be Equal ${fw_ver_file} ${fw_ver_bootblock}
40+
Should Be Equal ${fw_ver_bootblock} ${fw_ver_romstage}
41+
ELSE
42+
Variable Should Exist ${fw_file}
43+
Prepare Test Suite
44+
Flash firmware ${fw_file}
45+
Power Cycle On
46+
${version}= Get firmware version
47+
${coreboot_version}= Get firmware version from binary ${fw_file}
48+
Log Out And Close Connection
49+
Should Contain ${coreboot_version} ${version}
50+
END

‎dasharo-compatibility/audio-subsystem.robot

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
*** Settings ***
2+
Library SSHLibrary timeout=90 seconds
3+
Library Telnet timeout=20 seconds connection_timeout=120 seconds
4+
Library Process
5+
Library OperatingSystem
6+
Library String
7+
Library RequestsLibrary
8+
Library Collections
9+
# TODO: maybe have a single file to include if we need to include the same
10+
# stuff in all test cases
11+
Resource ../sonoff-rest-api/sonoff-api.robot
12+
Resource ../rtectrl-rest-api/rtectrl.robot
13+
Resource ../variables.robot
14+
Resource ../keywords.robot
15+
Resource ../keys.robot
16+
17+
# TODO:
18+
# - document which setup/teardown keywords to use and what are they doing
19+
# - go threough them and make sure they are doing what the name suggest (not
20+
# exactly the case right now)
21+
Suite Setup Run Keyword Prepare Test Suite
22+
Suite Teardown Run Keyword Log Out And Close Connection
23+
24+
25+
*** Test Cases ***
26+
AUD001.001 Audio subsystem detection (Ubuntu 20.04)
27+
[Documentation] Check whether the audio subsystem is initialized correctly
28+
... and can be detected in Linux OS.
29+
Skip If not ${audio_subsystem_support} AUD001.001 not supported
30+
Skip If not ${tests_in_ubuntu_support} AUD001.001 not supported
31+
Power On
32+
Boot system or from connected disk ubuntu
33+
Login to Linux
34+
Switch to root user
35+
Detect or Install Package alsa-utils
36+
${out}= Execute Linux command cat /sys/class/sound/card0/hwC0D*/chip_name
37+
Should Contain ${out} ${device_audio1}
38+
Should Contain ${out} ${device_audio2}
39+
Exit from root user
40+
41+
AUD001.002 Audio subsystem detection (Windows 11)
42+
[Documentation] Check whether the audio subsystem is initialized correctly
43+
... and can be detected in Windows 11.
44+
Skip If not ${audio_subsystem_support} AUD001.002 not supported
45+
Skip If not ${tests_in_windows_support} AUD001.002 not supported
46+
Power On
47+
Boot system or from connected disk ${os_windows}
48+
Login to Windows
49+
${out}= Get Sound Devices Windows
50+
Should Contain ${out} ${device_audio1_win}
51+
Should Contain ${out} OK
52+
53+
## PI-KVM necessary
54+
# AUD002.001 Audio playback (Ubuntu 20.04)
55+
# [Documentation] Check whether the audio subsystem is able to playback
56+
# ... audio recordings.
57+
# Execute Linux command pactl set-sink-mute alsa_output.pci-0000_00_1f.3.analog-stereo 0
58+
# Telnet.Read Until Prompt
59+
# Execute Linux command pactl set-sink-volume alsa_output.pci-0000_00_1f.3.analog-stereo 65535
60+
# Telnet.Read Until Prompt
61+
# Execute Linux command speaker-test
62+
# Telnet.Read Until Prompt
63+
# ${out}= Execute Linux command arecord -qd 1 volt && sox volt -n stat &> volt.d && sed '4q;d' volt.d
64+
# Should Contain ${out} #TODO the output
65+
66+
# in fact tested in AUD002.001
67+
# AUD003.001 Audio capture (Ubuntu 20.04)
68+
# [Documentation] Check whether the audio subsystem is able to capture
69+
# ... audio.
70+
71+
AUD004.001 External headset recognition (Ubuntu 20.04)
72+
[Documentation] Check whether the external headset is recognized
73+
... properly after plugging in micro jack into slot.
74+
Skip If not ${audio_subsystem_support} AUD004.001 not supported
75+
Skip If not ${tests_in_ubuntu_support} AUD004.001 not supported
76+
Power On
77+
Boot system or from connected disk ubuntu
78+
Login to Linux
79+
Switch to root user
80+
${out}= Execute Linux command amixer -c 0 contents | grep -A 2 'Headphone' | cat
81+
${headset_string}= Set Variable values=on
82+
Should Contain ${out} ${headset_string}
83+
Exit from root user
84+
85+
# Work in progress
86+
# AUD004.002 External headset recognition (Windows 11)
87+
# [Documentation] Check whether the external headset is recognized
88+
# ... properly after plugging in micro jack into slot.
89+
# Skip If not ${audio_subsystem_support} AUD004.002 not supported
90+
# Skip If not ${tests_in_windows_support} AUD004.002 not supported
91+
# Power On
92+
# Boot system or from connected disk ${os_windows}
93+
# Login to Windows
94+
# Execute Command In Terminal Install-PackageProvider -Name NuGet -Force
95+
# Execute Command In Terminal Install-Module -Name AudioDeviceCmdlets -Force
96+
# ${out}= Execute Command In Terminal Get-AudioDevice -list | ft Index, Default, Type, Name
97+
# Should Contain ${out} ${headset_string}
98+
# Exit from root user

0 commit comments

Comments
 (0)