This section describes the format in which IDT generates console logs and test reports.
AWS IoT Device Tester uses a standard format for printing messages to the console when it starts a test suite. The following excerpt shows an example of a console message generated by IDT.
time="2000-01-02T03:04:05-07:00" level=info msg=Using suite: MyTestSuite_1.0.0
executionId=9a52f362-1227-11eb-86c9-8c8590419f30
Most console messages consist of the following fields:
time
A full ISO 8601 timestamp for the logged event.
level
The message level for the logged event. Typically, the logged message level is one of info, warn, or error. IDT issues a fatal or panic message if it encounters an expected event that causes it to exit early.
msg
The logged message.
executionId
A unique ID string for the current IDT process. This ID is used to differentiate between individual IDT runs.
Console messages generated from a test suite provide additional information about the device under test and the test suite, test group, and test cases that IDT runs. The following excerpt shows an example of a console message generated from a test suite.
time="2000-01-02T03:04:05-07:00" level=info msg=Hello world! suiteId=MyTestSuite
groupId=myTestGroup testCaseId=myTestCase deviceId=my-device
executionId=9a52f362-1227-11eb-86c9-8c8590419f30
The test-suite specific part of the console message contains the following fields:
suiteId
The name of the test suite currently running.
groupId
The ID of the test group currently running.
testCaseId
The ID of the test case current running.
deviceId
A ID of the device under test that the current test case is using.
The test summary contains information about the test suite, the test results for each group that was run, and the locations of the generated logs and report files. The following example shows a test summary message.
========== Test Summary ==========
Execution Time: 5m00s
Tests Completed: 4
Tests Passed: 3
Tests Failed: 1
Tests Skipped: 0
----------------------------------
Test Groups:
GroupA: PASSED
GroupB: FAILED
----------------------------------
Failed Tests:
Group Name: GroupB
Test Name: TestB1
Reason: Something bad happened
----------------------------------
Path to AWS IoT Device Tester Report: /path/to/awsiotdevicetester_report.xml
Path to Test Execution Logs: /path/to/logs
Path to Aggregated JUnit Report: /path/to/MyTestSuite_Report.xml
awsiotdevicetester_report.xml is a signed report that contains the following information:
- The IDT version.
- The test suite version.
- The report signature and key used to sign the report.
- The device SKU and the device pool name specified in the
device.jsonfile. - The product version and the device features that were tested.
- The aggregate summary of test results. This information is the same as that contained in the
suite-name_report.xmlfile.
<apnreport>
<awsiotdevicetesterversion>idt-version</awsiotdevicetesterversion>
<testsuiteversion>test-suite-version</testsuiteversion>
<signature>signature</signature>
<keyname>keyname</keyname>
<session>
<testsession>execution-id</testsession>
<starttime>start-time</starttime>
<endtime>end-time</endtime>
</session>
<awsproduct>
<name>product-name</name>
<version>product-version</version>
<features>
<feature name="<feature-name>" value="supported | not-supported | <feature-value>" type="optional | required"/>
</features>
</awsproduct>
<device>
<sku>device-sku</sku>
<name>device-name</name>
<features>
<feature name="<feature-name>" value="<feature-value>"/>
</features>
<executionMethod>ssh | uart | docker</executionMethod>
</device>
<devenvironment>
<os name="<os-name>"/>
</devenvironment>
<report>
<suite-name-report-contents>
</report>
</apnreport>
The awsiotdevicetester_report.xml file contains an <awsproduct> tag that contains information about the product being tested and the product features that were validated after running a suite of tests.Attributes used in the <awsproduct> tag
name
The name of the product being tested.
version
The version of the product being tested.
features
The features validated. Features marked as required are required for the test suite to validate the device. The following snippet shows how this information appears in the awsiotdevicetester_report.xml file.
<feature name="ssh" value="supported" type="required"></feature>
Features marked as optional are not required for validation. The following snippets show optional features.
<feature name="hsi" value="supported" type="optional"></feature>
<feature name="mqtt" value="not-supported" type="optional"></feature>
The suite-name_Result.xml report is in JUnit XML format. You can integrate it into continuous integration and deployment platforms like Jenkins, Bamboo, and so on. The report contains an aggregate summary of test results.
<testsuites name="<suite-name> results" time="<run-duration>" tests="<number-of-test>" failures="<number-of-tests>" skipped="<number-of-tests>" errors="<number-of-tests>" disabled="0">
<testsuite name="<test-group-id>" package="" tests="<number-of-tests>" failures="<number-of-tests>" skipped="<number-of-tests>" errors="<number-of-tests>" disabled="0">
<!--success-->
<testcase classname="<classname>" name="<name>" time="<run-duration>"/>
<!--failure-->
<testcase classname="<classname>" name="<name>" time="<run-duration>">
<failure type="<failure-type>">
reason
</failure>
</testcase>
<!--skipped-->
<testcase classname="<classname>" name="<name>" time="<run-duration>">
<skipped>
reason
</skipped>
</testcase>
<!--error-->
<testcase classname="<classname>" name="<name>" time="<run-duration>">
<error>
reason
</error>
</testcase>
</testsuite>
</testsuites>
The report section in both the awsiotdevicetester_report.xml or suite-name_report.xml lists the tests that were run and the results.
The first XML tag <testsuites> contains the summary of the test execution. For example:
<testsuites name="MyTestSuite results" time="2299" tests="28" failures="0" errors="0" disabled="0">
```Attributes used in the `<testsuites>` tag
**`name`**
The name of the test suite\.
**`time`**
The time, in seconds, it took to run the test suite\.
**`tests`**
The number of tests executed\.
**`failures`**
The number of tests that were run, but did not pass\.
**`errors`**
The number of tests that IDT couldn't execute\.
**`disabled`**
This attribute is not used and can be ignored\.
In the event of test failures or errors, you can identify the test that failed by reviewing the `<testsuites>` XML tags\. The `<testsuite>` XML tags inside the `<testsuites>` tag show the test result summary for a test group\. For example:
The format is similar to the <testsuites> tag, but with a skipped attribute that is not used and can be ignored. Inside each <testsuite> XML tag, there are <testcase> tags for each executed test for a test group. For example:
<testcase classname="Security Test" name="IP Change Tests" attempts="1"></testcase>
```Attributes used in the `<testcase>` tag
**`name`**
The name of the test\.
**`attempts`**
The number of times IDT executed the test case\.
When a test fails or an error occurs, `<failure>` or `<error>` tags are added to the `<testcase>` tag with information for troubleshooting\. For example: