Skip to content

Commit dea5f2f

Browse files
sfc-gh-jcarrollsfc-gh-dmatthews
sfc-gh-jcarroll
and
sfc-gh-dmatthews
authored
Add "Automate your tests" page to App Testing feature section (#943)
* Add "Automate your tests" page to App Testing feature section * Add results images / update action versions * Update the llm-examples links * Fix action versions in prose * Nits, typos, and clarity --------- Co-authored-by: Debbie Matthews <[email protected]>
1 parent 8054390 commit dea5f2f

File tree

7 files changed

+200
-6
lines changed

7 files changed

+200
-6
lines changed

‎content/library/advanced-features/app-testing/automate-tests.md

+162
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
---
2+
title: Automate your tests with CI
3+
slug: /library/advanced-features/app-testing/automate-tests
4+
---
5+
6+
# Automate your tests with CI
7+
8+
One of the key benefits of app testing is that tests can be automated using Continuous Integration (CI). By running tests automatically during development, you can validate that changes to your app don't break existing functionality. You can verify app code as you commit, catch bugs early, and prevent accidental breaks before deployment.
9+
10+
There are many popular CI tools, including GitHub Actions, Jenkins, GitLab CI, Azure DevOps, and Circle CI. Streamlit app testing will integrate easily with any of them similar to any other Python tests.
11+
12+
## GitHub Actions
13+
14+
Since many Streamlit apps (and all Community Cloud apps) are built in GitHub, this page uses examples from [GitHub Actions](https://docs.github.com/en/actions). For more information about GitHub Actions, see:
15+
16+
- [Quickstart for GitHub Actions](https://docs.github.com/en/actions/quickstart)
17+
- [GitHub Actions: About continuous integration](https://docs.github.com/en/actions/automating-builds-and-tests/about-continuous-integration)
18+
- [GitHub Actions: Build & test Python](https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python)
19+
20+
## Streamlit App Action
21+
22+
[Streamlit App Action](https://github.com/marketplace/actions/streamlit-app-action) provides an easy way to add automated testing to your app repository in GitHub. It also includes basic smoke testing for each page of your app without you writing any test code.
23+
24+
To install Streamlit App Action, add a workflow `.yml` file to your repository's `.github/workflows/` folder. For example:
25+
26+
```yaml
27+
# .github/workflows/streamlit-app.yml
28+
name: Streamlit app
29+
30+
on:
31+
push:
32+
branches: ["main"]
33+
pull_request:
34+
branches: ["main"]
35+
36+
permissions:
37+
contents: read
38+
39+
jobs:
40+
streamlit:
41+
runs-on: ubuntu-latest
42+
steps:
43+
- uses: actions/checkout@v4
44+
- uses: actions/setup-python@v5
45+
with:
46+
python-version: "3.11"
47+
- uses: streamlit/[email protected]
48+
with:
49+
app-path: streamlit_app.py
50+
```
51+
52+
Let's take a look in more detail at what this action workflow is doing.
53+
54+
### Triggering the workflow
55+
56+
```yaml
57+
on:
58+
push:
59+
branches: ["main"]
60+
pull_request:
61+
branches: ["main"]
62+
```
63+
64+
This workflow will be triggered and execute tests on pull requests targeting the `main` branch, as well as any new commits pushed to the `main` branch. Note that it will also execute the tests on subsequent commits to any open pull requests. See [GitHub Actions: Triggering a workflow](https://docs.github.com/en/actions/using-workflows/triggering-a-workflow) for more information and examples.
65+
66+
### Setting up the test environment
67+
68+
```yaml
69+
jobs:
70+
streamlit:
71+
runs-on: ubuntu-latest
72+
steps:
73+
- uses: actions/checkout@v4
74+
- uses: actions/setup-python@v5
75+
with:
76+
python-version: "3.11"
77+
```
78+
79+
The workflow has a `streamlit` job that executes a series of steps. The job runs on a Docker container with the `ubuntu-latest` image.
80+
81+
- `actions/checkout@v4` checks out the current repository code from GitHub and copies the code to the job environment.
82+
- `actions/setup-python@v5` installs Python version 3.11.
83+
84+
### Running the app tests
85+
86+
```yaml
87+
- uses: streamlit/[email protected]
88+
with:
89+
app-path: streamlit_app.py
90+
```
91+
92+
Streamlit App Action does the following:
93+
94+
- Install `pytest` and install any dependencies specified in `requirements.txt`.
95+
- Run the built-in app smoke tests.
96+
- Run any other Python tests found in the repository.
97+
98+
<Tip>
99+
100+
If your app doesn't include `requirements.txt` in the repository root directory, you will need to add a step to install dependencies with your chosen package manager before running Streamlit App Action.
101+
102+
</Tip>
103+
104+
The built-in smoke tests have the following behavior:
105+
106+
- Run the app specified at `app-path` as an AppTest.
107+
- Validate that it completes successfully and does not result in an uncaught exception.
108+
- Do the same for any additional `pages/` of the app relative to `app-path`.
109+
110+
If you want to run Streamlit App Action without the smoke tests, you can set `skip-smoke: true`.
111+
112+
### Linting your app code
113+
114+
Linting is the automated checking of source code for programmatic and stylistic errors. This is done by using a lint tool (otherwise known as a linter). Linting is important to reduce errors and improve the overall quality of your code, especially for repositories with multiple developers or public repositories.
115+
116+
You can add automated linting with [Ruff](https://docs.astral.sh/ruff/) by passing `ruff: true` to Streamlit App Action.
117+
118+
```yaml
119+
- uses: streamlit/[email protected]
120+
with:
121+
app-path: streamlit_app.py
122+
ruff: true
123+
```
124+
125+
<Tip>
126+
127+
You may want to add a pre-commit hook like [ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit) in your local development environment to fix linting errors before they get to CI.
128+
129+
</Tip>
130+
131+
### Viewing results
132+
133+
If tests fail, the CI workflow will fail and you will see the results in GitHub. Console logs are available by clicking into the workflow run [as described here](https://docs.github.com/en/actions/using-workflows/about-workflows#viewing-the-activity-for-a-workflow-run).
134+
135+
![](/images/test-results-logs.png)
136+
137+
For higher-level test results, you can use [pytest-results-action](https://github.com/marketplace/actions/pytest-results-actions). You can combine this with Streamlit App Action as follows:
138+
139+
```yaml
140+
# ... setup as above ...
141+
- uses: streamlit/[email protected]
142+
with:
143+
app-path: streamlit_app.py
144+
# Add pytest-args to output junit xml
145+
pytest-args: -v --junit-xml=test-results.xml
146+
- if: always()
147+
uses: pmeier/[email protected]
148+
with:
149+
path: test-results.xml
150+
summary: true
151+
display-options: fEX
152+
```
153+
154+
![](/images/test-results-summary.png)
155+
156+
## Writing your own actions
157+
158+
The above is just provided as an example. Streamlit App Action is a quick way to get started. Once you learn the basics of your CI tool of choice, it's easy to build and customize your own automated workflows. This is a great way to improve your overall productivity as a developer and the quality of your apps.
159+
160+
## Working example
161+
162+
As a final working example example, take a look at our [`streamlit/llm-examples` Actions](https://github.com/streamlit/llm-examples/actions), defined in [this workflow file](https://github.com/streamlit/llm-examples/blob/main/.github/workflows/app-testing.yml).

‎content/library/advanced-features/app-testing/beyond-the-basics.md

-6
Original file line numberDiff line numberDiff line change
@@ -128,9 +128,3 @@ def test_balloons():
128128
```
129129

130130
By setting the value `at.session_state["magic_word"] = "Balloons"` within the test, you can simulate a user navigating to `second.py` after entering and saving "Balloons" on `first.py`.
131-
132-
## Automate your tests with GitHub actions
133-
134-
One of the key benefits of app testing is that tests can be automated. GitHub Actions are commonly used to validate commits and prevent accidental breaks. As an example, take a look at our [`streamlit/llm-examples`](https://github.com/streamlit/llm-examples) repo. Within `.github/workflows`, a script creates a virtual Python environment and [runs `pytest`](https://github.com/streamlit/llm-examples/blob/bbcc2667cec2a347b34ab3420b57d6ecb42a3188/.github/workflows/python-app.yml#L38).
135-
136-
Check out GitHub docs to learn about [GitHub Actions](https://docs.github.com/en/actions) and [Automating Projects using Actions](https://docs.github.com/en/issues/planning-and-tracking-with-projects/automating-your-project/automating-projects-using-actions).

‎content/library/advanced-features/app-testing/examples.md

+30
Original file line numberDiff line numberDiff line change
@@ -137,3 +137,33 @@ def test_log_out():
137137
```
138138

139139
See how Session State was modified in the last test? Instead of fully simulating a user logging in, the test jumps straight to a logged-in state by setting `at.session_state["status"] = "verified"`. After running the app, the test proceeds to simulate the user logging out.
140+
141+
### Automating your tests
142+
143+
If `myproject/` was pushed to GitHub as a repository, you could add GitHub Actions test automation with [Streamlit App Action](https://github.com/marketplace/actions/streamlit-app-action). This is as simple as adding a workflow file at `myproject/.github/workflows/`:
144+
145+
```yaml
146+
# .github/workflows/streamlit-app.yml
147+
name: Streamlit app
148+
149+
on:
150+
push:
151+
branches: ["main"]
152+
pull_request:
153+
branches: ["main"]
154+
155+
permissions:
156+
contents: read
157+
158+
jobs:
159+
streamlit:
160+
runs-on: ubuntu-latest
161+
steps:
162+
- uses: actions/checkout@v4
163+
- uses: actions/setup-python@v5
164+
with:
165+
python-version: "3.11"
166+
- uses: streamlit/[email protected]
167+
with:
168+
app-path: app.py
169+
```

‎content/library/advanced-features/app-testing/index.md

+6
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ The provided class, AppTest, simulates a running app and provides methods to set
2222
bold="Beyond the basics"
2323
href="/library/advanced-features/app-testing/beyond-the-basics"
2424
>explains how to work with secrets and Session State within app tests, including how to test multipage apps.</InlineCallout>
25+
<InlineCallout
26+
color="indigo-70"
27+
icon="play_circle"
28+
bold="Automate your tests"
29+
href="/library/advanced-features/app-testing/automate-tests"
30+
>with Continuous Integration (CI) to validate app changes over time.</InlineCallout>
2531
<InlineCallout
2632
color="indigo-70"
2733
icon="quiz"

‎content/menu.md

+2
Original file line numberDiff line numberDiff line change
@@ -461,6 +461,8 @@ site_menu:
461461
url: /library/advanced-features/app-testing/get-started
462462
- category: Streamlit library / Advanced features / App testing / Beyond the basics
463463
url: /library/advanced-features/app-testing/beyond-the-basics
464+
- category: Streamlit library / Advanced features / App testing / Automate your tests
465+
url: /library/advanced-features/app-testing/automate-tests
464466
- category: Streamlit library / Advanced features / App testing / Example
465467
url: /library/advanced-features/app-testing/examples
466468
- category: Streamlit library / Advanced features / App testing / Cheat sheet

‎public/images/test-results-logs.png

115 KB
Loading

‎public/images/test-results-summary.png

52.3 KB
Loading

0 commit comments

Comments
 (0)