test: Update e2e docker template (#1427)

mattrunyon · web-flow · commit e3b4cf274ea2 · 2023-08-14T10:17:14.000-04:00
While helping Ethan debug e2e flaky tests in CI, Don mentioned the npm
script to run e2e in docker was ugly. I agreed and decided to clean it
up some.

This change makes the e2e tests in Docker also spin up the DHC server w/
anonymous auth so you don't have to. We could extend this to add a dev
mode for e2e testing possibly, but I don't think you can get headed mode
in Docker if you want it.

You can also pass arguments directly now. `npm run e2e:ci --
--project=chromium tests/someTest.js` and it will run in Docker

Also figured out a way to remove a COPY line for every package.json
using multi-stage builds. It caches properly still if no package.json
files change.

Removed the volume mount for test-results because Docker creates the
directories inside of it as root (I tried running the container as a
regular user and it complains). This causes an issue if you run the
snapshots in Docker w/ a failure and then try to run locally because the
folder is owned by root and can't be cleared as a regular user.

Still need to update the readme. And probably rename `e2e:ci` since it's
not actually run in CI
diff --git a/.dockerignore b/.dockerignore
@@ -1,50 +1,40 @@
-# Just copy the .gitignore file for files we don't want to copy to the docker image
-
 # dependencies
-/node_modules
+**/node_modules/
 
 # testing
 /coverage
+jest*
+__mocks__/
+.github/
 
 # production
-/build
-/dist
+**/build/
+**/dist/
 
 # misc
-.vscode/*
-!.vscode/settings.json
-!.vscode/tasks.json
-!.vscode/launch.json
-!.vscode/extensions.json
-!.vscode/*.code-snippets
+.git/
+.vscode/
 .DS_Store
-.env.local
-.env.development.local
-.env.test.local
-.env.production.local
+**/.env*.local
 .project
 .settings/
 .eslintcache
 .stylelintcache
 lerna-debug.log
 Lerna-Profile-*.json
 
-/public/vs
-
 npm-debug.log*
-yarn-debug.log*
-yarn-error.log*
-
-src/**/*.css
 
-tsconfig.tsbuildinfo
-packages/*/package-lock.json
-/test-results/
-/playwright-report/
-/playwright/.cache/
+**/tsconfig.tsbuildinfo
+test-results/
+playwright-report/
+playwright/.cache/
+**/CHANGELOG.md
 
 # Ignore Dockerfile as well, since we don't need to copy that into the snapshot container
-Dockerfile
+**/Dockerfile
+.dockerignore
+**/.gitignore
 
 # Tests are copied to the docker container, as it modifies them
-/tests
+tests/
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
@@ -53,7 +53,7 @@ jobs:
         run: PLAYWRIGHT_BROWSERS_PATH=0 npx playwright install --with-deps
 
       - name: Run Playwright tests
-        run: PLAYWRIGHT_BROWSERS_PATH=0 npx playwright test
+        run: PLAYWRIGHT_BROWSERS_PATH=0 npx playwright test --config=playwright-ci.config.ts
 
       - name: Upload Playwright report
         uses: actions/upload-artifact@v3
diff --git a/README.md b/README.md
@@ -41,18 +41,10 @@ We use Chrome for development with the React and Redux extensions.
 
 - `npm install` : Install all dependencies and automatically bootstrap packages. Should be run before any of the other steps.
 - `npm start`: Start building all packages and watching them (when possible). Use when you're developing, and your changes will be picked up automatically.
-- `npm test`: Start running tests in all packages and watching (when possible). Use when you're developing, and any breaking changes will be printed out automatically.
-
-  Log messages from our log package are disabled by default in tests in [jest.setup.ts](./jest.setup.ts). To change the log level, set the `DH_LOG_LEVEL` env variable. For example, to enable all log messages run `DH_LOG_LEVEL=4 npm test`.
-
-  Note that log messages from other sources such as react prop types will still be printed since they do not go through our logger.
-
-  If you want to collect coverage locally, run `npm test -- --coverage`
-
+- `npm test`: Start running tests in all packages and watching (when possible). Use when you're developing, and any breaking changes will be printed out automatically. See [Unit tests](#unit-tests) for more details.
 - `npm run build`: Create a production build of all packages. Mainly used by CI when packaging up a production version of the app.
 - `npm run preview`: Runs the Vite preview server for the built code-studio, embed-grid, and embed-chart. These will open on ports 4000, 4010, and 4020.
-- `npm run e2e`: Runs the Playwright end-to-end tests locally.
-- `npm run e2e:headed`: Runs end-to-end tests in headed mode debug mode. Useful if you need to debug why a particular test isn't work. For example, to debug the `table.spec.ts` test directly, you could run `npm run e2e:headed -- ./tests/table.spec.ts`.
+- `npm run e2e`: Runs the Playwright end-to-end tests locally. See [E2E Tests](#e2e-tests) for more details.
 
 If your DHC address is different from the default `http://localhost:10000`, edit `.env.local` in each package to point to your local DHC. This is needed for the session websocket and for things like notebooks to be proxied correctly by Vite.
 
@@ -88,13 +80,47 @@ A standalone application with it's own entry point. Recommend copying the [embed
 
 A component/library package that can be imported into other packages. Recommend copying the [components](./packages/components/) package, removing any dependencies and files not required.
 
-## Updating E2E Snapshots
+## Unit Tests
+
+We use [Jest](https://jestjs.io/) for running unit tests. We use an eslint runner in Jest to lint files as tests. We also use [react-testing-library](https://testing-library.com/docs/react-testing-library/intro/) and [react-hooks-testing-library](https://react-hooks-testing-library.com/) for testing react components and hooks.
+
+Run `npm test` to start Jest in watch mode. By default, this will only run tests that have changed relative to your git origin/main. Keeping your origin/main up to date will help keep the number of tests run to a minimum.
+
+While the tests are running, you can press `p` to filter by filename or test name. You can also press `Shift+P` to see a list of projects and disable certain projects like the eslint or stylelint.
+
+Log messages from our @deephaven/log package are disabled by default in tests in [jest.setup.ts](./jest.setup.ts). To change the log level, set the `DH_LOG_LEVEL` env variable. For example, to enable all log messages run `DH_LOG_LEVEL=4 npm test`.
+
+Note that log messages from other sources such as react prop types will still be printed since they do not go through our logger.
+
+If you want to collect coverage locally, run `npm test -- --coverage`
+
+## E2E Tests
+
+We use [Playwright](https://playwright.dev/) for end-to-end tests. We test against Chrome, Firefox, and Webkit (Safari). Snapshots from E2E tests are only run against Linux so they can be validated in CI.
+
+You should be able to pass arguments to these commands as if you were running Playwright via CLI directly. For example, to test only `table.spec.ts` you could run `npm run e2e -- ./tests/table.spec.ts`, or to test only `table.spec.ts` in Firefox, you could run `npm run e2e -- --project firefox ./tests/table.spec.ts`. See [Playwright CLI](https://playwright.dev/docs/test-cli) for more details.
+
+Snapshots are used by end-to-end tests to visually verify the output. Snapshots are both OS and broser specific. Sometimes changes are made requiring snapshots to be updated. Update snapshots locally by running `npm run e2e:update-snapshots`.
+
+Once you are satisfied with the snapshots and everything is passing locally, you need to use the docker image to update snapshots for CI (unless you are running the same platform as CI (Ubuntu)). Run `npm run e2e:update-ci-snapshots` to update the CI snapshots. The snapshots will be written to your local directories. The Linux snapshots should be committed to git (non-Linux snapshots should be automatically ignored by git).
+
+### Local
+
+These scripts are recommended while developing your tests as they will be quicker to iterate and offer headed mode for debugging. You will need to run `npx playwright install` before using these scripts. You may also need to update the browsers with the same command any time the Playwright version is updated. See [Playwright Browsers](https://playwright.dev/docs/browsers) for more details.
+
+You must have a web UI running on `localhost:4000/ide/` (dev mode and build preview are both fine), and `deephaven-core` running on port 10000 with anonymous authentication for E2E tests. See [this guide](https://deephaven.io/core/docs/how-to-guides/authentication/auth-anon/) for more details on anonymous authentication.
+
+- `npm run e2e`: Run end-to-end tests in headless mode.
+- `npm run e2e:headed`: Runs end-to-end tests in headed debug mode. Also ignores snapshots since a test suite will stop once 1 snapshot comparison fails. Useful if you need to debug why a particular test isn't working. For example, to debug the `table.spec.ts` test directly, you could run `npm run e2e:headed -- ./tests/table.spec.ts`.
+- `npm run e2e:codegen`: Runs Playwright in codegen mode which can help with creating tests. See [Playwright Codegen](https://playwright.dev/docs/codegen/) for more details.
+- `npm run e2e:update-snapshots`: Updates the E2E snapshots for your local OS.
 
-**Note:** You must have [Docker installed](https://docs.docker.com/get-docker/), and `deephaven-core` must already be running on port 10000 with anonymous authentication on your local machine for all e2e tests. To enable anonymous authentication, pass the flag through `START_OPTS`, e.g.: `docker run --detach --publish 10000:10000 --name dh-core-server --env "START_OPTS=-Xmx4g -DAuthHandlers=io.deephaven.auth.AnonymousAuthenticationHandler" ghcr.io/deephaven/server:edge` or `START_OPTS="-DAuthHandlers=io.deephaven.auth.AnonymousAuthenticationHandler" ./gradlew server-jetty-app:run`.
+### Docker
 
-Snapshots are used by end-to-end tests to visually verify the output. Sometimes changes are made requiring snapshots to be updated. Before running snapshots locally, you must be running the development server with `npm start` or have run `npm run build` recently. Run snapshots locally by running `npm run e2e:update-snapshots`.
+These scripts will create a production build from your current code and run the tests in a Docker image. There is no need to install any other dependencies or have any other services running as they are all contained within Docker. Since any source code changes will require a new build, these scripts are only recommended if having issues with CI or if you need to update the snapshots for CI.
 
-Once you are satisfied with the snapshots and everything is passing locally, you need to use a docker image to [update snapshots for CI](https://playwright.dev/docs/test-snapshots) (unless you are running the same platform as CI (Ubuntu)). Run `npm run e2e:update-ci-snapshots` which will mount the current directory into a docker image and re-run the tests from there. Test results will be written to your local directories.
+- `npm run e2e:docker`: Runs the E2E tests in a docker image. This is useful for testing how tests _should_ run in the CI environment locally.
+- `npm run e2e:update-ci-snapshots`: Updates the E2E snapshots for CI.
 
 # Releases
 
diff --git a/package.json b/package.json
@@ -46,12 +46,13 @@
     "test:unit": "jest --config jest.config.unit.cjs",
     "test:lint": "jest --config jest.config.lint.cjs",
     "test:golden-layout": "lerna run test:ci --scope=@deephaven/golden-layout --stream",
+    "version-bump": "lerna version --conventional-commits --create-release github",
     "e2e": "playwright test",
     "e2e:codegen": "playwright codegen http://localhost:4000",
     "e2e:headed": "playwright test --project=chromium --debug --ignore-snapshots",
     "e2e:update-snapshots": "playwright test --update-snapshots",
-    "version-bump": "lerna version --conventional-commits --create-release github",
-    "e2e:update-ci-snapshots": "docker build --tag web-client-ui-ci-snapshots --file ./tests/update-ci-snapshots/Dockerfile . && docker run --rm --network host --volume $(pwd)/tests:/work/tests/ --volume $(pwd)/playwright-report:/work/playwright-report --volume $(pwd)/test-results:/work/test-results web-client-ui-ci-snapshots npm run e2e:update-snapshots -- --config=playwright-ci.config.ts"
+    "e2e:docker": "./tests/docker-scripts/run.sh web-ui-tests",
+    "e2e:update-ci-snapshots": "./tests/docker-scripts/run.sh web-ui-update-snapshots"
   },
   "repository": "https://github.com/deephaven/web-client-ui",
   "devDependencies": {
diff --git a/playwright-ci.config.ts b/playwright-ci.config.ts
@@ -1,8 +1,16 @@
+import type { PlaywrightTestConfig } from '@playwright/test';
 import DefaultConfig from './playwright.config';
 
 const config: PlaywrightTestConfig = {
   ...DefaultConfig,
-  snapshotDir: './tests',
+  webServer: {
+    // Only start the main code-studio server right now
+    // To test embed-grid and embed-chart, should have an array set for `webServer` and run them all separately as there's a port check
+    command: 'BASE_URL=/ide/ npm run preview:app -- -- -- --no-open', // Passing flags through npm is fun
+    port: 4000,
+    timeout: 60 * 1000,
+    reuseExistingServer: false,
+  },
 };
 
 export default config;
diff --git a/playwright.config.ts b/playwright.config.ts
@@ -32,7 +32,8 @@ const config: PlaywrightTestConfig = {
   /* Only have one worker since we don't want tests running in parallel, trampling over each other on the backend */
   workers: 1,
   /* Reporter to use. See https://playwright.dev/docs/test-reporters */
-  reporter: 'html',
+  /* Use host 0.0.0.0 so it can be forwarded from within docker */
+  reporter: [['html', { host: '0.0.0.0', port: 9323 }]],
   /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
   use: {
     /* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */
@@ -73,48 +74,7 @@ const config: PlaywrightTestConfig = {
         ...devices['Desktop Safari'],
       },
     },
-
-    /* Test against mobile viewports. */
-    // {
-    //   name: 'Mobile Chrome',
-    //   use: {
-    //     ...devices['Pixel 5'],
-    //   },
-    // },
-    // {
-    //   name: 'Mobile Safari',
-    //   use: {
-    //     ...devices['iPhone 12'],
-    //   },
-    // },
-
-    /* Test against branded browsers. */
-    // {
-    //   name: 'Microsoft Edge',
-    //   use: {
-    //     channel: 'msedge',
-    //   },
-    // },
-    // {
-    //   name: 'Google Chrome',
-    //   use: {
-    //     channel: 'chrome',
-    //   },
-    // },
   ],
-
-  /* Folder for test artifacts such as screenshots, videos, traces, etc. */
-  // outputDir: 'test-results/',
-
-  /* Run your local dev server before starting the tests */
-  webServer: {
-    // Only start the main code-studio server right now
-    // To test embed-grid and embed-chart, should have an array set for `webServer` and run them all separately as there's a port check
-    command: 'BASE_URL=/ide/ npm run preview:app -- -- -- --no-open', // Passing flags through npm is fun
-    port: 4000,
-    timeout: 60 * 1000,
-    reuseExistingServer: !process.env.CI,
-  },
 };
 
 export default config;
diff --git a/tests/docker-scripts/Dockerfile b/tests/docker-scripts/Dockerfile
@@ -0,0 +1,48 @@
+# syntax=docker/dockerfile:1
+# Dockerfile for updating the snapshots.
+# Expects to be run from the root of the web-client-ui repo
+FROM mcr.microsoft.com/playwright:v1.30.0-jammy AS playwright
+WORKDIR /work/
+
+# Update packages list and install some build tools
+RUN set -eux; \
+    apt-get update; \
+    apt-get install build-essential --yes;
+
+# Copy just the .nvmrc first and install nvm/node/npm first as these will change the least often
+# https://docs.docker.com/build/cache/
+COPY .nvmrc .
+
+# Set the default shell so the correct node/npm is used after install
+# https://stackoverflow.com/a/60137919
+SHELL ["/bin/bash", "--login", "-i", "-c"]
+RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
+RUN source /root/.bashrc && nvm install
+SHELL ["/bin/bash", "--login", "-c"]
+
+# This is a workaround for copying all package.json files w/ directory structure
+# without needing to list every file as a COPY command
+# The copy --from=copy-packages command will be a cache hit if the package.json files didn't change
+FROM alpine AS copy-packages
+WORKDIR /work/
+COPY . /tmp/web-client-ui
+COPY package.json package-lock.json skip.js .
+# cd first so the cp doesn't include /tmp/web-client-ui in the paths
+RUN cd /tmp/web-client-ui && cp --parents ./packages/*/package.json /work/
+
+FROM playwright AS build
+WORKDIR /work/
+COPY --from=copy-packages /work/ .
+
+# Disable the postinstall script, or npm ci will try and build and the files won't be there
+# We don't need the postinstall since we're going to rebuild right after
+RUN SKIP_POSTINSTALL=1 npm ci
+
+# Copy the web-client-ui src folder to the docker image
+# We do this last because the source files are the most likely to change
+# This requires the Dockerfile to be built in the context of the root of the web-client-ui repository
+# https://stackoverflow.com/a/34300129
+COPY . .
+
+# Now build the app
+RUN npm run build
diff --git a/tests/docker-scripts/docker-compose.yml b/tests/docker-scripts/docker-compose.yml
@@ -0,0 +1,35 @@
+version: '3.8'
+
+services:
+  dhc-server:
+    image: ghcr.io/deephaven/server:${DHC_VERSION:-edge}
+    expose:
+      - 10000
+    environment:
+      - START_OPTS=-Xmx4g -DAuthHandlers=io.deephaven.auth.AnonymousAuthenticationHandler -Ddeephaven.console.type=python
+
+  web-ui-tests:
+    build:
+      context: ../../
+      dockerfile: ./tests/docker-scripts/Dockerfile
+    ipc: host
+    environment:
+      - VITE_PROXY_URL=http://dhc-server:10000
+    ports:
+      - 9323:9323
+    volumes:
+      - ../../tests:/work/tests
+      - ../../test-results:/work/test-results
+      - ../../playwright-report:/work/playwright-report
+    entrypoint: 'npm run e2e -- --config=playwright-ci.config.ts'
+    depends_on:
+      dhc-server:
+        condition: service_healthy
+
+  web-ui-update-snapshots:
+    extends:
+      service: web-ui-tests
+    entrypoint: 'npm run e2e:update-snapshots -- --config=playwright-ci.config.ts'
+    depends_on: # depends_on is not shared with extends
+      dhc-server:
+        condition: service_healthy
diff --git a/tests/docker-scripts/run.sh b/tests/docker-scripts/run.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# This script runs the docker-compose file with the given arguments
+# The argument should contain the container name to run as the first argument (e.g. web-ui-tests)
+# Once the container finishes its task, this script shuts down the docker containers
+# because docker compose run starts dependent containers, but does not stop them
+
+pushd "$(dirname "$0")" # Set pwd to this directory
+docker compose run --service-ports --build --rm "$@" # Passes all arguments to the compose file
+exit_code=$?
+docker compose down
+popd # Reset pwd
+exit $exit_code
diff --git a/tests/update-ci-snapshots/Dockerfile b/tests/update-ci-snapshots/Dockerfile
