Skip to content

Latest commit

 

History

History
247 lines (179 loc) · 9.37 KB

guide-release-testing.md

File metadata and controls

247 lines (179 loc) · 9.37 KB
Raw

Release Testing

Release testing is manual testing the release crew does to sanity check that our new releases build and run and don't regress core experiences.

Important

Make sure you have the right access/tokens set up. See onboarding.

Caution

Currently, this flow can only be done on macOS machines.

Note

In most cases we should have 2 release crew members running release testing, on two separate machines.

Checkout the version to test

When testing locally, first checkout and pull from the stable branch for the version you wish to test. For example, if testing 0.75.x:

  1. git checkout 0.75-stable
  2. git pull

Clean up the local state

When testing locally, we want to ensure that we start from a clean slate to avoid caches polluting our testing.

  1. Delete RNTester and RNTestProject from your Android emulator and iOS simulator if leftover from previous test.

  2. Remove any temporary files from the react-native repo:

     yarn test-e2e-local-clean

  3. Install dependencies:

     yarn install

Generating projects to test

We test on two apps

  • RNTester, a demonstration app that higlights the APIs and core components of React Native
  • RNTestProject, a project built from the React Native template
App New Architecture enabled
RNTester true
RNTestProject false

Note

If you need to enable New Architecture on RNTestProject, follow the instructions here.

To generate the the right project with the specific configuration desired, you can use the command:

yarn test-e2e-local [options]

Followed by the options:

  --help          Show help                                            [boolean]
  --version       Show version number                                  [boolean]
  -t, --target      [choices: "RNTester", "RNTestProject"] [default: "RNTester"]
  -p, --platform                    [choices: "iOS", "Android"] [default: "iOS"]
  -h, --hermes                                         [boolean] [default: true]
  -c, --ciToken                                                         [string]

Using the -c <your-token> option is recommended, as the script will download the required artifacts from Github, instead of having to build from source reducing the testing time. It's suggested to store your Github Personal Access Token (classic) using:

export GITHUB_TOKEN=<token>

Warning

If you are testing versions of React Native that are < 0.75, you need a CircleCI token rather than a Github Token. Follow these instruction to learn how to create one.

Use this link to generate a token with the Repo scope.

There are a couple of extra requirements to make this flow work:

  1. the CI must have run at least once in the current branch. We need to download artifacts generated by Github, so we need to have at least one run of the CI.
  2. The relevant jobs must have been finished. The required jobs depends on the task you are about to test:
    • if you are testing RNTester on iOS, it has to download Hermes for iOS. So the job that produces it must have finished successfully
    • if you are testing RNTester on Android, it has to download the right APK (JSC or Hermes). So the job that produces the APKs must have finished successfully.
    • if you are testing RNTestProject, it has to download the maven prebuilts and the Hermes engine for iOS. So the jobs that produce those artifacts must have finished successfully.

If any of those prerequisites is not met, the script should output a proper error message.

Warning

CI artifacts are only taken from the last commit. This means everytime you push to your release branch, you'll have to wait for the artifacts to build again. You cannot release test using artifacts from an "older" commit. Plan accordingly.

If you need to build React Native from source, you can skip the -c parameter. By not passing the Github token, the script falls back to the previous flow, building everything locally.

Dimensions to test

Here are the 8 dimensions we cover in manual testing:

1. RNTester + iOS + Hermes:

yarn test-e2e-local -t "RNTester" -p "iOS" -h true -c $GITHUB_TOKEN

2. RNTester + iOS + JSC:

yarn test-e2e-local -t "RNTester" -p "iOS" -h false -c $GITHUB_TOKEN

3. RNTester + Android + Hermes:

yarn test-e2e-local -t "RNTester" -p "Android" -h true -c $GITHUB_TOKEN

4. RNTester + Android + JSC:

yarn test-e2e-local -t "RNTester" -p "Android" -h false -c $GITHUB_TOKEN

5. RNTestProject + iOS + Hermes:

yarn test-e2e-local -t "RNTestProject" -p "iOS" -h true -c $GITHUB_TOKEN

6. RNTestProject + iOS + JSC:

yarn test-e2e-local -t "RNTestProject" -p "iOS" -h false -c $GITHUB_TOKEN

7. RNTestProject + Android + Hermes:

yarn test-e2e-local -t "RNTestProject" -p "Android" -h true -c $GITHUB_TOKEN

8. RNTestProject + Android + JSC:

yarn test-e2e-local -t "RNTestProject" -p "Android" -h false -c $GITHUB_TOKEN

What to test?

  • Depending on the nature of what is being picked in, you may only test very specific workflows.
  • At Meta, we have internal e2e tests but they don't capture interactive capabilities or developer experience workflows.
  • General rule of thumb is to prioritize
    • debugging
    • developer menu
    • TextInput
    • Lists
    • Keyboard interactions

How to test?

  1. Create a Test Report
  2. Fill out the table of test dimensions
  3. Refer to the spreadsheet of test cases, make any outstanding notes in the Test Report
  4. Close the Test Report issue if the release passes testing

Recommended Sequence for testing

Tip

Make sure the remote assets are built on your release branch to use the -c option (highly recommended)

react-native$ yarn test-e2e-local-clean # alias this command to make your life easier to something like "clean"

react-native$ yarn test-e2e-local -p iOS -t RNTester -c <your-circle-ci-token>
react-native$ cd packages/rn-tester && yarn start
# verify tests "what to test"
# kill metro
# delete RNTester from your iOS sim
react-native/packages/rn-tester$ cd ../../

react-native$ yarn test-e2e-local -p iOS -t RNTester --hermes false -c <your-circle-ci-token>
react-native$ cd packages/rn-tester && yarn start
# verify tests "what to test"
# kill metro
# delete RNTester from your iOS sim
react-native/packages/rn-tester$ cd ../../

react-native$ yarn test-e2e-local -p Android -t RNTester -c <your-circle-ci-token>
react-native$ cd packages/rn-tester && yarn start
# verify tests "what to test"
# kill metro
# delete RNTester from your Android emulator
react-native/packages/rn-tester$ cd ../../

react-native$ yarn test-e2e-local -p Android -t RNTester --hermes false -c <your-circle-ci-token>
react-native$ cd packages/rn-tester && yarn start
# verify tests "what to test"
# kill metro
# delete RNTester from your Android emulator
react-native/packages/rn-tester$ cd ../../

# by default, RNTestProject will use Hermes for iOS and Android
react-native$ yarn test-e2e-local -p iOS -t RNTestProject -c <your-circle-ci-token>
react-native$ cd /tmp/RNTestProject
tmp/RNTestProject$ yarn start
# verify tests "what to test"
# kill metro
# delete RNTestProject from your iOS sim

# test RNTestProject on Android with Hermes
tmp/RNTestProject$ yarn run android # this should build the Android app and run on your emulator
tmp/RNTestProject$ yarn start
# verify tests "what to test"
# kill metro
# delete RNTestProject from your Android emulator

# test RNTestProject with JSC for iOS
tmp/RNTestProject$ cd ios && bundle install && USE_HERMES=0 bundle exec pod install --ansi
tmp/RNTestProject/ios$ cd ../
tmp/RNTestProject$ yarn run ios
tmp/RNTestProject$ yarn start
# verify tests "what to test"
# kill metro
# delete RNTestProject from your iOS sim

# update RNTestProject with JSC forAndroid
tmp/RNTestProject$ cd android
tmp/RNTestProject$ vim gradle.properties # update `hermesEnabled` to false
tmp/RNTestProject$ ./gradlew clean
tmp/RNTestProject/ios$ cd ../
tmp/RNTestProject$ yarn run android # This should re-build Android app
tmp/RNTestProject$ yarn start
# verify tests "what to test"
# kill metro
# delete RNTestProject from your Android sim

Versions older than 71

You need to use the interactive script run you through the different variants in Test Dimensions:

./scripts/test-manual-e2e.sh

This script will ask you to select which platform and which project you want to test, and then to execute a series of extra steps during the process. Bear in mind, when testing RNTester on Android, you need to start the Android emulator ahead of time or it will fail.