Running tests locally: How to?

[mrsimpson]

When running tests locally, there are a couple of issues which we encounter

• typeorm drop schema which runs pre_test fails => fix: Make sure the env is properly being set prior to executing the tests, typeorm itself does not feature DOTENV-support ("typeorm": "export $(cat ../../.env | xargs) && ts-node ./node_modules/typeorm/cli.js")
• Some™️ api-tests fail due to unexpected HTTP-return-codes (particularly expected 403 are not met)

Is there something we need to configure locally in order to make the tests run?

[mrsimpson]

Also we noted, that running tests from a package-subdir fails (probably since jest is not a dev-dependency of each package). Finally, we noted that we cannot trigger a test with watch easily. Is there parameter in pnpm which we need to set in order to pass the watch-parameter to the module?

In other words: How the heck do you test locally in an efficient manner?

[mrsimpson]

/cc @jschirrmacher

[nessur]

hey @mrsimpson !
Repo maintainer here. I normally run tests from either the project root, or in the package directory (when I'm making a small change), as: pnpm install && pnpm build && pnpm test.
If the logs are too verbose, you can add QUIET=true pnpm test.
All you should need, outside of the npm dependencies, is a local installation of postgresql and redis, both running on default ports.
If your test run is still failing, can you post the full logs here? I'd be glad to help debug anything else that's missing.

[mrsimpson]

@nessur great to get to know you and thanks for the offered help.

I'll try to document what I understood and where things go wrong

Prerequisites

As per my understanding:

• Redis is only used in mds-cache
• a pnpm build was not necessary in order to run tests, since ts-jest transpiles the files under test on-the-fly
  Am I wrong with either of that?

I am running a (compose-based) redis and postgres locally.

Executing tests

Running
pnpm install && pnpm build && pnpm test on a832fdb results in the following errors (in the order of appearance)

1. Database mdstest does not exist

=> ✅ CREATE DATABASE mdstest; – well, that was easy ;

2. pretest script fails:

@mds-core/mds-attachment-service@0.5.24 pretest .../mds-core/packages/mds-attachment-service
pnpm typeorm -- schema:drop
[...]
Error during schema drop:
error: password authentication failed for user "oliverjaegle"

=> typeorm doesn't know about the environment and doesn't properly connect to the postgres => ❓ export $(cat ../../env | xargs) && pnpm typeorm -- schema:drop fixes that one

3. (Edge-case:) Sharp module not available in proper local architecture (M1 Mac, arm64)

@mds-core/mds-attachment-service@0.5.24 test .../mds-core/packages/mds-attachment-service
 DOTENV_CONFIG_PATH=../../.env jest --runInBand
FAIL  tests/index.spec.ts
 ● Test suite failed to run
Something went wrong installing the "sharp" module
 Cannot find module '../build/Release/sharp-darwin-arm64v8.node' from '../../node_modules/.pnpm/sharp@0.29.3/node_modules/sharp/lib/sharp.js'

=> skipping mds-attachment-service for now

4. Executing tests in packages fails

mds-collector-service is the next one to fail with the following logs

 FAIL  tests/service.spec.ts (6.194 s)1259","ISOTimestamp":"2022-03-23T22:58:11.259Z","namespace":"mds-service-helpers","message":"Terminating process @mds-core/mds-colle
  Collector Service
    ✓ Service Unavailable (15 ms)
    Repository Migrations
      ✓ Run Migrations (32 ms)
      ✓ Revert Migrations (219 ms)
    Service Methods
      ✓ Register Schema (OK) (89 ms)
      ✓ Register Schema (Error) (34 ms)
      ✓ Get Schema (OK) (21 ms)
      ✓ Get Schema (Invalid schema_id) (42 ms)
      ✕ Write Schema Messages (OK) (29 ms)
      ✓ Write Schema Messages (Invalid schema_id) (17 ms)
      ✓ Write Schema Messages (Invalid provider_id) (4 ms)
      ✓ Write Schema Messages (Invalid message) (9 ms)

  ● Collector Service › Service Methods › Write Schema Messages (OK)

    Error Writing Messages for Schema test
  [...the coverage report...]
Ran all test suites.
{"level":"info","timestamp":"1648076291260","ISOTimestamp":"2022-03-23T22:58:11.260Z","namespace":"mds-rpc-common","message":"Stopping RPC server listening for application/grpc-web+json requests"}
{"level":"info","timestamp":"1648076291260","ISOTimestamp":"2022-03-23T22:58:11.260Z","namespace":"mds-repository","message":"Terminating R/W repository: collector"}
{"level":"info","timestamp":"1648076291260","ISOTimestamp":"2022-03-23T22:58:11.260Z","namespace":"mds-repository","message":"Terminating R/O connection: collector-ro-78251964-cb7c-4008-afb1-6d129906a845"}
{"level":"info","timestamp":"1648076291260","ISOTimestamp":"2022-03-23T22:58:11.260Z","namespace":"mds-repository","message":"Terminating R/W connection: collector-rw-78251964-cb7c-4008-afb1-6d129906a845"}
/Users/oliverjaegle/projects/deutschebahn/fermata/mds-core/packages/mds-collector-service:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  @mds-core/mds-collector-service@0.4.24 test: `DOTENV_CONFIG_PATH=../../.env jest --runInBand`
 Exit status 1
 ELIFECYCLE  Command failed with exit code 1.
 ELIFECYCLE  Test failed. See above for more details.

=> I try to analyze this single problem by descending into its directory and calling pnpm test

 packages/mds-collector-service ‹a832fdbb●› 
╰─$ pnpm test

[...]
 ts-node ./node_modules/typeorm/cli.js "schema:drop"

 sh: ts-node: command not found

=> As far as I understood this is due to the sub-package not having dev-dependencies on their own, thus no local node_modules is available. The node_modules folder is not available unless I run all the tests from the workspace root.

In other packages, it's jest which cannot be located:

/packages/mds-geography-api ‹a832fdbb●› 
╰─$ pnpm test              

[...]

sh: jest: command not found

Those are the main issues, more to come ;)

[jschirrmacher]

Regarding 3, this problem occurs in our special environment because we use ignore-scripts setting in .npmrc to prevent scripts in some direct or indirect dependencies to run automatically. We normally solve that by adding npm rebuild <lib name> in a postinstall target in package.json, and will do that in our fork as well.

[nessur]

I don't have all the answers yet, but your DB config issue with (2. Pretest) is that PG_NAME is missing from your shell's ENV. My ~/.zshrc has:

export PG_HOST=localhost
export PG_NAME=mdstest
export PG_USER=mdsadmin
export PG_MIGRATIONS=true. (so that `pnpm start` in a package will also run migrations on boot )

where mdsadmin is a SUPERUSER account w/ no password. If your PG_USER value is empty, it'll use your default user oliverjaegle which has a password, so you need to add export PG_PASS=something as well.

[mrsimpson]

@nessur Thx, I'm well aware of that. And what you say actually confirms my guess: There is some external configuration (e. g. your zsh-environment) which I don't have.
Since we already configure the environment in .env, I think it should be respected in the pretest as well. I created #962 for that purpose.

[nessur]

Not a bad suggestion. I don't love relying both on mds-core/.env and exports in my .zshrc file.

Can you try removing ignore-scripts from your NPM config? It seems like that could mess up package.json scripts, as mentioned here: https://stackoverflow.com/questions/59471962/how-does-npm-behave-differently-with-ignore-scripts-set-to-true

[mrsimpson]

I removed the ignore-scripts (temporarily, we need this for security reasons) and re-triggered a pnpm clean.
The collector service tests still fail:

FAIL tests/service.spec.ts
│   Collector Service
│     ✓ Service Unavailable (11 ms)
│     Repository Migrations
│       ✓ Run Migrations (30 ms)
│       ✓ Revert Migrations (176 ms)
│     Service Methods
│       ✓ Register Schema (OK) (66 ms)
│       ✓ Register Schema (Error) (14 ms)
│       ✓ Get Schema (OK) (4 ms)
│       ✓ Get Schema (Invalid schema_id) (37 ms)
│       ✕ Write Schema Messages (OK) (18 ms)
│       ✓ Write Schema Messages (Invalid schema_id) (9 ms)
│       ✓ Write Schema Messages (Invalid provider_id) (4 ms)
│       ✓ Write Schema Messages (Invalid message) (4 ms)
│   ● Collector Service › Service Methods › Write Schema Messages (OK)
│     Error Writing Messages for Schema test

Remark: My containerized postgres does not allow for a passwordless login, but I use postgres/pwd in my .env-file and this seems to work fine in other tests, so I assume this is not related.

[jschirrmacher]

If we know, which scripts in dependencies we want to execute, it is still possible to do so by running npm rebuild <library name>, for example in package.json's postinstall script.

can-i-ignore-scripts Tool helps us to figure out, which scripts exist and which we want to be run.

[nessur]

I removed the ignore-scripts (temporarily, we need this for security reasons) and re-triggered a pnpm clean. The collector service tests still fail:

My concern was that you were unable to execute pnpm test within packages/mds-collector-service. Sounds like that's resolved now?

Jest isn't great about printing verbose errors, specifically it can hide any error-stacks that may be returned in an exception. Looking at the contents of 'Write Schema Messages (OK)', the only unique item that could fail here, but not cause other tests to also fail, is the check to instantiate a Kafka Producer.

If I try to make my tests fail this way, i can do KAFKA_HOST=xxyy pnpm test. This, however, produces a lot of hard to ignore errors on my test outputs, though it does produce the exact same list of successes/failures as you posted above.

Do you use an IDE that lets you set break points, run an interactive debugger? In VSCode, I can set a breakpoint inside an exception handler (like packages/mds-collector-service/service/producer.ts:155) to see what error details are being squelched.