The e2e test script now accepts a `--retry-until-failure` flag that can
be useful for locally reproducing intermittent failures.
Normally the `retries` option will only retry upon failure. But if
you want the test to fail to inspect the failure state, you want it to
keep running until the first failure then stop. This flag accomplishes
that, reversing the retry conditions.
ESLint rules have been added to enforce our JSDoc conventions. These
rules were introduced by updating `@metamask/eslint-config` to v9.
Some of the rules have been disabled because the effort to fix all lint
errors was too high. It might be easiest to enable these rules one
directory at a time, or one rule at a time.
Most of the changes in this PR were a result of running
`yarn lint:fix`. There were a handful of manual changes that seemed
obvious and simple to make. Anything beyond that and the rule was left
disabled.
There are a few issues encountered when running `yarn setup` on new
Apple Silicon (aka M1, aka arm64) Macs:
* The script halts when attempting to run the install step for
the `chromedriver` package with the message "Only Mac 64 bits
supported". This is somewhat misleading as it seems to indicate that
chromedriver can only be installed on a 64-bit Mac. However, what I
think is happening is that the installation script for `chromedriver`
is not able to detect that an arm64 CPU *is* a 64-bit CPU. After
looking through the `chromedriver` repo, it appears that 87.0.1 is the
first version that adds a proper check ([1]).
Note that upgrading chromedriver caused the Chrome-specific tests to
fail intermittently on CI. I was not able to 100% work out the reason
for this, but ensuring that X (which provides a way for Chrome to run
in a GUI setting from the command line) is available seems to fix
these issues.
* The script also halts when attempting to run the install step for
the `electron` package. This happens because for the version of
`electron` we are using (9.4.2), there is no available binary for
arm64. It appears that Electron 11.x was the first version to support
arm64 Macs ([2]). This is a bit trickier to resolve because we don't
explicitly rely on `electron` — that's brought in by `react-devtools`.
The first version of `react-devtools` that relies on `electron` 11.x
is 4.11.0 ([3]).
[1]: 469dd0a6ee
[2]: https://www.electronjs.org/blog/apple-silicon
[3]: https://github.com/facebook/react/blob/main/packages/react-devtools/CHANGELOG.md#4110-april-9-2021
The benchmark script can now be set to retry upon failure, like the E2E
tests do. The default is zero, just as with the E2E tests. A retry of 2
has been set in CI to match the E2E tests as well.
The `retry` module had to be adjusted to throw an error in the case of
failure. Previously it just set the exit code, but that only worked
because it was the last thing called before the process ended. That is
no longer the case.
This script makes it easier to run an individual E2E test. In the past
I've run individual scripts by editing `run-all.sh` manually, but now
that can be done more easily with this script. It also allows setting
the number of retries to use and the browser to use from the CLI.
This script has been added as an npm script as well, called
'test:e2e:single'.
The `run-all.sh` script was rewritten in JavaScript to make it easier
to pass through a `--retries` argument.
The default number of retries has been set to zero to make local
testing easier. It has been set to 2 on CI.
This was mainly done to consolidate the code used to run an E2E test in
one place, to make later improvements easier.
