As we convert parts of the codebase to TypeScript, we will want a way to
track progress. This commit adds a dashboard which displays all of the
files that we wish to convert to TypeScript and which files we've
already converted.
The list of all possible files to convert is predetermined by walking
the dependency graph of each entrypoint the build system uses to compile
the extension (the files that the entrypoint imports, the files that the
imports import, etc). The list should not need to be regenerated, but
you can do it by running:
yarn ts-migration:enumerate
The dashboard is implemented as a separate React app. The CircleCI
configuration has been updated so that when a new commit is pushed, the
React app is built and stored in the CircleCI artifacts. When a PR is
merged, the built files will be pushed to a separate repo whose sole
purpose is to serve the dashboard via GitHub Pages (this is the same
way that the Storybook works). All of the app code and script to build
the app are self-contained under
`development/ts-migration-dashboard`. To build this app yourself, you
can run:
yarn ts-migration:dashboard:build
or if you want to build automatically as you change files, run:
yarn ts-migration:dashboard:watch
Then open the following file in your browser (there is no server
component):
development/ts-migration-dashboard/build/index.html
Finally, although you shouldn't have to do this, to manually deploy the
dashboard once built, you can run:
git remote add ts-migration-dashboard git@github.com:MetaMask/metamask-extension-ts-migration-dashboard.git
yarn ts-migration:dashboard:deploy
We use the `rc` package to read the `.metamaskrc` configuration file,
which is in "ini" format. This package has been replaced by the `ini`
package.
The `rc` package was not actively maintained, and it has had recent
security vulnerabilities. But most importantly, the config object
returned by `rc` includes a bunch of extra information that made build
script validation [1] difficult to implement. Specifically, it made it
challenging to ensure no extra environment variables were present.
The `ini` package on the other hand is simple, well maintained, and
is simpler to use. This package doesn't add any extra properties to the
object it returns, making validation easy.
[1]: https://github.com/MetaMask/metamask-extension/issues/15003
This is mainly associated with an update in GridPlus SDK and enables
better strategies for fetching calldata decoder data.
`eth-lattice-keyring` changes:
GridPlus/eth-lattice-keyring@v0.7.3...v0.10.0
`gridplus-sdk` changes (which includes a codebase rewrite):
GridPlus/gridplus-sdk@v1.2.3...v2.2.2
The build script now uses `yargs` rather than `minimist`. The CLI is
now better documented, and we have additional validation for each
option.
A patch for `yargs` was required because it would blow up on the line
`Error.captureStackTrace`. For some reason when running under LavaMoat,
that property did not exist.
Closes#12766
* origin/develop: (131 commits)
Update `protobufjs` and remove obsolete advisory exclusion (#14841)
Include snap version in pill (#14803)
Update PULL_REQUEST_TEMPLATE.md (#14790)
fix: keystone transaction qrcode has no white spacing (#14798)
Snap notifications integration (#14605)
Upgrade @metamask/eth-ledger-bridge-keyring (#14799)
snaps-skunkworks@0.15.0 (#14772)
Fix proptype errors in network dropdown, tx list item details, and account details modal tests (#14747)
Ensure transaction type is correctly updated on edit (#14721)
Add fiat onboarding for AVAX and MATIC through Wyre (#14683)
Bump @metamask/contract-metadata from 1.33.0 to 1.35.0 (#14791)
Slight cleanup of constants/transactions, useTransactionDisplayData, and TransactionIcon (#14784)
Migrate the "estimateGas" API call to "getFees" for STX (#14767)
Ignore advisory GHSA-wm7h-9275-46v2 (#14789)
Adding flag for MV3 (#14762)
Add types to send state (#14740)
Remove site origin on snap install (#14752)
Update design tokens library from 1.5 to 1.6 WIP (#14732)
Enables the "Safe Transaction From" copy for safeTransferFrom transactions (#14769)
remove draft transaction (#14701)
...
* origin/master: (101 commits)
Updating changelog
Add token standard to custom token details (#14506)
Revert "Dark Mode: What's New Announcement (#14346)"
Ensure network name in confirm page container is defined (#14520)
Updating lavamoat policies
Fix the alerts toggles in settings (#14498)
Disable swaps whenever the environment is not development or testing, so that behaviour follows production for QA purposes (#14499)
[skip e2e] Updating changelog for v10.14.0 (#14487)
Version v10.14.0
Docs - segment metrics (#14435)
Add snaps view search (#14419)
Run main, flask and beta in sequence in generate-lavamoat-policies.sh (#14470)
Modify import SRP page (#14425)
Dark Mode: Implement Metrics (#14455)
HoldToRevealButton component (#13785)
e2e test import json file as import account strategy (#14449)
MetaMetrics: Identify 'number_of_tokens' user trait (#14427)
MetaMetrics: Identify 'nft_autodetection_enabled' & 'opensea_api_enabled' (#14367)
Swaps: Sort "token_from" dropdown tokens by their fiat value first and "token_to" by top tokens (#14436)
Update segment instantiation check. Only check if SEGMENT_WRITE_KEY exists (#14407)
...
* Create `.zip` files deterministically
Our build system now creates `.zip` archives deterministically.
Previously the `.zip` file would differ between builds even when the
files being archived were identical. This was because the order the
files were passed in was non-deterministic, and the `mtime` for each
file was different between builds.
The files are now sorted before being zipped, and the `mtime` for each
file has been set to the unix epoch.
* Update lavamoat build policy
* Create `.zip` files deterministically
Our build system now creates `.zip` archives deterministically.
Previously the `.zip` file would differ between builds even when the
files being archived were identical. This was because the order the
files were passed in was non-deterministic, and the `mtime` for each
file was different between builds.
The files are now sorted before being zipped, and the `mtime` for each
file has been set to the unix epoch.
* Update lavamoat build policy
* lavamoat - apply lavamoat protections to popup and notification
* build - enable lavamoat for home
* lavamoat - add missing ui overrides for react family
* deps/patches - patch zxcvbn for ses compat
We currently store the JSON-RPC request and response objects in the permission activity log. The utility of doing this was always rather dubious, but never problematic. Until now.
In Flask, as the restricted methods have expanded in number, user secrets may be included on JSON-RPC message objects. This PR removes these properties from the permission activity log, and adds a migration which does the same to existing log objects. We don't interact with the log objects anywhere in our codebase, but we don't want unexpected properties to cause errors in the future should any log objects be retained.
This PR also updates relevant tests and test data. It makes a minor functional change to how a request is designated as a success or failure, but this should not change any behavior in practice.
* origin/develop: (210 commits)
Dark Mode: Remove unwanted background for price quote (#14278)
Dark Mode: Fix colors in toggle button (#14280)
Ensure proper color for swaps edit link (#14273)
Dark Mode: Ensure actionable message button colors are the same color as previously (#14271)
Add token standard to Token Added event. (#14253)
Token Aggregators component for Tokens Detected page (#14157)
Ensure Metafox follows cursor on Fetching quotes screen (#14261)
TransactionsControllerTest: catch uncaught errors (#14196)
GasModalPageContainer story: convert knobs and actions to controls / args (#13516)
Show STX switch for wrapping / unwrapping (#14225)
Change over ImportToken stories to use controls instead of knobs, update props in stories (#14246)
Change over FeeCard stories to use controls instead of knobs, update props in stories (#13766)
Update What's new screen with Token Detection information (#14124)
Improvements for multi-layer fee UX (#13547)
metaMetricsEvent -> trackEvent (#14249)
E2e dapp interactions (#14149)
failing contract interaction e2e (#14227)
Removed metrics event (#14042)
Add TypeScript to the build system (#13489)
Build user traits object when metamask state changes (#14192)
...
An array of integers is now used to represent the SRP in three cases:
* In the import wallet flow, the UI uses it to pass the user-provided
SRP to the background (which converts the array to a buffer).
* In the create wallet flow, the UI uses it to retrieve the generated
SRP from the background.
* When persisting the wallet to state, the background uses it to
serialize the SRP.
Co-authored-by: Elliot Winkler <elliot.winkler@gmail.com>
This commit modifies the build system so that TypeScript files can be
transpiled into ES5 just like JavaScript files.
Note that this commit does NOT change the build system to run TypeScript
files through the TypeScript compiler. In other words, no files will be
type-checked at the build stage, as we expect type-checking to be
handled elsewhere (live, via your editor integration with `tsserver`,
and before a PR is merged, via `yarn lint`). Rather, we merely instruct
Babel to strip TypeScript-specific syntax from any files that have it,
as if those files had been written using JavaScript syntax alone.
Why take this approach? Because it prevents the build process from being
negatively impacted with respect to performance (as TypeScript takes a
significant amount of time to run).
It's worth noting the downside of this approach: because we aren't
running files through TypeScript, but relying on Babel's [TypeScript
transform][1] to identify TypeScript syntax, this transform has to keep
up with any syntax changes that TypeScript adds in the future. In fact
there are a few syntactical forms that Babel already does not recognize.
These forms are rare or are deprecated by TypeScript, so I don't
consider them to be a blocker, but it's worth noting just in case it
comes up later. Also, any settings we place in `tsconfig.json` will be
completely ignored by Babel. Again, this isn't a blocker because there
are some analogs for the most important settings reflected in the
options we can pass to the transform. These and other caveats are
detailed in the [documentation for the transform][2].
[1]: https://babeljs.io/docs/en/babel-plugin-transform-typescript
[2]: https://babeljs.io/docs/en/babel-plugin-transform-typescript#caveats
There were several issues related to a retry mechanism. The latest keyring
offers a significant speed and UX enhancement relative to the previous release.
For full details, see:
GridPlus/eth-lattice-keyring@v0.5.0...v0.6.1
There were several issues related to a retry mechanism. The latest keyring
offers a significant speed and UX enhancement relative to the previous release.
For full details, see:
GridPlus/eth-lattice-keyring@v0.5.0...v0.6.1
This commit allows developers to write TypeScript files and lint them
(either via a language server in their editor of choice or through the
`yarn lint` command).
The new TypeScript configuration as well as the updated ESLint
configuration not only includes support for parsing TypeScript files,
but also provides some compatibility between JavaScript and TypeScript.
That is, it makes it possible for a TypeScript file that imports a
JavaScript file or a JavaScript file that imports a TypeScript file to
be linted.
Note that this commit does not integrate TypeScript into the build
system yet, so we cannot start converting files to TypeScript and
pushing them to the repo until that final step is complete.
* deprecate extensionizer for webextension-polyfill
* fix tests
* remove extensionizer
* fix browser windows api calls
* fix broken on firefox
* fix getAcceptLanguages call
* update more browser apis that are now promisified
* remove unnecessary console error ignoring in e2e tests
An array of integers is now used to represent the SRP in three cases:
* In the import wallet flow, the UI uses it to pass the user-provided
SRP to the background (which converts the array to a buffer).
* In the create wallet flow, the UI uses it to retrieve the generated
SRP from the background.
* When persisting the wallet to state, the background uses it to
serialize the SRP.
Co-authored-by: Elliot Winkler <elliot.winkler@gmail.com>
We would like to insert TypeScript into the ESLint configuration, and
because of the way that the current config is organized, that is not
easy to do.
Most files are assumed to be files that are suited for running in a
browser context. This isn't correct, as we should expect most files to
work in a Node context instead. This is because all browser-based files
will be run through a transpiler that is able to make use of
Node-specific variables anyway.
There are a couple of important ways we can categories files which our
ESLint config should be capable of handling well:
* Is the file a script or a module? In other words, does the file run
procedurally or is the file intended to be brought into an existing
file?
* If the file is a module, does it use the CommonJS syntax (`require()`)
or does it use the ES syntax (`import`/`export`)?
When we introduce TypeScript, this set of questions will become:
* Is the file a script or a module?
* If the file is a module, is it a JavaScript module or a TypeScript
module?
* If the file is a JavaScript module, does it use the CommonJS syntax
(`require()`) or does it use the ES syntax (`import`/`export`)?
To represent these divisions, this commit removes global rules — so now
all of the rules are kept in `overrides` for explicitness — and sets up
rules for CommonJS- and ES-module-compatible files that intentionally do
not overlap with each other. This way TypeScript (which has its own set
of rules independent from JavaScript and therefore shouldn't overlap
with the other rules either) can be easily added later.
Finally, this commit splits up the ESLint config into separate files and
adds documentation to each section. This way sets of rules which are
connected to a particular plugin (`jsdoc`, `@babel`, etc.) can be easily
understood instead of being obscured.
This PR adds `snaps` under Flask build flags to the extension. This branch is mostly equivalent to the current production version of Flask, excepting some bug fixes and tweaks.
Closes#11626
ESLint rules have been added to enforce our JSDoc conventions. These
rules were introduced by updating `@metamask/eslint-config` to v9.
Some of the rules have been disabled because the effort to fix all lint
errors was too high. It might be easiest to enable these rules one
directory at a time, or one rule at a time.
Most of the changes in this PR were a result of running
`yarn lint:fix`. There were a handful of manual changes that seemed
obvious and simple to make. Anything beyond that and the rule was left
disabled.
The ESLint config has been updated to v8. The breaking changes are:
* The Prettier rule `quoteProps` has been changed from `consistent` to
`as-needed`, meaning that if one key requires quoting, only that key is
quoted rather than all keys.
* The ESLint rule `no-shadow` has been made more strict. It now
prevents globals from being shadowed as well.
Most of these changes were applied with `yarn lint:fix`. Only the
shadowing changes required manual fixing (shadowing variable names were
either replaced with destructuring or renamed).
The dependency `globalThis` was added to the list of dynamic
dependencies in the build system, where it should have been already.
This was causing `depcheck` to fail because the new lint rules required
removing the one place where `globalThis` had been erroneously imported
previously.
A rule requiring a newline between multiline blocks and expressions has
been disabled temporarily to make this PR smaller and to avoid
introducing conflicts with other PRs.
The `selectHooks` function has been replaced with the equivalent
function from the `@metamask/rpc-methods` package, which is
functionally equivalent.
The function was included in that package so that it could be used
elsewhere in the `snaps-skunkworks` repo. Eventually the goal is to
migrate much of our RPC logic into this package so that it can be
shared across products, and by our libraries as needed.
# Permission System 2.0
## Background
This PR migrates the extension permission system to [the new `PermissionController`](https://github.com/MetaMask/snaps-skunkworks/tree/main/packages/controllers/src/permissions).
The original permission system, based on [`rpc-cap`](https://github.com/MetaMask/rpc-cap), introduced [`ZCAP-LD`](https://w3c-ccg.github.io/zcap-ld/)-like permissions to our JSON-RPC stack.
We used it to [implement](https://github.com/MetaMask/metamask-extension/pull/7004) what we called "LoginPerSite" in [version 7.7.0](https://github.com/MetaMask/metamask-extension/releases/tag/v7.7.0) of the extension, which enabled the user to choose which accounts, if any, should be exposed to each dapp.
While that was a worthwhile feature in and of itself, we wanted a permission _system_ in order to enable everything we are going to with Snaps.
Unfortunately, the original permission system was difficult to use, and necessitated the creation of the original `PermissionsController` (note the "s"), which was more or less a wrapper for `rpc-cap`.
With this PR, we shake off the yoke of the original permission system, in favor of the modular, self-contained, ergonomic, and more mature permission system 2.0.
Note that [the `PermissionController` readme](https://github.com/MetaMask/snaps-skunkworks/tree/main/packages/controllers/src/permissions/README.md) explains how the new permission system works.
The `PermissionController` and `SubjectMetadataController` are currently shipped via `@metamask/snap-controllers`. This is a temporary state of affairs, and we'll move them to `@metamask/controllers` once they've landed in prod.
## Changes in Detail
First, the changes in this PR are not as big as they seem. Roughly half of the additions in this PR are fixtures in the test for the new migration (number 68), and a significant portion of the remaining ~2500 lines are due to find-and-replace changes in other test fixtures and UI files.
- The extension `PermissionsController` has been deleted, and completely replaced with the new `PermissionController` from [`@metamask/snap-controllers`](https://www.npmjs.com/package/@metamask/snap-controllers).
- The original `PermissionsController` "domain metadata" functionality is now managed by the new `SubjectMetadataController`, also from [`@metamask/snap-controllers`](https://www.npmjs.com/package/@metamask/snap-controllers).
- The permission activity and history log controller has been renamed `PermissionLogController` and has its own top-level state key, but is otherwise functionally equivalent to the existing implementation.
- Migration number 68 has been added to account for the new state changes.
- The tests in `app/scripts/controllers/permissions` have been migrated from `mocha` to `jest`.
Reviewers should focus their attention on the following files:
- `app/scripts/`
- `metamask-controller.js`
- This is where most of the integration work for the new `PermissionController` occurs.
Some functions that were internal to the original controller were moved here.
- `controllers/permissions/`
- `selectors.js`
- These selectors are for `ControllerMessenger` selector subscriptions. The actual subscriptions occur in `metamask-controller.js`. See the `ControllerMessenger` implementation for details.
- `specifications.js`
- The caveat and permission specifications are required by the new `PermissionController`, and are used to specify the `eth_accounts` permission and its JSON-RPC method implementation.
See the `PermissionController` readme for details.
- `migrations/068.js`
- The new state should be cross-referenced with the controllers that manage it.
The accompanying tests should also be thoroughly reviewed.
Some files may appear new but have just moved and/or been renamed:
- `app/scripts/lib/rpc-method-middleware/handlers/request-accounts.js`
- This was previously implemented in `controllers/permissions/permissionsMethodMiddleware.js`.
- `test/mocks/permissions.js`
- A truncated version of `test/mocks/permission-controller.js`.
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
* support qr based signer
* add CSP for fire fox
* get QR Hardware wallet name from device
* fix qrHardware state missing in runtime
* support qr based signer sign transaction
* refine Request Signature modal ui
* remove feature toggle
* refine ui
* fix notification is closing even there is a pending qr hardware transaction
* add chinese translation, refine ui, fix qr process was breaking in some case
* support import accounts by pubkeys
* refine qr-based wallet ui and fix bugs
* update @keystonehq/metamask-airgapped-keyring to fix that the signing hd path was inconsistent in some edge case
* fix: avoid unnecessay navigation, fix ci
* refactor qr-hardware-popover with @zxing/browser
* update lavamoat policy, remove firefox CSP
* refine qr reader ui, ignore unnecessary warning display
* code refactor, use async functions insteads promise
Co-authored-by: Soralit <soralitria@gmail.com>
* Add CollectiblesController
* bump controllers version
* add CollectibleDetectionController
* adapt to ERC1155 support changes in CollectiblesController
* update @metamask/controllers to v20.0.0
* update lavamoat policy files
* put collectibleDetectionController instantiation behind feature flag
This PR adds one LavaMoat background script policy or each build type. It also renames the build system policy directory from `node` to `build-system` to make its purpose more clear. Each build type has the original `policy-override.json` for `main` builds. The `.prettierignore` file has been updated to match the locations of the new auto-generated policy files.
We need to maintain separate policies for each build type because each type will produce different bundles with different internal and external modules.
Co-authored-by: Mark Stacey <markjstacey@gmail.com>
The LavaMoat policy has been updated in accordance with the recent
update to `eth-json-rpc-middleware` in #10738. These changes were
generated with `yarn lavamoat:auto`.
* Support for Layer 2 networks with transaction fees on both layers
* Use variable name in transaction-breakdown
* Add comment on code source to ui/helpers/utils/optimism/fetchEstimatedL1Fee.js
* Fix unit tests
* Ensure values passed to are defined
* Fix activity log
This PR enables the exclusion of JavaScript and JSON source by `buildType`, and enables the running of `eslint` under LavaMoat. 80-90% of the changes in this PR are `.patch` files and LavaMoat policy additions.
The file exclusion is designed to work in conjunction with our code fencing. If you forget to fence an import statement of an excluded file, the application will now error on boot. **This PR commits us to a particular naming convention for files intended only for certain builds.** Continue reading for details.
### Code Fencing and ESLint
When a file is modified by the code fencing transform, we run ESLint on it to ensure that we fail early for syntax-related issues. This PR adds the first code fences that will be actually be removed in production builds. As a consequence, this was also the first time we attempted to run ESLint under LavaMoat. Making that work required a lot of manual labor because of ESLint's use of dynamic imports, but the manual changes necessary were ultimately quite minor.
### File Exclusion
For all builds, any file in `app/`, `shared/` or `ui/` in a sub-directory matching `**/${otherBuildType}/**` (where `otherBuildType` is any build type except `main`) will be added to the list of excluded files, regardless of its file extension. For example, if we want to add one or more pages to the UI settings in Flask, we'd create the folder `ui/pages/settings/flask`, add any necessary files or sub-folders there, and fence the import statements for anything in that folder. If we wanted the same thing for Beta, we would name the directory `ui/pages/settings/beta`.
As it happens, we already organize some of our source files in this way, namely the logo JSON for Beta and Flask builds. See `ui/helpers/utils/build-types.js` to see how this works in practice.
Because the list of ignored filed is only passed to `browserify.exclude()`, any files not bundled by `browserify` will be ignored. For our purposes, this is mostly relevant for `.scss`. Since we don't have anything like code fencing for SCSS, we'll have to consider how to handle our styles separately.
On an M1 Mac, when running `yarn start`, CPU can spike to 100% CPU, and
sometimes a bunch of `mdworker` instances will get spawned. This seems
to be caused by the file-watching mechanism used in dev to automatically
regenerate the build when something is changed. More specifically, we
are using an older version of `watchify`, which uses an older version of
`chokidar`, which is the package that actually does the watching. v4.0.0
of `watchify` upgrades `chokidar` to v3.x ([1]), which comes with
"massive CPU & RAM consumption improvements" ([2]). After the upgrade,
CPU usage decreases to 20-40%.
[1]: https://github.com/browserify/watchify/blob/master/CHANGELOG.md#400
[2]: https://github.com/paulmillr/chokidar/releases/tag/3.0.0
The main `version` field in `package.json` will now include the beta
version (if present) rather than it being passed in via the CLI when
building. The `version` field is now a fully SemVer-compatible version,
with the added restriction that any prerelease portion of the version
must match the format `<build type>.<build version>`.
This brings the build in-line with the future release process we will
be using for the beta version. The plan is for each future release to
enter a "beta phase" where the version would get updated to reflect
that it's a beta, and we would increment this beta version over time as
we update the beta. The manifest gives us a place to store this beta
version. It was also important to replace the automatic minor bump
logic that was being used previously, because the version in beta might
not be a minor bump.
Additionally, the filename logic used for beta builds was updated to
be generic across all build types rather than beta-specific. This will
be useful for Flask builds in the future.
* lavamoat - add lavamoat to webapp background
* test:e2e - add delay to resolve failure
* test:e2e - add delay to resolve failure
* build - add a switch for applying lavamoat, currently off for all
* test/e2e - remove delays added for lavamoat
* Revert "test/e2e - remove delays added for lavamoat"
This reverts commit 79c3479f15c072ed362ba1d4f1af41ea11a17d63.
The warnings about use of the unsafe Buffer constructor have been
addressed by package updates and patches.
The updates were:
* `gulp-sourcemaps` was updated from v2 to v3, and was patched to
replace remaining uses of the `Buffer` constructor
* Upstream PR: https://github.com/gulp-sourcemaps/gulp-sourcemaps/pull/388
* The transitive dependency `yazl` was updated from v2.4.3 to v2.5.1
in the lockfile.
* The abandoned packages `combine-source-map` and `inline-source-map`
were patched.
A CI job has been added to ensure the `allow-scripts` config and the
LavaMoat auto-generated policy is up-to-date. This will only run on
release branches and the `master` branch, because it's too difficult a
requirement to meet for each PR for contributors on macOS, due to
differences in the dependency graph caused by optional dependencies.
The `allow-scripts` and LavaMoat policy have both been updated using
`yarn allow-scripts auto` and `yarn lavamoat:auto`.
