From 6268272c834250e9cc6eb92bc1d7d6293fe5980d Mon Sep 17 00:00:00 2001 From: Dan Finlay Date: Thu, 14 Sep 2017 13:31:10 -0700 Subject: [PATCH 1/7] Add guide to porting to new platforms Adds a new guide to porting MetaMask to new platforms. Intended for all those devs asking us how to make a mobile MetaMask. --- README.md | 1 + docs/porting_to_new_environment.md | 65 ++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) create mode 100644 docs/porting_to_new_environment.md diff --git a/README.md b/README.md index 075db79c2..b549ade08 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,7 @@ To write tests that will be run in the browser using QUnit, add your test files - [How to live reload on local dependency changes](./docs/developing-on-deps.md) - [How to add new networks to the Provider Menu](./docs/adding-new-networks.md) - [How to manage notices that appear when the app starts up](./docs/notices.md) +- [How to port MetaMask to a new platform](./docs/porting_to_new_environment.md) - [How to generate a visualization of this repository's development](./docs/development-visualization.md) [1]: http://www.nomnoml.com/#view/%5B%3Cactor%3Euser%5D%0A%0A%5Bmetamask-ui%7C%0A%20%20%20%5Btools%7C%0A%20%20%20%20%20react%0A%20%20%20%20%20redux%0A%20%20%20%20%20thunk%0A%20%20%20%20%20ethUtils%0A%20%20%20%20%20jazzicon%0A%20%20%20%5D%0A%20%20%20%5Bcomponents%7C%0A%20%20%20%20%20app%0A%20%20%20%20%20account-detail%0A%20%20%20%20%20accounts%0A%20%20%20%20%20locked-screen%0A%20%20%20%20%20restore-vault%0A%20%20%20%20%20identicon%0A%20%20%20%20%20config%0A%20%20%20%20%20info%0A%20%20%20%5D%0A%20%20%20%5Breducers%7C%0A%20%20%20%20%20app%0A%20%20%20%20%20metamask%0A%20%20%20%20%20identities%0A%20%20%20%5D%0A%20%20%20%5Bactions%7C%0A%20%20%20%20%20%5BaccountManager%5D%0A%20%20%20%5D%0A%20%20%20%5Bcomponents%5D%3A-%3E%5Bactions%5D%0A%20%20%20%5Bactions%5D%3A-%3E%5Breducers%5D%0A%20%20%20%5Breducers%5D%3A-%3E%5Bcomponents%5D%0A%5D%0A%0A%5Bweb%20dapp%7C%0A%20%20%5Bui%20code%5D%0A%20%20%5Bweb3%5D%0A%20%20%5Bmetamask-inpage%5D%0A%20%20%0A%20%20%5B%3Cactor%3Eui%20developer%5D%0A%20%20%5Bui%20developer%5D-%3E%5Bui%20code%5D%0A%20%20%5Bui%20code%5D%3C-%3E%5Bweb3%5D%0A%20%20%5Bweb3%5D%3C-%3E%5Bmetamask-inpage%5D%0A%5D%0A%0A%5Bmetamask-background%7C%0A%20%20%5Bprovider-engine%5D%0A%20%20%5Bhooked%20wallet%20subprovider%5D%0A%20%20%5Bid%20store%5D%0A%20%20%0A%20%20%5Bprovider-engine%5D%3C-%3E%5Bhooked%20wallet%20subprovider%5D%0A%20%20%5Bhooked%20wallet%20subprovider%5D%3C-%3E%5Bid%20store%5D%0A%20%20%5Bconfig%20manager%7C%0A%20%20%20%20%5Brpc%20configuration%5D%0A%20%20%20%20%5Bencrypted%20keys%5D%0A%20%20%20%20%5Bwallet%20nicknames%5D%0A%20%20%5D%0A%20%20%0A%20%20%5Bprovider-engine%5D%3C-%5Bconfig%20manager%5D%0A%20%20%5Bid%20store%5D%3C-%3E%5Bconfig%20manager%5D%0A%5D%0A%0A%5Buser%5D%3C-%3E%5Bmetamask-ui%5D%0A%0A%5Buser%5D%3C%3A--%3A%3E%5Bweb%20dapp%5D%0A%0A%5Bmetamask-contentscript%7C%0A%20%20%5Bplugin%20restart%20detector%5D%0A%20%20%5Brpc%20passthrough%5D%0A%5D%0A%0A%5Brpc%20%7C%0A%20%20%5Bethereum%20blockchain%20%7C%0A%20%20%20%20%5Bcontracts%5D%0A%20%20%20%20%5Baccounts%5D%0A%20%20%5D%0A%5D%0A%0A%5Bweb%20dapp%5D%3C%3A--%3A%3E%5Bmetamask-contentscript%5D%0A%5Bmetamask-contentscript%5D%3C-%3E%5Bmetamask-background%5D%0A%5Bmetamask-background%5D%3C-%3E%5Bmetamask-ui%5D%0A%5Bmetamask-background%5D%3C-%3E%5Brpc%5D%0A diff --git a/docs/porting_to_new_environment.md b/docs/porting_to_new_environment.md new file mode 100644 index 000000000..85670efa7 --- /dev/null +++ b/docs/porting_to_new_environment.md @@ -0,0 +1,65 @@ +# Guide to Porting MetaMask to a New Environment + +MetaMask has been under continuous development for nearly two years now, and we’ve gradually discovered some very useful abstractions, that have allowed us to grow more easily. A couple of those layers together allow MetaMask to be ported to new environments and contexts increasingly easily. + +### The MetaMask Controller + +The core functionality of MetaMask all lives in what we call [The MetaMask Controller](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js). Our goal for this file is for it to eventually be its own javascript module that can be imported into any JS-compatible context, allowing it to fully manage an app's relationship to Ethereum. + +The MM Controller exposes most of its functionality via two methods: + +- [metamask.getState()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L241) - This method returns a javascript object representing the current MetaMask state. This includes things like known accounts, sent transactions, current exchange rates, and more! The controller is also an event emitter, so you can subscribe to state updates via `metamask.on('update', handleStateUpdate)`. State examples available [here](https://github.com/MetaMask/metamask-extension/tree/master/development/states) under the `metamask` key. (Warning: some are outdated) +- [metamask.getApi()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L274-L335) - Returns a JavaScript object filled with callback functions representing every operation our user interface ever performs. Everything from creating new accounts, changing the current network, to sending a transaction, is provided via these API methods. We export this external API on an object because it allows us to easily expose this API over a port using [dnode](https://www.npmjs.com/package/dnode), which is how our WebExtension's UI works! + +### The UI + +The MetaMask UI is essentially just a website that can be configured by passing it the API and state subscriptions from above. Anyone could make a UI that consumes these, effectively reskinning MetaMask. + +You can see this in action in our file [ui/index.js](https://github.com/MetaMask/metamask-extension/blob/master/ui/index.js). There you can see an argument being passed in named `accountManager`, which is essentially a MetaMask controller (forgive its really outdated parameter name!). With access to that object, the UI is able to initialize a whole React/Redux app that relies on this API for its account/blockchain-related/persistent states. + +## Putting it Together + +As an example, a WebExtension is always defined by a `manifest.json` file. [In ours](https://github.com/MetaMask/metamask-extension/blob/master/app/manifest.json#L31), you can see that [background.js](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/background.js) is defined as a script to run in the background, and this is the file that we use to initialize the MetaMask controller. + +In that file, there's a lot going on, so it's maybe worth focusing on our MetaMask controller constructor to start. It looks something like this: + +```javascript +const controller = new MetamaskController({ + // User confirmation callbacks: + showUnconfirmedMessage: triggerUi, + unlockAccountMessage: triggerUi, + showUnapprovedTx: triggerUi, + // initial state + initState, + // platform specific api + platform, +}) +``` +Since `background.js` is essentially the Extension setup file, we can see it doing all the things specific to the extension platform: +- Defining how to open the UI for new messages, transactions, and even requests to unlock (reveal to the site) their account. +- Provide the instance's initial state, leaving MetaMask persistence to the platform. +- Providing a `platform` object. This is becoming our catch-all adapter for platforms to define a few other platform-variant features we require, like opening a web link. (Soon we will be moving encryption out here too, since our browser-encryption isn't portable enough!) + +## Ports, streams, and Web3! + +Everything so far has been enough to create a MetaMask wallet on virtually any platform that runs JS, but MetaMask's most unique feature isn't being a wallet, it's providing an Ethereum-enabled JavaScript context to websites. + +MetaMask has two kinds of [duplex stream APIs](https://github.com/substack/stream-handbook#duplex) that it exposes: +- [metamask.setupTrustedCommunication(connectionStream, originDomain)](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L352) - This stream is used to connect the user interface over a remote port, and may not be necessary for contexts where the interface and the metamask-controller share a process. +- [metamask.setupUntrustedCommunication(connectionStream, originDomain)](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L337) - This method is used to connect a new web site's web3 API to MetaMask's blockchain connection. Additionally, the `originDomain` is used to block detected phishing sites. + +### Web3 as a Stream + +If you are making a MetaMask-powered browser for a new platform, one of the trickiest tasks will be injecting the Web3 API into websites that are visited. On WebExtensions, we actually have to pipe data through a total of three JS contexts just to let sites talk to our background process (site -> contentscript -> background). + +To make this as easy as possible, we use one of our favorite internal tools, [web3-provider-engine](https://www.npmjs.com/package/web3-provider-engine) to construct a custom web3 provider object whose source of truth is a stream that we connect to remotely. + +To see how we do that, you can refer to the [inpage script](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/inpage.js) that we inject into every website. There you can see it creates a multiplex stream to the background, and uses it to initialize what we call the [inpage-provider](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/lib/inpage-provider.js), which you can see stubs a few methods out, but mostly just passes calls to `sendAsync` through the stream it's passed! That's really all the magic that's needed to create a web3-like API in a remote context, once you have a stream to MetaMask available. + +In `inpage.js` you can see we create a `PortStream`, that's just a class we use to wrap WebExtension ports as streams, so we can reuse our favorite stream abstraction over the more irregular API surface of the WebExtension. In a new platform, you will probably need to construct this stream differently. The key is that you need to construct a stream that talks from the site context to the background. Once you have that set up, it works like magic! + +If streams seem new and confusing to you, that's ok, they can seem strange at first. To help learn them, we highly recommend reading Substack's [Stream Handbook](https://github.com/substack/stream-handbook), or going through NodeSchool's interactive command-line class [Stream Adventure](https://github.com/workshopper/stream-adventure), also maintained by Substack. + +## Conclusion + +I hope this has been helpful to you! If you have any other questionsm, or points you think need clarification in this guide, please [open an issue on our GitHub](https://github.com/MetaMask/metamask-plugin/issues/new)! \ No newline at end of file From 8a874824a8e8dc5e16289f7753b13f96497379cd Mon Sep 17 00:00:00 2001 From: Dan Finlay Date: Thu, 21 Sep 2017 11:45:33 -0700 Subject: [PATCH 2/7] Version 3.10.3 --- CHANGELOG.md | 2 ++ app/manifest.json | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ff062cf8..e692c58dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,8 @@ ## Current Master +## 3.10.3 2017-9-21 + - Fix bug where metamask-dapp connections are lost on rpc error - Fix bug that would sometimes display transactions as failed that could be successfully mined. diff --git a/app/manifest.json b/app/manifest.json index 67fb543b9..fd07f15a9 100644 --- a/app/manifest.json +++ b/app/manifest.json @@ -1,7 +1,7 @@ { "name": "MetaMask", "short_name": "Metamask", - "version": "3.10.2", + "version": "3.10.3", "manifest_version": 2, "author": "https://metamask.io", "description": "Ethereum Browser Extension", From e9043f22dfa7856e3360b312ce480e71f36d9381 Mon Sep 17 00:00:00 2001 From: Dan Finlay Date: Thu, 21 Sep 2017 15:47:25 -0700 Subject: [PATCH 3/7] Allow custom encryptor to be passed to MetaMaskController and KeyringControllers. --- app/scripts/keyring-controller.js | 2 +- app/scripts/metamask-controller.js | 3 ++- test/unit/keyring-controller-test.js | 5 +---- 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/app/scripts/keyring-controller.js b/app/scripts/keyring-controller.js index fd57fac70..adfa4a813 100644 --- a/app/scripts/keyring-controller.js +++ b/app/scripts/keyring-controller.js @@ -36,7 +36,7 @@ class KeyringController extends EventEmitter { identities: {}, }) this.ethStore = opts.ethStore - this.encryptor = encryptor + this.encryptor = opts.encryptor || encryptor this.keyrings = [] this.getNetwork = opts.getNetwork } diff --git a/app/scripts/metamask-controller.js b/app/scripts/metamask-controller.js index fef16c3a9..42248827f 100644 --- a/app/scripts/metamask-controller.js +++ b/app/scripts/metamask-controller.js @@ -95,6 +95,7 @@ module.exports = class MetamaskController extends EventEmitter { initState: initState.KeyringController, ethStore: this.ethStore, getNetwork: this.networkController.getNetworkState.bind(this.networkController), + encryptor: opts.encryptor || undefined, }) this.keyringController.on('newAccount', (address) => { this.preferencesController.setSelectedAddress(address) @@ -674,4 +675,4 @@ module.exports = class MetamaskController extends EventEmitter { return Promise.resolve(rpcTarget) }) } -} \ No newline at end of file +} diff --git a/test/unit/keyring-controller-test.js b/test/unit/keyring-controller-test.js index 2d9a53723..8d0d75f12 100644 --- a/test/unit/keyring-controller-test.js +++ b/test/unit/keyring-controller-test.js @@ -27,12 +27,9 @@ describe('KeyringController', function () { ethStore: { addAccount (acct) { accounts.push(ethUtil.addHexPrefix(acct)) }, }, + encryptor: mockEncryptor, }) - // Stub out the browser crypto for a mock encryptor. - // Browser crypto is tested in the integration test suite. - keyringController.encryptor = mockEncryptor - keyringController.createNewVaultAndKeychain(password) .then(function (newState) { newState From b25d4d5cfbf9a49e2669188198abd7377e697206 Mon Sep 17 00:00:00 2001 From: Dan Finlay Date: Thu, 21 Sep 2017 15:56:44 -0700 Subject: [PATCH 4/7] Add platform docs including encryptor param --- docs/porting_to_new_environment.md | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/docs/porting_to_new_environment.md b/docs/porting_to_new_environment.md index 85670efa7..c6336b9f9 100644 --- a/docs/porting_to_new_environment.md +++ b/docs/porting_to_new_environment.md @@ -6,10 +6,31 @@ MetaMask has been under continuous development for nearly two years now, and we The core functionality of MetaMask all lives in what we call [The MetaMask Controller](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js). Our goal for this file is for it to eventually be its own javascript module that can be imported into any JS-compatible context, allowing it to fully manage an app's relationship to Ethereum. -The MM Controller exposes most of its functionality via two methods: +#### Constructor -- [metamask.getState()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L241) - This method returns a javascript object representing the current MetaMask state. This includes things like known accounts, sent transactions, current exchange rates, and more! The controller is also an event emitter, so you can subscribe to state updates via `metamask.on('update', handleStateUpdate)`. State examples available [here](https://github.com/MetaMask/metamask-extension/tree/master/development/states) under the `metamask` key. (Warning: some are outdated) -- [metamask.getApi()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L274-L335) - Returns a JavaScript object filled with callback functions representing every operation our user interface ever performs. Everything from creating new accounts, changing the current network, to sending a transaction, is provided via these API methods. We export this external API on an object because it allows us to easily expose this API over a port using [dnode](https://www.npmjs.com/package/dnode), which is how our WebExtension's UI works! +When calling `new MetaMask(opts)`, many platform-specific options are configured. The keys on `opts` are as follows: + +- initState: The last emitted state, used for restoring persistent state between sessions. +- platform: The `platform` object defines a variety of platform-specific functions, including opening the confirmation view, and customizing the encryption method. + +##### Platform Options + +The `platform` object has a variety of options: + +- reload (function) - Will be called when MetaMask would like to reload its own context. +- openWindow ({ url }) - Will be called when MetaMask would like to open a web page. It will be passed a single `options` object with a `url` key, with a string value. +- getVersion() - Should return the current MetaMask version, as described in the current `CHANGELOG.md` or `app/manifest.json`. +- encryptor - An object that includes two methods: + - encrypt(password, object) - returns a Promise of a string that is ready for storage. + - decrypt(password, encryptedString) - Accepts the encrypted output of `encrypt` and returns a Promise of a restored `object` as it was encrypted. + +#### [metamask.getState()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L241) + +This method returns a javascript object representing the current MetaMask state. This includes things like known accounts, sent transactions, current exchange rates, and more! The controller is also an event emitter, so you can subscribe to state updates via `metamask.on('update', handleStateUpdate)`. State examples available [here](https://github.com/MetaMask/metamask-extension/tree/master/development/states) under the `metamask` key. (Warning: some are outdated) + +#### [metamask.getApi()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L274-L335) + +Returns a JavaScript object filled with callback functions representing every operation our user interface ever performs. Everything from creating new accounts, changing the current network, to sending a transaction, is provided via these API methods. We export this external API on an object because it allows us to easily expose this API over a port using [dnode](https://www.npmjs.com/package/dnode), which is how our WebExtension's UI works! ### The UI @@ -62,4 +83,4 @@ If streams seem new and confusing to you, that's ok, they can seem strange at fi ## Conclusion -I hope this has been helpful to you! If you have any other questionsm, or points you think need clarification in this guide, please [open an issue on our GitHub](https://github.com/MetaMask/metamask-plugin/issues/new)! \ No newline at end of file +I hope this has been helpful to you! If you have any other questionsm, or points you think need clarification in this guide, please [open an issue on our GitHub](https://github.com/MetaMask/metamask-plugin/issues/new)! From e9b7fd901862f57a92614b19f7f2caa969d4a282 Mon Sep 17 00:00:00 2001 From: Dan Finlay Date: Fri, 22 Sep 2017 10:41:14 -0700 Subject: [PATCH 5/7] Patch security update --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index 0d02f1df5..bcfb6c1ac 100644 --- a/package.json +++ b/package.json @@ -129,7 +129,7 @@ "redux": "^3.0.5", "redux-logger": "^3.0.6", "redux-thunk": "^2.2.0", - "request-promise": "^4.1.1", + "request-promise": "^4.2.1", "sandwich-expando": "^1.0.5", "semaphore": "^1.0.5", "sw-stream": "^2.0.0", From 4c971ebfd1fad18368ec418c36c4d05a6bb37e6d Mon Sep 17 00:00:00 2001 From: Dan Finlay Date: Fri, 22 Sep 2017 13:25:08 -0700 Subject: [PATCH 6/7] Define encryptor in constructor params instead of platform object --- app/scripts/metamask-controller.js | 2 +- docs/porting_to_new_environment.md | 14 ++++++++++---- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/app/scripts/metamask-controller.js b/app/scripts/metamask-controller.js index e4fa3518f..42248827f 100644 --- a/app/scripts/metamask-controller.js +++ b/app/scripts/metamask-controller.js @@ -95,7 +95,7 @@ module.exports = class MetamaskController extends EventEmitter { initState: initState.KeyringController, ethStore: this.ethStore, getNetwork: this.networkController.getNetworkState.bind(this.networkController), - encryptor: opts.platform.encryptor || undefined, + encryptor: opts.encryptor || undefined, }) this.keyringController.on('newAccount', (address) => { this.preferencesController.setSelectedAddress(address) diff --git a/docs/porting_to_new_environment.md b/docs/porting_to_new_environment.md index c6336b9f9..729a28e5d 100644 --- a/docs/porting_to_new_environment.md +++ b/docs/porting_to_new_environment.md @@ -11,7 +11,16 @@ The core functionality of MetaMask all lives in what we call [The MetaMask Contr When calling `new MetaMask(opts)`, many platform-specific options are configured. The keys on `opts` are as follows: - initState: The last emitted state, used for restoring persistent state between sessions. -- platform: The `platform` object defines a variety of platform-specific functions, including opening the confirmation view, and customizing the encryption method. +- platform: The `platform` object defines a variety of platform-specific functions, including opening the confirmation view, and opening web sites. +- encryptor - An object that provides access to the desired encryption methods. + +##### Encryptor + +An object that provides two simple methods, which can encrypt in any format you prefer. This parameter is optional, and will default to the browser-native WebCrypto API. + +- encrypt(password, object) - returns a Promise of a string that is ready for storage. +- decrypt(password, encryptedString) - Accepts the encrypted output of `encrypt` and returns a Promise of a restored `object` as it was encrypted. + ##### Platform Options @@ -20,9 +29,6 @@ The `platform` object has a variety of options: - reload (function) - Will be called when MetaMask would like to reload its own context. - openWindow ({ url }) - Will be called when MetaMask would like to open a web page. It will be passed a single `options` object with a `url` key, with a string value. - getVersion() - Should return the current MetaMask version, as described in the current `CHANGELOG.md` or `app/manifest.json`. -- encryptor - An object that includes two methods: - - encrypt(password, object) - returns a Promise of a string that is ready for storage. - - decrypt(password, encryptedString) - Accepts the encrypted output of `encrypt` and returns a Promise of a restored `object` as it was encrypted. #### [metamask.getState()](https://github.com/MetaMask/metamask-extension/blob/master/app/scripts/metamask-controller.js#L241) From e52d52b22ed00b1a761bf57cff54c7276c6450a7 Mon Sep 17 00:00:00 2001 From: "greenkeeper[bot]" Date: Tue, 26 Sep 2017 08:22:48 +0000 Subject: [PATCH 7/7] chore(package): update sinon to version 4.0.0 --- package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package.json b/package.json index bcfb6c1ac..09490096b 100644 --- a/package.json +++ b/package.json @@ -196,7 +196,7 @@ "react-addons-test-utils": "^15.5.1", "react-test-renderer": "^15.5.4", "react-testutils-additions": "^15.2.0", - "sinon": "^3.2.0", + "sinon": "^4.0.0", "tape": "^4.5.1", "testem": "^1.10.3", "uglifyify": "^4.0.2",