GitBook: [#25] Deploying components section
This commit is contained in:
Akshay Patel
gitbook-bot
parent
c3c9b9df4d
commit
4605685b35
|
|
@ -26,12 +26,15 @@
|
|||
* [Writing Algorithms for Compute to Data](building-with-ocean/compute-to-data-algorithms.md)
|
||||
* [Set Up a Marketplace](building-with-ocean/marketplace.md)
|
||||
* [Polygon (ex Matic)](building-with-ocean/polygon-bridge.md)
|
||||
* [Projects using Ocean Protocol](building-with-ocean/projects-using-ocean.md)
|
||||
* [Compute-to-Data](building-with-ocean/compute-to-data/README.md)
|
||||
* [Architecture](building-with-ocean/compute-to-data/compute-to-data-architecture.md)
|
||||
* [Datasets & Algorithms](building-with-ocean/compute-to-data/compute-to-data-datasets-algorithms.md)
|
||||
* [Minikube Compute-to-Data Environment](building-with-ocean/compute-to-data/compute-to-data-minikube.md)
|
||||
* [Setting up private docker registry](building-with-ocean/compute-to-data/compute-to-data-docker-registry.md)
|
||||
* [Deploying components](building-with-ocean/deploying-components/README.md)
|
||||
* [Deploying Marketplace](building-with-ocean/deploying-components/deploying-marketplace.md)
|
||||
* [Deploying Aquarius](building-with-ocean/deploying-components/deploying-aquarius.md)
|
||||
* [Projects using Ocean Protocol](building-with-ocean/projects-using-ocean.md)
|
||||
* [Using Ocean Market](building-with-ocean/marketplace-introduction.md)
|
||||
* [Publish a data asset](building-with-ocean/marketplace-publish-data-asset.md)
|
||||
* [Download a data asset](building-with-ocean/marketplace-download-data-asset.md)
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ To publish assets on the Ocean Marketplace, publishers must provide a link(an UR
|
|||
|
||||
On Ocean Marketplace, a publisher must provide the link to the asset during publish step. Once the asset is published, this link cannot be changed. So, it is essential that the publisher correctly sets this field (shown in the below image).
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
### Hosting services
|
||||
|
||||
|
|
@ -27,7 +27,7 @@ Open https://drive.google.com and upload the file you want to publish on the Oce
|
|||
|
||||
The file URL will be of the form `https://drive.google.com/file/d/<FILE-ID>/view?usp=sharing`, where the `<FILE-ID>` is the unique alphanumeric string. Verify if the URL is correct by entering it in a browser and check if the file is downloaded.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
**Step 2 - Create a downloadable link**
|
||||
|
||||
|
|
@ -41,7 +41,7 @@ Note the `<FILE-ID>` from step 1 and create a URL as below.
|
|||
|
||||
After creating a downloadable file URL, fill the `File*` field with the downloadable URL created in step 2.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
_Note: Google Drive allows only shared files to be downloaded, as shown in the above steps. The above method does not work with the shared folder. As a workaround, publishers can upload a zip of a folder and upload it as a file._
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,2 @@
|
|||
# Deploying components
|
||||
|
||||
|
|
@ -0,0 +1,212 @@
|
|||
# Deploying Aquarius
|
||||
|
||||
### About Aquarius
|
||||
|
||||
Aquarius is an off-chain component with caches the asset metadata published on-chain. By deploying own Aquarius, developers can control which assets are visible in their marketplace. For example, having a custom Aquarius instance allows assets only from specific addresses to be visible on the marketplace. This tutorial will provide the steps to deploy Aquarius. Ocean Protocol provides Aquarius docker images which can be viewed [here](https://hub.docker.com/r/oceanprotocol/aquarius/tags). Visit [this](https://github.com/oceanprotocol/aquarius) page to view Aquarius source code.
|
||||
|
||||
Aquarius consists of two parts:\
|
||||
\- **API:** The Aquarius API offers a convenient way to access the medatata without scanning the chain yourself.\
|
||||
\- **Event monitor:** Aquarius continually monitors the chains for MetadataCreated and MetadataUpdated events, processes these events and adds them to the database.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
* Docker and Docker compose are installed. Click [here](https://docs.docker.com/engine/install/) to view guide on installing docker.
|
||||
* Ethereum API. Aquarius uses Ethereum api for monitoring on-chain events.\
|
||||
Choose any api provider of your choice. Some of the commonly used are:
|
||||
* [Infura](https://infura.io/)
|
||||
* [Alchemy](https://www.alchemy.com/)
|
||||
* [Moralis](https://moralis.io/)
|
||||
|
||||
### Create a working directory
|
||||
|
||||
```
|
||||
mkdir Aqaurius
|
||||
cd Aquarius
|
||||
```
|
||||
|
||||
### Create a \`.env\` file
|
||||
|
||||
Copy the below content into the \`.env\` file and edit the values as needed.
|
||||
|
||||
{% code title=".env" %}
|
||||
```
|
||||
# check the available versions: https://hub.docker.com/repository/docker/oceanprotocol/aquarius
|
||||
AQUARIUS_VERSION=latest
|
||||
ALLOWED_PUBLISHERS='[""]'
|
||||
# Elastic search credentials
|
||||
DB_USERNAME=username
|
||||
DB_PASSWORD=password
|
||||
|
||||
# Replace below value with the API provider of your choice
|
||||
EVENTS_RPC_RINKEBY=wss://rinkeby.infura.io/ws/v3/INFURA_ID
|
||||
EVENTS_RPC_POLYGON=wss://rinkeby.infura.io/ws/v3/INFURA_ID
|
||||
EVENTS_RPC_MAINNET=wss://rinkeby.infura.io/ws/v3/INFURA_ID
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
### Create docker-compose file
|
||||
|
||||
{% code title="docker-compose.yml" %}
|
||||
```yaml
|
||||
version: '3'
|
||||
services:
|
||||
elasticsearch:
|
||||
image: elasticsearch:6.8.17
|
||||
container_name: elasticsearch
|
||||
restart: on-failure
|
||||
environment:
|
||||
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
|
||||
MAX_MAP_COUNT: "64000"
|
||||
discovery.type: "single-node"
|
||||
volumes:
|
||||
- data:/usr/share/elasticsearch/data
|
||||
ports:
|
||||
- 9200:9200
|
||||
networks:
|
||||
- ocean_backend
|
||||
aquarius:
|
||||
image: oceanprotocol/aquarius:${AQUARIUS_VERSION}
|
||||
container_name: aquarius
|
||||
restart: on-failure
|
||||
ports:
|
||||
- 5000:5000
|
||||
networks:
|
||||
- ocean_backend
|
||||
depends_on:
|
||||
- elasticsearch
|
||||
environment:
|
||||
DB_MODULE: elasticsearch
|
||||
DB_HOSTNAME: elasticsearch
|
||||
DB_PORT: 9200
|
||||
DB_USERNAME: ${DB_USERNAME}
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
DB_NAME: aquarius
|
||||
DB_SCHEME: http
|
||||
DB_SSL : "false"
|
||||
LOG_LEVEL: "DEBUG"
|
||||
AQUARIUS_BIND_URL : "http://0.0.0.0:5000"
|
||||
AQUARIUS_WORKERS : "8"
|
||||
RUN_AQUARIUS_SERVER: "1"
|
||||
AQUARIUS_CONFIG_FILE: "config.ini"
|
||||
EVENTS_ALLOW: 0
|
||||
RUN_EVENTS_MONITOR: 0
|
||||
ALLOWED_PUBLISHERS: ${ALLOWED_PUBLISHERS}
|
||||
volumes:
|
||||
data:
|
||||
driver: local
|
||||
networks:
|
||||
ocean_backend:
|
||||
driver: bridge
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
### Create events monitor docker compose file
|
||||
|
||||
{% tabs %}
|
||||
{% tab title="Events monitor - Mainnet" %}
|
||||
{% code title="docker-compose-events-mainnet.yml" %}
|
||||
```yaml
|
||||
version: '3'
|
||||
services:
|
||||
aquarius-events-mainnet:
|
||||
image: oceanprotocol/aquarius:${AQUARIUS_VERISON}
|
||||
container_name: aquarius-events-mainnet
|
||||
restart: on-failure
|
||||
networks:
|
||||
- ocean_backend
|
||||
depends_on:
|
||||
- elasticsearch
|
||||
environment:
|
||||
DB_MODULE: elasticsearch
|
||||
DB_HOSTNAME: elasticsearch
|
||||
DB_PORT: 9200
|
||||
DB_USERNAME: ${DB_USERNAME}
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
DB_NAME: aquarius
|
||||
DB_SCHEME: http
|
||||
DB_SSL : "false"
|
||||
LOG_LEVEL: "DEBUG"
|
||||
AQUARIUS_BIND_URL: "http://0.0.0.0:5000"
|
||||
AQUARIUS_WORKERS : "1"
|
||||
RUN_AQUARIUS_SERVER : "0"
|
||||
AQUARIUS_CONFIG_FILE: "config.ini"
|
||||
NETWORK_NAME: "mainnet"
|
||||
EVENTS_RPC: ${EVENTS_RPC_MAINNET}
|
||||
BFACTORY_BLOCK: # TODO
|
||||
METADATA_CONTRACT_BLOCK: # TODO
|
||||
METADATA_UPDATE_ALL : "0"
|
||||
OCEAN_ADDRESS : 0x967da4048cD07aB37855c090aAF366e4ce1b9F48
|
||||
EVENTS_ALLOW: 0
|
||||
RUN_EVENTS_MONITOR: 1
|
||||
BLOCKS_CHUNK_SIZE: "5000"
|
||||
volumes:
|
||||
data:
|
||||
driver: local
|
||||
networks:
|
||||
ocean_backend:
|
||||
driver: bridge
|
||||
```
|
||||
{% endcode %}
|
||||
{% endtab %}
|
||||
|
||||
{% tab title="Events monitor - Polygon" %}
|
||||
{% code title="docker-compose-events-ploygon.yml" %}
|
||||
```yaml
|
||||
version: '3'
|
||||
services:
|
||||
aquarius-events-polygon:
|
||||
image: oceanprotocol/aquarius:${AQUARIUS_VERISON}
|
||||
container_name: aquarius-events-polygon
|
||||
restart: on-failure
|
||||
networks:
|
||||
- ocean_backend
|
||||
depends_on:
|
||||
- elasticsearch
|
||||
environment:
|
||||
DB_MODULE: elasticsearch
|
||||
DB_HOSTNAME: elasticsearch
|
||||
DB_PORT: 9200
|
||||
DB_USERNAME: ${DB_USERNAME}
|
||||
DB_PASSWORD: ${DB_PASSWORD}
|
||||
DB_NAME: aquarius
|
||||
DB_SCHEME: http
|
||||
DB_SSL : "false"
|
||||
LOG_LEVEL: "DEBUG"
|
||||
AQUARIUS_BIND_URL: "http://0.0.0.0:5000"
|
||||
AQUARIUS_WORKERS : "1"
|
||||
RUN_AQUARIUS_SERVER : "0"
|
||||
AQUARIUS_CONFIG_FILE: "config.ini"
|
||||
NETWORK_NAME: "polygon"
|
||||
EVENTS_RPC: ${EVENTS_RPC_POLYGON}
|
||||
BFACTORY_BLOCK: 11005239
|
||||
METADATA_CONTRACT_BLOCK: 11005247
|
||||
METADATA_UPDATE_ALL: "0"
|
||||
OCEAN_ADDRESS: "0x282d8efCe846A88B159800bd4130ad77443Fa1A1"
|
||||
EVENTS_ALLOW: 0
|
||||
RUN_EVENTS_MONITOR: 1
|
||||
METADATA_CONTRACT_ADDRESS: "0x80E63f73cAc60c1662f27D2DFd2EA834acddBaa8"
|
||||
BLOCKS_CHUNK_SIZE: "5000"
|
||||
volumes:
|
||||
data:
|
||||
driver: local
|
||||
networks:
|
||||
ocean_backend:
|
||||
driver: bridge
|
||||
```
|
||||
{% endcode %}
|
||||
{% endtab %}
|
||||
{% endtabs %}
|
||||
|
||||
### Start Aquarius
|
||||
|
||||
```
|
||||
docker-compose \
|
||||
-f docker-compose.yml \
|
||||
-f docker-events-mainnet.yml \
|
||||
-f docker-events-polygon.yml \
|
||||
--env-file .env \
|
||||
-d \
|
||||
up
|
||||
```
|
||||
|
||||
After pulling all the asset metadata from the blockchain, Aquarius can be used to query the assets using Elasticsearch query. Aquarius REST API are documented here.
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# Deploying Marketplace
|
||||
|
||||
### Prerequisites
|
||||
|
||||
* Docker and Docker compose are installed
|
||||
|
||||
### Create a directory
|
||||
|
||||
```
|
||||
mkdir my-marketplace
|
||||
cd my-marketplace
|
||||
```
|
||||
|
||||
### Create file with name \`.env\`
|
||||
|
||||
Copy the below content into the \`.env\` file.
|
||||
|
||||
<mark style="color:red;">TODO: explain ALLOWED\_PUBLISHERS and EVENTS\_RPC</mark>
|
||||
|
||||
{% code title=".env" %}
|
||||
```
|
||||
DB_USERNAME=username
|
||||
DB_PASSWORD=password
|
||||
# check the available versions: https://hub.docker.com/repository/docker/oceanprotocol/aquarius
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
### Build a Marketplace container
|
||||
|
||||
#### Create a \`Dockerfile\` file and copy the below content into it.
|
||||
|
||||
{% code title="Dockerfile" %}
|
||||
```
|
||||
FROM node:16
|
||||
RUN git clone https://github.com/oceanprotocol/market.git /usr/app/market
|
||||
WORKDIR /usr/app/market
|
||||
RUN npm ci --legacy-peer-deps
|
||||
RUN npm run build
|
||||
EXPOSE 3000
|
||||
CMD ["npx", "next", "start"]
|
||||
```
|
||||
{% endcode %}
|
||||
|
||||
Build a docker image
|
||||
|
||||
```bash
|
||||
docker build . -f Dockerfile -t market:latest
|
||||
```
|
||||
|
||||
### Start the marketplace
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 87 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 38 KiB |
|
|
@ -9,7 +9,7 @@ description: Data NFTs and datatokens architecture
|
|||
|
||||
Here is the Ocean architecture.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
Here’s an overview of the figure.
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ Publishers can choose this fixed pricing model when they do not want Automated M
|
|||
|
||||
The image below shows how to set the fixed pricing of an asset in the Ocean's Marketplace. Here, the price of the asset is set to 10 Ocean tokens.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
### Dynamic pricing
|
||||
|
||||
|
|
@ -39,7 +39,7 @@ Publishers can set the pricing model of an asset to Dynamic pricing if they want
|
|||
|
||||
The image below shows how to set the Dynamic pricing of an asset in the Ocean's Marketplace. Here, the asset price is initially set to 50 Ocean tokens.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
Ocean Protocol also allows publishers to set the pricing using ocean.js and ocean.py library.
|
||||
|
||||
|
|
@ -75,4 +75,4 @@ Free pricing is suitable for individuals and organizations working in the public
|
|||
|
||||
The image below shows how to set free access to an asset in the Ocean's Marketplace.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ A non-fungible token stored on the blockchain represents a unique asset. NFTs ca
|
|||
|
||||
Fungible tokens represent fungible assets. If you have 5 ETH and Alice has 5 ETH, you and Alice could swap your ETH and your final holdings remain the same. They're apples-to-apples. Licenses (contracts) to access a copyrighted asset are naturally fungible - they can be swapped with each other.
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
### High-Level Architecture
|
||||
|
||||
|
|
@ -37,7 +37,7 @@ ERC721 tokens are non-fungible, thus cannot be used for automatic price discover
|
|||
|
||||
### High-Level Behavior
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
Here's an example.
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ The DDO is stored on-chain as part of the NFT contract and stored in encrypted f
|
|||
|
||||
Here is the flow:
|
||||
|
||||
.png>)
|
||||
 (1).png>)
|
||||
|
||||
<details>
|
||||
|
||||
|
|
@ -222,35 +222,37 @@ where "files" contains one or more storage objects.
|
|||
|
||||
Type of objects supported :
|
||||
|
||||
| Type | Description | Example |
|
||||
| ----- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `url` | Static URL. Contains url and HTTP method | <pre class="language-json"><code class="lang-json">[
|
||||
{
|
||||
"type": "url",
|
||||
"url": "https://url.com/file1.csv",
|
||||
"method": "GET",
|
||||
"headers":
|
||||
[
|
||||
{"Authorization": "Bearer 123"},
|
||||
{"APIKEY": "124"},
|
||||
]
|
||||
}
|
||||
]</code></pre> |
|
||||
| Type | Description | Example |
|
||||
| ----- | ---------------------------------------- | ----------------------------------------------------------------- |
|
||||
| `url` | Static URL. Contains url and HTTP method | <pre class="language-json"><code class="lang-json">[</code></pre> |
|
||||
| { | | |
|
||||
|
||||
```
|
||||
"type": "url",
|
||||
"url": "https://url.com/file1.csv",
|
||||
"method": "GET",
|
||||
"headers":
|
||||
[
|
||||
{"Authorization": "Bearer 123"},
|
||||
{"APIKEY": "124"},
|
||||
]
|
||||
```
|
||||
|
||||
} ] |
|
||||
|
||||
First class integrations supported in the future :
|
||||
|
||||
| Type | Description | Example |
|
||||
| ---------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| `ipfs` | IPFS files | <pre class="language-json"><code class="lang-json">[
|
||||
{
|
||||
"type": "ipfs",
|
||||
"hash": "XXX"
|
||||
}
|
||||
]</code></pre> |
|
||||
| `filecoin` | Filecoin storage | |
|
||||
| `arwave` | Arwave | |
|
||||
| `storj` | Storj | |
|
||||
| `sql` | Sql connection, dataset is generated by a query | |
|
||||
| Type | Description | Example |
|
||||
| ------ | ----------- | ----------------------------------------------------------------- |
|
||||
| `ipfs` | IPFS files | <pre class="language-json"><code class="lang-json">[</code></pre> |
|
||||
| { | | |
|
||||
|
||||
```
|
||||
"type": "ipfs",
|
||||
"hash": "XXX"
|
||||
```
|
||||
|
||||
} ] | | `filecoin` | Filecoin storage | | | `arwave` | Arwave | | | `storj` | Storj | | | `sql` | Sql connection, dataset is generated by a query | |
|
||||
|
||||
A service can contain multiple files, using multiple storage types.
|
||||
|
||||
|
|
@ -301,15 +303,23 @@ An asset with a service of `type` `compute` has the following additional attribu
|
|||
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Type | Required | Description |
|
||||
| `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. |
|
||||
| Type | Required | Description |
|
||||
| `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. |
|
||||
| <p><strong><code>allowNetworkAccess</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td><code>boolean</code></td><td><strong>✓</strong></td><td>If <code>true</code>, the algorithm job will have network access.</td></tr></tbody></table> | | |
|
||||
| Type | Required | Description |
|
||||
| `boolean` | **✓** | If `true`, the algorithm job will have network access. |
|
||||
| Type | Required | Description |
|
||||
| `boolean` | **✓** | If `true`, the algorithm job will have network access. |
|
||||
| <p><strong><code>publisherTrustedAlgorithmPublishers</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Array of <code>string</code></td><td><strong>✓</strong></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed.</td></tr></tbody></table> | | |
|
||||
| Type | Required | Description |
|
||||
| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. |
|
||||
| Type | Required | Description |
|
||||
| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. |
|
||||
| <p><strong><code>publisherTrustedAlgorithms</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Array of <code>publisherTrustedAlgorithms</code></td><td><strong>✓</strong></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below).</td></tr></tbody></table> | | |
|
||||
| Type | Required | Description |
|
||||
| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). |
|
||||
| Type | Required | Description |
|
||||
| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). |
|
||||
|
||||
The `publisherTrustedAlgorithms` is an array of objects with the following structure:
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 62 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 39 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
Loading…
Reference in New Issue