1
0
mirror of https://github.com/oceanprotocol/docs.git synced 2024-11-26 19:49:26 +01:00

Merge pull request #1055 from oceanprotocol/issue-1043-hosting-aquarius

Create hosting aquarius, subgraph page
This commit is contained in:
Akshay 2022-07-26 11:25:44 +02:00 committed by GitHub
commit 88568fc23f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 372 additions and 7 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

BIN
.gitbook/assets/image.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

View File

@ -27,17 +27,16 @@
* [Writing Algorithms for Compute to Data](building-with-ocean/compute-to-data/compute-to-data-algorithms.md) * [Writing Algorithms for Compute to Data](building-with-ocean/compute-to-data/compute-to-data-algorithms.md)
* [Setting up private docker registry](building-with-ocean/compute-to-data/compute-to-data-docker-registry.md) * [Setting up private docker registry](building-with-ocean/compute-to-data/compute-to-data-docker-registry.md)
* [Deploying components](building-with-ocean/deploying-components/README.md) * [Deploying components](building-with-ocean/deploying-components/README.md)
* [Setup server](building-with-ocean/deploying-components/setup-server.md)
* [Deploying Marketplace](building-with-ocean/deploying-components/deploying-marketplace.md) * [Deploying Marketplace](building-with-ocean/deploying-components/deploying-marketplace.md)
* [Deploying Aquarius](building-with-ocean/deploying-components/deploying-aquarius.md) * [Deploying Aquarius](building-with-ocean/deploying-components/deploying-aquarius.md)
<<<<<<< HEAD
* [Deploying Provider](building-with-ocean/deploying-components/deploying-provider.md) * [Deploying Provider](building-with-ocean/deploying-components/deploying-provider.md)
======= * [Deploying Ocean subgraph](building-with-ocean/deploying-components/deploying-ocean-subgraph.md)
* [Contributing](core-concepts/contributing.md) * [Contributing](core-concepts/contributing.md)
* [Contributor Code of Conduct](core-concepts/code-of-conduct.md) * [Contributor Code of Conduct](core-concepts/code-of-conduct.md)
* [Legal Requirements when Contributing Code](core-concepts/legal-reqs.md) * [Legal Requirements when Contributing Code](core-concepts/legal-reqs.md)
* [Reporting Vulnerabilities](core-concepts/vulnerabilities.md) * [Reporting Vulnerabilities](core-concepts/vulnerabilities.md)
* [Funding](core-concepts/get-funding.md) * [Funding](core-concepts/get-funding.md)
>>>>>>> d687b4e2d8782db2ea7e3e61fa15e9608451e78d
* [Projects using Ocean Protocol](building-with-ocean/projects-using-ocean.md) * [Projects using Ocean Protocol](building-with-ocean/projects-using-ocean.md)
* [API references](api-references/README.md) * [API references](api-references/README.md)
* [Aquarius REST API](api-references/aquarius-rest-api.md) * [Aquarius REST API](api-references/aquarius-rest-api.md)

View File

@ -0,0 +1,208 @@
# Deploying Aquarius
### About Aquarius
Aquarius is an off-chain component with caches the asset metadata published on-chain. By deploying own Aquarius, developers can control which assets are visible in their marketplace. For example, having a custom Aquarius instance allows assets only from specific addresses to be visible on the marketplace. This tutorial will provide the steps to deploy Aquarius. Ocean Protocol provides Aquarius docker images which can be viewed [here](https://hub.docker.com/r/oceanprotocol/aquarius/tags). Visit [this](https://github.com/oceanprotocol/aquarius) page to view Aquarius source code.
Aquarius consists of two parts:\
\- **API:** The Aquarius API offers a convenient way to access the medatata without scanning the chain yourself.\
\- **Event monitor:** Aquarius continually monitors the chains for MetadataCreated and MetadataUpdated events, processes these events and adds them to the database.
### Prerequisites
* A server for hosting Aquarius. See [this guide](setup-server.md) on creating a server.
* Docker and Docker compose are installed. Click [here](https://docs.docker.com/engine/install/) to view guide on installing docker.
* Ethereum API. Aquarius uses Ethereum api for monitoring on-chain events.\
Choose any api provider of your choice. Some of the commonly used are:
* [Infura](https://infura.io/)
* [Alchemy](https://www.alchemy.com/)
* [Moralis](https://moralis.io/)
### Create a working directory
```
mkdir Aquarius
cd Aquarius
```
### Create a \`.env\` file
Copy the below content into the \`.env\` file and edit the values as needed.
{% code title=".env" %}
```
# check the available versions: https://hub.docker.com/repository/docker/oceanprotocol/aquarius
AQUARIUS_VERSION=latest
ALLOWED_PUBLISHERS='[""]'
# Elastic search credentials
DB_USERNAME=username
DB_PASSWORD=password
# Replace below value with the API provider of your choice
EVENTS_RPC_POLYGON=<polygon-key>
EVENTS_RPC_MAINNET=<mainnet-key>
```
{% endcode %}
### Create docker-compose file
{% code title="docker-compose.yml" %}
```yaml
version: '3'
services:
elasticsearch:
image: elasticsearch:6.8.17
container_name: elasticsearch
restart: on-failure
environment:
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
MAX_MAP_COUNT: "64000"
discovery.type: "single-node"
volumes:
- data:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- ocean_backend
aquarius:
image: oceanprotocol/aquarius:${AQUARIUS_VERSION}
container_name: aquarius
restart: on-failure
ports:
- 5000:5000
networks:
- ocean_backend
depends_on:
- elasticsearch
environment:
DB_MODULE: elasticsearch
DB_HOSTNAME: elasticsearch
DB_PORT: 9200
DB_USERNAME: ${DB_USERNAME}
DB_PASSWORD: ${DB_PASSWORD}
DB_NAME: aquarius
DB_SCHEME: http
DB_SSL : "false"
LOG_LEVEL: "DEBUG"
AQUARIUS_BIND_URL : "http://0.0.0.0:5000"
AQUARIUS_WORKERS : "8"
RUN_AQUARIUS_SERVER: "1"
AQUARIUS_CONFIG_FILE: "config.ini"
EVENTS_ALLOW: 0
RUN_EVENTS_MONITOR: 0
ALLOWED_PUBLISHERS: ${ALLOWED_PUBLISHERS}
volumes:
data:
driver: local
networks:
ocean_backend:
driver: bridge
```
{% endcode %}
### Create events monitor docker compose file
{% tabs %}
{% tab title="Events monitor - Mainnet" %}
{% code title="docker-compose-events-mainnet.yml" %}
```yaml
version: '3'
services:
aquarius-events-mainnet:
image: oceanprotocol/aquarius:${AQUARIUS_VERSION}
container_name: aquarius-events-mainnet
restart: on-failure
networks:
- ocean_backend
depends_on:
- elasticsearch
environment:
DB_MODULE: elasticsearch
DB_HOSTNAME: elasticsearch
DB_PORT: 9200
DB_USERNAME: ${DB_USERNAME}
DB_PASSWORD: ${DB_PASSWORD}
DB_NAME: aquarius
DB_SCHEME: http
DB_SSL : "false"
LOG_LEVEL: "DEBUG"
AQUARIUS_BIND_URL: "http://0.0.0.0:5000"
AQUARIUS_WORKERS : "1"
RUN_AQUARIUS_SERVER : "0"
AQUARIUS_CONFIG_FILE: "config.ini"
NETWORK_NAME: "mainnet"
EVENTS_RPC: ${EVENTS_RPC_MAINNET}
METADATA_UPDATE_ALL : "0"
OCEAN_ADDRESS : "0x967da4048cD07aB37855c090aAF366e4ce1b9F48"
EVENTS_ALLOW: 0
RUN_EVENTS_MONITOR: 1
BLOCKS_CHUNK_SIZE: "5000"
volumes:
data:
driver: local
networks:
ocean_backend:
driver: bridge
```
{% endcode %}
{% endtab %}
{% tab title="Events monitor - Polygon" %}
{% code title="docker-compose-events-ploygon.yml" %}
```yaml
version: '3'
services:
aquarius-events-polygon:
image: oceanprotocol/aquarius:${AQUARIUS_VERSION}
container_name: aquarius-events-polygon
restart: on-failure
networks:
- ocean_backend
depends_on:
- elasticsearch
environment:
DB_MODULE: elasticsearch
DB_HOSTNAME: elasticsearch
DB_PORT: 9200
DB_USERNAME: ${DB_USERNAME}
DB_PASSWORD: ${DB_PASSWORD}
DB_NAME: aquarius
DB_SCHEME: http
DB_SSL : "false"
LOG_LEVEL: "DEBUG"
AQUARIUS_BIND_URL: "http://0.0.0.0:5000"
AQUARIUS_WORKERS : "1"
RUN_AQUARIUS_SERVER : "0"
AQUARIUS_CONFIG_FILE: "config.ini"
NETWORK_NAME: "polygon"
EVENTS_RPC: ${EVENTS_RPC_POLYGON}
METADATA_UPDATE_ALL: "0"
OCEAN_ADDRESS: "0x282d8efCe846A88B159800bd4130ad77443Fa1A1"
EVENTS_ALLOW: 0
RUN_EVENTS_MONITOR: 1
METADATA_CONTRACT_ADDRESS: "0x80E63f73cAc60c1662f27D2DFd2EA834acddBaa8"
BLOCKS_CHUNK_SIZE: "5000"
volumes:
data:
driver: local
networks:
ocean_backend:
driver: bridge
```
{% endcode %}
{% endtab %}
{% endtabs %}
### Start Aquarius
```
docker-compose \
-f docker-compose.yml \
-f docker-compose-events-mainnet.yml \
-f docker-compose-events-polygon.yml \
--env-file .env \
-d \
up
```
After pulling all the asset metadata from the blockchain, Aquarius can be used to query the assets using Elasticsearch query. Aquarius REST API are documented here.

View File

@ -2,9 +2,9 @@
### Prerequisites ### Prerequisites
* Docker and Docker compose are installed * A server for hosting Ocean Marketplace. See [this guide](setup-server.md) on creating a server.
### Create a directory #### Create a directory
``` ```
mkdir my-marketplace mkdir my-marketplace
@ -39,8 +39,6 @@ NEXT_PUBLIC_METADATACACHE_URI=https://v4.aquarius.oceanprotocol.com
``` ```
{% endcode %} {% endcode %}
### Build a Marketplace container
#### Create a \`Dockerfile\` file and copy the below content into it. #### Create a \`Dockerfile\` file and copy the below content into it.
{% code title="Dockerfile" %} {% code title="Dockerfile" %}

View File

@ -0,0 +1,87 @@
# Deploying Ocean subgraph
### About Ocean subgraph
Ocean subgraph allows querying the datatoken, dataNFT, and all event information using GraphQL. Hosting the Ocean subgraph saves the cost and time required in querying the data directly from the blockchain. The steps in this tutorial will explain how to host Ocean subgraph for the EVM compatible chains supported by Ocean Protocol.
### Prerequisites
* A server for hosting Ocean subgraph. See [this guide](setup-server.md) on creating a server.
* Docker and Docker compose are installed. Click [here](https://docs.docker.com/engine/install/) to view guide on installing docker.
* Ethereum API. Aquarius uses Ethereum api for monitoring on-chain events.\
Choose any api provider of your choice. Some of the commonly used are:
* [Infura](https://infura.io/)
* [Alchemy](https://www.alchemy.com/)
* [Moralis](https://moralis.io/)
### Create a working directory
```
mkdir ocean-subgraph
cd ocean-subgraph
```
### Create a \`.env\` file
Copy the below content into the \`.env\` file and edit the values as needed.
{% code title=".env" %}
```
ETHEREUM_NODE_PROVIDER_API='rinkeby:https://rinkeby.infura.io/v3/${INFURA_PROJECT_ID}'
```
{% endcode %}
### Create docker-compose file
{% code title="docker-compose.yml" %}
```yaml
version: '3'
services:
graph-node:
image: graphprotocol/graph-node:v0.26.0
ports:
- '9000:8000'
- '8001:8001'
- '8020:8020'
- '8030:8030'
- '8040:8040'
depends_on:
- ipfs
- postgres
environment:
postgres_host: postgres
postgres_user: graph-node
postgres_pass: let-me-in
postgres_db: graph-node
ipfs: 'ipfs:5001'
ethereum: ${ETHEREUM_NODE_PROVIDER_API}
RUST_LOG: info
ipfs:
image: ipfs/go-ipfs:v0.4.23
ports:
- '5001:5001'
volumes:
- ./data/ipfs:/data/ipfs
postgres:
image: postgres
ports:
- '5432:5432'
command: ['postgres', '-cshared_preload_libraries=pg_stat_statements']
environment:
POSTGRES_USER: graph-node
POSTGRES_PASSWORD: let-me-in
POSTGRES_DB: graph-node
volumes:
- ./data/postgres:/var/lib/postgresql/data
```
{% endcode %}
### Start Ocean subgraph
```
docker-compose \
-f docker-compose.yml
--env-file .env \
-d \
up
```

View File

@ -0,0 +1,65 @@
---
description: >-
The following tutorial shows how to create a server ready for hosting Ocean
Protocol's components.
---
# Setup server
## **Using hosting services**
Ocean Protocol's components can be hosted on any infrastructure providers like AWS, Azure, Heroku, Digitalocean, and many others. The tutorial here explains how to create a server using Digitalocean and installing docker which will be required to host Ocean Protocol's components. Apart from steps for create a server, the remaining part of the tutorial will be same for all hosting providers.
#### Creating account and setting billing
Go to [https://www.digitalocean.com/](https://www.digitalocean.com/) and create an account. Provide the appropriate information for billing and accounting.
#### Create a droplet
Click on **`Create`** button and choose **`Droplets`** options from dropdown.
![](../../.gitbook/assets/image.png)
#### Configure droplet
Select Ubuntu OS and choose a plan. The required CPU, Memory depends on the number of requests Aquarius is expected to serve.&#x20;
![Configure droplet](<../../.gitbook/assets/image (8).png>)
Also, select the region where you want Aquarius to be hosted and a root password.
![](<../../.gitbook/assets/image (4).png>)
![Click Create Droplet](<../../.gitbook/assets/image (7).png>)
Finalize the parameters for the server, click on `Create Droplet.` After the server is ready, select the `Access console` option from the dropdown.
![Click Access Console](<../../.gitbook/assets/image (3).png>)
![Click Launch Droplet Console](<../../.gitbook/assets/image (9).png>)
A window will open with a terminal session. Now, the required infrastructure is ready for hosting Aquarius, Provider or the Subgraph. Let's install docker and docker-compose on the server. Follow the installation guide [here](https://docs.docker.com/engine/install/ubuntu/).
The below commands shows the commands executed by following the guide.
```bash
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
# Now install docker-compose
sudo apt-get update
sudo apt-get install docker-compose-plugin
```
Now that, the server is ready with all the required dependencies are installed for hosting Ocean Components, follow the instructions given in Component specific guide.&#x20;
* [Deploying Marketplace](deploying-marketplace.md)
* [ Deploying Aquarius ](deploying-aquarius.md)

View File

@ -307,6 +307,8 @@ An asset with a service of `type` `compute` has the following additional attribu
| `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. | | `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. |
| Type | Required | Description | | Type | Required | Description |
| `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. | | `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. |
| Type | Required | Description |
| `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. |
| <p><strong><code>allowNetworkAccess</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td><code>boolean</code></td><td><strong></strong></td><td>If <code>true</code>, the algorithm job will have network access.</td></tr></tbody></table> | | | | <p><strong><code>allowNetworkAccess</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td><code>boolean</code></td><td><strong></strong></td><td>If <code>true</code>, the algorithm job will have network access.</td></tr></tbody></table> | | |
| Type | Required | Description | | Type | Required | Description |
| `boolean` | **✓** | If `true`, the algorithm job will have network access. | | `boolean` | **✓** | If `true`, the algorithm job will have network access. |
@ -314,6 +316,8 @@ An asset with a service of `type` `compute` has the following additional attribu
| `boolean` | **✓** | If `true`, the algorithm job will have network access. | | `boolean` | **✓** | If `true`, the algorithm job will have network access. |
| Type | Required | Description | | Type | Required | Description |
| `boolean` | **✓** | If `true`, the algorithm job will have network access. | | `boolean` | **✓** | If `true`, the algorithm job will have network access. |
| Type | Required | Description |
| `boolean` | **✓** | If `true`, the algorithm job will have network access. |
| <p><strong><code>publisherTrustedAlgorithmPublishers</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Array of <code>string</code></td><td><strong></strong></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed.</td></tr></tbody></table> | | | | <p><strong><code>publisherTrustedAlgorithmPublishers</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Array of <code>string</code></td><td><strong></strong></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed.</td></tr></tbody></table> | | |
| Type | Required | Description | | Type | Required | Description |
| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. | | Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. |
@ -321,6 +325,8 @@ An asset with a service of `type` `compute` has the following additional attribu
| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. | | Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. |
| Type | Required | Description | | Type | Required | Description |
| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. | | Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. |
| Type | Required | Description |
| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. |
| <p><strong><code>publisherTrustedAlgorithms</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Array of <code>publisherTrustedAlgorithms</code></td><td><strong></strong></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below).</td></tr></tbody></table> | | | | <p><strong><code>publisherTrustedAlgorithms</code></strong></p><table><thead><tr><th>Type</th><th>Required</th><th>Description</th></tr></thead><tbody><tr><td>Array of <code>publisherTrustedAlgorithms</code></td><td><strong></strong></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below).</td></tr></tbody></table> | | |
| Type | Required | Description | | Type | Required | Description |
| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). | | Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). |
@ -328,6 +334,8 @@ An asset with a service of `type` `compute` has the following additional attribu
| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). | | Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). |
| Type | Required | Description | | Type | Required | Description |
| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). | | Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). |
| Type | Required | Description |
| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). |
The `publisherTrustedAlgorithms` is an array of objects with the following structure: The `publisherTrustedAlgorithms` is an array of objects with the following structure: