diff --git a/SUMMARY.md b/SUMMARY.md index 1f7d0eb9..2b074285 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -26,12 +26,15 @@ * [Writing Algorithms for Compute to Data](building-with-ocean/compute-to-data-algorithms.md) * [Set Up a Marketplace](building-with-ocean/marketplace.md) * [Polygon (ex Matic)](building-with-ocean/polygon-bridge.md) - * [Projects using Ocean Protocol](building-with-ocean/projects-using-ocean.md) * [Compute-to-Data](building-with-ocean/compute-to-data/README.md) * [Architecture](building-with-ocean/compute-to-data/compute-to-data-architecture.md) * [Datasets & Algorithms](building-with-ocean/compute-to-data/compute-to-data-datasets-algorithms.md) * [Minikube Compute-to-Data Environment](building-with-ocean/compute-to-data/compute-to-data-minikube.md) * [Setting up private docker registry](building-with-ocean/compute-to-data/compute-to-data-docker-registry.md) + * [Deploying components](building-with-ocean/deploying-components/README.md) + * [Deploying Marketplace](building-with-ocean/deploying-components/deploying-marketplace.md) + * [Deploying Aquarius](building-with-ocean/deploying-components/deploying-aquarius.md) + * [Projects using Ocean Protocol](building-with-ocean/projects-using-ocean.md) * [Using Ocean Market](building-with-ocean/marketplace-introduction.md) * [Publish a data asset](building-with-ocean/marketplace-publish-data-asset.md) * [Download a data asset](building-with-ocean/marketplace-download-data-asset.md) diff --git a/building-with-ocean/asset-hosting.md b/building-with-ocean/asset-hosting.md index add82dbb..97655e7e 100644 --- a/building-with-ocean/asset-hosting.md +++ b/building-with-ocean/asset-hosting.md @@ -11,7 +11,7 @@ To publish assets on the Ocean Marketplace, publishers must provide a link(an UR On Ocean Marketplace, a publisher must provide the link to the asset during publish step. Once the asset is published, this link cannot be changed. So, it is essential that the publisher correctly sets this field (shown in the below image). -![Publish - File URL field]() +![Publish - File URL field]() ### Hosting services @@ -27,7 +27,7 @@ Open https://drive.google.com and upload the file you want to publish on the Oce The file URL will be of the form `https://drive.google.com/file/d//view?usp=sharing`, where the `` is the unique alphanumeric string. Verify if the URL is correct by entering it in a browser and check if the file is downloaded. -![Google Drive link]() +![Google Drive link]() **Step 2 - Create a downloadable link** @@ -41,7 +41,7 @@ Note the `` from step 1 and create a URL as below. After creating a downloadable file URL, fill the `File*` field with the downloadable URL created in step 2. -![Publish - Google Drive file]() +![Publish - Google Drive file]() _Note: Google Drive allows only shared files to be downloaded, as shown in the above steps. The above method does not work with the shared folder. As a workaround, publishers can upload a zip of a folder and upload it as a file._ diff --git a/building-with-ocean/deploying-components/README.md b/building-with-ocean/deploying-components/README.md new file mode 100644 index 00000000..29efd695 --- /dev/null +++ b/building-with-ocean/deploying-components/README.md @@ -0,0 +1,2 @@ +# Deploying components + diff --git a/building-with-ocean/deploying-components/deploying-aquarius.md b/building-with-ocean/deploying-components/deploying-aquarius.md new file mode 100644 index 00000000..5819de12 --- /dev/null +++ b/building-with-ocean/deploying-components/deploying-aquarius.md @@ -0,0 +1,212 @@ +# Deploying Aquarius + +### About Aquarius + +Aquarius is an off-chain component with caches the asset metadata published on-chain. By deploying own Aquarius, developers can control which assets are visible in their marketplace. For example, having a custom Aquarius instance allows assets only from specific addresses to be visible on the marketplace. This tutorial will provide the steps to deploy Aquarius. Ocean Protocol provides Aquarius docker images which can be viewed [here](https://hub.docker.com/r/oceanprotocol/aquarius/tags). Visit [this](https://github.com/oceanprotocol/aquarius) page to view Aquarius source code. + +Aquarius consists of two parts:\ +\- **API:** The Aquarius API offers a convenient way to access the medatata without scanning the chain yourself.\ +\- **Event monitor:** Aquarius continually monitors the chains for MetadataCreated and MetadataUpdated events, processes these events and adds them to the database. + +### Prerequisites + +* Docker and Docker compose are installed. Click [here](https://docs.docker.com/engine/install/) to view guide on installing docker. +* Ethereum API. Aquarius uses Ethereum api for monitoring on-chain events.\ + Choose any api provider of your choice. Some of the commonly used are: + * [Infura](https://infura.io/) + * [Alchemy](https://www.alchemy.com/) + * [Moralis](https://moralis.io/) + +### Create a working directory + +``` +mkdir Aqaurius +cd Aquarius +``` + +### Create a \`.env\` file + +Copy the below content into the \`.env\` file and edit the values as needed. + +{% code title=".env" %} +``` +# check the available versions: https://hub.docker.com/repository/docker/oceanprotocol/aquarius +AQUARIUS_VERSION=latest +ALLOWED_PUBLISHERS='[""]' +# Elastic search credentials +DB_USERNAME=username +DB_PASSWORD=password + +# Replace below value with the API provider of your choice +EVENTS_RPC_RINKEBY=wss://rinkeby.infura.io/ws/v3/INFURA_ID +EVENTS_RPC_POLYGON=wss://rinkeby.infura.io/ws/v3/INFURA_ID +EVENTS_RPC_MAINNET=wss://rinkeby.infura.io/ws/v3/INFURA_ID +``` +{% endcode %} + +### Create docker-compose file + +{% code title="docker-compose.yml" %} +```yaml +version: '3' +services: + elasticsearch: + image: elasticsearch:6.8.17 + container_name: elasticsearch + restart: on-failure + environment: + ES_JAVA_OPTS: "-Xms512m -Xmx512m" + MAX_MAP_COUNT: "64000" + discovery.type: "single-node" + volumes: + - data:/usr/share/elasticsearch/data + ports: + - 9200:9200 + networks: + - ocean_backend + aquarius: + image: oceanprotocol/aquarius:${AQUARIUS_VERSION} + container_name: aquarius + restart: on-failure + ports: + - 5000:5000 + networks: + - ocean_backend + depends_on: + - elasticsearch + environment: + DB_MODULE: elasticsearch + DB_HOSTNAME: elasticsearch + DB_PORT: 9200 + DB_USERNAME: ${DB_USERNAME} + DB_PASSWORD: ${DB_PASSWORD} + DB_NAME: aquarius + DB_SCHEME: http + DB_SSL : "false" + LOG_LEVEL: "DEBUG" + AQUARIUS_BIND_URL : "http://0.0.0.0:5000" + AQUARIUS_WORKERS : "8" + RUN_AQUARIUS_SERVER: "1" + AQUARIUS_CONFIG_FILE: "config.ini" + EVENTS_ALLOW: 0 + RUN_EVENTS_MONITOR: 0 + ALLOWED_PUBLISHERS: ${ALLOWED_PUBLISHERS} +volumes: + data: + driver: local +networks: + ocean_backend: + driver: bridge +``` +{% endcode %} + +### Create events monitor docker compose file + +{% tabs %} +{% tab title="Events monitor - Mainnet" %} +{% code title="docker-compose-events-mainnet.yml" %} +```yaml +version: '3' +services: + aquarius-events-mainnet: + image: oceanprotocol/aquarius:${AQUARIUS_VERISON} + container_name: aquarius-events-mainnet + restart: on-failure + networks: + - ocean_backend + depends_on: + - elasticsearch + environment: + DB_MODULE: elasticsearch + DB_HOSTNAME: elasticsearch + DB_PORT: 9200 + DB_USERNAME: ${DB_USERNAME} + DB_PASSWORD: ${DB_PASSWORD} + DB_NAME: aquarius + DB_SCHEME: http + DB_SSL : "false" + LOG_LEVEL: "DEBUG" + AQUARIUS_BIND_URL: "http://0.0.0.0:5000" + AQUARIUS_WORKERS : "1" + RUN_AQUARIUS_SERVER : "0" + AQUARIUS_CONFIG_FILE: "config.ini" + NETWORK_NAME: "mainnet" + EVENTS_RPC: ${EVENTS_RPC_MAINNET} + BFACTORY_BLOCK: # TODO + METADATA_CONTRACT_BLOCK: # TODO + METADATA_UPDATE_ALL : "0" + OCEAN_ADDRESS : 0x967da4048cD07aB37855c090aAF366e4ce1b9F48 + EVENTS_ALLOW: 0 + RUN_EVENTS_MONITOR: 1 + BLOCKS_CHUNK_SIZE: "5000" +volumes: + data: + driver: local +networks: + ocean_backend: + driver: bridge +``` +{% endcode %} +{% endtab %} + +{% tab title="Events monitor - Polygon" %} +{% code title="docker-compose-events-ploygon.yml" %} +```yaml +version: '3' +services: + aquarius-events-polygon: + image: oceanprotocol/aquarius:${AQUARIUS_VERISON} + container_name: aquarius-events-polygon + restart: on-failure + networks: + - ocean_backend + depends_on: + - elasticsearch + environment: + DB_MODULE: elasticsearch + DB_HOSTNAME: elasticsearch + DB_PORT: 9200 + DB_USERNAME: ${DB_USERNAME} + DB_PASSWORD: ${DB_PASSWORD} + DB_NAME: aquarius + DB_SCHEME: http + DB_SSL : "false" + LOG_LEVEL: "DEBUG" + AQUARIUS_BIND_URL: "http://0.0.0.0:5000" + AQUARIUS_WORKERS : "1" + RUN_AQUARIUS_SERVER : "0" + AQUARIUS_CONFIG_FILE: "config.ini" + NETWORK_NAME: "polygon" + EVENTS_RPC: ${EVENTS_RPC_POLYGON} + BFACTORY_BLOCK: 11005239 + METADATA_CONTRACT_BLOCK: 11005247 + METADATA_UPDATE_ALL: "0" + OCEAN_ADDRESS: "0x282d8efCe846A88B159800bd4130ad77443Fa1A1" + EVENTS_ALLOW: 0 + RUN_EVENTS_MONITOR: 1 + METADATA_CONTRACT_ADDRESS: "0x80E63f73cAc60c1662f27D2DFd2EA834acddBaa8" + BLOCKS_CHUNK_SIZE: "5000" +volumes: + data: + driver: local +networks: + ocean_backend: + driver: bridge +``` +{% endcode %} +{% endtab %} +{% endtabs %} + +### Start Aquarius + +``` +docker-compose \ +-f docker-compose.yml \ +-f docker-events-mainnet.yml \ +-f docker-events-polygon.yml \ +--env-file .env \ +-d \ +up +``` + +After pulling all the asset metadata from the blockchain, Aquarius can be used to query the assets using Elasticsearch query. Aquarius REST API are documented here. diff --git a/building-with-ocean/deploying-components/deploying-marketplace.md b/building-with-ocean/deploying-components/deploying-marketplace.md new file mode 100644 index 00000000..27c57e75 --- /dev/null +++ b/building-with-ocean/deploying-components/deploying-marketplace.md @@ -0,0 +1,50 @@ +# Deploying Marketplace + +### Prerequisites + +* Docker and Docker compose are installed + +### Create a directory + +``` +mkdir my-marketplace +cd my-marketplace +``` + +### Create file with name \`.env\` + +Copy the below content into the \`.env\` file. + +TODO: explain ALLOWED\_PUBLISHERS and EVENTS\_RPC + +{% code title=".env" %} +``` +DB_USERNAME=username +DB_PASSWORD=password +# check the available versions: https://hub.docker.com/repository/docker/oceanprotocol/aquarius +``` +{% endcode %} + +### Build a Marketplace container + +#### Create a \`Dockerfile\` file and copy the below content into it. + +{% code title="Dockerfile" %} +``` +FROM node:16 +RUN git clone https://github.com/oceanprotocol/market.git /usr/app/market +WORKDIR /usr/app/market +RUN npm ci --legacy-peer-deps +RUN npm run build +EXPOSE 3000 +CMD ["npx", "next", "start"] +``` +{% endcode %} + +Build a docker image + +```bash +docker build . -f Dockerfile -t market:latest +``` + +### Start the marketplace diff --git a/building-with-ocean/images/marketplace/publish/marketplace-publish-file-field (1) (1).png b/building-with-ocean/images/marketplace/publish/marketplace-publish-file-field (1) (1).png new file mode 100644 index 00000000..706aba2d Binary files /dev/null and b/building-with-ocean/images/marketplace/publish/marketplace-publish-file-field (1) (1).png differ diff --git a/building-with-ocean/images/marketplace/publish/publish-google-drive (1) (1).png b/building-with-ocean/images/marketplace/publish/publish-google-drive (1) (1).png new file mode 100644 index 00000000..f472a96f Binary files /dev/null and b/building-with-ocean/images/marketplace/publish/publish-google-drive (1) (1).png differ diff --git a/building-with-ocean/images/marketplace/publish/publish-google-drive-2 (1) (1).png b/building-with-ocean/images/marketplace/publish/publish-google-drive-2 (1) (1).png new file mode 100644 index 00000000..9b126c47 Binary files /dev/null and b/building-with-ocean/images/marketplace/publish/publish-google-drive-2 (1) (1).png differ diff --git a/core-concepts/architecture.md b/core-concepts/architecture.md index 3b147d3f..6b17e1dd 100644 --- a/core-concepts/architecture.md +++ b/core-concepts/architecture.md @@ -9,7 +9,7 @@ description: Data NFTs and datatokens architecture Here is the Ocean architecture. -![Ocean Protocol tools architecture]() +![Ocean Protocol tools architecture]() Here’s an overview of the figure. diff --git a/core-concepts/asset-pricing.md b/core-concepts/asset-pricing.md index d119063d..fa943d9c 100644 --- a/core-concepts/asset-pricing.md +++ b/core-concepts/asset-pricing.md @@ -19,7 +19,7 @@ Publishers can choose this fixed pricing model when they do not want Automated M The image below shows how to set the fixed pricing of an asset in the Ocean's Marketplace. Here, the price of the asset is set to 10 Ocean tokens. -![fixed-asset-pricing]() +![fixed-asset-pricing]() ### Dynamic pricing @@ -39,7 +39,7 @@ Publishers can set the pricing model of an asset to Dynamic pricing if they want The image below shows how to set the Dynamic pricing of an asset in the Ocean's Marketplace. Here, the asset price is initially set to 50 Ocean tokens. -![dynamic-asset-pricing]() +![dynamic-asset-pricing]() Ocean Protocol also allows publishers to set the pricing using ocean.js and ocean.py library. @@ -75,4 +75,4 @@ Free pricing is suitable for individuals and organizations working in the public The image below shows how to set free access to an asset in the Ocean's Marketplace. -![free-asset-pricing]() +![free-asset-pricing]() diff --git a/core-concepts/datanft-and-datatoken.md b/core-concepts/datanft-and-datatoken.md index 1f3f4e99..99120369 100644 --- a/core-concepts/datanft-and-datatoken.md +++ b/core-concepts/datanft-and-datatoken.md @@ -11,7 +11,7 @@ A non-fungible token stored on the blockchain represents a unique asset. NFTs ca Fungible tokens represent fungible assets. If you have 5 ETH and Alice has 5 ETH, you and Alice could swap your ETH and your final holdings remain the same. They're apples-to-apples. Licenses (contracts) to access a copyrighted asset are naturally fungible - they can be swapped with each other. -![Data NFT and datatoken]() +![Data NFT and datatoken]() ### High-Level Architecture @@ -37,7 +37,7 @@ ERC721 tokens are non-fungible, thus cannot be used for automatic price discover ### High-Level Behavior -![Flow]() +![Flow]() Here's an example. diff --git a/core-concepts/did-ddo.md b/core-concepts/did-ddo.md index b3185bf5..15b0f06a 100644 --- a/core-concepts/did-ddo.md +++ b/core-concepts/did-ddo.md @@ -33,7 +33,7 @@ The DDO is stored on-chain as part of the NFT contract and stored in encrypted f Here is the flow: -![DDO flow]() +![DDO flow]()
@@ -222,35 +222,37 @@ where "files" contains one or more storage objects. Type of objects supported : -| Type | Description | Example | -| ----- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `url` | Static URL. Contains url and HTTP method | 
[
-  {
-    "type": "url",
-    "url": "https://url.com/file1.csv",
-    "method": "GET",
-    "headers":
-      [
-       {"Authorization": "Bearer 123"}, 
-       {"APIKEY": "124"},
-      ]
-  }
-]
| +| Type | Description | Example | +| ----- | ---------------------------------------- | ----------------------------------------------------------------- | +| `url` | Static URL. Contains url and HTTP method | 
[
| +| { | | | + +``` +"type": "url", +"url": "https://url.com/file1.csv", +"method": "GET", +"headers": + [ + {"Authorization": "Bearer 123"}, + {"APIKEY": "124"}, + ] +``` + +} ] | First class integrations supported in the future : -| Type | Description | Example | -| ---------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| `ipfs` | IPFS files | 
[
-  {
-    "type": "ipfs",
-    "hash": "XXX"
-  }
-]
| -| `filecoin` | Filecoin storage | | -| `arwave` | Arwave | | -| `storj` | Storj | | -| `sql` | Sql connection, dataset is generated by a query | | +| Type | Description | Example | +| ------ | ----------- | ----------------------------------------------------------------- | +| `ipfs` | IPFS files | 
[
| +| { | | | + +``` +"type": "ipfs", +"hash": "XXX" +``` + +} ] | | `filecoin` | Filecoin storage | | | `arwave` | Arwave | | | `storj` | Storj | | | `sql` | Sql connection, dataset is generated by a query | | A service can contain multiple files, using multiple storage types. @@ -301,15 +303,23 @@ An asset with a service of `type` `compute` has the following additional attribu | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Type | Required | Description | | `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. | +| Type | Required | Description | +| `boolean` | **✓** | If `true`, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be `false` by default in all implementations. | |

allowNetworkAccess

TypeRequiredDescription
boolean✓If true, the algorithm job will have network access.
| | | | Type | Required | Description | | `boolean` | **✓** | If `true`, the algorithm job will have network access. | +| Type | Required | Description | +| `boolean` | **✓** | If `true`, the algorithm job will have network access. | |

publisherTrustedAlgorithmPublishers

TypeRequiredDescription
Array of string✓If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed.
| | | | Type | Required | Description | | Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. | +| Type | Required | Description | +| Array of `string` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed. | |

publisherTrustedAlgorithms

TypeRequiredDescription
Array of publisherTrustedAlgorithms✓If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below).
| | | | Type | Required | Description | | Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). | +| Type | Required | Description | +| Array of `publisherTrustedAlgorithms` | **✓** | If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below). | The `publisherTrustedAlgorithms` is an array of objects with the following structure: diff --git a/core-concepts/images/architecture (1) (1).png b/core-concepts/images/architecture (1) (1).png new file mode 100644 index 00000000..97084f85 Binary files /dev/null and b/core-concepts/images/architecture (1) (1).png differ diff --git a/core-concepts/images/datanft-and-datatoken (1) (1).png b/core-concepts/images/datanft-and-datatoken (1) (1).png new file mode 100644 index 00000000..2a0c79fb Binary files /dev/null and b/core-concepts/images/datanft-and-datatoken (1) (1).png differ diff --git a/core-concepts/images/ddo-flow (1) (1).png b/core-concepts/images/ddo-flow (1) (1).png new file mode 100644 index 00000000..fab9a027 Binary files /dev/null and b/core-concepts/images/ddo-flow (1) (1).png differ diff --git a/core-concepts/images/dynamic-asset-pricing (1) (1).png b/core-concepts/images/dynamic-asset-pricing (1) (1).png new file mode 100644 index 00000000..156114e3 Binary files /dev/null and b/core-concepts/images/dynamic-asset-pricing (1) (1).png differ diff --git a/core-concepts/images/fixed-asset-pricing (1) (1).png b/core-concepts/images/fixed-asset-pricing (1) (1).png new file mode 100644 index 00000000..cc55112a Binary files /dev/null and b/core-concepts/images/fixed-asset-pricing (1) (1).png differ diff --git a/core-concepts/images/free-asset-pricing (1) (1).png b/core-concepts/images/free-asset-pricing (1) (1).png new file mode 100644 index 00000000..8876bc31 Binary files /dev/null and b/core-concepts/images/free-asset-pricing (1) (1).png differ diff --git a/core-concepts/images/use-case (1) (1).png b/core-concepts/images/use-case (1) (1).png new file mode 100644 index 00000000..a581963f Binary files /dev/null and b/core-concepts/images/use-case (1) (1).png differ