mirror of
https://github.com/oceanprotocol/docs.git
synced 2024-11-26 19:49:26 +01:00
313 lines
14 KiB
Markdown
313 lines
14 KiB
Markdown
# Deploying Provider
|
|
|
|
### About Provider
|
|
|
|
Provider encrypts the URL and metadata during publishing and decrypts the URL when the dataset is downloaded or a compute job is started. It enables access to the data assets by streaming data (and never the URL). It performs checks on-chain for buyer permissions and payments. It also provides compute services (connects to a C2D environment).
|
|
|
|
Provider is a multichain component, meaning that it can handle these tasks on multiple chains with the proper configurations. The source code of Provider can be accessed from [here](https://github.com/oceanprotocol/provider).
|
|
|
|
As mentioned in the Setup a Server document, all Ocean components can be deployed in two types of configurations: simple, based on Docker Engine and Docker Compose, and complex, based on Kubernetes with Docker Engine. In this document, we will present how to deploy Provider in each of these configurations.
|
|
|
|
|
|
## Deploying Provider using Docker Engine and Docker Compose
|
|
|
|
In this guide, we will deploy Provider for two chains: Goerli (Ethereum test network) and Mumbai (Polygon test network). Therefore, please note that in the following configuration files, "5" and "80001" are the chain IDs for Goerli and Mumbai respectively.
|
|
|
|
|
|
### Prerequisites
|
|
|
|
* A server for hosting Provider. See [this guide](setup-server.md) for how to create a server;
|
|
* Docker Compose and Docker Engine are installed and configured on the server. See [this guide](setup-server.md#install-docker-engine-and-docker-compose) for how to install these products.
|
|
* The RPC URLs and API keys for each of the networks to which the Provider will be connected. See [this guide](../developers/obtaining-api-keys-for-blockchain-access.md) for how to obtain the URL and the API key.
|
|
* The private key which will be used by Provider to encrypt/decrypt URLs.
|
|
|
|
### Steps
|
|
|
|
The steps to deploy the Provider using Docker Engine and Docker Compose are:
|
|
|
|
1. [Create the /etc/docker/compose/provider/docker-compose.yml file](deploying-provider.md#1-create-the-etcdockercomposeproviderdocker-composeyml-file)
|
|
2. [Create the /etc/systemd/system/docker-compose@provider.service file](deploying-provider.md#2-create-the-etcsystemdsystemdocker-composeproviderservice-file)
|
|
3. [Reload the systemd manager configuration](deploying-provider.md#3.-reload-the-systemd-manager-configuration)
|
|
4. [Start the Provider service](deploying-provider.md#4.-start-the-provider-service)
|
|
5. [Check the service's status](deploying-provider.md#5.-check-the-services-status)
|
|
6. [Confirm the Provider is accessible](deploying-provider.md#6.-confirm-the-provider-is-accessible)
|
|
7. [Check Provider service logs](deploying-provider.md#7.-check-provider-service-logs)
|
|
|
|
|
|
#### 1. Create the /etc/docker/compose/provider/docker-compose.yml file
|
|
|
|
From a terminal console, create /etc/docker/compose/provider/docker-compose.yml file, then copy and paste the following content to it. Check the comments in the file and replace the fields with the specific values of your implementation.
|
|
|
|
```yaml
|
|
version: '3'
|
|
services:
|
|
provider:
|
|
image: oceanprotocol/provider-py:latest =>(check on https://hub.docker.com/r/oceanprotocol/provider-py for specific tag)
|
|
container_name: provider
|
|
restart: on-failure
|
|
ports:
|
|
- 8030:8030
|
|
networks:
|
|
backend:
|
|
environment:
|
|
ARTIFACTS_PATH: "/ocean-contracts/artifacts"
|
|
NETWORK_URL: '{"5":"https://goerli.infura.io/v3/<your INFURA project id>","80001":"https://polygon-mumbai.infura.io/v3/<your INFURA project id>"}'
|
|
PROVIDER_PRIVATE_KEY: '{"5":"<your private key>","80001":"<your private key"}'
|
|
LOG_LEVEL: DEBUG
|
|
OCEAN_PROVIDER_URL: 'http://0.0.0.0:8030'
|
|
OCEAN_PROVIDER_WORKERS: "1"
|
|
IPFS_GATEWAY: "< your IPFS gateway >"
|
|
OCEAN_PROVIDER_TIMEOUT: "9000"
|
|
OPERATOR_SERVICE_URL: "https://stagev4.c2d.oceanprotocol.com" => (use custom value for Operator Service URL)
|
|
AQUARIUS_URL: "http//localhost:5000" => (use custom value Aquarius URL)
|
|
REQUEST_TIMEOUT: "10"
|
|
networks:
|
|
backend:
|
|
driver: bridge
|
|
```
|
|
|
|
|
|
#### 2. Create the _/etc/systemd/system/docker-compose@provider.service_ file
|
|
|
|
Create the _/etc/systemd/system/docker-compose@provider.service_ file then copy and paste the following content to it. This example file could be customized if needed.
|
|
|
|
```
|
|
[Unit]
|
|
Description=%i service with docker compose
|
|
Requires=docker.service
|
|
After=docker.service
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
RemainAfterExit=true
|
|
Environment="PROJECT=ocean"
|
|
WorkingDirectory=/etc/docker/compose/%i
|
|
ExecStartPre=/usr/bin/env docker-compose -p $PROJECT pull
|
|
ExecStart=/usr/bin/env docker-compose -p $PROJECT up -d
|
|
ExecStop=/usr/bin/env docker-compose -p $PROJECT stop
|
|
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
|
|
#### 3. Reload the systemd manager configuration
|
|
|
|
Run the following command to reload the systemd manager configuration
|
|
|
|
```bash
|
|
sudo systemctl daemon-reload
|
|
```
|
|
|
|
Optionally, you can enable the services to start at boot, using the following command:
|
|
|
|
```bash
|
|
sudo systemctl enable docker-compose@provider.service
|
|
```
|
|
|
|
|
|
#### 4. Start the Provider service
|
|
|
|
To start the Provider service, run the following command:
|
|
|
|
```bash
|
|
sudo systemctl start docker-compose@provider.service
|
|
```
|
|
|
|
|
|
#### 5. Check the service's status
|
|
|
|
Check the status of the service by running the following command. The output of the command should be similar to the one presented here.
|
|
|
|
```bash
|
|
$ sudo systemctl status docker-compose@provider.service
|
|
● docker-compose@provider.service - provider service with docker compose
|
|
Loaded: loaded (/etc/systemd/system/docker-compose@provider.service; disabled; vendor preset: enabled)
|
|
Active: active (exited) since Wed 2023-06-14 09:41:53 UTC; 20s ago
|
|
Process: 4118 ExecStartPre=/usr/bin/env docker-compose -p $PROJECT pull (code=exited, status=0/SUCCESS)
|
|
Process: 4126 ExecStart=/usr/bin/env docker-compose -p $PROJECT up -d (code=exited, status=0/SUCCESS)
|
|
Main PID: 4126 (code=exited, status=0/SUCCESS)
|
|
CPU: 93ms
|
|
|
|
Jun 14 09:41:52 testvm systemd[1]: Starting provider service with docker compose...
|
|
Jun 14 09:41:52 testvm env[4118]: provider Pulling
|
|
Jun 14 09:41:53 testvm env[4118]: provider Pulled
|
|
Jun 14 09:41:53 testvm env[4126]: Container provider Created
|
|
Jun 14 09:41:53 testvm env[4126]: Container provider Starting
|
|
Jun 14 09:41:53 testvm env[4126]: Container provider Started
|
|
Jun 14 09:41:53 testvm systemd[1]: Finished provider service with docker compose.
|
|
```
|
|
|
|
|
|
#### 6. Confirm the Provider is accessible
|
|
|
|
Once started, the Provider service is accessible on `localhost` port 8030/tcp. Run the following command to access the Provider. The output should be similar to the one displayed here.
|
|
|
|
```bash
|
|
$ curl localhost:8030
|
|
{"chainIds":[5,80001],"providerAddresses":{"5":"0x00c6A0BC5cD0078d6Cd0b659E8061B404cfa5704","80001":"0x4256Df50c94D9a7e04610976cde01aED91eB531E"},"serviceEndpoints":{"computeDelete":["DELETE","/api/services/compute"],"computeEnvironments":["GET","/api/services/computeEnvironments"],"computeResult":["GET","/api/services/computeResult"],"computeStart":["POST","/api/services/compute"],"computeStatus":["GET","/api/services/compute"],"computeStop":["PUT","/api/services/compute"],"create_auth_token":["GET","/api/services/createAuthToken"],"decrypt":["POST","/api/services/decrypt"],"delete_auth_token":["DELETE","/api/services/deleteAuthToken"],"download":["GET","/api/services/download"],"encrypt":["POST","/api/services/encrypt"],"fileinfo":["POST","/api/services/fileinfo"],"initialize":["GET","/api/services/initialize"],"initializeCompute":["POST","/api/services/initializeCompute"],"nonce":["GET","/api/services/nonce"],"validateContainer":["POST","/api/services/validateContainer"]},"software":"Provider","version":"2.0.2"}
|
|
```
|
|
|
|
|
|
#### 7. Check Provider service logs
|
|
|
|
If needed, use docker CLI to check provider service logs.
|
|
|
|
First, identify the container id:
|
|
|
|
```bash
|
|
$ docker ps
|
|
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
|
594415b13f8c oceanprotocol/provider-py:v2.0.2 "/ocean-provider/doc…" 12 minutes ago Up About a minute 0.0.0.0:8030->8030/tcp, :::8030->8030/tcp provider
|
|
|
|
```
|
|
|
|
Then, check the logs from the Provider's docker container:
|
|
|
|
```bash
|
|
$ docker logs --follow provider
|
|
[2023-06-14 09:31:02 +0000] [8] [INFO] Starting gunicorn 20.0.4
|
|
[2023-06-14 09:31:02 +0000] [8] [INFO] Listening at: http://0.0.0.0:8030 (8)
|
|
[2023-06-14 09:31:02 +0000] [8] [INFO] Using worker: sync
|
|
[2023-06-14 09:31:02 +0000] [10] [INFO] Booting worker with pid: 10
|
|
2023-06-14 09:31:02 594415b13f8c rlp.codec[10] DEBUG Consider installing rusty-rlp to improve pyrlp performance with a rust based backend
|
|
2023-06-14 09:31:12 594415b13f8c ocean_provider.run[10] INFO incoming request = http, GET, 172.18.0.1, /?
|
|
2023-06-14 09:31:12 594415b13f8c ocean_provider.run[10] INFO root endpoint called
|
|
2023-06-14 09:31:12 594415b13f8c ocean_provider.run[10] INFO root endpoint response = <Response 1031 bytes [200 OK]>
|
|
[2023-06-14 09:41:53 +0000] [8] [INFO] Starting gunicorn 20.0.4
|
|
[2023-06-14 09:41:53 +0000] [8] [INFO] Listening at: http://0.0.0.0:8030 (8)
|
|
[2023-06-14 09:41:53 +0000] [8] [INFO] Using worker: sync
|
|
[2023-06-14 09:41:53 +0000] [10] [INFO] Booting worker with pid: 10
|
|
2023-06-14 09:41:54 594415b13f8c rlp.codec[10] DEBUG Consider installing rusty-rlp to improve pyrlp performance with a rust based backend
|
|
2023-06-14 09:42:40 594415b13f8c ocean_provider.run[10] INFO incoming request = http, GET, 172.18.0.1, /?
|
|
2023-06-14 09:42:40 594415b13f8c ocean_provider.run[10] INFO root endpoint called
|
|
2023-06-14 09:42:40 594415b13f8c ocean_provider.run[10] INFO root endpoint response = <Response 1031 bytes [200 OK]>
|
|
|
|
```
|
|
|
|
|
|
## Deploying Provider using Kubernetes with Docker Engine
|
|
|
|
|
|
In this example, we will run Provider as a Kubernetes deployment resource. We will deploy Provider for two chains: Goerli (Ethereum test network) and Mumbai (Polygon test network). Therefore, please note that in the following configuration files, "5" and "80001" are the chain IDs for Goerli and Mumbai respectively.
|
|
|
|
### Prerequisites
|
|
|
|
* A server for hosting Ocean Marketplace. See [this guide](setup-server.md) for how to create a server;
|
|
* Kubernetes with Docker Engine is installed and configured on the server. See [this chapter](setup-server.md#install-kubernetes-with-docker-engine) for information on installing Kubernetes.
|
|
* The RPC URLs and API keys for each of the networks to which the Provider will be connected. See [this guide](../developers/obtaining-api-keys-for-blockchain-access.md) for how to obtain the URL and the API key.
|
|
* The private key that will be used by Provider to encrypt/decrypt URLs.
|
|
* Aquarius is up and running
|
|
|
|
### Steps
|
|
|
|
The steps to deploy the Provider in Kubernetes are:
|
|
|
|
[1. Create a YAML file for Provider configuration.](deploying-provider.md#1-create-a-yaml-file-for-provider-configuration)
|
|
|
|
[2. Deploy the configuration.](deploying-provider.md#2.-deploy-the-configuration)
|
|
|
|
[3. Create a Kubernetes service.](deploying-provider.md#3.-create-a-kubernetes-service)
|
|
|
|
|
|
#### 1. Create a YAML file for Provider configuration.
|
|
|
|
From a terminal window, create a YAML file (in our example the file is named provider-deploy.yaml) then copy and paste the following content. Check the comments in the file and replace the fields with the specific values of your implementation (RPC URLs, the private key etc.).
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
labels:
|
|
app: provider
|
|
name: provider
|
|
spec:
|
|
progressDeadlineSeconds: 2147483647
|
|
replicas: 1
|
|
revisionHistoryLimit: 2147483647
|
|
selector:
|
|
matchLabels:
|
|
app: provider
|
|
strategy:
|
|
rollingUpdate:
|
|
maxSurge: 25%
|
|
maxUnavailable: 25%
|
|
type: RollingUpdate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: provider
|
|
spec:
|
|
containers:
|
|
- env:
|
|
- name: ARTIFACTS_PATH
|
|
value: /ocean-provider/artifacts
|
|
- name: NETWORK_URL
|
|
value: |
|
|
{"5":"https://goerli.infura.io/v3/<your INFURA project id>","80001":"https://polygon-mumbai.infura.io/v3/<your INFURA project id>"}
|
|
- name: PROVIDER_PRIVATE_KEY
|
|
value: |
|
|
{"5":"<your private key>","80001":"<your private key>"}
|
|
- name: LOG_LEVEL
|
|
value: DEBUG
|
|
- name: OCEAN_PROVIDER_URL
|
|
value: http://0.0.0.0:8030
|
|
- name: OCEAN_PROVIDER_WORKERS
|
|
value: "4"
|
|
- name: IPFS_GATEWAY
|
|
value: < your IPFS gateway >
|
|
- name: OCEAN_PROVIDER_TIMEOUT
|
|
value: "9000"
|
|
- name: OPERATOR_SERVICE_URL
|
|
value: < Operator service URL>
|
|
- name: AQUARIUS_URL
|
|
value: < Aquarius URL >
|
|
- name: UNIVERSAL_PRIVATE_KEY
|
|
value: <your universal private key>
|
|
- name: REQUEST_TIMEOUT
|
|
value: "10"
|
|
image: oceanprotocol/provider-py:latest => (check on https://hub.docker.com/r/oceanprotocol/provider-py for specific tag)
|
|
imagePullPolicy: Always
|
|
name: provider
|
|
ports:
|
|
- containerPort: 8030
|
|
protocol: TCP
|
|
resources:
|
|
limits:
|
|
cpu: 500m
|
|
memory: 700Mi
|
|
requests:
|
|
cpu: 500m
|
|
memory: 700Mi
|
|
terminationMessagePath: /dev/termination-log
|
|
terminationMessagePolicy: File
|
|
dnsPolicy: ClusterFirst
|
|
restartPolicy: Always
|
|
schedulerName: default-scheduler
|
|
terminationGracePeriodSeconds: 30
|
|
```
|
|
|
|
Tip: before deployment, you can [validate](https://github.com/instrumenta/kubeval) the yaml file.
|
|
|
|
|
|
#### 2. Deploy the configuration
|
|
|
|
Deploy the configuration in Kubernetes using the following commands.
|
|
|
|
```bash
|
|
kubectl config set-context --current --namespace ocean
|
|
kubectl apply -f provider-deploy.yaml
|
|
deployment.apps/provider created
|
|
|
|
kubectl get pod -l app=provider
|
|
NAME READY STATUS RESTARTS AGE
|
|
provider-865cb8cf9d-r9xm4 1/1 Running 0 67s
|
|
```
|
|
|
|
|
|
#### 3. Create a Kubernetes service
|
|
|
|
The next step is to create a Kubernetes service (eg. ClusterIP, NodePort, Loadbalancer, ExternalName) for this deployment, depending on the environment specifications. Follow [this link](https://kubernetes.io/docs/concepts/services-networking/service/) for details on how to create a Kubernetes service.
|
|
|