mirror of
https://github.com/oceanprotocol/docs.git
synced 2024-11-26 19:49:26 +01:00
GITBOOK-606: Compute to data section improvements
This commit is contained in:
Jamie Hewitt
gitbook-bot
parent
48c7ee5e20
commit
b0acf2324f
@ -32,7 +32,7 @@
|
||||
* [Harvest More Yield Data Farming](user-guides/how-to-data-farm.md)
|
||||
* [Claim Rewards Data Farming](user-guides/claim-ocean-rewards.md)
|
||||
* [Liquidity Pools \[deprecated\]](user-guides/remove-liquidity-pools.md)
|
||||
* [💻 Developers](developers/README.md)
|
||||
* [👨💻 Developers](developers/README.md)
|
||||
* [Architecture Overview](developers/architecture.md)
|
||||
* [Contracts](developers/contracts/README.md)
|
||||
* [Data NFTs](developers/contracts/data-nfts.md)
|
||||
@ -122,4 +122,4 @@
|
||||
* [🤝 Contribute](contribute/README.md)
|
||||
* [Partners & Collaborators](contribute/projects-using-ocean.md)
|
||||
* [Contributor Code of Conduct](contribute/code-of-conduct.md)
|
||||
* [Legal Requirements](contribute/legal-reqs.md)
|
||||
* [Legal Requirements](contribute/legal-reqs.md)
|
||||
|
||||
@ -29,13 +29,13 @@ For all the super sleuths out there, you may be able to earn a bounty for report
|
||||
|
||||
### Suggest a new feature 🤔💭
|
||||
|
||||
Use the _Issues_ section of each repository and select _Feature request_ to suggest and discuss any features you would like to see added.
|
||||
Use the _Issues_ section of each repository and select _`Feature request`_ to suggest and discuss any features you would like to see added.
|
||||
|
||||
As with bug reports, don't forget to search existing open + closed issues + PRs to see if something has already been suggested.
|
||||
|
||||
### Improve core software
|
||||
|
||||
It takes a tribe of awesome coders to build our tech stack, and you're invited to pitch in 😊 We'd love to have you contribute to any repository within the `oceanprotocol` GitHub organization!
|
||||
It takes a tribe of awesome coders to build our tech stack, and you're invited to pitch in 😊 We'd love to have you contribute to any repository within the `oceanprotocol` [GitHub](https://github.com/oceanprotocol) organization!
|
||||
|
||||
Before you start coding, please follow these basic guidelines:
|
||||
|
||||
@ -59,7 +59,7 @@ Depending on the release management of each repository, your contribution will b
|
||||
|
||||
Except for GitHub, you can chat with most Ocean Protocol core developers in our [Discord](https://discord.gg/TnXjkR5) if you have further development questions.
|
||||
|
||||
### Develop an dApp or integration on top of Ocean Protocol
|
||||
### Develop a dApp or integration on top of Ocean Protocol
|
||||
|
||||
We LOVE builders of dApps on Ocean! Nothing makes us feel prouder than seeing you create awesome things with our open-source tools.
|
||||
|
||||
@ -71,7 +71,7 @@ Our docs repo can always be improved. If you found a mistake or have an improvem
|
||||
|
||||
### Apply for a developer job
|
||||
|
||||
Do you REALLY love building on Ocean Protocol? Consider joining us full time! Our openings are listed at [https://github.com/oceanprotocol/jobs](https://github.com/oceanprotocol/jobs).
|
||||
Do you REALLY love building on Ocean Protocol? Consider joining us full-time! Our openings are listed at [https://github.com/oceanprotocol/jobs](https://github.com/oceanprotocol/jobs).
|
||||
|
||||
### Get funding
|
||||
|
||||
@ -80,6 +80,4 @@ You might be able to get funding from us for contributing to our core software,
|
||||
* [**Shipyard**](https://oceanprotocol.com/shipyard) (Ocean curated grants).
|
||||
* [**Data Challenges**](https://oceanprotocol.com/bounties) (rewards for publishing algorithms and datasets).
|
||||
|
||||
### Talk to us!
|
||||
|
||||
Check our our [Community Page](https://www.oceanprotocol.com/community) for our social media links where you can join the buzz around Ocean or chat with us directly 😊 Toodles!
|
||||
Check our [Community Page](https://www.oceanprotocol.com/community) for our social media links where you can join the buzz around Ocean or chat with us directly 😊 Toodles!
|
||||
|
||||
@ -7,7 +7,7 @@ cover: ../.gitbook/assets/cover/developer_banner.png
|
||||
coverY: 0
|
||||
---
|
||||
|
||||
# 💻 Developers
|
||||
# 👨💻 Developers
|
||||
|
||||
With Ocean, crypto wallets transform into magical data wallets, where your data can roam freely and securely. Crypto exchanges? Well, they've taken on a new role as data marketplaces, where you can showcase and trade your valuable data treasures. And hold on tight because DAOs are here to create epic data co-ops, where collaboration and innovation reign supreme! 🤝
|
||||
|
||||
|
||||
@ -6,17 +6,13 @@ description: Monetise your data while preserving privacy
|
||||
|
||||
### Introduction
|
||||
|
||||
There are many datasets that are too sensitive to be sold, such as health records or other personal information. Compute-to-Data enables you to make money from these datasets while keeping the data private. Rather than selling the raw data, you can sell compute access to the private data. 
|
||||
Certain datasets, such as health records and personal information, are too sensitive to be directly sold. However, Compute-to-Data offers a solution that allows you to monetize these datasets while keeping the data private. Instead of selling the raw data itself, you can offer compute access to the private data. This means you have control over which algorithms can be run on your dataset. For instance, if you possess sensitive health records, you can permit an algorithm to calculate the average age of a patient without revealing any other details.
|
||||
|
||||
You decide which algorithms you allow to be run on your dataset. So, for example, if you own sensitive health records you could allow an algorithm that outputs the average age of a patient but no other information.
|
||||
Compute-to-Data effectively resolves the tradeoff between leveraging the benefits of private data and mitigating the risks associated with data exposure. It enables the data to remain on-premise while granting third parties the ability to perform specific compute tasks on it, yielding valuable results like statistical analysis or AI model development.
|
||||
|
||||
Compute-to-data resolves the tradeoff between the benefits of using private data, and the risks of exposing it. It lets the data stay on-premise, yet allows 3rd parties to run specific compute jobs on it to get useful compute results like averaging or building an AI model.
|
||||
Private data holds immense value as it can significantly enhance research and business outcomes. However, concerns regarding privacy and control often impede its accessibility. Compute-to-Data addresses this challenge by granting specific access to the private data without directly sharing it. This approach finds utility in various domains, including scientific research, technological advancements, and marketplaces where private data can be securely sold while preserving privacy. Companies can seize the opportunity to monetize their data assets while ensuring the utmost protection of sensitive information.
|
||||
|
||||
The most valuable data is private data — using it can improve research and business outcomes. But concerns over privacy and control make it hard to access. With Compute-to-Data, private data isn’t directly shared but rather specific access to it is granted.
|
||||
|
||||
It can be used for data sharing in science or technology contexts, or in marketplaces for selling private data while preserving privacy, as an opportunity for companies to monetize their data assets.
|
||||
|
||||
Private data can help research, leading to life-altering innovations in science and technology. For example, more data improves the predictive accuracy of modern Artificial Intelligence (AI) models. Private data is often considered the most valuable data because it’s so hard to get at, and using it can lead to potentially big payoffs.
|
||||
Private data has the potential to drive groundbreaking discoveries in science and technology, with increased data improving the predictive accuracy of modern AI models. Due to its scarcity and the challenges associated with accessing it, private data is often regarded as the most valuable. By utilizing private data through Compute-to-Data, significant rewards can be reaped, leading to transformative advancements and innovative breakthroughs.
|
||||
|
||||
We suggest reading these guides to get an understanding on how compute-to-data works:
|
||||
|
||||
|
||||
@ -8,7 +8,9 @@ description: Specification of compute options for assets in Ocean Protocol.
|
||||
|
||||
### Compute Options
|
||||
|
||||
An asset with a service of `type` `compute` has the following additional attributes under the `compute` object. This object is required if the asset is of `type` `compute`, but can be omitted for `type` of `access`.
|
||||
An asset categorized as a `compute type` incorporates additional attributes under the `compute object`. 
|
||||
|
||||
These attributes are specifically relevant to assets that fall within the compute category and are not required for assets classified under the `access type`. However, if an asset is designated as `compute`, it is essential to include these attributes to provide comprehensive information about the compute service associated with the asset.
|
||||
|
||||
<table><thead><tr><th width="404.3333333333333">Attribute</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><strong><code>allowRawAlgorithm</code></strong>*</td><td><code>boolean</code></td><td>If <code>true</code>, any passed raw text will be allowed to run. Useful for an algorithm drag & drop use case, but increases risk of data escape through malicious user input. Should be <code>false</code> by default in all implementations.</td></tr><tr><td><strong><code>allowNetworkAccess</code></strong>*</td><td><code>boolean</code></td><td>If <code>true</code>, the algorithm job will have network access.</td></tr><tr><td><strong><code>publisherTrustedAlgorithmPublishers</code></strong>*</td><td>Array of <code>string</code></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. If not empty any algo published by the defined publishers is allowed.</td></tr><tr><td><strong><code>publisherTrustedAlgorithms</code></strong>*</td><td>Array of <code>publisherTrustedAlgorithms</code></td><td>If not defined, then any published algorithm is allowed. If empty array, then no algorithm is allowed. Otherwise only the algorithms defined in the array are allowed. (see below).</td></tr></tbody></table>
|
||||
|
||||
@ -16,7 +18,9 @@ An asset with a service of `type` `compute` has the following additional attribu
|
||||
|
||||
### Trusted Algorithms
|
||||
|
||||
The `publisherTrustedAlgorithms` is an array of objects with the following structure:
|
||||
The `publisherTrustedAlgorithms` is an array of objects that specifies algorithm permissions. It controls which algorithms can be used for computation. If not defined, any published algorithm is allowed. If the array is empty, no algorithms are allowed. However, if the array is not empty, only algorithms published by the defined publishers are permitted. 
|
||||
|
||||
The structure of each object within the `publisherTrustedAlgorithms` array is as follows:
|
||||
|
||||
<table><thead><tr><th width="289.3333333333333">Attribute</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td><strong><code>did</code></strong></td><td><code>string</code></td><td>The DID of the algorithm which is trusted by the publisher.</td></tr><tr><td><strong><code>filesChecksum</code></strong></td><td><code>string</code></td><td>Hash of algorithm's files (as <code>string</code>).</td></tr><tr><td><strong><code>containerSectionChecksum</code></strong></td><td><code>string</code></td><td>Hash of algorithm's image details (as <code>string</code>).</td></tr></tbody></table>
|
||||
|
||||
|
||||
@ -9,15 +9,18 @@ description: >-
|
||||
|
||||
### Overview
|
||||
|
||||
An algorithm in the Ocean Protocol stack is another asset type, in addition to data sets. An algorithm for Compute to Data is composed of the following:
|
||||
\
|
||||
In the Ocean Protocol stack, algorithms are recognized as distinct asset types, alongside datasets. When it comes to Compute-to-Data, an algorithm comprises the following key components:
|
||||
|
||||
* an algorithm code
|
||||
* a Docker image (base image + tag)
|
||||
* an entry point
|
||||
* **Algorithm Code**: The algorithm code refers to the specific instructions and logic that define the computational steps to be executed on a dataset. It encapsulates the algorithms' functionalities, calculations, and transformations.
|
||||
* **Docker Image**: A Docker image plays a crucial role in encapsulating the algorithm code and its runtime dependencies. It consists of a base image, which provides the underlying environment for the algorithm, and a corresponding tag that identifies a specific version or variant of the image.
|
||||
* **Entry Point**: The entry point serves as the starting point for the algorithm's execution within the compute environment. It defines the initial actions to be performed when the algorithm is invoked, such as loading necessary libraries, setting up configurations, or calling specific functions.
|
||||
|
||||
Collectively, these components form the foundation of an algorithm in the context of Compute-to-Data.
|
||||
|
||||
### Environment
|
||||
|
||||
When creating an algorithm asset in Ocean Protocol, the additional `algorithm` object needs to be included in its metadata service to define the Docker container environment:
|
||||
When creating an algorithm asset in Ocean Protocol, it is essential to include the additional algorithm object in its metadata service. This algorithm object plays a crucial role in defining the Docker container environment associated with the algorithm. By specifying the necessary details within the algorithm object, such as the base image, tags, runtime configurations, and dependencies, the metadata service ensures that the algorithm asset is properly configured for execution within a Docker container.
|
||||
|
||||
<details>
|
||||
|
||||
|
||||
@ -11,9 +11,13 @@ Here's the sequence diagram for starting a new compute job.
|
||||
|
||||
<figure><img src="../../.gitbook/assets/c2d/c2d_compute_job.png" alt=""><figcaption></figcaption></figure>
|
||||
|
||||
The Consumer calls the Provider with `start(did, algorithm, additionalDIDs)`. It returns job id `XXXX`. The Provider oversees the rest of the work. At any point, the Consumer can query the Provider for the job status via `getJobDetails(XXXX)`.
|
||||
The interaction between the Consumer and the Provider follows a specific workflow. To initiate the process, the Consumer contacts the Provider by invoking the `start(did, algorithm, additionalDIDs)` function with parameters such as the data identifier (DID), algorithm, and additional DIDs if required. Upon receiving this request, the Provider generates a unique job identifier (`XXXX`) and returns it to the Consumer. The Provider then assumes the responsibility of overseeing the remaining steps.
|
||||
|
||||
Here's how Provider works. First, it ensures that the Consumer has sent the appropriate datatokens to get access. Then, it calls asks the Operator-Service (a microservice) to start the job, which passes on the request to Operator-Engine (the actual compute system). Operator-Engine runs Kubernetes compute jobs etc as needed. Operator-Engine reports when to Operator-Service when the job has finished.
|
||||
Throughout the computation process, the Consumer has the ability to check the status of the job by making a query to the Provider using the `getJobDetails(XXXX)` function, providing the job identifier (`XXXX`) as a reference.
|
||||
|
||||
Now, let's delve into the inner workings of the Provider. Initially, it verifies whether the Consumer has sent the appropriate datatokens to gain access to the desired data. Once validated, the Provider interacts with the Operator-Service, a microservice responsible for coordinating the job execution. The Provider submits a request to the Operator-Service, which subsequently forwards the request to the Operator-Engine, the actual compute system in operation.
|
||||
|
||||
The Operator-Engine, equipped with functionalities like running Kubernetes compute jobs, carries out the necessary computations as per the requirements. Throughout the computation process, the Operator-Engine informs the Operator-Service of the job's progress. Finally, when the job reaches completion, the Operator-Engine signals the Operator-Service, ensuring that the Provider receives notification of the job's successful conclusion.
|
||||
|
||||
Here's the actors/components:
|
||||
|
||||
@ -30,9 +34,9 @@ Before the flow can begin, these pre-conditions must be met:
|
||||
|
||||
### Access Control using Ocean Provider
|
||||
|
||||
As with the `access` service, the `compute` service requires the **Ocean Provider** as a component handled by Publishers. Ocean Provider is in charge of interacting with users and managing the basics of a Publisher's infrastructure to integrate this infrastructure into Ocean Protocol. The direct interaction with the infrastructure where the data resides happens through this component only.
|
||||
Similar to the `access service`, the `compute service` within Ocean Protocol relies on the [Ocean Provider](../provider/), which is a crucial component managed by Publishers. The role of the Ocean Provider is to facilitate interactions with users and handle the fundamental aspects of a Publisher's infrastructure, enabling seamless integration into the Ocean Protocol ecosystem. It serves as the primary interface for direct interaction with the infrastructure where the data is located.
|
||||
|
||||
Ocean Provider includes the credentials to interact with the infrastructure (initially in cloud providers, but it could be on-premise).
|
||||
The [Ocean Provider](../provider/) encompasses the necessary credentials to establish secure and authorized interactions with the underlying infrastructure. Initially, this infrastructure may be hosted in cloud providers, although it also has the flexibility to extend to on-premise environments if required. By encompassing the necessary credentials, the Ocean Provider ensures the smooth and controlled access to the infrastructure, allowing Publishers to effectively leverage the compute service within Ocean Protocol.
|
||||
|
||||
### Operator Service
|
||||
|
||||
@ -70,20 +74,22 @@ The Operator Engine is in charge of retrieving all the workflows registered in a
|
||||
|
||||
### Pod Configuration
|
||||
|
||||
The Pod-Configuration repository operates in conjunction with the Operator Engine, and it initiates at the beginning of a job. It performs crucial functions that set up the environment for the job execution.
|
||||
The Pod-Configuration repository works hand in hand with the Operator Engine, playing a vital role in the initialization phase of a job. It carries out essential functions that establish the environment for job execution.
|
||||
|
||||
The Pod-Configuration is a node.js script that dynamically manages the environment set-up at the start of a job by the operator-engine. Its role involves fetching and preparing necessary assets and files to ensure a seamless job execution.
|
||||
At the core of the Pod-Configuration is a node.js script that dynamically manages the setup process when a job begins within the operator-engine. Its primary responsibility revolves around fetching and preparing the required assets and files, ensuring a smooth and seamless execution of the job. By meticulously handling the environment configuration, the Pod-Configuration script guarantees that all necessary components are in place, setting the stage for a successful job execution.
|
||||
|
||||
1. **Fetching Dataset Assets**: It fetches the files corresponding to datasets and saves them in the location `/data/inputs/DID/`. The files are named based on their array index ranging from 0 to X, depending on the total number of files associated with the dataset.
|
||||
2. **Fetching Algorithm Files**: The script then retrieves the algorithm files and stores them in the `/data/transformations/` directory. The first file is named 'algorithm', and the subsequent files are indexed from 1 to X, based on the number of files present for the algorithm.
|
||||
3. **Fetching DDOS**: Additionally, the Pod-Configuration fetches Decentralized Document Oriented Storage (DDOS) and saves them to the disk at the location `/data/ddos/`.
|
||||
4. **Error Handling**: In case of any provisioning failures, whether during data fetching or algorithm processing, the script updates the job status in a PostgreSQL database, and logs the relevant error messages.
|
||||
|
||||
Once the Pod-Configuration successfully completes these tasks, it closes and signals the operator-engine to initiate the algorithm pod for the next steps. This repository provides the basis for smooth job processing by effectively managing assets and algorithm files, and handling potential provisioning error.
|
||||
Upon the successful completion of its tasks, the Pod-Configuration gracefully concludes its operations and sends a signal to the operator-engine, prompting the initiation of the algorithm pod for subsequent steps. This repository serves as a fundamental component in ensuring the seamless processing of jobs by efficiently managing assets, algorithm files, and addressing potential provisioning errors. By effectively handling these crucial aspects, the Pod-Configuration establishes a solid foundation for smooth job execution and enables the efficient progression of the overall workflow.
|
||||
|
||||
### Pod Publishing
|
||||
|
||||
Pod Publishing is a command-line utility for processing, logging, and uploading workflow outputs, functioning in conjunction with the Operator Service and Operator Engine within a Kubernetes-based compute infrastructure. The primary functionality divided across three areas:
|
||||
Pod Publishing is a command-line utility that seamlessly integrates with the Operator Service and Operator Engine within a Kubernetes-based compute infrastructure. It serves as a versatile tool for efficiently processing, logging, and uploading workflow outputs. By working in tandem with the Operator Service and Operator Engine, Pod Publishing streamlines the workflow management process, enabling easy and reliable handling of output data generated during computation tasks. Whether it's processing complex datasets or logging crucial information, Pod Publishing simplifies these tasks and enhances the overall efficiency of the compute infrastructure.
|
||||
|
||||
The primary functionality of Pod Publishing can be divided into three key areas:
|
||||
|
||||
1. **Interaction with Operator Service**: Pod Publishing uploads the outputs of compute workflows initiated by the Operator Service to a designated AWS S3 bucket or the InterPlanetary File System (IPFS). It logs all processing steps and updates a PostgreSQL database.
|
||||
2. **Role in Publishing Pod**: Within the compute infrastructure orchestrated by the Operator Engine on Kubernetes (K8s), Pod Publishing is integral to the Publishing Pod. The Publishing Pod handles the creation of new assets in the Ocean Protocol network after a workflow execution.
|
||||
|
||||
@ -7,17 +7,21 @@ description: Datasets and Algorithms
|
||||
|
||||
### Datasets & Algorithms
|
||||
|
||||
With Compute-to-Data, datasets are not allowed to leave the premises of the data holder, only algorithms can be permitted to run on them under certain conditions within an isolated and secure environment. Algorithms are an asset type just like datasets and can be priced in the same way.
|
||||
Compute-to-Data introduces a paradigm where datasets remain securely within the premises of the data holder, ensuring strict data privacy and control. Only authorized algorithms are granted access to operate on these datasets, subject to specific conditions, within a secure and isolated environment. In this context, algorithms are treated as valuable assets, comparable to datasets, and can be priced accordingly. This approach enables data holders to maintain control over their sensitive data while allowing for valuable computations to be performed on them, fostering a balanced and secure data ecosystem.
|
||||
|
||||
Algorithms can be public or private by setting `"attributes.main.type"` value in DDO as follows:
|
||||
To define the accessibility of algorithms, their classification as either public or private can be specified by setting the `attributes.main.type` value in the Decentralized Data Object (DDO): 
|
||||
|
||||
* `"access"` - public. The algorithm can be downloaded, given appropriate datatoken.
|
||||
* `"compute"` - private. The algorithm is only available to use as part of a compute job without any way to download it. The Algorithm must be published on the same Ocean Provider as the dataset it's targeted to run on.
|
||||
|
||||
For each dataset, publishers can choose to allow various permission levels for algorithms to run:
|
||||
This flexibility allows for fine-grained control over algorithm usage, ensuring data privacy and enabling fair pricing mechanisms within the Compute-to-Data framework.
|
||||
|
||||
For each dataset, Publishers have the flexibility to define permission levels for algorithms to execute on their datasets, offering granular control over data access. 
|
||||
|
||||
There are several options available for publishers to configure these permissions:
|
||||
|
||||
* allow selected algorithms, referenced by their DID
|
||||
* allow all algorithms published within a network or marketplace
|
||||
* allow raw algorithms, for advanced use cases circumventing algorithm as an asset type, but most prone to data escape
|
||||
|
||||
All implementations should set permissions to private by default: upon publishing a compute dataset, no algorithms should be allowed to run on it. This is to prevent data escape by a rogue algorithm being written in a way to extract all data from a dataset.
|
||||
All implementations default to `private`, meaning that no algorithms are allowed to run on a compute dataset upon publishing. This precautionary measure helps prevent data leakage by thwarting rogue algorithms that could be designed to extract all data from a dataset. By establishing private permissions as the default setting, publishers ensure a robust level of protection for their data assets and mitigate the risk of unauthorized data access.
|
||||
|
||||
@ -5,9 +5,9 @@ description: >-
|
||||
algorithms in a C2D environment.
|
||||
---
|
||||
|
||||
# Setting up private docker registry
|
||||
# C2D - Private Docker Registry
|
||||
|
||||
The document is intended for a production setup. The tutorial provides the steps to setup a private docker registry on the server for the following scenarios:
|
||||
The document is intended for a production setup. The tutorial provides the steps to set up a private docker registry on the server for the following scenarios:
|
||||
|
||||
* Allow registry access only to the C2D environment.
|
||||
* Anyone can pull the image from the registry but, only authenticated users will push images to the registry.
|
||||
@ -22,10 +22,10 @@ _Note: Please change the domain names to your application-specific domain names.
|
||||
|
||||
#### 1.1 Prerequisites
|
||||
|
||||
* Running docker environment on the linux server.
|
||||
* A docker environment running on a Linux server.
|
||||
* Docker compose is installed.
|
||||
* C2D environment is running.
|
||||
* The domain names is mapped to the server hosting the registry.
|
||||
* The domain names are mapped to the server hosting the registry.
|
||||
|
||||
#### 1.2 Generate certificates
|
||||
|
||||
@ -34,9 +34,9 @@ _Note: Please change the domain names to your application-specific domain names.
|
||||
sudo certbot certonly --standalone --cert-name example.com -d example.com
|
||||
```
|
||||
|
||||
_Note: Do check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._
|
||||
_Note: Check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._
|
||||
|
||||
#### 1.3 Generate password file
|
||||
#### 1.3 Generate a password file
|
||||
|
||||
Replace content in `<>` with appropriate content.
|
||||
|
||||
@ -48,7 +48,7 @@ docker run \
|
||||
|
||||
#### 1.4 Docker compose template file for registry
|
||||
|
||||
Copy the below yml content to `docker-compose.yml` file and replace content in `<>`.
|
||||
Copy the below `yml` content to `docker-compose.yml` file and replace content in `<>`.
|
||||
|
||||
```yml
|
||||
version: '3'
|
||||
@ -114,9 +114,9 @@ http {
|
||||
}
|
||||
```
|
||||
|
||||
#### 1.6 Create kubernetes secret in C2D server
|
||||
#### 1.6 Create Kubernetes secret in C2D server
|
||||
|
||||
Login into Compute-to-data enviroment and run the following command with appropriate credentials:
|
||||
Login into the compute-to-data enviroment and run the following command with the appropriate credentials:
|
||||
|
||||
```bash
|
||||
kubectl create secret docker-registry regcred --docker-server=example.com --docker-username=<username> --docker-password=<password> --docker-email=<email_id> -n ocean-compute
|
||||
@ -124,7 +124,7 @@ kubectl create secret docker-registry regcred --docker-server=example.com --dock
|
||||
|
||||
#### 1.7 Update operator-engine configuration
|
||||
|
||||
Add `PULL_SECRET` property with value `regcred` in the [operator.yml](https://github.com/oceanprotocol/operator-engine/blob/main/kubernetes/operator.yml) file of operator-engine configuration. For more detials on operator-engine properties refer this [link](https://github.com/oceanprotocol/operator-engine/blob/177ca7185c34aa2a503afbe026abb19c62c69e6d/README.md?plain=1#L106)
|
||||
Add `PULL_SECRET` property with value `regcred` in the [operator.yml](https://github.com/oceanprotocol/operator-engine/blob/main/kubernetes/operator.yml) file of operator-engine configuration. For more details on operator-engine properties refer to the [operator-engine readme](https://github.com/oceanprotocol/operator-engine/blob/v4main/README.md).
|
||||
|
||||
Apply updated operator-engine configuration.
|
||||
|
||||
@ -133,20 +133,20 @@ kubectl config set-context --current --namespace ocean-compute
|
||||
kubectl apply -f operator-engine/kubernetes/operator.yml
|
||||
```
|
||||
|
||||
### Steup 2: Allow anyonymous `pull` operations
|
||||
### Steup 2: Allow anonymous `pull` operations
|
||||
|
||||
To implement this use case, 2 domains will be required:
|
||||
|
||||
* **example.com**: This domain will allow image push/pull operations only to the authenticated users.
|
||||
* **example.com**: This domain will only allow image push/pull operations from authenticated users.
|
||||
* **readonly.example.com**: This domain will allow only image pull operations
|
||||
|
||||
_Note: Please change the domain names to your application-specific domain names._
|
||||
|
||||
#### 2.1 Prerequisites
|
||||
|
||||
* Running docker environment on the linux server.
|
||||
* Running docker environment on the Linux server.
|
||||
* Docker compose is installed.
|
||||
* 2 domain names is mapped to the same server IP address.
|
||||
* 2 domain names are mapped to the same server IP address.
|
||||
|
||||
#### 2.2 Generate certificates
|
||||
|
||||
@ -158,7 +158,7 @@ sudo certbot certonly --standalone --cert-name readonly.example.com -d readonly.
|
||||
|
||||
_Note: Do check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._
|
||||
|
||||
#### 2.3 Generate password file
|
||||
#### 2.3 Generate a password file
|
||||
|
||||
Replace content in `<>` with appropriate content.
|
||||
|
||||
@ -170,7 +170,7 @@ docker run \
|
||||
|
||||
#### 2.4 Docker compose template file for registry
|
||||
|
||||
Copy the below yml content to `docker-compose.yml` file and replace content in `<>`. Here, we will be creating two services of the docker registry so that anyone can `pull` the images from the registry but, only authenticated users can `push` the images.
|
||||
Copy the below `yml` content to `docker-compose.yml` file and replace content in `<>`. Here, we will be creating two services of the docker registry so that anyone can `pull` the images from the registry but, only authenticated users can `push` the images.
|
||||
|
||||
```yml
|
||||
version: '3'
|
||||
@ -305,7 +305,7 @@ docker image pull readonly.example.com/my-algo:latest
|
||||
|
||||
#### Next step
|
||||
|
||||
You can publish an algorithm asset with the metadata containing registry URL, image, and tag information to enable users to run C2D jobs.
|
||||
You can publish an algorithm asset with the metadata containing the registry URL, image, and tag information to enable users to run C2D jobs.
|
||||
|
||||
### Further references
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user