oceanprotocol/docs
1
0
Fork 0
mirror of https://github.com/oceanprotocol/docs.git synced 2024-11-26 19:49:26 +01:00
Code Releases Activity

Added C2D workflow details

Browse Source
This commit is contained in:
Ana Loznianu 2023-10-19 15:07:12 +03:00
parent 4147bba980
commit a3d05219e9
3 changed files with 63 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
SUMMARY.md
View File

@ -92,7 +92,7 @@
- [Compute to data](developers/compute-to-data/README.md)
- [Architecture](developers/compute-to-data/compute-to-data-architecture.md)
- [Datasets & Algorithms](developers/compute-to-data/compute-to-data-datasets-algorithms.md)
- [Jobs](developers/compute-to-data/compute-job.md)
- [Jobs](developers/compute-to-data/compute-workflow.md)
- [Writing Algorithms](developers/compute-to-data/compute-to-data-algorithms.md)
- [Compute Options](developers/compute-to-data/compute-options.md)
- [Aquarius](developers/aquarius/README.md)

9
developers/compute-to-data/compute-job.md
View File

@ -1,9 +0,0 @@
---
title: Compute Jobs
section: developers
description: Specification of compute options for assets in Ocean Protocol.
---
<figure><img src="../../.gitbook/assets/c2d/c2d_compute_job.png" alt=""><figcaption><p>Compute detailed flow diagram</p></figcaption></figure>

62
developers/compute-to-data/compute-workflow.md Normal file
View File

@ -0,0 +1,62 @@
---
title: Compute Workflow
section: developers
description: Understanding the Compute-to-Data (C2D) Workflow
---
🚀 Now that we've introduced the key actors and provided an overview of the process, it's time to delve into the nitty-gritty of the compute workflow.🌟 We'll dissect each step, examining the inner workings of Compute-to-Data (C2D). From data selection to secure computations, we'll leave no stone unturned in this exploration. 💫 Get ready to uncover the intricacies of C2D and how it utilizes various actors and components for a blend of privacy and efficiency.
Ready for the deep dive? Let's get started! 🌊
## Starting a C2D Job
1. The consumer initiates a compute-to-data job by selecting the desired data asset and algorithm.
2. The provider checks the orders on the blockchain.
3. If the orders are valid, the provider can commence the compute flow.
4. The provider informs the consumer of the job number's successful creation.
5. With the job ID and confirmation of the orders, the consumer starts the job by calling the operator service.
6. The operator service communicates with the operator engine to initiate a new job.
## Creating the K8 Cluster and Allocating Job Volumes
7. As a new job begins, volumes are created on the Kubernetes cluster, a task handled by the operator engine.
8. The cluster creates and allocates volumes for the job using the job volumes
9. The volumes are created and allocated to the pod
10. After volume creation and allocation, the operator engine initiates the `run script` on the pod configuration.
## Loading Assets and Algorithms
11. The pod configuration requires the data asset and algorithm, prompting a request to the provider for retrieval.
12. The provider uploads assets to the allocated job volume.
13. Upon completion of file uploads, the provider notifies the pod configuration that the assets are ready for the job.
14. The pod configuration informs the operator engine that it's ready to start the job.
## Running the Algorithm on Data Asset(s)
15. The operator engine launches the algorithm pod on the Kubernetes cluster.
16. Kubernetes runs the algorithm pod.
17. When the algorithm completes processing the dataset, the operator engine receives confirmation.
18. Now that the results are available, the operator engine runs the script on the pod publishing component.
19. The pod publishing uploads the results, logs, and admin logs to the output volume.
20. Upon successful upload, the operator engine receives notification from the pod publishing, allowing it to clean up the job volumes.
## Cleaning Up Volumes and Allocated Space
21. The operator engine deletes the K8 volumes.
22. The Kubernetes cluster removes all used volumes.
23. Once volumes are deleted, the operator engine finalizes the job.
24. The operator engine informs the operator service that the job is completed, and the results are now accessible.
## Retrieving Job Details
25. The consumer retrieves job details by calling the provider's `get job details`.
26. The provider communicates with the operator service to fetch job details.
27. The operator service returns the job details to the provider.
28. With the job details, the provider can share them with the asset consumer.
## Retrieving Job Results
29. Equipped with job details, the asset consumer can retrieve the results from the recently executed job.
30. The provider engages the operator engine to access the job results.
31. As the operator service lacks access to this information, it uses the output volume to fetch the results.
32. The output volume provides the stored job results to the operator service.
33. The operator service shares the results with the provider.
34. The provider then delivers the results to the asset consumer.
For visual clarity, here's an image of the workflow in action! 🖼️✨
<figure><img src="../../.gitbook/assets/c2d/c2d_compute_job.png" alt=""><figcaption><p>Compute detailed flow diagram</p></figcaption></figure>