From a3d05219e95ec880cf27019829a3205a816339b7 Mon Sep 17 00:00:00 2001 From: Ana Loznianu Date: Thu, 19 Oct 2023 15:07:12 +0300 Subject: [PATCH] Added C2D workflow details --- SUMMARY.md | 2 +- developers/compute-to-data/compute-job.md | 9 --- .../compute-to-data/compute-workflow.md | 62 +++++++++++++++++++ 3 files changed, 63 insertions(+), 10 deletions(-) delete mode 100644 developers/compute-to-data/compute-job.md create mode 100644 developers/compute-to-data/compute-workflow.md diff --git a/SUMMARY.md b/SUMMARY.md index 4fe8f8a6..c8c98034 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -92,7 +92,7 @@ - [Compute to data](developers/compute-to-data/README.md) - [Architecture](developers/compute-to-data/compute-to-data-architecture.md) - [Datasets & Algorithms](developers/compute-to-data/compute-to-data-datasets-algorithms.md) - - [Jobs](developers/compute-to-data/compute-job.md) + - [Jobs](developers/compute-to-data/compute-workflow.md) - [Writing Algorithms](developers/compute-to-data/compute-to-data-algorithms.md) - [Compute Options](developers/compute-to-data/compute-options.md) - [Aquarius](developers/aquarius/README.md) diff --git a/developers/compute-to-data/compute-job.md b/developers/compute-to-data/compute-job.md deleted file mode 100644 index e8190a89..00000000 --- a/developers/compute-to-data/compute-job.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: Compute Jobs -section: developers -description: Specification of compute options for assets in Ocean Protocol. ---- - -

Compute detailed flow diagram

- - diff --git a/developers/compute-to-data/compute-workflow.md b/developers/compute-to-data/compute-workflow.md new file mode 100644 index 00000000..217c98a9 --- /dev/null +++ b/developers/compute-to-data/compute-workflow.md @@ -0,0 +1,62 @@ +--- +title: Compute Workflow +section: developers +description: Understanding the Compute-to-Data (C2D) Workflow +--- + +🚀 Now that we've introduced the key actors and provided an overview of the process, it's time to delve into the nitty-gritty of the compute workflow.🌟 We'll dissect each step, examining the inner workings of Compute-to-Data (C2D). From data selection to secure computations, we'll leave no stone unturned in this exploration. 💫 Get ready to uncover the intricacies of C2D and how it utilizes various actors and components for a blend of privacy and efficiency. + +Ready for the deep dive? Let's get started! 🌊 + +## Starting a C2D Job +1. The consumer initiates a compute-to-data job by selecting the desired data asset and algorithm. +2. The provider checks the orders on the blockchain. +3. If the orders are valid, the provider can commence the compute flow. +4. The provider informs the consumer of the job number's successful creation. +5. With the job ID and confirmation of the orders, the consumer starts the job by calling the operator service. +6. The operator service communicates with the operator engine to initiate a new job. + +## Creating the K8 Cluster and Allocating Job Volumes +7. As a new job begins, volumes are created on the Kubernetes cluster, a task handled by the operator engine. +8. The cluster creates and allocates volumes for the job using the job volumes +9. The volumes are created and allocated to the pod +10. After volume creation and allocation, the operator engine initiates the `run script` on the pod configuration. + +## Loading Assets and Algorithms +11. The pod configuration requires the data asset and algorithm, prompting a request to the provider for retrieval. +12. The provider uploads assets to the allocated job volume. +13. Upon completion of file uploads, the provider notifies the pod configuration that the assets are ready for the job. +14. The pod configuration informs the operator engine that it's ready to start the job. + +## Running the Algorithm on Data Asset(s) +15. The operator engine launches the algorithm pod on the Kubernetes cluster. +16. Kubernetes runs the algorithm pod. +17. When the algorithm completes processing the dataset, the operator engine receives confirmation. +18. Now that the results are available, the operator engine runs the script on the pod publishing component. +19. The pod publishing uploads the results, logs, and admin logs to the output volume. +20. Upon successful upload, the operator engine receives notification from the pod publishing, allowing it to clean up the job volumes. + +## Cleaning Up Volumes and Allocated Space +21. The operator engine deletes the K8 volumes. +22. The Kubernetes cluster removes all used volumes. +23. Once volumes are deleted, the operator engine finalizes the job. +24. The operator engine informs the operator service that the job is completed, and the results are now accessible. + +## Retrieving Job Details +25. The consumer retrieves job details by calling the provider's `get job details`. +26. The provider communicates with the operator service to fetch job details. +27. The operator service returns the job details to the provider. +28. With the job details, the provider can share them with the asset consumer. + +## Retrieving Job Results +29. Equipped with job details, the asset consumer can retrieve the results from the recently executed job. +30. The provider engages the operator engine to access the job results. +31. As the operator service lacks access to this information, it uses the output volume to fetch the results. +32. The output volume provides the stored job results to the operator service. +33. The operator service shares the results with the provider. +34. The provider then delivers the results to the asset consumer. + + +For visual clarity, here's an image of the workflow in action! 🖼️✨ + +

Compute detailed flow diagram