diff --git a/content/tutorials/compute-to-data-docker-registry.md b/content/tutorials/compute-to-data-docker-registry.md index 8224f544..2a296a24 100644 --- a/content/tutorials/compute-to-data-docker-registry.md +++ b/content/tutorials/compute-to-data-docker-registry.md @@ -3,32 +3,37 @@ title: Setting up private docker registry for Compute-to-Data environment description: Learn how to setup your own docker registry and push images for running algorithms in a C2D environment. --- -The tutorial provides the steps to setup a private docker registry on the server. The document is intended for a production setup where anyone can pull the image from the registry but, only authenticated users will push images to the registry. +The document is intended for a production setup. The tutorial provides the steps to setup a private docker registry on the server for the following scenarios: -To implement this use case, 2 domains will be required: +- Allow registry access only to the C2D environment. +- Anyone can pull the image from the registry but, only authenticated users will push images to the registry. + +## Setup 1: Allow registry access only to the C2D environment + + +To implement this use case, 1 domain will be required: - **example.com**: This domain will allow only image pull operations -- **admin.example.com**: This domain will allow image push/pull operations only to the authenticated users. _Note: Please change the domain names to your application-specific domain names._ -## Prerequisites +### 1.1 Prerequisites - Running docker environment on the linux server. - Docker compose is installed. -- 2 domain names is mapped to the same server IP address. +- C2D environment is running. +- The domain names is mapped to the server hosting the registry. -## Generate certificates +### 1.2 Generate certificates ```bash # install certbot: https://certbot.eff.org/ sudo certbot certonly --standalone --cert-name example.com -d example.com -sudo certbot certonly --standalone --cert-name admin.example.com -d admin.example.com ``` _Note: Do check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._ -## Generate password file +### 1.3 Generate password file Replace content in `<>` with appropriate content. @@ -38,7 +43,131 @@ docker run \ httpd:2 -Bbn > /auth/htpasswd ``` -## Docker compose template file for registry +### 1.4 Docker compose template file for registry + +Copy the below yml content to `docker-compose.yml` file and replace content in `<>`. + +```yml +version: '3' + +services: + registry: + restart: always + container_name: my-docker-registry + image: registry:2 + ports: + - 5050:5000 + environment: + REGISTRY_AUTH: htpasswd + REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd + REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm + REGISTRY_HTTP_SECRET: + volumes: + - /data:/var/lib/registry + - /auth:/auth + nginx: + image: nginx:latest + container_name: nginx + volumes: + - /nginx/logs:/app/logs/ + - nginx.conf:/etc/nginx/nginx.conf + - /etc/letsencrypt/:/etc/letsencrypt/ + ports: + - 80:80 + - 443:443 + depends_on: + - registry +``` + +### 1.5 Nginx configuration + +Copy the below nginx configuration to a `nginx.conf` file. + +```conf +events {} +http { + access_log /app/logs/access.log; + error_log /app/logs/error.log; + + server { + client_max_body_size 4096M; + listen 80 default_server; + server_name _; + return 301 https://$host$request_uri; + } + + server { + # Allowed request size should be large enough to allow pull operations + client_max_body_size 4096M; + listen 443 ssl; + server_name example.com; + ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; + location / { + proxy_connect_timeout 75s; + proxy_pass http://registry-read-only:5000; + } + } +} + +``` + +### 1.6 Create kubernetes secret in C2D server + +Login into Compute-to-data enviroment and run the following command with appropriate credentials: + +```bash +kubectl create secret docker-registry regcred --docker-server=example.com --docker-username= --docker-password= --docker-email= -n ocean-compute +``` + +### 1.7 Update operator-engine configuration + +Add `PULL_SECRET` property with value `regcred` in the [operator.yml](https://github.com/oceanprotocol/operator-engine/blob/main/kubernetes/operator.yml) file of operator-engine configuration. +For more detials on operator-engine properties refer this [link](https://github.com/oceanprotocol/operator-engine/blob/177ca7185c34aa2a503afbe026abb19c62c69e6d/README.md?plain=1#L106) + +Apply updated operator-engine configuration. + +```bash +kubectl config set-context --current --namespace ocean-compute +kubectl apply -f operator-engine/kubernetes/operator.yml +``` + +## Steup 2: Allow anyonymous `pull` operations + +To implement this use case, 2 domains will be required: + +- **example.com**: This domain will allow image push/pull operations only to the authenticated users. +- **readonly.example.com**: This domain will allow only image pull operations + +_Note: Please change the domain names to your application-specific domain names._ + +### 2.1 Prerequisites + +- Running docker environment on the linux server. +- Docker compose is installed. +- 2 domain names is mapped to the same server IP address. + +### 2.2 Generate certificates + +```bash +# install certbot: https://certbot.eff.org/ +sudo certbot certonly --standalone --cert-name example.com -d example.com +sudo certbot certonly --standalone --cert-name readonly.example.com -d readonly.example.com +``` + +_Note: Do check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._ + +### 2.3 Generate password file + +Replace content in `<>` with appropriate content. + +```bash +docker run \ + --entrypoint htpasswd \ + httpd:2 -Bbn > /auth/htpasswd +``` + +### 2.4 Docker compose template file for registry Copy the below yml content to `docker-compose.yml` file and replace content in `<>`. Here, we will be creating two services of the docker registry so that anyone can `pull` the images from the registry but, only authenticated users can `push` the images. @@ -88,7 +217,7 @@ services: - registry-read-only ``` -## Nginx configuration +### 2.5 Nginx configuration Copy the below nginx configuration to a `nginx.conf` file. @@ -109,9 +238,9 @@ http { # Allowed request size should be large enough to allow push operations client_max_body_size 4096M; listen 443 ssl; - server_name admin.example.com; - ssl_certificate /etc/letsencrypt/live/admin.example.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/admin.example.com/privkey.pem; + server_name readonly.example.com; + ssl_certificate /etc/letsencrypt/live/readonly.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/readonly.example.com/privkey.pem; location / { proxy_connect_timeout 75s; proxy_pass http://registry:5000; @@ -145,7 +274,7 @@ docker-compose -f docker-compose.yml up ### Login to registry ```bash -docker login admin.example.com -u -p +docker login example.com -u -p ``` ### Build and push an image to the registry @@ -153,8 +282,8 @@ docker login admin.example.com -u -p Use the commands below to build an image from a `Dockerfile` and push it to your private registry. ```bash -docker build . -t admin.example.com/my-algo:latest -docker image push admin.example.com/my-algo:latest +docker build . -t example.com/my-algo:latest +docker image push example.com/my-algo:latest ``` ### List images in the registry @@ -168,17 +297,19 @@ curl -X GET -u : https://example.com/v2/_catalog Use the commands below to build an image from a `Dockerfile` and push it to your private registry. ```bash +# requires login docker image pull example.com/my-algo:latest + +# allows anonymous pull if 2nd setup scenario is implemented +docker image pull readonly.example.com/my-algo:latest + + ``` -## Next step +### Next step You can publish an algorithm asset with the metadata containing registry URL, image, and tag information to enable users to run C2D jobs. -## Other alternatives - -Some use cases might require that image `pull` operations are also restricted to authenticated servers/users. To allow the compute-to-data environment to pull images from the private registry, add `imagePullSecrets` to the compute job template file [here](https://github.com/oceanprotocol/operator-engine/blob/main/operator_engine/templates/configure-job-template.yaml). -You can generate the registry secret in the compute environment namespace by following these [steps](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/). ## Further references