Merge pull request #853 from oceanprotocol/issue-852-docker-registry-tutorials

Issue 852 docker registry tutorials

content/tutorials/compute-to-data-docker-registry.md

@@ -1,15 +1,39 @@
 ---
 title: Setting up private docker registry for Compute-to-Data environment
-description: Learn how to setup own docker registry and push images for running algorithms in C2D environment.
+description: Learn how to setup your own docker registry and push images for running algorithms in a C2D environment.
 ---
 
-## Prerequisites
+The document is intended for a production setup. The tutorial provides the steps to setup a private docker registry on the server for the following scenarios:
 
-1. Running docker environment on the server.
+- Allow registry access only to the C2D environment.
-2. Domain name is mapped to the server IP address.
+- Anyone can pull the image from the registry but, only authenticated users will push images to the registry.
-3. SSL certificate
 
-## Generate password file
+## Setup 1: Allow registry access only to the C2D environment
+
+
+To implement this use case, 1 domain will be required:
+
+- **example.com**: This domain will allow only image pull operations
+
+_Note: Please change the domain names to your application-specific domain names._
+
+### 1.1 Prerequisites
+
+- Running docker environment on the linux server.
+- Docker compose is installed.
+- C2D environment is running.
+- The domain names is mapped to the server hosting the registry.
+
+### 1.2 Generate certificates
+
+```bash
+# install certbot: https://certbot.eff.org/
+sudo certbot certonly --standalone --cert-name example.com -d example.com
+```
+
+_Note: Do check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._
+
+### 1.3 Generate password file
 
 Replace content in `<>` with appropriate content.
 
@@ -19,7 +43,7 @@ docker run \
   httpd:2 -Bbn <username> <password> > <path>/auth/htpasswd
 ```
 
-## Docker compose template file for registry
+### 1.4 Docker compose template file for registry
 
 Copy the below yml content to `docker-compose.yml` file and replace content in `<>`.
 
@@ -34,8 +58,6 @@ services:
     ports:
       - 5050:5000
     environment:
-      REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt
-      REGISTRY_HTTP_TLS_KEY: /certs/domain.key
       REGISTRY_AUTH: htpasswd
       REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
       REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
@@ -43,7 +65,201 @@ services:
     volumes:
       - <path>/data:/var/lib/registry
       - <path>/auth:/auth
-      - <path>/certs:/certs
+  nginx:
+        image: nginx:latest
+        container_name: nginx
+        volumes:
+            -  <path>/nginx/logs:/app/logs/
+            -  nginx.conf:/etc/nginx/nginx.conf
+            -  /etc/letsencrypt/:/etc/letsencrypt/
+        ports:
+            - 80:80
+            - 443:443
+        depends_on:
+            - registry
+```
+
+### 1.5 Nginx configuration
+
+Copy the below nginx configuration to a `nginx.conf` file.
+
+```conf
+events {}
+http {
+  access_log /app/logs/access.log;
+  error_log /app/logs/error.log;
+	
+	server {
+        client_max_body_size 4096M;
+    		listen 80 default_server;
+    		server_name _;
+    		return 301 https://$host$request_uri;
+	}
+
+  server {
+          # Allowed request size should be large enough to allow pull operations
+          client_max_body_size 4096M;
+          listen 443 ssl;
+          server_name example.com;
+          ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
+          ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
+          location / {
+                  proxy_connect_timeout 75s;
+                  proxy_pass http://registry-read-only:5000;
+          }
+  }
+}
+
+```
+
+### 1.6 Create kubernetes secret in C2D server
+
+Login into Compute-to-data enviroment and run the following command with appropriate credentials:
+
+```bash
+kubectl create secret docker-registry regcred --docker-server=example.com --docker-username=<username> --docker-password=<password> --docker-email=<email_id> -n ocean-compute
+```
+
+### 1.7 Update operator-engine configuration
+
+Add `PULL_SECRET` property with value `regcred` in the [operator.yml](https://github.com/oceanprotocol/operator-engine/blob/main/kubernetes/operator.yml) file of operator-engine configuration.
+For more detials on operator-engine properties refer this [link](https://github.com/oceanprotocol/operator-engine/blob/177ca7185c34aa2a503afbe026abb19c62c69e6d/README.md?plain=1#L106)
+
+Apply updated operator-engine configuration.
+
+```bash
+kubectl config set-context --current --namespace ocean-compute
+kubectl apply  -f operator-engine/kubernetes/operator.yml
+```
+
+## Steup 2: Allow anyonymous `pull` operations
+
+To implement this use case, 2 domains will be required:
+
+- **example.com**: This domain will allow image push/pull operations only to the authenticated users.
+- **readonly.example.com**: This domain will allow only image pull operations
+
+_Note: Please change the domain names to your application-specific domain names._
+
+### 2.1 Prerequisites
+
+- Running docker environment on the linux server.
+- Docker compose is installed.
+- 2 domain names is mapped to the same server IP address.
+
+### 2.2 Generate certificates
+
+```bash
+# install certbot: https://certbot.eff.org/
+sudo certbot certonly --standalone --cert-name example.com -d example.com
+sudo certbot certonly --standalone --cert-name readonly.example.com -d readonly.example.com
+```
+
+_Note: Do check the access right of the files/directories where certificates are stored. Usually, they are at `/etc/letsencrypt/`._
+
+### 2.3 Generate password file
+
+Replace content in `<>` with appropriate content.
+
+```bash
+docker run \
+  --entrypoint htpasswd \
+  httpd:2 -Bbn <username> <password> > <path>/auth/htpasswd
+```
+
+### 2.4 Docker compose template file for registry
+
+Copy the below yml content to `docker-compose.yml` file and replace content in `<>`.
+Here, we will be creating two services of the docker registry so that anyone can `pull` the images from the registry but, only authenticated users can `push` the images.
+
+```yml
+version: '3'
+
+services:
+  registry:
+    restart: always
+    container_name: my-docker-registry
+    image: registry:2
+    ports:
+      - 5050:5000
+    environment:
+      REGISTRY_AUTH: htpasswd
+      REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
+      REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
+      REGISTRY_HTTP_SECRET: <secret>
+    volumes:
+      - <path>/data:/var/lib/registry
+      - <path>/auth:/auth
+  registry-read-only:
+      restart: always
+      container_name: my-registry-read-only
+      image: registry:2
+      read_only: true
+      ports:
+        - 5051:5000
+      environment:
+        REGISTRY_HTTP_SECRET: ${REGISTRY_HTTP_SECRET}
+      volumes:
+        - <path>/docker-registry/data:/var/lib/registry:ro
+      depends_on:
+        - registry
+  nginx:
+        image: nginx:latest
+        container_name: nginx
+        volumes:
+            -  <path>/nginx/logs:/app/logs/
+            -  nginx.conf:/etc/nginx/nginx.conf
+            -  /etc/letsencrypt/:/etc/letsencrypt/
+        ports:
+            - 80:80
+            - 443:443
+        depends_on:
+            - registry-read-only
+```
+
+### 2.5 Nginx configuration
+
+Copy the below nginx configuration to a `nginx.conf` file.
+
+```conf
+events {}
+http {
+  access_log /app/logs/access.log;
+  error_log /app/logs/error.log;
+	
+	server {
+        client_max_body_size 4096M;
+    		listen 80 default_server;
+    		server_name _;
+    		return 301 https://$host$request_uri;
+	}
+
+  server {
+           # Allowed request size should be large enough to allow push operations
+           client_max_body_size 4096M;
+           listen 443 ssl;
+           server_name readonly.example.com;
+           ssl_certificate /etc/letsencrypt/live/readonly.example.com/fullchain.pem;
+           ssl_certificate_key /etc/letsencrypt/live/readonly.example.com/privkey.pem;
+           location / {
+                   proxy_connect_timeout 75s;
+                   proxy_pass http://registry:5000;
+           }
+  }
+
+  server {
+          # Allowed request size should be large enough to allow pull operations
+          client_max_body_size 4096M;
+          listen 443 ssl;
+          server_name example.com;
+          ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
+          ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
+          location / {
+                  proxy_connect_timeout 75s;
+                  proxy_pass http://registry-read-only:5000;
+          }
+  }
+}
 
 ```
 
@@ -53,31 +269,50 @@ services:
 docker-compose -f docker-compose.yml up 
 ```
 
-## List images in the registry
+## Working with registry
 
-```bash
+### Login to registry
-curl -X GET -u <username>:<password> https://example.com/v2/_catalog
-```
-
-## Other useful commands
-
-
-## Login to registry
 
 ```bash
 docker login example.com -u <username> -p <password>
 ```
 
-## Build and push image to registry
+### Build and push an image to the registry
 
-Use the commands below to build an image from a  `Dockerfile` and push to your own private registry.
+Use the commands below to build an image from a `Dockerfile` and push it to your private registry.
 
 ```bash
 docker build . -t example.com/my-algo:latest
-
+docker image push example.com/my-algo:latest
-docker image tag example.com/my-algo:latest
 ```
 
-## Next step
+### List images in the registry
 
-You can publish an algorithm asset with the metadata containing registry url, image, and tag information to enable users to run C2D jobs. 
+```bash
+curl -X GET -u <username>:<password> https://example.com/v2/_catalog
+```
+
+### Pull an image from the registry
+
+Use the commands below to build an image from a `Dockerfile` and push it to your private registry.
+
+```bash
+# requires login
+docker image pull example.com/my-algo:latest
+
+# allows anonymous pull if 2nd setup scenario is implemented
+docker image pull readonly.example.com/my-algo:latest
+
+
+```
+
+### Next step
+
+You can publish an algorithm asset with the metadata containing registry URL, image, and tag information to enable users to run C2D jobs.
+
+
+## Further references
+
+- [Setup Compute-to-Data environment](/tutorials/compute-to-data-minikube/)
+- [Writing algorithms](/tutorials/compute-to-data-algorithms/)
+- [C2D example](/references/read-the-docs/ocean-py/READMEs/c2d-flow.md)