1
0
mirror of https://github.com/oceanprotocol/docs.git synced 2024-11-26 19:49:26 +01:00

Fix #721 - All 3 Storage Setup pages reference Brizo lots, it should be Provider

This commit is contained in:
trentmc 2021-08-23 11:41:15 +02:00
parent e44cf80881
commit 2555b33278
10 changed files with 220 additions and 33 deletions

View File

@ -75,6 +75,18 @@ module.exports = {
{
from: '/concepts/oeps-asset-ddo/',
to: '/concepts/ddo-metadata/'
},
{
from: '/tutorials/azure-for-brizo/',
to: '/tutorials/azure-for-provider/'
},
{
from: '/tutorials/amazon-s3-for-brizo/',
to: '/tutorials/amazon-s3-for-provider/'
},
{
from: '/tutorials/on-premise-for-brizo/',
to: '/tutorials/on-premise-for-provider/'
}
],
swaggerComponents: [

View File

@ -101,7 +101,7 @@ Here is an example DDO service:
{
"index": 1,
"type": "access",
"serviceEndpoint": "http://localhost:8030/api/v1/brizo/services/consume",
"serviceEndpoint": "http://localhost:8030/api/v1/provider/services/consume",
"attributes": {
"main": {
"cost":"10",
@ -113,7 +113,7 @@ Here is an example DDO service:
{
"index": 2,
"type": "compute",
"serviceEndpoint": "http://localhost:8030/api/v1/brizo/services/compute",
"serviceEndpoint": "http://localhost:8030/api/v1/provider/services/compute",
"attributes": {
"main": {
"cost":"10",

View File

@ -1,16 +0,0 @@
---
title: Set Up Amazon S3 Storage
description: Tutorial about how to set up Amazon S3 storage for use with Ocean Protocol.
---
*Note: This needs updating for Ocean V3. As a workaround: Brizo has been renamed to provider-py; it should work similarly.*
To enable Brizo to use files stored in Amazon S3 (i.e. files with an URL containing `s3://`), you must:
1. have an Amazon AWS user account (IAM account) with permission to read those files from S3, and
1. set the AWS credentials on the machine where Brizo is running to those of the AWS user in question. Instructions are given below.
1. Note that you don't have to set any Brizo-specific configuration settings, e.g. in the `[osmosis]` section of the Brizo config file or in some special Brizo environment variables.
Under the hood, Brizo uses [boto3](https://aws.amazon.com/sdk-for-python/) (the Python library for interacting with AWS) to interact with AWS and boto3 has a whole process for determining AWS credentials. The easiest way to set the AWS credentials on the machine where Brizo is running is to install the [AWS CLI](https://aws.amazon.com/cli/) and then use the `aws configure` command.
For more details, see [the boto3 user guide about credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

View File

@ -0,0 +1,15 @@
---
title: Set Up Amazon S3 Storage
description: Tutorial about how to set up Amazon S3 storage for use with Ocean Protocol.
---
*Note: This needs updating for Ocean V3.*
To enable Provider to use files stored in Amazon S3 (i.e. files with an URL containing `s3://`), you must:
1. have an Amazon AWS user account (IAM account) with permission to read those files from S3, and
1. set the AWS credentials on the machine where Provider is running to those of the AWS user in question. Instructions are given below.
1. Note that you don't have to set any Provider-specific configuration settings, e.g. in the `[osmosis]` section of the Provider config file or in some special Provider environment variables.
Under the hood, Provider uses [boto3](https://aws.amazon.com/sdk-for-python/) (the Python library for interacting with AWS) to interact with AWS and boto3 has a whole process for determining AWS credentials. The easiest way to set the AWS credentials on the machine where Provider is running is to install the [AWS CLI](https://aws.amazon.com/cli/) and then use the `aws configure` command.
For more details, see [the boto3 user guide about credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).

View File

@ -0,0 +1,177 @@
---
title: Set Up Amazon S3 Storage
description: Tutorial about how to set up Azure storage for use with Ocean.
---
*Note: This needs updating for Ocean V3.*
This tutorial is for publishers who want to get started using Azure to store some of their data assets. (Some data assets could also be stored in other places.)
Publishers must run [Provider](https://github.com/oceanprotocol/provider) to mediate consumer access to data assets stored in Azure Storage. Provider needs the following Azure credentials from the publisher:
- `AZURE_ACCOUNT_NAME`: Azure Storage Account Name (for storing files)
- `AZURE_ACCOUNT_KEY`: Azure Storage Account key
- `AZURE_RESOURCE_GROUP`: Azure resource group
- `AZURE_LOCATION`: Azure Region
- `AZURE_CLIENT_ID`: Azure Application ID
- `AZURE_CLIENT_SECRET`: Azure Application Secret
- `AZURE_TENANT_ID`: Azure Tenant ID
- `AZURE_SUBSCRIPTION_ID`: Azure Subscription ID
If you go through this tutorial, then you will get all the Azure credentials listed above.
If you already have data assets stored in Azure, then you might already have, or be able to get, the above information. You could use this tutorial to get a sense of where to look (but don't create anything new).
To give the above Azure credentials to Provider, you either put them in a Provider config file or in environment variables with the above names. Environment variables should be used if you're running Provider inside a container. If you want to use the config file option, see [Provider README](https://github.com/oceanprotocol/provider).
If you're using [Barge](https://github.com/oceanprotocol/barge) to run Provider and other Ocean Protocol components, then the above Azure credentials should go in the file `barge/provider.env`. (That file gets used to set environment variables.)
This tutorial uses the [Microsoft Azure Portal](https://azure.microsoft.com/en-us/features/azure-portal/), but [there are many other ways to interact with Azure](https://docs.microsoft.com/en-us/azure/#pivot=sdkstools).
**Note: Azure is constantly changing. For that reason, we give try to give links to official Azure documentation, since it _should_ stay up-to-date.**
## Sign in to Azure Portal
If you don't already have an Azure account, then you will have to create one. Go to the [Microsoft Azure website](https://azure.microsoft.com) and follow the links.
Once you have an Azure account, go to [https://portal.azure.com/](https://portal.azure.com/) and sign in.
## Get Your Subscription ID
The [Azure docs say](https://docs.microsoft.com/en-us/azure/guides/developer/azure-developer-guide), "A subscription is a logical grouping of Azure services that is linked to an Azure account. A single Azure account can contain multiple subscriptions."
If you see **Subscriptions** in the left sidebar of Azure Portal, then click that. If you don't see it, just type "Subscriptinos" into the search bar at the top, then click on **Subscriptions** under the SERVICES heading.
You should see a list of one or more subscriptions. Click on the one you want to use for Azure storage. Remember to use that one for the rest of this tutorial (whenever you are asked for a subscription name).
Copy the `Subscription ID`. That's what Provider calls `AZURE_SUBSCRIPTION_ID`. You now have one of the Azure credentials!
```text
# Example AZURE_SUBSCRIPTION_ID (Azure Subscription ID)
479284be-0104-421a-8488-1aeac0caecaa
```
## Create an Azure Active Directory (AD) Application
See the Azure docs page:
[How to: Use the portal to create an Azure AD application and service principal that can access resources](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
The first step there is to **Create an Azure Active Directory application**. Do that.
The app `Name` and `Sign-on URL` can be totally made up. The URL doesn't need to be real.
Once the app is created, copy the `Application ID`: that's what Provider calls the `AZURE_CLIENT_ID`. It should look something like this:
```text
# Example AZURE_CLIENT_ID (Application ID)
5d25ee8a-da2c-4e6f-8fba-09b6dd091038
```
## Get Authentication Key for Your AD Application
On [the same Azure docs page](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal), find the section titled **Get application ID and authentication key** or similar. You already have your application ID, but you still need generate an authentication key by following the instructions in that section.
You can make up whatever you like for the key's `Description`.
Once the application key is generated, copy its value: that's what Provider calls the `AZURE_CLIENT_SECRET`. It should look something like this:
```text
# Example AZURE_CLIENT_SECRET (Application key)
RVJ1H5gYOmnMitikmM5ehszqmgrY5BFkoalnjfWMuDM
```
## Get Tenant ID
On [the same Azure docs page](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal), find the section titled **Get tenant ID** or similar. Follow the instructions.
The tenant ID is what Provider calls `AZURE_TENANT_ID`.
```text
# Example AZURE_TENANT_ID (tenant ID, Directory ID)
2a4a3887-4e2e-4a31-8006-6e2b5877640e
```
## Create a Resource Group for Your Data Storage
See the Azure docs page:
[Manage Azure resources through portal](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-portal)
That page says how to create a new empty resource group. Do that.
You can make up whatever name you like, but it's good practice to avoid special characters and to include:
- some words to indicate what it's for, e.g. `Storage`
- your name
- the month and year it was created, e.g. `Nov2018`
to help you and others manage it. The Resource group name is what Provider calls the `AZURE_RESOURCE_GROUP` and the Resource group location is what Provider calls the `AZURE_LOCATION`. Here are examples of both:
```text
# Example AZURE_RESOURCE_GROUP (Resource group name)
StorageCreatedNov2018ByTroy
```
```text
# Example AZURE_LOCATION (Resource group location)
West Europe
```
## Give Your AD Application Access to Your Resource Group
Inside your new resource group:
- click **Access control (IAM)**
- click **+ Add role assignment**
- In the `Role` field, select `Contributor`. See the note below.
- Assign access to `Azure AD user, group, or service principal`
- In the `Select` field, begin entering the name of your AD application (created earlier). When it appears in the list, click on it there. It should now be listed as one of the "Selected members".
- Click **Save**
Note: You might want to give your application fewer permissions than what a `Contributor` role gets. The Azure docs have [a list of all the built-in roles for Azure resources](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles).
## Create a Storage Account
Follow the instructions in the Azure docs page:
[Create a storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal)
except you should use the _existing_ resource group you created earlier, i.e. don't create a new one.
The Storage account name you choose is what Provider calls the `AZURE_ACCOUNT_NAME`.
```text
# Example AZURE_ACCOUNT_NAME (Storage account name)
troystorageaccount1
```
Use the same `Location` as your resource group.
The other fields can be left with their default values unless you want to change them.
Wait for it to say, "Your deployment is complete."
## Get a Storage Account Access Key
See the Azure docs page:
[Manage storage account settings in the Azure portal](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-manage)
Go to the subsection about access keys and follow the instructions to view your new storage account's credentials.
Copy the value of one of the keys (e.g. key1, not the connection string). That's what Provider calls `AZURE_ACCOUNT_KEY`.
```text
# Example AZURE_ACCOUNT_KEY (Storage account access key)
93uKDkbjfnSUNPKw2tpe0LOM+3Wk+OSkNmgwhzjvzDw1d3sKVhMRTC5ikvN0r3zsx8eQrmT9Wgjz22iLPu3aGw==
```
You now have all the Azure credentials Provider needs. See the instructions near the top of this page about how to give those Azure credentials to Provider.
## Store Some Data in Azure Storage
You now have a storage account, but you don't have any data stored under it yet. To get some data stored in [Azure Storage](https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction), the easiest option is to use [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/), a free desktop app that works on Windows, macOS and Linux.
[Get Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/).
Azure Storage can store blobs, files, queues and tables. To work with Ocean Network, you should store your files in [Azure Blob storage (also called object storage)](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction), not Azure Files.
Besides Azure Storage Explorer, there are [many other Azure Storage APIs, libraries and tools](https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction#storage-apis-libraries-and-tools).

View File

@ -1,10 +0,0 @@
---
title: Set Up On-Premise Storage
description: Tutorial about how to set up on-premise storage for use with Ocean.
---
*Note: This needs updating for Ocean V3. As a workaround: Brizo has been renamed to provider-py; it should work similarly.*
To enable Brizo to use files stored in on-premise storage (i.e. files with an URL not containing `core.windows.net` or `s3://`), there is _nothing to do, other than make sure Brizo can resolve the URLs_. In particular, you don't have to set any Brizo-specific configuration settings, e.g. in the `[osmosis]` section of the Brizo config file or in some special Brizo environment variables.
Local and private network URLs are fine so long as they can be resolved by Brizo. Potential examples include `http://localhost/helicopter_data.xls`, `http://192.168.12.34/almond_sales_2012.csv` and `http://10.12.34.56/duck_photos.zip`.

View File

@ -0,0 +1,9 @@
---
title: Set Up On-Premise Storage
description: Tutorial about how to set up on-premise storage for use with Ocean.
---
*Note: This needs updating for Ocean V3.*
To enable Provider to use files stored in on-premise storage (i.e. files with an URL not containing `core.windows.net` or `s3://`), there is _nothing to do, other than make sure Provider can resolve the URLs_. In particular, you don't have to set any Provider-specific configuration settings, e.g. in the `[osmosis]` section of the Provider config file or in some special Provider environment variables.
Local and private network URLs are fine so long as they can be resolved by Provider. Potential examples include `http://localhost/helicopter_data.xls`, `http://192.168.12.34/almond_sales_2012.csv` and `http://10.12.34.56/duck_photos.zip`.

View File

@ -45,11 +45,11 @@
- group: Storage Setup
items:
- title: Set Up Azure Storage
link: /tutorials/azure-for-brizo/
link: /tutorials/azure-for-provider/
- title: Set Up Amazon S3 Storage
link: /tutorials/amazon-s3-for-brizo/
link: /tutorials/amazon-s3-for-provider/
- title: Set Up On-Premise Storage
link: /tutorials/on-premise-for-brizo/
link: /tutorials/on-premise-for-provider/
- group: Fine-Grained Permissions
items:

View File

@ -17,7 +17,7 @@ The sidebar for those generated reference pages will automatically switch to inc
Reference pages based on Swagger specs are sourced from remotely hosted Swagger specs:
- [`https://aquarius.test.oceanprotocol.com/spec`](https://aquarius.test.oceanprotocol.com/spec)
- [`https://brizo.test.oceanprotocol.com/spec`](https://brizo.test.oceanprotocol.com/spec)
- [`https://provider.test.oceanprotocol.com/spec`](https://provider.test.oceanprotocol.com/spec)
They are fetched and updated automatically upon every site build. For more information about stylistic issues, take a look at the section in the test page:

View File

@ -14,7 +14,7 @@ The documentation is split in multiple sections whose content lives in this repo
- **Core concepts**: high-level explanation of concepts, assumptions, and components
- **Setup**: getting started for various stakeholders and use cases
- **Tutorials**: detailed tutorials
- **API References**: docs for the Aquarius & Brizo REST APIs, and docs for various Squid libraries
- **API References**: docs for ocean.js, ocean.py, Aquarius REST API, and Provider REST API
Those sections are defined in the [`/data/sections.yml`](../data/sections.yml) file.