Copied code from ascribe/bigchain-depoloyment repo

bigchain-deployment/Readme.md

@@ -0,0 +1,19 @@
+## Create and configure the storage backend in Amazon's Cloud
+
+#### Getting started
+- Checkout bigchaindb and copy bigchain-deployment to bigchaindb repository
+
+#### Prerequesites
+ - Valid AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY is needed, both are exported as variables to the shell
+ - awscli,
+ - boto
+ - fabric w/ fabtools
+
+#### Cluster Installation
+ - Got to the DEPLOY-directory and run './startup.sh' with two parameters (tag and number of nodes)...that's it!
+        e.g.: ./startup.sh bro 7 to install a cluster tagged as bro with seven nodes.
+
+#### If an error occurs...
+There are some issues during the rollout on Amazon (presumably also in other cloud/virtual environments): if you tested with a high sequence it might be possible, that you run into:
+ - NetworkError: Host key for ec2-xx-xx-xx-xx.eu-central-1.compute.amazonaws.com did not match pre-existing key! Server's key was changed recently, or possible man-in-the-middle attack.
+If so, just clean up your known_hosts file and start again.

bigchain-deployment/bigchain/bigchaindb.conf

@@ -0,0 +1,107 @@
+#
+# RethinkDB instance configuration sample
+#
+# - Give this file the extension .conf and put it in /etc/rethinkdb/instances.d in order to enable it.
+# - See http://www.rethinkdb.com/docs/guides/startup/ for the complete documentation
+# - Uncomment an option to change its value.
+#
+
+###############################
+## RethinkDB configuration
+###############################
+
+### Process options
+
+## User and group used to run rethinkdb
+## Command line default: do not change user or group
+## Init script default: rethinkdb user and group
+# runuser=rethinkdb
+# rungroup=rethinkdb
+
+## Stash the pid in this file when the process is running
+## Note for systemd users: Systemd uses its own internal mechanism. Do not set this parameter.
+## Command line default: none
+## Init script default: /var/run/rethinkdb/<name>/pid_file (where <name> is the name of this config file without the extension)
+# pid-file=/var/run/rethinkdb/rethinkdb.pid
+
+### File path options
+
+## Directory to store data and metadata
+## Command line default: ./rethinkdb_data
+## Init script default: /var/lib/rethinkdb/<name>/ (where <name> is the name of this file without the extension)
+directory=/data
+
+## Log file options
+## Default: <directory>/log_file
+#log-file=/var/log/rethinkdb
+
+### Network options
+
+## Address of local interfaces to listen on when accepting connections
+## May be 'all' or an IP address, loopback addresses are enabled by default
+## Default: all local addresses
+# bind=127.0.0.1
+bind=all
+
+## Address that other rethinkdb instances will use to connect to this server.
+## It can be specified multiple times
+# canonical-address=
+
+## The port for rethinkdb protocol for client drivers
+## Default: 28015 + port-offset
+# driver-port=28015
+
+## The port for receiving connections from other nodes
+## Default: 29015 + port-offset
+# cluster-port=29015
+
+## The host:port of a node that rethinkdb will connect to
+## This option can be specified multiple times.
+## Default: none
+# join=example.com:29015
+
+## All ports used locally will have this value added
+## Default: 0
+# port-offset=0
+
+## r.http(...) queries will use the given server as a web proxy
+## Default: no proxy
+# reql-http-proxy=socks5://example.com:1080
+
+### Web options
+
+## Port for the http admin console
+## Default: 8080 + port-offset
+# http-port=8080
+
+## Disable web administration console
+# no-http-admin
+
+### CPU options
+
+## The number of cores to use
+## Default: total number of cores of the CPU
+# cores=2
+
+### Memory options
+
+## Size of the cache in MB
+## Default: Half of the available RAM on startup
+# cache-size=1024
+
+### Disk
+
+## How many simultaneous I/O operations can happen at the same time
+# io-threads=64
+#io-threads=128
+
+## Enable direct I/O
+direct-io
+
+### Meta
+
+## The name for this server (as will appear in the metadata).
+## If not specified, it will be randomly chosen from a short list of names.
+# server-name=server1
+## The host:port of a node that rethinkdb will connect to
+join=ec2-52-58-94-210.eu-central-1.compute.amazonaws.com:29015

bigchain-deployment/bigchain/bigchaindb.conf.template

@@ -0,0 +1,105 @@
+#
+# RethinkDB instance configuration sample
+#
+# - Give this file the extension .conf and put it in /etc/rethinkdb/instances.d in order to enable it.
+# - See http://www.rethinkdb.com/docs/guides/startup/ for the complete documentation
+# - Uncomment an option to change its value.
+#
+
+###############################
+## RethinkDB configuration
+###############################
+
+### Process options
+
+## User and group used to run rethinkdb
+## Command line default: do not change user or group
+## Init script default: rethinkdb user and group
+# runuser=rethinkdb
+# rungroup=rethinkdb
+
+## Stash the pid in this file when the process is running
+## Note for systemd users: Systemd uses its own internal mechanism. Do not set this parameter.
+## Command line default: none
+## Init script default: /var/run/rethinkdb/<name>/pid_file (where <name> is the name of this config file without the extension)
+# pid-file=/var/run/rethinkdb/rethinkdb.pid
+
+### File path options
+
+## Directory to store data and metadata
+## Command line default: ./rethinkdb_data
+## Init script default: /var/lib/rethinkdb/<name>/ (where <name> is the name of this file without the extension)
+directory=/data
+
+## Log file options
+## Default: <directory>/log_file
+#log-file=/var/log/rethinkdb
+
+### Network options
+
+## Address of local interfaces to listen on when accepting connections
+## May be 'all' or an IP address, loopback addresses are enabled by default
+## Default: all local addresses
+# bind=127.0.0.1
+bind=all
+
+## Address that other rethinkdb instances will use to connect to this server.
+## It can be specified multiple times
+# canonical-address=
+
+## The port for rethinkdb protocol for client drivers
+## Default: 28015 + port-offset
+# driver-port=28015
+
+## The port for receiving connections from other nodes
+## Default: 29015 + port-offset
+# cluster-port=29015
+
+## The host:port of a node that rethinkdb will connect to
+## This option can be specified multiple times.
+## Default: none
+# join=example.com:29015
+
+## All ports used locally will have this value added
+## Default: 0
+# port-offset=0
+
+## r.http(...) queries will use the given server as a web proxy
+## Default: no proxy
+# reql-http-proxy=socks5://example.com:1080
+
+### Web options
+
+## Port for the http admin console
+## Default: 8080 + port-offset
+# http-port=8080
+
+## Disable web administration console
+# no-http-admin
+
+### CPU options
+
+## The number of cores to use
+## Default: total number of cores of the CPU
+# cores=2
+
+### Memory options
+
+## Size of the cache in MB
+## Default: Half of the available RAM on startup
+# cache-size=1024
+
+### Disk
+
+## How many simultaneous I/O operations can happen at the same time
+# io-threads=64
+#io-threads=128
+
+## Enable direct I/O
+direct-io
+
+### Meta
+
+## The name for this server (as will appear in the metadata).
+## If not specified, it will be randomly chosen from a short list of names.
+# server-name=server1

bigchain-deployment/create_hostlist.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+import json
+import argparse
+import boto.ec2
+import os
+
+AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
+AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--tag", help="tag instances in aws")
+args = parser.parse_args()
+
+conn = boto.ec2.connect_to_region("eu-central-1",
+            aws_access_key_id=AWS_ACCESS_KEY_ID,
+            aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
+
+PUBLIC_LIST = []
+PRIVATE_LIST = []
+INSTANCE_IDS = []
+
+if args.tag:
+    tag = args.tag
+else:
+    # reading credentials from config for remote connection
+    print('usage: python3 create_hostlist.py --tag <tag>')
+    print('reason: tag missing!!!')
+    exit(1)
+
+
+def prepare_list(tag):
+    reservations = conn.get_all_instances(filters={"tag:Name" : tag})
+    instances = [i for r in reservations for i in r.instances]
+    for i in instances:
+        inst = i.__dict__
+        publdns = inst.get('public_dns_name')
+        privdns = inst.get('private_dns_name')
+        inst_id = inst.get('id')
+        PUBLIC_LIST.append(publdns)
+        PRIVATE_LIST.append(privdns)
+        INSTANCE_IDS.append(inst_id)
+    return PUBLIC_LIST, PRIVATE_LIST, INSTANCE_IDS
+
+
+# get lists from amazon
+publist, privlist, instlist = prepare_list(tag)
+
+# create shellscript for adding remote keys to known_hosts
+localFile = open('add2known_hosts.sh', 'w')
+localFile.write('#! /bin/bash\n')
+for entry in range(0,len(publist)):
+    localFile.write('ssh-keyscan ' + publist[entry] + ' >> ~/.ssh/known_hosts\n')
+localFile.close()
+
+# hostliste und id-liste aus json erzeugen
+hosts = publist
+localFile = open('add2dbconf', 'w')
+before = 'join='
+after = ':29015'
+localFile.write('## The host:port of a node that rethinkdb will connect to\n')
+for entry in range(0,int(len(hosts)/2)):
+    localFile.write(before + hosts[entry] + after + '\n')
+
+
+# printout hostlist
+print ("hosts_dev = ", publist)

bigchain-deployment/fab_prepare_chain.py

@@ -0,0 +1,28 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+""" Generating genesis block
+"""
+
+from __future__ import with_statement
+
+from fabric import colors as c
+from fabric.api import *
+from fabric.api import local, puts, settings, hide, abort, lcd, prefix
+from fabric.api import run, sudo, cd, get, local, lcd, env, hide
+from fabric.api import task, parallel
+from fabric.contrib import files
+from fabric.contrib.files import append, exists
+from fabric.contrib.console import confirm
+from fabric.contrib.project import rsync_project
+from fabric.operations import run, put
+from fabric.context_managers import settings
+from fabric.decorators import roles
+from fabtools import *
+
+env.user = 'ubuntu'
+env.key_filename = 'pem/bigchain.pem'
+
+@task
+def init_bigchaindb():
+    run('bigchaindb -y start &', pty = False)

bigchain-deployment/fabfile.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+""" Preparing, installing and configuring
+    bigchain and the storage backend
+"""
+
+from __future__ import with_statement
+
+import requests
+from time import *
+import os
+from datetime import datetime, timedelta
+import json
+from pprint import pprint
+
+from fabric import colors as c
+from fabric.api import *
+from fabric.api import local, puts, settings, hide, abort, lcd, prefix
+from fabric.api import run, sudo, cd, get, local, lcd, env, hide
+from fabric.api import task, parallel
+from fabric.contrib import files
+from fabric.contrib.files import append, exists
+from fabric.contrib.console import confirm
+from fabric.contrib.project import rsync_project
+from fabric.operations import run, put
+from fabric.context_managers import settings
+from fabric.decorators import roles
+from fabtools import *
+
+from hostlist import hosts_dev
+
+env.hosts = hosts_dev
+env.roledefs = {
+    "role1": hosts_dev,
+    "role2": [hosts_dev[0]],
+    }
+env.roles = ["role1"]
+env.user = 'ubuntu'
+env.key_filename = 'pem/bigchain.pem'
+
+
+################################################################################
+
+# base softwarestack rollout
+@task
+@parallel
+def install_base_software():
+    sudo('apt-get -y update')
+    sudo('dpkg --configure -a')
+    sudo('apt-get -y -f install')
+    sudo('apt-get -y install build-essential wget bzip2 ca-certificates \
+                     libglib2.0-0 libxext6 libsm6 libxrender1 libssl-dev \
+                     git gcc g++ python-dev libboost-python-dev libffi-dev \
+                     software-properties-common python-software-properties \
+                     python3-pip ipython3 sysstat s3cmd')
+
+
+# RethinkDB
+@task
+@parallel
+def install_rethinkdb():
+    """Installation of RethinkDB"""
+    with settings(warn_only=True):
+        # preparing filesystem
+        sudo("mkdir -p /data")
+        # Locally mounted storage (m3.2xlarge, aber auch c3.xxx)
+        try:
+            sudo("umount /mnt")
+            sudo("mkfs -t ext4 /dev/xvdb")
+            sudo("mount /dev/xvdb /data")
+        except:
+            pass
+
+        # persist settings to fstab
+        sudo("rm -rf /etc/fstab")
+        sudo("echo 'LABEL=cloudimg-rootfs	/	 ext4     defaults,discard    0   0' >> /etc/fstab")
+        sudo("echo '/dev/xvdb  /data        ext4    defaults,noatime    0   0' >> /etc/fstab")
+        # activate deadline scheduler
+        with settings(sudo_user='root'):
+            sudo("echo deadline > /sys/block/xvdb/queue/scheduler")
+        # install rethinkdb
+        sudo("echo 'deb http://download.rethinkdb.com/apt trusty main' | sudo tee /etc/apt/sources.list.d/rethinkdb.list")
+        sudo("wget -qO- http://download.rethinkdb.com/apt/pubkey.gpg | sudo apt-key add -")
+        sudo("apt-get update")
+        sudo("apt-get -y install rethinkdb")
+        # change fs to user
+        sudo('chown -R rethinkdb:rethinkdb /data')
+        # copy config file to target system
+        put('bigchain/bigchaindb.conf',
+            '/etc/rethinkdb/instances.d/instance1.conf', mode=0600, use_sudo=True)
+        # initialize data-dir
+        sudo('rm -rf /data/*')
+        # finally restart instance
+        sudo('/etc/init.d/rethinkdb restart')
+
+
+# bigchain deployment
+@task
+@parallel
+def install_bigchain():
+    sudo('python3 -m pip install bigchaindb')
+
+
+# startup all nodes of bigchaindb in cluster
+@task
+@parallel
+def start_bigchain_nodes():
+    sudo('screen -d -m bigchaindb -y start &', pty = False)
+
+
+@task
+def install_newrelic():
+    with settings(warn_only=True):
+        sudo('echo deb http://apt.newrelic.com/debian/ newrelic non-free >> /etc/apt/sources.list')
+        #sudo('apt-key adv --keyserver hkp://subkeys.pgp.net --recv-keys 548C16BF')
+        sudo('apt-get update')
+        sudo('apt-get -y --force-yes install newrelic-sysmond')
+        sudo('nrsysmond-config --set license_key=c88af00c813983f8ee12e9b455aa13fde1cddaa8')
+        sudo('/etc/init.d/newrelic-sysmond restart')
+
+
+###############################
+# Security / FirewallStuff next
+###############################
+
+@task
+def harden_sshd():
+    """Security harden sshd."""
+
+    # Disable password authentication
+    sed('/etc/ssh/sshd_config',
+        '#PasswordAuthentication yes',
+        'PasswordAuthentication no',
+        use_sudo=True)
+    # Deny root login
+    sed('/etc/ssh/sshd_config',
+        'PermitRootLogin yes',
+        'PermitRootLogin no',
+        use_sudo=True)
+
+
+@task
+def disable_root_login():
+    """Disable `root` login for even more security. Access to `root` account
+    is now possible by first connecting with your dedicated maintenance
+    account and then running ``sudo su -``."""
+    sudo('passwd --lock root')
+
+
+@task
+def set_fw():
+    # snmp
+    sudo('iptables -A INPUT -p tcp --dport 161 -j ACCEPT')
+    sudo('iptables -A INPUT -p udp --dport 161 -j ACCEPT')
+    # dns
+    sudo('iptables -A OUTPUT -p udp -o eth0 --dport 53 -j ACCEPT')
+    sudo('iptables -A INPUT -p udp -i eth0 --sport 53 -j ACCEPT')
+    # rethinkdb
+    sudo('iptables -A INPUT -p tcp --dport 28015 -j ACCEPT')
+    sudo('iptables -A INPUT -p udp --dport 28015 -j ACCEPT')
+    sudo('iptables -A INPUT -p tcp --dport 29015 -j ACCEPT')
+    sudo('iptables -A INPUT -p udp --dport 29015 -j ACCEPT')
+    sudo('iptables -A INPUT -p tcp --dport 8080 -j ACCEPT')
+    sudo('iptables -A INPUT -i eth0 -p tcp --dport 8080 -j DROP')
+    sudo('iptables -I INPUT -i eth0 -s 127.0.0.1 -p tcp --dport 8080 -j ACCEPT')
+    # save rules
+    sudo('iptables-save >  /etc/sysconfig/iptables')
+
+
+#########################################################
+# some helper-functions to handle bad behavior of cluster
+#########################################################
+
+# rebuild indexes
+@task
+@parallel
+def rebuild_indexes():
+    run('rethinkdb index-rebuild -n 2')
+
+
+@task
+def stopdb():
+    sudo('service rethinkdb stop')
+
+
+@task
+def startdb():
+    sudo('service rethinkdb start')
+
+
+@task
+def restartdb():
+    sudo('/etc/init.d/rethinkdb restart')

bigchain-deployment/get_elastic_ips.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+# create Elastic IPs and assign them to instances if needed
+import json
+import os
+import boto.ec2
+import argparse
+import time
+
+AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
+AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--tag", help="tag instances in aws")
+args = parser.parse_args()
+
+if args.tag:
+    tag = args.tag
+else:
+    # reading credentials from config for remote connection
+    print('usage: python3 get_elastic_ips.py --tag <tag>')
+    print('reason: tag missing!!!')
+    exit(1)
+
+conn = boto.ec2.connect_to_region("eu-central-1",
+            aws_access_key_id=AWS_ACCESS_KEY_ID,
+            aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
+
+INSTANCE_IDS = []
+
+
+def prepare_list(tag):
+    reservations = conn.get_all_instances(filters={"tag:Name" : tag})
+    instances = [i for r in reservations for i in r.instances]
+    for i in instances:
+        inst = i.__dict__
+        #print (inst)
+        #break
+        inst_id = inst.get('id')
+
+        INSTANCE_IDS.append(inst_id)
+    return INSTANCE_IDS
+
+
+def get_new_pubDNS():
+    eip = conn.allocate_address()
+    return eip
+
+if __name__ == "__main__":
+    # hostlist.tmp (JSON) erzeugen
+    instlist = prepare_list(tag)
+
+    for entry in range(0,len(instlist)):
+
+        instance_id = instlist[entry]
+        print(instance_id)
+        newpubDNS = get_new_pubDNS()
+        inID = str(newpubDNS).split(':')[1]
+        print(inID)
+        conn.associate_address(instance_id, public_ip=inID)
+
+    # make sure all addresse are assigned...
+    time.sleep(30)

bigchain-deployment/get_instance_status.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import json
+import os
+import boto.ec2
+import time
+import argparse
+
+
+AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
+AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--tag", help="tag instances in aws")
+args = parser.parse_args()
+
+if args.tag:
+    tag = args.tag
+else:
+    # reading credentials from config for remote connection
+    print('usage: python3 get_instance_status.py --tag <tag>')
+    print('reason: tag missing!!!')
+    exit(1)
+
+conn = boto.ec2.connect_to_region("eu-central-1",
+            aws_access_key_id=AWS_ACCESS_KEY_ID,
+            aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
+
+PASSING = []
+RUNNING = []
+
+
+# get list of instance ids from amazon
+def list_of_ids(tag):
+    # TODO: CHANGE TO PROPER DOCSTRING
+    # Returns a list of ids of all instances with the given tag
+    reservations = conn.get_all_instances(filters={"tag:Name": tag})
+    # There are several reservations
+    # and each reservation can have several instances
+    id_list = []
+    for reservation in reservations:
+        for instance in reservation.instances:
+            if instance.id is not None:
+                id_list.append(instance.id)
+    return id_list
+
+
+# Andreas' old code:
+"""
+INSTANCE_IDS = []
+
+def prepare_list(tag):
+    reservations = conn.get_all_instances(filters={"tag:Name" : tag})
+    instances = [i for r in reservations for i in r.instances]
+    for i in instances:
+        inst = i.__dict__
+        inst_id = inst.get('id')
+        INSTANCE_IDS.append(inst_id)
+    return INSTANCE_IDS
+"""
+
+
+# get statuses from amazon
+def create_run_pass_list(tag):
+    # instlist_old = prepare_list(tag)
+    # print("instlist_old = {}".format(instlist_old))
+    instlist_new = list_of_ids(tag)
+    print("instlist_new = {}".format(instlist_new))
+
+    instlist = instlist_new
+
+    for entry in range(0, len(instlist)):  # 0, 1, ..., [len(instlist) - 1]
+        instances = conn.get_only_instances(instance_ids=instlist[entry])
+        status = conn.get_all_instance_status(instance_ids=instlist[entry])
+        for instance in instances:
+            reachability = status[0].system_status.details["reachability"]
+            PASSING.append(reachability)
+    return instlist, PASSING, RUNNING
+
+
+if __name__ == "__main__":
+    # get lists from amazon
+    try:
+        instlist, passlist, runlist = create_run_pass_list(tag)
+        print("instlist = {}".format(instlist))
+        print("passlist = {}".format(passlist))
+        print("runlist = {}".format(runlist))
+    except IndexError:
+        print("Searching for matching cluster-tag...")
+        exit(1)
+
+    for entry in range(0,len(instlist)):
+        if "passed" in passlist and len(set(passlist)) == 1:
+            print("up and running")
+            exit(0)
+
+    # exit with error code for continous check if nothing found
+    exit(1)

bigchain-deployment/hostlist.py

@@ -0,0 +1 @@
+hosts_dev =  ['ec2-52-58-94-210.eu-central-1.compute.amazonaws.com', 'ec2-52-29-236-2.eu-central-1.compute.amazonaws.com']

bigchain-deployment/run_and_tag.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import boto.ec2
+import time
+import argparse
+import os
+
+AWS_ACCESS_KEY_ID = os.environ["AWS_ACCESS_KEY_ID"]
+AWS_SECRET_ACCESS_KEY = os.environ["AWS_SECRET_ACCESS_KEY"]
+
+parser = argparse.ArgumentParser()
+parser.add_argument("--tag", help="tag instances in aws")
+parser.add_argument("--nodes", help="number of nodes in the cluster")
+args = parser.parse_args()
+
+if args.tag:
+    tag = args.tag
+else:
+    # reading credentials from config for remote connection
+    print('usage: python3 run_and_tag.py --tag <tag> --nodes <number of nodes in cluster>')
+    print('reason: tag missing!!!')
+    exit(1)
+
+if args.nodes:
+    nodes = int(args.nodes)
+else:
+    print('usage: python3 run_and_tag.py --tag <tag> --nodes <number of nodes in cluster>')
+    print('reason: nodes missing!!!')
+    exit(1)
+
+conn = boto.ec2.connect_to_region("eu-central-1",
+                aws_access_key_id=AWS_ACCESS_KEY_ID,
+                aws_secret_access_key=AWS_SECRET_ACCESS_KEY)
+
+for _ in range(nodes):  # 0, 1, ..., (nodes-1) = nodes items
+    reservation = conn.run_instances(
+            'ami-accff2b1',                    # ubuntu-image
+            #'ami-596b7235',                    # ubuntu w/ iops storage
+            key_name='bigchain',
+            # IMPORTANT!!!! - here you change the machine type for the cluster
+            instance_type='m3.2xlarge',
+            #instance_type='c3.8xlarge',
+            #instance_type='c4.8xlarge',
+            security_groups=['bigchain'])
+
+    for instance in reservation.instances:
+        time.sleep(5)
+        instance.add_tag('Name', tag)

bigchain-deployment/startup.sh

@@ -0,0 +1,97 @@
+#! /bin/bash
+
+function printErr()
+    {
+        echo "usage: ./startup.sh <tag> <number_of_nodes_in_cluster>"
+        echo "No argument $1 supplied"
+    }
+
+if [ -z "$1" ]
+  then
+    printErr "<tag>"
+    exit 1
+fi
+
+if [ -z "$2" ]
+  then
+    printErr "<number_of_nodes_in_cluster>"
+    exit 1
+fi
+
+TAG=$1
+NODES=$2
+AWS=`which aws`
+FAB=`which fab`
+DEPLOYDIR=`pwd`
+BIGCHAINDIR=`dirname $DEPLOYDIR`
+export BIGCHAINDIR
+
+# check if awscli is installed
+if [ ! -f "$AWS" ]
+    then
+        echo "awscli is not installed!!!"
+        exit 1
+fi
+
+# checck if python-fabric is installed
+if [ ! -f "$FAB" ]
+    then
+        echo "python-fabric is not installed!!!"
+        exit 1
+fi
+
+# checking pem-file and changing access rights
+if [ ! -f "pem/bigchain.pem" ]
+    then
+        echo "Valid key is missing!!!"
+        exit 1
+fi
+# 0400 for pem/bigchain.pem
+chmod 0400 pem/bigchain.pem
+
+# starting and tagging instances
+python3 run_and_tag.py --tag $TAG --nodes $NODES
+# let's wait a minute to get the nodes ready and in status initializing
+#sleep 60
+
+# checking if instances are up and running (every 5 secs.)
+RET=1
+until [ ${RET} -eq 0 ]; do
+    python3 get_instance_status.py --tag $TAG
+    RET=$?
+    sleep 5
+done
+
+# in case of elastic ips...
+python3 get_elastic_ips.py --tag $TAG
+
+# everything prepared. now wait until instances up and running!
+# generate hostlist.py and add_keys.sh
+python3 create_hostlist.py --tag $TAG > hostlist.py
+# make add_keys executable and execute
+chmod +x add2known_hosts.sh
+./add2known_hosts.sh
+
+# resetting the rethinkdb initfile and adding the nodes to join...
+cp bigchain/bigchaindb.conf.template bigchain/bigchaindb.conf
+cat add2dbconf >> bigchain/bigchaindb.conf
+
+# rollout base packages for installation of storage and bigchain
+fab install_base_software
+
+# rollout storagebackend (rethinkdb)
+fab install_rethinkdb
+
+# rollout bigchain-reporitory
+fab install_bigchain
+
+# generate genesisblock
+HORST=`tail -1 bigchain/bigchaindb.conf|cut -d: -f1|cut -d= -f2`
+fab -H $HORST -f fab_prepare_chain.py init_bigchaindb
+# initiate sharding
+fab start_bigchain_nodes
+
+# now cleanup!
+rm add2known_hosts.sh add2dbconf
+
+# DONE!