Fixed merge conflict in bigchaindb/__init__.py

.gitignore

@@ -77,3 +77,7 @@ benchmarking-tests/ssh_key.py
 # Ansible-specific files
 ntools/one-m/ansible/hosts
 ntools/one-m/ansible/ansible.cfg
+
+# Just in time documentation
+docs/server/source/schema
+docs/server/source/drivers-clients/samples

CHANGELOG.md

@@ -1,4 +1,5 @@
 # Change Log (Release Notes)
+
 All _notable_ changes to this project will be documented in this file (`CHANGELOG.md`).
 This project adheres to [Semantic Versioning](http://semver.org/) (or at least we try).
 Contributors to this file, please follow the guidelines on [keepachangelog.com](http://keepachangelog.com/).
@@ -15,10 +16,74 @@ For reference, the possible headings are:
 * **Notes**
 
 
+## [0.8.0] - 2016-11-29
+Tag name: v0.8.0
+= commit:
+committed:
+
+### Added
+- The big new thing in version 0.8.0 is support for divisible assets, i.e. assets like carrots or thumbtacks, where the initial CREATE transaction can register/create some amount (e.g. 542 carrots), the first TRANSFER transaction can split that amount across multiple owners, and so on. [Pull Request #794](https://github.com/bigchaindb/bigchaindb/pull/794)
+- Wrote a formal schema for the JSON structure of transactions. Transactions are now checked against that schema. [Pull Request #798](https://github.com/bigchaindb/bigchaindb/pull/798)
+- New configuration parameter: `backlog_reassign_delay`. [Pull Request #883](https://github.com/bigchaindb/bigchaindb/pull/883)
+
+### Changed
+- CREATE transactions must now be signed by all `owners_before` (rather than by a federation node). [Pull Request #794](https://github.com/bigchaindb/bigchaindb/pull/794)
+- The user-provided timestamp was removed from the transaction data model (schema). [Pull Request #817](https://github.com/bigchaindb/bigchaindb/pull/817)
+- `get_transaction()` will now return a transaction from the backlog, even if there are copies of the transaction in invalid blocks. [Pull Request #793](https://github.com/bigchaindb/bigchaindb/pull/793)
+- Several pull requests to introduce a generalized database interface, to move RethinkDB calls into a separate implementation of that interface, and to work on a new MongoDB implementation of that interface. Pull Requests
+[#754](https://github.com/bigchaindb/bigchaindb/pull/754),
+[#783](https://github.com/bigchaindb/bigchaindb/pull/783),
+[#799](https://github.com/bigchaindb/bigchaindb/pull/799),
+[#806](https://github.com/bigchaindb/bigchaindb/pull/806),
+[#809](https://github.com/bigchaindb/bigchaindb/pull/809),
+[#853](https://github.com/bigchaindb/bigchaindb/pull/853)
+- Renamed "verifying key" to "public key". Renamed "signing key" to "private key". Renamed "vk" to "pk". [Pull Request #807](https://github.com/bigchaindb/bigchaindb/pull/807)
+- `get_transaction_by_asset_id` now ignores invalid transactions. [Pull Request #810](https://github.com/bigchaindb/bigchaindb/pull/810)
+- `get_transaction_by_metadata_id` now ignores invalid transactions. [Pull Request #811](https://github.com/bigchaindb/bigchaindb/pull/811)
+- Updates to the configs and scripts for deploying a test network on AWS. The example config file deploys virtual machines running Ubuntu 16.04 now. Pull Requests 
+[#771](https://github.com/bigchaindb/bigchaindb/pull/771),
+[#813](https://github.com/bigchaindb/bigchaindb/pull/813)
+- Changed logging of transactions on block creation so now it just says the length of the list of transactions, rather than listing all the transactions. [Pull Request #861](https://github.com/bigchaindb/bigchaindb/pull/861)
+
+### Fixed
+- Equality checks with AssetLinks. [Pull Request #825](https://github.com/bigchaindb/bigchaindb/pull/825)
+- Bug in `bigchaindb load`. [Pull Request #824](https://github.com/bigchaindb/bigchaindb/pull/824)
+- Two issues found with timestamp indexes. [Pull Request #816](https://github.com/bigchaindb/bigchaindb/pull/816)
+- Hard-coded `backlog_reassign_delay`. [Pull Request #854](https://github.com/bigchaindb/bigchaindb/pull/854)
+- Race condition in `test_stale_monitor.py`. [Pull Request #846](https://github.com/bigchaindb/bigchaindb/pull/846)
+- When creating a signed vote, decode the vote signature to a `str`. [Pull Request #869](https://github.com/bigchaindb/bigchaindb/pull/869)
+- Bug in AWS deployment scripts. Setting `BIND_HTTP_TO_LOCALHOST` to `False` didn't actually work. It does now. [Pull Request #870](https://github.com/bigchaindb/bigchaindb/pull/870)
+
+### External Contributors
+- @najlachamseddine - [Pull Request #528](https://github.com/bigchaindb/bigchaindb/pull/528)
+- @ChristianGaertner - [Pull Request #659](https://github.com/bigchaindb/bigchaindb/pull/659)
+- @MinchinWeb - [Pull Request #695](https://github.com/bigchaindb/bigchaindb/pull/695)
+- @ckeyer - [Pull Request #785](https://github.com/bigchaindb/bigchaindb/pull/785)
+
+### Notes
+- @ChristianGaertner added a Python style checker (Flake8) to Travis CI, so external contributors should be aware that the Python code in their pull requests will be checked. See [our Python Style Guide](PYTHON_STYLE_GUIDE.md).
+- Several additions and changes to the documentation, e.g. Pull Requests
+[#690](https://github.com/bigchaindb/bigchaindb/pull/690),
+[#764](https://github.com/bigchaindb/bigchaindb/pull/764),
+[#766](https://github.com/bigchaindb/bigchaindb/pull/766),
+[#769](https://github.com/bigchaindb/bigchaindb/pull/769),
+[#777](https://github.com/bigchaindb/bigchaindb/pull/777),
+[#800](https://github.com/bigchaindb/bigchaindb/pull/800),
+[#801](https://github.com/bigchaindb/bigchaindb/pull/801),
+[#802](https://github.com/bigchaindb/bigchaindb/pull/802),
+[#803](https://github.com/bigchaindb/bigchaindb/pull/803),
+[#819](https://github.com/bigchaindb/bigchaindb/pull/819),
+[#827](https://github.com/bigchaindb/bigchaindb/pull/827),
+[#859](https://github.com/bigchaindb/bigchaindb/pull/859),
+[#872](https://github.com/bigchaindb/bigchaindb/pull/872),
+[#882](https://github.com/bigchaindb/bigchaindb/pull/882),
+[#883](https://github.com/bigchaindb/bigchaindb/pull/883)
+
+
 ## [0.7.0] - 2016-10-28
 Tag name: v0.7.0
-= commit: 
-committed: 
+= commit: 2dd7f1af27478c529e6d2d916f64daa3fbda3885
+committed: Oct 28, 2016, 4:00 PM GMT+2
 
 ### Added
 - Stale transactions in the `backlog` table now get reassigned to another node (for inclusion in a new block): [Pull Request #359](https://github.com/bigchaindb/bigchaindb/pull/359)

CONTRIBUTING.md

@@ -121,8 +121,6 @@ git merge upstream/master
 
 Once you're done commiting a set of new things and you're ready to submit them for inclusion, please be sure to run all the tests (as per the instructions at the end of our [Python Style Guide](PYTHON_STYLE_GUIDE.md)).
 
-If your addition or change is substantial, then please add a line or two to the [CHANGELOG.md file](https://github.com/bigchaindb/bigchaindb/blob/master/CHANGELOG.md), following the guidelines given at the top of that file.
-
 (When you submit your pull request [following the instructions below], we run all the tests automatically, so we will see if some are failing. If you don't know why some tests are failing, you can still submit your pull request, but be sure to note the failing tests and to ask for help with resolving them.)
 
 ### Step 9 - Push Your New Branch to origin

Release_Process.md

@@ -2,16 +2,17 @@
 
 This is a summary of the steps we go through to release a new version of BigchainDB Server.
 
+1. Run `python docs/server/generate_schema_documentation.py` and commit the changes in `docs/server/source/schema/`, if any.
 1. Update the `CHANGELOG.md` file
-2. Update the version numbers in `bigchaindb/version.py`. Note that we try to use [semantic versioning](http://semver.org/) (i.e. MAJOR.MINOR.PATCH)
-3. Go to the [bigchaindb/bigchaindb Releases page on GitHub](https://github.com/bigchaindb/bigchaindb/releases)
+1. Update the version numbers in `bigchaindb/version.py`. Note that we try to use [semantic versioning](http://semver.org/) (i.e. MAJOR.MINOR.PATCH)
+1. Go to the [bigchaindb/bigchaindb Releases page on GitHub](https://github.com/bigchaindb/bigchaindb/releases)
    and click the "Draft a new release" button
-4. Name the tag something like v0.7.0
-5. The target should be a specific commit: the one when the update of `bigchaindb/version.py` got merged into master
-6. The release title should be something like v0.7.0
-7. The description should be copied from the `CHANGELOG.md` file updated above
-8. Generate and send the latest `bigchaindb` package to PyPI. Dimi and Sylvain can do this, maybe others
-9. Login to readthedocs.org as a maintainer of the BigchainDB Server docs.
+1. Name the tag something like v0.7.0
+1. The target should be a specific commit: the one when the update of `bigchaindb/version.py` got merged into master
+1. The release title should be something like v0.7.0
+1. The description should be copied from the `CHANGELOG.md` file updated above
+1. Generate and send the latest `bigchaindb` package to PyPI. Dimi and Sylvain can do this, maybe others
+1. Login to readthedocs.org as a maintainer of the BigchainDB Server docs.
    Go to Admin --> Versions and under **Choose Active Versions**, make sure that the new version's tag is
    "Active" and "Public"
 

bigchaindb/__init__.py

@@ -29,7 +29,7 @@ config = {
         'port': 8125,
         'rate': 0.01,
     },
-    'backlog_reassign_delay': 30
+    'backlog_reassign_delay': 120
 }
 
 # We need to maintain a backup copy of the original config dict in case

bigchaindb/commands/bigchain.py

@@ -1,6 +1,5 @@
 """Implementation of the `bigchaindb` command,
-which is one of the commands in the BigchainDB
-command-line interface.
+the command-line interface (CLI) for BigchainDB Server.
 """
 
 import os
@@ -194,7 +193,7 @@ def _run_load(tx_left, stats):
     b = bigchaindb.Bigchain()
 
     while True:
-        tx = Transaction.create([b.me], [b.me])
+        tx = Transaction.create([b.me], [([b.me], 1)])
         tx = tx.sign([b.me_private])
         b.write_transaction(tx)
 

bigchaindb/common/crypto.py

@@ -15,5 +15,5 @@ def generate_key_pair():
     return private_key.decode(), public_key.decode()
 
 
-SigningKey = crypto.Ed25519SigningKey
-VerifyingKey = crypto.Ed25519VerifyingKey
+PrivateKey = crypto.Ed25519SigningKey
+PublicKey = crypto.Ed25519VerifyingKey

bigchaindb/common/exceptions.py

@@ -22,11 +22,19 @@ class DoubleSpend(Exception):
     """Raised if a double spend is found"""
 
 
-class InvalidHash(Exception):
+class ValidationError(Exception):
+    """Raised if there was an error in validation"""
+
+
+class InvalidHash(ValidationError):
     """Raised if there was an error checking the hash for a particular
     operation"""
 
 
+class SchemaValidationError(ValidationError):
+    """Raised if there was any error validating an object's schema"""
+
+
 class InvalidSignature(Exception):
     """Raised if there was an error checking the signature for a particular
     operation"""

bigchaindb/common/schema/README.md

@@ -0,0 +1,30 @@
+# Introduction
+
+This directory contains the schemas for the different JSON documents BigchainDB uses.
+
+The aim is to provide:
+ - a strict definition/documentation of the data structures used in BigchainDB
+ - a language independent tool to validate the structure of incoming/outcoming
+   data (there are several ready to use
+   [implementations](http://json-schema.org/implementations.html) written in
+   different languages)
+
+## Learn about JSON Schema
+
+A good resource is [Understanding JSON Schema](http://spacetelescope.github.io/understanding-json-schema/index.html).
+It provides a *more accessible documentation for JSON schema* than the [specs](http://json-schema.org/documentation.html).
+
+## If it's supposed to be JSON, why's everything in YAML D:?
+
+YAML is great for its conciseness and friendliness towards human-editing in comparision to JSON.
+
+Although YAML is a superset of JSON, at the end of the day, JSON Schema processors, like
+[json-schema](http://python-jsonschema.readthedocs.io/en/latest/), take in a native object (e.g.
+Python dicts or JavaScript objects) as the schema used for validation. As long as we can serialize
+the YAML into what the JSON Schema processor expects (almost always as simple as loading the YAML
+like you would with a JSON file), it's the same as using JSON.
+
+Specific advantages of using YAML:
+ - Legibility, especially when nesting
+ - Multi-line string literals, that make it easy to include descriptions that can be [auto-generated
+   into Sphinx documentation](/docs/server/generate_schema_documentation.py)

bigchaindb/common/schema/__init__.py

@@ -0,0 +1,24 @@
+""" Schema validation related functions and data """
+import os.path
+
+import jsonschema
+import yaml
+
+from bigchaindb.common.exceptions import SchemaValidationError
+
+
+TX_SCHEMA_PATH = os.path.join(os.path.dirname(__file__), 'transaction.yaml')
+with open(TX_SCHEMA_PATH) as handle:
+    TX_SCHEMA_YAML = handle.read()
+TX_SCHEMA = yaml.safe_load(TX_SCHEMA_YAML)
+
+
+def validate_transaction_schema(tx_body):
+    """ Validate a transaction dict against a schema """
+    try:
+        jsonschema.validate(tx_body, TX_SCHEMA)
+    except jsonschema.ValidationError as exc:
+        raise SchemaValidationError(str(exc))
+
+
+__all__ = ['TX_SCHEMA', 'TX_SCHEMA_YAML', 'validate_transaction_schema']

bigchaindb/common/schema/transaction.yaml

@@ -0,0 +1,242 @@
+---
+"$schema": "http://json-schema.org/draft-04/schema#"
+id: "http://www.bigchaindb.com/schema/transaction.json"
+type: object
+additionalProperties: false
+title: Transaction Schema
+description: |
+    A transaction represents the creation or transfer of assets in BigchainDB.
+required:
+- id
+- fulfillments
+- conditions
+- operation
+- metadata
+- asset
+- version
+properties:
+  id:
+    "$ref": "#/definitions/sha3_hexdigest"
+    description: |
+        A sha3 digest of the transaction. The ID is calculated by removing all
+        derived hashes and signatures from the transaction, serializing it to
+        JSON with keys in sorted order and then hashing the resulting string
+        with sha3.
+  operation:
+    "$ref": "#/definitions/operation"
+  asset:
+    "$ref": "#/definitions/asset"
+    description: |
+      Description of the asset being transacted.
+
+      See: `Asset`_.
+  fulfillments:
+    type: array
+    title: "Fulfillments list"
+    description: |
+        Array of the fulfillments (inputs) of a transaction.
+
+        See: Fulfillment_.
+    items:
+      "$ref": "#/definitions/fulfillment"
+  conditions:
+    type: array
+    description: |
+        Array of conditions (outputs) provided by this transaction.
+
+        See: Condition_.
+    items:
+      "$ref": "#/definitions/condition"
+  metadata:
+    "$ref": "#/definitions/metadata"
+    description: |
+      User provided transaction metadata. This field may be ``null`` or may
+      contain an id and an object with freeform metadata.
+
+      See: `Metadata`_.
+  version:
+    type: integer
+    minimum: 1
+    maximum: 1
+    description: |
+        BigchainDB transaction schema version.
+definitions:
+  offset:
+    type: integer
+    minimum: 0
+  base58:
+    pattern: "[1-9a-zA-Z^OIl]{43,44}"
+    type: string
+  owners_list:
+    anyOf:
+    - type: array
+      items:
+        "$ref": "#/definitions/base58"
+    - type: 'null'
+  sha3_hexdigest:
+    pattern: "[0-9a-f]{64}"
+    type: string
+  uuid4:
+    pattern: "[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}"
+    type: string
+    description: |
+        A `UUID <https://tools.ietf.org/html/rfc4122.html>`_
+        of type 4 (random).
+  operation:
+    type: string
+    description: |
+        Type of the transaction:
+
+        A ``CREATE`` transaction creates an asset in BigchainDB. This
+        transaction has outputs (conditions) but no inputs (fulfillments),
+        so a dummy fulfillment is used.
+
+        A ``TRANSFER`` transaction transfers ownership of an asset, by providing
+        fulfillments to conditions of earlier transactions.
+
+        A ``GENESIS`` transaction is a special case transaction used as the
+        sole member of the first block in a BigchainDB ledger.
+    enum:
+    - CREATE
+    - TRANSFER
+    - GENESIS
+  asset:
+    type: object
+    description: |
+        Description of the asset being transacted. In the case of a ``TRANSFER``
+        transaction, this field contains only the ID of asset. In the case
+        of a ``CREATE`` transaction, this field may contain properties:
+    additionalProperties: false
+    required:
+    - id
+    properties:
+      id:
+        "$ref": "#/definitions/uuid4"
+      divisible:
+        type: boolean
+        description: |
+            Whether or not the asset has a quantity that may be partially spent.
+      updatable:
+        type: boolean
+        description: |
+            Whether or not the description of the asset may be updated. Defaults to false.
+      refillable:
+        type: boolean
+        description: |
+          Whether the amount of the asset can change after its creation. Defaults to false.
+      data:
+        description: |
+          User provided metadata associated with the asset. May also be ``null``.
+        anyOf:
+        - type: object
+          additionalProperties: true
+        - type: 'null'
+  condition:
+    type: object
+    description: |
+        An output of a transaction. A condition describes a quantity of an asset
+        and what conditions must be met in order for it to be fulfilled. See also:
+        fulfillment_.
+    additionalProperties: false
+    required:
+    - owners_after
+    - condition
+    - amount
+    properties:
+      cid:
+        "$ref": "#/definitions/offset"
+        description: |
+            Index of this condition's appearance in the `Transaction.conditions`_
+            array. In a transaction with 2 conditions, the ``cid``s will be 0 and 1.
+      condition:
+        description: |
+            Body of the condition. Has the properties:
+
+            - **details**: Details of the condition.
+            - **uri**: Condition encoded as an ASCII string.
+        type: object
+        additionalProperties: false
+        required:
+        - details
+        - uri
+        properties:
+          details:
+            type: object
+            additionalProperties: true
+          uri:
+            type: string
+            pattern: "^cc:([1-9a-f][0-9a-f]{0,3}|0):[1-9a-f][0-9a-f]{0,15}:[a-zA-Z0-9_-]{0,86}:([1-9][0-9]{0,17}|0)$"
+      owners_after:
+        "$ref": "#/definitions/owners_list"
+        description: |
+            List of public keys associated with asset ownership at the time
+            of the transaction.
+      amount:
+        type: integer
+        description: |
+            Integral amount of the asset represented by this condition.
+            In the case of a non divisible asset, this will always be 1.
+  fulfillment:
+    type: "object"
+    description:
+      A fulfillment is an input to a transaction, named as such because it
+      fulfills a condition of a previous transaction. In the case of a
+      ``CREATE`` transaction, a fulfillment may provide no ``input``.
+    additionalProperties: false
+    required:
+    - owners_before
+    - input
+    - fulfillment
+    properties:
+      fid:
+        "$ref": "#/definitions/offset"
+        description: |
+            The offset of the fulfillment within the fulfillents array.
+      owners_before:
+        "$ref": "#/definitions/owners_list"
+        description: |
+            List of public keys of the previous owners of the asset.
+      fulfillment:
+        anyOf:
+        - type: object
+          additionalProperties: false
+          properties:
+            bitmask:
+              type: integer
+            public_key:
+              type: string
+            type:
+              type: string
+            signature:
+              anyOf:
+              - type: string
+              - type: 'null'
+            type_id:
+              type: integer
+          description: |
+              Fulfillment of a condition_, or put a different way, this is a
+              payload that satisfies a condition in order to spend the associated
+              asset.
+        - type: string
+          pattern: "^cf:([1-9a-f][0-9a-f]{0,3}|0):[a-zA-Z0-9_-]*$"
+      input:
+        anyOf:
+        - type: 'object'
+          description: |
+            Reference to a condition of a previous transaction
+          additionalProperties: false
+          properties:
+            cid:
+              "$ref": "#/definitions/offset"
+            txid:
+              "$ref": "#/definitions/sha3_hexdigest"
+        - type: 'null'
+  metadata:
+    anyOf:
+    - type: object
+      description: |
+        User provided transaction metadata. This field may be ``null`` or may
+        contain an non empty object with freeform metadata.
+      additionalProperties: true
+      minProperties: 1
+    - type: 'null'

bigchaindb/common/transaction.py

@@ -6,7 +6,7 @@ from cryptoconditions import (Fulfillment as CCFulfillment,
                               ThresholdSha256Fulfillment, Ed25519Fulfillment)
 from cryptoconditions.exceptions import ParsingError
 
-from bigchaindb.common.crypto import SigningKey, hash_data
+from bigchaindb.common.crypto import PrivateKey, hash_data
 from bigchaindb.common.exceptions import (KeypairMismatchException,
                                           InvalidHash, InvalidSignature,
                                           AmountError, AssetIdMismatch)
@@ -541,7 +541,7 @@ class AssetLink(Asset):
 
     def __eq__(self, other):
         return isinstance(other, AssetLink) and \
-                self.to_dict() == self.to_dict()
+                self.to_dict() == other.to_dict()
 
     @classmethod
     def from_dict(cls, link):
@@ -572,67 +572,6 @@ class AssetLink(Asset):
             }
 
 
-class Metadata(object):
-    """Metadata is used to store a dictionary and its hash in a Transaction."""
-
-    def __init__(self, data=None, data_id=None):
-        """Metadata stores a payload `data` as well as data's hash, `data_id`.
-
-            Note:
-                When no `data_id` is provided, one is being generated by
-                this method.
-
-            Args:
-                data (dict): A dictionary to be held by Metadata.
-                data_id (str): A hash corresponding to the contents of
-                    `data`.
-        """
-        if data is not None and not isinstance(data, dict):
-            raise TypeError('`data` must be a dict instance or None')
-
-        # TODO: Rename `payload_id` to `id`
-        self.data_id = data_id if data_id is not None else self.to_hash()
-        self.data = data
-
-    def __eq__(self, other):
-        # TODO: If `other !== Data` return `False`
-        return self.to_dict() == other.to_dict()
-
-    @classmethod
-    def from_dict(cls, data):
-        """Transforms a Python dictionary to a Metadata object.
-
-            Args:
-                data (dict): The dictionary to be serialized.
-
-            Returns:
-                :class:`~bigchaindb.common.transaction.Metadata`
-        """
-        try:
-            return cls(data['data'], data['id'])
-        except TypeError:
-            return cls()
-
-    def to_dict(self):
-        """Transforms the object to a Python dictionary.
-
-            Returns:
-                (dict|None): The Metadata object as an alternative
-                    serialization format.
-        """
-        if self.data is None:
-            return None
-        else:
-            return {
-                'data': self.data,
-                'id': self.data_id,
-            }
-
-    def to_hash(self):
-        """A hash corresponding to the contents of `payload`."""
-        return str(uuid4())
-
-
 class Transaction(object):
     """A Transaction is used to create and transfer assets.
 
@@ -647,9 +586,8 @@ class Transaction(object):
                 spend.
             conditions (:obj:`list` of :class:`~bigchaindb.common.
                 transaction.Condition`, optional): Define the assets to lock.
-            metadata (:class:`~bigchaindb.common.transaction.Metadata`):
+            metadata (dict):
                 Metadata to be stored along with the Transaction.
-            timestamp (int): Defines the time a Transaction was created.
             version (int): Defines the version number of a Transaction.
     """
     CREATE = 'CREATE'
@@ -659,11 +597,11 @@ class Transaction(object):
     VERSION = 1
 
     def __init__(self, operation, asset, fulfillments=None, conditions=None,
-                 metadata=None, timestamp=None, version=None):
+                 metadata=None, version=None):
         """The constructor allows to create a customizable Transaction.
 
             Note:
-                When no `version` or `timestamp`, is provided, one is being
+                When no `version` is provided, one is being
                 generated by this method.
 
             Args:
@@ -676,9 +614,8 @@ class Transaction(object):
                 conditions (:obj:`list` of :class:`~bigchaindb.common.
                     transaction.Condition`, optional): Define the assets to
                     lock.
-                metadata (:class:`~bigchaindb.common.transaction.Metadata`):
+                metadata (dict):
                     Metadata to be stored along with the Transaction.
-                timestamp (int): Defines the time a Transaction was created.
                 version (int): Defines the version number of a Transaction.
 
         """
@@ -698,11 +635,10 @@ class Transaction(object):
         if fulfillments and not isinstance(fulfillments, list):
             raise TypeError('`fulfillments` must be a list instance or None')
 
-        if metadata is not None and not isinstance(metadata, Metadata):
-            raise TypeError('`metadata` must be a Metadata instance or None')
+        if metadata is not None and not isinstance(metadata, dict):
+            raise TypeError('`metadata` must be a dict or None')
 
         self.version = version if version is not None else self.VERSION
-        self.timestamp = timestamp if timestamp else gen_timestamp()
         self.operation = operation
         self.asset = asset if asset else Asset()
         self.conditions = conditions if conditions else []
@@ -754,7 +690,6 @@ class Transaction(object):
         if len(owners_after) == 0:
             raise ValueError('`owners_after` list cannot be empty')
 
-        metadata = Metadata(metadata)
         fulfillments = []
         conditions = []
 
@@ -829,7 +764,6 @@ class Transaction(object):
             pub_keys, amount = owner_after
             conditions.append(Condition.generate(pub_keys, amount))
 
-        metadata = Metadata(metadata)
         inputs = deepcopy(inputs)
         return cls(cls.TRANSFER, asset, inputs, conditions, metadata)
 
@@ -933,8 +867,8 @@ class Transaction(object):
             # to decode to convert the bytestring into a python str
             return public_key.decode()
 
-        key_pairs = {gen_public_key(SigningKey(private_key)):
-                     SigningKey(private_key) for private_key in private_keys}
+        key_pairs = {gen_public_key(PrivateKey(private_key)):
+                     PrivateKey(private_key) for private_key in private_keys}
 
         for index, fulfillment in enumerate(self.fulfillments):
             # NOTE: We clone the current transaction but only add the condition
@@ -942,7 +876,7 @@ class Transaction(object):
             #       previously signed ones.
             tx_partial = Transaction(self.operation, self.asset, [fulfillment],
                                      self.conditions, self.metadata,
-                                     self.timestamp, self.version)
+                                     self.version)
 
             tx_partial_dict = tx_partial.to_dict()
             tx_partial_dict = Transaction._remove_signatures(tx_partial_dict)
@@ -1104,8 +1038,7 @@ class Transaction(object):
             Transactions.
             """
             tx = Transaction(self.operation, self.asset, [fulfillment],
-                             self.conditions, self.metadata, self.timestamp,
-                             self.version)
+                             self.conditions, self.metadata, self.version)
             tx_dict = tx.to_dict()
             tx_dict = Transaction._remove_signatures(tx_dict)
             tx_serialized = Transaction._to_str(tx_dict)
@@ -1171,31 +1104,21 @@ class Transaction(object):
             Returns:
                 dict: The Transaction as an alternative serialization format.
         """
-        try:
-            metadata = self.metadata.to_dict()
-        except AttributeError:
-            # NOTE: metadata can be None and that's OK
-            metadata = None
-
         if self.operation in (self.__class__.GENESIS, self.__class__.CREATE):
             asset = self.asset.to_dict()
         else:
             # NOTE: An `asset` in a `TRANSFER` only contains the asset's id
             asset = {'id': self.asset.data_id}
 
-        tx_body = {
+        tx = {
             'fulfillments': [fulfillment.to_dict(fid) for fid, fulfillment
                              in enumerate(self.fulfillments)],
             'conditions': [condition.to_dict(cid) for cid, condition
                            in enumerate(self.conditions)],
             'operation': str(self.operation),
-            'timestamp': self.timestamp,
-            'metadata': metadata,
+            'metadata': self.metadata,
             'asset': asset,
-        }
-        tx = {
             'version': self.version,
-            'transaction': tx_body,
         }
 
         tx_no_signatures = Transaction._remove_signatures(tx)
@@ -1220,7 +1143,7 @@ class Transaction(object):
         # NOTE: We remove the reference since we need `tx_dict` only for the
         #       transaction's hash
         tx_dict = deepcopy(tx_dict)
-        for fulfillment in tx_dict['transaction']['fulfillments']:
+        for fulfillment in tx_dict['fulfillments']:
             # NOTE: Not all Cryptoconditions return a `signature` key (e.g.
             #       ThresholdSha256Fulfillment), so setting it to `None` in any
             #       case could yield incorrect signatures. This is why we only
@@ -1248,16 +1171,12 @@ class Transaction(object):
         tx = Transaction._remove_signatures(self.to_dict())
         return Transaction._to_str(tx)
 
-    @classmethod
-    # TODO: Make this method more pretty
-    def from_dict(cls, tx_body):
-        """Transforms a Python dictionary to a Transaction object.
+    @staticmethod
+    def validate_structure(tx_body):
+        """Validate the transaction ID of a transaction
 
             Args:
                 tx_body (dict): The Transaction to be transformed.
-
-            Returns:
-                :class:`~bigchaindb.common.transaction.Transaction`
         """
         # NOTE: Remove reference to avoid side effects
         tx_body = deepcopy(tx_body)
@@ -1272,17 +1191,26 @@ class Transaction(object):
 
         if proposed_tx_id != valid_tx_id:
             raise InvalidHash()
-        else:
-            tx = tx_body['transaction']
-            fulfillments = [Fulfillment.from_dict(fulfillment) for fulfillment
-                            in tx['fulfillments']]
-            conditions = [Condition.from_dict(condition) for condition
-                          in tx['conditions']]
-            metadata = Metadata.from_dict(tx['metadata'])
-            if tx['operation'] in [cls.CREATE, cls.GENESIS]:
-                asset = Asset.from_dict(tx['asset'])
-            else:
-                asset = AssetLink.from_dict(tx['asset'])
 
-            return cls(tx['operation'], asset, fulfillments, conditions,
-                       metadata, tx['timestamp'], tx_body['version'])
+    @classmethod
+    def from_dict(cls, tx):
+        """Transforms a Python dictionary to a Transaction object.
+
+            Args:
+                tx_body (dict): The Transaction to be transformed.
+
+            Returns:
+                :class:`~bigchaindb.common.transaction.Transaction`
+        """
+        cls.validate_structure(tx)
+        fulfillments = [Fulfillment.from_dict(fulfillment) for fulfillment
+                        in tx['fulfillments']]
+        conditions = [Condition.from_dict(condition) for condition
+                      in tx['conditions']]
+        if tx['operation'] in [cls.CREATE, cls.GENESIS]:
+            asset = Asset.from_dict(tx['asset'])
+        else:
+            asset = AssetLink.from_dict(tx['asset'])
+
+        return cls(tx['operation'], asset, fulfillments, conditions,
+                   tx['metadata'], tx['version'])

bigchaindb/core.py

@@ -330,41 +330,29 @@ class Bigchain(object):
         else:
             return None
 
-    def get_tx_by_metadata_id(self, metadata_id):
-        """Retrieves transactions related to a metadata.
+    def get_transactions_by_asset_id(self, asset_id):
+        """Retrieves valid or undecided transactions related to a particular
+        asset.
 
-        When creating a transaction one of the optional arguments is the `metadata`. The metadata is a generic
-        dict that contains extra information that can be appended to the transaction.
-
-        To make it easy to query the bigchain for that particular metadata we create a UUID for the metadata and
-        store it with the transaction.
+        A digital asset in bigchaindb is identified by an uuid. This allows us
+        to query all the transactions related to a particular digital asset,
+        knowing the id.
 
         Args:
-            metadata_id (str): the id for this particular metadata.
+            asset_id (str): the id for this particular asset.
 
         Returns:
-            A list of transactions containing that metadata. If no transaction exists with that metadata it
-            returns an empty list `[]`
+            A list of valid or undecided transactions related to the asset.
+            If no transaction exists for that asset it returns an empty list
+            `[]`
         """
-        cursor = self.backend.get_transactions_by_metadata_id(metadata_id)
-        return [Transaction.from_dict(tx) for tx in cursor]
-
-    def get_txs_by_asset_id(self, asset_id):
-        """Retrieves transactions related to a particular asset.
-
-        A digital asset in bigchaindb is identified by an uuid. This allows us to query all the transactions
-        related to a particular digital asset, knowing the id.
-
-        Args:
-            asset_id (str): the id for this particular metadata.
-
-        Returns:
-            A list of transactions containing related to the asset. If no transaction exists for that asset it
-            returns an empty list `[]`
-        """
-
-        cursor = self.backend.get_transactions_by_asset_id(asset_id)
-        return [Transaction.from_dict(tx) for tx in cursor]
+        txids = self.backend.get_txids_by_asset_id(asset_id)
+        transactions = []
+        for txid in txids:
+            tx = self.get_transaction(txid)
+            if tx:
+                transactions.append(tx)
+        return transactions
 
     def get_asset_by_id(self, asset_id):
         """Returns the asset associated with an asset_id.
@@ -379,7 +367,7 @@ class Bigchain(object):
         cursor = self.backend.get_asset_by_id(asset_id)
         cursor = list(cursor)
         if cursor:
-            return Asset.from_dict(cursor[0]['transaction']['asset'])
+            return Asset.from_dict(cursor[0]['asset'])
 
     def get_spent(self, txid, cid):
         """Check if a `txid` was already used as an input.
@@ -409,9 +397,10 @@ class Bigchain(object):
                 if self.get_transaction(transaction['id']):
                     num_valid_transactions += 1
                 if num_valid_transactions > 1:
-                    raise exceptions.DoubleSpend(
-                        '`{}` was spent more then once. There is a problem with the chain'.format(
-                            txid))
+                    raise exceptions.DoubleSpend(('`{}` was spent more than'
+                                                  ' once. There is a problem'
+                                                  ' with the chain')
+                                                 .format(txid))
 
             if num_valid_transactions:
                 return Transaction.from_dict(transactions[0])
@@ -447,7 +436,7 @@ class Bigchain(object):
             # use it after the execution of this function.
             # a transaction can contain multiple outputs (conditions) so we need to iterate over all of them
             # to get a list of outputs available to spend
-            for index, cond in enumerate(tx['transaction']['conditions']):
+            for index, cond in enumerate(tx['conditions']):
                 # for simple signature conditions there are no subfulfillments
                 # check if the owner is in the condition `owners_after`
                 if len(cond['owners_after']) == 1:
@@ -605,11 +594,11 @@ class Bigchain(object):
         }
 
         vote_data = serialize(vote)
-        signature = crypto.SigningKey(self.me_private).sign(vote_data.encode())
+        signature = crypto.PrivateKey(self.me_private).sign(vote_data.encode())
 
         vote_signed = {
             'node_pubkey': self.me,
-            'signature': signature,
+            'signature': signature.decode(),
             'vote': vote
         }
 

bigchaindb/db/backends/rethinkdb.py

@@ -138,48 +138,29 @@ class RethinkDBBackend:
                 .get_all(transaction_id, index='transaction_id')
                 .pluck('votes', 'id', {'block': ['voters']}))
 
-    def get_transactions_by_metadata_id(self, metadata_id):
-        """Retrieves transactions related to a metadata.
+    def get_txids_by_asset_id(self, asset_id):
+        """Retrieves transactions ids related to a particular asset.
 
-        When creating a transaction one of the optional arguments is the `metadata`. The metadata is a generic
-        dict that contains extra information that can be appended to the transaction.
-
-        To make it easy to query the bigchain for that particular metadata we create a UUID for the metadata and
-        store it with the transaction.
-
-        Args:
-            metadata_id (str): the id for this particular metadata.
-
-        Returns:
-            A list of transactions containing that metadata. If no transaction exists with that metadata it
-            returns an empty list `[]`
-        """
-        return self.connection.run(
-                r.table('bigchain', read_mode=self.read_mode)
-                .get_all(metadata_id, index='metadata_id')
-                .concat_map(lambda block: block['block']['transactions'])
-                .filter(lambda transaction: transaction['transaction']['metadata']['id'] == metadata_id))
-
-    def get_transactions_by_asset_id(self, asset_id):
-        """Retrieves transactions related to a particular asset.
-
-        A digital asset in bigchaindb is identified by an uuid. This allows us to query all the transactions
-        related to a particular digital asset, knowing the id.
+        A digital asset in bigchaindb is identified by an uuid. This allows us
+        to query all the transactions related to a particular digital asset,
+        knowing the id.
 
         Args:
             asset_id (str): the id for this particular metadata.
 
         Returns:
-            A list of transactions containing related to the asset. If no transaction exists for that asset it
-            returns an empty list `[]`
+            A list of transactions ids related to the asset. If no transaction
+            exists for that asset it returns an empty list `[]`
         """
 
+        # here we only want to return the transaction ids since later on when
+        # we are going to retrieve the transaction with status validation
         return self.connection.run(
             r.table('bigchain', read_mode=self.read_mode)
              .get_all(asset_id, index='asset_id')
              .concat_map(lambda block: block['block']['transactions'])
-             .filter(lambda transaction:
-                     transaction['transaction']['asset']['id'] == asset_id))
+             .filter(lambda transaction: transaction['asset']['id'] == asset_id)
+             .get_field('id'))
 
     def get_asset_by_id(self, asset_id):
         """Returns the asset associated with an asset_id.
@@ -195,10 +176,10 @@ class RethinkDBBackend:
              .get_all(asset_id, index='asset_id')
              .concat_map(lambda block: block['block']['transactions'])
              .filter(lambda transaction:
-                     transaction['transaction']['asset']['id'] == asset_id)
+                     transaction['asset']['id'] == asset_id)
              .filter(lambda transaction:
-                     transaction['transaction']['operation'] == 'CREATE')
-             .pluck({'transaction': 'asset'}))
+                     transaction['operation'] == 'CREATE')
+             .pluck('asset'))
 
     def get_spent(self, transaction_id, condition_id):
         """Check if a `txid` was already used as an input.
@@ -218,7 +199,7 @@ class RethinkDBBackend:
         return self.connection.run(
                 r.table('bigchain', read_mode=self.read_mode)
                 .concat_map(lambda doc: doc['block']['transactions'])
-                .filter(lambda transaction: transaction['transaction']['fulfillments'].contains(
+                .filter(lambda transaction: transaction['fulfillments'].contains(
                     lambda fulfillment: fulfillment['input'] == {'txid': transaction_id, 'cid': condition_id})))
 
     def get_owned_ids(self, owner):
@@ -235,7 +216,7 @@ class RethinkDBBackend:
         return self.connection.run(
                 r.table('bigchain', read_mode=self.read_mode)
                 .concat_map(lambda doc: doc['block']['transactions'])
-                .filter(lambda tx: tx['transaction']['conditions'].contains(
+                .filter(lambda tx: tx['conditions'].contains(
                     lambda c: c['owners_after'].contains(owner))))
 
     def get_votes_by_block_id(self, block_id):

bigchaindb/db/utils.py

@@ -116,15 +116,10 @@ def create_bigchain_secondary_index(conn, dbname):
         .index_create('transaction_id',
                       r.row['block']['transactions']['id'], multi=True)\
         .run(conn)
-    # secondary index for payload data by UUID
-    r.db(dbname).table('bigchain')\
-        .index_create('metadata_id',
-                      r.row['block']['transactions']['transaction']['metadata']['id'], multi=True)\
-        .run(conn)
     # secondary index for asset uuid
     r.db(dbname).table('bigchain')\
                 .index_create('asset_id',
-                              r.row['block']['transactions']['transaction']['asset']['id'], multi=True)\
+                              r.row['block']['transactions']['asset']['id'], multi=True)\
                 .run(conn)
 
     # wait for rethinkdb to finish creating secondary indexes
@@ -133,15 +128,10 @@ def create_bigchain_secondary_index(conn, dbname):
 
 def create_backlog_secondary_index(conn, dbname):
     logger.info('Create `backlog` secondary index.')
-    # to order transactions by timestamp
-    r.db(dbname).table('backlog')\
-        .index_create('transaction_timestamp',
-                      r.row['transaction']['timestamp'])\
-        .run(conn)
     # compound index to read transactions from the backlog per assignee
     r.db(dbname).table('backlog')\
         .index_create('assignee__transaction_timestamp',
-                      [r.row['assignee'], r.row['transaction']['timestamp']])\
+                      [r.row['assignee'], r.row['assignment_timestamp']])\
         .run(conn)
 
     # wait for rethinkdb to finish creating secondary indexes

bigchaindb/models.py

@@ -1,4 +1,4 @@
-from bigchaindb.common.crypto import hash_data, VerifyingKey, SigningKey
+from bigchaindb.common.crypto import hash_data, PublicKey, PrivateKey
 from bigchaindb.common.exceptions import (InvalidHash, InvalidSignature,
                                           OperationError, DoubleSpend,
                                           TransactionDoesNotExist,
@@ -208,11 +208,11 @@ class Block(object):
 
         return self
 
-    def sign(self, signing_key):
+    def sign(self, private_key):
         """Create a signature for the Block and overwrite `self.signature`.
 
         Args:
-            signing_key (str): A signing key corresponding to
+            private_key (str): A private key corresponding to
                 `self.node_pubkey`.
 
         Returns:
@@ -220,8 +220,8 @@ class Block(object):
         """
         block_body = self.to_dict()
         block_serialized = serialize(block_body['block'])
-        signing_key = SigningKey(signing_key)
-        self.signature = signing_key.sign(block_serialized.encode()).decode()
+        private_key = PrivateKey(private_key)
+        self.signature = private_key.sign(block_serialized.encode()).decode()
         return self
 
     def is_signature_valid(self):
@@ -233,11 +233,11 @@ class Block(object):
         block = self.to_dict()['block']
         # cc only accepts bytestring messages
         block_serialized = serialize(block).encode()
-        verifying_key = VerifyingKey(block['node_pubkey'])
+        public_key = PublicKey(block['node_pubkey'])
         try:
             # NOTE: CC throws a `ValueError` on some wrong signatures
             #       https://github.com/bigchaindb/cryptoconditions/issues/27
-            return verifying_key.verify(block_serialized, self.signature)
+            return public_key.verify(block_serialized, self.signature)
         except (ValueError, AttributeError):
             return False
 
@@ -261,7 +261,7 @@ class Block(object):
         block = block_body['block']
         block_serialized = serialize(block)
         block_id = hash_data(block_serialized)
-        verifying_key = VerifyingKey(block['node_pubkey'])
+        public_key = PublicKey(block['node_pubkey'])
 
         try:
             signature = block_body['signature']
@@ -275,8 +275,8 @@ class Block(object):
             # NOTE: CC throws a `ValueError` on some wrong signatures
             #       https://github.com/bigchaindb/cryptoconditions/issues/27
             try:
-                signature_valid = verifying_key\
-                    .verify(block_serialized.encode(), signature)
+                signature_valid = public_key\
+                        .verify(block_serialized.encode(), signature)
             except ValueError:
                 signature_valid = False
             if signature_valid is False:

bigchaindb/pipelines/block.py

@@ -116,7 +116,7 @@ class BlockPipeline:
         Returns:
             :class:`~bigchaindb.models.Block`: The Block.
         """
-        logger.info('Write new block %s with %s transactions', block.id, block.transactions)
+        logger.info('Write new block %s with %s transactions', block.id, len(block.transactions))
         self.bigchain.write_block(block)
         return block
 

bigchaindb/pipelines/election.py

@@ -6,7 +6,6 @@ is specified in ``create_pipeline``.
 """
 import logging
 
-import rethinkdb as r
 from multipipes import Pipeline, Node
 
 from bigchaindb.pipelines.utils import ChangeFeed
@@ -31,9 +30,8 @@ class Election:
             next_vote: The next vote.
 
         """
-        next_block = self.bigchain.connection.run(
-                r.table('bigchain')
-                .get(next_vote['vote']['voting_for_block']))
+        next_block = self.bigchain.get_block(
+            next_vote['vote']['voting_for_block'])
 
         block_status = self.bigchain.block_election_status(next_block['id'],
                                                            next_block['block']['voters'])

bigchaindb/pipelines/stale.py

@@ -67,7 +67,7 @@ def create_pipeline(timeout=5, backlog_reassign_delay=5):
     return monitor_pipeline
 
 
-def start(timeout=5, backlog_reassign_delay=5):
+def start(timeout=5, backlog_reassign_delay=None):
     """Create, start, and return the block pipeline."""
     pipeline = create_pipeline(timeout=timeout,
                                backlog_reassign_delay=backlog_reassign_delay)

bigchaindb/util.py

@@ -130,13 +130,13 @@ def verify_vote_signature(voters, signed_vote):
     """
 
     signature = signed_vote['signature']
-    vk_base58 = signed_vote['node_pubkey']
+    pk_base58 = signed_vote['node_pubkey']
 
     # immediately return False if the voter is not in the block voter list
-    if vk_base58 not in voters:
+    if pk_base58 not in voters:
         return False
 
-    public_key = crypto.VerifyingKey(vk_base58)
+    public_key = crypto.PublicKey(pk_base58)
     return public_key.verify(serialize(signed_vote['vote']).encode(), signature)
 
 
@@ -156,4 +156,4 @@ def is_genesis_block(block):
     try:
         return block.transactions[0].operation == 'GENESIS'
     except AttributeError:
-        return block['block']['transactions'][0]['transaction']['operation'] == 'GENESIS'
+        return block['block']['transactions'][0]['operation'] == 'GENESIS'

bigchaindb/version.py

@@ -1,2 +1,2 @@
-__version__ = '0.8.0.dev'
-__short_version__ = '0.8.dev'
+__version__ = '0.9.0.dev'
+__short_version__ = '0.9.dev'

bigchaindb/web/server.py

@@ -49,19 +49,22 @@ class StandaloneApplication(gunicorn.app.base.BaseApplication):
         return self.application
 
 
-def create_app(settings):
+def create_app(*, debug=False, threads=4):
     """Return an instance of the Flask application.
 
     Args:
         debug (bool): a flag to activate the debug mode for the app
             (default: False).
+        threads (int): number of threads to use
+    Return:
+        an instance of the Flask application.
     """
 
     app = Flask(__name__)
 
-    app.debug = settings.get('debug', False)
+    app.debug = debug
 
-    app.config['bigchain_pool'] = util.pool(Bigchain, size=settings.get('threads', 4))
+    app.config['bigchain_pool'] = util.pool(Bigchain, size=threads)
     app.config['monitor'] = Monitor()
 
     app.register_blueprint(info_views, url_prefix='/')
@@ -88,6 +91,7 @@ def create_server(settings):
     if not settings.get('threads'):
         settings['threads'] = (multiprocessing.cpu_count() * 2) + 1
 
-    app = create_app(settings)
+    app = create_app(debug=settings.get('debug', False),
+                     threads=settings['threads'])
     standalone = StandaloneApplication(app, settings)
     return standalone

bigchaindb/web/views/transactions.py

@@ -6,7 +6,7 @@ For more information please refer to the documentation on ReadTheDocs:
 from flask import current_app, request, Blueprint
 from flask_restful import Resource, Api
 
-from bigchaindb.common.exceptions import InvalidHash, InvalidSignature
+from bigchaindb.common.exceptions import ValidationError, InvalidSignature
 
 import bigchaindb
 from bigchaindb.models import Transaction
@@ -98,7 +98,7 @@ class TransactionListApi(Resource):
 
         try:
             tx_obj = Transaction.from_dict(tx)
-        except (InvalidHash, InvalidSignature):
+        except (ValidationError, InvalidSignature):
             return make_error(400, 'Invalid transaction')
 
         with pool() as bigchain:

deploy-cluster-aws/awsdeploy.sh

@@ -44,7 +44,8 @@ echo "IMAGE_ID = "$IMAGE_ID
 echo "INSTANCE_TYPE = "$INSTANCE_TYPE
 echo "SECURITY_GROUP = "$SECURITY_GROUP
 echo "USING_EBS = "$USING_EBS
-if [ "$USING_EBS" = True ]; then
+# Treat booleans as strings which must be either "True" or "False"
+if [ "$USING_EBS" == "True" ]; then
     echo "EBS_VOLUME_SIZE = "$EBS_VOLUME_SIZE
     echo "EBS_OPTIMIZED = "$EBS_OPTIMIZED
 fi
@@ -117,7 +118,11 @@ fab upgrade_setuptools
 
 if [ "$WHAT_TO_DEPLOY" == "servers" ]; then
     # (Re)create the RethinkDB configuration file conf/rethinkdb.conf
-    python create_rethinkdb_conf.py --bind-http-to-localhost $BIND_HTTP_TO_LOCALHOST
+    if [ "$BIND_HTTP_TO_LOCALHOST" == "True" ]; then
+        python create_rethinkdb_conf.py --bind-http-to-localhost
+    else
+        python create_rethinkdb_conf.py
+    fi
     # Rollout RethinkDB and start it
     fab prep_rethinkdb_storage:$USING_EBS
     fab install_rethinkdb

deploy-cluster-aws/create_rethinkdb_conf.py

@@ -13,14 +13,14 @@ from hostlist import public_dns_names
 
 # Parse the command-line arguments
 parser = argparse.ArgumentParser()
-parser.add_argument("--bind-http-to-localhost",
-                    help="should RethinkDB web interface be bound to localhost?",
-                    required=True)
+# The next line isn't strictly necessary, but it clarifies the default case:
+parser.set_defaults(bind_http_to_localhost=False)
+parser.add_argument('--bind-http-to-localhost',
+                    action='store_true',
+                    help='should RethinkDB web interface be bound to localhost?')
 args = parser.parse_args()
 bind_http_to_localhost = args.bind_http_to_localhost
 
-print('bind_http_to_localhost = {}'.format(bind_http_to_localhost))
-
 # cwd = current working directory
 old_cwd = os.getcwd()
 os.chdir('conf')

deploy-cluster-aws/example_deploy_conf.py

@@ -43,9 +43,9 @@ USE_KEYPAIRS_FILE=False
 # Canonical (the company behind Ubuntu) generates many AMIs
 # and you can search for one that meets your needs at:
 # https://cloud-images.ubuntu.com/locator/ec2/
-# Example:
-# (eu-central-1 Ubuntu 14.04 LTS amd64 hvm:ebs-ssd 20161020)
-IMAGE_ID="ami-9c09f0f3"
+# Example: ami-8504fdea is what you get if you search for:
+# eu-central-1 16.04 LTS amd64 hvm:ebs-ssd
+IMAGE_ID="ami-8504fdea"
 
 # INSTANCE_TYPE is the type of AWS instance to launch
 # i.e. How many CPUs do you want? How much storage? etc.

deploy-cluster-aws/fabfile.py

@@ -156,7 +156,12 @@ def prep_rethinkdb_storage(USING_EBS):
 @parallel
 def install_rethinkdb():
     """Install RethinkDB"""
-    sudo("echo 'deb http://download.rethinkdb.com/apt trusty main' | sudo tee /etc/apt/sources.list.d/rethinkdb.list")
+    # Old way:
+    # sudo("echo 'deb http://download.rethinkdb.com/apt trusty main' | sudo tee /etc/apt/sources.list.d/rethinkdb.list")
+    # New way: (from https://www.rethinkdb.com/docs/install/ubuntu/ )
+    sudo('source /etc/lsb-release && '
+         'echo "deb http://download.rethinkdb.com/apt $DISTRIB_CODENAME main" | '
+         'sudo tee /etc/apt/sources.list.d/rethinkdb.list')
     sudo("wget -qO- http://download.rethinkdb.com/apt/pubkey.gpg | sudo apt-key add -")
     sudo("apt-get update")
     sudo("apt-get -y install rethinkdb")

deploy-cluster-aws/write_keypairs_file.py

@@ -10,8 +10,8 @@ Using the list in other Python scripts:
     # in a Python 2 script:
     from keypairs import keypairs_list
     # keypairs_list is a list of (sk, pk) tuples
-    # sk = signing key (private key)
-    # pk = verifying key (public key)
+    # sk = private key
+    # pk = public key
 """
 
 import argparse

docs/root/requirements.txt

@@ -1,4 +1,4 @@
-Sphinx>=1.3.5
+Sphinx>=1.4.8
 recommonmark>=0.4.0
 sphinx-rtd-theme>=0.1.9
 sphinxcontrib-napoleon>=0.4.4

docs/root/source/assets.rst

@@ -3,8 +3,8 @@ How BigchainDB is Good for Asset Registrations & Transfers
 
 BigchainDB can store data of any kind (within reason), but it's designed to be particularly good for storing asset registrations and transfers:
 
-* The fundamental thing that one submits to a BigchainDB federation to be checked and stored (if valid) is a *transaction*, and there are two kinds: creation transactions and transfer transactions.
-* A creation transaction can be use to register any kind of indivisible asset, along with arbitrary metadata.
+* The fundamental thing that one submits to a BigchainDB federation to be checked and stored (if valid) is a *transaction*, and there are two kinds: CREATE transactions and TRANSFER transactions.
+* A CREATE transaction can be use to register any kind of asset (divisible or indivisible), along with arbitrary metadata.
 * An asset can have zero, one, or several owners.
 * The owners of an asset can specify (crypto-)conditions which must be satisified by anyone wishing transfer the asset to new owners. For example, a condition might be that at least 3 of the 5 current owners must cryptographically sign a transfer transaction.
 * BigchainDB verifies that the conditions have been satisified as part of checking the validity of transfer transactions. (Moreover, anyone can check that they were satisfied.)

docs/root/source/data-models/transaction-model.md

@@ -1,45 +0,0 @@
-# The Transaction Model
-
-A transaction has the following structure:
-
-```json
-{
-    "id": "<hash of transaction, excluding signatures (see explanation)>",
-    "version": "<version number of the transaction model>",
-    "transaction": {
-        "fulfillments": ["<list of fulfillments>"],
-        "conditions": ["<list of conditions>"],
-        "operation": "<string>",
-        "timestamp": "<timestamp from client>",
-        "asset": "<digital asset description (explained in the next section)>",
-        "metadata": {
-            "id": "<uuid>",
-            "data": "<any JSON document>"
-        }
-    }
-}
-```
-
-Here's some explanation of the contents of a transaction:
-
-- `id`: The hash of everything inside the serialized `transaction` body (i.e. `fulfillments`, `conditions`, `operation`, `timestamp` and `data`; see below), with one wrinkle: for each fulfillment in `fulfillments`, `fulfillment` is set to `null`. The `id` is also the database primary key.
-- `version`: Version number of the transaction model, so that software can support different transaction models.
-- `transaction`:
-    - `fulfillments`: List of fulfillments. Each _fulfillment_ contains a pointer to an unspent asset
-    and a _crypto fulfillment_ that satisfies a spending condition set on the unspent asset. A _fulfillment_
-    is usually a signature proving the ownership of the asset.
-    See the page about [Crypto-Conditions and Fulfillments](crypto-conditions.html).
-    - `conditions`: List of conditions. Each _condition_ is a _crypto-condition_ that needs to be fulfilled by a transfer transaction in order to transfer ownership to new owners.
-    See the page about [Crypto-Conditions and Fulfillments](crypto-conditions.html).
-    - `operation`: String representation of the operation being performed (currently either "CREATE", "TRANSFER" or "GENESIS"). It determines how the transaction should be validated.
-    - `timestamp`: The Unix time when the transaction was created. It's provided by the client. See the page about [timestamps in BigchainDB](../timestamps.html).
-	- `asset`: Definition of the digital asset. See next section.
-    - `metadata`:
-        - `id`: UUID version 4 (random) converted to a string of hex digits in standard form.
-        - `data`: Can be any JSON document. It may be empty in the case of a transfer transaction.
-
-Later, when we get to the models for the block and the vote, we'll see that both include a signature (from the node which created it). You may wonder why transactions don't have signatures... The answer is that they do! They're just hidden inside the `fulfillment` string of each fulfillment. A creation transaction is signed by the node that created it. A transfer transaction is signed by whoever currently controls or owns it.
-
-What gets signed? For each fulfillment in the transaction, the "fullfillment message" that gets signed includes the `operation`, `timestamp`, `data`, `version`, `id`, corresponding `condition`, and the fulfillment itself, except with its fulfillment string set to `null`. The computed signature goes into creating the `fulfillment` string of the fulfillment.
-
-One other note: Currently, transactions contain only the public keys of asset-owners (i.e. who own an asset or who owned an asset in the past), inside the conditions and fulfillments. A transaction does _not_ contain the public key of the client (computer) which generated and sent it to a BigchainDB node. In fact, there's no need for a client to _have_ a public/private keypair. In the future, each client may also have a keypair, and it may have to sign each sent transaction (using its private key); see [Issue #347 on GitHub](https://github.com/bigchaindb/bigchaindb/issues/347). In practice, a person might think of their keypair as being both their "ownership-keypair" and their "client-keypair," but there is a difference, just like there's a difference between Joe and Joe's computer.

docs/root/source/index.rst

@@ -58,6 +58,9 @@ At a high level, one can communicate with a BigchainDB cluster (set of nodes) us
    <div class="buttondiv">
      <a class="button" href="http://docs.bigchaindb.com/projects/py-driver/en/latest/index.html">Python Driver Docs</a>
    </div>
+   <div class="buttondiv">
+     <a class="button" href="https://docs.bigchaindb.com/projects/cli/en/latest/">Command Line Transaction Tool</a>
+   </div>
    <div class="buttondiv">
      <a class="button" href="http://docs.bigchaindb.com/projects/server/en/latest/index.html">Server Docs</a>
    </div>
@@ -83,4 +86,4 @@ More About BigchainDB
    smart-contracts
    transaction-concepts
    timestamps
-   data-models/index
+   Data Models <https://docs.bigchaindb.com/projects/server/en/latest/data-models/index.html>

docs/root/source/timestamps.md

@@ -1,15 +1,13 @@
 # Timestamps in BigchainDB
 
-Each transaction, block and vote has an associated timestamp. Interpreting those timestamps is tricky, hence the need for this section.
+Each block and vote has an associated timestamp. Interpreting those timestamps is tricky, hence the need for this section.
 
 
 ## Timestamp Sources & Accuracy
 
-A transaction's timestamp is provided by the client which created and submitted the transaction to a BigchainDB node. A block's timestamp is provided by the BigchainDB node which created the block. A vote's timestamp is provided by the BigchainDB node which created the vote.
+Timestamps in BigchainDB are provided by the node which created the block and the node that created the vote.
 
-When a BigchainDB client or node needs a timestamp, it calls a BigchainDB utility function named `timestamp()`. There's a detailed explanation of how that function works below, but the short version is that it gets the [Unix time](https://en.wikipedia.org/wiki/Unix_time) from its system clock, rounded to the nearest second.
-
-We can't say anything about the accuracy of the system clock on clients. Timestamps from clients are still potentially useful, however, in a statistical sense. We say more about that below.
+When a BigchainDB node needs a timestamp, it calls a BigchainDB utility function named `timestamp()`. There's a detailed explanation of how that function works below, but the short version is that it gets the [Unix time](https://en.wikipedia.org/wiki/Unix_time) from its system clock, rounded to the nearest second.
 
 We advise BigchainDB nodes to run special software (an "NTP daemon") to keep their system clock in sync with standard time servers. (NTP stands for [Network Time Protocol](https://en.wikipedia.org/wiki/Network_Time_Protocol).)
 
@@ -36,35 +34,26 @@ In all likelihood, you will never have to worry about leap seconds because they
 There's another gotcha with (Unix time) timestamps: you can't calculate the real-world elapsed time between two timestamps (correctly) by subtracting the smaller timestamp from the larger one. The result won't include any of the leap seconds that occured between the two timestamps. You could look up how many leap seconds happened between the two timestamps and add that to the result. There are many library functions for working with timestamps; those are beyond the scope of this documentation.
 
 
-## Avoid Doing Transactions Around Leap Seconds
-
-Because of the ambiguity and confusion that arises with Unix time around leap seconds, we advise users to avoid creating transactions around leap seconds.
-
-
 ## Interpreting Sets of Timestamps
 
 You can look at many timestamps to get a statistical sense of when something happened. For example, a transaction in a decided-valid block has many associated timestamps:
 
-* its own timestamp
-* the timestamps of the other transactions in the block; there could be as many as 999 of those
 * the timestamp of the block
 * the timestamps of all the votes on the block
 
-Those timestamps come from many sources, so you can look at all of them to get some statistical sense of when the transaction "actually happened." The timestamp of the block should always be after the timestamp of the transaction, and the timestamp of the votes should always be after the timestamp of the block.
-
 
 ## How BigchainDB Uses Timestamps
 
 BigchainDB _doesn't_ use timestamps to determine the order of transactions or blocks. In particular, the order of blocks is determined by RethinkDB's changefeed on the bigchain table.
 
-BigchainDB does use timestamps for some things. It uses them to determine if a transaction has been waiting in the backlog for too long (i.e. because the node assigned to it hasn't handled it yet). It also uses timestamps to determine the status of timeout conditions (used by escrow).
+BigchainDB does use timestamps for some things. When a Transaction is written to the backlog, a timestamp is assigned called the `assignment_timestamp`, to determine if it has been waiting in the backlog for too long (i.e. because the node assigned to it hasn't handled it yet).
 
 
 ## Including Trusted Timestamps
 
 If you want to create a transaction payload with a trusted timestamp, you can.
 
-One way to do that would be to send a payload to a trusted timestamping service. They will send back a timestamp, a signature, and their public key. They should also explain how you can verify the signature. You can then include the original payload, the timestamp, the signature, and the service's public key in your transaction. That way, anyone with the verification instructions can verify that the original payload was signed by the trusted timestamping service.
+One way to do that would be to send a payload to a trusted timestamping service. They will send back a timestamp, a signature, and their public key. They should also explain how you can verify the signature. You can then include the original payload, the timestamp, the signature, and the service's public key in your transaction metadata. That way, anyone with the verification instructions can verify that the original payload was signed by the trusted timestamping service.
 
 
 ## How the timestamp() Function Works

docs/root/source/transaction-concepts.md

@@ -1,29 +1,83 @@
 # Transaction Concepts
 
-In BigchainDB, _Transactions_ are used to register, issue, create or transfer things (e.g. assets).
+In BigchainDB, _Transactions_ are used to register, issue, create or transfer
+things (e.g. assets).
 
-Transactions are the most basic kind of record stored by BigchainDB. There are two kinds: creation transactions and transfer transactions.
+Transactions are the most basic kind of record stored by BigchainDB. There are
+two kinds: CREATE transactions and TRANSFER transactions.
 
-A _creation transaction_ can be used to register, issue, create or otherwise initiate the history of a single thing (or asset) in BigchainDB. For example, one might register an identity or a creative work. The things are often called "assets" but they might not be literal assets.
+## CREATE Transactions
 
-Currently, BigchainDB only supports indivisible assets. You can't split an asset apart into multiple assets, nor can you combine several assets together into one. [Issue #129](https://github.com/bigchaindb/bigchaindb/issues/129) is an enhancement proposal to support divisible assets.
+A CREATE transaction can be used to register, issue, create or otherwise
+initiate the history of a single thing (or asset) in BigchainDB. For example,
+one might register an identity or a creative work. The things are often called
+"assets" but they might not be literal assets.
 
-A creation transaction also establishes the conditions that must be met to transfer the asset. For example, there may be a condition that any transfer must be signed (cryptographically) by the signing/private key associated with a given verifying/public key. More sophisticated conditions are possible. BigchainDB's conditions are based on the crypto-conditions of the [Interledger Protocol (ILP)](https://interledger.org/).
+BigchainDB supports divisible assets as of BigchainDB Server v0.8.0.
+That means you can create/register an asset with an initial quantity,
+e.g. 700 oak trees. Divisible assets can be split apart or recombined
+by transfer transactions (described more below).
 
-A _transfer transaction_ can transfer an asset by fulfilling the current conditions on the asset. It can also specify new transfer conditions.
+A CREATE transaction also establishes the conditions that must be met to
+transfer the asset(s). For example, there may be a condition that any transfer
+must be signed (cryptographically) by the private key associated with a
+given public key. More sophisticated conditions are possible.
+BigchainDB's conditions are based on the crypto-conditions of the [Interledger
+Protocol (ILP)](https://interledger.org/).
 
-Today, every transaction contains one fulfillment-condition pair. The fulfillment in a transfer transaction must fulfill a condition in a previous transaction.
+## TRANSFER Transactions
 
-When a node is asked to check if a transaction is valid, it checks several things. Some things it checks are:
+A TRANSFER transaction can transfer an asset
+by fulfilling the current conditions on the asset.
+It must also specify new transfer conditions.
 
-* Are all the fulfillments valid? (Do they correctly satisfy the conditions they claim to satisfy?)
+**Example 1:** Suppose a red car is owned and controlled by Joe.
+Suppose the current transfer condition on the car says
+that any valid transfer must be signed by Joe.
+Joe and a buyer named Rae could build a TRANSFER transaction containing
+Joe's signature (to fulfill the current transfer condition)
+plus a new transfer condition saying that any valid transfer
+must be signed by Rae.
+
+**Example 2:** Someone might construct a TRANSFER transaction
+that fulfills the transfer conditions on four
+previously-untransferred assets of the same asset type
+e.g. paperclips. The amounts might be 20, 10, 45 and 25, say,
+for a total of 100 paperclips.
+The TRANSFER transaction would also set up new transfer conditions.
+For example, maybe a set of 60 paperclips can only be transferred
+if Gertrude signs, and a separate set of 40 paperclips can only be
+transferred if both Jack and Kelly sign.
+Note how the sum of the incoming paperclips must equal the sum
+of the outgoing paperclips (100).
+
+## Transaction Validity
+
+When a node is asked to check if a transaction is valid, it checks several
+things. Some things it checks are:
+
+* Are all the fulfillments valid? (Do they correctly satisfy the conditions
+  they claim to satisfy?)
 * If it's a creation transaction, is the asset valid?
 * If it's a transfer transaction:
    * Is it trying to fulfill a condition in a nonexistent transaction?
-   * Is it trying to fulfill a condition that's not in a valid transaction? (It's okay if the condition is in a transaction in an invalid block; those transactions are ignored. Transactions in the backlog or undecided blocks are not ignored.)
-   * Is it trying to fulfill a condition that has already been fulfilled, or that some other pending transaction (in the backlog or an undecided block) also aims to fulfill?
-   * Is the asset ID in the transaction the same as the asset ID in all transactions whose conditions are being fulfilled?
+   * Is it trying to fulfill a condition that's not in a valid transaction?
+     (It's okay if the condition is in a transaction in an invalid block; those
+     transactions are ignored. Transactions in the backlog or undecided blocks
+     are not ignored.)
+   * Is it trying to fulfill a condition that has already been fulfilled, or
+     that some other pending transaction (in the backlog or an undecided block)
+     also aims to fulfill?
+   * Is the asset ID in the transaction the same as the asset ID in all
+     transactions whose conditions are being fulfilled?
+   * Is the sum of the amounts in the fulfillments equal
+     to the sum of the amounts in the new conditions?
 
-If you're curious about the details of transaction validation, the code is in the `validate` method of the `Transaction` class, in `bigchaindb/models.py` (at the time of writing).
+If you're curious about the details of transaction validation, the code is in
+the `validate` method of the `Transaction` class, in `bigchaindb/models.py` (at
+the time of writing).
 
-Note: The check to see if the transaction ID is equal to the hash of the transaction body is actually done whenever the transaction is converted from a Python dict to a Transaction object, which must be done before the `validate` method can be called (since it's called on a Transaction object).
+Note: The check to see if the transaction ID is equal to the hash of the
+transaction body is actually done whenever the transaction is converted from a
+Python dict to a Transaction object, which must be done before the `validate`
+method can be called (since it's called on a Transaction object).

docs/server/generate_http_server_api_documentation.py

@@ -0,0 +1,87 @@
+""" Script to build http examples for http server api docs """
+
+import json
+import os
+import os.path
+
+from bigchaindb.common.transaction import Asset, Transaction
+
+
+TPLS = {}
+
+TPLS['post-tx-request'] = """\
+POST /transactions/ HTTP/1.1
+Host: example.com
+Content-Type: application/json
+
+%(tx)s
+"""
+
+
+TPLS['post-tx-response'] = """\
+HTTP/1.1 201 Created
+Content-Type: application/json
+
+%(tx)s
+"""
+
+
+TPLS['get-tx-status-request'] = """\
+GET /transactions/%(txid)s/status HTTP/1.1
+Host: example.com
+
+"""
+
+
+TPLS['get-tx-status-response'] = """\
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+{
+  "status": "valid"
+}
+"""
+
+
+TPLS['get-tx-request'] = """\
+GET /transactions/%(txid)s HTTP/1.1
+Host: example.com
+
+"""
+
+
+TPLS['get-tx-response'] = """\
+HTTP/1.1 200 OK
+Content-Type: application/json
+
+%(tx)s
+"""
+
+
+def main():
+    """ Main function """
+    pubkey = '9RaWxppkP9UyYWA7NJb5FcgkzfJNPfvPX3FCNw2T5Pwb'
+    asset = Asset(None, 'e6969f87-4fc9-4467-b62a-f0dfa1c85002')
+    tx = Transaction.create([pubkey], [([pubkey], 1)], asset=asset)
+    tx_json = json.dumps(tx.to_dict(), indent=2, sort_keys=True)
+
+    base_path = os.path.join(os.path.dirname(__file__),
+                             'source/drivers-clients/samples')
+
+    if not os.path.exists(base_path):
+        os.makedirs(base_path)
+
+    for name, tpl in TPLS.items():
+        path = os.path.join(base_path, name + '.http')
+        code = tpl % {'tx': tx_json, 'txid': tx.id}
+        with open(path, 'w') as handle:
+            handle.write(code)
+
+
+def setup(*_):
+    """ Fool sphinx into think it's an extension muahaha """
+    main()
+
+
+if __name__ == '__main__':
+    main()

docs/server/generate_schema_documentation.py

@@ -0,0 +1,178 @@
+""" Script to render transaction schema into .rst document """
+
+from collections import OrderedDict
+import os.path
+
+import yaml
+
+from bigchaindb.common.schema import TX_SCHEMA_YAML
+
+
+TPL_PROP = """\
+%(title)s
+%(underline)s
+
+**type:** %(type)s
+
+%(description)s
+"""
+
+
+TPL_DOC = """\
+..  This file was auto generated by %(file)s
+
+==================
+Transaction Schema
+==================
+
+* `Transaction`_
+
+* Condition_
+
+* Fulfillment_
+
+* Asset_
+
+* Metadata_
+
+.. raw:: html
+
+    <style>
+    #transaction-schema h2 {
+         border-top: solid 3px #6ab0de;
+         background-color: #e7f2fa;
+         padding: 5px;
+    }
+    #transaction-schema h3 {
+         background: #f0f0f0;
+         border-left: solid 3px #ccc;
+         font-weight: bold;
+         padding: 6px;
+         font-size: 100%%;
+         font-family: monospace;
+    }
+    </style>
+
+Transaction
+-----------
+
+%(transaction)s
+
+Condition
+----------
+
+%(condition)s
+
+Fulfillment
+-----------
+
+%(fulfillment)s
+
+Asset
+-----
+
+%(asset)s
+
+Metadata
+--------
+
+%(metadata)s
+"""
+
+
+def ordered_load_yaml(stream):
+    """ Custom YAML loader to preserve key order """
+    class OrderedLoader(yaml.SafeLoader):
+        pass
+
+    def construct_mapping(loader, node):
+        loader.flatten_mapping(node)
+        return OrderedDict(loader.construct_pairs(node))
+    OrderedLoader.add_constructor(
+        yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG,
+        construct_mapping)
+    return yaml.load(stream, OrderedLoader)
+
+
+TX_SCHEMA = ordered_load_yaml(TX_SCHEMA_YAML)
+
+
+DEFINITION_BASE_PATH = '#/definitions/'
+
+
+def render_section(section_name, obj):
+    """ Render a domain object and it's properties """
+    out = [obj['description']]
+    for name, prop in obj.get('properties', {}).items():
+        try:
+            title = '%s.%s' % (section_name, name)
+            out += [TPL_PROP % {
+                'title': title,
+                'underline': '^' * len(title),
+                'description': property_description(prop),
+                'type': property_type(prop),
+            }]
+        except Exception as exc:
+            raise ValueError("Error rendering property: %s" % name, exc)
+    return '\n\n'.join(out + [''])
+
+
+def property_description(prop):
+    """ Get description of property """
+    if 'description' in prop:
+        return prop['description']
+    if '$ref' in prop:
+        return property_description(resolve_ref(prop['$ref']))
+    if 'anyOf' in prop:
+        return property_description(prop['anyOf'][0])
+    raise KeyError("description")
+
+
+def property_type(prop):
+    """ Resolve a string representing the type of a property """
+    if 'type' in prop:
+        if prop['type'] == 'array':
+            return 'array (%s)' % property_type(prop['items'])
+        return prop['type']
+    if 'anyOf' in prop:
+        return ' or '.join(property_type(p) for p in prop['anyOf'])
+    if '$ref' in prop:
+        return property_type(resolve_ref(prop['$ref']))
+    raise ValueError("Could not resolve property type")
+
+
+def resolve_ref(ref):
+    """ Resolve definition reference """
+    assert ref.startswith(DEFINITION_BASE_PATH)
+    return TX_SCHEMA['definitions'][ref[len(DEFINITION_BASE_PATH):]]
+
+
+def main():
+    """ Main function """
+    defs = TX_SCHEMA['definitions']
+    doc = TPL_DOC % {
+        'transaction': render_section('Transaction', TX_SCHEMA),
+        'condition': render_section('Condition', defs['condition']),
+        'fulfillment': render_section('Fulfillment', defs['fulfillment']),
+        'asset': render_section('Asset', defs['asset']),
+        'metadata': render_section('Metadata', defs['metadata']['anyOf'][0]),
+        'file': os.path.basename(__file__),
+    }
+
+    base_path = os.path.join(os.path.dirname(__file__), 'source/schema')
+    path = os.path.join(base_path, 'transaction.rst')
+
+    if not os.path.exists(base_path):
+        os.makedirs(base_path)
+
+    with open(path, 'w') as handle:
+        handle.write(doc)
+
+
+def setup(*_):
+    """ Fool sphinx into think it's an extension muahaha """
+    main()
+
+
+if __name__ == '__main__':
+    main()

docs/server/requirements.txt

@@ -1,4 +1,4 @@
-Sphinx>=1.3.5
+Sphinx>=1.4.8
 recommonmark>=0.4.0
 sphinx-rtd-theme>=0.1.9
 sphinxcontrib-napoleon>=0.4.4

docs/server/source/appendices/cryptography.md

@@ -1,12 +1,16 @@
 # Cryptography
 
-The section documents the cryptographic algorithms and Python implementations that we use.
+The section documents the cryptographic algorithms and Python implementations
+that we use.
 
-Before hashing or computing the signature of a JSON document, we serialize it as described in [the section on JSON serialization](json-serialization.html).
+Before hashing or computing the signature of a JSON document, we serialize it
+as described in [the section on JSON serialization](json-serialization.html).
 
 ## Hashes
 
-We compute hashes using the SHA3-256 algorithm and [pysha3](https://bitbucket.org/tiran/pykeccak) as the Python implementation. We store the hex-encoded hash in the database. For example:
+We compute hashes using the SHA3-256 algorithm and
+[pysha3](https://bitbucket.org/tiran/pykeccak) as the Python implementation. We
+store the hex-encoded hash in the database. For example:
 
 ```python
 import hashlib
@@ -19,8 +23,16 @@ tx_hash = hashlib.sha3_256(data).hexdigest()
 
 ## Signature Algorithm and Keys
 
-BigchainDB uses the [Ed25519](https://ed25519.cr.yp.to/) public-key signature system for generating its public/private key pairs (also called verifying/signing keys). Ed25519 is an instance of the [Edwards-curve Digital Signature Algorithm (EdDSA)](https://en.wikipedia.org/wiki/EdDSA). As of April 2016, EdDSA was in ["Internet-Draft" status with the IETF](https://tools.ietf.org/html/draft-irtf-cfrg-eddsa-05) but was [already widely used](https://ianix.com/pub/ed25519-deployment.html).
+BigchainDB uses the [Ed25519](https://ed25519.cr.yp.to/) public-key signature
+system for generating its public/private key pairs. Ed25519 is an instance of
+the [Edwards-curve Digital Signature Algorithm
+(EdDSA)](https://en.wikipedia.org/wiki/EdDSA). As of April 2016, EdDSA was in
+["Internet-Draft" status with the
+IETF](https://tools.ietf.org/html/draft-irtf-cfrg-eddsa-05) but was [already
+widely used](https://ianix.com/pub/ed25519-deployment.html).
 
-BigchainDB uses the the [ed25519](https://github.com/warner/python-ed25519) Python package, overloaded by the [cryptoconditions library](https://github.com/bigchaindb/cryptoconditions).
+BigchainDB uses the the [ed25519](https://github.com/warner/python-ed25519)
+Python package, overloaded by the [cryptoconditions
+library](https://github.com/bigchaindb/cryptoconditions).
 
-All keys are represented with the base58 encoding by default.
+All keys are represented with the base58 encoding by default.

docs/server/source/appendices/json-serialization.md

@@ -52,5 +52,5 @@ signature = sk.sign(tx_serialized)
 
 # verify signature
 tx_serialized = bytes(serialize(tx))
-vk.verify(signature, tx_serialized)
+pk.verify(signature, tx_serialized)
 ```

docs/server/source/clusters-feds/aws-testing-cluster.md

@@ -32,7 +32,7 @@ What did you just install?
 * [The aws-cli package](https://pypi.python.org/pypi/awscli), which is an AWS Command Line Interface (CLI).
 
 
-## Basic AWS Setup
+## Setting up in AWS
 
 See the page about [basic AWS Setup](../appendices/aws-setup.html) in the Appendices.
 
@@ -126,7 +126,7 @@ BRANCH="master"
 WHAT_TO_DEPLOY="servers"
 SSH_KEY_NAME="not-set-yet"
 USE_KEYPAIRS_FILE=False
-IMAGE_ID="ami-9c09f0f3"
+IMAGE_ID="ami-8504fdea"
 INSTANCE_TYPE="t2.medium"
 SECURITY_GROUP="bigchaindb"
 USING_EBS=True
@@ -137,6 +137,8 @@ BIND_HTTP_TO_LOCALHOST=True
 
 Make a copy of that file and call it whatever you like (e.g. `cp example_deploy_conf.py my_deploy_conf.py`). You can leave most of the settings at their default values, but you must change the value of `SSH_KEY_NAME` to the name of your private SSH key. You can do that with a text editor. Set `SSH_KEY_NAME` to the name you used for `<key-name>` when you generated an RSA key pair for SSH (in basic AWS setup).
 
+You'll also want to change the `IMAGE_ID` to one that's up-to-date and available in your AWS region. If you don't remember your AWS region, then look in your `$HOME/.aws/config` file. You can find an up-to-date Ubuntu image ID for your region at [https://cloud-images.ubuntu.com/locator/ec2/](https://cloud-images.ubuntu.com/locator/ec2/). An example search string is "eu-central-1 16.04 LTS amd64 hvm:ebs-ssd". You should replace "eu-central-1" with your region name.
+
 If you want your nodes to have a predictable set of pre-generated keypairs, then you should 1) set `USE_KEYPAIRS_FILE=True` in the AWS deployment configuration file, and 2) provide a `keypairs.py` file containing enough keypairs for all of your nodes. You can generate a `keypairs.py` file using the `write_keypairs_file.py` script. For example:
 ```text
 # in a Python 3 virtual environment where bigchaindb is installed

docs/server/source/conf.py

@@ -35,6 +35,10 @@ _version = {}
 with open('../../../bigchaindb/version.py') as fp:
     exec(fp.read(), _version)
 
+import os.path
+import sys
+
+sys.path.insert(0, os.path.abspath(os.path.dirname(__file__) + '/..'))
 
 extensions = [
     'sphinx.ext.autodoc',
@@ -43,6 +47,11 @@ extensions = [
     'sphinx.ext.viewcode',
     'sphinx.ext.napoleon',
     'sphinxcontrib.httpdomain',
+    'sphinx.ext.autosectionlabel',
+    # Below are actually build steps made to look like sphinx extensions.
+    # It was the easiest way to get it running with ReadTheDocs.
+    'generate_schema_documentation',
+    'generate_http_server_api_documentation',
 ]
 
 # autodoc settings

docs/server/source/data-models/asset-model.md

@@ -28,5 +28,4 @@ For `TRANSFER` transactions we only keep the asset id.
 - `data`: A user supplied JSON document with custom information about the asset. Defaults to null.
 - _amount_: The amount of "shares". Only relevant if the asset is marked as divisible. Defaults to 1. The amount is not specified in the asset, but in the conditions (see next section).
 
-At the time of this writing divisible, updatable, and refillable assets are not yet implemented.
-See [Issue #487 on Github](https://github.com/bigchaindb/bigchaindb/issues/487)
+At the time of this writing, updatable and refillable assets are not yet implemented.

docs/server/source/data-models/block-model.rst

@@ -10,8 +10,8 @@ A block has the following structure:
       "block": {
         "timestamp": "<block-creation timestamp>",
         "transactions": ["<list of transactions>"],
-        "node_pubkey": "<public/verifying key of the node creating the block>",
-        "voters": ["<list of federation nodes verifying keys>"]
+        "node_pubkey": "<public key of the node creating the block>",
+        "voters": ["<list of federation nodes public keys>"]
       },
       "signature": "<signature of block>"
     }
@@ -20,14 +20,14 @@ A block has the following structure:
 - ``id``: The hash of the serialized ``block`` (i.e. the ``timestamp``, ``transactions``, ``node_pubkey``, and ``voters``). This is also a database primary key; that's how we ensure that all blocks are unique.
 
 - ``block``:
-    - ``timestamp``: The Unix time when the block was created. It's provided by the node that created the block. See `the page about timestamps <https://docs.bigchaindb.com/en/latest/timestamps.html>`_.
+    - ``timestamp``: The Unix time when the block was created. It's provided by the node that created the block.
     - ``transactions``: A list of the transactions included in the block.
-    - ``node_pubkey``: The public/verifying key of the node that created the block.
-    - ``voters``: A list of the verifying keys of federation nodes at the time the block was created.
+    - ``node_pubkey``: The public key of the node that created the block.
+    - ``voters``: A list of the public keys of federation nodes at the time the block was created.
       It's the list of federation nodes which can cast a vote on this block.
       This list can change from block to block, as nodes join and leave the federation.
 
-- ``signature``: Cryptographic signature of the block by the node that created the block. (To create the signature, the node serializes the block contents and signs that with its signing key.)
+- ``signature``: Cryptographic signature of the block by the node that created the block. (To create the signature, the node serializes the block contents and signs it with its private key.)
 
 
 Working with Blocks

docs/server/source/data-models/transaction-model.rst

@@ -0,0 +1,57 @@
+.. raw:: html
+
+   <style>
+    .rst-content a.internal[href*='/schema/'] {
+        border: solid 1px #e1e4e5;
+        font-family: monospace;
+        font-size: 12px;
+        color: blue;
+        padding: 2px 4px;
+        background-color: white;
+    }
+   </style>
+
+=====================
+The Transaction Model
+=====================
+
+A transaction has the following structure:
+
+.. code-block:: json
+
+    {
+        "id": "<hash of transaction, excluding signatures (see explanation)>",
+        "version": "<version number of the transaction model>",
+        "fulfillments": ["<list of fulfillments>"],
+        "conditions": ["<list of conditions>"],
+        "operation": "<string>",
+        "asset": "<digital asset description (explained in the next section)>",
+        "metadata": {
+            "id": "<uuid>",
+            "data": "<any JSON document>"
+        }
+    }
+
+Here's some explanation of the contents of a :ref:`transaction <transaction>`:
+
+- id: The :ref:`id <transaction.id>` of the transaction, and also the database primary key.
+- version: :ref:`Version <transaction.version>` number of the transaction model, so that software can support different transaction models.
+- **fulfillments**: List of fulfillments. Each :ref:`fulfillment <Fulfillment>` contains a pointer to an unspent asset
+  and a *crypto fulfillment* that satisfies a spending condition set on the unspent asset. A *fulfillment*
+  is usually a signature proving the ownership of the asset.
+  See :doc:`./crypto-conditions`.
+
+- **conditions**: List of conditions. Each :ref:`condition <Condition>` is a *crypto-condition* that needs to be fulfilled by a transfer transaction in order to transfer ownership to new owners.
+  See :doc:`./crypto-conditions`.
+
+- **operation**: String representation of the :ref:`operation <transaction.operation>` being performed (currently either "CREATE", "TRANSFER" or "GENESIS"). It determines how the transaction should be validated.
+
+- **asset**: Definition of the digital :ref:`asset <Asset>`. See next section.
+
+- **metadata**:
+    - :ref:`id <metadata.id>`: UUID version 4 (random) converted to a string of hex digits in standard form.
+    - :ref:`data <metadata.data>`: Can be any JSON document. It may be empty in the case of a transfer transaction.
+
+Later, when we get to the models for the block and the vote, we'll see that both include a signature (from the node which created it). You may wonder why transactions don't have signatures... The answer is that they do! They're just hidden inside the ``fulfillment`` string of each fulfillment. A creation transaction is signed by whoever created it. A transfer transaction is signed by whoever currently controls or owns it.
+
+What gets signed? For each fulfillment in the transaction, the "fullfillment message" that gets signed includes the ``operation``, ``data``, ``version``, ``id``, corresponding ``condition``, and the fulfillment itself, except with its fulfillment string set to ``null``. The computed signature goes into creating the ``fulfillment`` string of the fulfillment.

docs/server/source/data-models/vote-model.md

@@ -13,7 +13,7 @@ A vote has the following structure:
         "invalid_reason": "<None|DOUBLE_SPEND|TRANSACTIONS_HASH_MISMATCH|NODES_PUBKEYS_MISMATCH",
         "timestamp": "<Unix time when the vote was generated, provided by the voting node>"
     },
-    "signature": "<signature of vote>",
+    "signature": "<signature of vote>"
 }
 ```
 

docs/server/source/drivers-clients/http-client-server-api.rst

@@ -74,119 +74,23 @@ POST /transactions/
 
    Push a new transaction.
 
-   Note: The posted transaction should be valid `transaction
+   Note: The posted transaction should be a valid and signed `transaction
    <https://bigchaindb.readthedocs.io/en/latest/data-models/transaction-model.html>`_.
    The steps to build a valid transaction are beyond the scope of this page.
    One would normally use a driver such as the `BigchainDB Python Driver
    <https://docs.bigchaindb.com/projects/py-driver/en/latest/index.html>`_ to
-   build a valid transaction.
+   build a valid transaction. The exact contents of a valid transaction depend 
+   on the associated public/private keypairs.
 
    **Example request**:
 
-   .. sourcecode:: http
-
-      POST /transactions/ HTTP/1.1
-      Host: example.com
-      Content-Type: application/json
-
-      {
-        "transaction": {
-          "conditions": [
-            {
-              "cid": 0,
-              "condition": {
-                "uri": "cc:4:20:GG-pi3CeIlySZhQoJVBh9O23PzrOuhnYI7OHqIbHjkk:96",
-                "details": {
-                  "signature": null,
-                  "type": "fulfillment",
-                  "type_id": 4,
-                  "bitmask": 32,
-                  "public_key": "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-                }
-              },
-              "amount": 1,
-              "owners_after": [
-                "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-              ]
-            }
-          ],
-          "operation": "CREATE",
-          "asset": {
-            "divisible": false,
-            "updatable": false,
-            "data": null,
-            "id": "aebeab22-e672-4d3b-a187-bde5fda6533d",
-            "refillable": false
-          },
-          "metadata": null,
-          "timestamp": "1477578978",
-          "fulfillments": [
-            {
-              "fid": 0,
-              "input": null,
-              "fulfillment": "cf:4:GG-pi3CeIlySZhQoJVBh9O23PzrOuhnYI7OHqIbHjkn2VnQaEWvecO1x82Qr2Va_JjFywLKIOEV1Ob9Ofkeln2K89ny2mB-s7RLNvYAVzWNiQnp18_nQEUsvwACEXTYJ",
-              "owners_before": [
-                "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-              ]
-            }
-          ]
-        },
-        "id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e",
-        "version": 1
-      }
+   .. literalinclude:: samples/post-tx-request.http
+      :language: http
 
    **Example response**:
 
-   .. sourcecode:: http
-
-      HTTP/1.1 201 Created
-      Content-Type: application/json
-
-      {
-        "id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e",
-        "version": 1,
-        "transaction": {
-          "conditions": [
-            {
-              "amount": 1,
-              "condition": {
-                "uri": "cc:4:20:GG-pi3CeIlySZhQoJVBh9O23PzrOuhnYI7OHqIbHjkk:96",
-                "details": {
-                  "signature": null,
-                  "type_id": 4,
-                  "type": "fulfillment",
-                  "bitmask": 32,
-                  "public_key": "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-                }
-              },
-              "owners_after": [
-                "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-              ],
-              "cid": 0
-            }
-          ],
-          "fulfillments": [
-            {
-              "input": null,
-              "fulfillment": "cf:4:GG-pi3CeIlySZhQoJVBh9O23PzrOuhnYI7OHqIbHjkn2VnQaEWvecO1x82Qr2Va_JjFywLKIOEV1Ob9Ofkeln2K89ny2mB-s7RLNvYAVzWNiQnp18_nQEUsvwACEXTYJ",
-              "fid": 0,
-              "owners_before": [
-                "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-              ]
-            }
-          ],
-          "operation": "CREATE",
-          "timestamp": "1477578978",
-          "asset": {
-            "updatable": false,
-            "refillable": false,
-            "divisible": false,
-            "data": null,
-            "id": "aebeab22-e672-4d3b-a187-bde5fda6533d"
-          },
-          "metadata": null
-        }
-      }
+   .. literalinclude:: samples/post-tx-response.http
+      :language: http
 
    :statuscode 201: A new transaction was created.
    :statuscode 400: The transaction was invalid and not created.
@@ -208,21 +112,13 @@ GET /transactions/{tx_id}/status
 
    **Example request**:
 
-   .. sourcecode:: http
-
-      GET /transactions/7ad5a4b83bc8c70c4fd7420ff3c60693ab8e6d0e3124378ca69ed5acd2578792/status HTTP/1.1
-      Host: example.com
+   .. literalinclude:: samples/get-tx-status-request.http
+      :language: http
 
    **Example response**:
 
-   .. sourcecode:: http
-
-      HTTP/1.1 200 OK
-      Content-Type: application/json
-
-      {
-        "status": "valid"
-      }
+   .. literalinclude:: samples/get-tx-status-response.http
+      :language: http
 
    :statuscode 200: A transaction with that ID was found and the status is returned.
    :statuscode 404: A transaction with that ID was not found.
@@ -243,63 +139,13 @@ GET /transactions/{tx_id}
 
    **Example request**:
 
-   .. sourcecode:: http
-
-      GET /transactions/2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e HTTP/1.1
-      Host: example.com
+   .. literalinclude:: samples/get-tx-request.http
+      :language: http
 
    **Example response**:
 
-   .. sourcecode:: http
-
-      HTTP/1.1 200 OK
-      Content-Type: application/json
-
-      {
-        "transaction": {
-          "conditions": [
-            {
-              "cid": 0,
-              "condition": {
-                "uri": "cc:4:20:GG-pi3CeIlySZhQoJVBh9O23PzrOuhnYI7OHqIbHjkk:96",
-                "details": {
-                  "signature": null,
-                  "type": "fulfillment",
-                  "type_id": 4,
-                  "bitmask": 32,
-                  "public_key": "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-                }
-              },
-              "amount": 1,
-              "owners_after": [
-                "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-              ]
-            }
-          ],
-          "operation": "CREATE",
-          "asset": {
-            "divisible": false,
-            "updatable": false,
-            "data": null,
-            "id": "aebeab22-e672-4d3b-a187-bde5fda6533d",
-            "refillable": false
-          },
-          "metadata": null,
-          "timestamp": "1477578978",
-          "fulfillments": [
-            {
-              "fid": 0,
-              "input": null,
-              "fulfillment": "cf:4:GG-pi3CeIlySZhQoJVBh9O23PzrOuhnYI7OHqIbHjkn2VnQaEWvecO1x82Qr2Va_JjFywLKIOEV1Ob9Ofkeln2K89ny2mB-s7RLNvYAVzWNiQnp18_nQEUsvwACEXTYJ",
-              "owners_before": [
-                "2ePYHfV3yS3xTxF9EE3Xjo8zPwq2RmLPFAJGQqQKc3j6"
-              ]
-            }
-          ]
-        },
-        "id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e",
-        "version": 1
-      }
+   .. literalinclude:: samples/get-tx-response.http
+      :language: http
 
    :statuscode 200: A transaction with that ID was found.
    :statuscode 404: A transaction with that ID was not found.
@@ -308,6 +154,11 @@ GET /transactions/{tx_id}
 GET /unspents/
 -------------------------
 
+.. note::
+
+   This endpoint (unspents) is not yet implemented. We published it here for preview and comment.
+   
+
 .. http:get:: /unspents?owner_after={owner_after}
 
    Get a list of links to transactions' conditions that have not been used in

docs/server/source/drivers-clients/index.rst

@@ -1,10 +1,17 @@
 Drivers & Clients
 =================
 
+Currently, the only language-native driver is written in the Python language.
+
+We also provide the Transaction CLI to be able to script the building of
+transactions. You may be able to wrap this tool inside the language of
+your choice, and then use the HTTP API directly to post transactions.
+
+
 .. toctree::
    :maxdepth: 1
 
    http-client-server-api
-   python-driver
+   The Python Driver <https://docs.bigchaindb.com/projects/py-driver/en/latest/index.html>
+   Transaction CLI <https://docs.bigchaindb.com/projects/cli/en/latest/>
    example-apps
-   

docs/server/source/drivers-clients/python-driver.md

@@ -1,7 +0,0 @@
-# The Python Driver
-
-The BigchainDB Python Driver is a Python wrapper around the [HTTP Client-Server API](http-client-server-api.html). A developer can use it to develop a Python app that communicates with one or more BigchainDB clusters.
-
-The BigchainDB Python Driver documentation is at:
-
-[http://docs.bigchaindb.com/projects/py-driver/en/latest/index.html](http://docs.bigchaindb.com/projects/py-driver/en/latest/index.html)

docs/server/source/index.rst

@@ -14,5 +14,7 @@ BigchainDB Server Documentation
    drivers-clients/index
    clusters-feds/index
    topic-guides/index
+   data-models/index
+   schema/transaction
    release-notes
    appendices/index

docs/server/source/server-reference/configuration.md

@@ -21,6 +21,7 @@ For convenience, here's a list of all the relevant environment variables (docume
 `BIGCHAINDB_STATSD_PORT`<br>
 `BIGCHAINDB_STATSD_RATE`<br>
 `BIGCHAINDB_CONFIG_PATH`<br>
+`BIGCHAINDB_BACKLOG_REASSIGN_DELAY`<br>
 
 The local config file is `$HOME/.bigchaindb` by default (a file which might not even exist), but you can tell BigchainDB to use a different file by using the `-c` command-line option, e.g. `bigchaindb -c path/to/config_file.json start`
 or using the `BIGCHAINDB_CONFIG_PATH` environment variable, e.g. `BIGHAINDB_CONFIG_PATH=.my_bigchaindb_config bigchaindb start`.
@@ -154,3 +155,17 @@ export BIGCHAINDB_STATSD_RATE=0.01
 ```js
 "statsd": {"host": "localhost", "port": 8125, "rate": 0.01}
 ```
+
+## backlog_reassign_delay
+
+Specifies how long, in seconds, transactions can remain in the backlog before being reassigned.  Long-waiting transactions must be reassigned because the assigned node may no longer be responsive.  The default duration is 120 seconds.
+
+**Example using environment variables**
+```text
+export BIGCHAINDB_BACKLOG_REASSIGN_DELAY=30
+``` 
+
+**Default value (from a config file)**
+```js
+"backlog_reassign_delay": 120 
+```

setup.py

@@ -27,6 +27,18 @@ def check_setuptools_features():
 
 check_setuptools_features()
 
+dev_require = [
+    'ipdb',
+    'ipython',
+]
+
+docs_require = [
+    'Sphinx>=1.4.8',
+    'recommonmark>=0.4.0',
+    'sphinx-rtd-theme>=0.1.9',
+    'sphinxcontrib-httpdomain>=1.5.0',
+    'sphinxcontrib-napoleon>=0.4.4',
+]
 
 tests_require = [
     'coverage',
@@ -37,19 +49,7 @@ tests_require = [
     'pytest-cov>=2.2.1',
     'pytest-xdist',
     'pytest-flask',
-]
-
-dev_require = [
-    'ipdb',
-    'ipython',
-]
-
-docs_require = [
-    'Sphinx>=1.3.5',
-    'recommonmark>=0.4.0',
-    'sphinx-rtd-theme>=0.1.9',
-    'sphinxcontrib-httpdomain>=1.5.0',
-]
+] + docs_require
 
 benchmarks_require = [
     'line-profiler==1.0',
@@ -67,6 +67,8 @@ install_requires = [
     'requests~=2.9',
     'gunicorn~=19.0',
     'multipipes~=0.1.0',
+    'jsonschema~=2.5.1',
+    'pyyaml~=3.12',
 ]
 
 setup(
@@ -110,4 +112,5 @@ setup(
         'dev': dev_require + tests_require + docs_require + benchmarks_require,
         'docs': docs_require,
     },
+    package_data={'bigchaindb.common.schema': ['transaction.yaml']},
 )

tests/assets/test_digital_assets.py

@@ -5,13 +5,13 @@ from ..db.conftest import inputs  # noqa
 
 
 @pytest.mark.usefixtures('inputs')
-def test_asset_transfer(b, user_vk, user_sk):
+def test_asset_transfer(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
 
-    tx_input = b.get_owned_ids(user_vk).pop()
+    tx_input = b.get_owned_ids(user_pk).pop()
     tx_create = b.get_transaction(tx_input.txid)
 
-    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_vk], 1)],
+    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_pk], 1)],
                                        tx_create.asset)
     tx_transfer_signed = tx_transfer.sign([user_sk])
 
@@ -19,11 +19,11 @@ def test_asset_transfer(b, user_vk, user_sk):
     assert tx_transfer_signed.asset.data_id == tx_create.asset.data_id
 
 
-def test_validate_bad_asset_creation(b, user_vk):
+def test_validate_bad_asset_creation(b, user_pk):
     from bigchaindb.models import Transaction, Asset
 
     # `divisible` needs to be a boolean
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx.asset.divisible = 1
     with patch.object(Asset, 'validate_asset', return_value=None):
         tx_signed = tx.sign([b.me_private])
@@ -31,7 +31,7 @@ def test_validate_bad_asset_creation(b, user_vk):
         tx_signed.validate(b)
 
     # `refillable` needs to be a boolean
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx.asset.refillable = 1
     with patch.object(Asset, 'validate_asset', return_value=None):
         tx_signed = tx.sign([b.me_private])
@@ -39,7 +39,7 @@ def test_validate_bad_asset_creation(b, user_vk):
         b.validate_transaction(tx_signed)
 
     # `updatable` needs to be a boolean
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx.asset.updatable = 1
     with patch.object(Asset, 'validate_asset', return_value=None):
         tx_signed = tx.sign([b.me_private])
@@ -47,7 +47,7 @@ def test_validate_bad_asset_creation(b, user_vk):
         b.validate_transaction(tx_signed)
 
     # `data` needs to be a dictionary
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx.asset.data = 'a'
     with patch.object(Asset, 'validate_asset', return_value=None):
         tx_signed = tx.sign([b.me_private])
@@ -56,13 +56,13 @@ def test_validate_bad_asset_creation(b, user_vk):
 
 
 @pytest.mark.usefixtures('inputs')
-def test_validate_transfer_asset_id_mismatch(b, user_vk, user_sk):
+def test_validate_transfer_asset_id_mismatch(b, user_pk, user_sk):
     from bigchaindb.common.exceptions import AssetIdMismatch
     from bigchaindb.models import Transaction
 
-    tx_create = b.get_owned_ids(user_vk).pop()
+    tx_create = b.get_owned_ids(user_pk).pop()
     tx_create = b.get_transaction(tx_create.txid)
-    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_vk], 1)],
+    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_pk], 1)],
                                        tx_create.asset)
     tx_transfer.asset.data_id = 'aaa'
     tx_transfer_signed = tx_transfer.sign([user_sk])
@@ -70,23 +70,23 @@ def test_validate_transfer_asset_id_mismatch(b, user_vk, user_sk):
         tx_transfer_signed.validate(b)
 
 
-def test_get_asset_id_create_transaction(b, user_vk):
+def test_get_asset_id_create_transaction(b, user_pk):
     from bigchaindb.models import Transaction, Asset
 
-    tx_create = Transaction.create([b.me], [([user_vk], 1)])
+    tx_create = Transaction.create([b.me], [([user_pk], 1)])
     asset_id = Asset.get_asset_id(tx_create)
 
     assert asset_id == tx_create.asset.data_id
 
 
 @pytest.mark.usefixtures('inputs')
-def test_get_asset_id_transfer_transaction(b, user_vk, user_sk):
+def test_get_asset_id_transfer_transaction(b, user_pk, user_sk):
     from bigchaindb.models import Transaction, Asset
 
-    tx_create = b.get_owned_ids(user_vk).pop()
+    tx_create = b.get_owned_ids(user_pk).pop()
     tx_create = b.get_transaction(tx_create.txid)
     # create a transfer transaction
-    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_vk], 1)],
+    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_pk], 1)],
                                        tx_create.asset)
     tx_transfer_signed = tx_transfer.sign([user_sk])
     # create a block
@@ -100,32 +100,32 @@ def test_get_asset_id_transfer_transaction(b, user_vk, user_sk):
     assert asset_id == tx_transfer.asset.data_id
 
 
-def test_asset_id_mismatch(b, user_vk):
+def test_asset_id_mismatch(b, user_pk):
     from bigchaindb.models import Transaction, Asset
     from bigchaindb.common.exceptions import AssetIdMismatch
 
-    tx1 = Transaction.create([b.me], [([user_vk], 1)])
-    tx2 = Transaction.create([b.me], [([user_vk], 1)])
+    tx1 = Transaction.create([b.me], [([user_pk], 1)])
+    tx2 = Transaction.create([b.me], [([user_pk], 1)])
 
     with pytest.raises(AssetIdMismatch):
         Asset.get_asset_id([tx1, tx2])
 
 
 @pytest.mark.usefixtures('inputs')
-def test_get_txs_by_asset_id(b, user_vk, user_sk):
+def test_get_transactions_by_asset_id(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
 
-    tx_create = b.get_owned_ids(user_vk).pop()
+    tx_create = b.get_owned_ids(user_pk).pop()
     tx_create = b.get_transaction(tx_create.txid)
     asset_id = tx_create.asset.data_id
-    txs = b.get_txs_by_asset_id(asset_id)
+    txs = b.get_transactions_by_asset_id(asset_id)
 
     assert len(txs) == 1
     assert txs[0].id == tx_create.id
     assert txs[0].asset.data_id == asset_id
 
     # create a transfer transaction
-    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_vk], 1)],
+    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_pk], 1)],
                                        tx_create.asset)
     tx_transfer_signed = tx_transfer.sign([user_sk])
     # create the block
@@ -135,7 +135,7 @@ def test_get_txs_by_asset_id(b, user_vk, user_sk):
     vote = b.vote(block.id, b.get_last_voted_block().id, True)
     b.write_vote(vote)
 
-    txs = b.get_txs_by_asset_id(asset_id)
+    txs = b.get_transactions_by_asset_id(asset_id)
 
     assert len(txs) == 2
     assert tx_create.id in [t.id for t in txs]
@@ -145,15 +145,44 @@ def test_get_txs_by_asset_id(b, user_vk, user_sk):
 
 
 @pytest.mark.usefixtures('inputs')
-def test_get_asset_by_id(b, user_vk, user_sk):
+def test_get_transactions_by_asset_id_with_invalid_block(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
 
-    tx_create = b.get_owned_ids(user_vk).pop()
+    tx_create = b.get_owned_ids(user_pk).pop()
+    tx_create = b.get_transaction(tx_create.txid)
+    asset_id = tx_create.asset.data_id
+    txs = b.get_transactions_by_asset_id(asset_id)
+
+    assert len(txs) == 1
+    assert txs[0].id == tx_create.id
+    assert txs[0].asset.data_id == asset_id
+
+    # create a transfer transaction
+    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_pk], 1)],
+                                       tx_create.asset)
+    tx_transfer_signed = tx_transfer.sign([user_sk])
+    # create the block
+    block = b.create_block([tx_transfer_signed])
+    b.write_block(block, durability='hard')
+    # vote the block invalid
+    vote = b.vote(block.id, b.get_last_voted_block().id, False)
+    b.write_vote(vote)
+
+    txs = b.get_transactions_by_asset_id(asset_id)
+
+    assert len(txs) == 1
+
+
+@pytest.mark.usefixtures('inputs')
+def test_get_asset_by_id(b, user_pk, user_sk):
+    from bigchaindb.models import Transaction
+
+    tx_create = b.get_owned_ids(user_pk).pop()
     tx_create = b.get_transaction(tx_create.txid)
     asset_id = tx_create.asset.data_id
 
     # create a transfer transaction
-    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_vk], 1)],
+    tx_transfer = Transaction.transfer(tx_create.to_inputs(), [([user_pk], 1)],
                                        tx_create.asset)
     tx_transfer_signed = tx_transfer.sign([user_sk])
     # create the block
@@ -163,14 +192,14 @@ def test_get_asset_by_id(b, user_vk, user_sk):
     vote = b.vote(block.id, b.get_last_voted_block().id, True)
     b.write_vote(vote)
 
-    txs = b.get_txs_by_asset_id(asset_id)
+    txs = b.get_transactions_by_asset_id(asset_id)
     assert len(txs) == 2
 
     asset = b.get_asset_by_id(asset_id)
     assert asset == tx_create.asset
 
 
-def test_create_invalid_divisible_asset(b, user_vk, user_sk):
+def test_create_invalid_divisible_asset(b, user_pk, user_sk):
     from bigchaindb.models import Transaction, Asset
     from bigchaindb.common.exceptions import AmountError
 
@@ -178,19 +207,19 @@ def test_create_invalid_divisible_asset(b, user_vk, user_sk):
     # Transaction.__init__ should raise an exception
     asset = Asset(divisible=False)
     with pytest.raises(AmountError):
-        Transaction.create([user_vk], [([user_vk], 2)], asset=asset)
+        Transaction.create([user_pk], [([user_pk], 2)], asset=asset)
 
     # divisible assets need to have an amount > 1
     # Transaction.__init__ should raise an exception
     asset = Asset(divisible=True)
     with pytest.raises(AmountError):
-        Transaction.create([user_vk], [([user_vk], 1)], asset=asset)
+        Transaction.create([user_pk], [([user_pk], 1)], asset=asset)
 
     # even if a transaction is badly constructed the server should raise the
     # exception
     asset = Asset(divisible=False)
     with patch.object(Asset, 'validate_asset', return_value=None):
-        tx = Transaction.create([user_vk], [([user_vk], 2)], asset=asset)
+        tx = Transaction.create([user_pk], [([user_pk], 2)], asset=asset)
         tx_signed = tx.sign([user_sk])
     with pytest.raises(AmountError):
         tx_signed.validate(b)
@@ -198,17 +227,17 @@ def test_create_invalid_divisible_asset(b, user_vk, user_sk):
 
     asset = Asset(divisible=True)
     with patch.object(Asset, 'validate_asset', return_value=None):
-        tx = Transaction.create([user_vk], [([user_vk], 1)], asset=asset)
+        tx = Transaction.create([user_pk], [([user_pk], 1)], asset=asset)
         tx_signed = tx.sign([user_sk])
     with pytest.raises(AmountError):
         tx_signed.validate(b)
     assert b.is_valid_transaction(tx_signed) is False
 
 
-def test_create_valid_divisible_asset(b, user_vk, user_sk):
+def test_create_valid_divisible_asset(b, user_pk, user_sk):
     from bigchaindb.models import Transaction, Asset
 
     asset = Asset(divisible=True)
-    tx = Transaction.create([user_vk], [([user_vk], 2)], asset=asset)
+    tx = Transaction.create([user_pk], [([user_pk], 2)], asset=asset)
     tx_signed = tx.sign([user_sk])
     assert b.is_valid_transaction(tx_signed)

tests/assets/test_divisible_assets.py

@@ -10,12 +10,12 @@ from ..db.conftest import inputs  # noqa
 # Single owners_before
 # Single output
 # Single owners_after
-def test_single_in_single_own_single_out_single_own_create(b, user_vk):
+def test_single_in_single_own_single_out_single_own_create(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     asset = Asset(divisible=True)
-    tx = Transaction.create([b.me], [([user_vk], 100)], asset=asset)
+    tx = Transaction.create([b.me], [([user_pk], 100)], asset=asset)
     tx_signed = tx.sign([b.me_private])
 
     assert tx_signed.validate(b) == tx_signed
@@ -29,12 +29,12 @@ def test_single_in_single_own_single_out_single_own_create(b, user_vk):
 # Single owners_before
 # Multiple outputs
 # Single owners_after per output
-def test_single_in_single_own_multiple_out_single_own_create(b, user_vk):
+def test_single_in_single_own_multiple_out_single_own_create(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     asset = Asset(divisible=True)
-    tx = Transaction.create([b.me], [([user_vk], 50), ([user_vk], 50)],
+    tx = Transaction.create([b.me], [([user_pk], 50), ([user_pk], 50)],
                             asset=asset)
     tx_signed = tx.sign([b.me_private])
 
@@ -50,12 +50,12 @@ def test_single_in_single_own_multiple_out_single_own_create(b, user_vk):
 # Single owners_before
 # Single output
 # Multiple owners_after
-def test_single_in_single_own_single_out_multiple_own_create(b, user_vk):
+def test_single_in_single_own_single_out_multiple_own_create(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     asset = Asset(divisible=True)
-    tx = Transaction.create([b.me], [([user_vk, user_vk], 100)], asset=asset)
+    tx = Transaction.create([b.me], [([user_pk, user_pk], 100)], asset=asset)
     tx_signed = tx.sign([b.me_private])
 
     assert tx_signed.validate(b) == tx_signed
@@ -75,13 +75,13 @@ def test_single_in_single_own_single_out_multiple_own_create(b, user_vk):
 # Multiple outputs
 # Mix: one output with a single owners_after, one output with multiple
 #      owners_after
-def test_single_in_single_own_multiple_out_mix_own_create(b, user_vk):
+def test_single_in_single_own_multiple_out_mix_own_create(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     asset = Asset(divisible=True)
     tx = Transaction.create([b.me],
-                            [([user_vk], 50), ([user_vk, user_vk], 50)],
+                            [([user_pk], 50), ([user_pk, user_pk], 50)],
                             asset=asset)
     tx_signed = tx.sign([b.me_private])
 
@@ -101,13 +101,13 @@ def test_single_in_single_own_multiple_out_mix_own_create(b, user_vk):
 # Single input
 # Multiple owners_before
 # Output combinations already tested above
-def test_single_in_multiple_own_single_out_single_own_create(b, user_vk,
+def test_single_in_multiple_own_single_out_single_own_create(b, user_pk,
                                                              user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     asset = Asset(divisible=True)
-    tx = Transaction.create([b.me, user_vk], [([user_vk], 100)], asset=asset)
+    tx = Transaction.create([b.me, user_pk], [([user_pk], 100)], asset=asset)
     tx_signed = tx.sign([b.me_private, user_sk])
     assert tx_signed.validate(b) == tx_signed
     assert len(tx_signed.conditions) == 1
@@ -129,14 +129,14 @@ def test_single_in_multiple_own_single_out_single_own_create(b, user_vk,
 #       fail.
 #       Is there a better way of doing this?
 @pytest.mark.usefixtures('inputs')
-def test_single_in_single_own_single_out_single_own_transfer(b, user_vk,
+def test_single_in_single_own_single_out_single_own_transfer(b, user_pk,
                                                              user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 100)], asset=asset)
+    tx_create = Transaction.create([b.me], [([user_pk], 100)], asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
     block = b.create_block([tx_create_signed])
@@ -163,14 +163,14 @@ def test_single_in_single_own_single_out_single_own_transfer(b, user_vk,
 # Multiple output
 # Single owners_after
 @pytest.mark.usefixtures('inputs')
-def test_single_in_single_own_multiple_out_single_own_transfer(b, user_vk,
+def test_single_in_single_own_multiple_out_single_own_transfer(b, user_pk,
                                                                user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 100)], asset=asset)
+    tx_create = Transaction.create([b.me], [([user_pk], 100)], asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
     block = b.create_block([tx_create_signed])
@@ -199,14 +199,14 @@ def test_single_in_single_own_multiple_out_single_own_transfer(b, user_vk,
 # Single output
 # Multiple owners_after
 @pytest.mark.usefixtures('inputs')
-def test_single_in_single_own_single_out_multiple_own_transfer(b, user_vk,
+def test_single_in_single_own_single_out_multiple_own_transfer(b, user_pk,
                                                                user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 100)], asset=asset)
+    tx_create = Transaction.create([b.me], [([user_pk], 100)], asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
     block = b.create_block([tx_create_signed])
@@ -240,14 +240,14 @@ def test_single_in_single_own_single_out_multiple_own_transfer(b, user_vk,
 # Mix: one output with a single owners_after, one output with multiple
 #      owners_after
 @pytest.mark.usefixtures('inputs')
-def test_single_in_single_own_multiple_out_mix_own_transfer(b, user_vk,
+def test_single_in_single_own_multiple_out_mix_own_transfer(b, user_pk,
                                                             user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 100)], asset=asset)
+    tx_create = Transaction.create([b.me], [([user_pk], 100)], asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
     block = b.create_block([tx_create_signed])
@@ -281,14 +281,14 @@ def test_single_in_single_own_multiple_out_mix_own_transfer(b, user_vk,
 # Single output
 # Single owners_after
 @pytest.mark.usefixtures('inputs')
-def test_single_in_multiple_own_single_out_single_own_transfer(b, user_vk,
+def test_single_in_multiple_own_single_out_single_own_transfer(b, user_pk,
                                                                user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([b.me, user_vk], 100)],
+    tx_create = Transaction.create([b.me], [([b.me, user_pk], 100)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -320,14 +320,14 @@ def test_single_in_multiple_own_single_out_single_own_transfer(b, user_vk,
 # Single output
 # Single owners_after
 @pytest.mark.usefixtures('inputs')
-def test_multiple_in_single_own_single_out_single_own_transfer(b, user_vk,
+def test_multiple_in_single_own_single_out_single_own_transfer(b, user_pk,
                                                                user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 50), ([user_vk], 50)],
+    tx_create = Transaction.create([b.me], [([user_pk], 50), ([user_pk], 50)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -355,7 +355,7 @@ def test_multiple_in_single_own_single_out_single_own_transfer(b, user_vk,
 # Single output
 # Single owners_after
 @pytest.mark.usefixtures('inputs')
-def test_multiple_in_multiple_own_single_out_single_own_transfer(b, user_vk,
+def test_multiple_in_multiple_own_single_out_single_own_transfer(b, user_pk,
                                                                  user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
@@ -363,8 +363,8 @@ def test_multiple_in_multiple_own_single_out_single_own_transfer(b, user_vk,
     # CREATE divisible asset
     asset = Asset(divisible=True)
     tx_create = Transaction.create([b.me],
-                                   [([user_vk, b.me], 50),
-                                    ([user_vk, b.me], 50)],
+                                   [([user_pk, b.me], 50),
+                                    ([user_pk, b.me], 50)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -400,7 +400,7 @@ def test_multiple_in_multiple_own_single_out_single_own_transfer(b, user_vk,
 # Single output
 # Single owners_after
 @pytest.mark.usefixtures('inputs')
-def test_muiltiple_in_mix_own_multiple_out_single_own_transfer(b, user_vk,
+def test_muiltiple_in_mix_own_multiple_out_single_own_transfer(b, user_pk,
                                                                user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
@@ -408,8 +408,8 @@ def test_muiltiple_in_mix_own_multiple_out_single_own_transfer(b, user_vk,
     # CREATE divisible asset
     asset = Asset(divisible=True)
     tx_create = Transaction.create([b.me],
-                                   [([user_vk], 50),
-                                    ([user_vk, b.me], 50)],
+                                   [([user_pk], 50),
+                                    ([user_pk, b.me], 50)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -445,7 +445,7 @@ def test_muiltiple_in_mix_own_multiple_out_single_own_transfer(b, user_vk,
 # Mix: one output with a single owners_after, one output with multiple
 #      owners_after
 @pytest.mark.usefixtures('inputs')
-def test_muiltiple_in_mix_own_multiple_out_mix_own_transfer(b, user_vk,
+def test_muiltiple_in_mix_own_multiple_out_mix_own_transfer(b, user_pk,
                                                             user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
@@ -453,8 +453,8 @@ def test_muiltiple_in_mix_own_multiple_out_mix_own_transfer(b, user_vk,
     # CREATE divisible asset
     asset = Asset(divisible=True)
     tx_create = Transaction.create([b.me],
-                                   [([user_vk], 50),
-                                    ([user_vk, b.me], 50)],
+                                   [([user_pk], 50),
+                                    ([user_pk, b.me], 50)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -467,7 +467,7 @@ def test_muiltiple_in_mix_own_multiple_out_mix_own_transfer(b, user_vk,
 
     # TRANSFER
     tx_transfer = Transaction.transfer(tx_create.to_inputs(),
-                                       [([b.me], 50), ([b.me, user_vk], 50)],
+                                       [([b.me], 50), ([b.me, user_pk], 50)],
                                        asset=tx_create.asset)
     tx_transfer_signed = tx_transfer.sign([b.me_private, user_sk])
 
@@ -496,16 +496,16 @@ def test_muiltiple_in_mix_own_multiple_out_mix_own_transfer(b, user_vk,
 # Single output
 # Single owners_after
 @pytest.mark.usefixtures('inputs')
-def test_multiple_in_different_transactions(b, user_vk, user_sk):
+def test_multiple_in_different_transactions(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset
     # `b` creates a divisible asset and assigns 50 shares to `b` and
-    # 50 shares to `user_vk`
+    # 50 shares to `user_pk`
     asset = Asset(divisible=True)
     tx_create = Transaction.create([b.me],
-                                   [([user_vk], 50),
+                                   [([user_pk], 50),
                                     ([b.me], 50)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
@@ -518,11 +518,11 @@ def test_multiple_in_different_transactions(b, user_vk, user_sk):
     b.write_vote(vote)
 
     # TRANSFER divisible asset
-    # `b` transfers its 50 shares to `user_vk`
-    # after this transaction `user_vk` will have a total of 100 shares
+    # `b` transfers its 50 shares to `user_pk`
+    # after this transaction `user_pk` will have a total of 100 shares
     # split across two different transactions
     tx_transfer1 = Transaction.transfer(tx_create.to_inputs([1]),
-                                        [([user_vk], 50)],
+                                        [([user_pk], 50)],
                                         asset=tx_create.asset)
     tx_transfer1_signed = tx_transfer1.sign([b.me_private])
     # create block
@@ -534,7 +534,7 @@ def test_multiple_in_different_transactions(b, user_vk, user_sk):
     b.write_vote(vote)
 
     # TRANSFER
-    # `user_vk` combines two different transaction with 50 shares each and
+    # `user_pk` combines two different transaction with 50 shares each and
     # transfers a total of 100 shares back to `b`
     tx_transfer2 = Transaction.transfer(tx_create.to_inputs([0]) +
                                         tx_transfer1.to_inputs([0]),
@@ -557,14 +557,14 @@ def test_multiple_in_different_transactions(b, user_vk, user_sk):
 # inputs needs to match the amount being sent in the outputs.
 # In other words `amount_in_inputs - amount_in_outputs == 0`
 @pytest.mark.usefixtures('inputs')
-def test_amount_error_transfer(b, user_vk, user_sk):
+def test_amount_error_transfer(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
     from bigchaindb.common.exceptions import AmountError
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 100)], asset=asset)
+    tx_create = Transaction.create([b.me], [([user_pk], 100)], asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
     block = b.create_block([tx_create_signed])
@@ -593,7 +593,7 @@ def test_amount_error_transfer(b, user_vk, user_sk):
 
 @pytest.mark.skip(reason='Figure out how to handle this case')
 @pytest.mark.usefixtures('inputs')
-def test_threshold_same_public_key(b, user_vk, user_sk):
+def test_threshold_same_public_key(b, user_pk, user_sk):
     # If we try to fulfill a threshold condition where each subcondition has
     # the same key get_subcondition_from_vk will always return the first
     # subcondition. This means that only the 1st subfulfillment will be
@@ -606,7 +606,7 @@ def test_threshold_same_public_key(b, user_vk, user_sk):
 
     # CREATE divisible asset
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk, user_vk], 100)],
+    tx_create = Transaction.create([b.me], [([user_pk, user_pk], 100)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -626,16 +626,16 @@ def test_threshold_same_public_key(b, user_vk, user_sk):
 
 
 @pytest.mark.usefixtures('inputs')
-def test_sum_amount(b, user_vk, user_sk):
+def test_sum_amount(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset with 3 outputs with amount 1
     asset = Asset(divisible=True)
     tx_create = Transaction.create([b.me],
-                                   [([user_vk], 1),
-                                    ([user_vk], 1),
-                                    ([user_vk], 1)],
+                                   [([user_pk], 1),
+                                    ([user_pk], 1),
+                                    ([user_pk], 1)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -658,13 +658,13 @@ def test_sum_amount(b, user_vk, user_sk):
 
 
 @pytest.mark.usefixtures('inputs')
-def test_divide(b, user_vk, user_sk):
+def test_divide(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
 
     # CREATE divisible asset with 1 output with amount 3
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 3)],
+    tx_create = Transaction.create([b.me], [([user_pk], 3)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -690,14 +690,14 @@ def test_divide(b, user_vk, user_sk):
 
 # Check that negative inputs are caught when creating a TRANSFER transaction
 @pytest.mark.usefixtures('inputs')
-def test_non_positive_amounts_on_transfer(b, user_vk):
+def test_non_positive_amounts_on_transfer(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
     from bigchaindb.common.exceptions import AmountError
 
     # CREATE divisible asset with 1 output with amount 3
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 3)],
+    tx_create = Transaction.create([b.me], [([user_pk], 3)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -716,14 +716,14 @@ def test_non_positive_amounts_on_transfer(b, user_vk):
 
 # Check that negative inputs are caught when validating a TRANSFER transaction
 @pytest.mark.usefixtures('inputs')
-def test_non_positive_amounts_on_transfer_validate(b, user_vk, user_sk):
+def test_non_positive_amounts_on_transfer_validate(b, user_pk, user_sk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
     from bigchaindb.common.exceptions import AmountError
 
     # CREATE divisible asset with 1 output with amount 3
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 3)],
+    tx_create = Transaction.create([b.me], [([user_pk], 3)],
                                    asset=asset)
     tx_create_signed = tx_create.sign([b.me_private])
     # create block
@@ -748,7 +748,7 @@ def test_non_positive_amounts_on_transfer_validate(b, user_vk, user_sk):
 
 # Check that negative inputs are caught when creating a CREATE transaction
 @pytest.mark.usefixtures('inputs')
-def test_non_positive_amounts_on_create(b, user_vk):
+def test_non_positive_amounts_on_create(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
     from bigchaindb.common.exceptions import AmountError
@@ -756,20 +756,20 @@ def test_non_positive_amounts_on_create(b, user_vk):
     # CREATE divisible asset with 1 output with amount 3
     asset = Asset(divisible=True)
     with pytest.raises(AmountError):
-        Transaction.create([b.me], [([user_vk], -3)],
+        Transaction.create([b.me], [([user_pk], -3)],
                            asset=asset)
 
 
 # Check that negative inputs are caught when validating a CREATE transaction
 @pytest.mark.usefixtures('inputs')
-def test_non_positive_amounts_on_create_validate(b, user_vk):
+def test_non_positive_amounts_on_create_validate(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.transaction import Asset
     from bigchaindb.common.exceptions import AmountError
 
     # CREATE divisible asset with 1 output with amount 3
     asset = Asset(divisible=True)
-    tx_create = Transaction.create([b.me], [([user_vk], 3)],
+    tx_create = Transaction.create([b.me], [([user_pk], 3)],
                                    asset=asset)
     tx_create.conditions[0].amount = -3
     with patch.object(Asset, 'validate_asset', return_value=None):

tests/common/conftest.py

@@ -19,6 +19,8 @@ DATA = {
 }
 DATA_ID = '872fa6e6f46246cd44afdb2ee9cfae0e72885fb0910e2bcf9a5a2a4eadb417b8'
 
+UUID4 = 'dc568f27-a113-46b4-9bd4-43015859e3e3'
+
 
 @pytest.fixture
 def user_priv():
@@ -130,9 +132,8 @@ def data_id():
 
 
 @pytest.fixture
-def metadata(data, data_id):
-    from bigchaindb.common.transaction import Metadata
-    return Metadata(data, data_id)
+def uuid4():
+    return UUID4
 
 
 @pytest.fixture

tests/common/test_schema.py

@@ -0,0 +1,50 @@
+from pytest import raises
+
+from bigchaindb.common.exceptions import SchemaValidationError
+from bigchaindb.common.schema import TX_SCHEMA, validate_transaction_schema
+
+
+def test_validate_transaction_create(create_tx):
+    validate_transaction_schema(create_tx.to_dict())
+
+
+def test_validate_transaction_signed_create(signed_create_tx):
+    validate_transaction_schema(signed_create_tx.to_dict())
+
+
+def test_validate_transaction_signed_transfer(signed_transfer_tx):
+    validate_transaction_schema(signed_transfer_tx.to_dict())
+
+
+def test_validate_fails_metadata_empty_dict(create_tx):
+    create_tx.metadata = {'a': 1}
+    validate_transaction_schema(create_tx.to_dict())
+    create_tx.metadata = None
+    validate_transaction_schema(create_tx.to_dict())
+    create_tx.metadata = {}
+    with raises(SchemaValidationError):
+        validate_transaction_schema(create_tx.to_dict())
+
+
+def test_validation_fails():
+    with raises(SchemaValidationError):
+        validate_transaction_schema({})
+
+
+def test_addition_properties_always_set():
+    """
+    Validate that each object node has additionalProperties set, so that
+    transactions with junk keys do not pass as valid.
+    """
+    def walk(node, path=''):
+        if isinstance(node, list):
+            for i, nnode in enumerate(node):
+                walk(nnode, path + str(i) + '.')
+        if isinstance(node, dict):
+            if node.get('type') == 'object':
+                assert 'additionalProperties' in node, \
+                    ("additionalProperties not set at path:" + path)
+            for name, val in node.items():
+                walk(val, path + name + '.')
+
+    walk(TX_SCHEMA)

tests/common/test_transaction.py

@@ -274,6 +274,8 @@ def test_invalid_transaction_initialization():
 
 def test_create_default_asset_on_tx_initialization():
     from bigchaindb.common.transaction import Transaction, Asset
+    from bigchaindb.common.exceptions import ValidationError
+    from .util import validate_transaction_model
 
     with patch.object(Asset, 'validate_asset', return_value=None):
         tx = Transaction(Transaction.CREATE, None)
@@ -284,31 +286,33 @@ def test_create_default_asset_on_tx_initialization():
     asset.data_id = None
     assert asset == expected
 
+    # Fails because no asset hash
+    with raises(ValidationError):
+        validate_transaction_model(tx)
+
 
 def test_transaction_serialization(user_ffill, user_cond, data, data_id):
     from bigchaindb.common.transaction import Transaction, Asset
+    from bigchaindb.common.exceptions import ValidationError
+    from .util import validate_transaction_model
 
     tx_id = 'l0l'
-    timestamp = '66666666666'
 
     expected = {
         'id': tx_id,
         'version': Transaction.VERSION,
-        'transaction': {
-            # NOTE: This test assumes that Fulfillments and Conditions can
-            #       successfully be serialized
-            'fulfillments': [user_ffill.to_dict(0)],
-            'conditions': [user_cond.to_dict(0)],
-            'operation': Transaction.CREATE,
-            'timestamp': timestamp,
-            'metadata': None,
-            'asset': {
-                'id': data_id,
-                'divisible': False,
-                'updatable': False,
-                'refillable': False,
-                'data': data,
-            }
+        # NOTE: This test assumes that Fulfillments and Conditions can
+        #       successfully be serialized
+        'fulfillments': [user_ffill.to_dict(0)],
+        'conditions': [user_cond.to_dict(0)],
+        'operation': Transaction.CREATE,
+        'metadata': None,
+        'asset': {
+            'id': data_id,
+            'divisible': False,
+            'updatable': False,
+            'refillable': False,
+            'data': data,
         }
     }
 
@@ -316,38 +320,38 @@ def test_transaction_serialization(user_ffill, user_cond, data, data_id):
                      [user_cond])
     tx_dict = tx.to_dict()
     tx_dict['id'] = tx_id
-    tx_dict['transaction']['asset']['id'] = data_id
-    tx_dict['transaction']['timestamp'] = timestamp
+    tx_dict['asset']['id'] = data_id
 
     assert tx_dict == expected
 
+    # Fails because asset id is not a uuid4
+    with raises(ValidationError):
+        validate_transaction_model(tx)
 
-def test_transaction_deserialization(user_ffill, user_cond, data, data_id):
+
+def test_transaction_deserialization(user_ffill, user_cond, data, uuid4):
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
-    timestamp = '66666666666'
 
-    expected_asset = Asset(data, data_id)
+    expected_asset = Asset(data, uuid4)
     expected = Transaction(Transaction.CREATE, expected_asset, [user_ffill],
-                           [user_cond], None, timestamp, Transaction.VERSION)
+                           [user_cond], None, Transaction.VERSION)
 
     tx = {
         'version': Transaction.VERSION,
-        'transaction': {
-            # NOTE: This test assumes that Fulfillments and Conditions can
-            #       successfully be serialized
-            'fulfillments': [user_ffill.to_dict()],
-            'conditions': [user_cond.to_dict()],
-            'operation': Transaction.CREATE,
-            'timestamp': timestamp,
-            'metadata': None,
-            'asset': {
-                'id': data_id,
-                'divisible': False,
-                'updatable': False,
-                'refillable': False,
-                'data': data,
-            }
+        # NOTE: This test assumes that Fulfillments and Conditions can
+        #       successfully be serialized
+        'fulfillments': [user_ffill.to_dict()],
+        'conditions': [user_cond.to_dict()],
+        'operation': Transaction.CREATE,
+        'metadata': None,
+        'asset': {
+            'id': uuid4,
+            'divisible': False,
+            'updatable': False,
+            'refillable': False,
+            'data': data,
         }
     }
     tx_no_signatures = Transaction._remove_signatures(tx)
@@ -356,21 +360,18 @@ def test_transaction_deserialization(user_ffill, user_cond, data, data_id):
 
     assert tx == expected
 
+    validate_transaction_model(tx)
+
 
 def test_tx_serialization_with_incorrect_hash(utx):
     from bigchaindb.common.transaction import Transaction
     from bigchaindb.common.exceptions import InvalidHash
 
     utx_dict = utx.to_dict()
-    utx_dict['id'] = 'abc'
+    utx_dict['id'] = 'a' * 64
     with raises(InvalidHash):
         Transaction.from_dict(utx_dict)
     utx_dict.pop('id')
-    with raises(InvalidHash):
-        Transaction.from_dict(utx_dict)
-    utx_dict['id'] = []
-    with raises(InvalidHash):
-        Transaction.from_dict(utx_dict)
 
 
 def test_invalid_fulfillment_initialization(user_ffill, user_pub):
@@ -382,34 +383,6 @@ def test_invalid_fulfillment_initialization(user_ffill, user_pub):
         Fulfillment(user_ffill, [], tx_input='somethingthatiswrong')
 
 
-def test_invalid_metadata_initialization():
-    from bigchaindb.common.transaction import Metadata
-
-    with raises(TypeError):
-        Metadata([])
-
-
-def test_metadata_serialization(data, data_id):
-    from bigchaindb.common.transaction import Metadata
-
-    expected = {
-        'data': data,
-        'id': data_id,
-    }
-    metadata = Metadata(data, data_id)
-
-    assert metadata.to_dict() == expected
-
-
-def test_metadata_deserialization(data, data_id):
-    from bigchaindb.common.transaction import Metadata
-
-    expected = Metadata(data, data_id)
-    metadata = Metadata.from_dict({'data': data, 'id': data_id})
-
-    assert metadata == expected
-
-
 def test_transaction_link_serialization():
     from bigchaindb.common.transaction import TransactionLink
 
@@ -516,6 +489,16 @@ def test_cast_asset_link_to_boolean():
     assert bool(AssetLink(False)) is True
 
 
+def test_eq_asset_link():
+    from bigchaindb.common.transaction import AssetLink
+
+    asset_id_1 = 'asset_1'
+    asset_id_2 = 'asset_2'
+
+    assert AssetLink(asset_id_1) == AssetLink(asset_id_1)
+    assert AssetLink(asset_id_1) != AssetLink(asset_id_2)
+
+
 def test_add_fulfillment_to_tx(user_ffill):
     from bigchaindb.common.transaction import Transaction, Asset
 
@@ -537,6 +520,7 @@ def test_add_fulfillment_to_tx_with_invalid_parameters():
 
 def test_add_condition_to_tx(user_cond):
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     with patch.object(Asset, 'validate_asset', return_value=None):
         tx = Transaction(Transaction.CREATE, Asset())
@@ -544,6 +528,8 @@ def test_add_condition_to_tx(user_cond):
 
     assert len(tx.conditions) == 1
 
+    validate_transaction_model(tx)
+
 
 def test_add_condition_to_tx_with_invalid_parameters():
     from bigchaindb.common.transaction import Transaction, Asset
@@ -563,18 +549,21 @@ def test_sign_with_invalid_parameters(utx, user_priv):
 
 def test_validate_tx_simple_create_signature(user_ffill, user_cond, user_priv):
     from copy import deepcopy
-    from bigchaindb.common.crypto import SigningKey
+    from bigchaindb.common.crypto import PrivateKey
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     tx = Transaction(Transaction.CREATE, Asset(), [user_ffill], [user_cond])
     expected = deepcopy(user_cond)
-    expected.fulfillment.sign(str(tx).encode(), SigningKey(user_priv))
+    expected.fulfillment.sign(str(tx).encode(), PrivateKey(user_priv))
     tx.sign([user_priv])
 
     assert tx.fulfillments[0].to_dict()['fulfillment'] == \
         expected.fulfillment.serialize_uri()
     assert tx.fulfillments_valid() is True
 
+    validate_transaction_model(tx)
+
 
 def test_invoke_simple_signature_fulfillment_with_invalid_params(utx,
                                                                  user_ffill):
@@ -621,8 +610,9 @@ def test_validate_fulfillment_with_invalid_parameters(utx):
 def test_validate_multiple_fulfillments(user_ffill, user_cond, user_priv):
     from copy import deepcopy
 
-    from bigchaindb.common.crypto import SigningKey
+    from bigchaindb.common.crypto import PrivateKey
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     tx = Transaction(Transaction.CREATE, Asset(divisible=True),
                      [user_ffill, deepcopy(user_ffill)],
@@ -635,10 +625,10 @@ def test_validate_multiple_fulfillments(user_ffill, user_cond, user_priv):
 
     expected_first_bytes = str(expected_first).encode()
     expected_first.fulfillments[0].fulfillment.sign(expected_first_bytes,
-                                                    SigningKey(user_priv))
+                                                    PrivateKey(user_priv))
     expected_second_bytes = str(expected_second).encode()
     expected_second.fulfillments[0].fulfillment.sign(expected_second_bytes,
-                                                     SigningKey(user_priv))
+                                                     PrivateKey(user_priv))
     tx.sign([user_priv])
 
     assert tx.fulfillments[0].to_dict()['fulfillment'] == \
@@ -647,6 +637,8 @@ def test_validate_multiple_fulfillments(user_ffill, user_cond, user_priv):
         expected_second.fulfillments[0].fulfillment.serialize_uri()
     assert tx.fulfillments_valid() is True
 
+    validate_transaction_model(tx)
+
 
 def test_validate_tx_threshold_create_signature(user_user2_threshold_ffill,
                                                 user_user2_threshold_cond,
@@ -656,22 +648,25 @@ def test_validate_tx_threshold_create_signature(user_user2_threshold_ffill,
                                                 user2_priv):
     from copy import deepcopy
 
-    from bigchaindb.common.crypto import SigningKey
+    from bigchaindb.common.crypto import PrivateKey
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     tx = Transaction(Transaction.CREATE, Asset(), [user_user2_threshold_ffill],
                      [user_user2_threshold_cond])
     expected = deepcopy(user_user2_threshold_cond)
     expected.fulfillment.subconditions[0]['body'].sign(str(tx).encode(),
-                                                       SigningKey(user_priv))
+                                                       PrivateKey(user_priv))
     expected.fulfillment.subconditions[1]['body'].sign(str(tx).encode(),
-                                                       SigningKey(user2_priv))
+                                                       PrivateKey(user2_priv))
     tx.sign([user_priv, user2_priv])
 
     assert tx.fulfillments[0].to_dict()['fulfillment'] == \
         expected.fulfillment.serialize_uri()
     assert tx.fulfillments_valid() is True
 
+    validate_transaction_model(tx)
+
 
 def test_multiple_fulfillment_validation_of_transfer_tx(user_ffill, user_cond,
                                                         user_priv, user2_pub,
@@ -681,6 +676,7 @@ def test_multiple_fulfillment_validation_of_transfer_tx(user_ffill, user_cond,
     from bigchaindb.common.transaction import (Transaction, TransactionLink,
                                                Fulfillment, Condition, Asset)
     from cryptoconditions import Ed25519Fulfillment
+    from .util import validate_transaction_model
 
     tx = Transaction(Transaction.CREATE, Asset(divisible=True),
                      [user_ffill, deepcopy(user_ffill)],
@@ -699,6 +695,8 @@ def test_multiple_fulfillment_validation_of_transfer_tx(user_ffill, user_cond,
 
     assert transfer_tx.fulfillments_valid(tx.conditions) is True
 
+    validate_transaction_model(tx)
+
 
 def test_validate_fulfillments_of_transfer_tx_with_invalid_params(transfer_tx,
                                                                   cond_uri,
@@ -725,47 +723,43 @@ def test_validate_fulfillments_of_transfer_tx_with_invalid_params(transfer_tx,
         transfer_tx.fulfillments_valid([utx.conditions[0]])
 
 
-def test_create_create_transaction_single_io(user_cond, user_pub, data,
-                                             data_id):
+def test_create_create_transaction_single_io(user_cond, user_pub, data, uuid4):
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     expected = {
-        'transaction': {
-            'conditions': [user_cond.to_dict(0)],
-            'metadata': {
-                'data': data,
-            },
-            'asset': {
-                'id': data_id,
-                'divisible': False,
-                'updatable': False,
-                'refillable': False,
-                'data': data,
-            },
-            'fulfillments': [
-                {
-                    'owners_before': [
-                        user_pub
-                    ],
-                    'fid': 0,
-                    'fulfillment': None,
-                    'input': None
-                }
-            ],
-            'operation': 'CREATE',
+        'conditions': [user_cond.to_dict(0)],
+        'metadata': data,
+        'asset': {
+            'id': uuid4,
+            'divisible': False,
+            'updatable': False,
+            'refillable': False,
+            'data': data,
         },
-        'version': 1
+        'fulfillments': [
+            {
+                'owners_before': [
+                    user_pub
+                ],
+                'fid': 0,
+                'fulfillment': None,
+                'input': None
+            }
+        ],
+        'operation': 'CREATE',
+        'version': 1,
     }
 
-    asset = Asset(data, data_id)
-    tx = Transaction.create([user_pub], [([user_pub], 1)],
-                            data, asset).to_dict()
-    tx.pop('id')
-    tx['transaction']['metadata'].pop('id')
-    tx['transaction'].pop('timestamp')
-    tx['transaction']['fulfillments'][0]['fulfillment'] = None
+    asset = Asset(data, uuid4)
+    tx = Transaction.create([user_pub], [([user_pub], 1)], data, asset)
+    tx_dict = tx.to_dict()
+    tx_dict['fulfillments'][0]['fulfillment'] = None
+    tx_dict.pop('id')
 
-    assert tx == expected
+    assert tx_dict == expected
+
+    validate_transaction_model(tx)
 
 
 def test_validate_single_io_create_transaction(user_pub, user_priv, data):
@@ -786,16 +780,12 @@ def test_create_create_transaction_multiple_io(user_cond, user2_cond, user_pub,
     ffill = Fulfillment.generate([user_pub, user2_pub]).to_dict()
     ffill.update({'fid': 0})
     expected = {
-        'transaction': {
-            'conditions': [user_cond.to_dict(0), user2_cond.to_dict(1)],
-            'metadata': {
-                'data': {
-                    'message': 'hello'
-                }
-            },
-            'fulfillments': [ffill],
-            'operation': 'CREATE',
+        'conditions': [user_cond.to_dict(0), user2_cond.to_dict(1)],
+        'metadata': {
+            'message': 'hello'
         },
+        'fulfillments': [ffill],
+        'operation': 'CREATE',
         'version': 1
     }
     asset = Asset(divisible=True)
@@ -804,9 +794,7 @@ def test_create_create_transaction_multiple_io(user_cond, user2_cond, user_pub,
                             asset=asset,
                             metadata={'message': 'hello'}).to_dict()
     tx.pop('id')
-    tx['transaction']['metadata'].pop('id')
-    tx['transaction'].pop('timestamp')
-    tx['transaction'].pop('asset')
+    tx.pop('asset')
 
     assert tx == expected
 
@@ -814,6 +802,7 @@ def test_create_create_transaction_multiple_io(user_cond, user2_cond, user_pub,
 def test_validate_multiple_io_create_transaction(user_pub, user_priv,
                                                  user2_pub, user2_priv):
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     tx = Transaction.create([user_pub, user2_pub],
                             [([user_pub], 1), ([user2_pub], 1)],
@@ -822,48 +811,44 @@ def test_validate_multiple_io_create_transaction(user_pub, user_priv,
     tx = tx.sign([user_priv, user2_priv])
     assert tx.fulfillments_valid() is True
 
+    validate_transaction_model(tx)
+
 
 def test_create_create_transaction_threshold(user_pub, user2_pub, user3_pub,
                                              user_user2_threshold_cond,
                                              user_user2_threshold_ffill, data,
-                                             data_id):
+                                             uuid4):
     from bigchaindb.common.transaction import Transaction, Asset
 
     expected = {
-        'transaction': {
-            'conditions': [user_user2_threshold_cond.to_dict(0)],
-            'metadata': {
-                'data': data,
-            },
-            'asset': {
-                'id': data_id,
-                'divisible': False,
-                'updatable': False,
-                'refillable': False,
-                'data': data,
-            },
-            'fulfillments': [
-                {
-                    'owners_before': [
-                        user_pub,
-                    ],
-                    'fid': 0,
-                    'fulfillment': None,
-                    'input': None
-                },
-            ],
-            'operation': 'CREATE',
+        'conditions': [user_user2_threshold_cond.to_dict(0)],
+        'metadata': data,
+        'asset': {
+            'id': uuid4,
+            'divisible': False,
+            'updatable': False,
+            'refillable': False,
+            'data': data,
         },
+        'fulfillments': [
+            {
+                'owners_before': [
+                    user_pub,
+                ],
+                'fid': 0,
+                'fulfillment': None,
+                'input': None
+            },
+        ],
+        'operation': 'CREATE',
         'version': 1
     }
-    asset = Asset(data, data_id)
+    asset = Asset(data, uuid4)
     tx = Transaction.create([user_pub], [([user_pub, user2_pub], 1)],
                             data, asset)
     tx_dict = tx.to_dict()
     tx_dict.pop('id')
-    tx_dict['transaction']['metadata'].pop('id')
-    tx_dict['transaction'].pop('timestamp')
-    tx_dict['transaction']['fulfillments'][0]['fulfillment'] = None
+    tx_dict['fulfillments'][0]['fulfillment'] = None
 
     assert tx_dict == expected
 
@@ -871,12 +856,15 @@ def test_create_create_transaction_threshold(user_pub, user2_pub, user3_pub,
 def test_validate_threshold_create_transaction(user_pub, user_priv, user2_pub,
                                                data):
     from bigchaindb.common.transaction import Transaction, Asset
+    from .util import validate_transaction_model
 
     tx = Transaction.create([user_pub], [([user_pub, user2_pub], 1)],
                             data, Asset())
     tx = tx.sign([user_priv])
     assert tx.fulfillments_valid() is True
 
+    validate_transaction_model(tx)
+
 
 def test_create_create_transaction_with_invalid_parameters(user_pub):
     from bigchaindb.common.transaction import Transaction
@@ -906,56 +894,55 @@ def test_conditions_to_inputs(tx):
 
 
 def test_create_transfer_transaction_single_io(tx, user_pub, user2_pub,
-                                               user2_cond, user_priv, data_id):
+                                               user2_cond, user_priv, uuid4):
     from copy import deepcopy
-    from bigchaindb.common.crypto import SigningKey
+    from bigchaindb.common.crypto import PrivateKey
     from bigchaindb.common.transaction import Transaction, Asset
     from bigchaindb.common.util import serialize
+    from .util import validate_transaction_model
 
     expected = {
-        'transaction': {
-            'conditions': [user2_cond.to_dict(0)],
-            'metadata': None,
-            'asset': {
-                'id': data_id,
-            },
-            'fulfillments': [
-                {
-                    'owners_before': [
-                        user_pub
-                    ],
-                    'fid': 0,
-                    'fulfillment': None,
-                    'input': {
-                        'txid': tx.id,
-                        'cid': 0
-                    }
-                }
-            ],
-            'operation': 'TRANSFER',
+        'conditions': [user2_cond.to_dict(0)],
+        'metadata': None,
+        'asset': {
+            'id': uuid4,
         },
+        'fulfillments': [
+            {
+                'owners_before': [
+                    user_pub
+                ],
+                'fid': 0,
+                'fulfillment': None,
+                'input': {
+                    'txid': tx.id,
+                    'cid': 0
+                }
+            }
+        ],
+        'operation': 'TRANSFER',
         'version': 1
     }
     inputs = tx.to_inputs([0])
-    asset = Asset(None, data_id)
+    asset = Asset(None, uuid4)
     transfer_tx = Transaction.transfer(inputs, [([user2_pub], 1)], asset=asset)
     transfer_tx = transfer_tx.sign([user_priv])
     transfer_tx = transfer_tx.to_dict()
-    transfer_tx_body = transfer_tx['transaction']
 
     expected_input = deepcopy(inputs[0])
     expected['id'] = transfer_tx['id']
-    expected['transaction']['timestamp'] = transfer_tx_body['timestamp']
     expected_input.fulfillment.sign(serialize(expected).encode(),
-                                    SigningKey(user_priv))
+                                    PrivateKey(user_priv))
     expected_ffill = expected_input.fulfillment.serialize_uri()
-    transfer_ffill = transfer_tx_body['fulfillments'][0]['fulfillment']
+    transfer_ffill = transfer_tx['fulfillments'][0]['fulfillment']
 
     assert transfer_ffill == expected_ffill
 
     transfer_tx = Transaction.from_dict(transfer_tx)
     assert transfer_tx.fulfillments_valid([tx.conditions[0]]) is True
 
+    validate_transaction_model(transfer_tx)
+
 
 def test_create_transfer_transaction_multiple_io(user_pub, user_priv,
                                                  user2_pub, user2_priv,
@@ -968,34 +955,32 @@ def test_create_transfer_transaction_multiple_io(user_pub, user_priv,
     tx = tx.sign([user_priv])
 
     expected = {
-        'transaction': {
-            'conditions': [user2_cond.to_dict(0), user2_cond.to_dict(1)],
-            'metadata': None,
-            'fulfillments': [
-                {
-                    'owners_before': [
-                        user_pub
-                    ],
-                    'fid': 0,
-                    'fulfillment': None,
-                    'input': {
-                        'txid': tx.id,
-                        'cid': 0
-                    }
-                }, {
-                    'owners_before': [
-                        user2_pub
-                    ],
-                    'fid': 1,
-                    'fulfillment': None,
-                    'input': {
-                        'txid': tx.id,
-                        'cid': 1
-                    }
+        'conditions': [user2_cond.to_dict(0), user2_cond.to_dict(1)],
+        'metadata': None,
+        'fulfillments': [
+            {
+                'owners_before': [
+                    user_pub
+                ],
+                'fid': 0,
+                'fulfillment': None,
+                'input': {
+                    'txid': tx.id,
+                    'cid': 0
                 }
-            ],
-            'operation': 'TRANSFER',
-        },
+            }, {
+                'owners_before': [
+                    user2_pub
+                ],
+                'fid': 1,
+                'fulfillment': None,
+                'input': {
+                    'txid': tx.id,
+                    'cid': 1
+                }
+            }
+        ],
+        'operation': 'TRANSFER',
         'version': 1
     }
 
@@ -1010,11 +995,10 @@ def test_create_transfer_transaction_multiple_io(user_pub, user_priv,
     assert transfer_tx.fulfillments_valid(tx.conditions) is True
 
     transfer_tx = transfer_tx.to_dict()
-    transfer_tx['transaction']['fulfillments'][0]['fulfillment'] = None
-    transfer_tx['transaction']['fulfillments'][1]['fulfillment'] = None
-    transfer_tx['transaction'].pop('timestamp')
+    transfer_tx['fulfillments'][0]['fulfillment'] = None
+    transfer_tx['fulfillments'][1]['fulfillment'] = None
+    transfer_tx.pop('asset')
     transfer_tx.pop('id')
-    transfer_tx['transaction'].pop('asset')
 
     assert expected == transfer_tx
 

tests/common/util.py

@@ -0,0 +1,9 @@
+def validate_transaction_model(tx):
+    from bigchaindb.common.transaction import Transaction
+    from bigchaindb.common.schema import validate_transaction_schema
+
+    tx_dict = tx.to_dict()
+    # Check that a transaction is valid by re-serializing it
+    # And calling validate_transaction_schema
+    validate_transaction_schema(tx_dict)
+    Transaction.from_dict(tx_dict)

tests/conftest.py

@@ -25,8 +25,8 @@ CONFIG = {
 }
 
 # Test user. inputs will be created for this user. Cryptography Keys
-USER_SIGNING_KEY = '8eJ8q9ZQpReWyQT5aFCiwtZ5wDZC4eDnCen88p3tQ6ie'
-USER_VERIFYING_KEY = 'JEAkEJqLbbgDRAtMm8YAjGp759Aq2qTn9eaEHUj2XePE'
+USER_PRIVATE_KEY = '8eJ8q9ZQpReWyQT5aFCiwtZ5wDZC4eDnCen88p3tQ6ie'
+USER_PUBLIC_KEY = 'JEAkEJqLbbgDRAtMm8YAjGp759Aq2qTn9eaEHUj2XePE'
 
 
 # We need this function to avoid loading an existing
@@ -54,12 +54,12 @@ def node_config():
 
 @pytest.fixture
 def user_sk():
-    return USER_SIGNING_KEY
+    return USER_PRIVATE_KEY
 
 
 @pytest.fixture
-def user_vk():
-    return USER_VERIFYING_KEY
+def user_pk():
+    return USER_PUBLIC_KEY
 
 
 @pytest.fixture
@@ -70,9 +70,9 @@ def b(request, node_config):
 
 
 @pytest.fixture
-def create_tx(b, user_vk):
+def create_tx(b, user_pk):
     from bigchaindb.models import Transaction
-    return Transaction.create([b.me], [([user_vk], 1)])
+    return Transaction.create([b.me], [([user_pk], 1)])
 
 
 @pytest.fixture
@@ -81,8 +81,8 @@ def signed_create_tx(b, create_tx):
 
 
 @pytest.fixture
-def signed_transfer_tx(signed_create_tx, user_vk, user_sk):
+def signed_transfer_tx(signed_create_tx, user_pk, user_sk):
     from bigchaindb.models import Transaction
     inputs = signed_create_tx.to_inputs()
-    tx = Transaction.transfer(inputs, [([user_vk], 1)], signed_create_tx.asset)
+    tx = Transaction.transfer(inputs, [([user_pk], 1)], signed_create_tx.asset)
     return tx.sign([user_sk])

tests/db/conftest.py

@@ -14,7 +14,7 @@ from bigchaindb.db import get_conn, init_database
 from bigchaindb.common import crypto
 from bigchaindb.common.exceptions import DatabaseAlreadyExists
 
-USER2_SK, USER2_VK = crypto.generate_key_pair()
+USER2_SK, USER2_PK = crypto.generate_key_pair()
 
 
 @pytest.fixture(autouse=True)
@@ -70,7 +70,7 @@ def cleanup_tables(request, node_config):
 
 
 @pytest.fixture
-def inputs(user_vk):
+def inputs(user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.exceptions import GenesisBlockAlreadyExistsError
     # 1. create the genesis block
@@ -84,7 +84,7 @@ def inputs(user_vk):
     prev_block_id = g.id
     for block in range(4):
         transactions = [
-            Transaction.create([b.me], [([user_vk], 1)]).sign([b.me_private])
+            Transaction.create([b.me], [([user_pk], 1)]).sign([b.me_private])
             for i in range(10)
         ]
         block = b.create_block(transactions)
@@ -102,12 +102,12 @@ def user2_sk():
 
 
 @pytest.fixture
-def user2_vk():
-    return USER2_VK
+def user2_pk():
+    return USER2_PK
 
 
 @pytest.fixture
-def inputs_shared(user_vk, user2_vk):
+def inputs_shared(user_pk, user2_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.common.exceptions import GenesisBlockAlreadyExistsError
     # 1. create the genesis block
@@ -122,7 +122,7 @@ def inputs_shared(user_vk, user2_vk):
     for block in range(4):
         transactions = [
             Transaction.create(
-                [b.me], [user_vk, user2_vk], payload={'i': i}).sign([b.me_private])
+                [b.me], [user_pk, user2_pk], payload={'i': i}).sign([b.me_private])
             for i in range(10)
         ]
         block = b.create_block(transactions)

tests/db/test_bigchain_api.py

@@ -30,7 +30,7 @@ def dummy_block():
 
 class TestBigchainApi(object):
     def test_get_last_voted_block_cyclic_blockchain(self, b, monkeypatch):
-        from bigchaindb.common.crypto import SigningKey
+        from bigchaindb.common.crypto import PrivateKey
         from bigchaindb.common.exceptions import CyclicBlockchainError
         from bigchaindb.common.util import serialize
         from bigchaindb.models import Transaction
@@ -47,7 +47,7 @@ class TestBigchainApi(object):
         vote = b.vote(block1.id, b.get_last_voted_block().id, True)
         vote['vote']['previous_block'] = block1.id
         vote_data = serialize(vote['vote'])
-        vote['signature'] = SigningKey(b.me_private).sign(vote_data.encode())
+        vote['signature'] = PrivateKey(b.me_private).sign(vote_data.encode())
         b.write_vote(vote)
 
         with pytest.raises(CyclicBlockchainError):
@@ -88,11 +88,6 @@ class TestBigchainApi(object):
 
         assert b.has_previous_vote(block.id, block.voters) is True
 
-
-    def test_get_transactions_for_metadata_mismatch(self, b):
-        matches = b.get_tx_by_metadata_id('missing')
-        assert not matches
-
     def test_get_spent_with_double_spend(self, b, monkeypatch):
         from bigchaindb.common.exceptions import DoubleSpend
         from bigchaindb.models import Transaction
@@ -183,31 +178,14 @@ class TestBigchainApi(object):
         assert b.get_transaction(tx1.id) is None
         assert b.get_transaction(tx2.id) == tx2
 
-    def test_get_transactions_for_metadata(self, b, user_vk):
-        from bigchaindb.models import Transaction
-
-        metadata = {'msg': 'Hello BigchainDB!'}
-        tx = Transaction.create([b.me], [([user_vk], 1)], metadata=metadata)
-
-        block = b.create_block([tx])
-        b.write_block(block, durability='hard')
-
-        matches = b.get_tx_by_payload_uuid(tx.metadata.data_id)
-        assert len(matches) == 1
-        assert matches[0].id == tx.id
-
-    def test_get_transactions_for_metadata(self, b, user_vk):
-        matches = b.get_tx_by_metadata_id('missing')
-        assert not matches
-
     @pytest.mark.usefixtures('inputs')
-    def test_write_transaction(self, b, user_vk, user_sk):
+    def test_write_transaction(self, b, user_pk, user_sk):
         from bigchaindb.models import Transaction
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_vk], 1)], input_tx.asset)
+        tx = Transaction.transfer(inputs, [([user_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
         response = b.write_transaction(tx)
 
@@ -219,13 +197,13 @@ class TestBigchainApi(object):
         assert response['inserted'] == 1
 
     @pytest.mark.usefixtures('inputs')
-    def test_read_transaction(self, b, user_vk, user_sk):
+    def test_read_transaction(self, b, user_pk, user_sk):
         from bigchaindb.models import Transaction
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_vk], 1)], input_tx.asset)
+        tx = Transaction.transfer(inputs, [([user_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
         b.write_transaction(tx)
 
@@ -239,13 +217,13 @@ class TestBigchainApi(object):
         assert status == b.TX_UNDECIDED
 
     @pytest.mark.usefixtures('inputs')
-    def test_read_transaction_invalid_block(self, b, user_vk, user_sk):
+    def test_read_transaction_invalid_block(self, b, user_pk, user_sk):
         from bigchaindb.models import Transaction
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_vk], 1)], input_tx.asset)
+        tx = Transaction.transfer(inputs, [([user_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
         # There's no need to b.write_transaction(tx) to the backlog
 
@@ -263,13 +241,13 @@ class TestBigchainApi(object):
         assert response is None
 
     @pytest.mark.usefixtures('inputs')
-    def test_read_transaction_invalid_block_and_backlog(self, b, user_vk, user_sk):
+    def test_read_transaction_invalid_block_and_backlog(self, b, user_pk, user_sk):
         from bigchaindb.models import Transaction
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_vk], 1)], input_tx.asset)
+        tx = Transaction.transfer(inputs, [([user_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
 
         # Make sure there's a copy of tx in the backlog
@@ -292,44 +270,24 @@ class TestBigchainApi(object):
 
     @pytest.mark.usefixtures('inputs')
     def test_genesis_block(self, b):
-        import rethinkdb as r
-        from bigchaindb.util import is_genesis_block
-        from bigchaindb.db.utils import get_conn
+        block = b.backend.get_genesis_block()
 
-        response = list(r.table('bigchain')
-                        .filter(is_genesis_block)
-                        .run(get_conn()))
-
-        assert len(response) == 1
-        block = response[0]
         assert len(block['block']['transactions']) == 1
-        assert block['block']['transactions'][0]['transaction']['operation'] == 'GENESIS'
-        assert block['block']['transactions'][0]['transaction']['fulfillments'][0]['input'] is None
+        assert block['block']['transactions'][0]['operation'] == 'GENESIS'
+        assert block['block']['transactions'][0]['fulfillments'][0]['input'] is None
 
     def test_create_genesis_block_fails_if_table_not_empty(self, b):
-        import rethinkdb as r
         from bigchaindb.common.exceptions import GenesisBlockAlreadyExistsError
-        from bigchaindb.util import is_genesis_block
-        from bigchaindb.db.utils import get_conn
 
         b.create_genesis_block()
 
         with pytest.raises(GenesisBlockAlreadyExistsError):
             b.create_genesis_block()
 
-        genesis_blocks = list(r.table('bigchain')
-                              .filter(is_genesis_block)
-                              .run(get_conn()))
-
-        assert len(genesis_blocks) == 1
-
     @pytest.mark.skipif(reason='This test may not make sense after changing the chainification mode')
     def test_get_last_block(self, b):
-        import rethinkdb as r
-        from bigchaindb.db.utils import get_conn
-
         # get the number of blocks
-        num_blocks = r.table('bigchain').count().run(get_conn())
+        num_blocks = b.backend.count_blocks()
 
         # get the last block
         last_block = b.get_last_block()
@@ -381,15 +339,10 @@ class TestBigchainApi(object):
         assert status == b.BLOCK_UNDECIDED
 
     def test_get_last_voted_block_returns_genesis_if_no_votes_has_been_casted(self, b):
-        import rethinkdb as r
-        from bigchaindb import util
         from bigchaindb.models import Block
-        from bigchaindb.db.utils import get_conn
 
         b.create_genesis_block()
-        genesis = list(r.table('bigchain')
-                       .filter(util.is_genesis_block)
-                       .run(get_conn()))[0]
+        genesis = b.backend.get_genesis_block()
         genesis = Block.from_dict(genesis)
         gb = b.get_last_voted_block()
         assert gb == genesis
@@ -452,29 +405,25 @@ class TestBigchainApi(object):
         assert b.get_last_voted_block().id == block_3.id
 
     def test_no_vote_written_if_block_already_has_vote(self, b):
-        import rethinkdb as r
         from bigchaindb.models import Block
-        from bigchaindb.db.utils import get_conn
 
         genesis = b.create_genesis_block()
         block_1 = dummy_block()
         b.write_block(block_1, durability='hard')
 
         b.write_vote(b.vote(block_1.id, genesis.id, True))
-        retrieved_block_1 = r.table('bigchain').get(block_1.id).run(get_conn())
+        retrieved_block_1 = b.get_block(block_1.id)
         retrieved_block_1 = Block.from_dict(retrieved_block_1)
 
         # try to vote again on the retrieved block, should do nothing
         b.write_vote(b.vote(retrieved_block_1.id, genesis.id, True))
-        retrieved_block_2 = r.table('bigchain').get(block_1.id).run(get_conn())
+        retrieved_block_2 = b.get_block(block_1.id)
         retrieved_block_2 = Block.from_dict(retrieved_block_2)
 
         assert retrieved_block_1 == retrieved_block_2
 
     def test_more_votes_than_voters(self, b):
-        import rethinkdb as r
         from bigchaindb.common.exceptions import MultipleVotesError
-        from bigchaindb.db.utils import get_conn
 
         b.create_genesis_block()
         block_1 = dummy_block()
@@ -483,8 +432,8 @@ class TestBigchainApi(object):
         vote_1 = b.vote(block_1.id, b.get_last_voted_block().id, True)
         vote_2 = b.vote(block_1.id, b.get_last_voted_block().id, True)
         vote_2['node_pubkey'] = 'aaaaaaa'
-        r.table('votes').insert(vote_1).run(get_conn())
-        r.table('votes').insert(vote_2).run(get_conn())
+        b.write_vote(vote_1)
+        b.write_vote(vote_2)
 
         with pytest.raises(MultipleVotesError) as excinfo:
             b.block_election_status(block_1.id, block_1.voters)
@@ -492,16 +441,14 @@ class TestBigchainApi(object):
             .format(block_id=block_1.id, n_votes=str(2), n_voters=str(1))
 
     def test_multiple_votes_single_node(self, b):
-        import rethinkdb as r
         from bigchaindb.common.exceptions import MultipleVotesError
-        from bigchaindb.db.utils import get_conn
 
         genesis = b.create_genesis_block()
         block_1 = dummy_block()
         b.write_block(block_1, durability='hard')
         # insert duplicate votes
         for i in range(2):
-            r.table('votes').insert(b.vote(block_1.id, genesis.id, True)).run(get_conn())
+            b.write_vote(b.vote(block_1.id, genesis.id, True))
 
         with pytest.raises(MultipleVotesError) as excinfo:
             b.block_election_status(block_1.id, block_1.voters)
@@ -514,9 +461,7 @@ class TestBigchainApi(object):
             .format(block_id=block_1.id, n_votes=str(2), me=b.me)
 
     def test_improper_vote_error(selfs, b):
-        import rethinkdb as r
         from bigchaindb.common.exceptions import ImproperVoteError
-        from bigchaindb.db.utils import get_conn
 
         b.create_genesis_block()
         block_1 = dummy_block()
@@ -524,37 +469,33 @@ class TestBigchainApi(object):
         vote_1 = b.vote(block_1.id, b.get_last_voted_block().id, True)
         # mangle the signature
         vote_1['signature'] = 'a' * 87
-        r.table('votes').insert(vote_1).run(get_conn())
+        b.write_vote(vote_1)
         with pytest.raises(ImproperVoteError) as excinfo:
             b.has_previous_vote(block_1.id, block_1.id)
         assert excinfo.value.args[0] == 'Block {block_id} already has an incorrectly signed ' \
                                         'vote from public key {me}'.format(block_id=block_1.id, me=b.me)
 
     @pytest.mark.usefixtures('inputs')
-    def test_assign_transaction_one_node(self, b, user_vk, user_sk):
-        import rethinkdb as r
+    def test_assign_transaction_one_node(self, b, user_pk, user_sk):
         from bigchaindb.models import Transaction
-        from bigchaindb.db.utils import get_conn
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user_vk], 1)], input_tx.asset)
+        tx = Transaction.transfer(inputs, [([user_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
         b.write_transaction(tx)
 
         # retrieve the transaction
-        response = r.table('backlog').get(tx.id).run(get_conn())
+        response = list(b.backend.get_stale_transactions(0))[0]
 
         # check if the assignee is the current node
         assert response['assignee'] == b.me
 
     @pytest.mark.usefixtures('inputs')
-    def test_assign_transaction_multiple_nodes(self, b, user_vk, user_sk):
-        import rethinkdb as r
+    def test_assign_transaction_multiple_nodes(self, b, user_pk, user_sk):
         from bigchaindb.common.crypto import generate_key_pair
         from bigchaindb.models import Transaction
-        from bigchaindb.db.utils import get_conn
 
         # create 5 federation nodes
         for _ in range(5):
@@ -562,22 +503,23 @@ class TestBigchainApi(object):
 
         # test assignee for several transactions
         for _ in range(20):
-            input_tx = b.get_owned_ids(user_vk).pop()
+            input_tx = b.get_owned_ids(user_pk).pop()
             input_tx = b.get_transaction(input_tx.txid)
             inputs = input_tx.to_inputs()
-            tx = Transaction.transfer(inputs, [([user_vk], 1)], input_tx.asset)
+            tx = Transaction.transfer(inputs, [([user_pk], 1)], input_tx.asset)
             tx = tx.sign([user_sk])
             b.write_transaction(tx)
 
-            # retrieve the transaction
-            response = r.table('backlog').get(tx.id).run(get_conn())
+        # retrieve the transaction
+        response = b.backend.get_stale_transactions(0)
 
-            # check if the assignee is one of the _other_ federation nodes
-            assert response['assignee'] in b.nodes_except_me
+        # check if the assignee is one of the _other_ federation nodes
+        for tx in response:
+            assert tx['assignee'] in b.nodes_except_me
 
 
     @pytest.mark.usefixtures('inputs')
-    def test_non_create_input_not_found(self, b, user_vk):
+    def test_non_create_input_not_found(self, b, user_pk):
         from cryptoconditions import Ed25519Fulfillment
         from bigchaindb.common.exceptions import TransactionDoesNotExist
         from bigchaindb.common.transaction import (Fulfillment, Asset,
@@ -586,27 +528,27 @@ class TestBigchainApi(object):
         from bigchaindb import Bigchain
 
         # Create a fulfillment for a non existing transaction
-        fulfillment = Fulfillment(Ed25519Fulfillment(public_key=user_vk),
-                                  [user_vk],
+        fulfillment = Fulfillment(Ed25519Fulfillment(public_key=user_pk),
+                                  [user_pk],
                                   TransactionLink('somethingsomething', 0))
-        tx = Transaction.transfer([fulfillment], [([user_vk], 1)], Asset())
+        tx = Transaction.transfer([fulfillment], [([user_pk], 1)], Asset())
 
         with pytest.raises(TransactionDoesNotExist):
             tx.validate(Bigchain())
 
-    def test_count_backlog(self, b, user_vk):
+    def test_count_backlog(self, b, user_pk):
         from bigchaindb.models import Transaction
 
         for _ in range(4):
             tx = Transaction.create([b.me],
-                                    [([user_vk], 1)]).sign([b.me_private])
+                                    [([user_pk], 1)]).sign([b.me_private])
             b.write_transaction(tx)
 
         assert b.backend.count_backlog() == 4
 
 
 class TestTransactionValidation(object):
-    def test_create_operation_with_inputs(self, b, user_vk, create_tx):
+    def test_create_operation_with_inputs(self, b, user_pk, create_tx):
         from bigchaindb.common.transaction import TransactionLink
 
         # Manipulate fulfillment so that it has a `tx_input` defined even
@@ -616,7 +558,7 @@ class TestTransactionValidation(object):
             b.validate_transaction(create_tx)
         assert excinfo.value.args[0] == 'A CREATE operation has no inputs'
 
-    def test_transfer_operation_no_inputs(self, b, user_vk,
+    def test_transfer_operation_no_inputs(self, b, user_pk,
                                           signed_transfer_tx):
         signed_transfer_tx.fulfillments[0].tx_input = None
         with pytest.raises(ValueError) as excinfo:
@@ -624,7 +566,7 @@ class TestTransactionValidation(object):
 
         assert excinfo.value.args[0] == 'Only `CREATE` transactions can have null inputs'
 
-    def test_non_create_input_not_found(self, b, user_vk, signed_transfer_tx):
+    def test_non_create_input_not_found(self, b, user_pk, signed_transfer_tx):
         from bigchaindb.common.exceptions import TransactionDoesNotExist
         from bigchaindb.common.transaction import TransactionLink
 
@@ -633,15 +575,15 @@ class TestTransactionValidation(object):
             b.validate_transaction(signed_transfer_tx)
 
     @pytest.mark.usefixtures('inputs')
-    def test_non_create_valid_input_wrong_owner(self, b, user_vk):
+    def test_non_create_valid_input_wrong_owner(self, b, user_pk):
         from bigchaindb.common.crypto import generate_key_pair
         from bigchaindb.common.exceptions import InvalidSignature
         from bigchaindb.models import Transaction
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_transaction = b.get_transaction(input_tx.txid)
-        sk, vk = generate_key_pair()
-        tx = Transaction.create([vk], [([user_vk], 1)])
+        sk, pk = generate_key_pair()
+        tx = Transaction.create([pk], [([user_pk], 1)])
         tx.operation = 'TRANSFER'
         tx.asset = input_transaction.asset
         tx.fulfillments[0].tx_input = input_tx
@@ -671,21 +613,21 @@ class TestTransactionValidation(object):
 
         sleep(1)
 
-        signed_transfer_tx.timestamp = 123
+        signed_transfer_tx.metadata = {'different': 1}
         # FIXME: https://github.com/bigchaindb/bigchaindb/issues/592
         with pytest.raises(DoubleSpend):
             b.validate_transaction(signed_transfer_tx)
 
     @pytest.mark.usefixtures('inputs')
     def test_valid_non_create_transaction_after_block_creation(self, b,
-                                                               user_vk,
+                                                               user_pk,
                                                                user_sk):
         from bigchaindb.models import Transaction
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
-        transfer_tx = Transaction.transfer(inputs, [([user_vk], 1)],
+        transfer_tx = Transaction.transfer(inputs, [([user_pk], 1)],
                                            input_tx.asset)
         transfer_tx = transfer_tx.sign([user_sk])
 
@@ -701,16 +643,16 @@ class TestTransactionValidation(object):
         assert transfer_tx == b.validate_transaction(transfer_tx)
 
     @pytest.mark.usefixtures('inputs')
-    def test_transaction_not_in_valid_block(self, b, user_vk, user_sk):
+    def test_transaction_not_in_valid_block(self, b, user_pk, user_sk):
         from bigchaindb.models import Transaction
         from bigchaindb.common.exceptions import TransactionNotInValidBlock
 
-        input_tx = b.get_owned_ids(user_vk).pop()
+        input_tx = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(input_tx.txid)
         inputs = input_tx.to_inputs()
 
         # create a transaction that's valid but not in a voted valid block
-        transfer_tx = Transaction.transfer(inputs, [([user_vk], 1)],
+        transfer_tx = Transaction.transfer(inputs, [([user_pk], 1)],
                                            input_tx.asset)
         transfer_tx = transfer_tx.sign([user_sk])
 
@@ -722,7 +664,7 @@ class TestTransactionValidation(object):
 
         # create transaction with the undecided input
         tx_invalid = Transaction.transfer(transfer_tx.to_inputs(),
-                                          [([user_vk], 1)],
+                                          [([user_pk], 1)],
                                           transfer_tx.asset)
         tx_invalid = tx_invalid.sign([user_sk])
 
@@ -733,7 +675,7 @@ class TestTransactionValidation(object):
 class TestBlockValidation(object):
     @pytest.mark.skipif(reason='Separated tx validation from block creation.')
     @pytest.mark.usefixtures('inputs')
-    def test_invalid_transactions_in_block(self, b, user_vk):
+    def test_invalid_transactions_in_block(self, b, user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.common.exceptions import TransactionOwnerError
         from bigchaindb.common.util import gen_timestamp
@@ -741,7 +683,7 @@ class TestBlockValidation(object):
         from bigchaindb import util
 
         # invalid transaction
-        valid_input = b.get_owned_ids(user_vk).pop()
+        valid_input = b.get_owned_ids(user_pk).pop()
         tx_invalid = b.create_transaction('a', 'b', valid_input, 'c')
 
         block = b.create_block([tx_invalid])
@@ -758,7 +700,7 @@ class TestBlockValidation(object):
         #       skipped
         block_data = util.serialize_block(block)
         block_hash = crypto.hash_data(block_data)
-        block_signature = crypto.SigningKey(b.me_private).sign(block_data)
+        block_signature = crypto.PrivateKey(b.me_private).sign(block_data)
 
         block = {
             'id': block_hash,
@@ -782,7 +724,7 @@ class TestBlockValidation(object):
         block = dummy_block()
 
         # replace the block signature with an invalid one
-        block.signature = crypto.SigningKey(b.me_private).sign(b'wrongdata')
+        block.signature = crypto.PrivateKey(b.me_private).sign(b'wrongdata')
 
         # check that validate_block raises an InvalidSignature exception
         with pytest.raises(InvalidSignature):
@@ -797,10 +739,10 @@ class TestBlockValidation(object):
         block = dummy_block()
 
         # create some temp keys
-        tmp_sk, tmp_vk = crypto.generate_key_pair()
+        tmp_sk, tmp_pk = crypto.generate_key_pair()
 
         # change the block node_pubkey
-        block.node_pubkey = tmp_vk
+        block.node_pubkey = tmp_pk
 
         # just to make sure lets re-hash the block and create a valid signature
         # from a non federation node
@@ -812,16 +754,16 @@ class TestBlockValidation(object):
 
 
 class TestMultipleInputs(object):
-    def test_transfer_single_owner_single_input(self, b, inputs, user_vk,
+    def test_transfer_single_owner_single_input(self, b, inputs, user_pk,
                                                 user_sk):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
-        tx_link = b.get_owned_ids(user_vk).pop()
+        tx_link = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(tx_link.txid)
         inputs = input_tx.to_inputs()
-        tx = Transaction.transfer(inputs, [([user2_vk], 1)], input_tx.asset)
+        tx = Transaction.transfer(inputs, [([user2_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
 
         # validate transaction
@@ -831,19 +773,19 @@ class TestMultipleInputs(object):
 
     def test_single_owner_before_multiple_owners_after_single_input(self, b,
                                                                     user_sk,
-                                                                    user_vk,
+                                                                    user_pk,
                                                                     inputs):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
-        user3_sk, user3_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
+        user3_sk, user3_pk = crypto.generate_key_pair()
 
-        owned_inputs = b.get_owned_ids(user_vk)
+        owned_inputs = b.get_owned_ids(user_pk)
         tx_link = owned_inputs.pop()
         input_tx = b.get_transaction(tx_link.txid)
         tx = Transaction.transfer(input_tx.to_inputs(),
-                                  [([user2_vk, user3_vk], 1)], input_tx.asset)
+                                  [([user2_pk, user3_pk], 1)], input_tx.asset)
         tx = tx.sign([user_sk])
 
         assert b.is_valid_transaction(tx) == tx
@@ -853,14 +795,14 @@ class TestMultipleInputs(object):
     @pytest.mark.usefixtures('inputs')
     def test_multiple_owners_before_single_owner_after_single_input(self, b,
                                                                     user_sk,
-                                                                    user_vk):
+                                                                    user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
-        user3_sk, user3_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
+        user3_sk, user3_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk, user2_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk, user2_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
@@ -869,11 +811,11 @@ class TestMultipleInputs(object):
         vote = b.vote(block.id, b.get_last_voted_block().id, True)
         b.write_vote(vote)
 
-        owned_input = b.get_owned_ids(user_vk).pop()
+        owned_input = b.get_owned_ids(user_pk).pop()
         input_tx = b.get_transaction(owned_input.txid)
         inputs = input_tx.to_inputs()
 
-        transfer_tx = Transaction.transfer(inputs, [([user3_vk], 1)],
+        transfer_tx = Transaction.transfer(inputs, [([user3_pk], 1)],
                                            input_tx.asset)
         transfer_tx = transfer_tx.sign([user_sk, user2_sk])
 
@@ -885,15 +827,15 @@ class TestMultipleInputs(object):
     @pytest.mark.usefixtures('inputs')
     def test_multiple_owners_before_multiple_owners_after_single_input(self, b,
                                                                        user_sk,
-                                                                       user_vk):
+                                                                       user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
-        user3_sk, user3_vk = crypto.generate_key_pair()
-        user4_sk, user4_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
+        user3_sk, user3_pk = crypto.generate_key_pair()
+        user4_sk, user4_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk, user2_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk, user2_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
@@ -903,55 +845,55 @@ class TestMultipleInputs(object):
         b.write_vote(vote)
 
         # get input
-        tx_link = b.get_owned_ids(user_vk).pop()
+        tx_link = b.get_owned_ids(user_pk).pop()
         tx_input = b.get_transaction(tx_link.txid)
 
         tx = Transaction.transfer(tx_input.to_inputs(),
-                                  [([user3_vk, user4_vk], 1)], tx_input.asset)
+                                  [([user3_pk, user4_pk], 1)], tx_input.asset)
         tx = tx.sign([user_sk, user2_sk])
 
         assert b.is_valid_transaction(tx) == tx
         assert len(tx.fulfillments) == 1
         assert len(tx.conditions) == 1
 
-    def test_get_owned_ids_single_tx_single_output(self, b, user_sk, user_vk):
+    def test_get_owned_ids_single_tx_single_output(self, b, user_sk, user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.common.transaction import TransactionLink
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
         assert owned_inputs_user1 == [TransactionLink(tx.id, 0)]
         assert owned_inputs_user2 == []
 
-        tx = Transaction.transfer(tx.to_inputs(), [([user2_vk], 1)], tx.asset)
+        tx = Transaction.transfer(tx.to_inputs(), [([user2_pk], 1)], tx.asset)
         tx = tx.sign([user_sk])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
         assert owned_inputs_user1 == []
         assert owned_inputs_user2 == [TransactionLink(tx.id, 0)]
 
     def test_get_owned_ids_single_tx_single_output_invalid_block(self, b,
                                                                  user_sk,
-                                                                 user_vk):
+                                                                 user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.common.transaction import TransactionLink
         from bigchaindb.models import Transaction
 
         genesis = b.create_genesis_block()
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
@@ -960,14 +902,14 @@ class TestMultipleInputs(object):
         vote = b.vote(block.id, genesis.id, True)
         b.write_vote(vote)
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
         assert owned_inputs_user1 == [TransactionLink(tx.id, 0)]
         assert owned_inputs_user2 == []
 
         # NOTE: The transaction itself is valid, still will mark the block
         #       as invalid to mock the behavior.
-        tx_invalid = Transaction.transfer(tx.to_inputs(), [([user2_vk], 1)],
+        tx_invalid = Transaction.transfer(tx.to_inputs(), [([user2_pk], 1)],
                                           tx.asset)
         tx_invalid = tx_invalid.sign([user_sk])
         block = b.create_block([tx_invalid])
@@ -977,33 +919,33 @@ class TestMultipleInputs(object):
         vote = b.vote(block.id, b.get_last_voted_block().id, False)
         b.write_vote(vote)
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
 
         # should be the same as before (note tx, not tx_invalid)
         assert owned_inputs_user1 == [TransactionLink(tx.id, 0)]
         assert owned_inputs_user2 == []
 
     def test_get_owned_ids_single_tx_multiple_outputs(self, b, user_sk,
-                                                      user_vk):
+                                                      user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.common.transaction import TransactionLink, Asset
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
         # create divisible asset
         asset = Asset(divisible=True)
         tx_create = Transaction.create([b.me],
-                                       [([user_vk], 1), ([user_vk], 1)],
+                                       [([user_pk], 1), ([user_pk], 1)],
                                        asset=asset)
         tx_create_signed = tx_create.sign([b.me_private])
         block = b.create_block([tx_create_signed])
         b.write_block(block, durability='hard')
 
         # get input
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
 
         expected_owned_inputs_user1 = [TransactionLink(tx_create.id, 0),
                                        TransactionLink(tx_create.id, 1)]
@@ -1012,60 +954,60 @@ class TestMultipleInputs(object):
 
         # transfer divisible asset divided in two outputs
         tx_transfer = Transaction.transfer(tx_create.to_inputs(),
-                                           [([user2_vk], 1), ([user2_vk], 1)],
+                                           [([user2_pk], 1), ([user2_pk], 1)],
                                            asset=tx_create.asset)
         tx_transfer_signed = tx_transfer.sign([user_sk])
         block = b.create_block([tx_transfer_signed])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
         assert owned_inputs_user1 == []
         assert owned_inputs_user2 == [TransactionLink(tx_transfer.id, 0),
                                       TransactionLink(tx_transfer.id, 1)]
 
-    def test_get_owned_ids_multiple_owners(self, b, user_sk, user_vk):
+    def test_get_owned_ids_multiple_owners(self, b, user_sk, user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.common.transaction import TransactionLink
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
-        user3_sk, user3_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
+        user3_sk, user3_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk, user2_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk, user2_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
         expected_owned_inputs_user1 = [TransactionLink(tx.id, 0)]
 
         assert owned_inputs_user1 == owned_inputs_user2
         assert owned_inputs_user1 == expected_owned_inputs_user1
 
-        tx = Transaction.transfer(tx.to_inputs(), [([user3_vk], 1)], tx.asset)
+        tx = Transaction.transfer(tx.to_inputs(), [([user3_pk], 1)], tx.asset)
         tx = tx.sign([user_sk, user2_sk])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
-        owned_inputs_user2 = b.get_owned_ids(user2_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
+        owned_inputs_user2 = b.get_owned_ids(user2_pk)
         assert owned_inputs_user1 == owned_inputs_user2
         assert owned_inputs_user1 == []
 
-    def test_get_spent_single_tx_single_output(self, b, user_sk, user_vk):
+    def test_get_spent_single_tx_single_output(self, b, user_sk, user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk).pop()
+        owned_inputs_user1 = b.get_owned_ids(user_pk).pop()
 
         # check spents
         input_txid = owned_inputs_user1.txid
@@ -1074,7 +1016,7 @@ class TestMultipleInputs(object):
         assert spent_inputs_user1 is None
 
         # create a transaction and block
-        tx = Transaction.transfer(tx.to_inputs(), [([user2_vk], 1)], tx.asset)
+        tx = Transaction.transfer(tx.to_inputs(), [([user2_pk], 1)], tx.asset)
         tx = tx.sign([user_sk])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
@@ -1082,16 +1024,16 @@ class TestMultipleInputs(object):
         spent_inputs_user1 = b.get_spent(input_txid, input_cid)
         assert spent_inputs_user1 == tx
 
-    def test_get_spent_single_tx_single_output_invalid_block(self, b, user_sk, user_vk):
+    def test_get_spent_single_tx_single_output_invalid_block(self, b, user_sk, user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
 
         genesis = b.create_genesis_block()
 
         # create a new users
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
@@ -1100,7 +1042,7 @@ class TestMultipleInputs(object):
         vote = b.vote(block.id, genesis.id, True)
         b.write_vote(vote)
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk).pop()
+        owned_inputs_user1 = b.get_owned_ids(user_pk).pop()
 
         # check spents
         input_txid = owned_inputs_user1.txid
@@ -1109,7 +1051,7 @@ class TestMultipleInputs(object):
         assert spent_inputs_user1 is None
 
         # create a transaction and block
-        tx = Transaction.transfer(tx.to_inputs(), [([user2_vk], 1)], tx.asset)
+        tx = Transaction.transfer(tx.to_inputs(), [([user2_pk], 1)], tx.asset)
         tx = tx.sign([user_sk])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')
@@ -1124,26 +1066,26 @@ class TestMultipleInputs(object):
         # Now there should be no spents (the block is invalid)
         assert spent_inputs_user1 is None
 
-    def test_get_spent_single_tx_multiple_outputs(self, b, user_sk, user_vk):
+    def test_get_spent_single_tx_multiple_outputs(self, b, user_sk, user_pk):
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
         from bigchaindb.common.transaction import Asset
 
         # create a new users
-        user2_sk, user2_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
 
         # create a divisible asset with 3 outputs
         asset = Asset(divisible=True)
         tx_create = Transaction.create([b.me],
-                                       [([user_vk], 1),
-                                        ([user_vk], 1),
-                                        ([user_vk], 1)],
+                                       [([user_pk], 1),
+                                        ([user_pk], 1),
+                                        ([user_pk], 1)],
                                        asset=asset)
         tx_create_signed = tx_create.sign([b.me_private])
         block = b.create_block([tx_create_signed])
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
 
         # check spents
         for input_tx in owned_inputs_user1:
@@ -1151,7 +1093,7 @@ class TestMultipleInputs(object):
 
         # transfer the first 2 inputs
         tx_transfer = Transaction.transfer(tx_create.to_inputs()[:2],
-                                           [([user2_vk], 1), ([user2_vk], 1)],
+                                           [([user2_pk], 1), ([user2_pk], 1)],
                                            asset=tx_create.asset)
         tx_transfer_signed = tx_transfer.sign([user_sk])
         block = b.create_block([tx_transfer_signed])
@@ -1166,25 +1108,25 @@ class TestMultipleInputs(object):
         # spendable by BigchainDB
         assert b.get_spent(tx_create.to_inputs()[2].tx_input.txid, 2) is None
 
-    def test_get_spent_multiple_owners(self, b, user_sk, user_vk):
+    def test_get_spent_multiple_owners(self, b, user_sk, user_pk):
         import random
         from bigchaindb.common import crypto
         from bigchaindb.models import Transaction
 
-        user2_sk, user2_vk = crypto.generate_key_pair()
-        user3_sk, user3_vk = crypto.generate_key_pair()
+        user2_sk, user2_pk = crypto.generate_key_pair()
+        user3_sk, user3_pk = crypto.generate_key_pair()
 
         transactions = []
         for i in range(3):
             payload = {'somedata': random.randint(0, 255)}
-            tx = Transaction.create([b.me], [([user_vk, user2_vk], 1)],
+            tx = Transaction.create([b.me], [([user_pk, user2_pk], 1)],
                                     payload)
             tx = tx.sign([b.me_private])
             transactions.append(tx)
         block = b.create_block(transactions)
         b.write_block(block, durability='hard')
 
-        owned_inputs_user1 = b.get_owned_ids(user_vk)
+        owned_inputs_user1 = b.get_owned_ids(user_pk)
 
         # check spents
         for input_tx in owned_inputs_user1:
@@ -1192,7 +1134,7 @@ class TestMultipleInputs(object):
 
         # create a transaction
         tx = Transaction.transfer(transactions[0].to_inputs(),
-                                  [([user3_vk], 1)], transactions[0].asset)
+                                  [([user3_pk], 1)], transactions[0].asset)
         tx = tx.sign([user_sk, user2_sk])
         block = b.create_block([tx])
         b.write_block(block, durability='hard')

tests/db/test_utils.py

@@ -33,7 +33,6 @@ def test_init_creates_db_tables_and_indexes():
         'block_timestamp').run(conn) is True
 
     assert r.db(dbname).table('backlog').index_list().contains(
-        'transaction_timestamp',
         'assignee__transaction_timestamp').run(conn) is True
 
 
@@ -78,8 +77,6 @@ def test_create_bigchain_secondary_index():
         'block_timestamp').run(conn) is True
     assert r.db(dbname).table('bigchain').index_list().contains(
         'transaction_id').run(conn) is True
-    assert r.db(dbname).table('bigchain').index_list().contains(
-        'metadata_id').run(conn) is True
 
 
 def test_create_backlog_table():
@@ -108,8 +105,6 @@ def test_create_backlog_secondary_index():
     utils.create_table(conn, dbname, 'backlog')
     utils.create_backlog_secondary_index(conn, dbname)
 
-    assert r.db(dbname).table('backlog').index_list().contains(
-        'transaction_timestamp').run(conn) is True
     assert r.db(dbname).table('backlog').index_list().contains(
         'assignee__transaction_timestamp').run(conn) is True
 

tests/doc/run_doc_python_server_api_examples.py

@@ -1,492 +0,0 @@
-import json
-from time import sleep
-
-import cryptoconditions as cc
-from bigchaindb.common.util import gen_timestamp
-
-from bigchaindb import Bigchain, util, crypto, exceptions
-
-
-b = Bigchain()
-
-"""
-Create a Digital Asset
-"""
-
-# create a test user
-testuser1_priv, testuser1_pub = crypto.generate_key_pair()
-
-# define a digital asset data payload
-digital_asset_payload = {'msg': 'Hello BigchainDB!'}
-
-# a create transaction uses the operation `CREATE` and has no inputs
-tx = b.create_transaction(b.me, testuser1_pub, None, 'CREATE', payload=digital_asset_payload)
-
-# all transactions need to be signed by the user creating the transaction
-tx_signed = b.sign_transaction(tx, b.me_private)
-
-# write the transaction to the bigchain
-# the transaction will be stored in a backlog where it will be validated,
-# included in a block, and written to the bigchain
-b.write_transaction(tx_signed)
-
-sleep(8)
-
-"""
-Read the Creation Transaction from the DB
-"""
-
-tx_retrieved = b.get_transaction(tx_signed['id'])
-
-print(json.dumps(tx_retrieved, sort_keys=True, indent=4, separators=(',', ':')))
-
-print(testuser1_pub)
-print(b.me)
-
-print(tx_retrieved['id'])
-
-"""
-Transfer the Digital Asset
-"""
-
-# create a second testuser
-testuser2_priv, testuser2_pub = crypto.generate_key_pair()
-
-# retrieve the transaction with condition id
-tx_retrieved_id = b.get_owned_ids(testuser1_pub).pop()
-print(json.dumps(tx_retrieved_id, sort_keys=True, indent=4, separators=(',', ':')))
-
-# create a transfer transaction
-tx_transfer = b.create_transaction(testuser1_pub, testuser2_pub, tx_retrieved_id, 'TRANSFER')
-
-# sign the transaction
-tx_transfer_signed = b.sign_transaction(tx_transfer, testuser1_priv)
-
-
-b.validate_transaction(tx_transfer_signed)
-# write the transaction
-b.write_transaction(tx_transfer_signed)
-
-sleep(8)
-
-# check if the transaction is already in the bigchain
-tx_transfer_retrieved = b.get_transaction(tx_transfer_signed['id'])
-print(json.dumps(tx_transfer_retrieved, sort_keys=True, indent=4, separators=(',', ':')))
-
-"""
-Double Spends
-"""
-
-# create another transfer transaction with the same input
-tx_transfer2 = b.create_transaction(testuser1_pub, testuser2_pub, tx_retrieved_id, 'TRANSFER')
-
-# sign the transaction
-tx_transfer_signed2 = b.sign_transaction(tx_transfer2, testuser1_priv)
-
-# check if the transaction is valid
-try:
-    b.validate_transaction(tx_transfer_signed2)
-except exceptions.DoubleSpend as e:
-    print(e)
-
-"""
-Multiple Owners
-"""
-
-# Create a new asset and assign it to multiple owners
-tx_multisig = b.create_transaction(b.me, [testuser1_pub, testuser2_pub], None, 'CREATE')
-
-# Have the federation sign the transaction
-tx_multisig_signed = b.sign_transaction(tx_multisig, b.me_private)
-
-b.validate_transaction(tx_multisig_signed)
-b.write_transaction(tx_multisig_signed)
-
-# wait a few seconds for the asset to appear on the blockchain
-sleep(8)
-
-# retrieve the transaction
-tx_multisig_retrieved = b.get_transaction(tx_multisig_signed['id'])
-assert tx_multisig_retrieved is not None
-
-print(json.dumps(tx_multisig_retrieved, sort_keys=True, indent=4, separators=(',', ':')))
-
-testuser3_priv, testuser3_pub = crypto.generate_key_pair()
-
-tx_multisig_retrieved_id = b.get_owned_ids(testuser2_pub).pop()
-tx_multisig_transfer = b.create_transaction([testuser1_pub, testuser2_pub], testuser3_pub, tx_multisig_retrieved_id, 'TRANSFER')
-tx_multisig_transfer_signed = b.sign_transaction(tx_multisig_transfer, [testuser1_priv, testuser2_priv])
-
-try:
-    b.validate_transaction(tx_multisig_transfer_signed)
-except exceptions.InvalidSignature:
-    # import ipdb; ipdb.set_trace()
-    b.validate_transaction(tx_multisig_transfer_signed)
-b.write_transaction(tx_multisig_transfer_signed)
-
-# wait a few seconds for the asset to appear on the blockchain
-sleep(8)
-
-# retrieve the transaction
-tx_multisig_transfer_retrieved = b.get_transaction(tx_multisig_transfer_signed['id'])
-assert tx_multisig_transfer_retrieved is not None
-print(json.dumps(tx_multisig_transfer_retrieved, sort_keys=True, indent=4, separators=(',', ':')))
-
-"""
-Multiple Inputs and Outputs
-"""
-for i in range(3):
-    tx_mimo_asset = b.create_transaction(b.me, testuser1_pub, None, 'CREATE')
-    tx_mimo_asset_signed = b.sign_transaction(tx_mimo_asset, b.me_private)
-    b.validate_transaction(tx_mimo_asset_signed)
-    b.write_transaction(tx_mimo_asset_signed)
-
-sleep(8)
-
-# get inputs
-owned_mimo_inputs = b.get_owned_ids(testuser1_pub)
-print(len(owned_mimo_inputs))
-
-# create a transaction
-tx_mimo = b.create_transaction(testuser1_pub, testuser2_pub, owned_mimo_inputs, 'TRANSFER')
-
-tx_mimo_signed = b.sign_transaction(tx_mimo, testuser1_priv)
-# write the transaction
-b.validate_transaction(tx_mimo_signed)
-b.write_transaction(tx_mimo_signed)
-
-print(json.dumps(tx_mimo_signed, sort_keys=True, indent=4, separators=(',', ':')))
-
-sleep(8)
-
-"""
-Threshold Conditions
-"""
-
-# create some new testusers
-thresholduser1_priv, thresholduser1_pub = crypto.generate_key_pair()
-thresholduser2_priv, thresholduser2_pub = crypto.generate_key_pair()
-thresholduser3_priv, thresholduser3_pub = crypto.generate_key_pair()
-
-# retrieve the last transaction of testuser2
-tx_retrieved_id = b.get_owned_ids(testuser2_pub).pop()
-
-# create a base template for a 1-input/3-output transaction
-threshold_tx = b.create_transaction(testuser2_pub, [thresholduser1_pub, thresholduser2_pub, thresholduser3_pub],
-                                    tx_retrieved_id, 'TRANSFER')
-
-# create a 2-out-of-3 Threshold Cryptocondition
-threshold_condition = cc.ThresholdSha256Fulfillment(threshold=2)
-threshold_condition.add_subfulfillment(cc.Ed25519Fulfillment(public_key=thresholduser1_pub))
-threshold_condition.add_subfulfillment(cc.Ed25519Fulfillment(public_key=thresholduser2_pub))
-threshold_condition.add_subfulfillment(cc.Ed25519Fulfillment(public_key=thresholduser3_pub))
-
-# update the condition in the newly created transaction
-threshold_tx['transaction']['conditions'][0]['condition'] = {
-    'details': threshold_condition.to_dict(),
-    'uri': threshold_condition.condition.serialize_uri()
-}
-
-# conditions have been updated, so hash needs updating
-threshold_tx['id'] = util.get_hash_data(threshold_tx)
-
-# sign the transaction
-threshold_tx_signed = b.sign_transaction(threshold_tx, testuser2_priv)
-
-b.validate_transaction(threshold_tx_signed)
-# write the transaction
-b.write_transaction(threshold_tx_signed)
-
-sleep(8)
-
-# check if the transaction is already in the bigchain
-tx_threshold_retrieved = b.get_transaction(threshold_tx_signed['id'])
-print(json.dumps(tx_threshold_retrieved, sort_keys=True, indent=4, separators=(',', ':')))
-
-thresholduser4_priv, thresholduser4_pub = crypto.generate_key_pair()
-
-# retrieve the last transaction of thresholduser1_pub
-tx_retrieved_id = b.get_owned_ids(thresholduser1_pub).pop()
-
-# create a base template for a 2-input/1-output transaction
-threshold_tx_transfer = b.create_transaction([thresholduser1_pub, thresholduser2_pub, thresholduser3_pub],
-                                             thresholduser4_pub, tx_retrieved_id, 'TRANSFER')
-
-# parse the threshold cryptocondition
-threshold_fulfillment = cc.Fulfillment.from_dict(threshold_tx['transaction']['conditions'][0]['condition']['details'])
-
-subfulfillment1 = threshold_fulfillment.get_subcondition_from_vk(thresholduser1_pub)[0]
-subfulfillment2 = threshold_fulfillment.get_subcondition_from_vk(thresholduser2_pub)[0]
-subfulfillment3 = threshold_fulfillment.get_subcondition_from_vk(thresholduser3_pub)[0]
-
-
-# get the fulfillment message to sign
-threshold_tx_fulfillment_message = util.get_fulfillment_message(threshold_tx_transfer,
-                                                                threshold_tx_transfer['transaction']['fulfillments'][0],
-                                                                serialized=True)
-
-# clear the subconditions of the threshold fulfillment, they will be added again after signing
-threshold_fulfillment.subconditions = []
-
-# sign and add the subconditions until threshold of 2 is reached
-subfulfillment1.sign(threshold_tx_fulfillment_message, crypto.SigningKey(thresholduser1_priv))
-threshold_fulfillment.add_subfulfillment(subfulfillment1)
-subfulfillment2.sign(threshold_tx_fulfillment_message, crypto.SigningKey(thresholduser2_priv))
-threshold_fulfillment.add_subfulfillment(subfulfillment2)
-
-# Add remaining (unfulfilled) fulfillment as a condition
-threshold_fulfillment.add_subcondition(subfulfillment3.condition)
-
-assert threshold_fulfillment.validate(threshold_tx_fulfillment_message) == True
-
-threshold_tx_transfer['transaction']['fulfillments'][0]['fulfillment'] = threshold_fulfillment.serialize_uri()
-
-assert b.validate_fulfillments(threshold_tx_transfer) == True
-
-assert b.validate_transaction(threshold_tx_transfer) == threshold_tx_transfer
-
-b.write_transaction(threshold_tx_transfer)
-
-print(json.dumps(threshold_tx_transfer, sort_keys=True, indent=4, separators=(',', ':')))
-
-"""
-Hashlocked Conditions
-"""
-
-# Create a hash-locked asset without any owners_after
-hashlock_tx = b.create_transaction(b.me, None, None, 'CREATE')
-
-# Define a secret that will be hashed - fulfillments need to guess the secret
-secret = b'much secret! wow!'
-first_tx_condition = cc.PreimageSha256Fulfillment(preimage=secret)
-
-# The conditions list is empty, so we need to append a new condition
-hashlock_tx['transaction']['conditions'].append({
-    'condition': {
-        'uri': first_tx_condition.condition.serialize_uri()
-    },
-    'cid': 0,
-    'owners_after': None
-})
-
-# Conditions have been updated, so hash needs updating
-hashlock_tx['id'] = util.get_hash_data(hashlock_tx)
-
-# The asset needs to be signed by the owner_before
-hashlock_tx_signed = b.sign_transaction(hashlock_tx, b.me_private)
-
-# Some validations
-assert b.validate_transaction(hashlock_tx_signed) == hashlock_tx_signed
-assert b.is_valid_transaction(hashlock_tx_signed) == hashlock_tx_signed
-
-b.write_transaction(hashlock_tx_signed)
-print(json.dumps(hashlock_tx_signed, sort_keys=True, indent=4, separators=(',', ':')))
-
-sleep(8)
-
-hashlockuser_priv, hashlockuser_pub = crypto.generate_key_pair()
-
-# create hashlock fulfillment tx
-hashlock_fulfill_tx = b.create_transaction(None, hashlockuser_priv, {'txid': hashlock_tx['id'], 'cid': 0}, 'TRANSFER')
-
-# try a wrong secret
-hashlock_fulfill_tx_fulfillment = cc.PreimageSha256Fulfillment(preimage=b'')
-hashlock_fulfill_tx['transaction']['fulfillments'][0]['fulfillment'] = \
-    hashlock_fulfill_tx_fulfillment.serialize_uri()
-
-assert b.is_valid_transaction(hashlock_fulfill_tx) == False
-
-# provide the right secret
-hashlock_fulfill_tx_fulfillment = cc.PreimageSha256Fulfillment(preimage=secret)
-hashlock_fulfill_tx['transaction']['fulfillments'][0]['fulfillment'] = \
-    hashlock_fulfill_tx_fulfillment.serialize_uri()
-
-assert b.validate_transaction(hashlock_fulfill_tx) == hashlock_fulfill_tx
-assert b.is_valid_transaction(hashlock_fulfill_tx) == hashlock_fulfill_tx
-
-b.write_transaction(hashlock_fulfill_tx)
-print(json.dumps(hashlock_fulfill_tx, sort_keys=True, indent=4, separators=(',', ':')))
-
-
-"""
-Timeout Conditions
-"""
-# Create transaction template
-tx_timeout = b.create_transaction(b.me, None, None, 'CREATE')
-
-# Set expiry time (12 secs from now)
-time_sleep = 12
-time_expire = str(float(gen_timestamp()) + time_sleep)
-
-# only valid if the server time <= time_expire
-condition_timeout = cc.TimeoutFulfillment(expire_time=time_expire)
-
-# The conditions list is empty, so we need to append a new condition
-tx_timeout['transaction']['conditions'].append({
-    'condition': {
-        'details': condition_timeout.to_dict(),
-        'uri': condition_timeout.condition.serialize_uri()
-    },
-    'cid': 0,
-    'owners_after': None
-})
-
-# conditions have been updated, so hash needs updating
-tx_timeout['id'] = util.get_hash_data(tx_timeout)
-
-# sign transaction
-tx_timeout_signed = b.sign_transaction(tx_timeout, b.me_private)
-
-b.write_transaction(tx_timeout_signed)
-print(json.dumps(tx_timeout, sort_keys=True, indent=4, separators=(',', ':')))
-sleep(8)
-
-# Retrieve the transaction id of tx_timeout
-tx_timeout_id = {'txid': tx_timeout['id'], 'cid': 0}
-
-# Create a template to transfer the tx_timeout
-tx_timeout_transfer = b.create_transaction(None, testuser1_pub, tx_timeout_id, 'TRANSFER')
-
-# Parse the threshold cryptocondition
-timeout_fulfillment = cc.Fulfillment.from_dict(
-    tx_timeout['transaction']['conditions'][0]['condition']['details'])
-
-tx_timeout_transfer['transaction']['fulfillments'][0]['fulfillment'] = timeout_fulfillment.serialize_uri()
-
-# no need to sign transaction, like with hashlocks
-for i in range(time_sleep - 4):
-    tx_timeout_valid = b.is_valid_transaction(tx_timeout_transfer) == tx_timeout_transfer
-    seconds_to_timeout = int(float(time_expire) - float(gen_timestamp()))
-    print('tx_timeout valid: {} ({}s to timeout)'.format(tx_timeout_valid, seconds_to_timeout))
-    sleep(1)
-
-"""
-Escrow Conditions
-"""
-# retrieve the last transaction of testuser2
-tx_retrieved_id = b.get_owned_ids(testuser2_pub).pop()
-
-# Create escrow template with the execute and abort address
-tx_escrow = b.create_transaction(testuser2_pub, [testuser2_pub, testuser1_pub], tx_retrieved_id, 'TRANSFER')
-
-# Set expiry time (12 secs from now)
-time_sleep = 12
-time_expire = str(float(gen_timestamp()) + time_sleep)
-
-# Create escrow and timeout condition
-condition_escrow = cc.ThresholdSha256Fulfillment(threshold=1)  # OR Gate
-condition_timeout = cc.TimeoutFulfillment(expire_time=time_expire)  # only valid if now() <= time_expire
-condition_timeout_inverted = cc.InvertedThresholdSha256Fulfillment(threshold=1)
-condition_timeout_inverted.add_subfulfillment(condition_timeout)
-
-# Create execute branch
-condition_execute = cc.ThresholdSha256Fulfillment(threshold=2)  # AND gate
-condition_execute.add_subfulfillment(cc.Ed25519Fulfillment(public_key=testuser1_pub))  # execute address
-condition_execute.add_subfulfillment(condition_timeout)  # federation checks on expiry
-condition_escrow.add_subfulfillment(condition_execute)
-
-# Create abort branch
-condition_abort = cc.ThresholdSha256Fulfillment(threshold=2)  # AND gate
-condition_abort.add_subfulfillment(cc.Ed25519Fulfillment(public_key=testuser2_pub))  # abort address
-condition_abort.add_subfulfillment(condition_timeout_inverted)
-condition_escrow.add_subfulfillment(condition_abort)
-
-# Update the condition in the newly created transaction
-tx_escrow['transaction']['conditions'][0]['condition'] = {
-    'details': condition_escrow.to_dict(),
-    'uri': condition_escrow.condition.serialize_uri()
-}
-
-# conditions have been updated, so hash needs updating
-tx_escrow['id'] = util.get_hash_data(tx_escrow)
-
-# sign transaction
-tx_escrow_signed = b.sign_transaction(tx_escrow, testuser2_priv)
-
-# some checks
-assert b.validate_transaction(tx_escrow_signed) == tx_escrow_signed
-assert b.is_valid_transaction(tx_escrow_signed) == tx_escrow_signed
-
-print(json.dumps(tx_escrow_signed, sort_keys=True, indent=4, separators=(',', ':')))
-b.write_transaction(tx_escrow_signed)
-sleep(8)
-
-# Retrieve the last transaction of thresholduser1_pub
-tx_escrow_id = {'txid': tx_escrow_signed['id'], 'cid': 0}
-
-# Create a base template for output transaction
-tx_escrow_execute = b.create_transaction([testuser2_pub, testuser1_pub], testuser1_pub, tx_escrow_id, 'TRANSFER')
-
-# Parse the threshold cryptocondition
-escrow_fulfillment = cc.Fulfillment.from_dict(
-    tx_escrow['transaction']['conditions'][0]['condition']['details'])
-
-subfulfillment_testuser1 = escrow_fulfillment.get_subcondition_from_vk(testuser1_pub)[0]
-subfulfillment_testuser2 = escrow_fulfillment.get_subcondition_from_vk(testuser2_pub)[0]
-subfulfillment_timeout = escrow_fulfillment.subconditions[0]['body'].subconditions[1]['body']
-subfulfillment_timeout_inverted = escrow_fulfillment.subconditions[1]['body'].subconditions[1]['body']
-
-# Get the fulfillment message to sign
-tx_escrow_execute_fulfillment_message = \
-    util.get_fulfillment_message(tx_escrow_execute,
-                                 tx_escrow_execute['transaction']['fulfillments'][0],
-                                 serialized=True)
-
-escrow_fulfillment.subconditions = []
-
-# fulfill execute branch
-fulfillment_execute = cc.ThresholdSha256Fulfillment(threshold=2)
-subfulfillment_testuser1.sign(tx_escrow_execute_fulfillment_message, crypto.SigningKey(testuser1_priv))
-fulfillment_execute.add_subfulfillment(subfulfillment_testuser1)
-fulfillment_execute.add_subfulfillment(subfulfillment_timeout)
-escrow_fulfillment.add_subfulfillment(fulfillment_execute)
-
-# do not fulfill abort branch
-condition_abort = cc.ThresholdSha256Fulfillment(threshold=2)
-condition_abort.add_subfulfillment(subfulfillment_testuser2)
-condition_abort.add_subfulfillment(subfulfillment_timeout_inverted)
-escrow_fulfillment.add_subcondition(condition_abort.condition)
-
-# create fulfillment and append to transaction
-tx_escrow_execute['transaction']['fulfillments'][0]['fulfillment'] = escrow_fulfillment.serialize_uri()
-
-# Time has expired, hence the abort branch can redeem
-tx_escrow_abort = b.create_transaction([testuser2_pub, testuser1_pub], testuser2_pub, tx_escrow_id, 'TRANSFER')
-
-# Parse the threshold cryptocondition
-escrow_fulfillment = cc.Fulfillment.from_dict(
-    tx_escrow['transaction']['conditions'][0]['condition']['details'])
-
-subfulfillment_testuser1 = escrow_fulfillment.get_subcondition_from_vk(testuser1_pub)[0]
-subfulfillment_testuser2 = escrow_fulfillment.get_subcondition_from_vk(testuser2_pub)[0]
-subfulfillment_timeout = escrow_fulfillment.subconditions[0]['body'].subconditions[1]['body']
-subfulfillment_timeout_inverted = escrow_fulfillment.subconditions[1]['body'].subconditions[1]['body']
-
-tx_escrow_abort_fulfillment_message = \
-    util.get_fulfillment_message(tx_escrow_abort,
-                                 tx_escrow_abort['transaction']['fulfillments'][0],
-                                 serialized=True)
-escrow_fulfillment.subconditions = []
-
-# Do not fulfill execute branch
-condition_execute = cc.ThresholdSha256Fulfillment(threshold=2)
-condition_execute.add_subfulfillment(subfulfillment_testuser1)
-condition_execute.add_subfulfillment(subfulfillment_timeout)
-escrow_fulfillment.add_subcondition(condition_execute.condition)
-
-# Fulfill abort branch
-fulfillment_abort = cc.ThresholdSha256Fulfillment(threshold=2)
-subfulfillment_testuser2.sign(tx_escrow_abort_fulfillment_message, crypto.SigningKey(testuser2_priv))
-fulfillment_abort.add_subfulfillment(subfulfillment_testuser2)
-fulfillment_abort.add_subfulfillment(subfulfillment_timeout_inverted)
-escrow_fulfillment.add_subfulfillment(fulfillment_abort)
-
-tx_escrow_abort['transaction']['fulfillments'][0]['fulfillment'] = escrow_fulfillment.serialize_uri()
-
-for i in range(time_sleep - 4):
-    valid_execute = b.is_valid_transaction(tx_escrow_execute) == tx_escrow_execute
-    valid_abort = b.is_valid_transaction(tx_escrow_abort) == tx_escrow_abort
-
-    seconds_to_timeout = int(float(time_expire) - float(gen_timestamp()))
-    print('tx_execute valid: {} - tx_abort valid {} ({}s to timeout)'.format(valid_execute, valid_abort, seconds_to_timeout))
-    sleep(1)

tests/pipelines/test_block_creation.py

@@ -1,8 +1,6 @@
 import time
 from unittest.mock import patch
 
-import rethinkdb as r
-
 from multipipes import Pipe
 
 
@@ -38,14 +36,14 @@ def test_validate_transaction(b, create_tx):
     assert block_maker.validate_tx(valid_tx.to_dict()) == valid_tx
 
 
-def test_create_block(b, user_vk):
+def test_create_block(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines.block import BlockPipeline
 
     block_maker = BlockPipeline()
 
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         block_maker.create(tx)
 
@@ -55,7 +53,7 @@ def test_create_block(b, user_vk):
     assert len(block_doc.transactions) == 100
 
 
-def test_write_block(b, user_vk):
+def test_write_block(b, user_pk):
     from bigchaindb.models import Block, Transaction
     from bigchaindb.pipelines.block import BlockPipeline
 
@@ -63,26 +61,26 @@ def test_write_block(b, user_vk):
 
     txs = []
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         txs.append(tx)
 
     block_doc = b.create_block(txs)
     block_maker.write(block_doc)
-    expected = b.connection.run(r.table('bigchain').get(block_doc.id))
+    expected = b.backend.get_block(block_doc.id)
     expected = Block.from_dict(expected)
 
     assert expected == block_doc
 
 
-def test_duplicate_transaction(b, user_vk):
+def test_duplicate_transaction(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines import block
     block_maker = block.BlockPipeline()
 
     txs = []
     for i in range(10):
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         txs.append(tx)
 
@@ -90,26 +88,27 @@ def test_duplicate_transaction(b, user_vk):
     block_maker.write(block_doc)
 
     # block is in bigchain
-    assert b.connection.run(r.table('bigchain').get(block_doc.id)) == block_doc.to_dict()
+    assert b.backend.get_block(block_doc.id) == block_doc.to_dict()
 
     b.write_transaction(txs[0])
 
     # verify tx is in the backlog
-    assert b.connection.run(r.table('backlog').get(txs[0].id)) is not None
+    assert b.get_transaction(txs[0].id) is not None
 
     # try to validate a transaction that's already in the chain; should not work
     assert block_maker.validate_tx(txs[0].to_dict()) is None
 
     # duplicate tx should be removed from backlog
-    assert b.connection.run(r.table('backlog').get(txs[0].id)) is None
+    response, status = b.get_transaction(txs[0].id, include_status=True)
+    assert status != b.TX_IN_BACKLOG
 
 
-def test_delete_tx(b, user_vk):
+def test_delete_tx(b, user_pk):
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines.block import BlockPipeline
     block_maker = BlockPipeline()
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         block_maker.create(tx)
         # make sure the tx appears in the backlog
@@ -119,9 +118,7 @@ def test_delete_tx(b, user_vk):
     block_doc = block_maker.create(None, timeout=True)
 
     for tx in block_doc.to_dict()['block']['transactions']:
-        returned_tx = b.connection.run(r.table('backlog').get(tx['id']))
-        returned_tx.pop('assignee')
-        returned_tx.pop('assignment_timestamp')
+        returned_tx = b.get_transaction(tx['id']).to_dict()
         assert returned_tx == tx
 
     returned_block = block_maker.delete_tx(block_doc)
@@ -129,16 +126,17 @@ def test_delete_tx(b, user_vk):
     assert returned_block == block_doc
 
     for tx in block_doc.to_dict()['block']['transactions']:
-        assert b.connection.run(r.table('backlog').get(tx['id'])) is None
+        returned_tx, status = b.get_transaction(tx['id'], include_status=True)
+        assert status != b.TX_IN_BACKLOG
 
 
-def test_prefeed(b, user_vk):
+def test_prefeed(b, user_pk):
     import random
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines.block import initial
 
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)],
+        tx = Transaction.create([b.me], [([user_pk], 1)],
                                 {'msg': random.random()})
         tx = tx.sign([b.me_private])
         b.write_transaction(tx)
@@ -159,26 +157,22 @@ def test_start(create_pipeline):
     assert pipeline == create_pipeline.return_value
 
 
-def test_full_pipeline(b, user_vk):
+def test_full_pipeline(b, user_pk):
     import random
     from bigchaindb.models import Block, Transaction
     from bigchaindb.pipelines.block import create_pipeline, get_changefeed
 
     outpipe = Pipe()
-
-    count_assigned_to_me = 0
+    # include myself here, so that some tx are actually assigned to me
+    b.nodes_except_me = [b.me, 'aaa', 'bbb', 'ccc']
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)],
+        tx = Transaction.create([b.me], [([user_pk], 1)],
                                 {'msg': random.random()})
-        tx = tx.sign([b.me_private]).to_dict()
-        assignee = random.choice([b.me, 'aaa', 'bbb', 'ccc'])
-        tx['assignee'] = assignee
-        tx['assignment_timestamp'] = time.time()
-        if assignee == b.me:
-            count_assigned_to_me += 1
-        b.connection.run(r.table('backlog').insert(tx, durability='hard'))
+        tx = tx.sign([b.me_private])
 
-    assert b.connection.run(r.table('backlog').count()) == 100
+        b.write_transaction(tx)
+
+    assert b.backend.count_backlog() == 100
 
     pipeline = create_pipeline()
     pipeline.setup(indata=get_changefeed(), outdata=outpipe)
@@ -188,9 +182,9 @@ def test_full_pipeline(b, user_vk):
     pipeline.terminate()
 
     block_doc = outpipe.get()
-    chained_block = b.connection.run(r.table('bigchain').get(block_doc.id))
+    chained_block = b.backend.get_block(block_doc.id)
     chained_block = Block.from_dict(chained_block)
 
-    assert len(block_doc.transactions) == count_assigned_to_me
+    block_len = len(block_doc.transactions)
     assert chained_block == block_doc
-    assert b.connection.run(r.table('backlog').count()) == 100 - count_assigned_to_me
+    assert b.backend.count_backlog() == 100 - block_len 

tests/pipelines/test_election.py

@@ -2,20 +2,19 @@ import time
 from unittest.mock import patch
 
 from bigchaindb.common import crypto
-import rethinkdb as r
 from multipipes import Pipe, Pipeline
 
 from bigchaindb import Bigchain
 from bigchaindb.pipelines import election
 
 
-def test_check_for_quorum_invalid(b, user_vk):
+def test_check_for_quorum_invalid(b, user_pk):
     from bigchaindb.models import Transaction
 
     e = election.Election()
 
     # create blocks with transactions
-    tx1 = Transaction.create([b.me], [([user_vk], 1)])
+    tx1 = Transaction.create([b.me], [([user_pk], 1)])
     test_block = b.create_block([tx1])
 
     # simulate a federation with four voters
@@ -33,18 +32,19 @@ def test_check_for_quorum_invalid(b, user_vk):
                    [member.vote(test_block.id, 'abc', False) for member in test_federation[2:]]
 
     # cast votes
-    b.connection.run(r.table('votes').insert(votes, durability='hard'))
+    for vote in votes:
+        b.write_vote(vote)
 
     # since this block is now invalid, should pass to the next process
     assert e.check_for_quorum(votes[-1]) == test_block
 
 
-def test_check_for_quorum_invalid_prev_node(b, user_vk):
+def test_check_for_quorum_invalid_prev_node(b, user_pk):
     from bigchaindb.models import Transaction
     e = election.Election()
 
     # create blocks with transactions
-    tx1 = Transaction.create([b.me], [([user_vk], 1)])
+    tx1 = Transaction.create([b.me], [([user_pk], 1)])
     test_block = b.create_block([tx1])
 
     # simulate a federation with four voters
@@ -62,19 +62,20 @@ def test_check_for_quorum_invalid_prev_node(b, user_vk):
                    [member.vote(test_block.id, 'def', True) for member in test_federation[2:]]
 
     # cast votes
-    b.connection.run(r.table('votes').insert(votes, durability='hard'))
+    for vote in votes:
+        b.write_vote(vote)
 
     # since nodes cannot agree on prev block, the block is invalid
     assert e.check_for_quorum(votes[-1]) == test_block
 
 
-def test_check_for_quorum_valid(b, user_vk):
+def test_check_for_quorum_valid(b, user_pk):
     from bigchaindb.models import Transaction
 
     e = election.Election()
 
     # create blocks with transactions
-    tx1 = Transaction.create([b.me], [([user_vk], 1)])
+    tx1 = Transaction.create([b.me], [([user_pk], 1)])
     test_block = b.create_block([tx1])
 
     # simulate a federation with four voters
@@ -91,26 +92,29 @@ def test_check_for_quorum_valid(b, user_vk):
     votes = [member.vote(test_block.id, 'abc', True)
              for member in test_federation]
     # cast votes
-    b.connection.run(r.table('votes').insert(votes, durability='hard'))
+    for vote in votes:
+        b.write_vote(vote)
 
     # since this block is valid, should go nowhere
     assert e.check_for_quorum(votes[-1]) is None
 
 
-def test_check_requeue_transaction(b, user_vk):
+def test_check_requeue_transaction(b, user_pk):
     from bigchaindb.models import Transaction
 
     e = election.Election()
 
     # create blocks with transactions
-    tx1 = Transaction.create([b.me], [([user_vk], 1)])
+    tx1 = Transaction.create([b.me], [([user_pk], 1)])
     test_block = b.create_block([tx1])
 
     e.requeue_transactions(test_block)
-    backlog_tx = b.connection.run(r.table('backlog').get(tx1.id))
-    backlog_tx.pop('assignee')
-    backlog_tx.pop('assignment_timestamp')
-    assert backlog_tx == tx1.to_dict()
+
+    backlog_tx, status = b.get_transaction(tx1.id, include_status=True)
+    #backlog_tx = b.connection.run(r.table('backlog').get(tx1.id))
+    assert status == b.TX_IN_BACKLOG
+    assert backlog_tx == tx1
+
 
 
 @patch.object(Pipeline, 'start')
@@ -122,7 +126,7 @@ def test_start(mock_start):
     mock_start.assert_called_with()
 
 
-def test_full_pipeline(b, user_vk):
+def test_full_pipeline(b, user_pk):
     import random
     from bigchaindb.models import Transaction
 
@@ -131,7 +135,7 @@ def test_full_pipeline(b, user_vk):
     # write two blocks
     txs = []
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)],
+        tx = Transaction.create([b.me], [([user_pk], 1)],
                                 {'msg': random.random()})
         tx = tx.sign([b.me_private])
         txs.append(tx)
@@ -141,7 +145,7 @@ def test_full_pipeline(b, user_vk):
 
     txs = []
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)],
+        tx = Transaction.create([b.me], [([user_pk], 1)],
                                 {'msg': random.random()})
         tx = tx.sign([b.me_private])
         txs.append(tx)
@@ -157,16 +161,16 @@ def test_full_pipeline(b, user_vk):
     vote_valid = b.vote(valid_block.id, 'abc', True)
     vote_invalid = b.vote(invalid_block.id, 'abc', False)
 
-    b.connection.run(r.table('votes').insert(vote_valid, durability='hard'))
-    b.connection.run(r.table('votes').insert(vote_invalid, durability='hard'))
+    b.write_vote(vote_valid)
+    b.write_vote(vote_invalid)
 
     outpipe.get()
     pipeline.terminate()
 
     # only transactions from the invalid block should be returned to
     # the backlog
-    assert b.connection.run(r.table('backlog').count()) == 100
+    assert b.backend.count_backlog() == 100
     # NOTE: I'm still, I'm still tx from the block.
     tx_from_block = set([tx.id for tx in invalid_block.transactions])
-    tx_from_backlog = set([tx['id'] for tx in list(b.connection.run(r.table('backlog')))])
+    tx_from_backlog = set([tx['id'] for tx in list(b.backend.get_stale_transactions(0))])
     assert tx_from_block == tx_from_backlog

tests/pipelines/test_stale_monitor.py

@@ -1,16 +1,14 @@
-import rethinkdb as r
 from bigchaindb import Bigchain
 from bigchaindb.pipelines import stale
 from multipipes import Pipe, Pipeline
 from unittest.mock import patch
 from bigchaindb import config_utils
-import time
 import os
 
 
-def test_get_stale(b, user_vk):
+def test_get_stale(b, user_pk):
     from bigchaindb.models import Transaction
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx = tx.sign([b.me_private])
     b.write_transaction(tx, durability='hard')
 
@@ -24,10 +22,10 @@ def test_get_stale(b, user_vk):
         assert tx.to_dict() == _tx
 
 
-def test_reassign_transactions(b, user_vk):
+def test_reassign_transactions(b, user_pk):
     from bigchaindb.models import Transaction
     # test with single node
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx = tx.sign([b.me_private])
     b.write_transaction(tx, durability='hard')
 
@@ -36,33 +34,33 @@ def test_reassign_transactions(b, user_vk):
     stm.reassign_transactions(tx.to_dict())
 
     # test with federation
-    tx = Transaction.create([b.me], [([user_vk], 1)])
+    tx = Transaction.create([b.me], [([user_pk], 1)])
     tx = tx.sign([b.me_private])
     b.write_transaction(tx, durability='hard')
 
     stm = stale.StaleTransactionMonitor(timeout=0.001,
                                         backlog_reassign_delay=0.001)
     stm.bigchain.nodes_except_me = ['aaa', 'bbb', 'ccc']
-    tx = list(b.connection.run(r.table('backlog')))[0]
+    tx = list(b.backend.get_stale_transactions(0))[0]
     stm.reassign_transactions(tx)
 
-    reassigned_tx = b.connection.run(r.table('backlog').get(tx['id']))
+    reassigned_tx = list(b.backend.get_stale_transactions(0))[0]
     assert reassigned_tx['assignment_timestamp'] > tx['assignment_timestamp']
     assert reassigned_tx['assignee'] != tx['assignee']
 
     # test with node not in federation
-    tx = Transaction.create([b.me], [([user_vk], 1)])
-    tx = tx.sign([b.me_private]).to_dict()
-    tx.update({'assignee': 'lol'})
-    tx.update({'assignment_timestamp': time.time()})
-    b.connection.run(r.table('backlog').insert(tx, durability='hard'))
+    tx = Transaction.create([b.me], [([user_pk], 1)])
+    tx = tx.sign([b.me_private])
+    stm.bigchain.nodes_except_me = ['lol']
+    b.write_transaction(tx, durability='hard')
+    stm.bigchain.nodes_except_me = None
 
-    tx = list(b.connection.run(r.table('backlog')))[0]
+    tx = list(b.backend.get_stale_transactions(0))[0]
     stm.reassign_transactions(tx)
-    assert b.connection.run(r.table('backlog').get(tx['id']))['assignee'] != 'lol'
+    assert tx['assignee'] != 'lol'
 
 
-def test_full_pipeline(monkeypatch, user_vk):
+def test_full_pipeline(monkeypatch, user_pk):
     from bigchaindb.models import Transaction
     CONFIG = {
         'database': {
@@ -77,7 +75,6 @@ def test_full_pipeline(monkeypatch, user_vk):
     }
     config_utils.set_config(CONFIG)
     b = Bigchain()
-    outpipe = Pipe()
 
     original_txs = {}
     original_txc = []
@@ -85,20 +82,27 @@ def test_full_pipeline(monkeypatch, user_vk):
     monkeypatch.setattr('time.time', lambda: 1)
 
     for i in range(100):
-        tx = Transaction.create([b.me], [([user_vk], 1)])
+        tx = Transaction.create([b.me], [([user_pk], 1)])
         tx = tx.sign([b.me_private])
         original_txc.append(tx.to_dict())
 
         b.write_transaction(tx)
-        original_txs[tx.id] = b.connection.run(r.table('backlog').get(tx.id))
+    original_txs = list(b.backend.get_stale_transactions(0))
+    original_txs = {tx['id']: tx for tx in original_txs}
 
-    assert b.connection.run(r.table('backlog').count()) == 100
+    assert len(original_txs) == 100
 
     monkeypatch.undo()
 
+    inpipe = Pipe()
+    # Each time the StaleTransactionMonitor pipeline runs, it reassigns
+    # all eligible transactions. Passing this inpipe prevents that from
+    # taking place more than once.
+    inpipe.put(())
+    outpipe = Pipe()
     pipeline = stale.create_pipeline(backlog_reassign_delay=1,
                                      timeout=1)
-    pipeline.setup(outdata=outpipe)
+    pipeline.setup(indata=inpipe, outdata=outpipe)
     pipeline.start()
 
     # to terminate
@@ -107,8 +111,8 @@ def test_full_pipeline(monkeypatch, user_vk):
 
     pipeline.terminate()
 
-    assert b.connection.run(r.table('backlog').count()) == 100
-    reassigned_txs = list(b.connection.run(r.table('backlog')))
+    assert len(list(b.backend.get_stale_transactions(0))) == 100
+    reassigned_txs= list(b.backend.get_stale_transactions(0))
 
     # check that every assignment timestamp has increased, and every tx has a new assignee
     for reassigned_tx in reassigned_txs:

tests/pipelines/test_vote.py

@@ -2,7 +2,6 @@ import time
 
 from unittest.mock import patch
 
-import rethinkdb as r
 from multipipes import Pipe, Pipeline
 
 
@@ -33,7 +32,8 @@ def test_vote_creation_valid(b):
     assert vote['vote']['is_block_valid'] is True
     assert vote['vote']['invalid_reason'] is None
     assert vote['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialize(vote['vote']).encode(),
+    assert isinstance(vote['signature'], str)
+    assert crypto.PublicKey(b.me).verify(serialize(vote['vote']).encode(),
                                             vote['signature']) is True
 
 
@@ -52,7 +52,7 @@ def test_vote_creation_invalid(b):
     assert vote['vote']['is_block_valid'] is False
     assert vote['vote']['invalid_reason'] is None
     assert vote['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialize(vote['vote']).encode(),
+    assert crypto.PublicKey(b.me).verify(serialize(vote['vote']).encode(),
                                             vote['signature']) is True
 
 
@@ -166,7 +166,7 @@ def test_valid_block_voting_sequential(b, monkeypatch):
         last_vote = vote_obj.vote(*vote_obj.validate_tx(tx, block_id, num_tx))
 
     vote_obj.write_vote(last_vote)
-    vote_rs = b.connection.run(r.table('votes').get_all([block.id, b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block_id, b.me)
     vote_doc = vote_rs.next()
 
     assert vote_doc['vote'] == {'voting_for_block': block.id,
@@ -177,7 +177,7 @@ def test_valid_block_voting_sequential(b, monkeypatch):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
@@ -200,7 +200,7 @@ def test_valid_block_voting_multiprocessing(b, monkeypatch):
     vote_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block.id, b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block.id, b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block.id,
@@ -211,7 +211,7 @@ def test_valid_block_voting_multiprocessing(b, monkeypatch):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
@@ -241,7 +241,7 @@ def test_valid_block_voting_with_create_transaction(b, monkeypatch):
     vote_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block.id, b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block.id, b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block.id,
@@ -252,7 +252,7 @@ def test_valid_block_voting_with_create_transaction(b, monkeypatch):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
@@ -296,7 +296,7 @@ def test_valid_block_voting_with_transfer_transactions(monkeypatch, b):
     vote2_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block.id, b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block.id, b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block.id,
@@ -307,10 +307,10 @@ def test_valid_block_voting_with_transfer_transactions(monkeypatch, b):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
-    vote2_rs = b.connection.run(r.table('votes').get_all([block2.id, b.me], index='block_and_voter'))
+    vote2_rs = b.backend.get_votes_by_block_id_and_voter(block2.id, b.me)
     vote2_doc = vote2_rs.next()
     assert vote2_out['vote'] == vote2_doc['vote']
     assert vote2_doc['vote'] == {'voting_for_block': block2.id,
@@ -321,11 +321,11 @@ def test_valid_block_voting_with_transfer_transactions(monkeypatch, b):
 
     serialized_vote2 = util.serialize(vote2_doc['vote']).encode()
     assert vote2_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote2,
+    assert crypto.PublicKey(b.me).verify(serialized_vote2,
                                             vote2_doc['signature']) is True
 
 
-def test_unsigned_tx_in_block_voting(monkeypatch, b, user_vk):
+def test_unsigned_tx_in_block_voting(monkeypatch, b, user_pk):
     from bigchaindb.common import crypto, util
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines import vote
@@ -347,7 +347,7 @@ def test_unsigned_tx_in_block_voting(monkeypatch, b, user_vk):
     vote_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block.id, b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block.id, b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block.id,
@@ -358,11 +358,11 @@ def test_unsigned_tx_in_block_voting(monkeypatch, b, user_vk):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
-def test_invalid_id_tx_in_block_voting(monkeypatch, b, user_vk):
+def test_invalid_id_tx_in_block_voting(monkeypatch, b, user_pk):
     from bigchaindb.common import crypto, util
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines import vote
@@ -386,7 +386,7 @@ def test_invalid_id_tx_in_block_voting(monkeypatch, b, user_vk):
     vote_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block['id'], b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block['id'], b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block['id'],
@@ -397,11 +397,11 @@ def test_invalid_id_tx_in_block_voting(monkeypatch, b, user_vk):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
-def test_invalid_content_in_tx_in_block_voting(monkeypatch, b, user_vk):
+def test_invalid_content_in_tx_in_block_voting(monkeypatch, b, user_pk):
     from bigchaindb.common import crypto, util
     from bigchaindb.models import Transaction
     from bigchaindb.pipelines import vote
@@ -425,7 +425,7 @@ def test_invalid_content_in_tx_in_block_voting(monkeypatch, b, user_vk):
     vote_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block['id'], b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block['id'], b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block['id'],
@@ -436,11 +436,11 @@ def test_invalid_content_in_tx_in_block_voting(monkeypatch, b, user_vk):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
-def test_invalid_block_voting(monkeypatch, b, user_vk):
+def test_invalid_block_voting(monkeypatch, b, user_pk):
     from bigchaindb.common import crypto, util
     from bigchaindb.pipelines import vote
 
@@ -460,7 +460,7 @@ def test_invalid_block_voting(monkeypatch, b, user_vk):
     vote_out = outpipe.get()
     vote_pipeline.terminate()
 
-    vote_rs = b.connection.run(r.table('votes').get_all([block['id'], b.me], index='block_and_voter'))
+    vote_rs = b.backend.get_votes_by_block_id_and_voter(block['id'], b.me)
     vote_doc = vote_rs.next()
     assert vote_out['vote'] == vote_doc['vote']
     assert vote_doc['vote'] == {'voting_for_block': block['id'],
@@ -471,7 +471,7 @@ def test_invalid_block_voting(monkeypatch, b, user_vk):
 
     serialized_vote = util.serialize(vote_doc['vote']).encode()
     assert vote_doc['node_pubkey'] == b.me
-    assert crypto.VerifyingKey(b.me).verify(serialized_vote,
+    assert crypto.PublicKey(b.me).verify(serialized_vote,
                                             vote_doc['signature']) is True
 
 
@@ -483,13 +483,16 @@ def test_voter_considers_unvoted_blocks_when_single_node(monkeypatch, b):
     monkeypatch.setattr('time.time', lambda: 1)
     b.create_genesis_block()
 
+    block_ids = []
     # insert blocks in the database while the voter process is not listening
     # (these blocks won't appear in the changefeed)
     monkeypatch.setattr('time.time', lambda: 2)
     block_1 = dummy_block(b)
+    block_ids.append(block_1.id)
     b.write_block(block_1, durability='hard')
     monkeypatch.setattr('time.time', lambda: 3)
     block_2 = dummy_block(b)
+    block_ids.append(block_2.id)
     b.write_block(block_2, durability='hard')
 
     vote_pipeline = vote.create_pipeline()
@@ -504,6 +507,7 @@ def test_voter_considers_unvoted_blocks_when_single_node(monkeypatch, b):
     # create a new block that will appear in the changefeed
     monkeypatch.setattr('time.time', lambda: 4)
     block_3 = dummy_block(b)
+    block_ids.append(block_3.id)
     b.write_block(block_3, durability='hard')
 
     # Same as before with the two `get`s
@@ -511,19 +515,9 @@ def test_voter_considers_unvoted_blocks_when_single_node(monkeypatch, b):
 
     vote_pipeline.terminate()
 
-    # retrieve blocks from bigchain
-    blocks = list(b.connection.run(
-                r.table('bigchain')
-                .order_by(r.asc((r.row['block']['timestamp'])))))
-
-    # FIXME: remove genesis block, we don't vote on it
-    # (might change in the future)
-    blocks.pop(0)
-    vote_pipeline.terminate()
-
     # retrieve vote
-    votes = b.connection.run(r.table('votes'))
-    votes = list(votes)
+    votes = [list(b.backend.get_votes_by_block_id(_id))[0]
+             for _id in block_ids]
 
     assert all(vote['node_pubkey'] == b.me for vote in votes)
 
@@ -536,12 +530,15 @@ def test_voter_chains_blocks_with_the_previous_ones(monkeypatch, b):
     monkeypatch.setattr('time.time', lambda: 1)
     b.create_genesis_block()
 
+    block_ids = []
     monkeypatch.setattr('time.time', lambda: 2)
     block_1 = dummy_block(b)
+    block_ids.append(block_1.id)
     b.write_block(block_1, durability='hard')
 
     monkeypatch.setattr('time.time', lambda: 3)
     block_2 = dummy_block(b)
+    block_ids.append(block_2.id)
     b.write_block(block_2, durability='hard')
 
     vote_pipeline = vote.create_pipeline()
@@ -555,15 +552,13 @@ def test_voter_chains_blocks_with_the_previous_ones(monkeypatch, b):
     vote_pipeline.terminate()
 
     # retrive blocks from bigchain
-    blocks = list(b.connection.run(
-                     r.table('bigchain')
-                     .order_by(r.asc((r.row['block']['timestamp'])))))
-
+    blocks = [b.get_block(_id) for _id in block_ids]
     # retrieve votes
-    votes = list(b.connection.run(r.table('votes')))
+    votes = [list(b.backend.get_votes_by_block_id(_id))[0]
+             for _id in block_ids]
 
-    assert votes[0]['vote']['voting_for_block'] in (blocks[1]['id'], blocks[2]['id'])
-    assert votes[1]['vote']['voting_for_block'] in (blocks[1]['id'], blocks[2]['id'])
+    assert votes[0]['vote']['voting_for_block'] in (blocks[0]['id'], blocks[1]['id'])
+    assert votes[1]['vote']['voting_for_block'] in (blocks[0]['id'], blocks[1]['id'])
 
 
 def test_voter_checks_for_previous_vote(monkeypatch, b):
@@ -578,8 +573,7 @@ def test_voter_checks_for_previous_vote(monkeypatch, b):
     monkeypatch.setattr('time.time', lambda: 2)
     block_1 = dummy_block(b)
     inpipe.put(block_1.to_dict())
-
-    assert b.connection.run(r.table('votes').count()) == 0
+    assert len(list(b.backend.get_votes_by_block_id(block_1.id))) == 0
 
     vote_pipeline = vote.create_pipeline()
     vote_pipeline.setup(indata=inpipe, outdata=outpipe)
@@ -594,14 +588,16 @@ def test_voter_checks_for_previous_vote(monkeypatch, b):
 
     # queue another block
     monkeypatch.setattr('time.time', lambda: 4)
-    inpipe.put(dummy_block(b).to_dict())
+    block_2 = dummy_block(b)
+    inpipe.put(block_2.to_dict())
 
     # wait for the result of the new block
     outpipe.get()
 
     vote_pipeline.terminate()
 
-    assert b.connection.run(r.table('votes').count()) == 2
+    assert len(list(b.backend.get_votes_by_block_id(block_1.id))) == 1
+    assert len(list(b.backend.get_votes_by_block_id(block_2.id))) == 1
 
 
 @patch.object(Pipeline, 'start')

tests/test_docs.py

@@ -0,0 +1,16 @@
+
+import subprocess
+
+
+def test_build_server_docs():
+    proc = subprocess.Popen(['bash'], stdin=subprocess.PIPE)
+    proc.stdin.write('cd docs/server; make html'.encode())
+    proc.stdin.close()
+    assert proc.wait() == 0
+
+
+def test_build_root_docs():
+    proc = subprocess.Popen(['bash'], stdin=subprocess.PIPE)
+    proc.stdin.write('cd docs/root; make html'.encode())
+    proc.stdin.close()
+    assert proc.wait() == 0

tests/test_models.py

@@ -142,7 +142,7 @@ class TestBlockModel(object):
         assert Block(transactions) == Block(transactions)
 
     def test_sign_block(self, b):
-        from bigchaindb.common.crypto import SigningKey, VerifyingKey
+        from bigchaindb.common.crypto import PrivateKey, PublicKey
         from bigchaindb.common.util import gen_timestamp, serialize
         from bigchaindb.models import Block, Transaction
 
@@ -156,13 +156,13 @@ class TestBlockModel(object):
             'voters': voters,
         }
         expected_block_serialized = serialize(expected_block).encode()
-        expected = SigningKey(b.me_private).sign(expected_block_serialized)
+        expected = PrivateKey(b.me_private).sign(expected_block_serialized)
         block = Block(transactions, b.me, timestamp, voters)
         block = block.sign(b.me_private)
         assert block.signature == expected.decode()
 
-        verifying_key = VerifyingKey(b.me)
-        assert verifying_key.verify(expected_block_serialized, block.signature)
+        public_key = PublicKey(b.me)
+        assert public_key.verify(expected_block_serialized, block.signature)
 
     def test_validate_already_voted_on_block(self, b, monkeypatch):
         from unittest.mock import Mock

tests/web/conftest.py

@@ -25,7 +25,7 @@ def app(request, node_config):
     restore_config(request, node_config)
 
     from bigchaindb.web import server
-    app = server.create_app({'debug': True})
+    app = server.create_app(debug=True)
     return app
 
 
@@ -33,5 +33,5 @@ def app(request, node_config):
 # NOTE: In order to have a database setup as well as the `input` fixture,
 #       we have to proxy `db.conftest.input` here.
 # TODO: If possible replace this function with something nicer.
-def inputs(user_vk):
-    conftest.inputs(user_vk)
+def inputs(user_pk):
+    conftest.inputs(user_pk)

tests/web/test_transactions.py

@@ -8,8 +8,8 @@ TX_ENDPOINT = '/api/v1/transactions/'
 
 
 @pytest.mark.usefixtures('inputs')
-def test_get_transaction_endpoint(b, client, user_vk):
-    input_tx = b.get_owned_ids(user_vk).pop()
+def test_get_transaction_endpoint(b, client, user_pk):
+    input_tx = b.get_owned_ids(user_pk).pop()
     tx = b.get_transaction(input_tx.txid)
     res = client.get(TX_ENDPOINT + tx.id)
     assert tx.to_dict() == res.json
@@ -33,8 +33,8 @@ def test_post_create_transaction_endpoint(b, client):
     tx = tx.sign([user_priv])
 
     res = client.post(TX_ENDPOINT, data=json.dumps(tx.to_dict()))
-    assert res.json['transaction']['fulfillments'][0]['owners_before'][0] == user_pub
-    assert res.json['transaction']['conditions'][0]['owners_after'][0] == user_pub
+    assert res.json['fulfillments'][0]['owners_before'][0] == user_pub
+    assert res.json['conditions'][0]['owners_after'][0] == user_pub
 
 
 def test_post_create_transaction_with_invalid_id(b, client):
@@ -43,7 +43,7 @@ def test_post_create_transaction_with_invalid_id(b, client):
 
     tx = Transaction.create([user_pub], [([user_pub], 1)])
     tx = tx.sign([user_priv]).to_dict()
-    tx['id'] = 'invalid id'
+    tx['id'] = 'abcd' * 16
 
     res = client.post(TX_ENDPOINT, data=json.dumps(tx))
     assert res.status_code == 400
@@ -55,20 +55,25 @@ def test_post_create_transaction_with_invalid_signature(b, client):
 
     tx = Transaction.create([user_pub], [([user_pub], 1)])
     tx = tx.sign([user_priv]).to_dict()
-    tx['transaction']['fulfillments'][0]['fulfillment'] = 'invalid signature'
+    tx['fulfillments'][0]['fulfillment'] = 'cf:0:0'
 
     res = client.post(TX_ENDPOINT, data=json.dumps(tx))
     assert res.status_code == 400
 
 
+def test_post_create_transaction_with_invalid_structure(client):
+    res = client.post(TX_ENDPOINT, data='{}')
+    assert res.status_code == 400
+
+
 @pytest.mark.usefixtures('inputs')
-def test_post_transfer_transaction_endpoint(b, client, user_vk, user_sk):
-    sk, vk = crypto.generate_key_pair()
+def test_post_transfer_transaction_endpoint(b, client, user_pk, user_sk):
+    sk, pk = crypto.generate_key_pair()
     from bigchaindb.models import Transaction
 
     user_priv, user_pub = crypto.generate_key_pair()
 
-    input_valid = b.get_owned_ids(user_vk).pop()
+    input_valid = b.get_owned_ids(user_pk).pop()
     create_tx = b.get_transaction(input_valid.txid)
     transfer_tx = Transaction.transfer(create_tx.to_inputs(),
                                        [([user_pub], 1)], create_tx.asset)
@@ -76,17 +81,17 @@ def test_post_transfer_transaction_endpoint(b, client, user_vk, user_sk):
 
     res = client.post(TX_ENDPOINT, data=json.dumps(transfer_tx.to_dict()))
 
-    assert res.json['transaction']['fulfillments'][0]['owners_before'][0] == user_vk
-    assert res.json['transaction']['conditions'][0]['owners_after'][0] == user_pub
+    assert res.json['fulfillments'][0]['owners_before'][0] == user_pk
+    assert res.json['conditions'][0]['owners_after'][0] == user_pub
 
 
 @pytest.mark.usefixtures('inputs')
-def test_post_invalid_transfer_transaction_returns_400(b, client, user_vk, user_sk):
+def test_post_invalid_transfer_transaction_returns_400(b, client, user_pk, user_sk):
     from bigchaindb.models import Transaction
 
     user_priv, user_pub = crypto.generate_key_pair()
 
-    input_valid = b.get_owned_ids(user_vk).pop()
+    input_valid = b.get_owned_ids(user_pk).pop()
     create_tx = b.get_transaction(input_valid.txid)
     transfer_tx = Transaction.transfer(create_tx.to_inputs(),
                                        [([user_pub], 1)], create_tx.asset)
@@ -96,8 +101,8 @@ def test_post_invalid_transfer_transaction_returns_400(b, client, user_vk, user_
 
 
 @pytest.mark.usefixtures('inputs')
-def test_get_transaction_status_endpoint(b, client, user_vk):
-    input_tx = b.get_owned_ids(user_vk).pop()
+def test_get_transaction_status_endpoint(b, client, user_pk):
+    input_tx = b.get_owned_ids(user_pk).pop()
     tx, status = b.get_transaction(input_tx.txid, include_status=True)
     res = client.get(TX_ENDPOINT + input_tx.txid + "/status")
     assert status == res.json['status']