populse_db.engine package

This Python package contains engines that contains the minimum API to connect an external database engine to a Populse database.

class populse_db.engine.Engine

Bases: object

Base class for database engines. All methods of this base class raise NotImplementedError. This class exists to list all the methods that must be implemented by an engine.

Parameters passed to __init__ may differ between engine classes. It is up to engine_factory to extract appropriate parameter(s) from URL.

add_collection(collection, primary_key)

Create a new collection given the name of a field that will be created with a text type. This field is the primary key of each document. It means that, in a single collection, a value must uniquely identify a document. In other word, two documents cannot have the same value for this field.

Parameters
  • collection – collection name (str)

  • primary_key – name of the primary key field (str)

add_field(collection, field, type, description, index)

Adds a new field in a collection.

Parameters
  • collection – collection name (str, must be existing)

  • field – new field name (str, must not be existing)

  • type – field type, in (‘string’, ‘int’, ‘float’, ‘boolean’, ‘date’, ‘datetime’, ‘time’, ‘json’, ‘list_string’, ‘list_int’, ‘list_float’, ‘list_boolean’, ‘list_date’, ‘list_datetime’, ‘list_time’, ‘list_json’)

  • description – field description (str or None)

  • index – boolean indicating if a database index must be created for this field to speed-ud queries involving values for this field.

clear()

Erase database data and schema.

collection(collection)

Returns a Row Object with at least the following items: collection_name: name of the collection primary_key: name of the primary key field More engine specific items may be present but they must not be used in the general API. Returns None if the collection does not exist.

Parameters

collection – collection name (str)

collections()

Return an iterable (e.g. list, generator, etc.) browsing the collections presents in the database. See collection() method for the format of each collection object.

commit()

Store in the database all modifications done since the last call to either __enter__, commit or rollback.

document(collection, document_id, fields=None, as_list=False)

Returns a Row Object corresponding to a document in the collection. The object has one item per selected fields. If fields is not given, fields returned by the fields() method are used. If as_list is True, a list of values is returned (one value per selected field). Returns None if the collection or document does not exist.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

  • fields – list of fields to get values from (other fields are ignored) (list of str or None)

  • as_list – if True, return a list of values insted of a Row Object (str)

field(collection, field)

Returns a Row Object corresponding to a collection field with at least the following items: collection_name: name of the collection field_name: field name field_type: type of values for this field description: text describing the field usage has_index: boolean indicating if an index was created for this field More engine specific items may be present but they must not be used in the general API. Returns None if the collection or the field does not exist.

Parameters
  • collection – collection name (str)

  • field – field name (str)

fields(collection=None)

Return an iterable (e.g. list, generator, etc.) browsing the fields presents in the given collection (or all collections if collection parameter is None). See field() method for the format of each field object.

Parameters

collection – collection name (str)

filter_documents(parsed_filter, fields=None, as_list=False)

Iterate over document selected by a filter. See document() method for the format of a document object.

Parameters
  • parsed_filter – internal object representing a filter on a collection (filter object returned by parse_filter())

  • fields – list of fields to get values from (other fields are ignored) (list of str or None)

  • as_list – if True, return a list of values insted of a Row Object (str)

has_collection(collection)

Checks existence of a collection. May be called often, must be fast.

Parameters

collection – collection name (str)

has_document(collection, document_id)

Checks existence of a document in a collection.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

has_field(collection, field)

Checks existence of a field in a collection. May be called often, must be fast. Returns False if collection does not exist.

Parameters
  • collection – collection name (str)

  • field – field name (str)

has_value(collection, document_id, field)

Check if a document has a not null value for a given field.

parse_filter(collection, filter)

Given a filter string, return a internal query representation that can be used with filter_documents() to select documents

Parameters
  • collection – the collection for which the filter is intended (str, must be existing)

  • filter – the selection string using the populse_db selection language.

primary_key(collection)

Return the name of the primary key of a collection. Returns None if the collection does not exist.

Parameters

collection – collection name (str)

remove_collection(collection)

Delete a collection and its data.

Parameters

collection – collection name (str)

remove_document(collection, document_id)

Remove a document from a collection.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

remove_fields(collection, fields)

Remove given fields from a collection as well as all corresponding data.

Parameters
  • collection – collection name (str)

  • fields – field name (str)

remove_value(collection, document_id, field)

Remove a value from a document (setting its value to None).

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

  • fields – field name (str)

rollback()

Store discards all database modifications done since the last call to either __enter__, commit or rollback.

set_values(collection, document_id, values)

Change some values in an existing document.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

  • values – dictionary with field/value pairs (dict)

populse_db.engine.engine_factory(database_url)

Method that interprets an URL and creates the corresponding database engine.

Submodules

populse_db.engine.sqlite module

class populse_db.engine.sqlite.FilterToSqliteQuery(engine, collection)

Bases: populse_db.filter.FilterToQuery

Implements required methods to produce a SQLite query given a document selection filter. This class returns either None (all documents are selected) or an SQL WHERE clause (without the WHERE keyword) as a list of string (that must be joined with spaces). This WHERE clause is useable with a SELECT from the table containing the collection documents. Using a list for combining strings is supposed to be more efficient (especially for long queries).

Create a parser for a givent engine and collection

build_condition_all()

Return a selection query that select all documents. This query is directly given to the engine and never combined with other queries.

build_condition_combine_conditions(left_condition, operator_str, right_condition)

Builds a condition that combines two conditions with an operator.

Parameters
  • left_condition – condition object returned by one of the build_condition_*() method (except build_condition_all)

  • operator_str – string containing one of the BOOLEAN_OPERATOR defined in the grammar (in lowercase)

  • right_condition – condition object returned by one of the build_condition_*() method (except build_condition_all)

build_condition_field_in_list(field, list_value)

Builds a condition checking if a field value is a constant list value

Parameters
  • field – field object as returned by Database.get_field

  • list_value – Pyton list containing literals

build_condition_field_in_list_field(field, list_field)

Builds a condition checking if a field value is in another list field value

Parameters
  • field – field object as returned by Database.get_field

  • list_field – field object as returned by Database.get_field

build_condition_field_op_field(left_field, operator_str, right_field)

Builds a condition comparing the content of two fields with an operator.

Parameters
  • left_field – field object as returned by Database.get_field

  • operator – string containing one of the CONDITION_OPERATOR defined in the grammar (in lowercase)

  • right_field – field object as returned by Database.get_field

build_condition_field_op_value(field, operator_str, value)

Builds a condition comparing the content of a field with a constant value using an operator.

Parameters
  • field – field object as returned by Database.get_field

  • operator_str – string containing one of the CONDITION_OPERATOR defined in the grammar (in lowercase)

  • value – Python value (None, string number, boolean or date/time)

build_condition_literal_in_list_field(value, list_field)

Builds a condition checking if a constant value is in a list field

Parameters
  • value – Python literal

  • list_field – field object as returned by Database.get_field

build_condition_negation(condition)

Builds a condition inverting another condition.

Parameters

condition – condition object returned by one of the build_condition_*() method (except build_condition_all)

build_condition_value_op_field(value, operator_str, field)

Builds a condition comparing a constant value with the content of a field withusing an operator.

Parameters
  • value – Python value (None, string number, boolean or date/time)

  • operator_str – string containing one of the CONDITION_OPERATOR defined in the grammar (in lowercase)

  • field – field object as returned by Database.get_field

get_column(field)
Returns

The SQL representation of a field object.

get_column_value(python_value)

Converts a Python value to a value suitable to put in a database column

no_list_operator = {'<', '<=', '>', '>=', 'ilike', 'like'}
sql_operators = {'!=': 'IS NOT', '==': 'IS', 'ilike': 'LIKE'}
class populse_db.engine.sqlite.SQLiteEngine(database)

Bases: populse_db.engine.Engine

Parameters passed to __init__ may differ between engine classes. It is up to engine_factory to extract appropriate parameter(s) from URL.

add_collection(collection, primary_key)

Create a new collection given the name of a field that will be created with a text type. This field is the primary key of each document. It means that, in a single collection, a value must uniquely identify a document. In other word, two documents cannot have the same value for this field.

Parameters
  • collection – collection name (str)

  • primary_key – name of the primary key field (str)

add_document(collection, document, create_missing_fields)
add_field(collection, field, type, description, index)

Adds a new field in a collection.

Parameters
  • collection – collection name (str, must be existing)

  • field – new field name (str, must not be existing)

  • type – field type, in (‘string’, ‘int’, ‘float’, ‘boolean’, ‘date’, ‘datetime’, ‘time’, ‘json’, ‘list_string’, ‘list_int’, ‘list_float’, ‘list_boolean’, ‘list_date’, ‘list_datetime’, ‘list_time’, ‘list_json’)

  • description – field description (str or None)

  • index – boolean indicating if a database index must be created for this field to speed-ud queries involving values for this field.

clear()

Erase database data and schema.

collection(collection)

Returns a Row Object with at least the following items: collection_name: name of the collection primary_key: name of the primary key field More engine specific items may be present but they must not be used in the general API. Returns None if the collection does not exist.

Parameters

collection – collection name (str)

collections()

Return an iterable (e.g. list, generator, etc.) browsing the collections presents in the database. See collection() method for the format of each collection object.

static column_to_python(field_type, value)
commit()

Store in the database all modifications done since the last call to either __enter__, commit or rollback.

document(collection, document_id, fields=None, as_list=False)

Returns a Row Object corresponding to a document in the collection. The object has one item per selected fields. If fields is not given, fields returned by the fields() method are used. If as_list is True, a list of values is returned (one value per selected field). Returns None if the collection or document does not exist.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

  • fields – list of fields to get values from (other fields are ignored) (list of str or None)

  • as_list – if True, return a list of values insted of a Row Object (str)

field(collection, field)

Returns a Row Object corresponding to a collection field with at least the following items: collection_name: name of the collection field_name: field name field_type: type of values for this field description: text describing the field usage has_index: boolean indicating if an index was created for this field More engine specific items may be present but they must not be used in the general API. Returns None if the collection or the field does not exist.

Parameters
  • collection – collection name (str)

  • field – field name (str)

fields(collection=None)

Return an iterable (e.g. list, generator, etc.) browsing the fields presents in the given collection (or all collections if collection parameter is None). See field() method for the format of each field object.

Parameters

collection – collection name (str)

filter_documents(parsed_filter, fields=None, as_list=False)

Iterate over document selected by a filter. See document() method for the format of a document object.

Parameters
  • parsed_filter – internal object representing a filter on a collection (filter object returned by parse_filter())

  • fields – list of fields to get values from (other fields are ignored) (list of str or None)

  • as_list – if True, return a list of values insted of a Row Object (str)

has_collection(collection)

Checks existence of a collection. May be called often, must be fast.

Parameters

collection – collection name (str)

has_document(collection, document_id)

Checks existence of a document in a collection.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

has_field(collection, field)

Checks existence of a field in a collection. May be called often, must be fast. Returns False if collection does not exist.

Parameters
  • collection – collection name (str)

  • field – field name (str)

has_table(table)
has_value(collection, document_id, field)

Check if a document has a not null value for a given field.

static list_hash(list)
name_to_sql(name)

Transforms the name into a valid and unique SQLite table/column name. Since all names are quoted in SQL with ‘[]’, there is no restriction on character that can be used. However, case is insignificant in SQLite. Therefore, all upper case characters are prefixed with ‘!’.

Parameters

name – Name (str)

Returns

Valid and unique table/column name

parse_filter(collection, filter)

Given a filter string, return a internal query representation that can be used with filter_documents() to select documents

Parameters
  • collection – the collection for which the filter is intended (str, must be existing)

  • filter – the selection string using the populse_db selection language.

primary_key(collection)

Return the name of the primary key of a collection. Returns None if the collection does not exist.

Parameters

collection – collection name (str)

static python_to_column(field_type, value)

Converts a python value into a suitable value to put in a database column.

remove_collection(collection)

Delete a collection and its data.

Parameters

collection – collection name (str)

remove_document(collection, document_id)

Remove a document from a collection.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

remove_fields(collection, fields)

Remove given fields from a collection as well as all corresponding data.

Parameters
  • collection – collection name (str)

  • fields – field name (str)

remove_value(collection, document_id, field)

Remove a value from a document (setting its value to None).

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

  • fields – field name (str)

rollback()

Store discards all database modifications done since the last call to either __enter__, commit or rollback.

set_values(collection, document_id, values)

Change some values in an existing document.

Parameters
  • collection – collection name (str)

  • document_id – document identifier: the value of the primary key field (str)

  • values – dictionary with field/value pairs (dict)

sql_type(type)
type_to_sql = {'boolean': 'BOOLEAN', 'date': 'TEXT', 'datetime': 'TEXT', 'float': 'REAL', 'int': 'INT', 'json': 'STRING', 'string': 'TEXT', 'time': 'TEXT'}