The exact API of all methods and classes, generated thanks to the docstrings.

populse_db documentation master file, created by
sphinx-quickstart on Thu Jul  5 16:12:12 2018.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

Populse_db is ensured to work on the platforms Linux, OSX, and Windows

The relational schema of populse_db is generic, and completely dynamic.

The methods of the API populse_db mainly allow you to add and remove collections, fields, and documents.

Beware that all table and column names (except collection and field tables) are hashed with md5, in order to avoid issues with forbidden characters.

When a collection is added, it is added to the list of collections, and the collection table is created.

When a document is added to a collection, a row is inserted in the collection table.

When a field is added to a collection, it is added in the list of fields, and a column is created in the collection table.

Exact call: session.add_field("collection_test", "field", populse_db.database.FIELD_TYPE_STRING, "field test")

Exact call: session.add_field(“collection_test”, “field”, populse_db.database.FIELD_TYPE_STRING, “field test”)

When a value is added to <collection, field, document>, the corresponding cell is set.

Exact call: session.new_value("collection_test", "document", "field", "new_value")

Exact call: session.new_value(“collection_test”, “document”, “field”, “new_value”)

This documentation website has been generated with Sphinx (See Sphinx website for more informations `here <http://www.sphinx-doc.org/en/master/>`__)

This documentation website has been generated with Sphinx (See Sphinx website for more informations 

The source code of this website is in docs/ directory (The website is actually in docs/html/ directory, but docs/index.html is redirecting to the website)

cd docs/
make html # The website generated will be in docs/html/ directory
cd ..

The website is deployed automatically thanks to GitHub pages (More informations are available on their website `here <https://pages.github.com/>`__)

The website is deployed automatically thanks to GitHub pages (More informations are available on their website 

The option is available in the GitHub repository populse_db settings: See GitHub Pages subsection in Options section, it is currently set to master branch /docs folder

The empty file docs/.nojekyll must be there in order to have CSS on the deployed GitHub website at https://populse.github.io/populse_db/

The empty file docs/.nojekyll must be there in order to have CSS on the deployed GitHub website at 

The structure of the website that can be modified is in docs/source/ directory (for example, docs/source/index.rst is this current page)

In order to install Sphinx, the following commands have been executed (from populse_db root directory):

pip install sphinx
mkdir docs/
cd docs/
sphinx-quickstart
"""
Default values, except the following ones:
 - Separate source and build directories (y/n) [n]: y
 - Project name: populse_db
 - Author name(s): populse
 - Project version: 1
 - Project release: 1.0.0
 - autodoc: automatically insert docstrings from modules (y/n) [n]: y
 - Create Windows command file? (y/n) [y]: n
"""
"""
The following line (22) must be uncommented from docs/source/conf.py:
 - sys.path.insert(0, os.path.abspath('../..'))
The following line (8) must be modified from docs/Makefile:
 - BUILDDIR      = BUILD  =>  BUILDDIR      = .
"""
sphinx-apidoc -f -o source/ ../python/populse_db/

After that, the documentation can be modified in docs/source directory, and built with make html command in docs/ directory

The documentation is directly using the docstring from the source code to automatically generate the API

Populse_db is precisely released under the CeCILL-B software license.

You can find all the details on the license `here
<http://www.cecill.info/licences/Licence_CeCILL-B_V1-en.html>`__, or refer to the license file `here
<https://github.com/populse/populse_db/blob/master/LICENSE.en>`__.

Database(database_url, caches=None, list_tables=None, query_type=None)

attributes:
- string_engine: String engine of the database
- engine: database engine

methods:
- __enter__: Creates or gets a DatabaseSession instance
- __exit__: Releases the latest created DatabaseSession
- clear: Clears the database

database_url -- Database engineThe engine is constructed this way: dialect://user:password@host/dbname[?key=value..]The dialect can be sqlite or postgresqlFor sqlite databases, the file can be not existing yet, it will be created in this caseExamples:

"sqlite:///foo.db"

"postgresql://scott:tiger@localhost/test"

The engine is constructed this way: dialect://user:password@host/dbname[?key=value..]

For sqlite databases, the file can be not existing yet, it will be created in this case

Examples:
- "sqlite:///foo.db"
- "postgresql://scott:tiger@localhost/test"

list_tables -- obsolete parameter kept for backward compatibility

query_type -- obsolete parameter kept for backward compatibility

ValueError -- If database_url is invalid

If the schema is not coherent with the API (the database is not a populse_db database)

- database: Database instance
- session: Session related to the database
- table_classes: List of all table classes, generated automatically
- base: Database base
- metadata: Database metadata

table_classes: List of all table classes, generated automatically

methods:
- add_collection: Adds a collection
- remove_collection: Removes a collection
- get_collection: Gives the collection row
- get_collections: Gives all collection rows
- get_collections_names: Gives all collection names
- add_field: Adds a field to a collection
- add_fields: Adds a list of fields to a collection
- remove_field: Removes a field from a collection
- get_field: Gives all fields rows given a collection
- get_fields_names: Gives all fields names given a collection
  to the name
- get_value: Gives the value of <collection, document, field>
- set_value: Sets the value of <collection, document, field>
- set_values: Sets several values of <collection, document, field>
- remove_value: Removes the value of <collection, document, field>
- add_value: Adds a value to <collection, document, field>
- get_document: Gives the document row given a document name and a collection
- get_documents: Gives all document rows given a collection
- get_documents_names: Gives all document names given a collection
- add_document: Adds a document to a collection
- remove_document: Removes a document from a collection
- save_modifications: Saves the pending modifications
- unsave_modifications: Unsaves the pending modifications
- has_unsaved_modifications: To know if there are unsaved
  modifications
- filter_documents: Gives the list of documents matching the filter

get_fields_names: Gives all fields names given a collection
to the name

set_values: Sets several values of <collection, document, field>

remove_value: Removes the value of <collection, document, field>

get_document: Gives the document row given a document name and a collection

get_documents_names: Gives all document names given a collection

has_unsaved_modifications: To know if there are unsaved
modifications

filter_documents: Gives the list of documents matching the filter

primary_key -- New collection primary_key column (str) => "index" by default

ValueError -- If the collection is already existing

If the collection name is invalid

If the primary_key is invalid

DatabaseSession.add_document(collection, document, create_missing_fields=True, flush=None)

document -- Dictionary of document values (dict), or document primary_key (str)The primary_key must not be existing

Dictionary of document values (dict), or document primary_key (str)

create_missing_fields -- Boolean to know if the missing fields must be createdIf True, fields that are in the document but not in the collection are created if the type can be guessed from the value in the document
(possible for all valid values except None and []).

If True, fields that are in the document but not in the collection are created if the type can be guessed from the value in the document

ValueError -- If the collection does not exist

If the document already exists

If document is invalid (invalid name or no primary_key)

DatabaseSession.add_field(collection, name, field_type, description=None, index=False, flush=None)

collection, name, field_type, description=None, index=False, flush=None

field_type -- Field type, in ('string', 'int', 'float', 'boolean', 'date', 'datetime',
'time', 'json', 'list_string', 'list_int', 'list_float', 'list_boolean', 'list_date',
'list_datetime', 'list_time', 'list_json')

Field type, in (‘string’, ‘int’, ‘float’, ‘boolean’, ‘date’, ‘datetime’,
‘time’, ‘json’, ‘list_string’, ‘list_int’, ‘list_float’, ‘list_boolean’, ‘list_date’,
‘list_datetime’, ‘list_time’, ‘list_json’)

description -- Field description (str or None) => None by default

index -- Bool to know if indexing must be done => False by default

If the field already exists

If the field name is invalid

If the field type is invalid

If the field description is invalid

DatabaseSession.add_value(collection, document_id, field, value, checks=True)

If the field does not exist

If the document does not exist

If the value is invalid

If <collection, document_id, field> already has a value

check_value_type() (populse_db.database.DatabaseSession class method)

DatabaseSession.filter_documents(collection, filter_query, fields=None, as_list=False)

filter_query can be the result of self.filter_query() or a string containing a filter
(in this case self.fliter_query() is called to get the actual query)

filter_query -- Filter query (str)A filter row must be written this way: {<field>} <operator> "<value>"

The operator must be in ('==', '!=', '<=', '>=', '<', '>', 'IN', 'ILIKE', 'LIKE')

The filter rows can be linked with ' AND ' or ' OR '

Example: "((({BandWidth} == "50000")) AND (({FileName} LIKE "%G1%")))"

A filter row must be written this way: {<field>} <operator> "<value>"

A filter row must be written this way: {<field>} <operator> “<value>”

The operator must be in (‘==’, ‘!=’, ‘<=’, ‘>=’, ‘<’, ‘>’, ‘IN’, ‘ILIKE’, ‘LIKE’)

Example: “((({BandWidth} == “50000”)) AND (({FileName} LIKE “%G1%”)))”

get_collections_names() (populse_db.database.DatabaseSession method)

DatabaseSession.get_document(collection, document_id, fields=None, as_list=False)

Gives a Document instance given a collection and a document identifier

DatabaseSession.get_documents(collection, fields=None, as_list=False, document_ids=None)

List of all document rows of the collection if it exists, None otherwise

get_documents_names() (populse_db.database.DatabaseSession method)

List of all document names of the collection if it exists, None otherwise

List of all fields rows of the collection if it exists, None otherwise

List of all fields names of the collection if it exists, None otherwise

The current value of <collection, document, field> if it exists, None otherwise

remove_collection() (populse_db.database.DatabaseSession method)

field -- Field name (str, must be existing), or list of fields (list of str, must all be existing)

Field name (str, must be existing), or list of fields (list of str, must all be existing)

DatabaseSession.remove_value(collection, document_id, field, flush=None)

DatabaseSession.set_value(collection, document_id, field, new_value, flush=None)

Sets the value associated to <collection, document, field> if it exists

If trying to set the primary_key

DatabaseSession.set_values(collection, document_id, values, flush=None)

If the values are invalid

Reprsents a list of value of fixed size with a key string for each value.
It allows to access to values with their index or with their key.
It is also possible to acess to values as attributes.
The function list_with_keys() is used to create derived classes
with a fixed set of item names.

Return a new instance of ListWithNames with
a given list of keys

Returns the field type corresponding to a Python value.
This type can be used in add_field(s) method.
For list values, only the first item is considered to get the type.
Type cannot be determined for empty list.
If value is None, the result is None.

Instance of this class are passed to Lark parser when parsing a document
selection filter string in order to create an object that can be used to
select items from the database. FilterToQuery implements methods that are
common to all engines and does not produce anything because the query
objectis specific to each engine. Therefore, engine class must use a 
subclass of FilterToQuery that implements the following methods:

build_condition_all
build_condition_literal_in_list_field
build_condition_field_in_list_field
build_condition_field_in_list
build_condition_field_op_field
build_condition_value_op_field
build_condition_negation
build_condition_combine_conditions

Return a selection query that select all documents. This query
is directly given to the engine and never combined with other
queries.

build_condition_combine_conditions() (populse_db.filter.FilterToQuery method)

Builds a condition that combines two conditions with an operator.

left_condition -- condition object returned by one of the 
build_condition_*() method (except 
build_condition_all)

condition object returned by one of the 

operator_str -- string containing one of the BOOLEAN_OPERATOR
defined in the grammar (in lowercase)

string containing one of the BOOLEAN_OPERATOR

right_condition -- condition object returned by one of the 

build_condition_field_in_list() (populse_db.filter.FilterToQuery method)

Builds a condition checking if a field value is a
constant list value

build_condition_field_in_list_field() (populse_db.filter.FilterToQuery method)

Builds a condition checking if a field value is in another
list field value

build_condition_field_op_field() (populse_db.filter.FilterToQuery method)

Builds a condition comparing the content of two fields with an operator.

operator -- string containing one of the CONDITION_OPERATOR

string containing one of the CONDITION_OPERATOR

build_condition_field_op_value() (populse_db.filter.FilterToQuery method)

FilterToQuery.build_condition_field_op_value(field, operator_str, value)

Builds a condition comparing the content of a field with a constant
value using an operator.

operator_str -- string containing one of the CONDITION_OPERATOR

value -- Python value (None, string number, boolean or date/time)

build_condition_literal_in_list_field() (populse_db.filter.FilterToQuery method)

Builds a condition checking if a constant value is in a list field

build_condition_negation() (populse_db.filter.FilterToQuery method)

condition -- condition object returned by one of the 

build_condition_value_op_field() (populse_db.filter.FilterToQuery method)

FilterToQuery.build_condition_value_op_field(value, operator_str, field)

Builds a condition comparing a constant value with the content of a
field withusing an operator.

A singleton instance of Lark grammar parser for filter expression

An instance of Lark grammar parser for parsing only a literal
value (int, string, list, date, etc.) from a filter expression. This
is used for testing the parsing of these literals.

Create an instance of the class that will use the named test
method when executed. Raises a ValueError if the instance does
not have a method with the specified name.

test_add_get_document() (populse_db.test.TestsSQLiteInMemory method)

This Python package contains engines that contains the minimum API to connect
an external database engine to a Populse database.

Base class for database engines. All methods of this base class raise
NotImplementedError. This class exists to list all the methods that
must be implemented by an engine.

Parameters passed to __init__ may differ between engine classes.
It is up to engine_factory to extract appropriate parameter(s)
from URL.

Create a new collection given the name of a field that will be created
with a text type. This field is the primary key of each document. It
means that, in a single collection, a value must uniquely identify a 
document. In other word, two documents cannot have the same value for
this field.

type -- field type, in ('string', 'int', 'float', 
'boolean', 'date', 'datetime', 'time', 'json', 'list_string',
'list_int', 'list_float', 'list_boolean', 'list_date',

field type, in (‘string’, ‘int’, ‘float’, 
‘boolean’, ‘date’, ‘datetime’, ‘time’, ‘json’, ‘list_string’,
‘list_int’, ‘list_float’, ‘list_boolean’, ‘list_date’,

index -- boolean indicating if a database index must be created
for this field to speed-ud queries involving values for this field.

boolean indicating if a database index must be created

Returns a Row Object with at least the following items:
collection_name: name of the collection
primary_key: name of the primary key field
More engine specific items may be present but they must
not be used in the general API.
Returns None if the collection does not exist.

Return an iterable (e.g. list, generator, etc.) browsing the
collections presents in the database. See collection() method
for the format of each collection object.

Store in the database all modifications done since the last call to
either __enter__, commit or rollback.

Engine.document(collection, document_id, fields=None, as_list=False)

Returns a Row Object corresponding to a document in the collection.
The object has one item per selected fields. If fields is not given,
fields returned by the fields() method are used.
If as_list is True, a list of values is returned (one value per selected
field).
Returns None if the collection or document does not exist.

document_id -- document identifier: the value of the primary
key field (str)

fields -- list of fields to get values from (other fields are
ignored) (list of str or None)

list of fields to get values from (other fields are

as_list -- if True, return a list of values insted of a Row Object (str)

Returns a Row Object corresponding to a collection field with at
least the following items:
field_name: field name
field_type: type of values for this field
description: text describing the field usage
has_index: boolean indicating if an index was created for this field
Returns None if the collection or the field does not exist.

fields presents in the given collection (or all collections if
collection parameter is None). See field() method
for the format of each field object.

Engine.filter_documents(parsed_filter, fields=None, as_list=False)

Iterate over document selected by a filter. See document() method
for the format of a document object.

parsed_filter -- internal object representing a filter on a
collection (filter object returned by parse_filter())

internal object representing a filter on a

Checks existence of a collection. May be called often,
must be fast.

Checks existence of a field in a collection. May be called often,
Returns False if collection does not exist.

Given a filter string, return a internal query representation that
can be used with filter_documents() to select documents

collection -- the collection for which the filter is intended 
(str, must be existing)

the collection for which the filter is intended 

filter -- the selection string using the populse_db selection
language.

Return the name of the primary key of a collection.

Remove given fields from a collection as well as all corresponding data.

Store discards all database modifications done since the last call to

Method that interprets an URL and creates the corresponding database
engine.

Implements required methods to produce a SQLite query given a document
selection filter. This class returns either None (all documents are 
selected) or an SQL WHERE clause (without the WHERE keyword) as a list 
of string (that must be joined with spaces). This WHERE clause is useable 
with a SELECT from the table containing the collection documents. Using a 
list for combining strings is supposed to be more efficient (especially 
for long queries).

build_condition_all() (populse_db.engine.sqlite.FilterToSqliteQuery method)

FilterToSqliteQuery.build_condition_field_op_value(field, operator_str, value)

build_condition_negation() (populse_db.engine.sqlite.FilterToSqliteQuery method)

FilterToSqliteQuery.build_condition_value_op_field(value, operator_str, field)

get_column() (populse_db.engine.sqlite.FilterToSqliteQuery method)

get_column_value() (populse_db.engine.sqlite.FilterToSqliteQuery method)

Converts a Python value to a value suitable to put in a database column

no_list_operator (populse_db.engine.sqlite.FilterToSqliteQuery attribute)

sql_operators (populse_db.engine.sqlite.FilterToSqliteQuery attribute)

SQLiteEngine.add_document(collection, document, create_missing_fields)

SQLiteEngine.add_field(collection, field, type, description, index)

column_to_python() (populse_db.engine.sqlite.SQLiteEngine static method)

SQLiteEngine.document(collection, document_id, fields=None, as_list=False)

filter_documents() (populse_db.engine.sqlite.SQLiteEngine method)

SQLiteEngine.filter_documents(parsed_filter, fields=None, as_list=False)

list_hash() (populse_db.engine.sqlite.SQLiteEngine static method)

Transforms the name into a valid and unique SQLite table/column name.
Since all names are quoted in SQL with '[]', there is no restriction
on character that can be used. However, case is insignificant in
SQLite. Therefore, all upper case characters are prefixed with '!'.

Since all names are quoted in SQL with ‘[]’, there is no restriction
SQLite. Therefore, all upper case characters are prefixed with ‘!’.

python_to_column() (populse_db.engine.sqlite.SQLiteEngine static method)

Converts a python value into a suitable value to put in a
database column.

remove_collection() (populse_db.engine.sqlite.SQLiteEngine method)

remove_document() (populse_db.engine.sqlite.SQLiteEngine method)

{'boolean': 'BOOLEAN', 'date': 'TEXT', 'datetime': 'TEXT', 'float': 'REAL', 'int': 'INT', 'json': 'STRING', 'string': 'TEXT', 'time': 'TEXT'}

# A compatible version of Python and Pip must be installed

To install pip:
sudo apt-get install python-pip # Python2
sudo apt-get install python3-pip # Python3

# To install the module:
sudo pip install populse-db # Beware that it is the Pip version corresponding to the good Python version
sudo pipx.x install populse-db # With a precise Pip version
sudo pythonx.x -m pip install populse-db # With a precise Python version

# A compatible version of Python must be installed
sudo apt-get install git
git clone https://github.com/populse/populse_db.git /tmp/populse_db
cd /tmp/populse_db
sudo python setup.py install # Beware that it is the good Python version (use pythonx.x to be sure)
cd /tmp
sudo rm -r /tmp/populse_db

import populse_db
from populse_db.database import Database
import populse_db.database

Continuous integration has been deployed with Travis (Linux, OSX), and AppVeyor (Windows)

The script of tests is python/populse_db/test.py, so the following command launches the tests:

python test.py # (if python/populse_db/ directory has been added to $PATH, or if $PWD in the terminal)
python python/populse_db/test.py # (from populse_db root directory)
python -m populse_db.test

Small script to show the initialization and some examples of calls:

import os
import tempfile

import shutil

from populse_db.database import Database, FIELD_TYPE_STRING, FIELD_TYPE_INTEGER

# Generating the database in a temp directory
temp_folder = tempfile.mkdtemp()
path = os.path.join(temp_folder, "test.db")
database_url = 'sqlite:///' + path
database = Database(database_url)

# Creating the session and working with it
with database as session:

    # Creating a profile table
    session.add_collection("Profile")

    # Adding several properties
    session.add_field("Profile", "First name", FIELD_TYPE_STRING)
    session.add_field("Profile", "Last name", FIELD_TYPE_STRING)
    session.add_field("Profile", "Age", FIELD_TYPE_INTEGER)

    # Filling the table
    profile1 = {}
    profile1["index"] = "profile1"
    profile1["First name"] = "Jules"
    profile1["Last name"] = "CESAR"
    profile1["Age"] = 55
    session.add_document("Profile", profile1)

    session.add_document("Profile", "profile2")
    session.add_value("Profile", "profile2", "First name", "Louis")
    session.add_value("Profile", "profile2", "Last name", "XIV")
    session.add_value("Profile", "profile2", "Age", 76)

    # Setting a value
    result = session.filter_documents("Profile", "({Age} > 50) AND ({First name} == \"Jules\")")
    for document in result: # profile1 is displayed, as it's the only document with the value Age greater than 50, and the value First name being Jules
        print(document.index)

shutil.rmtree(temp_folder)

The database file test.py resulting from the script is available `here <https://github.com/populse/populse_db/blob/master/docs/databases/test.db>`_.

The database file test.py resulting from the script is available 

As you can see in the collection table, the collection Profile has been created.

As you can see in the field table, the three fields First name, Last name, and Age have been created, in addition to the primary key.

As you can see in the collection profile table, the two profiles Jules CESAR and Louis XIV have been created.