Welcome to pepys-import


Pepys-import is a Python library to be used in the parsing and collation of data to be used in platform based spatial analysis.

The library provides tools to help work through text data-files, and then push data measurements (and supporting metadata) to a databaes.

PostGres and SQLite databases are currently supported, with SQLite being the low-friction option favoured for getting started with Pepys-Import, or for project maintainers.

Here’s a working demonstrator of datafile import: binder

Online documentation

Learn more here: docs


Code coverage

We’re aiming for 100% code coverage on the project, track our progress here: code_cov

Upstream security

We have continuous vulnerability testing on the Open Source libraries we depend upon for development: dev_req and production: plain_req

Code Style

Black is used on the project: black

It is suggested to install a pre-commit hook in order to apply Black before pushing commits:

$ pip install pre-commit
$ pre-commit install

Project Progress

View the project Kanban board here


To prepare for running ensure Python 3.6 or later are installed in your system. You can check your Python 3 version with the following command:

$ python3 --version

If you don’t have Python 3.6+ in your system, please download it from python.org

Pip is also necessary to create virtual environment. If you don’t have it in your system, please download it:

$ sudo apt-get install python3-pip

It is possible to verify pip installation with the following command:

$ pip3 --version

Ubuntu 18.04 LTS Development Instructions

Installing Spatialite

1. Easy Way

Open a terminal and run the following command:

$ sudo apt install spatialite-bin
$ sudo apt-get install -y libsqlite3-mod-spatialite
2. Alternative Way (Compiling From Source)

On Debian-based distributions the following libraries are necessary to run SpatiaLite:

$ sudo apt install zlib1g-dev libfreexl1
$ sudo apt-get install sqlite3 libsqlite3-dev
$ sudo apt-get install -y libsqlite3-mod-spatialite
$ sudo apt-get install binutils libproj-dev gdal-bin libgeos-dev libxml2-dev

After all libraries are installed, it is necessary to download and run the latest SpatiaLite version:

$ wget https://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-X.Y.Z.tar.gz
$ tar xaf libspatialite-X.Y.Z.tar.gz
$ cd libspatialite-X.Y.Z
$ ./configure
$ make
$ sudo make install

Installing PostGIS

The best way to install PostGIS is running the codes as follows:

sudo apt-get install libpq-dev python-dev
sudo add-apt-repository ppa:ubuntugis/ubuntugis-unstable
sudo apt-get update
sudo apt-get install postgis

Cloning the Repository

Navigate to folder you would like to download the repository and run the following commands:

$ sudo apt install git
$ git clone https://github.com/debrief/pepys-import.git

Creating Python Environment

Virtual Environment might be used to run the project. For creating a proper one, the following commands must be executed respectively in your project directory (please change the version with your Python version):

$ python3.6 -m virtualenv venv

Note: If you downloaded virtualenv in the current session, virtualenv command won’t work, please try to run it after rebooting your machine.

When environment is created successfully, please run the following commands:

$ source venv/bin/activate
$ pip install -r requirements.txt

Unit tests

  • In order to run the tests, please install requirements_dev: pip install -r requirements_dev.txt
  • Run the unit test suite with: coverage3 run -m unittest discover -v
  • View the unit test coverage with: coverage report

Command Line Instructions

To run from the command line go to the top level directory of the library in your bash shell or terminal program

Run by specifying the program as a module with -m and leaving off the .py file extension

The exact executable name for invoking python will depend how you have it installed, but most commonly it’s just python

For example run the Sqlite example using: python -m Experiments.DataStore_sqliteExperiment

IntelliJ Instructions

To run from inside IntelliJ open the project Mark the Store package as source by right clicking on the directory and selecting Mark Directory as -> Source Root

Open any python module you want to run in the main editor window, right click anywhere in the editor and choose the Run or Debug option

Indices and tables