pepys_import.file package

Subpackages

• pepys_import.file.highlighter package
  • Subpackages
    • pepys_import.file.highlighter.support package
      • Submodules
      • pepys_import.file.highlighter.support.char module
      • pepys_import.file.highlighter.support.color_picker module
      • pepys_import.file.highlighter.support.combine module
      • pepys_import.file.highlighter.support.export module
      • pepys_import.file.highlighter.support.line module
      • pepys_import.file.highlighter.support.test_utils module
      • pepys_import.file.highlighter.support.token module
      • pepys_import.file.highlighter.support.usages module
      • Module contents
  • Submodules
  • pepys_import.file.highlighter.highlighter module
  • Module contents

Submodules

pepys_import.file.file_processor module

class pepys_import.file.file_processor.FileProcessor(filename=None, archive=False, skip_validation=False, archive_path=None, local_parsers=None)

Bases: object

static display_import_summary(import_summary)

static get_file_contents(full_path: str)

static get_first_line(file_path: str)

Retrieve the first line from the file

Parameters

file_path (String) – Full file path

Returns

First line of text

Return type

String

load_importers_dynamically(path='/home/docs/checkouts/readthedocs.org/user_builds/pepys-import/checkouts/latest/importers')

Dynamically adds all the importers in the given path.

It loads core importers by default.

Parameters

path (String) – Path of a folder that has importers

process(path: str, data_store: Optional[pepys_import.core.store.data_store.DataStore] = None, descend_tree: bool = True)

Process the data in the given path

Parameters

• path (String) – File/Folder path

• data_store (DataStore) – Database

• descend_tree (bool) – Whether to recursively descend through the folder tree

process_file(file_object, current_path, data_store, processed_ctr, import_summary, file_number, total_files)

register_importer(importer)

Adds the supplied importer to the list of import modules

Parameters

importer – An importer module that must define the functions defined

in the Importer base class :type importer: Importer

pepys_import.file.importer module

class pepys_import.file.importer.Importer(name, validation_level, short_name, datafile_type, default_privacy=None)

Bases: abc.ABC

abstract can_load_this_file(file_contents) → bool

Whether this parser can handle this whole file

Parameters

file_contents (String) – Whole file contents

Returns

True/False

Return type

bool

abstract can_load_this_filename(filename) → bool

Whether this importer can load a file with the provided filename

Parameters

filename (String) – Full filename

Returns

True/False

Return type

bool

abstract can_load_this_header(header) → bool

Whether this importer can load a file with this first line of text

Parameters

header (String) – The initial line of text

Returns

True/False

Return type

bool

abstract can_load_this_type(suffix) → bool

Whether this importer can load files with the specified suffix.

Parameters

suffix (String) – File suffix (e.g. “.doc”)

Returns

True/False

Return type

bool

get_cached_platform(data_store, platform_name, change_id, quadgraph=None, unknown=False)

get_cached_sensor(data_store, sensor_name, sensor_type, platform_id, change_id)

Gets the sensor object for the given platform - either from the cache if it exists there, or resolving it.

If the sensor_name and sensor_type are both None, and this platform_id is present in the self.platform_sensor_mapping dict, then get the Sensor object from there - otherwise use the resolver to resolve it.

Parameters

data_store – DataStore instance :type data_Store: DataStore :param platform_id: ID of

the platform for which you want to get the sensor :type platform_id: int

load_this_file(data_store, path, file_object, datafile, change_id)

Loads the specified file.

Performs the common administrative operations that must be performed before the _load_this_file method is called, then calls that function.

Parameters

• data_store (DataStore) – The DataStore object connected to the database into which the file should be loaded

• path (String) – The full path to the file to import

• file_object (HighlightedFile) – The HighlightedFile object representing the file, allowing access to lines and tokens

• datafile (Datafile) – The Datafile object referring to the file to be imported

• change_id (Integer or UUID) – ID of the Change object that represents the change which will occur when importing this file

set_highlighting_level(level)

Sets the HighlightLevel of recording highlighted extractions. Can be one of:

• NONE: No highlighting or recording of extractions

• HTML: Produce a highlighted html file showing the extractions

• DATABASE: Produce a highlighted html file and record extractions to the database

Module contents