corelay.io.storage

‘io module contains classes to load and dump different files like hdf5, etc.

Classes

DataStorageBase	Implements a key, value storage object.
HDF5Storage	HDF5 storage that stores data under different keys.
HashedHDF5	Hashed storage of Processor data in HDF5 files
NoStorage	Stub class when no Storage is used.
PickleStorage	Experimental pickle storage that uses pickle to store data.
Storable	Abstract class to check for write/ read attributes via isinstance
StorableMeta	Meta class to check for write/ read attributes via isinstance

Exceptions

NoDataSource	Raise when no data source available.
NoDataTarget	Raise when no target source available.

class corelay.io.storage.DataStorageBase(**kwargs)

Bases: Plugboard

Implements a key, value storage object.

at(**kwargs)

Return a copy of the instance where kwargs become the attributes of the class. I.e. a specific self.data_key is set so that self.write(data) automatically writes the data to correct key.

close()

Close opened io file object.

abstract exists()

Return True if data exists.

abstract keys()

Return keys of the io file object.

abstract read(data_in=None, meta=None)

Should implement read functionality.

abstract write(data_out, data_in=None, meta=None)

Should implement write functionality.

class corelay.io.storage.HDF5Storage(path, mode='r', **kwargs)

Bases: DataStorageBase

HDF5 storage that stores data under different keys.

exists()

Returns True if key exists in self.io.

keys()

Return keys of the storage.

read(data_in=None, meta=None)

Return type:

data for a given key

write(data_out, data_in=None, meta=None)

Parameters:

data_out (np.ndarray, dict) – Data being stored. Dictionaries are pickled and stored as strings.

class corelay.io.storage.HashedHDF5(h5group)

Bases: object

Hashed storage of Processor data in HDF5 files

read(data_in, meta)

Read output from a hashed h5 group, with hash of (data_in, meta)

write(data_out, data_in, meta)

Write output to a hashed h5 group, with hash of (data_in, meta)

exception corelay.io.storage.NoDataSource(message='No Data Source available.')

Bases: Exception

Raise when no data source available.

exception corelay.io.storage.NoDataTarget

Bases: Exception

Raise when no target source available.

class corelay.io.storage.NoStorage(**kwargs)

Bases: DataStorageBase

Stub class when no Storage is used.

exists()

Return True if data exists.

keys()

Return keys of the io file object.

read(data_in=None, meta=None)

Should implement read functionality.

write(data_out, data_in=None, meta=None)

Should implement write functionality.

class corelay.io.storage.PickleStorage(path, mode='r', **kwargs)

Bases: DataStorageBase

Experimental pickle storage that uses pickle to store data.

exists()

Return True if key exists in self.keys().

keys()

Return keys from self.data. Need to load the complete pickle at first read.

read(data_in=None, meta=None)

Return data for a given key. Need to load the complete pickle at first read. After the data is cached.

Return type:

data for a given key

write(data_out, data_in=None, meta=None)

Write and pickle the data as: {“data”: data, “key”: key}

Parameters:

data_out (np.ndarray, dict) – Data being stored.

class corelay.io.storage.Storable

Bases: object

Abstract class to check for write/ read attributes via isinstance

class corelay.io.storage.StorableMeta

Bases: type

Meta class to check for write/ read attributes via isinstance