bsb.storage package

Subpackages

Submodules

bsb.storage.interfaces module

class bsb.storage.interfaces.ConnectivityIterator(cs: ConnectivitySet, direction, lchunks=None, gchunks=None, scoped=True)

Bases: object

Parameters:

cs (ConnectivitySet)

all()

as_globals()

as_scoped()

chunk_iter()

Iterate over the connection data chunk by chunk.

Returns:

The presynaptic chunk, presynaptic locations, postsynaptic chunk, and postsynaptic locations.

Return type:

Tuple[Chunk, numpy.ndarray, Chunk, numpy.ndarray]

from_(chunks)

incoming()

outgoing()

to(chunks)

class bsb.storage.interfaces.ConnectivitySet(engine)

Bases: Interface

Stores the connections between 2 types of cell as local and global locations. A location is a cell id, referring to the n-th cell in the chunk, a branch id, and a point id, to specify the location on the morphology. Local locations refer to cells on this chunk, while global locations can come from any chunk and is associated to a certain chunk id as well.

Locations are either placement-context or chunk dependent: You may form connections between the n-th cells of a placement set (using connect()), or of the n-th cells of 2 chunks (using chunk_connect()).

A cell has both incoming and outgoing connections; when speaking of incoming connections, the local locations are the postsynaptic cells, and when speaking of outgoing connections they are the presynaptic cells. Vice versa for the global connections.

abstract chunk_connect(src_chunk, dst_chunk, src_locs, dst_locs)

Must connect the src_locs to the dest_locs, interpreting the cell ids (first column of the locs) as the cell rank in the chunk.

abstract clear(chunks=None)

Must clear (some chunks of) the placement set

abstract connect(pre_set, post_set, src_locs, dest_locs)

Must connect the src_locs to the dest_locs, interpreting the cell ids (first column of the locs) as the cell rank in the placement set.

abstract classmethod create(engine, tag)

Must create the placement set.

abstract static exists(engine, tag)

Must check the existence of the connectivity set

abstract flat_iter_connections(direction=None, local_=None, global_=None)

Must iterate over the connectivity data, yielding the direction, local chunk, global chunk, and data:

for dir, lchunk, gchunk, data in self.flat_iter_connections():
    print(f"Flat {dir} block between {lchunk} and {gchunk}")

If a keyword argument is given, that axis is not iterated over, and the value is fixed in each iteration.

abstract get_global_chunks(direction, local_)

Must list all the global chunks that contain data coming from a local chunk in the given direction

abstract get_local_chunks(direction)

Must list all the local chunks that contain data in the given direction ("inc" or "out").

abstract classmethod get_tags(engine)

Must return the tags of all existing connectivity sets.

Parameters:

engine – Storage engine to inspect.

abstract load_block_connections(direction, local_, global_)

Must load the connections from direction perspective between local_ and global_.

Returns:

The local and global connections locations

Return type:

Tuple[numpy.ndarray, numpy.ndarray]

load_connections()

Loads connections as a CSIterator.

Returns:

A connectivity set iterator, that will load data

abstract load_local_connections(direction, local_)

Must load all the connections from direction perspective in local_.

Returns:

The local connection locations, a vector of the global connection chunks (1 chunk id per connection), and the global connections locations. To identify a cell in the global connections, use the corresponding chunk id from the second return value.

Return type:

Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]

abstract nested_iter_connections(direction=None, local_=None, global_=None)

Must iterate over the connectivity data, leaving room for the end-user to set up nested for loops:

for dir, itr in self.nested_iter_connections():
    for lchunk, itr in itr:
        for gchunk, data in itr:
            print(f"Nested {dir} block between {lchunk} and {gchunk}")

If a keyword argument is given, that axis is not iterated over, and the amount of nested loops is reduced.

post_type: CellType

post_type_name: str

pre_type: CellType

pre_type_name: str

require(engine, tag)

Must make sure the connectivity set exists. The default implementation uses the class’s exists and create methods.

tag: str

class bsb.storage.interfaces.Engine(root, comm)

Bases: Interface

Engines perform the transactions that come from the storage object, and read/write data in a specific format. They can perform collective or individual actions.

Warning

Collective actions can only be performed from all nodes, or deadlocks occur. This means in particular that they may not be called from component code.

abstract clear_connectivity()

collective Must clear existing connectivity data.

abstract clear_placement()

collective Must clear existing placement data.

property comm

The communicator in charge of collective operations.

abstract copy(new_root)

collective Must copy the storage object to the new root.

abstract create()

collective Must create the storage engine.

abstract exists()

Must check existence of the storage object.

property format

Name of the type of engine. Automatically set through the plugin system.

abstract get_chunk_stats()

readonly Must return a dictionary with all chunk statistics.

abstract move(new_root)

collective Must move the storage object to the new root.

classmethod peek_exists(root)

Must peek at the existence of the given root, without instantiating anything.

read_only()

A context manager that enters the engine into readonly mode. In readonly mode the engine does not perform any locking, write-operations or network synchronization, and errors out if a write operation is attempted.

readwrite()

abstract recognizes(root)

Must return whether the given argument is recognized as a valid storage object.

abstract remove()

collective Must remove the storage object.

property root

The unique identifier for the storage. Usually pathlike, but can be anything.

abstract property root_slug

Must return a pathlike unique identifier for the root of the storage object.

set_comm(comm)

collective Set a new communicator in charge of collective operations.

abstract property versions

Must return a dictionary containing the version of the engine package, and bsb package, used to last write to this storage object.

class bsb.storage.interfaces.FileStore(engine)

Bases: Interface

Interface for the storage and retrieval of files essential to the network description.

abstract all()

Return all ids and associated metadata in the file store.

find_file(predicate)

find_files(predicate)

find_id(id)

find_meta(key, value)

get(id) → StoredFile

Return a StoredFile wrapper

Return type:

StoredFile

abstract get_encoding(id)

Must return the encoding of the file with the given id, or None if it is unspecified binary data.

abstract get_meta(id) → Mapping[str, Any]

Must return the metadata of the given id.

Return type:

Mapping[str, Any]

abstract get_mtime(id)

Must return the last modified timestamp of file with the given id.

abstract has(id)

Must return whether the file store has a file with the given id.

abstract load(id)

Load the content of an object in the file store.

Parameters:

id (str) – id of the content to be loaded.

Returns:

The content of the stored object

Return type:

str

Raises:

FileNotFoundError – The given id doesn’t exist in the file store.

abstract load_active_config()

Load the active configuration stored in the file store.

Returns:

The active configuration

Return type:

Configuration

Raises:

Exception – When there’s no active configuration in the file store.

abstract remove(id)

Remove the content of an object in the file store.

Parameters:

id (str) – id of the content to be removed.

Raises:

FileNotFoundError – The given id doesn’t exist in the file store.

abstract store(content, id=None, meta=None, encoding=None, overwrite=False)

Store content in the file store. Should also store the current timestamp as mtime meta.

Parameters:

• content (str) – Content to be stored

• id (str) – Optional specific id for the content to be stored under.

• meta (dict) – Metadata for the content

• encoding (str) – Optional encoding

• overwrite (bool) – Overwrite existing file

Returns:

The id the content was stored under

Return type:

str

abstract store_active_config(config)

Store configuration in the file store and mark it as the active configuration of the stored network.

Parameters:

config (Configuration) – Configuration to be stored

Returns:

The id the config was stored under

Return type:

str

class bsb.storage.interfaces.GeneratedMorphology(name, generated, meta)

Bases: StoredMorphology

class bsb.storage.interfaces.Interface(engine)

Bases: ABC

class bsb.storage.interfaces.MorphologyRepository(engine)

Bases: Interface

abstract all()

Fetch all of the stored morphologies.

Returns:

List of the stored morphologies.

Return type:

List[StoredMorphology]

abstract get_all_meta()

Get the metadata of all stored morphologies. :returns: Metadata dictionary :rtype: dict

abstract get_meta(name)

Get the metadata of a stored morphology.

Parameters:

name (str) – Key of the stored morphology.

Returns:

Metadata dictionary

Return type:

dict

abstract has(name)

Check whether a morphology under the given name exists

Parameters:

name (str) – Key of the stored morphology.

Returns:

Whether the key exists in the repo.

Return type:

bool

list()

List all the names of the morphologies in the repository.

abstract load(name)

Load a stored morphology as a constructed morphology object.

Parameters:

name (str) – Key of the stored morphology.

Returns:

A morphology

Return type:

Morphology

abstract preload(name)

Load a stored morphology as a morphology loader.

Parameters:

name (str) – Key of the stored morphology.

Returns:

The stored morphology

Return type:

StoredMorphology

abstract save(name, morphology, overwrite=False)

Store a morphology

Parameters:

• name (str) – Key to store the morphology under.

• morphology (bsb.morphologies.Morphology) – Morphology to store

• overwrite (bool) – Overwrite any stored morphology that already exists under that name

Returns:

The stored morphology

Return type:

StoredMorphology

abstract select(*selectors)

Select stored morphologies.

Parameters:

selectors (List[bsb.morphologies.selector.MorphologySelector]) – Any number of morphology selectors.

Returns:

All stored morphologies that match at least one selector.

Return type:

List[StoredMorphology]

abstract set_all_meta(all_meta)

Set the metadata of all stored morphologies. :param all_meta: Metadata dictionary. :type all_meta: dict

abstract update_all_meta(meta)

Update the metadata of stored morphologies with the provided key values

Parameters:

meta (str) – Metadata dictionary.

class bsb.storage.interfaces.NetworkDescription(engine)

Bases: Interface

class bsb.storage.interfaces.NoopLock

Bases: object

class bsb.storage.interfaces.PlacementSet(engine, cell_type)

Bases: Interface

Interface for the storage of placement data of a cell type.

abstract append_additional(name, chunk, data)

Append arbitrary user data to the placement set. The length of the data must match that of the placement set, and must be storable by the engine.

Parameters:

• name

• chunk (Chunk) – The chunk to store data in.

• data (numpy.ndarray) – Arbitrary user data. You decide ❤️

abstract append_data(chunk, positions=None, morphologies=None, rotations=None, additional=None, count=None)

Append data to the placement set. If any of positions, morphologies, or rotations is given, the arguments to its left must also be given (e.g. passing morphologies, but no positions, is not allowed, passing just positions is allowed)

Parameters:

• chunk (Chunk) – The chunk to store data in.

• positions (numpy.ndarray) – Cell positions

• rotations (RotationSet) – Cell rotations

• morphologies (MorphologySet) – Cell morphologies

• additional (Dict[str, numpy.ndarray]) – Additional datasets with 1 value per cell, will be stored under its key in the dictionary

• count (int) – Amount of entities to place. Excludes the use of any positional, rotational or morphological data.

property cell_type

The associated cell type.

Returns:

The cell type

Return type:

CellType

abstract chunk_context(chunks)

abstract clear(chunks=None)

Clear (some chunks of) the placement set.

Parameters:

chunks (List[bsb.storage._chunks.Chunk]) – If given, the specific chunks to clear.

count_morphologies()

Must return the number of different morphologies used in the set.

abstract classmethod create(engine, cell_type)

Create a placement set.

Parameters:

• engine (bsb.storage.interfaces.Engine) – The engine that governs this PlacementSet.

• cell_type (bsb.cell_types.CellType) – The cell type whose data is stored in the placement set.

Returns:

A placement set

Return type:

bsb.storage.interfaces.PlacementSet

abstract static exists(engine, cell_type)

Check existence of a placement set.

Parameters:

• engine (bsb.storage.interfaces.Engine) – The engine that governs the existence check.

• cell_type (bsb.cell_types.CellType) – The cell type to look for.

Returns:

Whether the placement set exists.

Return type:

bool

abstract get_all_chunks()

Get all the chunks that exist in the placement set.

Returns:

List of existing chunks.

Return type:

List[bsb.storage._chunks.Chunk]

abstract get_chunk_stats()

Should return how many cells were placed in each chunk.

abstract get_label_mask(labels)

Should return a mask that fits the placement set for the cells with given labels.

Parameters:

• cells (numpy.ndarray) – Array of cells in this set to label.

• labels (list[str]) – List of labels

abstract get_labelled(labels)

Should return the cells labelled with given labels.

Parameters:

• cells (numpy.ndarray) – Array of cells in this set to label.

• labels (list[str]) – List of labels

abstract label(labels, cells)

Should label the cells with given labels.

Parameters:

• cells (numpy.ndarray) – Array of cells in this set to label.

• labels (list[str]) – List of labels

abstract load_additional(key=None)

load_box_tree(morpho_cache=None)

Load boxes, and form an RTree with them, for fast spatial lookup of rhomboid intersection.

Parameters:

morpho_cache – See load_boxes().

Returns:

A boxtree

Return type:

bsb.trees.BoxTree

load_boxes(morpho_cache=None)

Load the cells as axis aligned bounding box rhomboids matching the extension, orientation and position in space. This function loads morphologies, unless a morpho_cache is given, then that is used.

Parameters:

morpho_cache (MorphologySet) – If you’ve previously loaded morphologies with soft or hard caching enabled, you can pass the resulting morphology set here to reuse it. If afterwards you need the morphology set, you best call load_morphologies() first and reuse it here.

Returns:

An iterator with 6 coordinates per cell: 3 min and 3 max coords, the bounding box of that cell’s translated and rotated morphology.

Return type:

Iterator[Tuple[float, float, float, float, float, float]]

Raises:

DatasetNotFoundError if no morphologies are found.

abstract load_ids()

abstract load_morphologies(allow_empty=False)

Return a MorphologySet associated to the cells. Raises an error if there is no morphology data, unless allow_empty=True.

Parameters:

allow_empty (bool) – Silence missing morphology data error, and return an empty morphology set.

Returns:

Set of morphologies

Return type:

MorphologySet

abstract load_positions()

Return a dataset of cell positions.

Returns:

An (Nx3) dataset of positions.

Return type:

numpy.ndarray

abstract load_rotations()

Load the rotation data of the placement set :returns: A rotation set :rtype: ~bsb.morphologies.RotationSet

classmethod require(engine, type)

Return and create a placement set, if it didn’t exist before.

The default implementation uses the exists() and create() methods.

Parameters:

• engine (bsb.storage.interfaces.Engine) – The engine that governs this PlacementSet.

• cell_type (bsb.cell_types.CellType) – The cell type whose data is stored in the placement set.

Returns:

A placement set

Return type:

bsb.storage.interfaces.PlacementSet

abstract set_chunk_filter(chunks)

Should limit the scope of the placement set to the given chunks.

Parameters:

chunks (list[bsb.storage._chunks.Chunk]) – List of chunks

abstract set_label_filter(labels)

Should limit the scope of the placement set to the given labels.

Parameters:

labels (list[str]) – List of labels

abstract set_morphology_label_filter(morphology_labels)

Should limit the scope of the placement set to the given sub-cellular labels. The morphologies returned by load_morphologies() should return a filtered form of themselves if as_filtered() is called on them.

Parameters:

morphology_labels (list[str]) – List of labels

property tag

The unique identifier of the placement set.

Returns:

Unique identifier

Return type:

str

class bsb.storage.interfaces.ReadOnlyManager(engine)

Bases: object

class bsb.storage.interfaces.StorageNode(*args, _parent=None, _key=None, **kwargs)

Bases: object

engine

Base implementation of all the different configuration attributes. Call the factory function attr() instead.

get_node_name()

root: Any

class bsb.storage.interfaces.StoredFile(store, id)

Bases: object

load()

property meta

property mtime

class bsb.storage.interfaces.StoredMorphology(name, loader, meta)

Bases: object

cached_load(labels=None)

get_meta()

load()

Module contents

This module imports all supported storage engines, objects that read and write data, which are present as subfolders of the engine folder, and provides them transparently to the user, as a part of the Storage factory class. The module scans the storage.interfaces module for any class that inherits from Interface to collect all Feature Interfaces and then scans the storage.engines.* submodules for any class that provides an implementation of those features.

These features, because they all follow the same interface can then be passed on to consumers and can be used independent of the underlying storage engine, which is the end goal of this module.

bsb.storage._chunks.chunklist(chunks) → List[Chunk]

Convert an iterable of chunk like objects to a sorted unique chunklist

Return type:

List[Chunk]

class bsb.storage._chunks.Chunk(chunk, chunk_size)

Chunk identifier, consisting of chunk coordinates and size.

class bsb.storage.NotSupported(operation)

Bases: object

Utility class that throws a NotSupported error when it is used. This is the default “implementation” of every storage feature that isn’t provided by an engine.

class bsb.storage.Storage(engine, root, comm=None, main=0, missing_ok=True)

Bases: object

Factory class that produces all the features and shims the functionality of the underlying engine.

Create a Storage provider based on a specific engine uniquely identified by the root object.

Parameters:

• engine (str) – The name of the storage engine.

• root (object) – An object that uniquely describes the storage, such as a filename or path. The value to be provided depends on the engine. For the hdf5 engine the filename has to be provided.

• comm (mpi4py.MPI.Comm) – MPI communicator that shares control over this Storage.

• main – Rank of the MPI process that executes single-node tasks.

assert_support(feature)

clear_connectivity()

clear_placement(scaffold=None)

copy(new_root)

Move the storage to a new root.

create()

Create the minimal requirements at the root for other features to function and for the existence check to pass.

exists()

Check whether the storage exists at the root.

property files

property format

get_chunk_stats()

get_connectivity_set(tag)

Get a connection set.

Parameters:

tag (str) – Connection tag

Returns:

~bsb.storage.interfaces.ConnectivitySet

get_connectivity_sets()

Return a ConnectivitySet for the given type.

Parameters:

type (CellType) – Specific cell type.

Returns:

~bsb.storage.interfaces.ConnectivitySet

get_placement_set(type, chunks=None, labels=None, morphology_labels=None)

Return a PlacementSet for the given type.

Parameters:

• type (CellType) – Specific cell type.

• chunks (list[tuple[float, float, float]]) – Optionally load a specific list of chunks.

• labels (list[str]) – Labels to filter the placement set by.

Returns:

~bsb.storage.interfaces.PlacementSet

init(scaffold)

Initialize the storage to be ready for use by the specified scaffold.

init_placement(scaffold)

is_main_process()

load()

Load a scaffold from the storage.

Returns:

Scaffold

load_active_config()

Load the configuration object from the storage.

Returns:

Configuration

property morphologies

move(new_root)

Move the storage to a new root.

property preexisted

read_only()

remove()

Remove the storage and all data contained within. This is an irreversible destructive action!

renew(scaffold)

Remove and recreate an empty storage container for a scaffold.

require_connectivity_set(tag, pre=None, post=None)

Get a connection set.

Parameters:

tag (str) – Connection tag

Returns:

~bsb.storage.interfaces.ConnectivitySet

require_placement_set(cell_type)

Get a placement set.

Parameters:

cell_type (CellType) – Connection cell_type

Returns:

~bsb.storage.interfaces.PlacementSet

property root

property root_slug

store_active_config(config)

Store a configuration object in the storage.

supports(feature)

bsb.storage.create_engine(name, root, comm)

Create an engine from the engine’s Engine interface.

Parameters:

• name (str) – The name of the engine to create.

• root (object) – An object that uniquely describes the storage, such as a filename or path. The value to be provided depends on the engine. For the hdf5 engine the filename has to be provided.

• comm (bsb.services.mpi.MPIService) – MPI communicator that shares control over the Engine interface.

bsb.storage.discover_engines()

Get a dictionary of all available storage engines.

bsb.storage.get_engine_node(engine_name)

bsb.storage.get_engines()

bsb.storage.open_storage(root, comm=None)

Load a Storage object from its root.

Parameters:

• root – Root (usually path) pointing to the storage object.

• comm (mpi4py.MPI.Comm) – MPI communicator that shares control over the Storage.

Returns:

A network scaffold

Return type:

Storage

bsb.storage.view_support(engine=None)

Return which storage engines support which features.

class bsb.storage._files.MorphologyOperationCallable(*args, **kwargs)

Hello

class bsb.storage._files.OperationCallable(*args, **kwargs)

class bsb.storage._files.CodeDependencyNode(*args, _parent=None, _key=None, **kwargs)

Allow the loading of external code during network loading.

attr: str

Attribute to extract from the loaded script

file: FileDependency

get_node_name()

load_object()

module: str

Should be either the path to a python file or a import like string

class bsb.storage._files.FileDependency(source: str | PathLike, file_store: FileStore = None, ext: str = None, cache=True)

Parameters:

• source (str | PathLike)

• file_store (FileStore)

• ext (str)

get_content(check_store=True)

get_meta(check_store=True)

get_stored_file()

provide_locally()

provide_stream()

should_update()

store_content(content, encoding=None, meta=None)

update(force=False)

property uri

class bsb.storage._files.FileDependencyNode(*args, _parent=None, _key=None, **kwargs)

file: FileDependency

Base implementation of all the different configuration attributes. Call the factory function attr() instead.

get_node_name()

get_stored_file()

load_object()

provide_locally()

provide_stream()

scaffold: Scaffold

class bsb.storage._files.FileScheme

find(file: FileDependency)

Parameters:

file (FileDependency)

get_content(file: FileDependency)

Parameters:

file (FileDependency)

get_local_path(file: FileDependency)

Parameters:

file (FileDependency)

get_meta(file: FileDependency)

Parameters:

file (FileDependency)

provide_stream(file: FileDependency)

Parameters:

file (FileDependency)

should_update(file: FileDependency, stored_file)

Parameters:

file (FileDependency)

class bsb.storage._files.MorphologyDependencyNode(*args, _parent=None, _key=None, **kwargs)

Configuration dependency node to load morphology files. The content of these files will be stored in bsb.morphologies.Morphology instances.

get_morphology_name()

Returns morphology name provided by the user or extract it from its file name.

Returns:

Morphology name

Return type:

str

get_node_name()

load_object(parser=None, save=True) → Morphology

Return type:

Morphology

name: str

Base implementation of all the different configuration attributes. Call the factory function attr() instead.

parser: MorphologyParser

Name associated to the morphology. If not provided, the program will use the name of the file in which the morphology is stored.

pipeline: cfglist[MorphologyOperation]

queue(pool)

Add the loading of the current morphology to a job queue.

Parameters:

pool (bsb.services.pool.JobPool) – Queue of jobs.

store_content(content, *args, encoding=None, meta=None)

store_object(morpho, hash_)

Save a morphology into the circuit file under the name of this instance morphology.

Parameters:

• hash (str) – Hash key to store as metadata with the morphology

• morpho (bsb.morphologies.Morphology) – Morphology to store

class bsb.storage._files.MorphologyOperation(*args, _parent=None, _key=None, **kwargs)

func: MorphologyOperationCallable

Base implementation of all the different configuration attributes. Call the factory function attr() instead.

get_node_name()

class bsb.storage._files.NeuroMorphoScheme

create_session()

get_base_url()

get_meta(file: FileDependency)

Parameters:

file (FileDependency)

get_nm_meta(file: FileDependency)

Parameters:

file (FileDependency)

resolve_uri(file: FileDependency)

Parameters:

file (FileDependency)

class bsb.storage._files.NrrdDependencyNode(*args, _parent=None, _key=None, **kwargs)

Configuration dependency node to load NRRD files.

get_data()

get_header()

get_node_name()

load_object()

class bsb.storage._files.Operation(*args, _parent=None, _key=None, **kwargs)

func: OperationCallable

Base implementation of all the different configuration attributes. Call the factory function attr() instead.

get_node_name()

parameters: dict[Any]

class bsb.storage._files.UriScheme

abstract find(file: FileDependency)

Parameters:

file (FileDependency)

abstract get_content(file: FileDependency)

Parameters:

file (FileDependency)

abstract get_local_path(file: FileDependency)

Parameters:

file (FileDependency)

abstract get_meta(file: FileDependency)

Parameters:

file (FileDependency)

abstract provide_stream(file)

abstract should_update(file: FileDependency, stored_file)

Parameters:

file (FileDependency)

class bsb.storage._files.UrlScheme

create_session()

find(file: FileDependency)

Parameters:

file (FileDependency)

get_base_url()

get_content(file: FileDependency)

Parameters:

file (FileDependency)

get_local_path(file: FileDependency)

Parameters:

file (FileDependency)

get_meta(file: FileDependency)

Parameters:

file (FileDependency)

provide_stream(file)

resolve_uri(file: FileDependency)

Parameters:

file (FileDependency)

should_update(file: FileDependency, stored_file)

Parameters:

file (FileDependency)