Extensions for lab-specific metadata: Extending LabMetaData
Use case
Here we address the use case of adding lab-specific metadata to a file, e.g., lab-specific information about experimental protocols, lab-specific identifiers and so on. This approach is intended for usually small metadata. Extension source
Approach
To include lab-specific metadata, NWB provides pynwb.file.LabMetaData
as a
a convenient base type, which makes it easy to add your data to an pynwb.file.NWBFile
without having to modify the pynwb.file.NWBFile
type itself
(since adding of pynwb.file.LabMetaData
is already implemented).
Note
NWB uses dynamically extensible table structures based on DynamicTable
to describe metadata and derived results, e.g., TimeIntervals
for epochs or trials
or ElectrodeTable
to describe extracellular electrodes. Depending on the
type of metadata, use of these existing dynamic table structures can help avoid the need for
custom extensions by including the data as additional, custom columns in the appropriate existing tables.
Creating the extension
1. Create a new repository for the extension using the ndx-template:
cookiecutter gh:nwb-extensions/ndx-template
2. Answer a few simple questions of the cookiecutter template. We can respond to many questions
with Enter
to accept the default response (e.g., to start with version=0.1.0
):
namespace [ndx-my-namespace]: ndx-labmetadata-example
description [My NWB extension]: Example extension to illustrate how to extend LabMetaData for adding lab-specific metadata
author [My Name]: Oliver Ruebel
email [my_email@example.com]: oruebel@lbl.gov
github_username [myname]: oruebel
copyright [2021, Oliver Ruebel]:
version [0.1.0]:
release [alpha]:
Select license:
1 - BSD-3
2 - MIT
3 - Apache Software License 2.0
4 - Other
Choose from 1, 2, 3, 4 [1]: 1
py_pkg_name [ndx_labmetadata_example]:
3. Edit ndx-my-brainlabsrc/spec/create_extension_spec.py
that was generated for you to define the
schema of your extension. See The Spec API section for details on how to use the specification API.
Add
LabMetaData
as an include type
ns_builder.include_type('LabMetaData', namespace='core')
Define your new
LabMetaData
type for your lab
labmetadata_ext = NWBGroupSpec(
name='custom_lab_metadata',
doc='Example extension type for storing lab metadata',
neurodata_type_def='LabMetaDataExtensionExample',
neurodata_type_inc='LabMetaData',
)
Add the
Groups
,Datasets
, andAttributes
with the metadata specific to our lab to ourLabMetaData
schema
labmetadata_ext.add_dataset(
name="tissue_preparation",
doc="Lab-specific description of the preparation of the tissue",
dtype='text',
quantity='?'
)
Add our new type definitions to the extension
new_data_types = [labmetadata_ext]
4. Generate the schema for the extension by running the create_extension_spec.py
script
cd ndx-labmetadata-example
python src/spec/create_extension_spec.py
5. Edit src/pynwb/__init__.py
to define Python API classes for our new extension data types via pynwb.get_class()
.
LabMetaDataExtensionExample = get_class('LabMetaDataExtensionExample', 'ndx-labmetadata-example')
6. Define unit tests for the extension. The ndx-template created an example test
module src/pynwb/tests/test_tetrodeseries.py
to illustrate how to implement tests. Here we simply remove
this file and replace it with our own tests test_labmetadata_example.py. More details below in Creating unit tests.
7. To make sure our extension schema and source code files are version controlled, we now add all the files we just created to the Git repo:
git add .
git commit -m "Added API classes, tests, and schema files"
8. Install your extension (Python only)(Optional)
pip install .
Now our extension is ready to use!
Creating custom Python API classes
We skip this step here, since this extension of LabMetaData
is simple enough that the
autogenerated class is sufficient. If the autogenerated class from pynwb.get_class()
for an extension
data types is not sufficient, then we can either customize the autogenerated class as described in
Generating a PyNWB API (recommended only for basic changes) or define our own custom API class as
described in Building a custom Python API for an extension (recommended for full customization).
Creating unit tests
from pynwb.testing.mock.file import mock_NWBFile
from pynwb.testing import TestCase
from ndx_labmetadata_example import LabMetaDataExtensionExample
class TestLabMetaDataExtensionExample(TestCase):
"""Test basic functionality of LabMetaDataExtensionExample without read/write"""
def setUp(self):
"""Set up an NWB file. Necessary because TetrodeSeries requires references to electrodes."""
self.nwbfile = mock_NWBFile()
def test_constructor(self):
"""Test that the constructor for TetrodeSeries sets values as expected."""
tissue_preparation = "Example tissue preparation"
lmdee_object = LabMetaDataExtensionExample(tissue_preparation=tissue_preparation)
self.assertEqual(lmdee_object.tissue_preparation, tissue_preparation)
from pynwb.testing.mock.file import mock_NWBFile
from pynwb.testing import TestCase
from pynwb.testing.testh5io import NWBH5IOMixin
from ndx_labmetadata_example import LabMetaDataExtensionExample
class TestLabMetaDataExtensionExampleRoundtrip(NWBH5IOMixin, TestCase):
"""
Roundtrip test for LabMetaDataExtensionExample to test read/write
This test class writes the LabMetaDataExtensionExample to an NWBFile, then
reads the data back from the file, and compares that the data read from file
is consistent with the original data. Using the pynwb.testing infrastructure
simplifies this complex test greatly by allowing to simply define how to
create the container, add to a file, and retrieve it form a file. The
task of writing, reading, and comparing the data is then taken care of
automatically by the NWBH5IOMixin.
"""
def setUpContainer(self):
"""set up example LabMetaDataExtensionExample object"""
self.lab_meta_data = LabMetaDataExtensionExample(tissue_preparation="Example tissue preparation")
return self.lab_meta_data
def addContainer(self, nwbfile):
"""Add the test LabMetaDataExtensionExample to the given NWBFile."""
nwbfile.add_lab_meta_data(lab_meta_data=self.lab_meta_data)
def getContainer(self, nwbfile):
"""Get the LabMetaDataExtensionExample object from the given NWBFile."""
return nwbfile.get_lab_meta_data(self.lab_meta_data.name)
cd ndx-labmetadata-example
pytest
Coming soon ...
Coming soon ...
Coming soon ...
Documenting the extension
REAME.md: Add instructions to the
README.md
file. This typically includes information on how to install the extension and an example on how to use the extensionSchema and user documentation:
Install the latest release of hdmf_docutils:
python -m pip install hdmf-docutils
Generate the documentation for your extension based on the YAML schema files via:
cd docs/ make html
To view the docs, simply open
docs/build/html/index.html
in your browserSee the docs/README.md for instructions on how to customize the documentation for your extension.
See Documenting Extensions for more details.
Writing data using the extension
from pynwb.file import NWBFile, Subject
from ndx_labmetadata_example import LabMetaDataExtensionExample
from pynwb import NWBHDF5IO
from uuid import uuid4
from datetime import datetime
# create an example NWBFile
nwbfile = NWBFile(
session_description="test session description",
identifier=str(uuid4()),
session_start_time=datetime(1970, 1, 1),
subject=Subject(
age="P50D",
description="example mouse",
sex="F",
subject_id="test_id")
)
# create our custom lab metadata
lab_meta_data = LabMetaDataExtensionExample(tissue_preparation="Example tissue preparation")
# Add the test LabMetaDataExtensionExample to the NWBFile
nwbfile.add_lab_meta_data(lab_meta_data=lab_meta_data)
# Write the file to disk
filename = "testfile.nwb"
with NWBHDF5IO(path=filename, mode="a") as io:
io.write(nwbfile)
Coming soon ...
Reading an NWB file that uses the extension
from pynwb import NWBHDF5IO
from ndx_labmetadata_example import LabMetaDataExtensionExample
# Read the file from disk
io = NWBHDF5IO(path=filename, mode="r")
nwbfile = io.read()
# Get the custom lab metadata object
lab_meta_data = nwbfile.get_lab_meta_data(name="custom_lab_metadata")
from pynwb import NWBHDF5IO
# Read the file from disk. Load the namespace from file to
# autogenerate classes from the schema
io = NWBHDF5IO(path=filename, mode="r", load_namespaces=True)
nwbfile = io.read()
# Get the custom lab metadata object
lab_meta_data = nwbfile.get_lab_meta_data(name="custom_lab_metadata")
Coming soon ...
Publishing the extension
The steps to publish an extension are the same for all extensions. We, therefore, here only briefly describe he main steps for publishing our extension. For a more in-depth guide, see the page Publishing extensions
GitHub (Open Source): To make the sources of your extension openly accessible, publish the extension on GitHub by following the instructions on Open Source: Releasing your extension Git repository.
PyPI (Open Access): Publish your extension on PyPI to make it easy for users to install it and to create a persistent release of the extension following the Open Access: Releasing your extension on PyPI guide.
NDX Catalog (Open Publication): The NDX Catalog serves as a central, community-led catalog for extensions to the NWB data standard. The NDX Catalog manages basic metadata about extensions while ownership of the source repositories for the extensions remain with the developers. For a step-by-step guide the Open Publication: Publishing your extension on the NDX Catalog guide.