Extensions for lab-specific metadata: Extending LabMetaData

Use case

Here we address the use case of adding lab-specific metadata to a file, e.g., lab-specific information about experimental protocols, lab-specific identifiers and so on. This approach is intended for usually small metadata. Extension source

Approach

To include lab-specific metadata, NWB provides pynwb.file.LabMetaData as a a convenient base type, which makes it easy to add your data to an pynwb.file.NWBFile without having to modify the pynwb.file.NWBFile type itself (since adding of pynwb.file.LabMetaData is already implemented).

Note

NWB uses dynamically extensible table structures based on DynamicTable to describe metadata and derived results, e.g., TimeIntervals for epochs or trials or ElectrodeTable to describe extracellular electrodes. Depending on the type of metadata, use of these existing dynamic table structures can help avoid the need for custom extensions by including the data as additional, custom columns in the appropriate existing tables.

Creating the extension

1. Create a new repository for the extension using the ndx-template:

cookiecutter gh:nwb-extensions/ndx-template

2. Answer a few simple questions of the cookiecutter template. We can respond to many questions with Enter to accept the default response (e.g., to start with version=0.1.0):

namespace [ndx-my-namespace]: ndx-labmetadata-example
description [My NWB extension]: Example extension to illustrate how to extend LabMetaData for adding lab-specific metadata
author [My Name]: Oliver Ruebel
email [my_email@example.com]: oruebel@lbl.gov
github_username [myname]: oruebel
copyright [2021, Oliver Ruebel]:
version [0.1.0]:
release [alpha]:
Select license:
1 - BSD-3
2 - MIT
3 - Apache Software License 2.0
4 - Other
Choose from 1, 2, 3, 4 [1]: 1
py_pkg_name [ndx_labmetadata_example]:

3. Edit ndx-my-brainlabsrc/spec/create_extension_spec.py that was generated for you to define the schema of your extension. See The Spec API section for details on how to use the specification API.

  • Add LabMetaData as an include type

ns_builder.include_type('LabMetaData', namespace='core')

  • Define your new LabMetaData type for your lab

 labmetadata_ext = NWBGroupSpec(
    name='custom_lab_metadata',
    doc='Example extension type for storing lab metadata',
    neurodata_type_def='LabMetaDataExtensionExample',
    neurodata_type_inc='LabMetaData',
)

  • Add the Groups, Datasets, and Attributes with the metadata specific to our lab to our LabMetaData schema

labmetadata_ext.add_dataset(
    name="tissue_preparation",
    doc="Lab-specific description of the preparation of the tissue",
    dtype='text',
    quantity='?'
)

  • Add our new type definitions to the extension

new_data_types = [labmetadata_ext]

4. Generate the schema for the extension by running the create_extension_spec.py script

cd ndx-labmetadata-example
python src/spec/create_extension_spec.py

5. Edit src/pynwb/__init__.py to define Python API classes for our new extension data types via pynwb.get_class().

LabMetaDataExtensionExample = get_class('LabMetaDataExtensionExample', 'ndx-labmetadata-example')

6. Define unit tests for the extension. The ndx-template created an example test module src/pynwb/tests/test_tetrodeseries.py to illustrate how to implement tests. Here we simply remove this file and replace it with our own tests test_labmetadata_example.py. More details below in Creating unit tests.

7. To make sure our extension schema and source code files are version controlled, we now add all the files we just created to the Git repo:

git add .
git commit -m "Added API classes, tests, and schema files"

8. Install your extension (Python only)(Optional)

pip install .

Now our extension is ready to use!

Creating custom Python API classes

We skip this step here, since this extension of LabMetaData is simple enough that the autogenerated class is sufficient. If the autogenerated class from pynwb.get_class() for an extension data types is not sufficient, then we can either customize the autogenerated class as described in Generating a PyNWB API (recommended only for basic changes) or define our own custom API class as described in Building a custom Python API for an extension (recommended for full customization).

Creating unit tests

from pynwb.testing.mock.file import mock_NWBFile
from pynwb.testing import TestCase
from ndx_labmetadata_example import LabMetaDataExtensionExample


class TestLabMetaDataExtensionExample(TestCase):
    """Test basic functionality of LabMetaDataExtensionExample without read/write"""

    def setUp(self):
        """Set up an NWB file. Necessary because TetrodeSeries requires references to electrodes."""
        self.nwbfile = mock_NWBFile()

    def test_constructor(self):
        """Test that the constructor for TetrodeSeries sets values as expected."""
        tissue_preparation = "Example tissue preparation"
        lmdee_object = LabMetaDataExtensionExample(tissue_preparation=tissue_preparation)
        self.assertEqual(lmdee_object.tissue_preparation, tissue_preparation)

Documenting the extension

  • REAME.md: Add instructions to the README.md file. This typically includes information on how to install the extension and an example on how to use the extension

  • Schema and user documentation:

    • Install the latest release of hdmf_docutils: python -m pip install hdmf-docutils

    • Generate the documentation for your extension based on the YAML schema files via:

    cd docs/
make html

    • To view the docs, simply open docs/build/html/index.html in your browser

    • See the docs/README.md for instructions on how to customize the documentation for your extension.

See Documenting Extensions for more details.

Writing data using the extension

from pynwb.file import NWBFile, Subject
from ndx_labmetadata_example import LabMetaDataExtensionExample
from pynwb import NWBHDF5IO
from uuid import uuid4
from datetime import datetime

# create an example NWBFile
nwbfile = NWBFile(
    session_description="test session description",
    identifier=str(uuid4()),
    session_start_time=datetime(1970, 1, 1),
    subject=Subject(
        age="P50D",
        description="example mouse",
        sex="F",
        subject_id="test_id")
)

# create our custom lab metadata
lab_meta_data = LabMetaDataExtensionExample(tissue_preparation="Example tissue preparation")

# Add the test LabMetaDataExtensionExample to the NWBFile
nwbfile.add_lab_meta_data(lab_meta_data=lab_meta_data)

# Write the file to disk
filename = "testfile.nwb"
with NWBHDF5IO(path=filename, mode="a") as io:
    io.write(nwbfile)

Reading an NWB file that uses the extension

from pynwb import NWBHDF5IO
from ndx_labmetadata_example import LabMetaDataExtensionExample

# Read the file from disk
io =  NWBHDF5IO(path=filename, mode="r")
nwbfile = io.read()
# Get the custom lab metadata object
lab_meta_data = nwbfile.get_lab_meta_data(name="custom_lab_metadata")

Publishing the extension

The steps to publish an extension are the same for all extensions. We, therefore, here only briefly describe he main steps for publishing our extension. For a more in-depth guide, see the page Publishing extensions