Developer guidelines

This webpage is intended to guide users through making making changes to tedana’s codebase, in particular working with tests. The worked example also offers some guidelines on approaching testing when adding new functions. Please check out our contributing guide for getting started.

Monthly Developer Calls

We run monthly developer calls via Zoom. You can see the schedule via the tedana google calendar.

Everyone is welcome. We look forward to meeting you there!

Adding and Modifying Tests

Testing is an important component of development. For simplicity, we have migrated all tests to pytest. There are two basic kinds of tests: unit and integration tests. Unit tests focus on testing individual functions, whereas integration tests focus on making sure that the whole workflow runs correctly.

Unit Tests

For unit tests, we try to keep tests from the same module grouped into one file. Make sure the function you’re testing is imported, then write your test. Good tests will make sure that edge cases are accounted for as well as common cases. You may also use pytest.raises to ensure that errors are thrown for invalid inputs to a function.

Integration Tests

Adding integration tests is relatively rare. An integration test will be a complete multi-echo dataset called with some set of options to ensure end-to-end pipeline functionality. These tests are relatively computationally expensive but aid us in making sure the pipeline is stable during large sets of changes. If you believe you have a dataset that will test tedana more completely, please open an issue before attempting to add an integration test. After securing the appropriate permission from the dataset owner to share it with tedana, you can use the following procedure:

(1) Make a tar.gz file which will unzip to be only the files you’d like to run a workflow on. You can do this with the following, which would make an archive my_data.tar.gz:

tar czf my_data.tar.gz my_data/*.nii.gz

(2) Run the workflow with a known-working version, and put the outputs into a text file inside $TEDANADIR/tedana/tests/data/, where TEDANADIR is your local tedana repository. We encourage using the convention <DATASET>_<n_echoes>_echo_outputs.txt, appending verbose to the filename if the integration test uses tedana in the verbose mode.

(3) Write a test function in test_integration.py. To write the test function you can follow the model of our five echo set, which takes the following steps:

  1. Check if a pytest user is skipping integration, skip if so
  2. Use download_test_data to retrieve the test data from OSF
  3. Run a workflow
  4. Use resources_filename and check_integration_outputs to compare your expected output to actual output.

(4) If you need to upload new data, you will need to contact the maintainers and ask them to either add it to the tedana OSF project or give you permission to add it.

(5) Once you’ve tested your integration test locally and it is working, you will need to add it to the CircleCI config and the Makefile. Following the model of the three-echo and five-echo sets, define a name for your integration test and on an indented line below put

@py.test --cov-append --cov-report term-missing --cov=tedana -k TEST

with TEST your test function’s name. This call basically adds code coverage reports to account for the new test, and runs the actual test in addition.

(6) Using the five-echo set as a template, you should then edit .circlec/config.yml to add your test, calling the same name you define in the Makefile.

Viewing CircleCI Outputs

If you need to take a look at a failed test on CircleCI rather than locally, you can use the following block to retrieve artifacts (see CircleCI documentation here)

export CIRCLE_TOKEN=':your_token'

curl https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/$build_number/artifacts?circle-token=$CIRCLE_TOKEN \
   | grep -o 'https://[^"]*' \
   | sed -e "s/$/?circle-token=$CIRCLE_TOKEN/" \
   | wget -v -i -

To get a CircleCI token, follow the instructions for getting one. You cannot do this unless you are part of the ME-ICA/tedana organization. If you don’t want all of the artifacts, you can go to the test details and use the browser to manually select the files you would like.

Worked Example

Suppose we want to add a function in tedana that creates a file called `hello_world.txt to be stored along the outputs of the tedana workflow.

First, we merge the repository’s master branch into our own to make sure we’re up to date, and then we make a new branch called something like feature/say_hello. Any changes we make will stay on this branch. We make the new function and call it say_hello and locate this function inside of io.py. We’ll also need to make a unit test. (Some developers actually make the unit test before the new function; this is a great way to make sure you don’t forget to create it!) Since the function lives in io.py, its unit test should go into test_io.py. The job of this test is exclusively to tell if the function we wrote does what it claims to do without errors. So, we define a new function in test_io.py that looks something like this:

def test_say_hello():
    # run the function
    say_hello()
    # test the function
    assert op.exists('hello_world.txt')
    # clean up
    os.remove('hello_world.txt')

We should see that our unit test is successful via

pytest $TEDANADIR/tedana/tests/test_io.py -k test_say_hello

If not, we should continue editing the function until it passes our test. Let’s suppose that suddenly, you realize that what would be even more useful is a function that takes an argument, place, so that the output filename is actually hello_PLACE, with PLACE the value passed and 'world' as the default value. We merge any changes from the upstream master branch into our branch via

git checkout feature/say_hello
git fetch upstream master
git merge upstream/master

and then begin work on our test. We need to our unit test to be more complete, so we update it to look more like the following, adding several cases to make sure our function is robust to the name supplied:

def test_say_hello():
    # prefix of all files to be checked
    prefix = 'hello_'
    # suffix of all files to be checked
    suffix  = '.txt'
    # run the function with several cases
    for x in ['world', 'solar system', 'galaxy', 'universe']:
        # current test name
        outname = prefix + x + suffix
        # call the function
        say_hello(x)
        # test the function
        assert op.exists(outname)
        # clean up from this call
        os.remove(outname)

Once that test is passing, we may need to adjust the integration test. Our program creates a file, hello_world.txt, which the older version would not have produced. Therefore, we need to add the file to $TEDANADIR/tedana/tests/data/tedana_outputs.txt and its counterpart, R2-D2– uh, we mean, tedana_outputs_verbose.txt. With that edit complete, we can run the full pytest suite via

pytest $TEDANADIR/tedana/tests

Once that filename is added, all of the tests should be passing and we should open a PR to have our change reviewed.

From here, others working on the project may request changes and we’ll have to make sure that our tests are kept up to date with any changes made as we did before updating the unit test. For example, if a new parameter is added, greeting, with a default of hello, we’ll need to adjust the unit test. However, since this doesn’t change the typical workflow of tedana, there’s no need to change the integration test; we’re still matching the original filename. Once we are happy with the changes and some members of tedana have approved the changes, our changes will be merged!

We should then do the following cleanup with our git repository:

git checkout master
git fetch upstream master
git merge upstream/master
git branch -d feature/say_hello
git push --delete origin feature/say_hello

and we’re good to go!