This webpage is intended to guide users through making making changes to
tedana’s codebase, in particular working with tests.
The worked example also offers some guidelines on approaching testing when
adding new functions.
Please check out our contributing guide for getting started.
Monthly Developer Calls¶
We run monthly developer calls via Zoom. You can see the schedule via the tedana google calendar.
Everyone is welcome. We look forward to meeting you there!
Adding and Modifying Tests¶
Testing is an important component of development.
For simplicity, we have migrated all tests to
There are two basic kinds of tests: unit and integration tests.
Unit tests focus on testing individual functions, whereas integration tests focus on making sure
that the whole workflow runs correctly.
For unit tests, we try to keep tests from the same module grouped into one file.
Make sure the function you’re testing is imported, then write your test.
Good tests will make sure that edge cases are accounted for as well as common cases.
You may also use
pytest.raises to ensure that errors are thrown for invalid inputs to a
Adding integration tests is relatively rare.
An integration test will be a complete multi-echo dataset called with some set of options to ensure
end-to-end pipeline functionality.
These tests are relatively computationally expensive but aid us in making sure the pipeline is
stable during large sets of changes.
If you believe you have a dataset that will test
tedana more completely, please open an issue
before attempting to add an integration test.
After securing the appropriate permission from the dataset owner to share it with
can use the following procedure:
(1) Make a
tar.gz file which will unzip to be only the files you’d like to
run a workflow on.
You can do this with the following, which would make an archive
tar czf my_data.tar.gz my_data/*.nii.gz
(2) Run the workflow with a known-working version, and put the outputs into a text file inside
TEDANADIR is your local
We encourage using the convention
to the filename if the integration test uses
tedana in the verbose mode.
(3) Write a test function in
To write the test function you can follow the model of our five echo set, which takes the following steps:
- Check if a pytest user is skipping integration, skip if so
download_test_datato retrieve the test data from OSF
- Run a workflow
check_integration_outputsto compare your expected output to actual output.
(4) If you need to upload new data, you will need to contact the maintainers and ask them to either add it to the tedana OSF project or give you permission to add it.
(5) Once you’ve tested your integration test locally and it is working, you will need to add it to the
CircleCI config and the
Following the model of the three-echo and five-echo sets, define a name for your integration test
and on an indented line below put
@py.test --cov-append --cov-report term-missing --cov=tedana -k TEST
TEST your test function’s name.
This call basically adds code coverage reports to account for the new test, and runs the actual
test in addition.
(6) Using the five-echo set as a template, you should then edit
.circlec/config.yml to add your
test, calling the same name you define in the
Viewing CircleCI Outputs¶
If you need to take a look at a failed test on CircleCI rather than locally, you can use the following block to retrieve artifacts (see CircleCI documentation here)
export CIRCLE_TOKEN=':your_token' curl https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/$build_number/artifacts?circle-token=$CIRCLE_TOKEN \ | grep -o 'https://[^"]*' \ | sed -e "s/$/?circle-token=$CIRCLE_TOKEN/" \ | wget -v -i -
To get a CircleCI token, follow the instructions for getting one. You cannot do this unless you are part of the ME-ICA/tedana organization. If you don’t want all of the artifacts, you can go to the test details and use the browser to manually select the files you would like.
Suppose we want to add a function in
tedana that creates a file called
be stored along the outputs of the
First, we merge the repository’s
master branch into our own to make sure we’re up to date, and
then we make a new branch called something like
Any changes we make will stay on this branch.
We make the new function and call it
say_hello and locate this function inside of
We’ll also need to make a unit test.
(Some developers actually make the unit test before the new function; this is a great way to make
sure you don’t forget to create it!)
Since the function lives in
io.py, its unit test should go into
The job of this test is exclusively to tell if the function we wrote does what it claims to do
So, we define a new function in
test_io.py that looks something like this:
def test_say_hello(): # run the function say_hello() # test the function assert op.exists('hello_world.txt') # clean up os.remove('hello_world.txt')
We should see that our unit test is successful via
pytest $TEDANADIR/tedana/tests/test_io.py -k test_say_hello
If not, we should continue editing the function until it passes our test.
Let’s suppose that suddenly, you realize that what would be even more useful is a function that
takes an argument,
place, so that the output filename is actually
PLACE the value passed and
'world' as the default value.
We merge any changes from the upstream master branch into our branch via
git checkout feature/say_hello git fetch upstream master git merge upstream/master
and then begin work on our test. We need to our unit test to be more complete, so we update it to look more like the following, adding several cases to make sure our function is robust to the name supplied:
def test_say_hello(): # prefix of all files to be checked prefix = 'hello_' # suffix of all files to be checked suffix = '.txt' # run the function with several cases for x in ['world', 'solar system', 'galaxy', 'universe']: # current test name outname = prefix + x + suffix # call the function say_hello(x) # test the function assert op.exists(outname) # clean up from this call os.remove(outname)
Once that test is passing, we may need to adjust the integration test.
Our program creates a file,
hello_world.txt, which the older version would not have produced.
Therefore, we need to add the file to
$TEDANADIR/tedana/tests/data/tedana_outputs.txt and its
counterpart, R2-D2– uh, we mean,
With that edit complete, we can run the full
pytest suite via
Once that filename is added, all of the tests should be passing and we should open a PR to have our change reviewed.
From here, others working on the project may request changes and we’ll have to make sure that our
tests are kept up to date with any changes made as we did before updating the unit test.
For example, if a new parameter is added,
greeting, with a default of
hello, we’ll need to
adjust the unit test.
However, since this doesn’t change the typical workflow of
tedana, there’s no need to change
the integration test; we’re still matching the original filename.
Once we are happy with the changes and some members of
tedana have approved the changes, our
changes will be merged!
We should then do the following cleanup with our git repository:
git checkout master git fetch upstream master git merge upstream/master git branch -d feature/say_hello git push --delete origin feature/say_hello
and we’re good to go!