Developer guidelines¶
This webpage is intended to guide users through making making changes to
tedana
’s codebase, in particular working with tests.
The worked example also offers some guidelines on approaching testing when
adding new functions.
Please check out our contributing guide for getting started.
Monthly Developer Calls¶
We run monthly developer calls via Zoom. You can see the schedule via the tedana google calendar.
Everyone is welcome. We look forward to meeting you there!
Adding and Modifying Tests¶
Testing is an important component of development.
For simplicity, we have migrated all tests to pytest
.
There are two basic kinds of tests: unit and integration tests.
Unit tests focus on testing individual functions, whereas integration tests focus on making sure
that the whole workflow runs correctly.
Unit Tests¶
For unit tests, we try to keep tests from the same module grouped into one file.
Make sure the function you’re testing is imported, then write your test.
Good tests will make sure that edge cases are accounted for as well as common cases.
You may also use pytest.raises
to ensure that errors are thrown for invalid inputs to a
function.
Integration Tests¶
Adding integration tests is relatively rare.
An integration test will be a complete multi-echo dataset called with some set of options to ensure
end-to-end pipeline functionality.
These tests are relatively computationally expensive but aid us in making sure the pipeline is
stable during large sets of changes.
If you believe you have a dataset that will test tedana
more completely, please open an issue
before attempting to add an integration test.
After securing the appropriate permission from the dataset owner to share it with tedana
, you
can use the following procedure:
(1) Make a tar.gz
file which will unzip to be only the files you’d like to
run a workflow on.
You can do this with the following, which would make an archive my_data.tar.gz
:
tar czf my_data.tar.gz my_data/*.nii.gz
(2) Run the workflow with a known-working version, and put the outputs into a text file inside
$TEDANADIR/tedana/tests/data/
, where TEDANADIR
is your local tedana repository
.
We encourage using the convention <DATASET>_<n_echoes>_echo_outputs.txt
, appending verbose
to the filename if the integration test uses tedana
in the verbose mode.
(3) Write a test function in test_integration.py
.
To write the test function you can follow the model of our five echo set, which takes the following steps:
Check if a pytest user is skipping integration, skip if so
Use
download_test_data
to retrieve the test data from OSFRun a workflow
Use
resources_filename
andcheck_integration_outputs
to compare your expected output to actual output.
(4) If you need to upload new data, you will need to contact the maintainers and ask them to either add it to the tedana OSF project or give you permission to add it.
(5) Once you’ve tested your integration test locally and it is working, you will need to add it to the
CircleCI config and the Makefile
.
Following the model of the three-echo and five-echo sets, define a name for your integration test
and on an indented line below put
@py.test --cov-append --cov-report term-missing --cov=tedana -k TEST
with TEST
your test function’s name.
This call basically adds code coverage reports to account for the new test, and runs the actual
test in addition.
(6) Using the five-echo set as a template, you should then edit .circlec/config.yml
to add your
test, calling the same name you define in the Makefile
.
Viewing CircleCI Outputs¶
If you need to take a look at a failed test on CircleCI rather than locally, you can use the following block to retrieve artifacts (see CircleCI documentation here)
export CIRCLE_TOKEN=':your_token'
curl https://circleci.com/api/v1.1/project/:vcs-type/:username/:project/$build_number/artifacts?circle-token=$CIRCLE_TOKEN \
| grep -o 'https://[^"]*' \
| sed -e "s/$/?circle-token=$CIRCLE_TOKEN/" \
| wget -v -i -
To get a CircleCI token, follow the instructions for getting one. You cannot do this unless you are part of the ME-ICA/tedana organization. If you don’t want all of the artifacts, you can go to the test details and use the browser to manually select the files you would like.
Worked Example¶
Suppose we want to add a function in tedana
that creates a file called `hello_world.txt
to
be stored along the outputs of the tedana
workflow.
First, we merge the repository’s main
branch into our own to make sure we’re up to date, and
then we make a new branch called something like feature/say_hello
.
Any changes we make will stay on this branch.
We make the new function and call it say_hello
and locate this function inside of io.py
.
We’ll also need to make a unit test.
(Some developers actually make the unit test before the new function; this is a great way to make
sure you don’t forget to create it!)
Since the function lives in io.py
, its unit test should go into test_io.py
.
The job of this test is exclusively to tell if the function we wrote does what it claims to do
without errors.
So, we define a new function in test_io.py
that looks something like this:
def test_say_hello():
# run the function
say_hello()
# test the function
assert op.exists('hello_world.txt')
# clean up
os.remove('hello_world.txt')
We should see that our unit test is successful via
pytest $TEDANADIR/tedana/tests/test_io.py -k test_say_hello
If not, we should continue editing the function until it passes our test.
Let’s suppose that suddenly, you realize that what would be even more useful is a function that
takes an argument, place
, so that the output filename is actually hello_PLACE
, with
PLACE
the value passed and 'world'
as the default value.
We merge any changes from the upstream main branch into our branch via
git checkout feature/say_hello
git fetch upstream main
git merge upstream/main
and then begin work on our test. We need to our unit test to be more complete, so we update it to look more like the following, adding several cases to make sure our function is robust to the name supplied:
def test_say_hello():
# prefix of all files to be checked
prefix = 'hello_'
# suffix of all files to be checked
suffix = '.txt'
# run the function with several cases
for x in ['world', 'solar system', 'galaxy', 'universe']:
# current test name
outname = prefix + x + suffix
# call the function
say_hello(x)
# test the function
assert op.exists(outname)
# clean up from this call
os.remove(outname)
Once that test is passing, we may need to adjust the integration test.
Our program creates a file, hello_world.txt
, which the older version would not have produced.
Therefore, we need to add the file to $TEDANADIR/tedana/tests/data/tedana_outputs.txt
and its
counterpart, R2-D2– uh, we mean, tedana_outputs_verbose.txt
.
With that edit complete, we can run the full pytest
suite via
pytest $TEDANADIR/tedana/tests
Once that filename is added, all of the tests should be passing and we should open a PR to have our change reviewed.
From here, others working on the project may request changes and we’ll have to make sure that our
tests are kept up to date with any changes made as we did before updating the unit test.
For example, if a new parameter is added, greeting
, with a default of hello
, we’ll need to
adjust the unit test.
However, since this doesn’t change the typical workflow of tedana
, there’s no need to change
the integration test; we’re still matching the original filename.
Once we are happy with the changes and some members of tedana
have approved the changes, our
changes will be merged!
We should then do the following cleanup with our git repository:
git checkout main
git fetch upstream main
git merge upstream/main
git branch -d feature/say_hello
git push --delete origin feature/say_hello
and we’re good to go!