External modules

The MRtrix3 build process allows for the easy development of separate modules, compiled against the MRtrix3 core (or indeed against any other MRtrix3 module). This allows developers to maintain their own repository, or compile stand-alone commands provided by developers / other users, without affecting their core MRtrix3 installation. The obvious benefit is that developers can keep their own developments private if they wish to, and the MRtrix3 core can be kept as lean as possible.

Filesystem structure

A module simply consists of a separate directory, which contains its own cmd/ folder, and potentially also its own src/ folder if required. The build process is then almost identical to that for the MRtrix3 core, with a few differences.

To demonstrate this, we construct a module residing alongside the core installation:

$ mkdir ~/mymodule
$ cd ~/mymodule/
$ mkdir cmd

Assuming our module consists of a single mycommand.cpp C++ file, it should be placed in the cmd/ folder (see below for details). This results in the following folder structure:

mymodule/
|-- cmd/
|   |-- mycommand.cpp

The most relevant difference is how the build script is invoked. For a module, compilation is started by invoking the MRtrix3 core’s build script, but with the module’s top-level folder being the current working directory. For example, if the MRtrix3 core resides in the directory ~/mrtrix3/, and the module resides in ~/mymodule, then the module can be compiled by typing:

$ cd ~/mymodule
$ ../mrtrix3/build

This will compile all .cpp files found in the cmd/ folder, along with any potentially dependencies in the src/ folder (if applicable), and place the resulting executables in the bin/ folder (which will be created if not already present).

Note

Once compiled, the newly created executables will only run if they remain in the same location relative to the MRtrix3 core folder. This is because the runtime search path for the MRtrix3 dynamic library is set within each executable to search in the core MRtrix3 lib/ folder (for our example, this is ../../mrtrix3/lib/). If you need to move your MRtrix3 and module installations, make sure to maintain their relative paths.

Linking to the MRtrix3 core (C++ code only)

For routine use of modules containing C++ code, it is more convenient to set up a reference to the core MRtrix3 installation. This can be done in two ways:

Using a symbolic link to the MRtrix3 core’s build script:
```
$ cd ~/mymodule
$ ln -s ../mrtrix3/build
```
This results in the following folder structure:
```
mymodule/
|-- bin/
|-- build -> ../mrtrix3/build
|-- cmd/
|   |-- mycommand.cpp
```
The link can then be invoked directly, and the build script will detect that it is compiling a module:
```
$ ./build
```
Note that this approach will NOT work on Windows / MSYS2 installations, due to the lack of support for symbolic links. In this case, use the alternative approach below.
Using a text file containing the path to the MRtrix3 core’s build script:
```
$ cd ~/mymodule
$ echo ../mrtrix3/build > build
$ chmod +x build
```
The last command ensures that the file is executable. This results in the following folder structure:
```
mymodule/
|-- bin/
|-- build
|-- cmd/
|   |-- mycommand.cpp
```
The new executable file can then be invoked directly, and the build script will again detect that it is compiling a module:
```
$ ./build
```

Handling Python commands

The instructions above relate to modules containing C++ code. Modules can also contain Python scripts, in which case additional steps will be required to ensure the module’s scripts use the core MRtrix3 python libraries. There are several ways to do this, depending on your circumstances. In all cases, the aim is to ensure that the correct mrtrix3.py module can be located and imported (i.e. that the import mrtrix3 line succeeds).

Symbolic link to the MRtrix3 core mrtrix3.py (recommended): the core installation contains a bin/mrtrix3.py file that will be imported preferentially for any script co-located within the same bin/ folder. It will in turn locate and import the actual mrtrix3 module, which is located in the core lib/mrtrix3 folder. If a symbolic link to that file is placed in the module’s bin/ folder, it will locate the correct modules. For example:
```
$ cd ~/mymodule/
$ ln -sr ../mrtrix3/bin/mrtrix3.py bin/
```
This results in the following folder structure:
```
mymodule/
|-- bin/
|   |-- mrtrix3.py -> ../../mrtrix3/bin/mrtrix3.py
|-- build -> ../mrtrix3/build
|-- cmd/
|   |-- mycommand.cpp
```
Copy of the MRtrix3 core mrtrix3.py file: in some cases, it may not be possible or convenient to use a symbolic link as described above. This is the case particularly on Windows / MSYS2 installations, or when distributing an independent module. In this case, a copy of the core MRtrix3 bin/mrtrix3.py can be placed in the module’s bin/ folder:
```
$ cd ~/mymodule/
$ cp ../mrtrix3/bin/mrtrix3.py bin/
```
This results in the following folder structure:
```
mymodule/
|-- bin/
|   |-- mrtrix3.py
|-- build -> ../mrtrix3/build
|-- cmd/
|   |-- mycommand.cpp
```
In this case, the script will fail to detect the MRtrix3 modules in the normal way, and will instead rely on the build symbolic link or file to locate the core libraries. For this to work, the module must therefore have been set up as suggested in the previous section: either with a build symbolic link pointing the core MRtrix3 build script, or with a build file containing the path to the core build script. The location of the core MRtrix3 build script is then sufficient to locate the core Python libraries, since they should reside in a known location relative to that script.
Use the PYTHONPATH environment variable: some users may prefer to set the PYTHONPATH environment variable to point to the core MRtrix3 lib/ folder. This is the more usual way of locating modules in Python, and will work here also:
```
$ export PYTHONPATH=~/mrtrix3/lib
```
Note

While the PYTHONPATH environment variable will work, there are good reasons not to use this approach. If you have multiple versions of MRtrix3 installed on one system, and use this approach, then the Python modules within whichever of those MRtrix3 versions is added to PYTHONPATH will always be imported, regardless of the version of MRtrix3 against which any particular external module is intended to run. Creation of a bin/mrtrix3.py symbolic link or copy is therefore preferable, as it allows different external modules to run against different MRtrix3 installations.

Adding code to the module

New code can be added to this new module as follows:

Stand-alone .cpp file: a single C++ code file destined to be compiled into a binary executable should have the .cpp file extension, and be placed into the cmd/ directory of the module. Execution of the build script in the module root directory should then detect the presence of this file, and generate an executable file in the corresponding bin/ directory.
Stand-alone Python file: A stand-alone Python script designed to make use of the MRtrix3 Python APIs will typically not have any file extension, and will have its first line set to #!/usr/bin/python3. Such files should be placed directly into the bin/ directory. It will also typically be necessary to mark the file as executable before the system will allow it to run:
```
$ chmod +x bin/example_script
```
(Replace example_script with the name of the script file you have added)
More complex modules: If the requisite code for a particular functionality cannot reasonably be fully encapsulated within a single file, additional files will need to be added to the module. For C++ code, these will need to be added to the src/ directory. For further details, refer to the relevant developer documentation.

For example, the following steps take the example_script Python script and example_binary.cpp C++ files, previously downloaded by the user into the ~/Downloads/ folder, place them in the appropriate locations in the module created as described above, ensure the Python script is executable, and build the C++ executable:

$ cd ~/mymodule
$ cp ~/Downloads/example_script bin/
$ cp ~/Downloads/example_binary.cpp cmd/
$ chmod +x bin/example_script
$ ./build
[1/2] [CC] tmp/cmd/example_binary.o
[2/2] [LD] bin/example_binary

This results in the following folder structure:

mymodule/
|-- bin/
|   |-- example_binary
|   |-- example_script
|   |-- mrtrix3.py -> ../../mrtrix3/bin/mrtrix3.py
|-- build -> ../mrtrix3/build
|-- cmd/
|   |-- example_binary.cpp
|-- tmp/
|   |-- (directories)

Both example command executables – example_binary and example_script – now reside in directory ~/mymodule/bin/. The example_binary executable will be linked against the core MRtrix3 library (in the ~/mrtrix3/lib folder), and the example_script Python script will import modules from the core MRtrix3 Python module (in the ~/mrtrix3/lib/mrtrix3 folder) – neither will run if these libraries are not found.

Adding modules to `PATH`

Because these binaries are not placed into the same directory as those provided as part of the core MRtrix3 installation, simply typing the name of the command into the terminal will not work, as your system will not yet be configured to look for executable files in this new location. You can solve this in one of three ways:

Provide the full path to the binary file when executing it. So for instance, instead of typing:
$ example_binary argument1 argument2 ...
you would use:
$ ~/mymodule/bin/example_binary argument1 argument2 ...
While this may be inconvenient in some circumstances, in others it can be beneficial, as it is entirely explicit and clear as to exactly which version of the command is being run. This is especially useful when experimenting with different versions of a command, where the name of the command has not changed.
Use the set_path script provided with MRtrix3 to automatically add the location of the module’s bin/ directory to PATH whenever a terminal session is created. To do this, execute your core MRtrix3 installation’s set_path script while residing in the top-level directory of the module:
$ cd ~/mymodule
$ ../mrtrix3/set_path
Manually add the location of the bin/ directory of this new module to your system’s PATH environment variable. Most likely you will want this location to be already stored within PATH whenever you open a new terminal; therefore you will most likely want to add a line such as that below to the appropriate configuration file for your system (e.g. ~/.bashrc or ~/.bash_profile; the appropriate file will depend on your particular system):
$ export PATH=~/mymodule/bin:$PATH
Obviously you will need to modify this line according to the location on your file system where you have installed the module.