RobustX
=======
The increasing use of machine learning models to aid decision-making
in high-stakes industries like finance and healthcare demands
explainability to build trust. Counterfactual Explanations (CEs)
provide valuable insights into model predictions by showing how
slight changes in input data could lead to different outcomes. A key
aspect of CEs is their robustness, which ensures that the desired
outcomes remain stable even with minor alterations to the input.
Robustness is important since produced CEs should hold up in the
future should the original model be altered or replaced.Despite the
importance of robustness, there has been a lack of standardised tools
to comprehensively evaluate and compare robust CE generation methods.
To address this, **RobustX** was developed as an open-source Python
library aimed at benchmarking the robustness of various CE methods.
RobustX provides a systematic framework for generating, evaluating,
and comparing CEs with a focus on robustness, enabling fair and
effective benchmarking. The library is highly extensible, allowing
for custom models, datasets, and tools to be integrated, making it an
essential tool for enhancing the reliability and interpretability of
machine learning models in critical applications.
Features
--------
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: html
|
.. raw:: html
.. raw:: html
.. raw:: html
-
Standardises the evaluation and benchmarking of robust CEs.
.. raw:: html
.. raw:: html
-
Supports multiple ML frameworks, including PyTorch, Keras, and
scikit-learn.
.. raw:: html
.. raw:: html
-
Extensible to incorporate custom models, datasets, CE methods, and
evaluation metrics.
.. raw:: html
.. raw:: html
-
Includes several robust CE generation algorithms (e.g., TRexNN, RNCE)
and non-robust baselines (e.g., MCE, BLS).
.. raw:: html
.. raw:: html
.. raw:: html
|
.. raw:: html
.. raw:: html
Setup
-----
The core required packages for working on RobustX are listed in
``environment.yml`` and ``requirements.txt``.
To set up a new virtual environment with Conda, use ``environment.yml``.
This will install the required packages into a new environment called
RobustX.
.. code:: bash
conda env create -f environment.yml
conda activate robustx
Then, start using the library by:
::
pip install -e .
Alternatively, if using an existing Python environment, directly run
``pip install -e .``
Note that one needs `Gurobi `__ optimizer to
run mixed integer programming-based methods. Gurobi offers `free
academic
licenses `__.
Examples
--------
A first example of RobustX is provided below.
.. code:: python
# first prepare a task
from robustx.datasets.ExampleDatasets import get_example_dataset
from robustx.lib.models.pytorch_models.SimpleNNModel import SimpleNNModel
from robustx.lib.tasks.ClassificationTask import ClassificationTask
data = get_example_dataset("ionosphere")
data.default_preprocess()
model = SimpleNNModel(34, [8], 1)
model.train(data.X, data.y)
task = ClassificationTask(model, data)
# specify the names of the methods and evaluations we want to use, run benchmarking
# This will find CEs for all instances predicted with the undesirable class (0) and compare
from robustx.lib import default_benchmark
methods = ["KDTreeNNCE", "MCE", "MCER", "RNCE", "STCE", "PROPLACE"]
evaluations = ["Validity", "Distance", "Delta-robustness"]
default_benchmark(task, methods, evaluations, neg_value=0, column_name="target", delta=0.005)
which will produce an output similar to:
+-----------+------------------+---------+---------+-----------------+
| Method | Execution Time | V | D | D |
| | (s) | alidity | istance | elta-robustness |
+===========+==================+=========+=========+=================+
| K | 0.21686 | 1 | 5.76588 | 0.515625 |
| DTreeNNCE | | | | |
+-----------+------------------+---------+---------+-----------------+
| MCE | 3.44478 | 1 | 3.26922 | 0 |
+-----------+------------------+---------+---------+-----------------+
| MCER | 137.563 | 1 | 5.14061 | 0.648438 |
+-----------+------------------+---------+---------+-----------------+
| RNCE | 3.98173 | 1 | 6.03255 | 1 |
+-----------+------------------+---------+---------+-----------------+
| STCE | 29.6889 | 1 | 6.86523 | 1 |
+-----------+------------------+---------+---------+-----------------+
| PROPLACE | 12.9444 | 1 | 5.96721 | 1 |
+-----------+------------------+---------+---------+-----------------+
A demonstration of how to use the library is available here:
.. code:: bash
conda activate robustx
cd RobustX/demo
streamlit run demo_main.py --theme.base="light"
Python notebooks demonstrating the usage of RobustX are available
`here `__.
The docs pages can be accessed by opening
``docs/build/html/index.html``.
Contributors
------------
- **Junqi Jiang** - junqi.jiang20@imperial.ac.uk
- **Luca Marzari** - luca.marzari@univr.it
- **Aaryan Purohit**
- **Francesco Leofante** - f.leofante@imperial.ac.uk
License
-------
RobustX is licensed under the MIT License. For more details, please
refer to the ``LICENSE`` file in this repository.
robustx documentation
=====================
WIP. Docs automatically generated in codes.
.. toctree::
:maxdepth: 2
:caption: Contents:
modules