NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks

Fahim Dalvi, Avery Nortonsmith, Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, James R. Glass

AAAI 2019 pp. 9851-9852

doi:10.1609/AAAI.V33I01.33019851 /aaai/2019/dalvi2019aaai-neurox/

Abstract

We present a toolkit to facilitate the interpretation and understanding of neural network models. The toolkit provides several methods to identify salient neurons with respect to the model itself or an external task. A user can visualize selected neurons, ablate them to measure their effect on the model accuracy, and manipulate them to control the behavior of the model at the test time. Such an analysis has a potential to serve as a springboard in various research directions, such as understanding the model, better architectural choices, model distillation and controlling data biases. The toolkit is available for download.1

PDF AAAI Semantic Scholar

Cite

Text

Dalvi et al. "NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks." AAAI Conference on Artificial Intelligence, 2019. doi:10.1609/AAAI.V33I01.33019851

Markdown

[Dalvi et al. "NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks." AAAI Conference on Artificial Intelligence, 2019.](https://mlanthology.org/aaai/2019/dalvi2019aaai-neurox/) doi:10.1609/AAAI.V33I01.33019851

BibTeX

@inproceedings{dalvi2019aaai-neurox,
  title     = {{NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks}},
  author    = {Dalvi, Fahim and Nortonsmith, Avery and Bau, Anthony and Belinkov, Yonatan and Sajjad, Hassan and Durrani, Nadir and Glass, James R.},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {2019},
  pages     = {9851-9852},
  doi       = {10.1609/AAAI.V33I01.33019851},
  url       = {https://mlanthology.org/aaai/2019/dalvi2019aaai-neurox/}
}