ML Anthology

Structured Regularizer for Neural Higher-Order Sequence Models

Martin Ratajczak, Sebastian Tschiatschek, Franz Pernkopf
ECML-PKDD 2015 pp. 168-183
doi:10.1007/978-3-319-23528-8_11 /ecmlpkdd/2015/ratajczak2015ecmlpkdd-structured/

Abstract

We introduce both joint training of neural higher-order linear-chain conditional random fields (NHO-LC-CRFs) and a new structured regularizer for sequence modelling. We show that this regularizer can be derived as lower bound from a mixture of models sharing parts, e.g. neural sub-networks, and relate it to ensemble learning. Furthermore, it can be expressed explicitly as regularization term in the training objective. We exemplify its effectiveness by exploring the introduced NHO-LC-CRFs for sequence labeling. Higher-order LC-CRFs with linear factors are well-established for that task, but they lack the ability to model non-linear dependencies. These non-linear dependencies, however, can be efficiently modeled by neural higher-order input-dependent factors. Experimental results for phoneme classification with NHO-LC-CRFs confirm this fact and we achieve state-of-the-art phoneme error rate of $16.7\%$ on TIMIT using the new structured regularizer.

PDF ECML-PKDD Semantic Scholar

Cite

Text

Ratajczak et al. "Structured Regularizer for Neural Higher-Order Sequence Models." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2015. doi:10.1007/978-3-319-23528-8_11

Markdown

[Ratajczak et al. "Structured Regularizer for Neural Higher-Order Sequence Models." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2015.](https://mlanthology.org/ecmlpkdd/2015/ratajczak2015ecmlpkdd-structured/) doi:10.1007/978-3-319-23528-8_11

BibTeX

@inproceedings{ratajczak2015ecmlpkdd-structured,
  title     = {{Structured Regularizer for Neural Higher-Order Sequence Models}},
  author    = {Ratajczak, Martin and Tschiatschek, Sebastian and Pernkopf, Franz},
  booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
  year      = {2015},
  pages     = {168-183},
  doi       = {10.1007/978-3-319-23528-8_11},
  url       = {https://mlanthology.org/ecmlpkdd/2015/ratajczak2015ecmlpkdd-structured/}
}