CUED-RNNLM Toolkit

Introduction

CUED-RNNLM toolkit present an implementation of RNNLM training (on GPU), and efficient evaluation(on CPU). We have recently released CUED-RNNLM v1.0, which support LSTM, GRU, Highway structure, as well as more flexible deep structure.

more recently, we released CUED-RNNLM v1.1, which support su-RNNLMs.

Code and example for lattice rescoring on HTK and Kaldi lattice are available as well, please feel free to send me email (chenxie95@gmail.com) if you observed any issues.

We strongly recommend you to use CUED-RNNLM v1.0 and CUED-RNNLM v1.1 (for su-RNNLM), and the old version (CUED-RNNLM v0.1) won't be maintained

CUED-RNNLM.v1.1 - CUED-RNNLM.v1.0 - CUED-RNNLM.v0.1 - Contact - Reference - License

CUED-RNNLM v1.1

cued-rnnlm code

cued-rnnlm evalation on CPU

HTK HLRescore

HTK Lattice Rescore example (including su-RNNLM lattice rescoring)

Highlights

Add support for su-RNNLMs (check out our ASRU2017 paper)

CUED-RNNLM v1.0 usage

cued-rnnlm code

cued-rnnlm evalation on CPU

HTK HLRescore, HTK Lattice Rescore example

Kaldi Patch (support to rescore Kaldi lattice directly), Kaldi Lattice Rescore example

CUED RNNLM Toolkit Document v1.0

Recipe for training and evaluation (PPL and N-best rescoring)

Highlights

• Support LSTM, GRU, Highway structure

• Support dropout

• Support various deep structure

CUED-RNNLM v0.1

cued-rnnlm v0.1 (both linux and windows supported)

cued-rnnlm v0.1a (modify gradient clipping for NCE training with shared noise sample(under testing))

tools to evaluate cued-rnnlm on CPU

HTK 3.4.1 patch(RNNLM latrescore supported in HLRescore) HTKLib change file list HTKTools change file list

Example of RNNLM lattice rescore

Kaldi lattice to HTK lattice Convert Tools HTK and Kaidi Lattices

External Link

HTK 3.5 supports RNNLMs generated by CUED-RNNLM toolkit for lattice rescoring now!

Highlights

• Improved RNNLM training criteria
  Cross Entropy
  Variance Regularization
  Noise Contrastive Estimation

• RNNLMs efficiently trained on GPU with spliced sentence bunch

• Unnormalized RNNLM probability to be applied for fast evaluation

• RNNLM trained with additional feature in input layer for better performance (e.g. LDA topic representation)

• ReLU supported as hidden node for better convergence

Document

CUED RNNLM Toolkit

AMI Data and Recipe

Recipe

log file

Usage

rnnlm is invoked by typing the commend line

rnnlm {-train | -ppl | -nbest | -sample} [options]

-train	RNNLM training (GPU supported only)
-ppl	RNNLM evaluation for calculation of perplexity (CPU supported only)
-nbest	RNNLM evaluation for N best rescoring (CPU supported only)
-sample	Sample a specified number of words from a well-trained RNNLM (GPU supported only)

Details about the options could be found from the toolkit document

Contact

Xie Chen - chenxie95 at gmail.com

References

• X. Chen, X. Liu, A. Ragni, Y. Wang, M.J.F. Gales and P.C. Woodland (2017). "Future Word Context in Neural Network Language Models" .
  In Proceedings of ASRU .

• X. Chen, X. Liu, Y. Qian, M.J.F. Gales and P.C. Woodland (2016). "CUED-RNNLM -- An Open-Source Toolkit for Efficient Training and Evaluation of Recurrent Neural Network Language Models" .
  In Proceedings of ICASSP.

• X. Chen, T. Tan, X. Liu, P. Lanchantin, M. Wan, M.J.F. Gales and P.C. Woodland (2015). Recurrent Neural Network Language Model Adaptation for Multi-Genre Broadcast Speech Recognition .
  In Proceedings of INTERSPEECH.

• X. Chen, X. Liu, M.J.F. Gales, P.C. Woodland (2015). Recurrent Neural Network Language Model Training with Noise Contrastive Estimation for Speech Recognition .
  In Proceedings of ICASSP.

X. Chen, X. Liu, M.J.F. Gales, P.C. Woodland (2015). Improving the Training and Evaluation Efficiency of Recurrent Neural Network Language Models .
In Proceedings of ICASSP.

• X. Chen, Y. Wang, X. Liu, M.J.F. Gales and P.C. Woodland (2014). Efficient GPU-based Training of Recurrent Neural Network Language Models Using Spliced Sentence Bunch
  In Proceedings of INTERSPEECH.

• X. Liu, Y. Wang, X. Chen, M.J.F. Gales and P.C. Woodland (2014). EEfficientLattice Rescoring Using Recurrent Neural Network Language Model In Proceedings of ICASSP.

Acknowledgement

This work is supported by Toshiba Research Europe Ltd, Cambridge Research Lab. It was also supported by EPSRC grant EP/I031022/1 (Natural Speech Technology).

We would like to thank Andreas Stolcke for porting it into windows (MS Visual C) and valuable advices.

We would like to thank Ricky Chan for providing the Kaldi lattice rescoring code for CUED-RNNLM v0.1

License

This toolkit is freely available under the BSD license and copyright from the RNNLM toolkit.