Download the bundle
tensorflow-lingvo_-_2019-02-26_02-33-30.bundle and run:
git clone tensorflow-lingvo_-_2019-02-26_02-33-30.bundle -b master
Lingvo
Lingvo
What is it?
Lingvo is a framework for building neural networks in Tensorflow, particularlysequence models.
A list of publications using Lingvo can be found here.
Quick start
Docker
The easiest way to get started is to use the providedDocker script. If instead you want to install itdirectly on your machine, skip to the section below.
First,install docker. Then,the following commands should give you a working shell with Lingvo installed.
shellLINGVO_DIR="/tmp/lingvo" # (change to the cloned lingvo directory, e.g. "$HOME/lingvo")LINGVO_DEVICE="gpu" # (Leave empty to build and run CPU only docker)sudo docker build --tag tensorflow:lingvo $(test "$LINGVO_DEVICE" = "gpu" && echo "--build-arg base_image=nvidia/cuda:10.0-cudnn7-runtime-ubuntu16.04") - < ${LINGVO_DIR}/docker/dev.dockerfilesudo docker run --rm $(test "$LINGVO_DEVICE" = "gpu" && echo "--runtime=nvidia") -it -v ${LINGVO_DIR}:/tmp/lingvo -v ${HOME}/.gitconfig:/home/${USER}/.gitconfig:ro -p 6006:6006 -p 8888:8888 --name lingvo tensorflow:lingvo bashbazel test -c opt //lingvo:trainer_test //lingvo:models_test
Installing directly
This is an alternative to using Docker as described in the section above.
The prerequisites are:
- a TensorFlow installation (for nowtf-nightly is required),
- a
C++ compiler (only g++ 4.8 is officially supported), and - the bazel build system.
Refer to docker/dev.dockerfile for more specificdetails.
Running the MNIST image model
Preparing the input data
shellmkdir -p /tmp/mnistbazel run -c opt //lingvo/tools:keras2ckpt -- --dataset=mnist --out=/tmp/mnist/mnist
You will get the following files in /tmp/mnist:
mnist.data-00000-of-00001: 53MB.mnist.index: 241 bytes.
Running the model
To run the trainer in single-machine mode, use
shellbazel build -c opt //lingvo:trainerbazel-bin/lingvo/trainer --run_locally=cpu --mode=sync --model=image.mnist.LeNet5 --logdir=/tmp/mnist/log --logtostderr
After a few seconds, the training accuracy should reach 85% at step 100, asseen in the following line.
INFO:tensorflow:step: 100 accuracy:0.85546875 log_pplx:0.46025506 loss:0.46025506 num_preds:256 num_samples_in_batch:256
The artifacts will be produced in /tmp/mnist/log/control:
params.txt: hyper-parameters.model_analysis.txt: model sizes for each layer.train.pbtxt: the training tf.GraphDef.events.*: a tensorboard events file.
In the /tmp/mnist/log/train directory, one will obtain:
ckpt-*: the checkpoint files.checkpoint: a text file containing information about the checkpoint files.
Running the machine translation model
To run a more elaborate model, you'll need a cluster with GPUs. Please refer tolingvo/tasks/mt/README.md for more information.
Current models
Automatic Speech Recogition
Image
Language Modelling
Machine Translation
[1]: Listen, Attend and Spell. WilliamChan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. ICASSP 2016.
[2]: End-to-end Continuous Speech Recognition using Attention-based RecurrentNN: First Results. Jan Chorowski, DzmitryBahdanau, Kyunghyun Cho, and Yoshua Bengio. arXiv 2014.
[3]:Gradient-based learning applied to document recognition.Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. IEEE 1998.
[4]:Exploring the Limits of Language Modeling.Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu.arXiv, 2016.
[5]: The Best of Both Worlds: Combining Recent Advances in Neural MachineTranslation. Mia X. Chen, Orhan Firat,Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, MikeSchuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, LukaszKaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. ACL 2018.
References
Please cite this paper when referencingLingvo.
@misc{shen2019lingvo, title={Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling}, author={Jonathan Shen and Patrick Nguyen and Yonghui Wu and Zhifeng Chen and others}, year={2019}, eprint={1902.08295}, archivePrefix={arXiv}, primaryClass={cs.LG}}
Source:
https://github.com/tensorflow/lingvoUploader:
tensorflowUpload date: 2018-07-24
