Why does my neural network run slowly

I downloaded a seq2seq model from https://github.com/google/seq2seq. I used it to train an NMT model with 2-layer encoder and 4-layer decoder. It takes about 22 seconds per 100 steps.

Then, I use tensorflow to implement a seq2seq model with 1-layer encoder and 1-layer decoder by myself, and use it to train a NMT model. It seems that the model is simpler, but it will take about 33 seconds to train 100 steps.

What are possible reasons of this situation? What should I do to make my neural network run faster?