Why coRNN converge slower than other models, for example GRU?

[alexxchen]

Hi, very good work here. I come across the slow convergence problem when using coRNN and UniCORNN. I also tried your code in adding task and sMNIST, they just converge slower than GRU.

For example in adding task of T=10000 (look the same for T=1000, 5000):

Are there any theoretical reasons? It is because the symplectic Euler method is a biased estimation.

The GRU class I used:

class GRU(nn.Module):
    def __init__(self, n_inp, n_hid, n_out):
        super(GRU, self).__init__()
        self.n_hid = n_hid
        self.rnn = nn.GRUCell(input_size=n_inp, hidden_size=n_hid)
        self.readout = nn.Linear(n_hid, n_out)

    def forward(self, x):
        ## initialize hidden states
        hidden = None

        for t in range(x.size(0)):
            hidden = self.rnn(x[t], hidden)
        output = self.readout(hidden)

        return output