NaN-Verlust im Tensorflow LSTM-Modell

Question

Der folgende Netzwerkcode, der dein klassisches einfaches LSTM-Sprachmodell sein sollte, beginnt nach einiger Zeit mit der Ausgabe von Nanoverlusten ... auf meinem Trainingset dauert es einige Stunden und ich konnte nicht replizieren es leicht auf kleineren Datensätzen. Aber es passiert immer im ernsthaften Training.

Sparse_softmax_with_cross_entropy sollte numerisch stabil sein, also kann es nicht die Ursache sein ... aber ansonsten sehe ich keinen anderen Knoten, der ein Problem im Graphen verursachen könnte. Was könnte das Problem sein?

class MyLM(): 
    def __init__(self, batch_size, embedding_size, hidden_size, vocab_size): 
     self.x = tf.placeholder(tf.int32, [batch_size, None]) # [batch_size, seq-len] 
     self.lengths = tf.placeholder(tf.int32, [batch_size]) # [batch_size] 

     # remove padding. [batch_size * seq_len] -> [batch_size * sum(lengths)] 
     mask = tf.sequence_mask(self.lengths) # [batch_size, seq_len] 
     mask = tf.cast(mask, tf.int32) # [batch_size, seq_len] 
     mask = tf.reshape(mask, [-1]) # [batch_size * seq_len] 

     # remove padding + last token. [batch_size * seq_len] -> [batch_size * sum(lengths-1)] 
     mask_m1 = tf.cast(tf.sequence_mask(self.lengths - 1, maxlen=tf.reduce_max(self.lengths)), tf.int32) # [batch_size, seq_len] 
     mask_m1 = tf.reshape(mask_m1, [-1]) # [batch_size * seq_len] 

     # remove padding + first token. [batch_size * seq_len] -> [batch_size * sum(lengths-1)] 
     m1_mask = tf.cast(tf.sequence_mask(self.lengths - 1), tf.int32) # [batch_size, seq_len-1] 
     m1_mask = tf.concat([tf.cast(tf.zeros([batch_size, 1]), tf.int32), m1_mask], axis=1) # [batch_size, seq_len] 
     m1_mask = tf.reshape(m1_mask, [-1]) # [batch_size * seq_len] 

     embedding = tf.get_variable("TokenEmbedding", shape=[vocab_size, embedding_size]) 
     x_embed = tf.nn.embedding_lookup(embedding, self.x) # [batch_size, seq_len, embedding_size] 

     lstm = tf.nn.rnn_cell.LSTMCell(hidden_size, use_peepholes=True) 

     # outputs shape: [batch_size, seq_len, hidden_size] 
     outputs, final_state = tf.nn.dynamic_rnn(lstm, x_embed, dtype=tf.float32, 
               sequence_length=self.lengths) 
     outputs = tf.reshape(outputs, [-1, hidden_size]) # [batch_size * seq_len, hidden_size] 

     w = tf.get_variable("w_out", shape=[hidden_size, vocab_size]) 
     b = tf.get_variable("b_out", shape=[vocab_size]) 
     logits_padded = tf.matmul(outputs, w) + b # [batch_size * seq_len, vocab_size] 
     self.logits = tf.dynamic_partition(logits_padded, mask_m1, 2)[1] # [batch_size * sum(lengths-1), vocab_size] 

     predict = tf.argmax(logits_padded, axis=1) # [batch_size * seq_len] 
     self.predict = tf.dynamic_partition(predict, mask, 2)[1] # [batch_size * sum(lengths)] 

     flat_y = tf.dynamic_partition(tf.reshape(self.x, [-1]), m1_mask, 2)[1] # [batch_size * sum(lengths-1)] 

     self.cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=self.logits, labels=flat_y) 
     self.cost = tf.reduce_mean(self.cross_entropy) 
     self.train_step = tf.train.AdamOptimizer(learning_rate=0.01).minimize(self.cost) 

Answer 1

Es kann der Fall von exploding gradients, wo Gradienten während Backpropagation in LSTMs explodieren kann, was Zahl überläuft. Eine übliche Technik, um mit explodierenden Gradienten umzugehen, ist Gradient Clipping durchzuführen.