Sind Impuls und (Mini-) Batch Training kompatibel?

Question

Ich hatte ein Backpropagation-Modell, das einwandfrei funktionierte, jedoch wollte ich Batch-Training implementieren.

-Code vor Batch-Training (in Backpropagation-Funktion), Pseudo-Code:

forevery(connection in this.connections.in){ 
    // Adjust weight 
    var deltaWeight = rate * gradient + momentum * connection.previousDeltaWeight; 
    connection.weight += deltaWeight; 
    connection.previousDeltaWeight = deltaWeight; 
} 

// Adjust bias 
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias; 
this.bias += deltaBias; 

this.previousDeltabias = deltaBias; 

Und der neue Code ist:

forevery(connection in this.connections.in){ 
    // Adjust weight 
    var deltaWeight = rate * gradient * this.mask + momentum * connection.previousDeltaWeight; 
    connection.totalDeltaWeight += deltaWeight; 
    if(update){ 
    connection.weight += connection.totalDeltaWeight; 
    connection.previousDeltaWeight = connection.totalDeltaWeight; 
    connection.totalDeltaWeight = 0; 
    } 
} 

// Adjust bias 
var deltaBias = rate * this.error.responsibility + momentum * this.previousDeltaBias; 
this.totalDeltaBias += deltaBias; 
if(update){ 
    this.bias += this.totalDeltaBias; 
    this.previousDeltaBias = this.totalDeltaBias; 
    this.totalDeltaBias = 0; 
} 

Wenn also die Losgröße 4 ist, wird das Backpropagation genannt 3x mit update=false und das 4. Mal mit update=true. Das Batch-Training funktioniert einwandfrei, aber wenn ich das Momentum (=0.9) einschalte, beginnen alle Werte zu überlaufen. Was könnte das Problem sein?

Answer 1

Wow. Ich sammelte den Impuls falsch. Ich habe es batch_size mal eingefügt was falsch war, also ist es jetzt nur einmal enthalten.

forevery(connection in this.connections.in){ 
    // Adjust weight 
    var deltaWeight = rate * gradient * this.mask; 
    connection.totalDeltaWeight += deltaWeight; 
    if(update){ 
    connection.totalDeltaWeight += momentum * connection.previousDeltaWeight; 
    connection.weight += connection.totalDeltaWeight; 
    connection.previousDeltaWeight = connection.totalDeltaWeight; 
    connection.totalDeltaWeight = 0; 
    } 
} 

// note: MINI_BATCH SHALL BE OPTIMIZED SOON 

// Adjust bias 
var deltaBias = rate * this.error.responsibility; 
this.totalDeltaBias += deltaBias; 
if(update){ 
    this.totalDeltaBias += momentum * this.previousDeltaBias; 
    this.bias += this.totalDeltaBias; 
    this.previousDeltaBias = this.totalDeltaBias; 
    this.totalDeltaBias = 0; 
}