Langsame Speicherleck in data.table wenn benannte Listen in j Rückkehr (versuchen, einen data.table neu zu gestalten)

Question

bearbeitet 3:

Ich habe ein viel kürzeres Beispiel für den Speicherverlust. Ich hoffe es macht es viel einfacher darüber nachzudenken, was vor sich geht. Wenn die Iterationen fortfahren, sehen Sie, dass die Speicherbelegung von gc() VCell stetig ansteigt, während die Speicherbenutzung, die von tables() gemeldet wird, gleich bleibt. Irgendwie scheint der Aufruf von unlist (.SD) verantwortlich zu sein. Hier ist sie:

DT = data.table(k = 1:100, g = 1:20, val = rnorm(2e6)) 
for (i in 1:100){ 
    tmp = DT[ , unlist(.SD), by = 'k'] 
    print(gc()) 
    tables() 
} 

Original-Beitrag:

Ich sehe einige Speicherverhalten Ich verstehe nicht, wenn das data.table Paket. Ich verwende R-2.13.0 mit data.table 1.8.8. Ich laufe auf 64-Bit-suse-Linux.

Mein ultimatives Ziel ist es, eine data.table von "long" zu "wide" zu formatieren und dabei so wenig Speicher wie möglich zu verwenden. Ich folgte einem Vorschlag in einer anderen [SO Post] (Nested if else statements over a number of columns). Im Grunde versuche ich eine data.table umzuformen, die eine benannte Liste im j Ausdruck zurückgibt.

Ich sehe einen stetig steigenden Speicherverbrauch, der wie ein Speicherleck aussieht. Der gesamte von data.tables oder anderen Objekten belegte Speicher berücksichtigt nicht, was in gc() angezeigt wird. Insbesondere beginnt Vcells bei etwa 17 MB und endet bei fast 30 MB, während die von tables() gemeldete Gesamtspeicherbelegung 19 MB beträgt (am Ende). Es gibt keine anderen Objekte (die ich sehen kann), die eine sinnvolle Menge an Speicher verwenden. Wenn Sie den folgenden Code ausführen, wird die zunehmende Speicherbelegung mit print (gc()) -Anweisungen wiederholt angezeigt.

Fehle ich etwas oder gibt es ein Problem mit der Speicherzuweisung in dogroups.c?

Hier ist Code, um das Problem zu reproduzieren, das ich sehe. Irgendwelche Ideen? Ich würde wirklich gerne in der Lage sein, eine data.table relativ effizient zu gestalten, wobei die Speichernutzung eine größere Rolle spielt als die Geschwindigkeit.

library(data.table) 

if(!exists('DT')){ 
    cat('creating DT\n') 
    # make a "long" matrix with 300 columns and keys v,d 
    v = 1:250 
    d = 1:50 
    grid = expand.grid(v,d) 
    DT = data.table(v = grid[,1], d = grid[,2])  
    # now add many columns 
    DT[,sprintf('col%s',1:100) := 1:nrow(DT)]; 
    # set d as key, we don't care much about v for this example 
    setkey(DT,'d') 
} 

# The following code attempts to cast a "long" data.table to "wide" format 
# it is the equivalent the reshape2 call: 
# 
# dcast(melt(DT, c('d','v')), d ~ v + variable, value_var='value') 
# 
# When I run the code I see ever-increasing memory use. sourcing the file 
# repeatedly shows that as well. The total memory used by the input 
# and result data.table or any other objects do not account for the total use. 


# casting patterned after 
# https://stackoverflow.com/questions/15510566/nested-if-else-statements-over-a-number-of-columns/15511689?noredirect=1#comment21968080_15511689 

paste.dash <- function(...){ paste(..., sep='-')}  

# assumes keys is a vector of characters 
dt.melt <- function(dt, keys) { 
    dt[, list(variable = names(.SD), value = unlist(.SD)), by = keys] 
} 

# assumes keys is a vector of characters. 
# all.names is all the column names we expect in the wide data.table 
# we accommodate for the possibility of missing wide table values 
# for some groups by appending NAs for any column names not present. 
# in the particular example above there are no missing values, 
# but the data I intend to run this on does. 
dt.recast<- function(dat, keys, all.names,verbose=FALSE){ 

    if (verbose){ 
    cat(sprintf('dt.recast(): keys = %s\n', paste(keys, collapse=','))) 
    print(gc()) 
    } 
    # id, variable, value 
    m = dt.melt(dat, keys) 

    # m.names will be the wide table column names. 
    m.names = do.call(paste.dash, m[, c(keys,'variable'), with=FALSE]) 

    #append anything that's missing in this group to end of list with NA values 
    missing.names = setdiff(all.names, m.names) 
    missing.vals = rep(NA_real_, length(missing.names)) 
    ret.val = c(m$value, missing.vals) 
    # set names and make a list as required by data.table to generate a wide row 
    ret.val = as.list(setattr(ret.val,'names', c(m.names,missing.names))) 

    if (verbose){ 
    print(gc()) 
    } 

    return(ret.val) 
} 

# turn to wide format row key 'd': columns are cartesian product of v and 
# current non-key columns 

all.wide.names = do.call(paste.dash, expand.grid(unique(DT$v), tail(names(DT),-2))) 

print (gc()) 

DT.wide = DT[ , dt.recast(.SD, 'v', all.wide.names, verbose = TRUE), 
    by = 'd', 
    verbose=TRUE ] 

print (gc()) 

Edit:

#Here is the output of sessionInfo 
> sessionInfo() 
R version 2.13.0 (2011-04-13) 
Platform: x86_64-unknown-linux-gnu (64-bit) 

locale: 
[1] LC_CTYPE=en_US.UTF-8  LC_NUMERIC=C    LC_TIME=en_US.UTF-8  LC_COLLATE=en_US.UTF-8  LC_MONETARY=C    LC_MESSAGES=en_US.UTF-8 LC_PAPER=en_US.UTF-8  LC_NAME=C \ 
       LC_ADDRESS=C 
[10] LC_TELEPHONE=C    LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C 

attached base packages: 
[1] stats  graphics grDevices utils  datasets methods base 

other attached packages: 
[1] data.table_1.8.8 
> 

Edit2: Hier ist eine Ausgabe von zwei aufeinanderfolgenden Läufen.

> source('memory-leak.R') 
data.table 1.8.8 For help type: help("data.table") 
creating DT 
     used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 231906 12.4  407500 21.8 350000 18.7 
Vcells 272022 2.1  786432 6.0 773683 6.0 
Finding groups (bysameorder=TRUE) ... done in 0.001secs. bysameorder=TRUE and o__ is length 0 
Optimization is on but j left unchanged as 'dt.recast(.SD, "v", all.wide.names, verbose = TRUE)' 
Starting dogroups ... dt.recast(): keys = v 
     used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 233168 12.5  467875 25 350000 18.7 
Vcells 292303 2.3  786432 6 773683 6.0 
     used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 258224 13.8  531268 28.4 350000 18.7 
Vcells 474776 3.7  905753 7.0 773683 6.0 
The result of j is a named list. It's very inefficient to create the same names over and over again for each group. When j=list(...), any names are detected, removed and put back after grouping has completed, for efficiency. Using j=transform(), for example, prevents that speedup (consider changing to :=). This message may be upgraded to warning in future. 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283206 15.2  531268 28.4 350000 18.7 
Vcells 1699595 13.0 2029708 15.5 1699607 13.0 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308232 16.5  597831 32 350000 18.7 
Vcells 1882303 14.4 2221551 17 2029708 15.5 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1732347 13.3 2412628 18.5 2029708 15.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32 350000 18.7 
Vcells 1915666 14.7 2613259 20 2284358 17.5 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1764847 13.5 2823921 21.6 2284358 17.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 1948166 14.9 3045117 23.3 2316858 17.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1797347 13.8 3045117 23.3 2316858 17.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 1980666 15.2 3277372 25.1 2349358 18.0 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1829847 14.0 3277372 25.1 2349358 18.0 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2013166 15.4 3277372 25.1 2381858 18.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1862347 14.3 3277372 25.1 2381858 18.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2045666 15.7 3277372 25.1 2414358 18.5 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1894847 14.5 3277372 25.1 2414358 18.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2078166 15.9 3277372 25.1 2446858 18.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1927347 14.8 3277372 25.1 2446858 18.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2110666 16.2 3277372 25.1 2479358 19.0 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1959847 15.0 3277372 25.1 2479358 19.0 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2143166 16.4 3521240 26.9 2511858 19.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 1992347 15.3 3521240 26.9 2511858 19.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2175666 16.6 3521240 26.9 2544358 19.5 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2024847 15.5 3521240 26.9 2544358 19.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2208166 16.9 3521240 26.9 2576858 19.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2057347 15.7 3521240 26.9 2576858 19.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2240666 17.1 3521240 26.9 2609358 20.0 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2089847 16.0 3521240 26.9 2609358 20.0 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2273166 17.4 3521240 26.9 2641858 20.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2122347 16.2 3521240 26.9 2641858 20.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2305666 17.6 3521240 26.9 2674358 20.5 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2154847 16.5 3521240 26.9 2674358 20.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2338166 17.9 3777302 28.9 2706858 20.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2187347 16.7 3777302 28.9 2706858 20.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2370666 18.1 3777302 28.9 2739358 20.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2219847 17.0 3777302 28.9 2739358 20.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2403166 18.4 3777302 28.9 2771858 21.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2252347 17.2 3777302 28.9 2771858 21.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2435666 18.6 3777302 28.9 2804358 21.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2284847 17.5 3777302 28.9 2804358 21.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2468166 18.9 3777302 28.9 2836858 21.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2317347 17.7 3777302 28.9 2836858 21.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2500666 19.1 4046167 30.9 2869358 21.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2349847 18.0 4046167 30.9 2869358 21.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2533166 19.4 4046167 30.9 2901858 22.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2382347 18.2 4046167 30.9 2901858 22.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2565666 19.6 4046167 30.9 2934358 22.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2414847 18.5 4046167 30.9 2934358 22.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2598166 19.9 4046167 30.9 2966858 22.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2447347 18.7 4046167 30.9 2966858 22.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2630666 20.1 4046167 30.9 2999358 22.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2479847 19.0 4046167 30.9 2999358 22.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2663166 20.4 4046167 30.9 3031858 23.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2512347 19.2 4046167 30.9 3031858 23.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2695666 20.6 4328475 33.1 3064358 23.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2544847 19.5 4328475 33.1 3064358 23.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2728166 20.9 4328475 33.1 3096858 23.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2577347 19.7 4328475 33.1 3096858 23.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2760666 21.1 4328475 33.1 3129358 23.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2609847 20.0 4328475 33.1 3129358 23.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2793166 21.4 4328475 33.1 3161858 24.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2642347 20.2 4328475 33.1 3161858 24.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2825666 21.6 4328475 33.1 3194358 24.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2674847 20.5 4328475 33.1 3194358 24.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2858166 21.9 4328475 33.1 3226858 24.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2707347 20.7 4328475 33.1 3226858 24.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2890666 22.1 4624898 35.3 3259358 24.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2739847 21.0 4624898 35.3 3259358 24.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2923166 22.4 4624898 35.3 3291858 25.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2772347 21.2 4624898 35.3 3291858 25.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2955666 22.6 4624898 35.3 3324358 25.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2804847 21.4 4624898 35.3 3324358 25.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 2988166 22.8 4624898 35.3 3356858 25.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2837347 21.7 4624898 35.3 3356858 25.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 3020666 23.1 4624898 35.3 3389358 25.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 2869847 21.9 4624898 35.3 3389358 25.9 

... <snip> ... 

dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 3162347 24.2 5262949 40.2 3681858 28.1 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 3345666 25.6 5262949 40.2 3714358 28.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 3194847 24.4 5262949 40.2 3714358 28.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 3378166 25.8 5262949 40.2 3746858 28.6 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 3227347 24.7 5262949 40.2 3746858 28.6 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 3410666 26.1 5262949 40.2 3779358 28.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 283211 15.2  597831 32.0 350000 18.7 
Vcells 3259847 24.9 5262949 40.2 3779358 28.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 308247 16.5  597831 32.0 350000 18.7 
Vcells 3443166 26.3 5262949 40.2 3811858 29.1 
done dogroups in 10.972 secs 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 258292 13.8  597831 32.0 350000 18.7 
Vcells 3247919 24.8 5262949 40.2 3811858 29.1 
> tables() 
    NAME  NROW MB COLS                    KEY 
[1,] DT  12,500 5 v,d,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,c d 
[2,] DT.wide  50 14 d,1-col1,1-col2,1-col3,1-col4,1-col5,1-col6,1-col7,1-col8,1-col9,1-col10,1-col11 d 
Total: 19MB 
> source('/memory-leak.R') 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 260024 13.9  597831 32.0 350000 18.7 
Vcells 3279245 25.1 5262949 40.2 3859228 29.5 
Finding groups (bysameorder=TRUE) ... done in 0.001secs. bysameorder=TRUE and o__ is length 0 
Optimization is on but j left unchanged as 'dt.recast(.SD, "v", all.wide.names, verbose = TRUE)' 
Starting dogroups ... dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 260400 14.0  597831 32.0 350000 18.7 
Vcells 3297670 25.2 5262949 40.2 3859228 29.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 285438 15.3  597831 32.0 350000 18.7 
Vcells 3480986 26.6 5262949 40.2 3859228 29.5 
The result of j is a named list. It's very inefficient to create the same names over and over again for each group. When j=list(...), any names are detected, removed and put back after grouping has completed, for efficiency. Using j=transform(), for example, prevents that speedup (consider changing to :=). This message may be upgraded to warning in future. 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32.0 350000 18.7 
Vcells 4705194 35.9 5606096 42.8 4781165 36.5 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32.0 374617 20.1 
Vcells 4888513 37.3 5966400 45.6 5257204 40.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32.0 374617 20.1 
Vcells 4737694 36.2 6344720 48.5 5257204 40.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32.0 374617 20.1 
Vcells 4921013 37.6 6741956 51.5 5289704 40.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32.0 374617 20.1 
Vcells 4770194 36.4 7159053 54.7 5289704 40.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 4953513 37.8 7597005 58 5322204 40.7 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4802694 36.7 7597005 58 5322204 40.7 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 4986013 38.1 7597005 58 5354704 40.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4835194 36.9 7597005 58 5354704 40.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 5018513 38.3 7597005 58 5387204 41.2 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4867694 37.2 7597005 58 5387204 41.2 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 5051013 38.6 7597005 58 5419704 41.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4900194 37.4 7597005 58 5419704 41.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 5083513 38.8 7597005 58 5452204 41.6 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4932694 37.7 7597005 58 5452204 41.6 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 5116013 39.1 7597005 58 5484704 41.9 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4965194 37.9 7597005 58 5484704 41.9 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32 374617 20.1 
Vcells 5148513 39.3 7597005 58 5517204 42.1 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32 374617 20.1 
Vcells 4997694 38.2 7597005 58 5517204 42.1 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32.0 374617 20.1 
Vcells 5181013 39.6 8056855 61.5 5549704 42.4 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32.0 374617 20.1 
Vcells 5030194 38.4 8056855 61.5 5549704 42.4 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32.0 374617 20.1 
Vcells 5213513 39.8 8056855 61.5 5582204 42.6 
dt.recast(): keys = v 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32.0 374617 20.1 
Vcells 5062694 38.7 8056855 61.5 5582204 42.6 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32.0 374617 20.1 
Vcells 5246013 40.1 8056855 61.5 5614704 42.9 
dt.recast(): keys = v 

... <snip> ... 

      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 310409 16.6  597831 32.0 374617 20.1 
Vcells 6265194 47.8 9579015 73.1 6784704 51.8 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 335445 18.0  597831 32.0 374617 20.1 
Vcells 6448513 49.2 9579015 73.1 6817204 52.1 
done dogroups in 11.53 secs 
      used (Mb) gc trigger (Mb) max used (Mb) 
Ncells 260003 13.9  597831 32.0 374617 20.1 
Vcells 4978149 38.0 9579015 73.1 6817204 52.1 
> tables() 
    NAME  NROW MB COLS                    KEY 
[1,] DT  12,500 5 v,d,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,c d 
[2,] DT.wide  50 14 d,1-col1,1-col2,1-col3,1-col4,1-col5,1-col6,1-col7,1-col8,1-col9,1-col10,1-col11 d 
Total: 19MB 
> 

Answer 1

UPDATE - Jetzt in v1.8.11 fixiert. Von NEWS:

Lange ausstehende (in der Regel kleine) Speicherverlust in Gruppierung behoben. Wenn die letzte Gruppe kleiner als die größte Gruppe ist, wurde der Unterschied in diese Größen nicht veröffentlicht. Auch in nicht-trivialen Aggregationen wo jede Gruppe eine andere Anzahl von Zeilen zurückgibt. Die meisten Benutzer führen eine Gruppierungsabfrage einmal durch und werden es nie bemerkt haben, aber jeder, der Aufrufe zur Gruppierung durchläuft (z. B. wenn er parallel läuft), kann unter # 2648 gelitten haben. Tests hinzugefügt.

Vielen Dank an Vc273, Y T und andere.

---

Das besondere (gut) Beispiel am Anfang der Frage ist eine „nicht-triviale“ Aggregation angesehen, wenn das Ergebnis jeder Gruppe eine andere Anzahl von Zeilen sein kann, nicht nur ein einzelne aggregiert in eine Reihe.Hinzufügen verbose=TRUE ergibt:

Geschrieben weniger Zeilen (4000000) als zugewiesen (4488000).

und das ist, wo das Leck in diesem Fall war. Nur wichtig, wenn Sie die Gruppierung mehrmals wiederholen müssen, wie es manchmal nötig ist. Das Ergebnis war korrekt.

---

vorherige Antwort erhalten für die Nachwelt ...

diesen Teil Bedenken Sie:

#now add many columns 
for (i in 1:100){ 
    DT[[sprintf('col%s',i)]] = 1:nrow(DT); 
} 

Das ist nicht := oder set() verwendet, das sind die data.table vorgesehen Weisen Spalten durch Verweis hinzufügen. = ist das gleiche wie <-; h., bei jeder Iteration dieser for-Schleife wird die gesamte DT kopiert, um Platz für die einzelne zusätzliche Spalte zu schaffen. Das Speicherleck, das Sie beschreiben, würde mit dieser for Schleife übereinstimmen.

Einige Optionen sind:

• die viele Spalten in einem Rutsch hinzufügen mit cbind
• die Spalten in einem Rutsch hinzufügen mit := z.B. DT[,sprintf('col%s',1:100):=1:nrow(DT)]
• die for Schleife halten aber
:= oder set() bei jeder Iteration

Ich habe verwenden nicht wirklich Ihren Code ausführen, dies zu überprüfen, auch dort später andere Probleme sein kann.

UPDATE: Ich habe jetzt Ihren Code ausgeführt und ich denke, dass ich in der Lage sein könnte zu erraten, was Sie über Speicherverbrauch meinen. Aber das Raten kann viel Zeit in Anspruch nehmen, besonders in solchen Bereichen. Kannst du bitte deutlich darüber hinausgehen:

sehe ich einen stetig steigenden Speicherverbrauch, der wie ein Speicherleck aussieht.

Was genau sehen Sie? d.h., was sind die Zahlen? Woran fängt es an und wo endet es? Wie oft hast du es ausgeführt? Bitte geben Sie auch die Ausgabe von sessionInfo(); obwohl Sie die Version von R (2.13.0) geben, die hilfreich ist, hilft es zu wissen, ob Sie 32bit oder 64bit Linux, Mac oder Windows sind.