Throttling FTP Download mit Python ftplib

Question

Wie drossle ich den FTP-Download mit Python ftplib? Zum Beispiel, setze die Geschwindigkeit auf 20Mb/s?

Ich verwende den folgenden Code-Dateien mit Python ftplib zum Download:

from ftplib import FTP 
import os 

download_list = 'testlist.txt' # inital list of directories to be downloaded 
path_list = [] # initalize a list of all the pathes from download_list 
local_folder = 'testStorage' #where files are going to be downloaded to 
downloaded_list = 'completedownload.txt' # list of completed downloads 
error_list = 'incomplete_downloads.txt' # list of paths that are incomplete 


ftp=FTP("ftp.address.com") 
ftp.login("user_name","password") #login to FTP account 
print "Successfully logged in" 

# make a list of files to download from a file 
with open(download_list, 'r') as f: 
    content = f.readlines() 
    path_list = [x.strip() for x in content] 

for path in path_list: 
    path = path.replace("*","") # strips the * found in the source file 
    print '\nChanging directory to ' + path + ':\n' 
    #ftp.cwd('/AAA/BBB/CCC/logic-1/') #the format to change into path note the * is omitted 
    #if ftp.cwd(path) == True: 


    try: # tries the path in the file 
     ftp.cwd(path) 
     #ftp.retrlines('LIST') 
     filenames = ftp.nlst() 
     for filename in filenames: 
      local_directory = local_folder+path # create the local path ie : testStorage/AAA/BBB/CCC/logic-1/ 
      local_filename = os.path.join(local_directory,filename) # 

      if os.path.exists(local_filename) == False: # checks if file already exists 
       if not os.path.exists(local_directory): # mimic the remote path locally 
        os.makedirs(local_directory) 

       file = open(local_filename,'wb') 
       ftp.retrbinary('RETR '+ filename, file.write) 
       print filename 
       file.close() 

      elif os.path.exists(local_filename) == True: # skip the file if it exits 
       print 'File ' +filename + ' already exists, skipping this file' 

    except: #if path in text file does not exist write to error_list.txt 

     print 'Path ' + path + ' does not exist writing path to error_list.txt' 
     with open(error_list, 'a') as f2: 
      f2.write(path+'\n') 
     continue 

print "all done closing connection" 
ftp.close() #CLOSE THE FTP CONNECTION 

Answer 1

um den Download zu drosseln, implementieren nur eine Funktion, die file.write und time.sleep je nach Bedarf der Fall ist. Übergeben Sie diese Funktion an ftp.retrbinary als callback (anstelle von file.write direkt).

Dieser Pseudo-Code (ich weiß nicht Python tun) sollten Ihnen eine Vorstellung geben:

total_length = 0 
start_time = time.time() 

def write_and_sleep(buf): 
    global file 
    global total_length 
    global start_time 
    file.write(buf) 
    total_length += sys.getsizeof(buf) 
    while (total_length/(time.time() - start_time)) > 100000000: 
     time.sleep(0.1) 

ftp.retrbinary('RETR '+ filename, write_and_sleep) 

Reduzierung maxblocksize (das dritte Argument von ftp.retrbinary) kann glatte "Download-Kurve" zu erreichen helfen.