Wie können URL-Variationen wie "www" und "https" am besten mit URL.openConnection() verarbeitet werden?

Question

Wie kann ich beim Öffnen einer Verbindung das beste zu verwendende URL-Format ermitteln?

Viele Websites geben unterschiedliche Ergebnisse zurück, je nachdem, ob die URL "www" und/oder "https" verwendet.

Zum Beispiel, hier ist ein Test, den ich schrieb einige der verschiedenen Ergebnisse zu sehen:

import java.util.Scanner; 
import java.util.ArrayList; 
import java.net.*; 
import java.io.*; 

public class Test { 

    public static void main(String[] args) 
    { 
     String baseURL = "google.com"; 

     try 
     { 
     java.net.URL url = new java.net.URL("http://" + baseURL); 
     java.net.URLConnection connection = url.openConnection(); 
     connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"); 
     BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream())); 

     String line; 
     int lineCount = 0; 

     while ((line = in.readLine()) != null) 
     { 
      lineCount++; 
     } 

     System.out.println("http://" + baseURL + " = " + lineCount + " lines"); 
     } 

     catch (Exception ex) 
     { 
     System.out.println("http://" + baseURL + " throws an error"); 
     } 



     try 
     { 
     java.net.URL url = new java.net.URL("http://www." + baseURL); 
     java.net.URLConnection connection = url.openConnection(); 
     connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"); 
     BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream())); 

     String line; 
     int lineCount = 0; 

     while ((line = in.readLine()) != null) 
     { 
      lineCount++; 
     } 

     System.out.println("http://www." + baseURL + " = " + lineCount + " lines"); 
     } 

     catch(Exception ex) 
     { 
     System.out.println("http://www." + baseURL + " throws an error"); 
     } 







     try 
     { 
     java.net.URL url = new java.net.URL("https://" + baseURL); 
     java.net.URLConnection connection = url.openConnection(); 
     connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"); 
     BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream())); 

     String line; 
     int lineCount = 0; 

     while ((line = in.readLine()) != null) 
     { 
      lineCount++; 
     } 

     System.out.println("https://" + baseURL + " = " + lineCount + " lines"); 
     } 

     catch (Exception ex) 
     { 
     System.out.println("https://" + baseURL + " throws an error"); 
     } 



     try 
     { 
     java.net.URL url = new java.net.URL("https://www." + baseURL); 
     java.net.URLConnection connection = url.openConnection(); 
     connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"); 
     BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream())); 

     String line; 
     int lineCount = 0; 

     while ((line = in.readLine()) != null) 
     { 
      lineCount++; 
     } 

     System.out.println("https://www." + baseURL + " = " + lineCount + " lines"); 
     } 

     catch (Exception ex) 
     { 
     System.out.println("https://www." + baseURL + " throws an error"); 
     } 
    } 
} 

Hier waren die Ergebnisse der es auf vier verschiedenen Websites ausgeführt wird:

http://stackoverflow.com = 4205 lines 
http://www.stackoverflow.com = 4205 lines 
https://stackoverflow.com = 4205 lines 
https://www.stackoverflow.com = 2 lines 

 

http://qvc.com = 2438 lines 
http://www.qvc.com = 2438 lines 
https://qvc.com throws an error 
https://www.qvc.com = 0 lines 

 

http://facebook.com = 0 lines 
http://www.facebook.com = 0 lines 
https://facebook.com = 25 lines 
https://www.facebook.com = 25 lines 

 

http://google.com = 6 lines 
http://www.google.com = 6 lines 
https://google.com = 343 lines 
https://www.google.com = 343 lines 

eine Basis-URL, wie "google.com" In Anbetracht, was ist der richtige Weg, welches Format zu sehen, zu prüfen ich für die Website verwenden sollte?

Answer 1

Nachdem Marc B Antwort zu lesen, ein paar andere Stackoverflow-Threads (die ich in der ursprünglichen Frage Kommentare verbunden sind), und this guide, hier ist, was ich kam mit:

String baseURL = "google.com"; 

try 
{ 
    java.net.URL url = new java.net.URL("http://" + baseURL); 
    java.net.HttpURLConnection connection = (HttpURLConnection)url.openConnection(); 
    connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"); 

    int response = connection.getResponseCode(); 
    System.out.println("Response code: " + response); 

    if (response == 301 || response == 302 || response == 303) 
    { 
      System.out.println("Redirect location: " + connection.getHeaderField("Location")); 
    } 

    else 
    { 
      BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream())); 

      String line; 
      int lineCount = 0; 

      while ((line = in.readLine()) != null) 
      { 
      lineCount++; 
      } 

      System.out.println("http://" + baseURL + " = " + lineCount + " lines\n"); 
    } 
} 

catch (Exception ex) 
{ 
    System.out.println("http://" + baseURL + " throws an error\n"); 
} 

Welche diese Ausgänge:

Response code: 302 
Redirect location: https://www.google.com/?gws_rd=ssl 

Sie können anstelle der numerischen Antwortcodes auch HttpURLConnection.HTTP_MOVED_TEMP, HttpURLConnection.HTTP_MOVED_PERM und HttpURLConnection.HTTP_SEE_OTHER verwenden. Das ist wahrscheinlich eine bessere Praxis.

Answer 2

Überprüfen Sie den HTTP-Antwortcode. Wenn Sie eine Weiterleitung erhalten, haben Sie wahrscheinlich das falsche Format verwendet. z.B. http://www.stackoverflow.com wird eine 301-Weiterleitung zu nur http://stackoverflow.com tun.