Extrahieren von Text aus einem HTML-Dokument mit BeautifulSoup in Python

Question

Ich versuche, Text von den Eltern Kommentaren auf der Website zu extrahieren songmeanings.com mit BeautifulSoup aus der folgenden HTML:

<div class="text" id="comment-73014911864"> 
 
<strong class="title"> 
 
    General Comment 
 
</strong> 
 
This is a beautiful song. I love it a lot. He is the ONLY, and yes, ONLY rapper I will listen to. Because, 
 
<br/> 
 
<br/> 
 
(a) His songs have meaning. They're not about sex and cars and bling blingin' rims. 
 
<br/> 
 
(b) He has talent. He can actually rap. I don't think d12 is any good. =/ 
 
<br/> 
 
<br/> 
 
Anyway. I love this song and I'm getting his new CD right now... hehe. 
 
<br/> 
 
-Sarah 
 
<div class="sign"> 
 
    <a class="author" href="/profiles/view/17067478/" id="userprofile-17067478" rel="me nofollow" title="xoDonnieDarko"> 
 
    xoDonnieDarko 
 
    </a> 
 
    <em class="date"> 
 
    on December 06, 2005 
 
    </em> 
 
    <a href="/songs/view/3530822107858560012/?&amp;specific_com=73014911864#comments" id="specific_com-73014911864" rel="nofollow" title="Permalink"> 
 
    Link 
 
    </a> 
 
</div> 
 
<ul class="answers"> 
 
    <li> 
 
    <div class="title"> 
 
    <a class="replies close-replies" href="#" id="showreplies-73014911864" rel="nofollow" title="3 Replies"> 
 
    3 Replies 
 
    </a> 
 
    <span class="login"> 
 
    <a class="lightbox" href="#popup-loginform" rel="nofollow"> 
 
     Log in to reply 
 
    </a> 
 
    </span> 
 
    <br> 
 
    </br> 
 
    </div> 
 
    <div id="formreply-73014911864" style="display: none;"> 
 
    <!-- comment-form --> 
 
    <form action="#" class="comment-form-reply" id="comment-form-reply-73014911864"> 
 
    <div class="area" id="reply-errors-box" style="display: none;"> 
 
     <label for="type"> 
 
     </label> 
 
     <span id="reply-errors" style="color: #ff0000;"> 
 
     There was an error. 
 
     </span> 
 
    </div> 
 
    <div class="area"> 
 
     <div class="textarea"> 
 
     <div class="holder"> 
 
     <div class="frame"> 
 
     <textarea class="frmreplycomment-73014911864" id="frmreplycomment" name="frmreplycomment"> 
 
      @xoDonnieDarko 
 
     </textarea> 
 
     </div> 
 
     </div> 
 
     </div> 
 
    </div> 
 
    <input id="frmreplylid" name="frmreplylid" type="hidden" value="3530822107858560012"> 
 
     <input id="frmaid" name="frmaid" type="hidden" value="94"> 
 
     <input id="frmreplycid" name="frmreplycid" type="hidden" value="73014911864"> 
 
     <input class="submit" type="submit" value="Add reply"/> 
 
     </input> 
 
     </input> 
 
    </input> 
 
    </form> 
 
    </div> 
 
    <div id="thesereplies-73014911864" style="display: none;"> 
 
    <div class="answer-holder" id="fullcomment-73015890665"> 
 
    <a name="comment-73015890665"> 
 
    </a> 
 
    <div id="rating-holder-73015890665"> 
 
     <div class="numb-holder"> 
 
     <span id="com-rating-73015890665"> 
 
     <strong class="numb" id="numb-rating-73015890665"> 
 
     +1 
 
     </strong> 
 
     </span> 
 
     <div class="com-whorated" id="com-whorated-73015890665" style="display: none; text-align: center;"> 
 
     <span class="processing"> 
 
     </span> 
 
     </div> 
 
     <div id="processing-73015890665" style="text-align: center; padding: 8px 8px 0px 12px; display: none;"> 
 
     <span class="processing"> 
 
     </span> 
 
     </div> 
 
     </div> 
 
    </div> 
 
    <div class="text"> 
 
     i agree he is the only rapper i can listen too. 
 
     <div class="sign"> 
 
     <span id="flagspan-73015890665"> 
 
     <a class="flag" href="#" id="flag-73015890665"> 
 
     Flag 
 
     </a> 
 
     </span> 
 
     <a class="author" href="/profiles/view/17374833/" id="userprofile-17374833" rel="me nofollow" title="byrdman1992"> 
 
     byrdman1992 
 
     </a> 
 
     <em class="date"> 
 
     on March 15, 2010 
 
     </em> 
 
     </div> 
 
    </div> 
 
    </div> 
 
    <div class="answer-holder" id="fullcomment-73015961779"> 
 
    <a name="comment-73015961779"> 
 
    </a> 
 
    <div id="rating-holder-73015961779"> 
 
     <div class="numb-holder"> 
 
     <span id="com-rating-73015961779"> 
 
     <strong class="numb" id="numb-rating-73015961779"> 
 
     0 
 
     </strong> 
 
     </span> 
 
     <div class="com-whorated" id="com-whorated-73015961779" style="display: none; text-align: center;"> 
 
     <span class="processing"> 
 
     </span> 
 
     </div> 
 
     <div id="processing-73015961779" style="text-align: center; padding: 8px 8px 0px 12px; display: none;"> 
 
     <span class="processing"> 
 
     </span> 
 
     </div> 
 
     </div> 
 
    </div> 
 
    <div class="text"> 
 
     same her the ONLY one...and sometimes lil' wayne! lol 
 
     <div class="sign"> 
 
     <span id="flagspan-73015961779"> 
 
     <a class="flag" href="#" id="flag-73015961779"> 
 
     Flag 
 
     </a> 
 
     </span> 
 
     <a class="author" href="/profiles/view/17418133/" id="userprofile-17418133" rel="me nofollow" title="dancer017"> 
 
     dancer017 
 
     </a> 
 
     <em class="date"> 
 
     on August 26, 2010 
 
     </em> 
 
     </div> 
 
    </div> 
 
    </div> 
 
    <div class="answer-holder" id="fullcomment-73016306033"> 
 
    <a name="comment-73016306033"> 
 
    </a> 
 
    <div id="rating-holder-73016306033"> 
 
     <div class="numb-holder"> 
 
     <span id="com-rating-73016306033"> 
 
     <strong class="numb" id="numb-rating-73016306033"> 
 
     0 
 
     </strong> 
 
     </span> 
 
     <div class="com-whorated" id="com-whorated-73016306033" style="display: none; text-align: center;"> 
 
     <span class="processing"> 
 
     </span> 
 
     </div> 
 
     <div id="processing-73016306033" style="text-align: center; padding: 8px 8px 0px 12px; display: none;"> 
 
     <span class="processing"> 
 
     </span> 
 
     </div> 
 
     </div> 
 
    </div> 
 
    <div class="text"> 
 
     <a href="/profiles/view/17067478/?mention=12eeb84af5d911243541dc3bf651fc7b" id="userprofile-17067478" rel="me nofollow" title="@xoDonnieDarko"> 
 
     @xoDonnieDarko 
 
     </a> 
 
     RIttz is pretty good.. Can listen to yela and tech too. 
 
     <div class="sign"> 
 
     <span id="flagspan-73016306033"> 
 
     <a class="flag" href="#" id="flag-73016306033"> 
 
     Flag 
 
     </a> 
 
     </span> 
 
     <a class="author" href="/profiles/view/17643918/" id="userprofile-17643918" rel="me nofollow" title="Heeltoehole"> 
 
     Heeltoehole 
 
     </a> 
 
     <em class="date"> 
 
     on September 05, 2015 
 
     </em> 
 
     </div> 
 
    </div> 
 
    </div> 
 
    </div> 
 
    </li> 
 
</ul> 
 
</div> 
 

 
<div class="text"> 
 
i agree he is the only rapper i can listen too. 
 
<div class="sign"> 
 
    <span id="flagspan-73015890665"> 
 
    <a class="flag" href="#" id="flag-73015890665"> 
 
    Flag 
 
    </a> 
 
    </span> 
 
    <a class="author" href="/profiles/view/17374833/" id="userprofile-17374833" rel="me nofollow" title="byrdman1992"> 
 
    byrdman1992 
 
    </a> 
 
    <em class="date"> 
 
    on March 15, 2010 
 
    </em> 
 
</div> 
 
</div> 
 

 
<div class="text"> 
 
same her the ONLY one...and sometimes lil' wayne! lol 
 
<div class="sign"> 
 
    <span id="flagspan-73015961779"> 
 
    <a class="flag" href="#" id="flag-73015961779"> 
 
    Flag 
 
    </a> 
 
    </span> 
 
    <a class="author" href="/profiles/view/17418133/" id="userprofile-17418133" rel="me nofollow" title="dancer017"> 
 
    dancer017 
 
    </a> 
 
    <em class="date"> 
 
    on August 26, 2010 
 
    </em> 
 
</div> 
 
</div> 
 

 
<div class="text"> 
 
<a href="/profiles/view/17067478/?mention=12eeb84af5d911243541dc3bf651fc7b" id="userprofile-17067478" rel="me nofollow" title="@xoDonnieDarko"> 
 
    @xoDonnieDarko 
 
</a> 
 
RIttz is pretty good.. Can listen to yela and tech too. 
 
<div class="sign"> 
 
    <span id="flagspan-73016306033"> 
 
    <a class="flag" href="#" id="flag-73016306033"> 
 
    Flag 
 
    </a> 
 
    </span> 
 
    <a class="author" href="/profiles/view/17643918/" id="userprofile-17643918" rel="me nofollow" title="Heeltoehole"> 
 
    Heeltoehole 
 
    </a> 
 
    <em class="date"> 
 
    on September 05, 2015 
 
    </em> 
 
</div> 
 
</div>

Verwendung Mit diesem Code kann ich den meisten Text aus den Kommentaren extrahieren, aber alle Kommentare mit Zeilenumbrüchen haben einen fehlenden Inhalt:

Was gibt die Ausgabe:

Dies ist ein schönes Lied. Ich liebe es sehr. Er ist der EINZIGE, und ja, NUR Rapper, den ich hören werde. Denn

Was ich will, ist:

Dies ist ein schönes Lied. Ich liebe es sehr. Er ist der EINZIGE, und ja, NUR Rapper, den ich hören werde. Denn,

(a) Seine Lieder haben eine Bedeutung. Es geht nicht um Sex und Autos und bling blingin Felgen.
(b) Er hat Talent. Er kann tatsächlich rappen. Ich denke nicht, dass D12 gut ist. =/

Wie auch immer. Ich liebe dieses Lied und ich bekomme gerade seine neue CD ... hehe.
-Sarah

Wie kann ich den gesamten Text von einem übergeordneten Kommentar zu extrahieren? Gibt es einen besseren Weg dies zu tun als das starke Tag?

Answer 1

ich leicht modifiziert https://stackoverflow.com/a/11809215/42346 (geben Sie ihm eine upvote!), Um diese Lösung zu erhalten:

def loop_until(text,first_elem): 
    try: 
    text += first_elem.string 
    if first_elem.next == first_elem.find_next('div'): 
     return text 
    else: 
     return loop_until(text,first_elem.next.next) 
    except TypeError: 
    pass 

Nennen Sie es wie folgt aus:

next_elem = soup.find_all('strong')[0].nextSibling 
loop_until('',next_elem) 

Ergebnis:

u"\n This is a beautiful song. I love it a lot. He is the ONLY, and yes, ONLY rapper I will listen to. Because,\n \n\n (a) His songs have meaning. They're not about sex and cars and bling blingin' rims.\n \n (b) He has talent. He can actually rap. I don't think d12 is any good. =/\n \n\n Anyway. I love this song and I'm getting his new CD right now... hehe.\n \n -Sarah\n "