Wie kann ich mehrzeiligen Datensatz aus einer Textdatei mit Spark

Question

Ich habe eine Textdatei. Die Datensätze sind alle Datensätze mit mehreren Zeilen, die durch "\ n \ n \ n \ n \ n" getrennt sind. Die Textdatei sieht wie folgt:

name: Steven 
gender: male 
title: mr. 
company: ABC 

cell 647-777-**** 
home 905-000-**** 
work 289-***-1111 





name: Al 
gender: male 
title: mr. 
company: DEF 

home 905-111-**** 
cell 289-991-**** 

Was habe ich den Code getan unten:

val contact_raw = sc.wholeTextFiles("/user/data/contact.txt").flatMap(x => x._2.split("\n\n\n\n\n")) 
val contact = contact_raw.map(contacts => { 
    val per_person = contacts.split("\n\n") 
    (per_person(0), per_person(1)) 
}).map(
    contact_info => { 
     val personal_info = contact_info._1.split("\n") 
     var name = "" 
     var company = "" 
     var gender = "" 
     var title = "" 
     for (x <- personal_info) { 
      if(x.startsWith("name:")){ 
       name = x.split("name:")(1).trim 
      } else if(x.startsWith("gender:")){ 
       gender = x.split("gender:")(1).trim 
      } else if(x.startsWith("title:")){ 
       title = x.split("title:")(1) 
      } else if(x.startsWith("company:")){ 
       company = x.split("company:")(1) 
      } 
     } 
     val phone_info = contact_info._2.split("\n").map(
       pair => { 
        val phone_pair = pair.split("\\s") 
        (phone_pair(0), phone_pair(1)) 
       } 
      ) 
     (name, gender, title, company, phone_info) 
    } 
).toDF("name", "gender", "title", "company", "phone_info") 

Die Ausgabe lautet:

scala> contact.show 
+------+------+-----+-------+--------------------+ 
| name|gender|title|company|   phone_info| 
+------+------+-----+-------+--------------------+ 
|Steven| male| mr.| ABC|[[cell,647-777-**...| 
| Al| male| mr.| DEF|[[home,905-111-**...| 
+------+------+-----+-------+--------------------+ 

Und das Schema:

scala> contact.printSchema 
root 
|-- name: string (nullable = true) 
|-- gender: string (nullable = true) 
|-- title: string (nullable = true) 
|-- company: string (nullable = true) 
|-- phone_info: array (nullable = true) 
| |-- element: struct (containsNull = true) 
| | |-- _1: string (nullable = true) 
| | |-- _2: string (nullable = true) 

Die erwartete Ausgabe ist:

+------+------+-----+-------+-------------+------------+ 
| name|gender|title|company| phone_type|number  | 
+------+------+-----+-------+-------------+------------+ 
|Steven| male| mr.| ABC|   cell|647-777-****| 
|Steven| male| mr.| ABC|   home|905-000-****| 
|Steven| male| mr.| ABC|   work|289-***-1111| 
| Al| male| mr.| DEF|   home|905-111-****| 
| Al| male| mr.| DEF|   cell|289-991-****| 
+------+------+-----+-------+-------------+------------+ 

Wer kann mir sagen, wie ich meinen Code ändern kann, um die Ausgabe zu erhalten, die ich brauche? Vielen Dank.

Answer 1

Die folgende funktionieren würde:

val contact = contact_raw.map(contacts => { 
     val per_person = contacts.split("\n\n") 
     (per_person(0), per_person(1)) 
    }).flatMap(
     contact_info => { 
     val personal_info = contact_info._1.split("\n") 
     var name = "" 
     var company = "" 
     var gender = "" 
     var title = "" 
     for (x <- personal_info) { 
      if (x.startsWith("name:")) { 
      name = x.split("name:")(1).trim 
      } else if (x.startsWith("gender:")) { 
      gender = x.split("gender:")(1).trim 
      } else if (x.startsWith("title:")) { 
      title = x.split("title:")(1) 
      } else if (x.startsWith("company:")) { 
      company = x.split("company:")(1) 
      } 
     } 
     contact_info._2.split("\n").map(
      pair => { 
      val phone_pair = pair.split("\\s") 
      (name, gender, title, company, phone_pair(0), phone_pair(1)) 
      } 
     ) 
     } 
    ).toDF("name", "gender", "title", "company", "phone_info")