Finde die am häufigsten vorkommenden Wörter in einem Text in R

Question

Kann mir jemand helfen, wie man die am häufigsten verwendeten zwei und drei Wörter in einem Text mit R findet?

Mein Text ist ...

text <- c("There is a difference between the common use of the term phrase and its technical use in linguistics. In common usage, a phrase is usually a group of words with some special idiomatic meaning or other significance, such as \"all rights reserved\", \"economical with the truth\", \"kick the bucket\", and the like. It may be a euphemism, a saying or proverb, a fixed expression, a figure of speech, etc. In grammatical analysis, particularly in theories of syntax, a phrase is any group of words, or sometimes a single word, which plays a particular role within the grammatical structure of a sentence. It does not have to have any special meaning or significance, or even exist anywhere outside of the sentence being analyzed, but it must function there as a complete grammatical unit. For example, in the sentence Yesterday I saw an orange bird with a white neck, the words an orange bird with a white neck form what is called a noun phrase, or a determiner phrase in some theories, which functions as the object of the sentence. Theorists of syntax differ in exactly what they regard as a phrase; however, it is usually required to be a constituent of a sentence, in that it must include all the dependents of the units that it contains. This means that some expressions that may be called phrases in everyday language are not phrases in the technical sense. For example, in the sentence I can't put up with Alex, the words put up with (meaning \'tolerate\') may be referred to in common language as a phrase (English expressions like this are frequently called phrasal verbs\ but technically they do not form a complete phrase, since they do not include Alex, which is the complement of the preposition with.") 

Answer 1

Hier ist eine einfache Basis R Ansatz für die 5 häufigsten Wörter:

head(sort(table(strsplit(gsub("[[:punct:]]", "", text), " ")), decreasing = TRUE), 5) 

#  a the  of  in phrase 
# 21  18  12  10  8 

Was es gibt eine ganze Zahl Vektor mit der Frequenzzahl und die Namen des Vektors entsprechen den Wörtern, die gezählt wurden.

• gsub("[[:punct:]]", "", text) Interpunktion zu entfernen, da Sie nicht darauf verlassen will, ich
• strsplit(gsub("[[:punct:]]", "", text), " ") erraten die Zeichenfolge auf Räume
• table() zu zählen einzigartige Elemente Frequenz
• sort(..., decreasing = TRUE) sortieren sie zu spalten in abnehmender Reihenfolge
• head(..., 5) auszuwählen nur die Top-5 häufigsten Wörter

Answer 2

Wir können die Worte aufgeteilt und verwenden Tabelle die Frequenz zusammenzufassen:

words <- strsplit(text, "[ ,.\\(\\)\"]") 
sort(table(words, exclude = ""), decreasing = T) 

Answer 3

Der tidytext Paket macht diese Art der Sache recht einfach:

library(tidytext) 
library(dplyr) 

data_frame(text = text) %>% 
    unnest_tokens(word, text) %>% # split words 
    anti_join(stop_words) %>% # take out "a", "an", "the", etc. 
    count(word, sort = TRUE) %>% # count occurrences 

# Source: local data frame [73 x 2] 
# 
#   word  n 
#   (chr) (int) 
# 1  phrase  8 
# 2  sentence  6 
# 3  words  4 
# 4  called  3 
# 5  common  3 
# 6 grammatical  3 
# 7  meaning  3 
# 8   alex  2 
# 9   bird  2 
# 10 complete  2 
# ..   ... ... 

---

Wenn die Frage nach Zählungen fragt von Bigrammen und Trigrammen, tokenizers::tokenize_ngrams ist nützlich:

library(tokenizers) 

tokenize_ngrams(text, n = 3L, n_min = 2L, simplify = TRUE) %>% # tokenize bigrams and trigrams 
    as_data_frame() %>% # structure 
    count(value, sort = TRUE) # count 

# Source: local data frame [531 x 2] 
# 
#   value  n 
#   (fctr) (int) 
# 1  of the  5 
# 2  a phrase  4 
# 3 the sentence  4 
# 4   as a  3 
# 5  in the  3 
# 6  may be  3 
# 7 a complete  2 
# 8 a phrase is  2 
# 9 a sentence  2 
# 10  a white  2 
# ..   ... ... 

Answer 4

Ihr Text ist:

text <- c("There is a difference between the common use of the term phrase and its technical use in linguistics. In common usage, a phrase is usually a group of words with some special idiomatic meaning or other significance, such as \"all rights reserved\", \"economical with the truth\", \"kick the bucket\", and the like. It may be a euphemism, a saying or proverb, a fixed expression, a figure of speech, etc. In grammatical analysis, particularly in theories of syntax, a phrase is any group of words, or sometimes a single word, which plays a particular role within the grammatical structure of a sentence. It does not have to have any special meaning or significance, or even exist anywhere outside of the sentence being analyzed, but it must function there as a complete grammatical unit. For example, in the sentence Yesterday I saw an orange bird with a white neck, the words an orange bird with a white neck form what is called a noun phrase, or a determiner phrase in some theories, which functions as the object of the sentence. Theorists of syntax differ in exactly what they regard as a phrase; however, it is usually required to be a constituent of a sentence, in that it must include all the dependents of the units that it contains. This means that some expressions that may be called phrases in everyday language are not phrases in the technical sense. For example, in the sentence I can't put up with Alex, the words put up with (meaning \'tolerate\') may be referred to in common language as a phrase (English expressions like this are frequently called phrasal verbs\ but technically they do not form a complete phrase, since they do not include Alex, which is the complement of the preposition with.") 

In Natural Language Processing, 2-Wort-Sätze werden als "Bigramm-" und 3-Wort-Sätze werden als "Tri-Gramm bezeichnet ", und so weiter. Im Allgemeinen wird eine gegebene Kombination von n Wörtern als "n-gram" bezeichnet.

Erstens haben wir das ngram Paket (auf CRAN) Dann

# Install package "ngram" 
install.packages("ngram") 

installieren, werden wir die häufigsten Zwei-Wort-und Drei-Wort-Sätze

library(ngram) 

# To find all two-word phrases in the test "text": 
ng2 <- ngram(text, n = 2) 

# To find all three-word phrases in the test "text": 
ng3 <- ngram(text, n = 3) 

Schließlich finden, werden wir drucken, um die Objekte (ngrams), wie unten mit verschiedenen Methoden:

print(ng, output="truncated") 

print(ngram(x), output="full") 

get.phrasetable(ng) 

ngram::ngram_asweka(text, min=2, max=3) 

We c eine Verwendung auch Markov-Ketten neue Sequenzen plappern:

# if we are using ng2 (bi-gram) 
lnth = 2 
babble(ng = ng2, genlen = lnth) 

# if we are using ng3 (tri-gram) 
lnth = 3 
babble(ng = ng3, genlen = lnth) 

Answer 5

Simplest?

require(quanteda) 

# bi-grams 
topfeatures(dfm(text, ngrams = 2, verbose = FALSE)) 
##  of_the  a_phrase the_sentence  may_be   as_a  in_the in_common phrase_is 
##   5   4   4   3   3   3   2   2 
## is_usually  group_of 
##   2   2 

# for tri-grams 
topfeatures(dfm(text, ngrams = 3, verbose = FALSE)) 
##  a_phrase_is group_of_words of_a_sentence of_the_sentence for_example_in example_in_the 
##    2    2    2    2    2    2 
## in_the_sentence an_orange_bird orange_bird_with  bird_with_a 
#    2    2    2    2