Tag Archives: r computing

Visual Representation of Complex Data Using R: The tm and wordcloud Packages: Part One Revisited and Updated


I am uploading this “revision of the Revisited and updated” post in a plain text format to better facilliate the use of RSS feed readers to access the article. Links to the PDF version of this posting and the SOTU text file are at the bottom of the my previous post.

After a hiatus during which I retired from a teaching career I am returning to a project that I started nearly five years ago. I have published on this site many tutorials using R analysis to examine large data sets with many variables. My last post on this was part one of a project that involves large data sets composed of text data such as speeches. Virtually any collection of text can be processed using R and numerous R packages have been developed to generate various types of statistical analysis such word frequency counts and correlations among word frequencies. These statistics help enhance the information that is presented visually in the wordcloud graphic generated by the packages discussed in the Part One tutorial.

In this updated version of the Part One tutorial I will review the use of these tools using updated R packages and an analysis using text of the 2026 State of the Union Address. As indicated in Part One, full texts of State of the Union Addresses are available in several formats that can be downloaded. These cover Presidents from George W. Bush to the current term of Donald Trump, and can be found at https://www.govinfo.gov/features/state-of-the-union

There are other sources for SOTU texts but the government website allows quick selection of a file and one click to download the file. I simply save the selected file into a Windows Notepad text file. Make sure when you view the downloaded file that it is in an 80 column format and not formatted to fit a small screen such as a cellphone or tablet. I have found at least a couple of these files that are in the latter format and are not read correctly by the R functions presented in this paper.

The 2026 address was formatted originally in a small screen format. I found a wide-column transcript that I have used in this analysis. A link to the file is found on this website at the end of the article.

This section contains the updated R code that was presented in part one of this project. As indicated above, the resulting wordcloud is from the 2026 State of the Union. Additional code is presented in this section to calculate and print a word frequency table and to print a matrix of word association coefficients for the most used words.

Before diving into the code make sure that you have installed the latest version of the R console or an IDE such as RStudio. As I have stated earlier, I highly recommend the use of RStudio as a development environment and all of the projects presented on this website were coded and refined on this platform. As is the case with any IDE there is a learning curve, but if you are more than a casual programmer the learning process is well worth the effort. If you are using the R console and want to make sure all packages are up to date you can use the following code
#################################################################################script to access and install the newest version of the R console
###############################################################################
#the console should be updated from the Windows R console not from RStudio
#use the following code from the console
################################################################################
install.packages(“installr”) #download installer
library(installr) #move installer to library
updateR() #run the update; follow prompts as needed
################################################################################

The updated wordcloud code as shown below has been extensively commented as to the purpose of the sections within the program structure. For the sake of brevity I will not repeat the explanations contained in the original posting of this document. You can refer to that document on this website if necessary. I should also point out that you can copy all or parts of the code as presented here and paste it into the R console or RStudio code section. It should run properly. I tried to present the code in sections delimited by comments that allow segments to run individually for debugging purposes. As noted in the comments, make sure you change the table function argument section to point to the location of your downloaded text file. The complete code is seen below.


#NOTE:THIS SECTION HAS BEEN UPDATED WITH TRUMP SOTU 2026 DATA
#this is an updated version of Wordcloud1 that will include
#analysis and graphic representation of word counts and association coefficients
#for a single SOTU, Trump 2026
####################################################
#Load required packages
#####################################################
install.packages(“tm”) #processes data
install.packages(“wordcloud”) #creates visual plot
install.packages(“tidyverse”) #graphics utilities
install.packages(“readr”) #to load text files
install.packages(“RColorBrewer”) #for color graphics
#####################################################
#####################################################
#Load and view raw 2026 SOTU text file ‘sotu26.txt’
#Use RStudio Import Dataset tab or code below
#NOTE: BE SURE TO CHANGE THE read_table ARGUMENT TO
#POINT TO THE LOCATION OF THE FILE YOU ARE USING
#####################################################
library(readr)
statu26 <- read_table(“e:/dellfiles/sotu26.txt”, col_names = FALSE) #make sure this points to your file location!
#View(statu26)
####################################################
#Take raw text file statu26 and convert to corpus format named docs26
#####################################################
library(tm)
docs26 <- Corpus(VectorSource(statu26))
####################################################
#Clean punctuation, stopwords, white space
#Three passes create corpus vector source from original file
#A corpus is a collection of text
#One of the functions removes ‘stopwords’, words that are pre-defined
####################################################
library(tm)
library(wordcloud)
data(docs26)
docs26 <- tm_map(docs26, function(x)removeWords(x,stopwords()))
docs26 <- tm_map(docs26,removePunctuation) #remove punctuation
################################################################
#remove stopwords using with ‘en’ or ‘SMART’ criteria
docs26 <- tm_map(docs26,removeWords,stopwords(“SMART”))
###############################################################
docs26 <- tm_map(docs26,stripWhitespace) #remove white space
####################################################
#Cleaned corpus is now formatted into text document matrix
#Then frequency count done for each word in matrix
#dmat <-create matrix; dval <-sort; dframe <-count word frequencies
###################################################
docmat <- TermDocumentMatrix(docs26)
dmat <- as.matrix(docmat)
dval <- sort(rowSums(dmat),decreasing=TRUE)
dframe <- data.frame(word=names(dval),freq=dval)
####################################################
#Final step is to use wordcloud to generate graphics
#There are a number of options that can be set
#Use RColorBrewer to generate a color wordcloud
####################################################
library(RColorBrewer)
set.seed(1234) #use if random.color=TRUE
par(bg=”white”) #background color
wordcloud(dframe$word,dframe$freq,colors=brewer.pal(8,”Set1″),random.order=FALSE,scale=c(2.75,0.35),min.freq=2,max.words=150,rot.per=0.35)
###################################################
####################################################
# Add a title above the plot
mtext(“Donald Trump SOTU 2026”, side = 3, line = 2, cex = 1.5)
##############################################################################
#this section contains code for printing word freq tables;
#word assoc analysis
#########################################################
#code to print 15 most used words with freq
####################################################
#for 2026 address
head(dframe,15)
######################################################
#code to find associations among specified terms
#drops assoc < .25
#initial setup formost frequent words; used between 10 and 25 times
#####################################################
#for 2026 address
findAssocs(docmat,terms = c(“and”,”but”,”budget”,”know”,”economy”,”health”,”plan”,”american”,”energy”,”people”,”care”),corlimit = .25)
######################################################


The output from the code will produce the wordcloud graphic display, a list of the most frequently used words in the document, and a matrix that displays association strengths equal to or greater than .25 for eleven of the most frequently used words in the document. A specific analysis of word frequencies and associations will be the topic of a future paper.

Shown below is the wordcloud graphic, the table of most frequently used words, and a portion of the association matrix generated by the code. I have only shown the portion of the matrix using the four most frequently used words.


Word Frequencies

word freq
and and 25
plan plan 16
but but 15
economy economy 14
now now 13
budget budget 12
health health 12
american american 12
education education 11
energy energy 11
people people 10
care care 10
country country 9
work work 9
america america 9



The association matrix:

$and
along already cmadam first now over second
0.85 0.85 0.85 0.85 0.85 0.85 0.85
speaking still thanks these third the this
0.85 0.85 0.85 0.85 0.85 0.79 0.73
for finally because there you its meet
0.71 0.64 0.50 0.44 0.44 0.41 0.29
bay career claims classroom commerce conscience earned
0.29 0.29 0.29 0.29 0.29 0.29 0.29
fallen ignore import industries ineffective layoffs marketbased
0.29 0.29 0.29 0.29 0.29 0.29 0.29
member message pad perseveres piles pride retooled
0.29 0.29 0.29 0.29 0.29 0.29 0.29
sacrifice succeed warera wealthy
0.29 0.29 0.29 0.29

$but
along already cmadam first over second
0.83 0.83 0.83 0.83 0.83 0.83
speaking still thanks these third now
0.83 0.83 0.83 0.83 0.83 0.81
the this for there finally because
0.81 0.65 0.64 0.64 0.53 0.43
you its agribusiness anger breaks cities
0.43 0.35 0.30 0.30 0.30 0.30
compete csomewhere lower messes pakistan percent
0.30 0.30 0.30 0.30 0.30 0.30
presidents pushed readying reality rebuilt set
0.30 0.30 0.30 0.30 0.30 0.30
transformation view
0.30 0.30

$plan
bank stand held full lose president
0.44 0.42 0.42 0.42 0.42 0.42
weakened builds challenge cmr intend preserve
0.42 0.41 0.41 0.41 0.41 0.41
progress proud respond tells creating cvice
0.41 0.41 0.41 0.41 0.41 0.41
deeds flow friends greensburg honest illusions
0.41 0.41 0.41 0.41 0.41 0.41
ive leonard outstanding prescription speak stopped
0.41 0.41 0.41 0.41 0.41 0.41
strain transform tysheoma boldly candidly doors
0.41 0.41 0.41 0.41 0.41 0.41
join men opinions recognizes showing stability
0.41 0.41 0.41 0.41 0.41 0.41
summit teachers team town save understand
0.41 0.41 0.41 0.41 0.35 0.34
asked recovery
0.30 0.28

$economy
assistance laughter abroad carry comforted costs
0.51 0.51 0.50 0.50 0.50 0.50
cunited cynical direct dramatically gains neighbors
0.50 0.50 0.50 0.50 0.50 0.50
oversight planet receive strategy taking teach
0.50 0.50 0.50 0.50 0.50 0.50
tornado troops auto industry energy act
0.50 0.50 0.37 0.37 0.32 0.30
held hold promise countries government ability
0.30 0.30 0.30 0.30 0.30 0.30
expanded build family social
0.30 0.30 0.30 0.30

In part two of this project I will expand the code to generate comparison clouds and associated statistics for two or more text files. Once again I will use State of the Union texts as the data for analysis. It is my hope that these R tutorials and examples have provided readers with some useful information and working models that can be adapted to their own research projects.

All R programming for this project was done using RStudio 2026.07.0+139 “Pacific Dogwood.”
This PDF document was produced using TeXstudio 4.9.5 (git 4.9.5)
Using Qt Version 6.11.0, compiled with Qt 6.11.0 R.
R, RStudio, and TeXstudio are free, open-source software.
Author: Douglas M. Wiig 7/31/2026
https://dmwiig.net

← Back

Thank you for your response. ✨

Visual Representation of Complex Data Using R: The tm and wordcloud Packages: Part One Revisited and Updated


An R Tutorial: Visual Representation of Complex Multivariate Relationships Using the R ‘qgraph’ Package, Part Two


An R programming tutorial by D.M. Wiig

This post is contained in a .pdf document.  To access the document click on the green link shown below.

qgraphpost3

R Tutorial: Visualizing Multivariate Relationships in Large Datasets


R Tutorial: Visualizing multivariate relationships in Large Datasets

A tutorial by D.M. Wiig

In two previous blog posts I discussed some techniques for visualizing relationships involving two or three variables and a large number of cases. In this tutorial I will extend that discussion to show some techniques that can be used on large datasets and complex multivariate relationships involving three or more variables.

In this tutorial I will use the R package nmle which contains the dataset MathAchieve. Use the code below to install the package and load it into the R environment:

####################################################
#code for visual large dataset MathAchieve
#first show 3d scatterplot; then show tableplot variations
####################################################
install.packages(“nmle”) #install nmle package
library(nlme) #load the package into the R environment
####################################################

Once the package is installed take a look at the structure of the data set by using:

####################################################
attach(MathAchieve) #take a look at the structure of the dataset
str(MathAchieve)
####################################################

Classes ‘nfnGroupedData’, ‘nfGroupedData’, ‘groupedData’ and ‘data.frame’: 7185 obs. of 6 variables:
$ School : Ord.factor w/ 160 levels “8367”<“8854″<..: 59 59 59 59 59 59 59 59 59 59 …
$ Minority: Factor w/ 2 levels “No”,”Yes”: 1 1 1 1 1 1 1 1 1 1 …
$ Sex : Factor w/ 2 levels “Male”,”Female”: 2 2 1 1 1 1 2 1 2 1 …
$ SES : num -1.528 -0.588 -0.528 -0.668 -0.158 …
$ MathAch : num 5.88 19.71 20.35 8.78 17.9 …
$ MEANSES : num -0.428 -0.428 -0.428 -0.428 -0.428 -0.428 -0.428 -0.428 -0.428 -0.428 …
– attr(*, “formula”)=Class ‘formula’ language MathAch ~ SES | School
.. ..- attr(*, “.Environment”)=<environment: R_GlobalEnv>
– attr(*, “labels”)=List of 2
..$ y: chr “Mathematics Achievement score”
..$ x: chr “Socio-economic score”
– attr(*, “FUN”)=function (x)
..- attr(*, “source”)= chr “function (x) max(x, na.rm = TRUE)”
– attr(*, “order.groups”)= logi TRUE
>

As can be seen from the output shown above the MathAchieve dataset consists of 7185 observations and six variables. Three of these variables are numeric and three are factors. This presents some difficulties when visualizing the data. With over 7000 cases a two-dimensional scatterplot showing bivariate correlations among the three numeric variables is of limited utility.

We can use a 3D scatterplot and a linear regression model to more clearly visualize and examine relationships among the three numeric variables. The variable SES is a vector measuring socio-economic status, MathAch is a numeric vector measuring mathematics achievment scores, and MEANSES is a vector measuring the mean SES for the school attended by each student in the sample.

We can look at the correlation matrix of these 3 variables to get a sense of the relationships among the variables:

> ####################################################
> #do a correlation matrix with the 3 numeric vars;
> ###################################################
> data(“MathAchieve”)
> cor(as.matrix(MathAchieve[c(4,5,6)]), method=”pearson”)  

SES MathAch MEANSES
SES 1.0000000 0.3607556 0.5306221
MathAch 0.3607556 1.0000000 0.3437221
MEANSES 0.5306221 0.3437221 1.0000000

In using the cor() function as seen above we can determine the variables used by specifying the column that each numeric variable is in as shown in the output from the str() function.  The 3 numeric variables, for example, are in columns 4, 5, and 6 of the matrix.

As discussed in previous tutorials we can visualize the relationship among these three variable by using a 3D scatterplot. Use the code as seen below:

####################################################
#install.packages(“nlme”)
install.packages(“scatterplot3d”)
library(scatterplot3d)
library(nlme) #load nmle package
attach(MathAchieve) #MathAchive dataset is in environment
scatterplot3d(SES, MEANSES, MathAch, main=”Basic 3D Scatterplot”) #do the plot with default options
####################################################

The resulting plot is:

mathach3dscatter

Even though the scatter plot lacks detail due to the large sample size it is still possible to see the moderate correlations shown in the correlation matrix by noting the shape and direction of the data points  .  A regression plane can be calculated and added to the plot using the following code:

scatterplot3d(SES, MEANSES, MathAch, main=”Basic 3D Scatterplot”) #do the plot with default options
####################################################
##use a linear regression model to plot a regression plane
#y=MathAchieve, SES, MEANSES are predictor variables
####################################################
model1=lm(MathAch ~ SES + MEANSES)    ## generate a regression
#take a look at the regression output
summary(model1)
#run scatterplot again putting results in model
model <- scatterplot3d(SES, MEANSES, MathAch, main=”Basic 3D Scatterplot”)     #do the plot with default options
#link the scatterplot and linear model using the plane3d function
model$plane3d(model1)        ## link the 3d scatterplot in ‘model’ to the ‘plane3d’ option with ‘model1’ regression information
####################################################

The resulting output is seen below:

Call:
lm(formula = MathAch ~ SES + MEANSES)

Residuals:
Min 1Q Median 3Q Max
-20.4242 -4.6365 0.1403 4.8534 17.0496

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 12.72590 0.07429 171.31 <2e-16 ***
SES 2.19115 0.11244 19.49 <2e-16 ***
MEANSES 3.52571 0.21190 16.64 <2e-16 ***

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 6.296 on 7182 degrees of freedom
Multiple R-squared: 0.1624, Adjusted R-squared: 0.1622
F-statistic: 696.4 on 2 and 7182 DF, p-value: < 2.2e-16

and the plot with the plane is:

mathachplot2

While the above analysis gives us useful information, it is limited by the mixture of numeric values and factors.  A more detailed visual analysis that will allow the display and comparison of all six of the variables is possible by using the functions available in the R package Tableplots.   This package was created to aid in the visualization and inspection of large datasets with multiple variables.

The MathAchieve contains a total of six variables and 7185 cases.  The Tableplots package can be used with datasets larger than 10,000 observations and up to 12 or so variables. It can be used visualize relationships among variables using the same measurement scale or mixed measurement types.

To look at a comparisons of each data type and then view all 6 together begin with the following:

####################################################
attach(MathAchieve) #attach the dataset
#set up 3 data frames with numeric, factors, and mixed
####################################################
mathmix <- data.frame(SES,MathAch,MEANSES,School=factor(School),Minority=factor(Minority),Sex=factor(Sex)) #all 6 vars
mathfact <- data.frame(School=factor(School),Minority=factor(Minority),Sex=factor(Sex)) #3 factor vars
mathnum <- data.frame(SES,MathAch,MEANSES) #3 numeric vars
####################################################

To view a comparison of the 3 numeric variables use:

####################################################
require(tabplot) #load tabplot package
tableplot(mathnum) #generate a table plot with numeric vars only
####################################################

resulting in the following output:

mathnumplot

To view only the 3 factor variables use:

####################################################
require(tabplot)   #load tabplot package
tableplot(mathfact)    #generate a table plot with factors only
####################################################

Resulting in:

mathfactplot

To view and compare table plots of all six variables use:

####################################################
require(tabplot)    #load tabplot package
tableplot(mathmix)    #generate a table plot with all six variables
####################################################

Resulting in:

mathmixplot

Using tableplots is useful in visualizing relationships among a set of variabes. The fact that comparisons can be made using mixed levels of measurement and very large sample sizes provides a tool that the researcher can use for initial exploratory data analysis.

The above visual table comparisons agree with the moderate correlation among the three numeric variables found in the correlation and regression models discussed above.  It is also possible to add some additional interpretation by viewing and comparing the mix of both factor and numeric variables.

In this tutorial I have provided a very basic introduction to the use of table plots in visualizing data. Interested readers can find an abundance of information about Tableplot options and interpretations in the CRAN documentation.

In my next tutorial I will continue a discussion of methods to visualize large and complex datasets by looking at some techniques that allow exploration of very large datasets and up to 12 variables or more.

R For Beginners: Some Simple R Code to do Common Statistical Procedures, Part Two


An R tutorial by D. M. Wiig

This posting contains an embedded Word document. To view the document full screen click on the icon in the lower right hand corner of the embedded document.

 

 

R For Beginners: Basic R Code for Common Statistical Procedures Part I


An R tutorial by D. M. Wiig

This section gives examples of code to perform some of the most common elementary statistical procedures. All code segments assume that the package ‘car’ has been loaded and the file ‘Freedman’ has been loaded as the active dataset. Use the menu from the R console to load the ’car’ dataset or use the following command line to access the CRAN site list and packages:


install.packages()

Once the ’car’ package has been downloaded and installed use the following command to make it the active library.

require(car)

Load the ‘Freedman’ data file from the dataset ‘car’

data(Freedman, package="car")

List basic descriptives of the variables:

summary(Freedman)

Perform a correlation between two variables using Pearson, Kendall or Spearman’s correlation:

cor(filename[,c("var1","var2")], use="complete.obs", method="pearson")

cor(filename[,c("var1","var2")], use="complete.obs", method="spearman")

cor(filename[,c("var1","var2")], use="complete.obs", method="kendall")

Example:

cor(Freedman[,c("crime","density")], use="complete.obs", method="pearson")

cor(Freedman[,c("crime","density")], use="complete.obs", method="kendall")

cor(Freedman[,c("crime","density")], use="complete.obs", method="spearman")

In the next post I will discuss basic code to produce multiple correlations and linear regression analysis.  See other tutorials on this blog for more R code examples for basic statistical analysis.

 

R Video Tutorial: Basic R Code to Load a Data File and Produce a Histogram


R For Beginners:  Some Simple R Code to Load a Data File and Produce a Histogram

A tutorial by D. M. Wiig

I have found that a good method for learning how to write R code is to examine complete code segments written to perform specific tasks and to modify these procedures to fit your specific needs. Trying to master R code in the abstract by reading a book or manual can be informative but is more often confusing.  Observing what various code segments do by observing the results allows you to learn with hands-on additions and modifications as needed for your purposes.

In this document I have included a short video tutorial that discusses  loading a dataset from the R library, examining the contents of the dataset and selecting one of the variables to examine using a basic histogram.  I have included an annotated code chunk of the procedures discussed in the video.

The video appears below with the code segment following.

Here is the annotated code used in the video:

###################################
#use the dataset mtcars from the ‘datasets’ package
#select the variable mpg to do a histogram
#show a frequency distribution of the scores
##########################################
#library is ‘datasets’
#########################################
library(“datasets”)
#########################################
#take a look at what is in ‘datasets’
#########################################
library(help=”datasets”)
#######################################
#take a look at the ‘mtcars’ data
#########################################
View(mtcars)
#######################################
#now do a basic histogram with the hist function
###########################################
hist(mtcars$mpg)
#############################################
#dress up the graph; not covered in the video but easy to do
############################################
hist(mtcars$mpg, col=”red”, xlab = “Miles per Gallon”, main = “Basic Histogram Using ‘mtcars’ Data”)
###################################################

 

R For Beginners: A Video Tutorial on Installing and Using the Deducer Statistics Package


R For Beginners:  A Video Tutorial on Installing and Using the Deducer Statistics Package with the R Console

In previous tutorials I have discussed the use of R Commander and Deducer statistical packages that provide a menu based GUI for R.  In this video tutorial I will discuss downloading and installing the Deducer statistics package.  This video is designed to support my previous tutorial on the same subject.

I have embedded the video below,   I hope you find this tutorial  a useful adjunct to installing and using the menu based Deducer package.

This document is an embedded Word document.  To view it full screen click on the icon in the lower right corner of the screen

 

R For Beginners: Installing the JGR GUI On a Linux Platform


A Tutorial by D. M. Wiig

This is an embedded Word document.  To view it full screen click on the icon in the lower right cornet of the document.

Watch for more tutorials discussing  R statistics on a Linux platform.

R Video Tutorial For Beginners: Installing And Using the Rcommander GUI


R Video Tutorial For Beginners: Installing And Using the Rcommander GUI

A tutorial video by D. M. Wiig

In my recent series of tutorials for those interested in the R statistical programming language I have discussed both the installation and use of the R console and R Commander statistics GUI.  Before viewing the tutorial make sure the R Commander package has been download into your R library via the Install Packages menu option.  This procedure was discussed in the previously posted R Commander tutorial.

Relative to this first tutorial I have have created a video that covers the initial installation of R Commander.  The video is seen below:

Click the icon in the lower right side of the screen to view the tutorial in full screen mode.

I hope that you find this useful in your pursuit of learning about  R statistics.