Stats 102A - Homework 3 Instructions
Homework questions and instructions copyright Miles Chen, Do not post, share, or distribute without permission.
Homework 3 Requirements
You will submit two files.
The files you submit will be:
1. 102a_hw_03_output_First_Last.Rmd Take the provided R Markdown file and make the necessary edits so
that it generates the requested output. The first line of your .Rmd file should be to source the R script file you
wrote.
2. 102a_hw_03_output_First_Last.pdf Your output PDF file. This is the primary file that will be graded. Make
sure all requested output is visible in the output file.
There is no script file to submit
Failure to submit all files will result in an automatic 40 point penalty.
Academic Integrity
At the top of your R markdown file, be sure to include the following statement after modifying it with your name.
“By including this statement, I, Joe Bruin, declare that all of the work in this assignment is my own original work.
At no time did I look at the code of other students nor did I search for code solutions online. I understand that
plagiarism on any single part of this assignment will result in a 0 for the entire assignment and that I will be referred
to the dean of students.”
If you collaborated verbally with other students, please also include the following line to credit them.
“I did discuss ideas related to the homework with Josephine Bruin for parts 2 and 3, with John Wooden for part 2,
and with Gene Block for part 5. At no point did I show another student my code, nor did I look at another student’s
code.”
Reading:
Read the following:
a. tidy data: https://r4ds.had.co.nz/tidy-data.html
b. data transformation: https://r4ds.had.co.nz/transform.html
c. How dplyr replaced my most common R idioms: http://www.onthelambda.com/2014/02/10/how-dplyr-replaced-my-most-common-r-idioms/
d. regular expressions tutorial http://regexone.com/
Part 1 - dplyr exercises
Using dplyr, answer the following questions about the vehicles dataset. You are allowed to use non-dplyr functions,
but I do believe dplyr makes the questions much easier.
a. How many unique vehicle makers (variable make) are included in the dataset?
b. How many vehicles made in 2014 are represented in the dataset?
c. For the year 2014, what was the average city mpg (gas mileage) for all compact cars? What was the average
city mpg for midsize cars in 2014?
d. For the year 2014, compare makers of midsize cars. Find the average city mpg of midsize cars for each
manufacturer. For example, in 2014, Acura has 5 midsize cars with an average city mpg of 20.6, while Audi
has 12 midsize cars with an average city mpg of 19.08. Produce a table showing the city mpg for 2014 midsize
cars for the 27 manufacturers represented in the table. Arrange the results in descending order, so that the
manufacturer with the highest average mpg will be listed first.
1
e. Finally, for the years 1994, 1999, 2004, 2009, and 2014, find the average city mpg of midsize cars for each
manufacturer for each year. Use tidyr to transform the resulting output so each manufacturer has one row, and
five columns (a column for each year). Print out all the rows of the resulting tibble. You can use print(tibble,
n = 40) to print 40 rows of a tibble.
Make sure your output is visible for each question
Part 2 - more dplyr
I have uploaded a dataset called dr4.Rdata. It contains the dates that fictional users visited a fictional website. The
website is able to track if the same user visited the site more than once. For the particular date range, the site had
395 visitors, and 130 of them visited more than once. Some of them (13 people) visited the site 5 times.
Using dplyr, find the average time between repeated visits to the site.
You will want to find the total average.
Be careful when calculating this.
For example, the first user to visit the site more than once (row 2, ,YPELGRZNOQUTNPOH) visited on 6-29, 7-27,
8-3, and 8-11. The time difference for the repeated visits are: 28 days, 7 days, and 8 days, respectively, for an average
of 14.33 days.
The next user with repeated visits is row 3 (SNTCUXUDIHCCSPJA). This person visited on 6-15 and 8-17, a
difference of 63 days.
If your dataset had only these two rows, the average time between visits would be (28 + 7 + 8 + 63) / 4 = 26.5 days.
It is not ( 14.33 + 63 ) / 2 = 38.66 days.
When I first attempted this, I used filter(), mutate(), rowwise(), ungroup(), and summarise(). Upon further
review, I realized that it is entirely possible to complete this task using only filter() and mutate() commands. I
do not care what combination of commands you use. I do care that you get the correct final result.
Make sure your final output shows the desired average number of days between visits.
Part 3 - rvest
You will use the package rvest to scrape data from the website baseball-reference.com.
Begin at the teams page http://www.baseball-reference.com/teams/.
For each active team (there are 30), visit each team’s page and download the “Franchise History” table. The node
you will want to use is #franchise_years. Combine all the tables in one. Note that some franchises have names and
locations. To keep track of the team, add a column to the dataframe called “current_name” which will contain the
current name of the team. (For example, in the ‘current_name’ column, the row for 1965 Milwaukee Braves will
contain the value ‘Atlanta Braves’)
Hint: After identifying the node, I used the function html_table() to extract the table from each team’s page.
Important: It is bad manners to repeatedly hit a site with http requests, and could cause your IP to become banned.
While you are testing out your code, be sure to test with only two or three teams at a time. Once you get your code
running, then you may expand your code to download data for all 30 teams.
After scraping the data, print out the dimensions of the resulting data.
Hint: When I ran my code, my table had 2684 rows and 22 columns.
Also print out the first few rows of the resulting data.
Part 4 - dplyr to summarize the baseball data
Unfortunately the baseball-reference site makes use the of the non-breaking space character and uses it in places like
the space in “Atlanta Braves.”
2
I’ve written some commands for you in the Rmd file that will replace all instances of the non-breaking space and
replace it with a standard space character in the baseball table. I’ve done this part for you. You just need to run the
code in the section Some light text clean up
Once you have created your table, use the data it contains to calculate some summary statistics.
For each franchise, filter the dataset to only include data from the years 2001 to 2019 (inclusive). If the franchise
changed team names during this period, include the previous team’s data as well. (e.g. the data for the Washington
Nationals will also include data for the 2001-2004 Montreal Expos)
Then calculate the following summary statistics for each team across the 19 seasons:
• total wins (TW)
• total losses (TL)
• total runs scored (TR)
• total runs allowed (TRA)
• total win percentage (wins / (wins + losses))
Sort the resulting table (should have a total of 30 rows) in descending order by total win percentage. Be sure to print
all rows and columns of the resulting summary table.
All requested columns must appear in the html to receive full credit.
Hint: At the top of my table, I had the NY Yankees, with a total win percentage of 1799 Total Wins, 1276 Total
Losses, and a Total Win Percentage of 0.5850407.
Part 5 - Regular expressions for the Manager column
Using regular expressions, extract the wins and losses for the managers listed in the managers column. Do not use
each season’s number of wins or losses. You must extract the information from the managers column using regular
expressions. That column has the information written in the form “F.LastName (82-80)”. You will need to use capture
groups in your regular expression to separate the different pieces of information.
Be careful as some of the rows contain information for more than one manager. Combine all of the manager information
to get a total wins and loss value for each of the managers. Many managers have managed more than one team. Be
sure to combine all of the win-loss information for the same manager. You may assume that entries that share the
same first initial and last name are the same person.
Create a summary table with one line for each manager. The table should contain the following columns, and should
be sorted descending by total number of games.
• Manager’s name (First initial and Last Name)
• Total number of games managed
• Total number of wins across career
• Total number of losses across career
• Total win percentage
You can independently verify if your information is correct on baseball-reference.com. Each manager has his own
page with a total count of wins and losses.
Figuring out the regular expression here is probably the trickiest part. There is also an instance where there are two
different people with the same first initial and the same last name. Unfortunately, their information will end up being
combined. For this homework assignment, that’s okay.
Regarding the regular expression, you will need to use capture groups, and thus str_match_all(). We use the _all
variant because some of the entries will have multiple managers.
All requested columns must appear in the html to receive full credit.
The first line of my table reads: C.Mack, 7679, 3731, 3948, 0.4858706, for manager, games, wins, losses, win percentage.
Also Watch out for T.La Russa who has a space in his name. He managed the second most number of games with a
final record of 2728-2365. If you report his name as T.La, you will not get full credit.
3
Part 6 - Extra credit
This is completely optional. Up to 10 points.
IMDB webscraping and summarization. You will need to add this section to the Rmd file yourself.
This is the IMDB page for actor Keanu Reeves. https://www.imdb.com/name/nm0000206/?ref_=nv_sr_srsg_0
The task is to follow the links to all of the projects he had a role in from 2019 and earlier (Hangin In through Between
Two Ferns: The Movie).
From each movie page, follow the link to See full cast. From that page, scrape all members of the cast (credited and
uncredited). Do not scrape director, writer, or crew information.
After gathering this data, create a summary table of the actors that Keanu Reeves has worked with.
+7 points if you can identify the actors that have appeared in 2+ projects with Keanu Reeves. The table should
include the total number of projects they appeared in together arranged in descending order. Actors will be arranged
alphabetically for ties.
+3 additional points if your table can list the names of the projects. How you format and present this is up to you.
For example, the entry for Laurence Fishburne might look like this: (by the way, this example might not be accurate)
## Laurence Fishburne 6
## ....
Would yield 7 points
## Laurence Fishburne 6 "The Matrix"
## "The Matrix Reloaded"
## "The Matrix Revolutions"
## "Enter the Matrix"
## "John Wick: Chapter 2"
## "John Wick: Chapter 3 - Parabellum"
## ....
Would yield 10 points
4