BUSINESS DATA MINING (IDS 572)
HOMEWORK 1
DUE DATE: WEDNESDAY, SEPTEMBER 16 AT 3:30 PM
• Please provide succinct answers to the questions below.
• You should submit an electronic pdf or word file in blackboard.
• Please include the names of all team-members in your write up and in the name of the file.
• One submission is sufficient for the entire group.
• You should include all the R functions you use in your pdf file.
• Please make sure your graphs have titles, labels, legends (if necessary), appropriate colors and
etc.
Problem 1. This question should be done without using R.
We have just put out a special promotion and would like to determine who responded to the mailing. We
have a sample of consumers, including both purchasers and non-purchasers (all received the promotion),
and would like to predict who is a purchaser. For each consumer, we have their age (bucketed into
ranges), their income which is a numerical variable, and whether or not they responded to last year’s
mailing.
Purchase? Age Income Last Year?
Yes Young 60K No
Yes Middle 60K Yes
Yes Old 100K Yes
Yes Young 60K Yes
Yes Young 100K Yes
Yes Middle 60K No
No Old 150K Yes
No Middle 100K No
No Young 150K Yes
No Middle 100K Yes
No Old 150K No
Please show your calculations in the following questions,
(a) Using the 1-rule method discussed in class, find the relevant sets of classification rules for the
target variable (Purchase?) by testing each of the input attributes. Which of these sets of rules
has the lowest misclassification rate?
(b) Now, we want to construct a decision tree using this data set. What is the entropy measure of
the entire data set?
(c) What is the information gain for splitting on age? On income? On last year? Which should be
the initial split variable?
1
2 HOMEWORK 1 DUE DATE: WEDNESDAY, SEPTEMBER 16 AT 3:30 PM
(d) Construct the entire tree using the information gain. What is the decision at each terminal
node?
(e) What is the accuracy of your decision tree on this data set? Explain your answer.
(f) What would your tree predict for a Middle aged person with 90K income who did not purchase
last year? Justify your answer. Consider the following instances as test data points.
Purchase? Age Income Last Year?
Yes Middle 60K No
No Young 90K Yes
Yes Old 100K No
Yes Young 60K Yes
Yes Middle 140K No
What is the accuracy of your decision tree model on the test data? Justify your answer.
(g) What are the support and confidence of the rules
– If Last Year = Yes, then Yes.
– If Age = Middle, and Income = 50K, then Yes.
(h) Based on your decision tree, which variable is considered the most important variable? Justify
your answer.
The goal of the remaining questions is to review statistical R programming and get comfortable with
coding in R, especially for those ones of you who did not learn R in IDS 570. The questions are asking
for simple tasks but try your best to code them in the most elegant way that you can. For example,
you can use the function “tibbles” instead of data-frame for a nicer looking data frames. Take care of
details and try to play with the functions arguments to check how they work. Again, since some of you
are new with R, please feel free to contact me or the TA if you have any specific questions.
Problem 2. In this question, we use the built-in R dataset called attitude which contains information
from a survey of the clerical employees of a large financial organization. To access this date set use
“data(“attitude”)”. Learn more about each variable by reading the variable description in ?attitude.
(a) Summarize the main statistics of all the variables in the data set.
(b) How many observations are in the attitude dataset? What function in R did you use to display
this information?
(c) Produce a scatterplot matrix of the variables in the attitude dataset. What seems to be most
correlated with the overall rating?
(d) Produce a scatterplot of rating (on the y-axis) vs. learning (on the x-axis). Add a title to the
plot.
BUSINESS DATA MINING (IDS 572) 3
(e) Produce 2 side-by-side histograms, one for rating and one for learning. You will need to use
par(mfrow=...) to get the two plots together.
Problem 3. Use hw1.xls to answer the following questions. Include all the charts with proper labels
in your report. Please, for each of the chart you produced, write a sentence or two explaining what you
see from the chart.
(a) Make a frequency distribution table for the gender variable to see the frequency distribution.
(b) Make a bar chart for gender variable.
(c) Make a histogram to display the distribution of the Height variable.
(d) Make a cluster bar chart (side-by-side bar chart) to examine the correlation between gender and
Ate Fried Food variables.
(e) Make a scatter plot to examine the correlation between Weight and Height variables, and write
a sentence to describe the trend you observed from the scatter plot.
(f) Find the 5-number summary for the Height data and make a boxplot for the Height data with
mild and extreme outliers identified using inner and outer fences. Draw the boxplot.
Problem 4. To do this question you need the following packages in R: MASS, plyr, dplyr, tibble, and
ggplot2. We are going to use the builtin data set “birthwt” (Risk Factors Associated with Low Infant
Birth Weight) from the MASS library. This dataset contains 189 instances and 10 variables. To learn
more about this data set you can use ?birthwt.
(a) All the variables are represented as integer. Write your own function that automatically converts
all the integer variables to factors (categorical).
(b) Repeat part (a) using mutate() and mapvalues() functions.
(c) Use the tapply() function to see what the average birthweight looks like when broken down
by race and smoking status. Does smoking status appear to have an effect on birth weight?
Does the effect of smoking status appear to be consistent across racial groups? What is the
association between race and birth weight?
(d) Use kable() function from knitr to dispaly the table you get in part (c).
(e) Use ddply() function to get the average birthweight by mother’s race and compare it with
tapply() function.
(f) Use ggplot2() to plot the average birthweight (computed in part (e)) for each race group in a
bar plot.
(g) Use ddply() function to look at the average birthweight and proportion of babies with low
birthweight broken down by smoking status.
(h) Split the data further by adding “mother smokes” to the ddply() formula used in part (g).
4 HOMEWORK 1 DUE DATE: WEDNESDAY, SEPTEMBER 16 AT 3:30 PM
(i) Is the mother’s age correlated with birth weight? Does the correlation vary with smoking status?
Problem 5. “ggplot() produces far better and more easily customizable graphics than anythor visual-
ization functions in R. There are two basic calls in ggplot:
• qplot(x, y, . . . , data): a “quick-plot” routine, which essentially replaces the base plot().
• ggplot(data, aes(x, y, ...), ...): defines a graphics object from which plots can be generated, along
with aesthetic mappings that specify how variables are mapped to visual properties.
In this question, we would like to quickly practice drawing different plots using ggplot2(). For this
purpose, we use the “diamonds” dataset in R. You can access this dataset by writing “data(diamonds)”.
(a) What type of variable is price? Would you expect its distribution to be symmetric, right-skewed,
or left-skewed? Why? Make a histogram of the distribution of diamond prices. Does the shape
of the distribution match your expectation? (Use geom histogram()).
(b) Visualize a few other numerical variables in the dataset and discuss any interesting features.
When describing distributions of numerical variables we might also want to view statistics like
mean, median, etc.
(c) What type of variable is color? Which color is most prominently represented in the dataset?
(d) Make a bar plot of the distribution of cut, and describe its distribution (Use georm bar())
(e) Make a histogram of the depths of diamonds, with binwidth of 0.2%, and add another variable
(say, cut) to the visualization. You can do this either using an aesthetic or a facet. Typical
diamonds of which cut have the highest depth? On average, does depth increase or decrease as
cut grade increase or decrease?
(f) Compare the distribution of price for the different cuts. Does anything seem unusual? Describe.
(g) Draw a scatterplot showing the price (y-axis) as a function of the carat (size).
(h) Shrink the points in your scatter plot in part (g) using the alpha argument in geom point.
(i) Use facet wrap(∼ factor1 + factor2 + ... + factorn) command to create scatter plots showing
how diamond price varies with carat size for different values of “cut” (use colour = color in
aes()).