IDS 561 Homework 1 Due: 02/23/2021 Tuesday 3:30pm (before class) In this homework you will mimic the process of MapReduce task. Specifically, you will write your own map and reduce functions (without distributing to several machines) to mimic the process of mapper and reducer. The task is to count the number of occurrences of each word in a text file. This program, known as Word Count, is the equivalent of the standard “Hello, world!” program you typically write when you learn a new programming language. While doing this homework you will learn: 1) how to prepare the data. 2) how to write map and reduce functions. 3) get a better understanding of how mapper and reducer work. Dataset The input of this homework is a text document (around 13,000 lines) which includes several paragraphs. It is a raw data. You need to do some data cleaning works to prepare it for next step. Task You are supposed to build several functions to mimic each step of MapReduce. They are: Function Description Input of this function Output of this function Data cleaning function Some data cleaning jobs, such as removing numbers, punctuations and special symbols, uppercase to lower case. Raw text data Clean text data Data split function Split the dataset into two parts: Part1 includes the first 5000 lines of the text file, Part2 includes the rest text. Output of data cleaning function Two separated subsets: Part1 and Part2. Mapper function Two mapper functions that produce a set of key-value pairs for Part1 and Part2 subsets respectively. Output of data split function Key-value pairs of Part1 and Part2. Sort function Sort by key of Part1 and Part2 together, with an ascending sort order Output of mapper function Sorted Key-value pairs for the whole dataset Partition function All the tokens (i.e., words) starting with letter “a” to “m” are sent to Reducer1, and the others (“n” to “z”) are sent to Reducer2. Output of sort function Two ascending ordered partitions. Reducer function Collect all values belonging to the key and count the frequency of words for the two ordered partitions. Output of partition function Word frequency of the ordered partitions. Main function Wrap all the steps together and combine the output of the two partitions together. Output of reducer function Final result of word counting. The figure below shows the basic workflow of this word count task. Note: ① Using multi-thread is highly encouraged. Here are some tutorials of Python multithreading: https://www.toptal.com/python/beginners-guide-to-concurrency-and-parallelism-in- python https://realpython.com/intro-to-python-threading/ https://www.geeksforgeeks.org/multithreading-python-set-1/ ② Same code for the two mapper functions or the two reducer functions is ok. ③ You can use any data types provided in Python, e.g., list, dictionary and so on. What to submit (one submission per group) You need to submit two files: a Python file and a CSV file. Python file: Your python code. Please add comments to make it readable. CSV file: Your final word count output. The format should look like this: Word Frequency apple 123 banana 45 … …
欢迎咨询51作业君
