10/6/2020 Assignment 2.1 (Due 19:00 CDT on Oct 11, 2020) - CS 242 - Illinois Wiki https://wiki.illinois.edu/wiki/pages/viewpage.action?pageId=616243850 1/5 Created by Wang, Ren-Jay, last modified by Song, Hang on Oct 05, 2020 Assignment 2.1 (Due 19:00 CDT on Oct 11, 2020) Assignment 2.1 - Extending your web scraper into a web application Overview This week, you will be expanding on the data you scraped from last week to include several important new features. Being the superstar senior software engineer that you are, you have decided that although your work last week was impeccable, there are still some features you can add to make it more presentable. Specifically, the new requirements you would like to add are: 1. Analysis - you want to be able to answer some meaningful questions about your data 2. API Creation - you want the public to have access to your data 3. Visualization - you want your data to be understandable via some graphs and charts Part I: API Creation You want to make your amazing data publicly accessible. To do this, you will write an API. What is an API? In computer programming, an application programming interface (API) is a set of subroutine definitions, protocols, and tools for building application software. In general terms, it's a set of clearly defined methods of communication between various software components...Just as a graphical user interface makes it easier for people to use programs, application programming interfaces make it easier for developers to use certain technologies in building applications. By abstracting the underlying implementation and only exposing objects or actions the developer needs, an API reduces the cognitive load on a programmer. - Wikipedia, Application programming interface APIs are used in many real-world applications and are common in software development, web development, personal projects, hackathon, and many other places. In this assignment, we are asking you to write a web API. In addition to making your data accessible, this will help give you a better understanding of how web structuring works. Your API should include the following endpoints: GET, PUT, POST and DELETE (CRUD). Best practices and resources to design APIs are listed in this blog post. Please read through this to get a sense of how to set up good API parameters and mechanisms. We likely will not utilize all but this is a solid resource. Your response should be formatted in JSON. You can use CURL or postman to demonstrate that your API is working Get: api/books?attr={attr_value} Example: /books?title=”The Pragmatic Programmer” Find books with the title that has the phrase, it does not need to be unique. It should support searching for: (1) titles, (2) authors, and (3) related books api/authors?attr={attr_value} Example: /authors?name=”Shawshank&Redemption” Find authors with the name that has the phrase, it does not need to be unique. It should support searching for: (1) name, (2) book title You should also have APIs that supports arithmetic operations: And & OR For instance, having an API that can search for titles that include both words: "change" and "design" For instance, having an API that can search for authors that fulfills one or the other: "John Green" or "design" Put: api/books?attr={attr_value} Put, or update the first entry of the item returned from the given attribute. For instance, PUT {"ISBN": 1234567890} to /books?title=”The Pragmatic Programmer” should update the ISBN value to that entry in the database. api/authors?attr={attr_value} Same as authors ents 10/6/2020 Assignment 2.1 (Due 19:00 CDT on Oct 11, 2020) - CS 242 - Illinois Wiki https://wiki.illinois.edu/wiki/pages/viewpage.action?pageId=616243850 2/5 An API that accepts JSON encoded POST, PUT & PATCH requests should also require the Content- Type header to be set to application/json or throw a 415 Unsupported Media Type HTTP status code. Post: api/book Leverage POST requests to ADD A book to the backend (database) api/books Leverage POST requests to ADD SEVERAL books to the backend (database) api/author Leverage POST requests to ADD AN author to the backend (database) api/authors Leverage POST requests to ADD SEVERAL authors to the backend (database) Delete: api/book?attr={attr_value} Leverage DELETE requests to REMOVE the first book of the filtered results from the database api/author?attr={attr_value} Leverage DELETE requests to REMOVE the first author of the filtered results content from the database Your API should return an HTTP Code for each request. At a minimum, you should include 200 OK, 201 Created, 415 Unsupported Media Type and 400 Bad Request. More code are in the blog post listed above. Your API should read and return JSON formatted results. A good tool to use is Postman. Your output to all endpoints should be in JSON format. You are not required to put your script on a live server. You can test your API on localhost using POSTMAN or the command line. Putting the software on a live server gives you 1 additional credit to the overall assignment. Some of these servers include but not limited to firebase, heroku, AWS, Azure, GCP... To get started on APIs, we recommend you using a lightweight library FLASK. You are free to use other tools to complete building APIs depending on your database design in the first part of this assignment, for example, using Django. Part II: Rendering content and form submission Now that you have the API, you might want others to make use of these APIs. One easy way is to place these APIs into HTML templates and return them as rendered HTML files. There are a lot of front end libraries that one can utilize, for instance React.js or Angular.js. However, to make things easier and straightforward, since you are writing in flask, you can use the Jinja templating engine. Of course, this is not mandatory as long as your data renders and fulfill the following requirements, you are fine. Rendering front-end API results. Instead of showing the results in JSON format, you should design a front-page that renders the results. You might want to make uses of form submission boxes or tables. The interface does not have to look pretty (though there are additional points for that) You are required to support the following operations: CRUD for /book/
: the book of the book_id CRUD for /author/ : the author of the author_id GET /books: a list of all the books in your database GET /authors: a list of authors in your database Part III: Data Visualization Finally, we want you to complete a set of queries and visualization. We would like you to answer 2 queries and render 2 visualizations. You are recommended to use D3.js to complete the visualization. A good tutorial is here and you should be able to find a lot more on the web. Query 1: Find the author that has the most books stored in your database. This query should be accessed by /query/most-book-authors. Query 2: Find the book that has the most similar books stored in your database. This query should be accessed by /query/most-similar-books. Visualization 1: Visualize the ranking of the authors based on their ratings, you can just display a subset of the ranking results, for instance, the top 15. This query should be accessed by /vis/rank-authors. Visualization 2: Visualize the ranking of the books based on the number of reviews they have, you can just display a subset of the ranking results, for instance, the top 15. This query should be accessed by /vis/rank-books. Of course, there are a ton of libraries that does great visualization and you can use any of them as long as they are rendered. You can also add additional parameters to the URLs listed above for additional filtering. ents 10/6/2020 Assignment 2.1 (Due 19:00 CDT on Oct 11, 2020) - CS 242 - Illinois Wiki https://wiki.illinois.edu/wiki/pages/viewpage.action?pageId=616243850 3/5 You are also encouraged to perform your own analysis of your data and may receive bonus points for interesting and/or well- presented analysis. Testing As usual, we require that you write extensive unit tests for each part of this assignment. Since it is difficult to write unit tests for your visualization, you should submit a manual test plan with relevant screenshots. For your API, we expect you to use pytests to verify your APIs. Summary Reading On the Criteria To Be Used in Decomposing Systems into Modules Submission Moderators are asked to grade styles according to PEP8 Standard, so please follow it. If you do follow an alternative style guide, please provide them to the moderator and TAs and demonstrate that you are following this certain convention. This implementation is to maintain grading standards across sections. (Alternatives, Airbnb Ruby Style Guide, Airbnb Javascript Style Guide) Please use Python 3.4+ and not Python 2. This assignment is due on 19:00 CDT Oct 11, 2020. Please be sure to submit in Gitlab, grant correct access, and ask your moderator or TA before the deadline if you have any questions. Please make sure you follow the repo naming conventions listed on the piazza. Please make sure that you create a branch with the name assignment-2.1 and merge it back to the master (while keeping the branch) Just to stress again, you do not need to complete the entire assignment to get 100 points. The rubric is designed for you to focus on writing good code first then complete the additional functionalities. Objectives Learn about data visualization Learn how to create and test a web API Learn to render results using HTML templates Resources Python Other relevant resources are listed in the descriptions above. Category Scoring Notes Basic Preparation Basic Preparation 2 2 pts - Comes to the section on time within 10 minutes and fully setup Presentation 3 -1 pt: Student does not exceed the allotted time -1 pt: Present the functionalities implemented that ents The readings are due before lecture as usual on Fridays. 10/6/2020 Assignment 2.1 (Due 19:00 CDT on Oct 11, 2020) - CS 242 - Illinois Wiki https://wiki.illinois.edu/wiki/pages/viewpage.action?pageId=616243850 4/5 week -1 pt: Student's presentation does not highlight how styling was taken into account while coding, how improvements were made from previous weeks in styling or did not show how current week’s rubric for styling was taken into account -1 pt: Student's presentation does not include what they think they had not done well and wants to improve Participation 3 +1 pt for each interaction: Interact with the group 3 times (ask a question, make a comment, help answer a question, etc.) Effort 2 -1 pt: The effort on complying with best coding practices. -1 pt: The effort on completing functional requirements. Commenting/Documentation 3 -1 pt for each infraction: Block comments should be written for all public functions and nontrivial private functions with clear documentation. Inline comments should be written to provide context to a non-intuitive solution Code submission 4 -1: Repo URL correct -1: Repo named correctly Submit code with multiple commits: -2pt with only one commit message -1pt for multiple bad commit messages +0.1pt for very exceptional commit messages Submit with a clear and complete readme: -2 pt for no readme -1 pt for incomplete or readme with less than 10 lines Decomposition/Overall Design 4 -1 pt per infraction: Any piece of the project must execute its core functionality well. The project should be well decomposed into classes and methods. The project is adequately decomposed into abstract classes and classes. The abstract classes and methods with shared functionality from different parts should generalized and reusable. Functions are not duplicated and you make an effort to ensure that duplicate pieces of code are refactored into methods. The project should be maintainable, which means it should be easy to make changes to parts of the project without breaking its functionality. Tests should also be up to date. The code should be readable. (If moderators need to re-read the line twice to understand what it does, it is not readable.) Environment variables should be protected and should not be hard-coded. (If API keys and secrets and other variables should be provided with an alternative env file, pts should be taken off if not done.) Style guide 4 -1 pt per infraction of a guideline: ents 10/6/2020 Assignment 2.1 (Due 19:00 CDT on Oct 11, 2020) - CS 242 - Illinois Wiki https://wiki.illinois.edu/wiki/pages/viewpage.action?pageId=616243850 5/5 Follow all PEP 8 guidelines. Presentation 3 Present the code clearly Functional Requirements (Total: 16) Requirements - API 8 0 points: Lacks any API For each CRUD API (GET, POST, PUT, DELETE), 2 points are rewarded. GET: +1 per route. Both routes need to support AND & OR POST: +0.5 per route PUT: +1 per route DETELE: +1 per route -0.5 point for each wrong return types (200, 201, 415, 404) for each route Requirements - Rendering 4 0 points: Lacks any rendering For completing each of the rendering listed, 1 point is rewarded Requirements - Data analysis: Query and Visualization 3 0 points: Lacks support for importing provided JSON 1 point for each query 0.5 points for each visualization Testing Requirements (Total: 10) Unit Tests - API & Data analysis 5 0 points: No unit tests for every 2 unit tests, gain 1 point Manual Test Plan 5 0 points: No manual tests for every 2 manual test, gain 1 point Bonus (Total: 2.5) Fleshed out rendering page 1 1 extra point toward the entire assignment if the rendering is very nice looking and easy to navigate Deployed to an external server 0.5 0.5 EC toward entire assignment if the content is deployed to an external server such as Heroku or firebase Utilize CI 1 1 EC toward the entire assignment if the repository is set-up with CI/CD on GitLab for your APIs. Resources can be found here. Total 51(+2.5) ents No labels 欢迎咨询51作业君