2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 1/10
Resources / Assignments (/COMP9321/20T1/resources/41975)
/ Week 7 (/COMP9321/20T1/resources/41978) / Assignment-2
Assignment-2


Data Service for World Bank Economic Indicators
In this assignment, you are asked to develop a Flask-Restplus data service that allows a client to read and
store some publicly available economic indicator data for countries around the world, and allow the consumers
to access the data through a REST API.
IMPORTANT : For this assignment , you will be asked to access the given Web content programmatically.
Some Web hosts do not allow their web pages to be accessed programmatically, some hosts may block your
IP if you access their pages too many times. During the implementation, download a few test pages and
access the content locally - try not to perform too many programmatically.
The source URL: http://api.worldbank.org/v2/ (http://api.worldbank.org/v2/)
(http://api.worldbank.org/v2/) Documentations on API Call Structure:
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
The World Bank indicators API provides 50 year of data over more than 1500 indicators for countries around
the world.
List of indicators can be found at: http://api.w o rldbank.org/v2/indicators
(http://api.worldbank.org/v2/indicators)
List of countries can be found at: (http://api.worldbank.org/v2/countries)
http://api.worldbank.org/v2/countries (http://api.worldbank.org/v2/countries)
As the documentation shows , you can construct URLs specifying different parameters to acquire necessary
data. Use a custom URL to get information about a specific indicator. For example, the data on the GDP of all
countries from 2012 to 2017 can be acquired via this URL:
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017)
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&per_page=1000
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&per_page=1000)
For this assignment we are interested in annual values of a user specified indicator from 2012 to 2017 for one
or all countries. The content of the file is quite straightforward. The data service specification will ask you to
'import' this data into a 'data storage'. You should inspect the content of the file carefully and decide on a data
model and storage method. You will find that a JSON format is suitable to accessing data and publishing it.

Specification Make Submission Check Submission Collect Submission
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 2/10
Assignment Specification
The data service should use JSON format to publish its data and implement following operations.
Question-1: Import a collection from the data service (2.5 marks)
This operation can be considered as an on-demand 'import' operation. The service will download the JSON
data for all countries (http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&per_page=1000) respective to the year 2012 to 2017 and identified by the
indicator id given by the user and process the content into an internal data format and store it in the database;
use sqlite for storing the data (the name of the database should be YOURZID.db ) .
For example, the following link can be used to obtain data for the indicator called: NY.GDP.MKTP.CD
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&per_page=1000)
http://api.worldbank.org/v2/countries/all/indicators/ NY.GDP.MKTP.CD ?
date=2012:2017&format=json&per_page=1000
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?
date=2012:2017&format=json&per_page=1000)
To import data your require an indicator as a parameter given by the API user. The parameter should be a
query parameter named:
indicator_id : an indicator (http://api.worldbank.org/v2/indicators) http://api.worldbank.org/v2/indicators
(http://api.worldbank.org/v2/indicators)
After importing the collection, the service should return a response containing at least the following information:
uri : the URL with which the imported collection can be retrieved
id : a unique integer identifier automatically generated
creation_time : the time the collection stored in the database
Example:
HTTP operation: POST /collections?indicator_id=NY.GDP.MKTP.CD
Returns: 201 Created
{
"uri" : "/collections/1",
"id" : 1,
"creation_time": "2019-04-08T12:06:11Z",
"indicator_id" : "NY.GDP.MKTP.CD"
}
You should return appropriate responses (JSON formatted) in case of already imported collections,
invalid indicators or any invalid attempts to use the endpoint ( e.g. If the input indicator id doesn't exist in
the data source, return error 404 )
UPDATE: Please remove those entries where the value is NONE
Question 2- Deleting a collection with the data service (2.5 marks)
This operation deletes an existing collection from the database. The interface should look like as below:
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 3/10
HTTP operation: DELETE /collections/{id}
Returns: 200 OK
{
"message" :"The collection 1 was removed from the database!",
"id": 1
}
Question 3 - Retrieve the list of available collections (2.5 marks)
This operation retrieves all available collections. The interface should look like as like below:
"order_by" is a comma separated string value to sort the collection based on the given criteria. Each segment
of this value indicates how the collection should be sorted, and it consists of two parts (+ or -, and the name of
column e.g., id). In each segment, + indicates ascending order, and - indicates descending order. Here are
some samples:
+id,+creation_time order by "id ascending" and then "creation_time ascending"
In this case sorting by "id" has priority over "creation_time "
-creation_time order by "creation_time descending"
Returns: 200 OK
[
{
"uri" : "/collections/1",
"id" : 1,
"creation_time": "2019-04-08T12:06:11Z",
"indicator" : "NY.GDP.MKTP.CD"
},
{
"uri" : "/collections/2",
"id" : 2,
"creation_time": "2019-05-08T12:16:11Z",
"indicator" : "2.0.cov.Math.pl_3.all"
},
...
Question 4 - Retrieve a collection (2.5 marks)
This operation retrieves a collection by its ID . The response of this operation will show the imported content
from world bank API for all 6 years. That is, the data model that you have designed is visible here inside a
JSON entry's content part.
The interface should look like as like below:
HTTP operation: GET /collections?order_by={+id,+creation_time,+indicator,-id,-creation_time,-in
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 4/10
HTTP operation: GET /collections/{id}
Returns: 200 OK
{
"id" : 1,
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"creation_time" : "2019-04-08T12:06:11Z"
"entries" : [
{"country": "Arab World", "date": 2016, "value": 2500164034395.78 },
{"country": "Australia", "date": 2016, "value": 780016444034.00 },
...
]
}
Question 5 - Retrieve economic indicator value for given country and a year (2.5
marks)
The interface should look like as like below:
HTTP operation: GET /collections/{id}/{year}/{country}
Returns: 200 OK
{
"id": 1,
"indicator" : "NY.GDP.MKTP.CD",
"country": "Arab World",
"year": 2016,
"value": 780016444034.00
}
Question 6 - Retrieve top/bottom economic indicator values for a given year
(2.5 marks)
The interface should look like as like below:
HTTP operation: GET /collections/{id}/{year}?q=
Returns: 200 OK
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 5/10
Resource created about a month ago (Thursday 27 February 2020, 12:54:52 PM), last modified about 15 hours ago (Tuesday 31
March 2020, 10:35:01 PM).
{
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"entries" : [
{
"country": "Arab World",
"value": 2500164034395.78
},
...
]
}
The is an optional integer parameter which can be either of following:
+N (or simply N) : Returns top N countries sorted by indicator value (highest first)
-N : Returns bottom N countries sorted by indicator value
where N can be an integer value between 1 and 100. For example, a request like " GET /collections/1 /2015?
q=+10 " should returns top 10 countries according to the indicator value.
Notice:
Your code must implemented in flask-restplus and automatically generate swagger doc for testing the
endpoints.
Your code must be executable in CSE machines
Your code must not require installing any software (python libraries are fine)
Your code must be written in Python 3.5 or newer versions.
Your operations (API methods) must return appropriate responses in JSON format, and appropriate
HTTP response code! e.g., 500 Internal Server Error is inappropriate response!
Make sure you are using right datatypes in the database and in you API methods (e.g., not string for
years '2017')
Install flask_restplus on cse machines by pip3 command, and make sure you install pip3 install
werkzeug~=0.16.1
Submission:
One and only one Python script file named " YOUR_ZID .py" which contains your code
How I can submit my assignment?
Go to the assignment page click on the "Make Submission" tab; pick your files which must be named
"YOUR_ZID.py". Make sure that the files are not empty, and submit the files together.
Can I submit my file after deadline?
Yes, you can. But 25% of your assignment will be deducted as a late penalty per day. In other words, if
you be late for more than 3 days, you will not be marked.
Comments
  (/COMP9321/20T1/forums/search?forum_choice=resource/43071)
 (/COMP9321/20T1/forums/resource/43071)
Add a comment
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 6/10
Sophia Lev (/users/z5178140) about 3 hours ago (Wed Apr 01 2020 10:28:31 GMT+1100 (澳大利亚东
部夏令时间))
Hi Mohammadali, I have trouble using flask-restplus on the cse machine. I run "pip3 install
flask-restplus" command, and the installation seems okay, but when I try running my code, I
get an error: "ModuleNotFoundError: No module named 'flask_restplus'". What am I doing
wrong?
Thanks!
Reply
Shubhankar Mathur (/users/z5229053) about 11 hours ago (Wed Apr 01 2020 01:37:39 GMT+1100
(澳大利亚东部夏令时间))
Is it possible to share a sample url for question 3 in a similar manner as question 1 and
question 6?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 5 hours ago (Wed Apr 01 2020
08:25:12 GMT+1100 (澳大利亚东部夏令时间))
GET /collections?order_by=+id,+creation_time
Reply
Sunreet Singh (/users/z5130780) about 12 hours ago (Wed Apr 01 2020 01:30:46 GMT+1100 (澳大利
亚东部夏令时间))
My Collections/{id} endpoint is just taking unsigned int (positive int) and returning 404 when
given any negative value.
Should the automatic error handling be okay? Or should I add my own error?

Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 5 hours ago (Wed Apr 01 2020
08:25:55 GMT+1100 (澳大利亚东部夏令时间))
The response is not a valid json! so you should have your own error message in json
format
Reply
Yu Xia (/users/z5212108) about 12 hours ago (Wed Apr 01 2020 00:48:46 GMT+1100 (澳大利亚东部夏
令时间)), last modified about 12 hours ago (Wed Apr 01 2020 00:55:41 GMT+1100 (澳大利亚东部夏令时间))
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 7/10
I notice that for some indicators, the number of entries is pretty large (e.g,
"NY.GDP.MKTP.CD" has 1584 entries). And it took me around 30 seconds to store them in
my database and show result in swagger doc. Would that be a problem? Will we lose marks
for that?
By the way, i don't know why that took so long. I think my code is pretty efficient (in terms of
time complexity). Is it normal for python to took that long, or is it because of my
implementation?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 5 hours ago (Wed Apr 01 2020
08:24:16 GMT+1100 (澳大利亚东部夏令时间))
Not sure why it takes too much time (e.g., if it is because of many inserts, use bullk insert
in SQL)
But there is not penalty for low performance.
Reply
Chenfan Zhu (/users/z5199811) about 13 hours ago (Wed Apr 01 2020 00:10:04 GMT+1100 (澳大利亚
东部夏令时间))
About Q1,
suppose we originally had [[id=1, id=2 ] as our collections, and then we delete collection 2.
Should a new collection have id=2 or id=3?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 5 hours ago (Wed Apr 01 2020
08:21:30 GMT+1100 (澳大利亚东部夏令时间))
in the assignment, this is not important. but in reality, it should be 3
Reply
Xavier Poon (/users/z5110093) about 15 hours ago (Tue Mar 31 2020 22:22:16 GMT+1100 (澳大利亚
东部夏令时间))
In Q6, it says 'q=', however in the example they have 'query=+10'. Should the query
parameter be named 'q' or 'query'?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 15 hours ago (Tue Mar 31 2020
22:35:15 GMT+1100 (澳大利亚东部夏令时间))
it should be "q"
Reply
Chenfan Zhu (/users/z5199811) about 18 hours ago (Tue Mar 31 2020 19:33:42 GMT+1100 (澳大利亚
东部夏令时间))
About Q3, will the query be { +id,-creation_time}?
And should we treat it as 400 Error
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 8/10
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 15 hours ago (Tue Mar 31 2020
22:35:43 GMT+1100 (澳大利亚东部夏令时间))
did not get the question
Reply
Chenfan Zhu (/users/z5199811) about 14 hours ago (Tue Mar 31 2020 22:43:25 GMT+1100
(澳大利亚东部夏令时间))
HTTP operation: GET /collections?order_by={+id,-creation_time}
is it possible that order by id ASC and then order by creation_time DESC?
Reply
Mark Timothy Tabios (/users/z5181061) about 18 hours ago (Tue Mar 31 2020 19:30:46 GMT+1100
(澳大利亚东部夏令时间))
for the creation_time,
Q1. is this following UTC time or Local Time?
Q2. Should we follow the same format as given above:
"2019-05-08T12:16:11Z"
or can we use default python datetime iso format which could be something like this?
"2020-03-31T00:00:32.367969"
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 15 hours ago (Tue Mar 31 2020
22:35:56 GMT+1100 (澳大利亚东部夏令时间))
Either is fine
Reply
Sunreet Singh (/users/z5130780) about 18 hours ago (Tue Mar 31 2020 18:49:19 GMT+1100 (澳大利
亚东部夏令时间)), last modified about 17 hours ago (Tue Mar 31 2020 19:49:25 GMT+1100 (澳大利亚东部
夏令时间))
Q6,

+N (or simply N) : Returns top N countries sorted by indicator value ( highest first )
Do we need to sort by value or indicator_value. If we are just sorting according to just value ,
highest first makes sense.
"value": 2500164034395.78
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 9/10
If it means indicator_value , I didn't get what highest first mean? Wouldn't it be just in
Alphabetical order?
"indicator_value": "GDP (current US$)"?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 15 hours ago (Tue Mar 31 2020
22:36:39 GMT+1100 (澳大利亚东部夏令时间))
sort by "value"
Reply
Bhawna Kundu (/users/z5130378) about 22 hours ago (Tue Mar 31 2020 15:27:50 GMT+1100 (澳大利
亚东部夏令时间))
Hi, for Q3 do we have to consider the inputs like:
{-id,} - an extra "," at the end of the input string
{-id,sdc,+id} - invalid middle input
If yes, do I have to just return the valid status codes (400, 404) there or do I have to ignore
the invalid values and then sort according to the valid inputs.
E.g: In case {-id,sdc,+id} should I ignore the value "sdc" and sort w.r.t "-id,+id"?
Thanks!

Reply
Mark Timothy Tabios (/users/z5181061) about 16 hours ago (Tue Mar 31 2020 21:36:35
GMT+1100 (澳大利亚东部夏令时间)), last modified about 15 hours ago (Tue Mar 31 2020 21:37:20
GMT+1100 (澳大利亚东部夏令时间))
good question on invalid middle input, waiting for tutor's response with you then but
personally, it feels like an invalid request and should throw an error. It does not follow
valid values in the request and can pose security risk to your application.
Reply
Junyu Ren (/users/z5195715) about 20 hours ago (Tue Mar 31 2020 16:37:04 GMT+1100 (澳大利
亚东部夏令时间))
I think even '-id,+id' is an error,you cannot sort by id asc and desc simuntaniously
Reply
Dominik Tobaben (/users/z5298989) about 16 hours ago (Tue Mar 31 2020 21:07:09
GMT+1100 (澳大利亚东部夏令时间))
You can do so in SQL (order by id ASC, id DESC), it just doesn't have any effect, but
still works fine
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 14 hours ago (Tue Mar 31
2020 22:37:43 GMT+1100 (澳大利亚东部夏令时间))
2020/4/1 Assignment-2 | COMP9321 20T1 | WebCMS3
https://webcms3.cse.unsw.edu.au/COMP9321/20T1/resources/43071 10/10
Load More Comments
True
Reply
Bhawna Kundu (/users/z5130378) about 19 hours ago (Tue Mar 31 2020 18:27:24
GMT+1100 (澳大利亚东部夏令时间))
How about {-id,sdc,+creation_time}?
In case {-id,sdc,+creation_time} should I ignore the value "sdc" and sort w.r.t "-
id,+creation_time"?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 14 hours ago (Tue Mar 31
2020 22:38:45 GMT+1100 (澳大利亚东部夏令时间))
this is an invalid request; and can be treated as a bad request
Reply
Bhawna Kundu (/users/z5130378) about 14 hours ago (Tue Mar 31 2020 22:43:04
GMT+1100 (澳大利亚东部夏令时间))
Thanks
Reply