ISYS3401 IT Evaluation (2022)
Individual Assignment 1 (30%): Plan for Evaluation
(Technology and User Acceptance)
Due date: Mid-semester break (April 21st)

The aim of IT Evaluation is to assess the technology as well as how the technology is used to achieve
desired outcome(s). In this assignment, your tasks include (1) appraisal of evidence from existing
evaluation studies and (2) to develop a detailed plan to evaluate (1) a piece of technology and it’s
acceptability with appropriate user experience approaches.
Part 1 (Technology focus): Reviewing evidence from existing evaluation of AI for
medical imagining (10%).
The deliverables included in this part 1 would require you to first perform a quick “review” on the
technology, with these questions in mind:
 Description of the Technology
 Who is it for?
 What does this technology set out to do?
 Has any evaluation been done on this technology? If so what were the key findings? What type
of evidence found (You need to look for published articles or evaluation reports)?
 What are the challenges in evaluating this technology?
Output: Summarise your findings into a powerpoint document and virtual presentation in zoom
recording.
Part 2 (User focus): Evaluation Plan for AI for medical imaging used in diagnosis of
Tuberculosis (20%).
In this part, you are interested in the behaviour of the users of how they use this AI technology to
help diagnostic decision making. You need to develop a user acceptance model with appropriate
constructs and metrics. Many reference models can be found in the Information Systems literature.
For example, you may use the Technology Acceptance Model (TAM) as a starting point, or other
appropriate models you might see fit your UX evaluation. The tasks included in part 2:
 Develop a Model for User Acceptability
 What user behaviour(s) you plan to measure? Describe and justify.
 How would you plan for formative assessment(s) and/or summative assessment(s)?
 What constructs and metrics you plan to measure?
 What instruments you plan to use for data collection? Survey questionnaires?
 What is your evaluation strategy? This is to ensure the data you collected is “adequate” and
“appropriate” to be used for analysis.
 Provide limitations of your evaluation plan. Hence, what you are not measuring but could be
relevant to User Acceptability Evaluation.
Output: An evaluation proposal in word document
Submission Details:
1. Assignment description to be made available on Canvas before 21st April. You have 4 weeks (1
extra week from mid-semester break) to complete this assignment.
2. This assignment is done individually.
3. Submit the following files to Canvas before 5pm, Thursday, 21st April 2021 (mid-semester break)

Deliverables:
 Part 1: Recorded presentation via zoom (up to 10 mins) (10%) including a powerpoint document.
 Part 2: A written Evaluation Plan (up to 2000 words) (20%). Suggested structure of your plan
document is shown below:
Structure of your evaluation plan % allocation
Abstract / Executive Summary
 Summarise and highlight of your evaluation plan.

10%
Introduction (motivation, objective(s) and background related work)
 Purpose of your evaluation, what would you like to measure? Why?
 Elaborate on your motivation and objectives behind the evaluation?

20%-30%
Evaluation Framework and Measurement Model
 Introduce your evaluation framework,
 Detail the construction of your measurement model,
 Discuss and articulate the variables used for each construct in your model,
including a question instruments you plan to use for data collection.
Justification might be needed.
 Detail your data collection plan including methods to collect data,
 Make sure the model you develop is appreciate in achieving the
objectives.

50%-60%
Conclusion and Limitations
 It is important to re-iterate your evaluation goals as well as what you have
excluded in the plan (things you do not plan to measure).

10%
References

Note:
 The case study we use for tutorial in week 6 could be of relevance to this assignment. Several
case studies can be found in Chapter 10 of the Measuring the User Experience book.
 A Canvas Discussion Board will be setup for sharing concerns and issues related to the
assignment.

欢迎咨询51作业君