IronClad Case Study
Due Monday by 3am
Points 100
Submitting a website url or a file upload
Start Assignment
"IronClad" Visual Search System Case Study
Objective: The objective of this assignment is to develop a machine-learning system prototype to perform
(nearest neighbor) searches for similar images for a given "probe" image. The ultimate goal is to provide you
with practical experience implementing, evaluating, and deploying a real-world computer vision visual search
system.
Deliverable A: Docker-enabled Software Package (30%)
Submit this deliverable through your provisioned repository on GitHub. Minimally, your system must include
the following capabilities:
Extraction Service:
The system must properly preprocess image inputs compliant with the embedding model's
specification.
The system must be able to extract "embeddings" from images.
Hint: Your probe directory should not be copied to your docker container.
Retrieval Service:
The system should index images in the catalog storage.
The system's nearest neighbor search must scale to billions of images.
The system should return the k-nearest neighbors of an input "probe."
Hint*: Let the user define the system parameter k. Argue how you would select the best k for the
system.
Interface Service:
The system should support an endpoint (@app.route("/identify")) to trigger the authentication process
and return the top-k predicted identities
To send a file to the Flask server using curl, use the example in `student-
resources/ironclad/app.py`.
The system should support an endpoint that incrementally adds identities to the gallery
(@app.route("/add")).
To send a file to the Flask server using curl, use the example in `student-
resources/ironclad/app.py`.
The system should support an endpoint that extracts historical predictions/searches and their
pertinent details (@app.route("/history")).
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...1/8
IronClad Case Study (1)
The system should be contained within a docker container.
Deliverable B: Written Report (70%)
Submit a report in the form of a Markdown file (called System_Report.md) detailing your analysis of system
design decisions. Submit supporting notebook files on GitHub. The report must include the following
sections:
System Design:
Use accompanying diagrams to describe your system components and processes.
Only one paragraph per service
Metrics Definition:
Provide a paragraph description of your offline metrics and their purpose.
Provide a paragraph description of your online metrics and their purpose. How would you monitor
these metrics?
Analysis of System Parameters and Configurations:
Identify FIVE significant design decisions for this system. Be sure to enumerate them for clarity. Argue
how each design decision is significant to the system (i.e., how it would impact the system as a
whole). Explain the methods you used to arrive at a decision. Use graphs, tables, etc., to substantiate
your arguments with evidence. You must have at least two analyses from each of the following
services:
Extraction Service
Retrieval Service (e.g., see Hint* in Deliverable A: Retrieval Service)
Given your evidence, provide how this would influence your design.
Note: You may upload your FAISS serialized file using the File Upload functionality.
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...2/8
CriteriaRatingsPts
30 pts
10 pts
Docker-
enabled
Software
Package
30 to >0.0 pts
Full Marks
0 pts
No Marks
System Design
10 pts
Excellent
(Top 50%)
Demonstrates
a deep
understanding
of the
problem,
addressing
complex
requirements
and
constraints
with clear
insights from
the analysis.
Presents an
innovative,
detailed, and
scalable
system
configuration.
Clearly
defines and
prioritizes
requirements.
Provides a
thorough
justification of
configuration
choices,
analyzing
trade-offs and
demonstrating
a strong
grasp of
system
complexity
and design
impact.
Critically
discusses
8 pts
Performant
(Middle 50-
10%)
Understands
the problem
and
addresses
most key
requirements,
though some
complex
constraints
may not be
fully
explored. The
system
configuration
is logical and
mostly
scalable but
lacks full
innovation.
Provides
adequate
justification of
configuration
choices with
some
analysis of
trade-offs,
though depth
may be
lacking in
some areas.
Mentions
system
weaknesses
or limitations
but provides
limited or
general
6 pts
Good (Bottom
10%)
Provides a
basic
understanding
of the problem
and
requirements,
though critical
aspects or
complexities
may be
overlooked.
The system
configuration is
functional but
lacks depth or
scalability
considerations.
Justification of
configuration
choices is
present but
with minimal
analysis of
trade-offs or
system impact.
Weaknesses
and limitations
are
acknowledged
but without
sufficient depth
or discussion
of mitigation.
4 pts
Needs
Improvement
(as needed)
Ina
adequately
defines the
problem or
requirements.
The system
configuration
is incomplete
or poorly
designed,
lacking
consideration
for scalability
or complexity.
Little
justification of
configuration
choices, with
trade-offs or
system
impacts left
unexplored.
Weaknesses
and
limitations are
either not
discussed or
are
inadequately
addressed.
2 pts
Poor
Fails to
adequately
define the
problem or
requirements.
The system
configuration
is incomplete
or poorly
designed,
lacking
consideration
for scalability
or complexity.
No
justification of
configuration
choices, with
trade-offs or
system
impacts left
unexplored.
Weaknesses
and
limitations
are either not
discussed or
are
inadequately
addressed.
0 pts
No
Marks
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...3/8
CriteriaRatingsPts
5 pts
2 pts
5 pts
3 pts
system
weaknesses
and
limitations
resulting from
the analysis
and provides
mitigation
strategies.
strategies for
mitigation.
Offline Metrics
5 pts
Full Marks
Provides a
clear, detailed
description of
offline metrics,
thoroughly
explaining their
role in
evaluating
system
performance
before
deployment.
Clearly outlines
how they are
used in testing,
validation, or
model training
stages.
Demonstrates
strong
understanding
of the specific
importance and
relevance of
these metrics to
the system.
4 pts
Performant
Defines
offline
metrics
clearly, with
a good
explanation
of their
purpose,
though
missing
some finer
points on
their
significance
or specific
usage in
testing.
3 pts
Good
Provides a
basic
explanation
of offline
metrics but
lacks depth
or a clear
connection
to system
evaluation
or its
purpose.
2 pts
Needs
Improvement
Incorrectly
defines offline
metrics clearly
or provides no
understanding
of their role or
purpose in
system
evaluation.
1 pts
Poor
Fails to define
offline metrics
clearly, or
provides little
to no
understanding
of their role or
purpose in
system
evaluation.
0 pts
No
Marks
Analysis 1:
Objective,
Motivation,
Methods
2 pts
Excellent
Thoroughly explains the objective,
purpose, and significance of the
analysis; fully connects to the stated
requirements, demonstrating deep
insight into the problem context.
1 pts
Good
Objectives and motivation
are explained at a basic
level, but significant
connections to the
requirements are missing.
0 pts
Poor
Objectives and
motivation are unclear,
poorly explained or
irrelevant to the
requirements.
Analysis 1:
Justification of
Design
Decisions with
Evidence
5 pts
Excellent
(Top 20%)
Provides a
detailed
analysis of
trade-offs and
strong, well-
substantiated
rationale for
decisions;
demonstrates
a solid
understanding
of system
configuration
and its impact
on
requirements;
and balances
cost and
performance.
(Awarded to
4 pts
Performant
(Middle 20-
40%)
Provides
clear
justification
for most
design
decisions,
addressing
trade-offs
with
reasonable
evidence,
though
lacking
depth in
some areas.
(Awarded to
20-40%
percentile of
submissions)
3 pts
Good
(Middle 40-
60%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
40-60%
percentile of
submissions)
2 pts
Needs
Improvement
(Middle 60-
80%)
Justification is
weak or
missing, with
little to no
discussion of
trade-offs or
rationale for
design
decisions.
(Awarded to
60-80%
percentile of
submissions)
1 pts
Poor
(Bottom
~20%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
bottom ~20%
percentile of
submissions)
0 pts
No
Marks
No
analysis
provided
Analysis 1:
Documentation
and Clarity
3 pts
Excellent
Case study is written,
logically structured, and
easy to follow; visual aids
such as diagrams and
charts enhance
2 pts
Good
Writing is mostly
clear and logical,
though some
complex concepts
may be hard to
1 pts
Poor
Case study is
understandable but lacks
clear structure or has
confusing sections;
0 pts
No Marks
No
analysis
provided.
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...4/8
CriteriaRatingsPts
2 pts
5 pts
3 pts
2 pts
5 pts
top 20%
percentile of
submissions)
understanding of complex
concepts.
follow; visuals are
present but not
entirely effective.
minimal or unclear visual
aids.
Analysis 2:
Objective,
Motivation,
Methods
2 pts
Excellent
Thoroughly explains the objective,
purpose, and significance of the
analysis; fully connects to the stated
requirements, demonstrating deep
insight into the problem context.
1 pts
Good
Objectives and motivation
are explained at a basic
level, but significant
connections to the
requirements are missing.
0 pts
Poor
Objectives and
motivation are unclear,
poorly explained or
irrelevant to the
requirements.
Analysis 2:
Justification of
Design
Decisions with
Evidence
5 pts
Excellent
(Top 20%)
Provides a
detailed
analysis of
trade-offs and
strong, well-
substantiated
rationale for
decisions;
demonstrates
a solid
understanding
of system
configuration
and its impact
on
requirements;
and balances
cost and
performance.
(Awarded to
top 20%
percentile of
submissions)
4 pts
Performant
(Middle 20-
40%)
Provides
clear
justification
for most
design
decisions,
addressing
trade-offs
with
reasonable
evidence,
though
lacking
depth in
some areas.
(Awarded to
20-40%
percentile of
submissions)
3 pts
Good
(Middle 40-
60%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
40-60%
percentile of
submissions)
2 pts
Needs
Improvement
(Middle 60-
80%)
Justification is
weak or
missing, with
little to no
discussion of
trade-offs or
rationale for
design
decisions.
(Awarded to
60-80%
percentile of
submissions)
1 pts
Poor
(Bottom
~20%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
bottom ~20%
percentile of
submissions)
0 pts
No
Marks
No
analysis
provided
Analysis 2:
Documentation
and Clarity
3 pts
Excellent
Case study is written, logically
structured, and easy to follow;
visual aids such as diagrams
and charts enhance
understanding of complex
concepts.
2 pts
Good
Writing is mostly clear and
logical, though some
complex concepts may be
hard to follow; visuals are
present but not entirely
effective.
0 pts
Poor
Case study is
understandable but lacks
clear structure or has
confusing sections; minimal
or unclear visual aids.
Analysis 3:
Objective,
Motivation,
Methods
2 pts
Excellent
Thoroughly explains the objective,
purpose, and significance of the
analysis; fully connects to the stated
requirements, demonstrating deep
insight into the problem context.
1 pts
Good
Objectives and motivation
are explained at a basic
level, but significant
connections to the
requirements are missing.
0 pts
No Marks
Objectives and
motivation are unclear,
poorly explained or
irrelevant to the
requirements.
Analysis 3:
Justification of
Design
Decisions with
Evidence
5 pts
Excellent
(Top 20%)
Provides a
detailed
4 pts
Performant
(Middle 20-
40%)
3 pts
Good
(Middle 40-
60%)
2 pts
Needs
Improvement
(Middle 60-
80%)
1 pts
Poor
(Bottom
~20%)
0 pts
No
Marks
No
analysis
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...5/8
CriteriaRatingsPts
3 pts
2 pts
5 pts
3 pts
analysis of
trade-offs and
strong, well-
substantiated
rationale for
decisions;
demonstrates
a solid
understanding
of system
configuration
and its impact
on
requirements;
and balances
cost and
performance.
(Awarded to
top 20%
percentile of
submissions)
Provides
clear
justification
for most
design
decisions,
addressing
trade-offs
with
reasonable
evidence,
though
lacking
depth in
some areas.
(Awarded to
20-40%
percentile of
submissions)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
40-60%
percentile of
submissions)
Justification is
weak or
missing, with
little to no
discussion of
trade-offs or
rationale for
design
decisions.
(Awarded to
60-80%
percentile of
submissions)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
bottom ~20%
percentile of
submissions)
provided
Analysis 3:
Documentation
and Clarity
3 pts
Full Marks
Case study is written,
logically structured, and
easy to follow; visual aids
such as diagrams and
charts enhance
understanding of complex
concepts.
2 pts
Good
Writing is mostly
clear and logical,
though some
complex concepts
may be hard to
follow; visuals are
present but not
entirely effective.
1 pts
Poor
Case study is
understandable but lacks
clear structure or has
confusing sections;
minimal or unclear visual
aids.
0 pts
No Marks
No
analysis
provided.
Analysis 4:
Objective,
Motivation,
Methods
2 pts
Excellent
Thoroughly explains the objective,
purpose, and significance of the
analysis; fully connects to the stated
requirements, demonstrating deep
insight into the problem context.
1 pts
Good
Objectives and motivation
are explained at a basic
level, but significant
connections to the
requirements are missing.
0 pts
Poor
Objectives and
motivation are unclear,
poorly explained or
irrelevant to the
requirements.
Analysis 4:
Justification of
Design
Decisions with
Evidence
5 pts
Excellent
(Top 20%)
Provides a
detailed
analysis of
trade-offs and
strong, well-
substantiated
rationale for
decisions;
demonstrates
a solid
understanding
of system
configuration
and its impact
on
requirements;
and balances
cost and
4 pts
Performant
(Middle 20-
40%)
Provides
clear
justification
for most
design
decisions,
addressing
trade-offs
with
reasonable
evidence,
though
lacking
depth in
some areas.
(Awarded to
20-40%
3 pts
Good
(Middle 40-
60%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
40-60%
percentile of
submissions)
2 pts
Needs
Improvement
(Middle 60-
80%)
Justification is
weak or
missing, with
little to no
discussion of
trade-offs or
rationale for
design
decisions.
(Awarded to
60-80%
percentile of
submissions)
1 pts
Poor
(Bottom
~20%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
bottom ~20%
percentile of
submissions)
0 pts
No
Marks
No
analysis
provided
Analysis 4:
Documentation
and Clarity
3 pts
Full Marks
Case study is written,
logically structured, and
2 pts
Good
Writing is mostly
clear and logical,
1 pts
Poor
Case study is
understandable but lacks
0 pts
No Marks
No
analysis
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...6/8
CriteriaRatingsPts
2 pts
5 pts
3 pts
5 pts
performance.
(Awarded to
top 20%
percentile of
submissions)
percentile of
submissions)
easy to follow; visual aids
such as diagrams and
charts enhance
understanding of complex
concepts.
though some
complex concepts
may be hard to
follow; visuals are
present but not
entirely effective.
clear structure or has
confusing sections;
minimal or unclear visual
aids.
provided.
Analysis 5:
Objective,
Motivation,
Methods
2 pts
Excellent
Thoroughly explains the objective,
purpose, and significance of the
analysis; fully connects to the stated
requirements, demonstrating deep
insight into the problem context.
1 pts
Good
Objectives and motivation
are explained at a basic
level, but significant
connections to the
requirements are missing.
0 pts
Poor
Objectives and
motivation are unclear,
poorly explained or
irrelevant to the
requirements.
Analysis 5:
Justification of
Design
Decisions with
Evidence
5 pts
Excellent
(Top 20%)
Provides a
detailed
analysis of
trade-offs and
strong, well-
substantiated
rationale for
decisions;
demonstrates
a solid
understanding
of system
configuration
and its impact
on
requirements;
and balances
cost and
performance.
(Awarded to
top 20%
percentile of
submissions)
4 pts
Performant
(Middle 20-
40%)
Provides
clear
justification
for most
design
decisions,
addressing
trade-offs
with
reasonable
evidence,
though
lacking
depth in
some areas.
(Awarded to
20-40%
percentile of
submissions)
3 pts
Good
(Middle 40-
60%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
40-60%
percentile of
submissions)
2 pts
Needs
Improvement
(Middle 60-
80%)
Justification is
weak or
missing, with
little to no
discussion of
trade-offs or
rationale for
design
decisions.
(Awarded to
60-80%
percentile of
submissions)
1 pts
Poor
(Bottom
~20%)
Offers basic
justification
for design
decisions,
with limited
consideration
of trade-offs
or lacking
solid
evidence for
key choices.
(Awarded to
bottom ~20%
percentile of
submissions)
0 pts
No
Marks
No
analysis
provided
Analysis 5:
Documentation
and Clarity
3 pts
Full Marks
Case study is written,
logically structured, and
easy to follow; visual aids
such as diagrams and
charts enhance
understanding of complex
concepts.
2 pts
Good
Writing is mostly
clear and logical,
though some
complex concepts
may be hard to
follow; visuals are
present but not
entirely effective.
1 pts
Poor
Case study is
understandable but lacks
clear structure or has
confusing sections;
minimal or unclear visual
aids.
0 pts
No Marks
No
analysis
provided.
Online Metrics
5 pts
Full Marks
Offers a
thorough
description of
online metrics
and their role in
real-time
system
monitoring.
Provides
4 pts
Performant
Describes
online
metrics with
relevant
examples,
but lacks
depth in
connecting
them to
3 pts
Good
Provides a
basic
explanation
of online
metrics, but
does not fully
connect them
to real-time
performance.
2 pts
Needs
Improvement
Fails to
explain online
metrics clearly
and their role
in live system
monitoring.
Monitoring
strategy is
1 pts
Poor
Does not
provide an
understanding
of their role in
live system
monitoring.
Monitoring
strategy is
poorly
0 pts
No
Marks
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...7/8
Total Points: 100
CriteriaRatingsPts
detailed
examples of
their purpose
and explains
how they relate
to user
experience,
system health,
or business
outcomes.
Clearly
explains how
online metrics
would be
monitored
using specific
tools,
strategies, or
processes.
Discusses how
monitoring
helps detect
issues and
improve
system
performance.
Demonstrates
a clear
understanding
of their
importance in
evaluating live
system
performance.
broader
system
goals or
their
significance
for real-time
monitoring.
Monitoring
strategy is
described
but lacks
specific
tools or
methods.
Monitoring
strategy is
present but
lacks clarity
or detail.
poorly
explained or
missing.
explained or
missing.
2024/10/24 20:48IronClad Case Study
https://jhu.instructure.com/courses/82966/assignments/889126?return_to=https%3A%2F%2Fjhu.instructure.com%2Fcalendar%23view_name%3Dmo...8/8
