Utilizing FSM and CFT Testing Techniques for Quality Assurance in the Amazon Shopping Application
Haoyang Guo, M.S. in Computer Science Prof. Jeff Tian ([email protected])
CS 7314: Software Testing and Quality Assurance
November 30, 2023
Abstract
This project aims to assess the quality and functionality of the Amazon Shopping Application through a comprehensive testing approach. By leveraging Finite State Machine (FSM) and Control Flow Testing (CFT) techniques, the study not only evaluates the application's behavior in various states and control paths but also aims to enhance the overall user experience. The benefits of these techniques include ensuring smoother navigation, quicker transaction processes, reduced errors, and increased reliability for users interacting with the Amazon Shopping Application. The report details the background, problem statement, testing strategies, activities, models, test cases, results, analysis, lessons learned, and follow-up actions. Besides these two primary methods, we also employed another additional technique in the appendix to conduct brief tests on other functionalities of the Amazon Shopping Application. Utilize UBST to test user feedback and the product review feature. UBST primarily examined user interaction with the functionalities of the Amazon Shopping Application. For detailed information, please refer to the appendix.
Keywords: Software Testing, Finite State Machines, Control Flow Testing, Control Flow Graph, Black Box Testing
Table of Content
1. Introduction ............................................4
2. Activities ...............................................5 2.1 Finite State Machine (FSM) Testing .............................5
2.1.1 Basic Reason Choosing FSM...................................5 2.1.2 FSM Model ................................................6 2.1.3 Sets Representation.........................................7 2.1.4 Table Representation........................................8 2.1.5 Test Cases.................................................9 2.1.6 Summary of FSM ..........................................11
2.2 Control Flow Testing (CFT) ...................................13
2.2.1 Basic Reason Choosing CFT ..................................13 2.2.2 CFG for Amazon ...........................................14 2.2.3 Test Cases................................................15 2.2.4 Test Cases Sensitize ........................................18 2.2.5 Summary of CFT ...........................................18
3. Result and Summary.....................................19 4. Lesson Learned .........................................20 5. Appendix..............................................22
5.1 CFT Test Cases Sensitize .....................................22 5.2 UBST for Amazon...........................................25
5.2.1 UBST Testing Objectives and Strategy ..........................25 5.2.2 Operational Profiles ........................................25 5.2.3 Summary of UBST .........................................28
1. Introduction
Amazon, with over 300 million active customer accounts and a global presence across 180+ countries, has established itself as a dominant e-commerce force. Its extensive product catalog and annual revenue exceeding $386 billion reflect its pivotal role in consumers' lives. With an intuitive interface and streamlined processes, Amazon processes billions of transactions annually, solidifying its position as a preferred online shopping destination. Our project aims to comprehensively evaluate the Amazon Shopping Application's quality and functionalities. Employing Finite State Machine (FSM) and Control Flow Testing (CFT) techniques, we seek to analyze the application's behavior in various states and control paths. FSM allows us to map out distinct user states, while CFT enables the evaluation of control paths within the application, identifying potential logic flaws and enhancing overall reliability. By conducting a thorough assessment using these methodologies, our objective is to identify and rectify potential defects, improve usability, and ensure a seamless shopping experience for Amazon's vast user community.
2. Activities
2.1 Finite State Machine (FSM) Testing 2.1.1 Basic Reason Choosing FSM
The basic idea of FSMs is to use an intermediate formalism to model the program execution or behavior that strikes a balance between expressive power and simplicity (Chow,1978). At one extreme, lists and partitions test only provide simple processing models that may not be expressive enough to represent complex program executions and behavior. On the other hand, the actual implementation, or the programs themselves, contains too much detail that needs to be abstracted into models so that specific aspects or features can be analyzed and tested. FSMs lie in between these two extremes, and possess some flexibility in the level of details that can be modeled by the number of states, the number of links among them, and related input/output (Tian). FSM models can be formalized and used to precisely specify the behavior and interactions for many systems and components and serve as the basis for testing. In addition, many of the sub-operations within end-to-end operations specified in checklists or associated with partitions may be in common. The construction of FSMs could highlight these commonly used core sub-operations. The use of FSMs could lead to more effective testing by focusing on these core sub-operations and their connections to the rest of the system operations. Similarly, more economical testing could also be achieved by avoiding exact repetition of some of these common sub- operations. Therefore, we conducted tests on the interaction and operations between
each page function of the Amazon shopping application by constructing FSM models. This was done to check for any end-to-end conversion issues or interaction problems within the Amazon shopping application. We envisioned how to improve the model to make the entire process more efficient and economically feasible for detection purposes.
2.1.2 FSM Model
Amazon is a vast and complex software with numerous Finite State Machines (FSMs). Here, we are specifically discussing some frequently accessed high-traffic pages that users encounter in the basic shopping process: the homepage, product category page, individual product page, shopping cart page, personal information page, and checkout page. The selected pages cover the basic shopping process quite comprehensively. By conducting Finite State Machine (FSM) analysis on these frequently used pages, we aim to extract crucial characteristics of the transitions between these key functional pages. We constructed the model as shown in Figure 1. In this model, we constructed six states, namely: Main page, Personal page, Shopping page, Category page, Product page, and Payment page. The transitions between each state are also indicated in the model.
2.1.3 Sets Representation
Figure 1
While having the graphical representation we can easily produce the other two representations. After constructing the Finite State Machine (FSM) model diagram, building Sets representation and table representation aids in achieving a more comprehensive understanding and utilization of the model. Sets representation typically refers to the state set, explicitly listing all the possible states the system can be in. This representation allows a clear understanding of the entire state space, aiding in validating the completeness of test cases to ensure coverage of all possible states. Sets representation is as follows.
States:
Main page; Personal page; Shopping page; Category page; Product page; Payment page
Transitions:
Main->Personal; Main->Shopping cart; Main->Category; Main->Product; Personal->Main; Personal->Shopping cart; Personal->Category; Personal->Product; Shopping cart->Main; Shopping cart->Personal; Shopping cart->Category; Shopping cart->Product; Shopping cart->Payment;
Category->Main; Category->Personal; Category->Shopping cart; Category->Product; Product->Main; Product->Personal; Product->Shopping cart; Product->Category; Product->Payment; Product->Product
Payment->Product; Payment->Shopping cart; Payment->main;
2.1.4 Table Representation
Table representation, whether in the form of a state transition table or state transition diagram, presents the system's states and corresponding transition rules in a tabular format. Tables offer a more structured and easily understandable way to describe the relationships between states and transitions. They clearly display transition conditions and associated behaviors between different states, making system behavior rules more explicit. Table representations facilitate easier analysis and review of system behavior. They provide an intuitive view of the system's states and their transition rules, aiding in identifying potential logical errors or behavioral anomalies. Also, based on table representation, it's easier to generate test cases for the system. Analyzing the state transition rules within the table allows the creation of test cases covering various scenarios and conditions, ensuring coverage of diverse
system behaviors. The table representation we constructed is shown in Table 1.
Table 1
2.1.5 Test Cases
After having sets representation and table representation, we constructing test cases based on them. Test case design encompasses two aspects. They are Node coverage test case and Link coverage test case. Node coverage test cases aim to cover all the state nodes in the FSM model diagram. Each state node should have at least one test case to validate that the system can reach or pass through each state. Ensure that the test cases can trigger the system to enter each state, including normal state transitions and edge or exceptional state transitions. Link coverage test cases aim to cover all transitions between states in the FSM model diagram. Ensure that test cases validate every possible transition path between states, encompassing primary pathways and transitions in edge cases. Developing these two types of test cases helps ensure comprehensive testing coverage of the system behavior. Node coverage test cases ensure that each state undergoes testing at least once, while Link coverage test
cases validate transitions between states, aiding in uncovering potential logic errors or abnormal behavior. Adhering to the principle of cost-effectiveness, here we aim to achieve comprehensive node and link coverage using as few test cases as possible. This involves employing one test case for Node Coverage Test and one test case for Link Coverage Test.
Node coverage test case:
Main page->Category page->Personal page->Shopping cart page->Product page->Payment page
Link coverage test case:
Main page->Category page->Main page->Personal page->Main page->Shopping cart page->Main page->Product page->Main page->Personal page->Shopping cart page->Personal page->Category page->Personal page->Product page->Personal page->Category page->Shopping cart page->Category page->Product page->Category page->Shopping cart page->Product page->Shopping cart page->Payment page->Shopping cart page->Product page->Product page->Payment page->Product page->Payment page->Main page
Due to the aim of minimizing the number of test cases, our design of the Link Coverage Test Case includes some duplicated pathways. The design concept of this test case begins with traversing all transitions from one node and then moving to the next node, removing the traversed node and all associated transitions. This process continues until all nodes are completely traversed. During this process, as we traverse
all transitions from one node before moving to the next, there is inevitable repetition when transitioning to the second node. This repetition is unavoidable due to the presence of interlinked transitions among many states in the FSM model we designed. For instance, in figure 2, when traversing node1, which has a transition2 pointing from node2 to node1, following the approach, to ensure complete coverage, we need to traverse all paths associated with node1 before moving to node2. Hence, while traversing node1, it becomes necessary to examine transition2, leading us through transition1 from node1 to node2. After completing the traversal of all nodes from node1, we then proceed to traverse node2, necessitating a repetition of transition1. Before commencing the traversal of node2, it's possible to disregard all transitions associated with node1. This is illustrated in Figure 3.
Figure 2 Figure 3
2.1.6 Summary of FSM
Test case analysis is a crucial step in the software testing process. It aids in uncovering potential issues within the system and promotes system refinement and optimization. Through the analysis of test case execution and outcomes, a comprehensive evaluation of the system's performance and stability is possible,
ensuring adherence to design specifications and user expectations. After illustrating all the FSM test case examples, the next step involves the analysis of these test cases. Analyzing the execution of test cases helps assess whether the system's behavior and functionalities align with expectations and meet the design specifications. When analyzing test cases, considerations include: Executing Test Cases, Comparing Expected Results, Analyzing Exceptional Situations, Coverage Analysis, Defect Identification and Resolution.
In conducting test case evaluations for Amazon, we have completed inspections covering all states and transitions within the system's Finite State Machine (FSM). Throughout this comprehensive testing process, no issues were detected. The transitions between each scenario and product information pages were seamless. The software design exhibited remarkable coherence, offering a user-friendly experience by enabling direct transitions from any given scenario to the desired one, avoiding the need to backtrack to a specific scenario before proceeding to the intended destination.
The FSM checks confirmed the robustness and logical coherence of the software, facilitating direct transitions between diverse scenarios without complications. This seamless navigation significantly enhances user experience by eliminating unnecessary steps and ensuring swift access to desired functionalities. In summary, the testing process validated the system's stability, rational design, and smooth navigational flow between different states, affirming its high usability and efficient operation.
2.2 Control Flow Testing (CFT) 2.2.1 Basic Reason Choosing CFT
The Amazon Shopping application is well-suited for Control Flow Testing (CFT) due to several reasons. The first reason is that it has complex functionality and logic. The Amazon Shopping app encompasses intricate functionalities and logic, involving various shopping scenarios such as product search, cart management, and checkout processes. CFT is suitable for testing these complex control flow paths, ensuring the correct execution of diverse scenarios. The second reason is that its diverse user Interaction paths. Users navigate the Amazon app through different interaction paths, including clicking various buttons, exploring different product categories, etc. CFT can validate these interaction paths, verifying the corresponding control flows for user actions. Besides, its validation of critical functions. Certain critical functions in the application, like order submission, payment processes, involve multiple conditions and decision points. CFT can validate various control flow paths associated with these crucial functions to ensure system correctness and stability. Also, its complex logic Scenarios: The Amazon Shopping application might contain intricate logic, such as promotional activities, coupon applications, user permissions, etc. CFT aids in verifying diverse execution paths for this complex logic to ensure expected operation.
Therefore, CFT is applicable to various sections of the Amazon Shopping application, especially those involving decision-making, branching, and conditional judgment. It allows thorough inspection and validation of control flow paths
associated with different user interactions, functional modules, and critical business processes, thereby ensuring the reliability and stability of the application.
2.2.2 CFG for Amazon
Amazon, as a shopping platform, is primarily used for selecting and purchasing items. In this section, we will employ control flow testing to establish models for how users filter products, make payments, and choose payment methods. By creating models using Control Flow Testing (CFT), we will conduct case testing to identify whether there are any issues in Amazon's process for these activities. Due to the complexity and multitude of categories in the Amazon catalog, we have chosen to establish models focusing on relatively common categories. We constructed the model as shown in Figure 4. In the Control Flow Testing (CFT) model we have constructed, there are multiple Control Flow Graphs (CFGs) nested within each other, including one loop CFG. The loop G1 represents a purchasing process where customers continuously add items to their shopping carts, resulting in repetitive iterations. G2 involves the selection of commonly used shopping categories, determined by conditional choices among various categories. G8 signifies the payment process and is connected to G1 in the sequence of events.
2.2.3 Test Cases
Figure 4
Due to an excessive number of nodes within the Control Flow Testing (CFT), we have substituted a portion that meets the standards with 'G'. In the Control Flow Testing (CFT) model we have constructed, there are multiple Control Flow Graphs (CFGs) nested within each other, including one loop CFG. Here I used a bottom-up
path method to find the test cases. Below are the listed test cases required for each G chart individually. The overall test cases for the entire model can be derived from the combinations of paths chosen for each G chart.
G6: P4-C4-P8-J1; P4-C4-P9-J1
G7: P5-C5-P10-J2; P5-C5-P11-J2
G3: P1-C2-G6-J3; P1-C2-G7-J3
G4: P2-P12-J4
G5: P3-C3-P6-J5-J6; P3-C3-P7-J5-J6
G2: P18-C1-G3-J7; P18-C1-G4-J7; P18-C1-G5-J7
G1: I-P14-C6-P13; I-P14-C6-G2-C6-P13; I-P14-C6-G2-C6-G2-C6-P13; I-P14-C6-G2- C6-G2-C6-G2-C6-P13
G8: P15-C7-P16-J8-F; P15-C7-P17-J8-F
The paths depicted on the up represent the paths derived from a bottom-up
approach. Since G1 involves a loop, assuming it has no higher boundary, we conducted testing for the loop in four cases: Bypass the loop (not entering the loop), Going through the loop once, Going through the loop twice, and Going through the loop “k” times. The “Bypass the loop” case is intended to verify if the loop can yield results. Going through loop once is to assess its capability to exit the loop. Going through the loop twice aims to evaluate if the loop can continue its iterations. Here we use going through three times, similar to entering “k” times, is primarily used to represent scenarios where the loop may encounter issues when traversed multiple times. This testing method for loops adopts the principles of partition testing.
Therefore, the number of our test cases should be: {[(G6+G7)+G4+G5]^0}*G8+{[(G6+G7)+G4+G5]^1}*G8+{[(G6+G7)+G4+G5]^2}*G8+{[( G6+G7)+G4+G5]^3}*G8=7^0*2+7^1*2+7^2*2+7^3*2=2+14+98+686=800.
Hence, to test all paths of this CFT model, a total of 800 test cases are required. Due to the large quantity, here we will list only the test cases involving going through the loop once.
I-P14-C6-P18-C1-P1-C2-P4-C4-P8-J1-J3-J7-C6-P13-P15-C7-P16-J8-F; I-P14-C6-P18-C1-P1-C2-P4-C4-P8-J1-J3-J7-C6-P13-P15-C7-P17-J8-F; I-P14-C6-P18-C1-P1-C2-P4-C4-P9-J1-J3-J7-C6-P13-P15-C7-P16-J8-F; I-P14-C6-P18-C1-P1-C2-P4-C4-P9-J1-J3-J7-C6-P13-P15-C7-P17-J8-F; I-P14-C6-P18-C1-P1-C2-P5-C5-P10-J2-J3-J7-C6-P13-P15-C7-P16-J8-F; I-P14-C6-P18-C1-P1-C2-P5-C5-P10-J2-J3-J7-C6-P13-P15-C7-P17-J8-F; I-P14-C6-P18-C1-P1-C2-P5-C5-P11-J2-J3-J7-C6-P13-P15-C7-P16-J8-F; I-P14-C6-P18-C1-P1-C2-P5-C5-P11-J2-J3-J7-C6-P13-P15-C7-P17-J8-F; I-P14-C6-P18-C1-G4 -J7-C6-P13-P15-C7-P16-J8-F;
I-P14-C6-P18-C1-G4 -J7-C6-P13-P15-C7-P17-J8-F;
I-P14-C6-P18-C1-P3-C3-P6-J5-J6 -J7-C6-P13-P15-C7-P16-J8-F; I-P14-C6-P18-C1-P3-C3-P6-J5-J6 -J7-C6-P13-P15-C7-P17-J8-F; I-P14-C6-P18-C1-P3-C3-P7-J5-J6 -J7-C6-P13-P15-C7-P16-J8-F; I-P14-C6-P18-C1-P3-C3-P7-J5-J6 -J7-C6-P13-P15-C7-P17-J8-F;
2.2.4 Test Cases Sensitize
These sensitizing test cases aim to assess the system's behavior under conditions, including testing boundary conditions, exceptional scenarios, or potential issues. These test cases might involve:
Triggering boundary conditions: Testing input boundaries, such as maximum values, minimum values, or invalid inputs, to evaluate how the system handles extreme scenarios.
Exceptional scenario testing: Verifying the system's response to exceptional inputs or scenarios, like inputting invalid data, network interruptions, or other external interferences.
Error path testing: Examining the system's response when errors occur, like incorrect user actions, system failures, or improper processing paths.
Here, we are also conducting test cases sensitization based on paths involving going loop once. Please refer to the appendix for detailed information.
2.2.5 Summary of CFT
In this chapter, we modeled a significantly extensive Control Flow Testing (CFT) model by focusing on specific functionalities within the shopping process of the Amazon shopping application. Constructing test cases for this model revealed that testing even a portion of its functionalities requires a considerable amount of resources. I believe that conducting a CFT on an entire software system containing nested loops is highly inefficient and uneconomical. It might be more efficient and cost-effective to divide a
complex software system into multiple parts and perform targeted CFT on each segment separately.
3. Result and Summary
Through the construction of FSM and CFT models and the creation of test cases for the Amazon shopping application, no issues were detected as all test cases passed successfully. The reason for this ease in achieving passing test results can be attributed to two main factors. Firstly, the software being tested is a production software already installed on mature smartphones that have been released into the market. This suggests that most defects have already been rectified, hence the comprehensive passing of test results is reasonable. Secondly, the functionalities being tested are fundamental features that must be implemented by smartphone operating systems, making these aspects less prone to issues during the maintenance phase. The results also indicate that each function under test aligns with expectations, which is a favorable outcome for the testing process.
After conducting tests using both FSM and CFT methods, we observed that FSM proved more effective in evaluating certain functionalities within the Amazon shopping application. It provided a clearer understanding of the overall behavior by focusing on the transition relationships between states. However, the CFT method was particularly useful in detecting control flows along specific paths, especially when dealing with nested loops. During the actual testing process, we found that relying solely on CFT for comprehensive testing is resource-intensive and uneconomical, as
even testing a fraction of functionalities demanded a significant number of test cases. In conclusion, we recommend combining FSM and CFT methods during software testing, selectively applying them according to specific requirements, in order to conduct targeted examinations of different sections. This approach aims to enhance efficiency and cost-effectiveness in the testing process.
4. Lesson Learned
From this project, we've gained valuable insights into the limitations of FSM (Finite State Machine) and CFT (Control Flow Testing) methodologies in software testing.
While FSM offers a systematic approach to model system behavior by defining states and transitions, it may fall short in representing complex real-world scenarios. FSMs struggle to handle systems with intricate behaviors or those featuring non- deterministic transitions, which limits their effectiveness in capturing nuanced interactions within software.
On the other hand, CFT, although beneficial in examining specific paths within a program, might encounter challenges when dealing with nested loops or extensive control flows. This method's reliance on exhaustive testing along control paths becomes resource-intensive and often impractical, especially in large-scale systems.
Moreover, both FSM and CFT have limitations when applied individually. FSM may overlook certain unexpected behaviors due to its static nature, while CFT may struggle with achieving comprehensive coverage in highly complex software due to the
exponential growth of test cases.
Therefore, in complex projects, employing a comprehensive testing approach is
crucial. Integrating multiple methodologies helps overcome the limitations of individual techniques. By combining FSM, CFT, and potentially other testing methods, we can enhance the efficiency and effectiveness of software testing, ensuring more robust evaluations and better software quality.
5. Appendix
5.1 CFT Test Cases Sensitize
Test What to Buy
Test How to Pay
Test Case#
Steps
Result
Buy vegetables and fruits from Amazon Fresh
By card
TC-1
1. Open Amazon shopping application
2. Select Groceries
3. Choose Amazon Fresh
4. Add a vegetable or fruit into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with card
8. Finish
no defect
Buy vegetables and fruits from Amazon Fresh
By PayPal
TC-2
1. Open Amazon shopping application
2. Select Groceries
3. Choose Amazon Fresh
4. Add a vegetable or fruit into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with PayPal
8. Finish
no defect
Buy meats from Amazon Fresh
By card
TC-3
1. Open Amazon shopping application
2. Select Groceries
3. Choose Amazon Fresh
4. Add a meat product into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with card
8. Finish
no defect
Buy meats from Amazon Fresh
By PayPal
TC-4
1. Open Amazon shopping application
2. Select Groceries
3. Choose Amazon Fresh
4. Add a meat product into cart
5. Choose no more things to buy
6. Click Pay
no defect
7. Choose pay with card 8. Finish
Buy vegetables and fruits from Whole Food Market
By card
TC-5
1. Open Amazon shopping application
2. Select Groceries
3. Choose Whole Food Market
4. Add a vegetable or fruit into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with card
8. Finish
no defect
Buy vegetables and fruits from Whole Food Market
By PayPal
TC-6
1. Open Amazon shopping application
2. Select Groceries
3. Choose Whole Food Market
4. Add a vegetable or fruit into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with PayPal
8. Finish
no defect
Buy meats from Whole Food Market
By card
TC-7
1. Open Amazon shopping application
2. Select Groceries
3. Choose Whole Food Market
4. Add a meat product into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with card
8. Finish
no defect
Buy meats from Whole Food Market
By PayPal
TC-8
1. Open Amazon shopping application
2. Select Groceries
3. Choose Whole Food Market
4. Add a meat product into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with PayPal
8. Finish
no defect
Buy medicine from Medical Care
By card
TC-9
1. Open Amazon shopping application
2. Select Medical Care
3. Add a medicine into cart
4. Choose no more things to buy
5. Click Pay
no defect
6. Choose pay with card 7. Finish
Buy medicine from Medical Care
By PayPal
TC-10
1. Open Amazon shopping application
2. Select Medical Care
3. Add a medicine into cart
4. Choose no more things to buy
5. Click Pay
6. Choose pay with PayPal
7. Finish
no defect
Buy clothes for men
By card
TC-11
1. Open Amazon shopping application
2. Select Clothing
3. Choose for men
4. Add a cloth into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with card
8. Finish
no defect
Buy clothes for men
By PayPal
TC-12
1. Open Amazon shopping application
2. Select Clothing
3. Choose for men
4. Add a cloth into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with PayPal
8. Finish
no defect
Buy clothes for women
By card
TC-13
1. Open Amazon shopping application
2. Select Clothing
3. Choose for women
4. Add a cloth into cart
5. Choose no more things to buy
6. Click Pay
7. Choose pay with card
8. Finish
no defect
Buy clothes for women
By PayPal
TC-14
1. Open Amazon shopping application
2. Select Clothing
3. Choose for women
4. Add a cloth into cart
5. Choose no more things to buy
6. Click Pay
no defect
7. Choose pay with PayPal 8. Finish
5.2 UBST for Amazon
5.2.1 UBST Testing Objectives and Strategy
The many sub-domains for large software systems like Amazon may include many different operations for each sub-domain. In such situations, the equivalence relation as represented by partition testing described earlier in this chapter represents a uniform sampling of one test point from each sub-domain. However, if operations associated with one particular sub-domain are used more often than others, each underlying defect related to this sub-domain is also more likely to cause more problems to users than problems associated with other sub-domains.
Usage-Based Statistical Testing (UBST) with Musa's Operational Profiles is an approach that involves creating test cases based on the observed or anticipated usage patterns of a system. In the context of Amazon, incorporating UBST with Musa's Operational Profiles can be a powerful strategy for testing scenarios that mimic real- world usage.
5.2.2 Operational Profiles
There are three generic methods for information gathering and OP construction: actually measurement of usage at customer installations, survey of target customers, and usage estimation based on expert opinions. For existing software products, the most precise way to obtain customer usage scenarios and associated probabilities is
through actual measurement of the in-field operations at the customer installations. In this experiment, we chose to gather data through a survey with a small sample size, as an alternative to surveying a large number of customers. The goal is to exercise
our testing capabilities by testing Amazon through a representative subset of users.
User Type
Weight
student
worker
retired
Table 2
In this survey, we interviewed a total of two students, two workers, and one retired individual, representing the proportions in our user demographics. However, the sample size is extremely small and lacks significance. We are cautiously attempting to use this as input for Operational Profiles in our Usage-Based Statistical Testing (UBST) experiments.
Function
Usage profile by user type
Overall profile
student
worker
retired
Main page
Shopping cart page
Category page
Personal page
Product page
Payment page
Table 3
Here, we employ a timing method, starting the timer when the participant opens the homepage during the survey. We record the time spent on each page and, in the end, calculate the usage proportion for each page by dividing the total time spent by the participant. The participants were asked to use the Amazon Shopping application in screen recording mode to make purchases of several items. The screen recordings captured the duration each participant spent on respective screens/pages. The time recorded here is accurate down to the minute for each interaction.
We can draw a simple conclusion: during a basic shopping process, the majority of individuals tend to spend the most time on product pages, delving into a more detailed understanding of the items they are considering for purchase. Conversely, during a straightforward purchasing process, people often do not allocate much time to the personal information page.
5.2.3 Summary of UBST
Although the Operational Profiles (op) from this experiment lack significant reference value, it is still evident from the data that each section of the program carries different weights. Usage-Based Statistical Testing (UBST) offers numerous benefits, one of which is providing developers and maintainers with a clear understanding of the weights associated with different parts of the program. This, in turn, enables better resource allocation for testing and maintenance purposes. Also, Introducing new products by implementing highly used features quickly to capture market share and then gradually adding lesser-used features in subsequent releases or product updates. Besides, through UBST, the develop can have better communications with customers and better customer relations.
Like most test activities, the actual testing is typically performed late in the overall product development process, and the model construction could be and should be started much earlier. Usage-based statistical testing actually pushes both these activities to the extremes at both ends as compared with most other testing techniques. On the one hand, the operational profiles (OPs) need to be constructed with customer and user input. It makes more sense to start them right at the requirement analysis phase, or even earlier, in the product planning and market assessment phase. On the other hand, testing according to customer usage scenarios and frequencies captured in OPs cannot be performed until most of the product components have been implemented. Therefore, such OP-based testing could only be performed in the very late sub-phases of testing, such as in the integration, system, or
acceptance testing sub-phases.
In contrast, FSMs strike a balance between simplicity and detail, offering a
moderate level of expressiveness. However, their intricate models might contain excessive details requiring abstraction. On the other hand, CFT provides simpler models that might lack the expressiveness needed for complex program behaviors. While CFT and FSMs offer simplicity or detailed models, respectively, they both have limitations in expressiveness and may necessitate abstraction or simplification in certain contexts.
In summary, UBST excels in understanding segment weights, optimizing resources, and refining product strategies, yet may be constrained by reliance on Operational Profiles (OP) data. FSMs offer balanced models but can be detail-heavy, while CFT provides simplicity at the cost of expressiveness, necessitating careful consideration based on the testing requirements and complexity of the software under scrutiny.
