Use HAST to determine the robustness of a system

Testing is now an accepted and vital part of any software development project. In fact, accurate testing is essential in order for the software to be implemented successfully.

Automated testing is recognized as a cost-efficient way to increase application reliability, while reducing the time and cost of software quality programs. Automated tests can be designed to replicate any user/application activity in a high-speed and easily replicated environment.

However, not all tests should, or can be, automated. A software tool cannot automate certain processes that require human intervention, such as loading special paper into a printer, image quality, etc. Manual tests are those that require human intervention to perform a test procedure.

Healthcare product testing requires more of domain knowledge, and testing in general is critical in the life cycle stage of any software development activity. In today's situation, healthcare software is gaining more importance and is becoming more complex with the incorporation of new features and design based on customer feedback and market needs.

During the ongoing product development phases, it is also necessary to focus on the CTQ (Critical To Quality) of the system, i.e. performance and reliability parameters that define the system availability to the end user. Testing for robustness, performance and reliability are the critical focus areas and are very much necessary when developing healthcare software.

In the new product development lifecycle, testing for robustness, performance and reliability takes a backseat in the early stages of the product development and is rigorous during the later and final stages of the testing cycle. To avoid repeatedly crashing and system non-availability issues during later phases, you should catch these issues during early phases of development. This can be achieved using an automated test framework called HAST -- Highly Accelerated Stress Testing.

HAST is an automated framework of running cycling scenarios (derived from system-level use cases) of an Application Under Test (AUT) on an independent test machine to derive performance and reliability growth during the product development cycle. HAST is an essential part of any product development cycle that aims to deliver a reliable product at the end. It is run continuously over a longer period of time to determine the robustness of the system.

HAST is meant solely for robustness and reliability testing. The whole idea is to have the system perform use case operations repeatedly until it breaks down. It is more significant in hardware systems than purely software systems. In healthcare scanner systems, HAST scans and generates images repeatedly to ensure the hardware/software/firmware interaction can withstand the continuous scanning.

The infrastructure for building HAST is simply the ability to execute the functionality repeatedly without manual intervention.

• For UI based systems, it is achieved by GUI tools such as Quick Test Pro without any code change.
• For non-UI subsystems, it is driven by script providing hooks (interface) within the code to achieve automation of the use cases.

HAST helps identify reliability issues early in the development phase. Later, when the application is stable, helps in defining the Mean Time Between Crashes (MTBC) for the systems.

Needs for HAST

• To beat the system by simulating simultaneous/concurrent operations on the system.
• To detect critical system issues such memory issues, response time, application hangs and crashes, recovery issues, unknown exceptions, deadlocks, and performance issues early in the product development phase, which are impossible to find in manual testing.
• To simulate workflow scenarios, increase load and simulate concurrency on the system.
• To derive performance and reliability growth of the system under load conditions.

Building HAST
Identify a simple use case/workflow scenario for the AUT and develop automated use case for the same using a test script/test tool. Start the application and run the use case for "N" number of iterations and parse data from log files.

HAST flowchart

Test Machine

Infrastructure for HAST setup

• Hardware and software -- Test System and AUT
• Create HAST scripts -- QTP test scripting or any other Scripting technique.
• Parser scripts-- Parse log files to extract perf & reli data and publish on dashboard.
• Application logs -- Availability of structured logging mechanism in recording transactions, delays, time outs, etc.

Benefits of HAST

• It helps determine the stability of the application.
• It covers complete functional use case coverage.
• It helps find critical issues during the early stages of product development.
• Any user familiar with the operation can operate HAST.
• It helps determine the performance of the system.
• It helps derive the performance and reliability growth of the system.
• As a framework, it can be deployed on any of the domain applications (platforms, Web-based, etc.)

Simple use case of healthcare application for HAST operation

1. Start AUT
2. Create New Examination for a patient
3. Prepare for scanning
4. Start the scan procedure
5. Select the scanned exam and perform import and export operation to network & archive hosts (DVD, PACS).
6. Open viewer and perform viewing operations (change layout, zoom, pan, window leveling)
7. Perform post-processing operations
8. Print the images
9. Clean up the job queue for finished jobs
10. Format media
11. Delete the patient from database
12. Repeat steps 2-12