Load tests are important for applications that will have a lot of people using them, such as a Web site like Amazon where millions of hits are generated in an hour or so on any given day.
It's important to have adequate setup before executing the load test for an application. It's especially important to decide on the metrics than you need to monitor for the load tests.
Take a scenario where you are testing an application for performance-related issues. You must monitor the four main things: processor, memory, disk and network. Once the load test completes, you should have an overview of the application performance.
Now you can move to next level of monitoring and so on until you are satisfied with the investigation. For example, if after the first load test you find that the page faults occur at a level higher than the accepted threshold -- indicating memory problems -- then you may want to investigate further the cause of the page fault, such as if it is a hard page fault or soft page fault. Therefore, you may want to monitor the transition faults, pages input and pages output to answer your doubts.
Keep in mind that further investigation requires knowledge of the application, server, network, and what the counters and metrics mean. It's also helpful if you have a good understanding of the internal working of the components.
So, keeping a good watch on the metrics during your load tests will build your confidence in the application.
