How to write Performance Requirements with Example

          The only way in which systems will meet their performance targets is for them to be specified clearly and unambiguously. It is a simple fact that if performance is not a stated criterion of the system requirements then the system designers will generally not consider performance issues. While loose or incorrectly defined performance specifications can lead to disputes between clients and suppliers. In many cases performance requirements are never ridged as system that does not fully meet its defined performance requirements may still be released as other consideration such as time to market.

In order to assess the performance of a system the following must be clearly specified:
• Response Time
• Workload
• Scalability
• Platform

Response Time

              In some cases the system response times are clearly identified as part of a business case for example a criminals fingerprint need to be identified while the criminal is still in custody (less than an hour). In some cases the response time will be dictated by legal requirements although this is rare.

For general applications asking users to what is an acceptable response time is like asking people how much salary they require!! The whole process is simply a process of negotiation.

The general advice on response time from Jakon Nielsen book on Usability is:

0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.

1.0 second is about the limit for the user’s flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.

10 seconds is about the limit for keeping the user’s attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.

For systems that have to support significant numbers of users the cost of response times delays can actually be measured in monetary terms and therefore can form part of trade-off studies between different architectures providing different levels of performance.

REMEMBER USERS HATE VARIATION IN RESPONSE TIMES

Whatever is chosen must be measurable in the real system. Care must be taken to ensure that the performance measurement is unambiguous, concise and completely defined. Response time specification should include the following information:

Measurement Points: The points at which you want the response time measured needs to be included. For example are you interested in response time at the data centre or from a branch network?

.

What is included: Define what is included in the measurement. For example does the measure for a web page include the browser render time or just the delivery time to the browser?

What is excluded: Ensure calls to 3rd parties beyond the control of the system developer are defined. This is particularly important when defining contractual response time requirements, as suppliers can only be responsible for areas they can influence.

Statistic Type: The statistic type needs to be defined. For example 95% of all response time should be less than 8.5 second.

Measurement Period: Define the period over which to measure the response time. This is particularly important where workload varies over the day.

Platform: Should it not be possible to test on the real production hardware any allowance for testing on test hardware needs to be stated.

Error Rate: Define the acceptable error rate allowed during the measurement of the response times. Some systems may produce errors under high workloads and therefore the acceptable error rate need to be defined.

Finally once you have defined the response times have them reviewed by somebody else to check if the definition is indeed clear and unambiguous. Remember also to state the workload at which the response time are to be met.

Workload

Again the business case or existing process should be the start of the workload definition. However, it is not enough to state that “the system should be capable of supporting 80,000 customers” or “the system should be able to support 4 pages/sec”. These statements are often good metrics at a high level management level but do not define the work that the system must support. This is particularly important as the mix of transaction affects the performance. For example a DB system may easily handle 10,000 read transaction per hour but only 3,000 update transactions per hour.

The most likely transactions to specify are the user initiated transactions but care must be taken to consider all the users of the system and the batch processes.

For example, a system may have external customers, internal staff providing data entry and batch processes such as backups.If the backup is not completed overnight then it may seriously disrupt the performance experienced by the users the next day.

The workload is often described as the scenarios that the users are likely to execute. The table below is an example that shows for a user scenario how many requests per day, what pages a user executes and the think time between pages. You will notice that a percentage is included before the exit page, this is to specify that only a percentage of users will view that page.

Scenario	Daily Total	Pages	Think Time
View Balance	2000	Login, Portal, 50% Exit	15 secs
Bill Pay	45	Login, Banking, Payment, Confirm, 75% Exit	20 secs
..	..	..	..

When completing a workload specification a check must be made to ensure that all relevant functions have been covered. This includes not just the obvious user workloads but special cases such as management requests, backups and error scenarios/handling. Once all loads have been considered, infrequent or inappropriate workloads can be eliminated.Inappropriate workloads might include error scenarios where it is understood that errors will be very infrequent although it should be appreciated that an appreciable workload can result from the need to provide adequate error and/or security logging.

The workload should be specified up to the date that you wish current hardware to support without upgrade.

Workload Profile

The performance of the system is dependent on how the load is delivered to the system. For example it is easier to achieve faster response times for a system that receive an regular arrival of work throughout the day compared to one that receive burst of traffic. Therefore it is important that the workload profile is defined

The arrival rate into the system is rarely going to be constant throughout the day or the week. The figure below shows the arrival rate for requests to a website.

As can be seen in the figure the workload peaks around lunchtime and late evening but activity is very quite during the night.The above example shows how workload varies over a 24 hour period but it may be more important to show how workload varies over a month for example batch or over an hour for example end of day share trading.

Defining a Peak Workload

             Many organizations’ workload may suffer a rare or unexpected increase in the workload of the system and this is often referred to as a “peak” workload. The decision facing system developers is whether to design a system to cope with a peak workload. The answer to this will depend on the consequence of failing. For example certain organizations design for a peak. The NY stock exchange in 1997 could process 3 billion transactions in a day while the daily average is less than 1.2 billion (Times, August 14 1997). This is because of the need to execute trades the same day and to maintain an accurate record of the state of the market.

When defining the workload for a new venture and little or no existing workload data exists and the system is to be developed by a supplier the temptation is to specify a high peak workload. This is a valid but often an expensive approach which may still not protect you from high demand as it can still be higher than you specify! An alternative solution is to specify the workload as your business model and analysis predicts it will be. Next, have a detailed specification of the scalability of the system to ensure that the supplier develops a system that is scalable quickly. Specify requirements for detecting and processing overload to ensure flood control mechanisms are in place to avoid the system crashing under intensive loads. Test that the system is scalable by renting in additional equipment or using vendors test environments. Finally ensure the operational staff have a procedure for increasing the scalability of the site and contracts are in place for additional hardware/bandwidth etc.

Scalability

In one respect scalability is simply specified as the increase in the system’s workload that the system should be able to process. The scalability required is often driven by the lifespan and the maturity of the system. For example, a new (and hence immature) system could suffer an unexpected growth in popularity and suffer from a significant increase in workload as it becomes popular with new users. More mature systems which represent improvements on older systems are likely to have more accurately defined workloads and thus be less likely to suffer in this respect

Remember to specify that the response time requirements should still be meet as the workload scales

A problem with scalability specification is that it may not be economically viable to test the scalability, as it often requires additional hardware. Therefore an alternative is either to rent in the additional hardware for the purpose of the tests or to use an extrapolation technique such as a simulation model.

Platform Considerations

A platform is defined as the underlying hardware and software (operating system and software utilities) which will house the system.It is not always the case that the designer will be given a “green field” choice of what platform on which to house the system.In some cases the customer may dictate this choice or there may be internal reasons (product strategy perhaps) that will constrain the designer’s freedom. It may also be the case that the system will require various generic products to be used in which case the performance of these must also be specified.

Consideration must be given as to whether the hardware will be used exclusively for the system or whether it must be shared with some other process.A new system might have to compete with an old system for the processor and its resources.The processor and resource requirements of the old system must be specified so the amount of free space and performance can be assessed.

When the customer specifies the platform for the system it is not only important that the project engineer knows what has been specified but that they also understand its capability in terms of performance.

When part of the platform consists of external resources such as connections to external databases or banking systems then the response times of these external resources must also be specified

Contractual Considerations

For any project where the system is developed by a supplier, considerable thought needs to be given to the wording of performance goals. The problem is that the customer will lose potential leverage if performance requirements are specified after the ink has dried. While the supplier will be wary of signing upto performance requirements before they have undertaken enough analysis to access how reasonable the performance requirements can be met. Problems are particularly acute for new ventures where no existing workload date is available to estimate demand accurately and therefore assumptions and projections can be challenged. This should not be used as an excuse for neither party to ignore specification of performance requirements.

Make every effort to ensure that during the contractual negotiations the performance requirements are discussed and negotated. This allows trade-offs to be made by both customer and supplier. A customer who is unsure of the system workload may specify high or even excessive volumes. The supplier should analyze these and explain how this is driving the cost of the system allowing the customer to decide if they want to pay for the comfort of a high workload margin.

Where time does not exist to fully explore the performance requirements then a task should be created to define these in the analysis phase of the project. If this is the case a clause should be added to the contract stating that this work will be undertaken and when.

An Example

This example is from an on-line bank and the supplier of the front end software. It was decided that the approach taken was that the software supplier would be responsible for a level of software performance within the test environment that closely resembled the production environment.

Response Time

The purpose of this section of the document is to outline the Software Performance Goals for Product X.These are the goals that Supplier Y and Customer Z minimally require to see in the Performance Test environment before putting an application into Production.These are not Data Centre SLA measures.

Software:

Supplier Y warrants that in supporting 300,000 customers it shall ensure that performance shall not fall below the following level:

95% of ALL visible pages for “normal” customers respond in 8 seconds or less, including infrastructure, excluding backends.

Measurement Points:

The response times will be measured using HP LoadRunner (or similar tool) located behind the firewall and in front of the web servers. The timer will measure the time from the request for a page to when the last bit required to render the page is returned. Backend response times will be measured using the application server log files.

Definitions

Backends are third party products and information providers such as Reuters share quotes not supplied by Supplier Y.

For the purpose of measuring the response time the performance tests should not exceed 60% CPU utilization during the busy hour

Visible Page shall mean a web page visible (non-blank) as seen by a customer.All redirect page times will be included in the response time of the page it redirects to

The test workload will be based on a normal business day, as defined in the workload section below, executed in the release acceptance test environment.

If the Supplier Y Software fails the response time criteria as set out above, then Supplier Y shall be liable to pay to Customer Z of the following amounts (depending on the level of failure):

Software Response Times for failure to meet 95% of all pages in less than 8 seconds

Score	0	1	2	3	4	5
Response Time expressed in seconds	8	8.01-9	9.01-10	10.01-11	11.01-14	14.01 and Over
Cumulative Total Compensation	£0	£8,000	£15,000	£45,000	£90,000	£180,000

Workload

The software must support 80,000 customers which will on a busy day generate 4500 customer interactions as outlined in the table below:

Ref No	Description	Pages	Daily Total
1	Portal	Login, Portal 50% Exit	2500
2	Transaction History (Statement)	Login, Portal, 50%Balances, Statement, 70% D Stat,50% Exit	500
3	Bank Viewer	Login, Portal, 50%Balances, Statement, 70% D Stat, balances,charges,balances,so list, dd list, Int Trf, Bal, portal,50% Exit	700
4	News Reader	Login, Portal, 50% Bal, Stat, 70% D Stat, Intra Day, News, Portal, News, Portal, BV add, 50% Exit	250
5	Portfolio edit	Login, Portal, Portfolio View, Portal Pref, Portfolio View, Add Shares, View Share, Share Quantity,Portfolio View, 50% Exit	100
6	OO Payment	Login, Portal, 50% Balances, Statement, 70% D Stat, OO List, New OO, New Payee, Confirm Payee, OO Details, Confirm OO, OO List, Payee List, Click Payee, Delete Payee, confirm Payee, 50% exit	245
7	Assign MM Category	Login, Portal, 50% Balances, Statement, 70% D Stats, GoMM, CreateCategoty, ConfirmCatagory, Balances, Statement, Click Item, Statement, GoMM, Click Report, Report, 50% Exit	105
8	Detail Bank Browser	Login, Portal, 50%Balances, Statement, 70% D Stats, In Progress, Portal, Alerts, View Alert, Portal, 50% Exit	100
		Total	4500

The percentage in front of certain page numbers represents the probability of that page being requested. The think time between all pages is 20 seconds with the exception of the think time between the login and portal pages which is 15 seconds. The profile for work arriving is shown in the figure below:

Scalability

Supplier Y warrants that the banking Software shall be capable of supporting at least 300,000 customers when implemented into a suitable production environment.