Testing is an important aspect of quality control. Being able to get a thorough understanding intowhat the role and benefits of testing are and important first step.
What is the Purpose of Software Testing? The purposes of software testing is to ensuree that products and services are developed to the standards expected and are not deficient or defective. That is, they don’t have any bugs, issues or errors which can be costly to repair after release.
This article introduces the key concepts and terminology, giving you an overall grounding on software testing.
What are the objectives of testing?
Testing allows products to be checked against the requirements specified to ensure that the product meets those requirements.
If during the testing of a product, it’s revealed that the product doesn’t meet the requirements set, then the product has not only failed the test but failed to meet the requirements.
If the testing of the product meets the requirements then the product has passed the test. Failure to meet the requirements could lead to the product not being fit for its intended market.
One of the aspects of testing is checking to see if the requirements have been achieved.
Consider the following:
A screw manufacturer develops different types of screws for the hardware industry. One of their best sellers is a 45mm screw which is popular with the building industry. The screw is designed to ensure adequate penetration of brick walls without damaging the brick strength.
The length of 45mm is part of the requirements for developing these screws. As even the slightest change in the length of the screw could cause damage to the brick work. During the production of these screws, the final checks involve testing the screw to ensure that it is of the correct length as per the requirements.
This is done by running a special laser light over the screw which is used to calculate the length. Any screws that are not 45mm fail this test and are scrapped. Those that are exactly 45mm pass this test and are packaged for delivery to the many hardware stores that sell this screw.
The above example highlights the importance of testing the requirement of a 45mm length. As any screws failing this requirement could cause damage not only to the brickwork but the screw manufacturer’s image.
Why is testing necessary?
Whilst the main aim of testing is to ensure that the requirements specified have been met. Testing can give much more than this.
Consider again a screw manufacturer and their brand of 45mm screws, the requirements state that the screws are made of steel and are exactly 45mm long. Adhering to both these requirements in theory, leads to screws that are ready for sale.
Testing can take this a step further by ensuring that the screws are of the quality expected. The last thing the screw manufacturer wants to do is sell screws that fail whilst being used.
They could be part of a batch made with poor quality steel or a batch where the hardening process hasn’t been used effectively enough. There’s nothing worse for the screw manufacturer to end up being sued by their customers for substandard screws.
Testing can help increase the quality of products produced, reducing the chances of being sued for product failure later on.
As well as checking requirements, testing can be used to check the quality of a product.
It’s important to realise that as humans there is a susceptibility to make mistakes at one point or another. Even the brightest, bravest and boldest amongst us can still succumb to making a mistake or two.
Throw in some external pressures such as time constraints, budget headaches, limited resources and the ugliest of them all, politics, the chances that mistakes will occur will dramatically increase.
Testing can help in isolating these mistakes and revealing them well before any real damage occurs. So if there’s an error made during ordering and the wrong quality steel is ordered, testing can be used to alert the organisation that the quality of steel used is suspect. Allowing the screw manufacturer to stop these screws from being sold to their customers.
Errors, Faults and Failure
When a mistake does occur, it will lead to a situation that wasn’t at all expected that is, it is incorrect and this is known as an error.
For example, a new central heating system is installed but the outlet for the waste gases (flue) from the boiler is slightly obscured by a supporting beam from a canopy, as shown in Figure 4 ‘Incorrect boiler installation’ on the next page.
Whilst the installation of the central heating system is correct, the engineer hasn’t left enough room from the flue to the supporting beam. This is an ERROR on the part of the engineer.
The whole central heating is switched off, to prevent further flooding. This is known as the FAILURE of the system, as it has to be switched off and can no longer provide heat around the house, as show in Figure 5 ‘Error, faults and failures’ below.
So why did this error occur?
Maybe the engineer was not competent enough to realise that a certain amount of clearance was required for the flue, so that hot waste gases could be released.
Maybe the engineer was in a rush to complete the job, as they also had other jobs to do that day.
Or maybe the boiler was not appropriate and the seals in the boiler were already faulty.
Whatever the reason for the error that caused the fault and ultimately the failure, the error could have been avoided simply by adopting a series of tests.
The engineer could have assessed the level of gas emissions which could have given an indication as to whether the system was running as expected.
The emissions data could have been compared to what would have been expected for such an installation, with any deviation from the expected emission levels pointing to an increased possibility of a failure of the central heating system when it was switched on.
A simple emissions test as such could have saved on a whole host of problems that were caused by the boiler system overheating.
In conclusion the engineer, in this example, overlooked the importance of testing and as a consequence caused the system to fail.
Summary
• Testing is used to check if the requirements have been met.
• Testing can be used to check the quality of a product.
• Errors are a result of mistakes.
• Faults are caused by errors.
• Failure is caused by faults.
Why Testing is needed?
The main reason why testing is so important is to reduce the impact of any losses that could arise if the product fails to meet certain requirements.
In the ‘Quality Mission’ section, the types of losses that organisations could experience were highlighted.
These losses can be substantially reduced by utilising quality improvement methods such as testing.
In this section the potential losses are examined again, along with examples and an insight into how testing could have reduced the losses.
1. Loss of revenue and ultimately profit.
Organisations are under intense pressure by their owners and shareholders to consistently produce more and more profits. Any reduction in profit will simply not be tolerated. To increase profits, an organisation must increase its revenue and reduce its costs.
If a product fails to meet expectations of the market then this product will not produce revenue as anticipated and the costs of remedying will increase, leading to a drop in potential profit.
The further in the products lifecycle that a defect is detected, the higher the cost of rectifying it. So whilst a defect detected during the development of the product may be rectified at little cost.
Defects later such as during the products use, will cost much more to rectify as they will involve expensive product recalls.
LOSS: Poor car sales due to unreliable and poor quality vehicles.
Many British car manufacturers have fallen by the wayside because they have failed to meet the expectations of the buying public, to whom quality and reliability have become an important factor in their purchasing decisions.
Poorly designed and built cars have not sold in the numbers required for profitability. Leaving little or no money available to invest into quality improvement and newer models.
REMEDY: Better testing of the vehicles could have increased reliability. Further testing of whether the vehicle designs were acceptable by the buying public could have helped develop profitable cars that people really wanted.
LOSS: Costly product recalls of faulty products.
Many manufactures have fallen foul of product recalls which have severely dented profitability. A chocolate manufacturer incurred costs in excess of $30 million having to recall millions of chocolate bars due to a fault in their production.
A laptop manufacturer found that the batteries they used could overheat and explode. The recall involved several million laptops.
A major car manufacturer had to spend over $2 billion to recall vehicles with suspect tyres that allegedly had the tread peel from the tyres causing blow-outs in hot conditions or during high-speed driving. There had been many deaths reported due to the alleged defect in the tyres.
REMEDY: More thorough testing could have isolated these problems in the products development stage, where the cost of rectifying any defects would have been marginal compared to the cost of having to recall millions of products later.
2. Losses due to litigation.
Organisations are all too aware of the implications of being sued should their product fail to meet expectations. The costs of fighting a law suit can be very demanding even for the largest of corporates and this is something that must be avoided at all costs.
LOSS: Costly compensation and legal costs incurred from poor product design.
In the USA a large motor manufacturer was subject to huge fines, legal costs and compensation payments due to the increased possibility of death to the vehicle occupants from a side impact.
The positioning of the fuel tank coupled with minimal safety testing increased the chances of an explosion during a side impact. As a result there were several incidents of explosions from side impacts killing and maiming the occupants of the vehicle.
REMEDY: By testing the safety of the fuel tanks positioning, the vehicle manufacturer could have realised that there was a safety issue. Resulting in redesigning the vehicle to incorporate the fuel tank in another part of the vehicle, where the fuel tanks safety would not have been compromised.
3. Loss of reputation, image and/or brand.
In an era when the media is king, anything that can create news irrespective of it being bad or good, is marketable. Organisations spend millions on Public Relations (PR) to ensure that their organisation is always seen in a good light.
LOSS: Market lead lost to rivals due to product failures.
The British in the 1950’s led the world in research in developing a commercial jet airliner. The British through Frank Whittle had developed the jet engine which went onto revolutionise air travel.
Frank Whittle’s work on developing the jet engine was eventually realised by the British government as an important step towards development of the next generation of airplanes, which could revolutionise the commercial travel markets and create thousands of jobs in the UK as a result of building large airliners.
UK plane maker De Havilland began designing and developing a commercial jet airliner. They had experience of designing and building commercial propeller powered planes but this was their first foray into developing jet powered commercial planes.
They were under intense pressure to develop the jet airliner, this pressure wasn’t merely just from other airplane manufacturers (who were still far behind on the jet technology but were still considered a threat).
The government was also putting pressure on De Havilland as this development would help the UK create jobs and increase Britain’s world prominence which had diminished since the end of the second world war.
On the 2nd May 1952 the first Comet entered service for the British Overseas Airways Corporation (predecessor of British Airways). The Comet timetable included flights from London to Beirut, Khartoum, Livingston, Johannesburg, Entebbe and Rome.
One year after entering commercial service, a Canadian Pacific Airline Comet crashed on take-off from Calcutta. On the 10th January 1954 a BOAC Comet crashed off the Mediterranean island of Elbe killing 35 people on board. A few months later on 8th April 1954 a South African Airways Comet crashed on route from Rome to Johannesburg killing all 14 passengers and seven crew.
The last accident resulted in the Comet being grounded until the cause of the crashes could be determined. The first crash was blamed on pilot error during take-off but the subsequent crashes blame was not so easy to pinpoint.
Eventually it was realised that the Comet had stress fractures from the incorrect shape of the windows and had suffered metal fatigue caused by the expansion and contraction of the fuselage structure during flight.
The latter led to a critical situation where the fuselage skin became so weakened that it could no longer contain the cabin pressure resulting in it ripping open.
By the time these faults had been rectified, the Comet had suffered enormously from the bad press it received in relation to its many accidents. Airlines opted instead to buy planes from American manufacturers such as Boeing and McDonnell Douglas. Who had learnt from the mistakes of the Comet and created much safer planes.
Subsequently these organisations became aviation giants, creating thousands of jobs and millions in sales and profits in the process.
The Comet never recovered from the damage caused by the accidents and today only the British defence forces use the Nimrod which is the current version of the Comet.
REMEDY: Pressurisation testing during the development of the Comet would have proven that there were design flaws. Not only in the shape of the windows but whether the metal skin of the fuselage was thick enough.
The effects of pressurisation could have been tested without actually flying the plane. As Comet engineers later found out when the plane was grounded.
They created a giant water tank around the fuselage of the plane. Which when filled with water would create pressure on the fuselage. By increasing and decreasing the water in the tank they could simulate the same effects of pressurisation and depressurisation that the Comet jet experienced whilst in the air.
During these tests engineers discovered the problems of the fuselage metal fatigue and stress fractures of using square windows. By the time these issues had been rectified and the plane allowed to fly again, Comet had suffered a serious blow to its reputation as a reliable and safe plane.
4. Losses from regulatory bodies.
Many industries and territories are governed by bodies that ensure that organisations adhere to certain standards and regulations. If this criteria is not met then that organisation maybe subject to penalties enforced by that authority or even restricted from operating in a particular territory.
LOSS: Penalties from failing to meet Foods and Drugs Administration regulations.
Schering-Plough, an American pharmaceutical organisation was fined $500 million by the Food and Drug Administration (FDA) for quality control problems at two of its plants.
The FDA had inspected the plants over a number of years and found problems with Quality Assurance and Quality Control.
Further losses were incurred from Schering-Plough having to suspend the production of many of its products to ensure that rectification of the issues highlighted by the FDA could take place.
REMEDY: By adopting higher levels of quality control and quality assurance, the organisation could have avoided the obscene fine.
5. Loss of human life.
The greatest loss of all is human life and any organisation must strive to reduce this loss, ideally eliminate it.
LOSS: Avoidable fatalities caused by improper repairs and testing.
A jumbo jet had damaged its tail during a landing in Tokyo, Japan. The damage didn’t require total grounding of the airplane, to replace the whole tail section. Instead several rivets could be applied to ensure that the tail didn’t disintegrate during flight until the tail could be repaired properly at an appropriate facility.
In the rush to get the plane back into service the maintenance engineer did not follow the recommended procedures of using double line of rivets. Instead he used a single line of rivets to secure the tail section.
Furthermore there was no one available to quality control the standard of the riveting to ensure that it met aircraft manufacturers guidelines. Unfortunately, the plane was allowed to go back into service.
Not long after taking off the rivets started to weaken causing the tail structure to come apart. This caused the plane to lose stability leading to a crash which killed 520 people.
REMEDY: By testing to see if the work had been carried out to the standards recommended by the airplane manufacturer. The accident could have been avoided.
Summary
• Testing can reduce the number of defects in a product leading to higher revenues and profits.
• Testing products can reduce litigation costs by determining product faults during development instead of during a products general sale.
• Testing can strengthen a products image and/or brand.
• Testing products to conform to regulatory regulations can avoid penalties.
• Testing can help avoid unnecessary loss of life from product defects.
Making The Unknown Known By Testing
When a product is not tested, it becomes difficult to verify whether the product will work as it’s expected to. This can lead to all matter of consequences whereby the potential failure of the untested product could not only be costly financially but could also be fatal.
Consider the humble smoke alarm, arguably one of the greatest inventions of the modern era. Many lives have been saved by this simple box of electronics. Giving people the time to escape before a fire can really start to take hold.
Whilst the smoke alarm has been built to many quality standards. The humble smoke alarm is only effective if it actually works. If the battery is flat or the electronics become suspect then the effectiveness of this electronic gizmo is severely impacted.
If the smoke alarm can’t detect the smoke from a fire then no alarm will be sounded and the chances of survival will be drastically reduced.
The majority of smoke alarms have a button which when pressed checks to see if the operation of the smoke alarm is satisfactory.
Normally a loud beep is sounded to indicate that the smoke alarm is working correctly. Ideally this should be checked at least once a week.
Pressing this button once a week, is a test. Without this test, the smoke alarms effectiveness cannot be guaranteed.
The effectiveness of a product cannot be guaranteed if it has not been tested.
Testing Is Verification
Any product which is to be marketed must be tested as its effectiveness needs to be verified. By merely building to the highest standards possible won’t necessarily guarantee that the product will work as expected.
Summary
• The effectiveness of a product cannot be guaranteed if it has not been tested.
• Building to the highest standards doesn’t guarantee that the product will work as expected.
Types of Testing
Black box testing works on the principle that the internal workings of the product under test are not known to the person carrying out the testing.
All the tester needs to know in a Black box test scenario, is what to put in (input) and what is expected at the other end (output). What happens in between is no concern to the black box tester.
If the product under test is an alarm clock. Then the black box testing methodology assumes that the internal workings of the alarm clock are not needed to be known.
The programming used to keep the time synchronised, the electronics used in the circuit board and so on, play no relevance to the black box testing.
Black box testing is putting something in and expecting something out without understanding what happens in between.
White Box Testing
White box testing is the opposite whereby the person carrying out the testing knows the internal workings of the product under test. What happens between the input and output of a test needs to be understood and analysed by the white box tester.
Consider, Formula 1 motor racing, the test driver’s remit is to see whether the car can perform as it should do. As a consequence, the test driver doesn’t need to understand or know the internal workings of the car’s engine.
The Formula 1 test engineers on the other hand, who monitor the engines performance need to know how the engine works in order to interpret the test data that their monitoring equipment produces whilst the test driver is testing the car.
The test driver is carrying out black box testing whilst the test engineers are carrying out white box testing.
White box testing requires an understanding of how the internals of the product work.
White box testing is beyond the scope of this course as it requires substantial knowledge of the internal workings of the product under test. Instead this course will concentrate on Black box testing.
Black Box Testing
In the world of Black box testing, there are generally two areas into which this form of testing can be divided,
• Non-functional testing
• Functional testing
Non-functional testing is where the product behaviour is assessed, whilst Functional testing is where a product is assessed on how it works.
Consider a 4×4 also known as a Sports Utility Vehicle (SUV). Functionally testing the vehicle would look at seeing how it works. Whether the 4×4 system, steering and so on work as they should.
Non Functional testing would look at how the vehicle behaves such as when it is taken off road. Does it behave as a typical off road vehicle or does it behave like a normal car, in that it gets stuck in the mud and can’t climb steep hills.
Both sets of tests are important in proving that the vehicle meets the required standards.
Summary
• Black box testing is putting something in and expecting something out without understanding what happens in between.
• White box testing is putting something in and expecting something out whilst understanding what happens in between.
• Functional testing involves assessing whether the product works as expected.
• Non-functional testing involves assessing whether the product behaves as expected.
What is Functional Testing?
Functional testing involves testing either an element of a product or the complete product. This testing is used to determine whether the product will work as planned.
More importantly functional testing can be used to determine whether a product will work when used by a typical customer.
The product can be logically broken down into various elements, that is its functions. Any product that is marketed today needs to undergo rigorous functional testing to ensure that it’s functions work correctly.
That is they work according to the specifications highlighted at the requirements stage of developing the product.
Alarm Clock
Consider the alarm clock, this must be able to perform certain functions in order to be marketable. Functions such as:
• display the time correctly
• the alarm works as set
• the date is correct
These are some of the functions of the alarm clock, without these functions working correctly it would be difficult to use the alarm clock.
This would therefore mean, the alarm clock would not be fit for the purpose it was designed for.
Cars
If the object under test is a car then for functional testing, some of the items listed below, will need testing, such as,
• switches for turning on headlights,
• seatbelts working in accordance with regulatory standards,
• the steering wheel turning the wheels correctly,
• the car reversing when reverse gear is engaged.
These are some important parts of the car’s functional tests. These items all need to be tested thoroughly to ensure that the car’s functions do work as they were designed and required to do so.
Airplane
If the object under test is an airplane then for functional testing, some of the items listed below, will need testing, such as,
• correct instrumentation readings,
• emergency oxygen releasing mechanisms work,
• external doors lock to leave airtight seals.
These are some important parts of the airplane’s functional tests. These items all need to be tested thoroughly to ensure that the airplane’s functions do work as they were designed and required to.
Detailed Functional Testing
Consider the purchase of an alarm clock (as shown in Figure 6, ‘Alarm Clock’ below). As part of quality checking, the alarm clock must be fit for the purpose it will be bought for.
What type of functional testing would the alarm clock have to undergo?
The standard tests would include checking whether the following functions work as expected,
• date
• time
• alarm
These functional tests would include setting the time and seeing whether they can be set as per the instruction manual.
#TEST 1 Set the time to 12:03.
Press and hold the TIME button,
Press the HOUR button twelve times,
Press the MINUTE button three times,
Release the TIME button
This test will be repeated with many different times to ensure that many different times can be set. Once these tests have been successfully completed then the next functional test could involve setting the alarm.
#TEST 2 Set the alarm time to 12:25.
Press and hold the ALARM button,
Press the HOUR button twelve times,
Press the MINUTE button twenty five times,
Release the alarm button
The final part of this test is to check if the alarm does actually go off at 12:25.
If the time cannot be set properly then there will be no point in testing the alarm, as the alarm functionality requires the time to be working correctly.
There may be other functions available on the alarm clock such as a radio, 24-hour clock display, a light and maybe even voice controlled alarm deactivation. These functions would also require functional testing.
Summary
• Functional testing involves checking that the functions of a product work as expected.
• A products functions are defined during the requirements stage.
Non-Functional Testing analysed
Non-Functional testing is used to assess,
• how a product behaves,
• whether the behaviour is appropriate and
• whether the behaviour is acceptable.
If a 4×4 (SUV) is undergoing non-functional testing, the behaviour being assessed could be the 4×4’s (SUV) off road ability, so how the 4×4 (SUV) behaves when it is driven off road is assessed, to see whether the behaviour is appropriate and whether it is acceptable.
So if the 4×4 (SUV) struggled when used off-road, this would be its behaviour which for a 4×4 (SUV) would not be appropriate and would be unacceptable.
Non-Functional Behaviours
Typical non-functional behaviours include the following:
• Performance
• Load
• Stress
• Usability
• Reliability
• Maintainability
• Configuration
• Security
• Recovery
• Interoperability
Performance Testing
As the name suggests this type of tests are designed to assess how well a product performs. This performance behaviour may be an important component of the product test and critical in terms of marketing that product. Therefore it is essential that the performance behaviour is assessed.
Performance car example
A car has been developed to be able to accelerate to sixty miles per hour in under six seconds. This performance behaviour needs to be tested and verified.
This car’s main marketing assets are its speed and its rapid acceleration ability. The performance testing would be carried out to assess whether the car could actually produce these behaviours to the levels expected.
As a result of performance testing the car engine may require further tuning to ensure that any bottlenecks in fuel delivery and power output are quickly found and rectified. To ensure that the car is capable of reaching the acceleration and speed levels expected.
Performance testing is concerned with finding and eliminating any bottlenecks that could affect performance.
Dry Cleaning example
A dry cleaning organisation wants to introduce a one hour cleaning service. They could simply start by advertising a ‘one hour’ service and then wait to see if they could cope with the demand and see whether they could actually deliver a ‘one hour’ service.
The best course of action for the dry cleaning organisation would be to carry out performance tests. Whereby they could find out if they can actually cope with providing a ‘one hour’ service and whether any potential customers would actually get a ‘one hour’ service.
Engineering organisation example
A small engineering organisation decides on whether it would be cost effective for it to build a new type of bolt. They know that they would need to build several thousand bolts a day to stand any chance of making any profit.
The engineering organisation can run performance tests to see whether they can manufacture enough bolts in a day to make it worthwhile. This saves them from ultimately starting to produce the bolts and then finding out later that they can’t produce enough bolts in a day to make any money.
In conclusion, performance testing is an important indicator not just for fast cars but for organisations to find out whether they can perform as expected.
Performance testing allows organisations to fine tune their products to ensure that they deliver the performance expected.
Load Testing
A product may work fine when it is used singularly, that is by one person but what happens when it is used by many people at the same time?
Load testing is used to determine how a product behaves when varying levels of loads are applied to the product.
Load testing ascertains whether the product can fulfil varying levels of usage.
A Multi-Purpose Vehicle (MPV) (also known as a minivan, minibus, people carrier or multi utility vehicle) undergoes a load test. The main marketing assets of the MPV are:
• it’s ability to carry five people along with their luggage
• by flipping a few buttons, carry a total of seven passengers efficiently
The load testing would be carried out to assess whether the vehicle can efficiently carry five passengers, then six passengers and then finally seven passengers.
If the vehicle is developed to carry seven passengers efficiently but during load testing the vehicle uses an excessive amount of fuel, then the vehicle can be construed as failing the load testing with seven passengers. The vehicle manufacturer could then decide to use a more powerful engine and see whether the fuel consumption figures improve.
This load test could also include the impact of carrying luggage and how this affects the fuel consumption, (as shown in the load test table below).
Additionally, a caravan could be added to the load test, to see whether the MPV can carry seven passengers as well as tow a caravan.
By running a series of load tests as shown below, it can be quickly determined through accurate measurements whether the vehicle can deliver what the marketing promises.
Load testing is an important indicator in proving whether a product behaves accordingly under normal conditions of use.
Stress Testing
How does a product behave when it is pushed to its limits?
Does it get damaged or can it still be used?
Stress testing is used to determine whether a product can be still used after it has been taken to its limit.
These limits then can be used to improve the product to overcome failure or mechanisms put in place to stop failure points being reached.
Consider two fast cars which undergo a stress test to determine whether their main marketing assets, their ability to reach speeds in excess of 150 mph can be justified.
During the stress tests, it is found that when the first vehicle reaches 141mph, it’s tyres explode, causing the vehicle to crash. The second vehicle when reaching 163mph, causes the engine to explode.
These are stresses at which the vehicles fail and from where it is not possible to use these vehicles again as they get damaged.
The manufacturer of these vehicles may decide that it would be far too expensive to develop new tyres and engines to overcome these limitations. Therefore devices that limit speed could be put onto the engines, reducing the maximum speed that could be reached.
The vehicle whose tyres explode past 141mph may be limited to a maximum speed of just 130mph by the devices that limit speed. Whilst the other vehicle whose engine explodes at 163mph, will have a speed limiting device added that inhibits the engine from producing the power that is required to take the vehicle over 150mph.
Stress testing determines the recoverable limits of a product.
In fact many cars today have speed limiting devices fitted, to protect the car from being damaged. Ultimately saving the organisation from accruing losses from litigation and compensation claims.
Another example for the benefits of stress testing could be a chocolate manufacturer, who accepts a very large order for specialist chocolates. After an hour of the production lines starting, two hundred chocolate pieces a minute is reached, things suddenly start to go wrong.
The electric motor on the chocolate mixer machine burns out and smoke starts appearing over the conveyor belt as the excessive heat caused from the belt moving even more quickly over the rollers builds up.
The chocolate manufacturer has exceeded the maximum capacity of what the production lines can produce. If adequate stress testing had been done prior, then the manufacturer would have been able to set limits on the size of the orders that they could accept. Instead of taking on orders which damaged their production facilities.
Reliability
How reliable is a product behaviour when it is used? Reliability testing is used to determine whether the product works reliably over a period of time and/or conditions. These failures then can be used to improve the product to overcome its reliability issues.
Consider a new bus which has to undergo reliability testing to ensure whether the vehicle is reliable enough to be used for cross country journeys across the Australian outback.
During the tests, the vehicle keeps breaking down and has problems primarily with its engine overheating. This unreliability needs to be overcome to make sure the vehicle can perform under these conditions for prolonged periods.
Another example could be an organisation that makes elevators. The organisation has finished building their latest style of elevator. The elevator undergoes a series of reliability tests to ensure that they work as expected.
By running a reliability test of the elevator moving up and down for a set period of time, any of the components used in its manufacture which are suspect, could be rapidly determined in a reliability test and these inadequacies corrected before the elevator is marketed.
Reliability testing checks whether the product is reliable.
A Swiss clock manufacturer has to ensure that their clocks are accurate and hence reliable. Their reputation of not only building accurate but reliable clocks is world renowned.
To ensure that this reputation remains intact, every clock produced is checked against a master clock whose reliability and accuracy has already been assured. Any clocks that deviate from the accuracy and reliability of the master clock are either repaired or discarded.
Security
How secure is a product’s behaviour when it is used? Security testing is used to determine whether the product works securely over a period of time and/or conditions. Any failures can then be used to improve the product to overcome its security deficiencies or mechanisms put in place to reduce the impact of these deficiencies.
Consider several new cars whose marketing stipulates that they are secure, are tested by an independent authority. For the testing, a reformed car thief is used and the time taken for this reformed offender to break into the cars noted.
Four of the cars security systems can be overcome within 30 seconds. The manufacturers are told that their security systems are weak and need to be readdressed before the independent authority can issue a security rating.
Security testing checks the security of a product.
A major bank develops an on-line banking system. Two days before the system is scheduled to go live, one of the directors asks the development team on whether the system passed all the testing. The development team assured the director that all the tests that they ran proved that the system was secure.
Unimpressed by this, the director authorises a third party security specialist organisation to carry out penetration tests. Within minutes, the security testers have opened up several security holes caused by poor development.
The director decides to postpone the launch of the online banking facility and orders corrections to be made to the issues highlighted by the security testers. Once the corrections are complete, the security testers test the system again. This time the testing reveals that the system is secure and the online banking system is ready to be used by the general public.
Inoperability
Many products and services have to operate with other products and services during their life of operation. Interoperability testing involves testing products and services against each other, as typical of their use. Interoperability testing can help determine issues that can be rectified.
Most large commercial airplane manufacturers build their planes without any engines. The engines are chosen by the airline and the final part of the jigsaw involves connecting the engines to the plane. The engines are designed and built by specialist jet engine manufacturers such as Rolls Royce, General Electric and Pratt & Whitney.
As part of the interoperability testing the engines have to be able to work with the airplane in question. Hence if Boeing is building a 747 then an appropriately powered engine is required that can not only produce enough thrust to achieve flight but can also communicate with the 747’s management systems.
Interoperability testing is done to check whether a product works with other products.
Pirate radio stations are the boon of the regulatory authorities because they can cause havoc with emergency air frequencies. Prior to a new radio station gaining a broadcast license, interoperability testing between the frequency allocated for the station and other frequencies is undertaken.
The last thing the authorities would want to do, is to allocate frequencies that stop emergency service’s signals. Leaving the police, ambulance and fire brigade without communications.
Usability
Is the product usable? Organisations don’t want to spend millions developing products that are not usable by their customers. As the customers will simply elect not to buy their products.
Usability testing is not the same as market research, which mainly focuses on how attractive the product is to a customer and how memorable the product logo was with the customer.
Usability testing is purely concerned with how easy to use a product is. The best way to test usability of a product is to give the product to ordinary people to use. These are people who are most likely to buy the particular product under test.
Consider a food manufacturer decides to cash in on the growing ready meal market by developing their own quick cook ready meals. However, for the customer to make the ready meals several steps are involved, which need to be done correctly and hence take time to do.
In terms of usability the ready meals have failed as the customer was required to undergo several steps which were not only tedious, but could affect the quality of the finished ready meals, if they were not adhered to properly.
In the end customers would be put off by the tedium of making this ready meal and as a result sales would have been affected.
Usability testing determines whether the product is easy to use.
Flat pack furniture is another bane of usability. Many items assembly instructions are poorly thought out and the time taken to assemble the items, leaves the customer frustrated. Ideally a usability test can be used to determine whether an item can be assembled quickly by a novice.
Video/DVD remote controls that require a science degree just to operate are another area of usability that requires thorough testing. Especially as the ‘grey’ generation is increasing in population and by making products and services that the elder generations find difficult to use, simply makes it more difficult to sell to that sector of the population.
Maintainability
Many products and services require regular maintenance to ensure that they can carry on working effectively. The ability to maintain a product requires testing to check whether this is indeed possible.
All car manufacturers stipulate service intervals for their vehicles. These service intervals help to prolong the life of the vehicle and ensure that it operates as required. Vehicles that are not serviced regularly start to lose their fuel efficiency and worse still, this can cause engine components to fail prematurely.
The design of the engines needs to include features that allow easy maintainability. If the engine needs an oil change then features must be designed for engine that allow this to be completed quickly. If it takes two hours to change the oil because the engine needs to be unbolted from the chassis and pulled out, then the vehicle’s maintainability is poor.
If the vehicle engine has been designed with a drain plug at the bottom of the engine which can be easily released, allowing the oil to easily drain out of the engine, then it becomes easier to change the engines oil which in turn makes the engine more maintainable.
Maintainability tests ensure that the product can be serviced and/or overhauled effectively.
Likewise commercial airplanes require maintenance after a certain period of flight. Whilst the plane is sitting on the ground it is effectively making no income for the airline, therefore it is vitally important that the plane can be overhauled as quickly and as safely as possible.
Maintainability testing can highlight the best course of action to take during the overhaul. Cutting down on unnecessary time which ultimately costs the airline potential revenue.
Configuration
Many products and services come with the ability to be configured in many different ways. Configuration testing involves checking multiple configurations available and evaluating whether the different configurations work as required. It would be foolhardy to simply test one configuration and then assume that the product could work in all its other configurations.
Typically many of today’s car manufacturers offer their vehicles in many different models. Each of which is a different configuration of the base model. So there may be a saloon as a base model, a hatchback, an open top and an estate version.
The manufacturer will test each of these configurations to ensure that they meet the common objectives highlighted in the requirements.
Multi-Purpose Vehicles (MPV) (also known as a minivan, minibus, people carrier or multi utility vehicle) have ingenious seating arrangements. Allowing the carriage of additional passengers above the normal five passengers.
The seating arrangement can be changed, however as part of a configuration testing approach. The MPV’s various seat configurations need to be tested to see whether they actually work as expected and/or whether they can be used as expected.
There’s no point in building a seven seater vehicle, when it’s very difficult for the passengers to clamber into the rear most seats. Under a configuration test, not only would the seven seat set up be tested but the access to this configuration would also need to be assessed.
Configuration testing determines whether products or services with multiple configurations work in each of their configurations.
Manufacturers that sell food products may also have to conduct configuration testing. A margarine manufacturer not only needs to test whether their product is safe and effective to use as a spread. But whether it can be used in other configurations, such as:
• to fry with
• to baste with
• to mix with (in cakes for example)
These other configurations of margarine may not be acceptable and the manufacturer can therefore highlight this on their packaging. Not only does this provide an indication to the customer on the uses of the margarine but also protects the manufacturer if the margarine is not used within highlighted guidelines.
Recovery
What happens when a product stops working?
Organisations have to seriously consider what would happen if one of their products stopped working. Would they be able to carry on functioning as an organisation or would they cease to operate.
The worst scenario for any organisation is when all their eggs are all in one basket. Leaving them vulnerable to disasters that can cause a complete meltdown of their ability to operate.
This was highlighted during the 9/11 terrorist attacks in New York. When one particular organisation not only lost several hundred employees but was completely wiped out of business. It’s main operations were in one tower with a backup operations centre in the other tower. When both towers collapsed so did the fortunes of the organisation in question.
Many corporates today duplicate their operational centres, data centres and the like. Ensuring that not only can they continue operating should a major event such as a terrorist strike occur but also when more commonly events such as a power failure occur.
Recovery testing is done to calculate whether the product can recover and what steps are needed for the recovery.
The ability of an organisation to recover requires testing and this recovery testing is a major event for many corporates.
Just like an emergency drill for employees which is used to practice evacuation procedures, recovery tests (known as Disaster Recovery tests) can be undertaken to ensure that the organisation can continue operating should an emergency occur.
Low Cost Airlines
Low cost airlines have to be very aware of recovery, especially in light of their tight schedules, which if interrupted can have a severe knock on effect to the airlines whole flight schedules. As such, many airlines have backup plans developed through recovery testing.
They know that a problem with a plane can cause severe disruption so they plan accordingly to bring in additional planes through agreements they’ve setup for this type of incident.
Without any recovery testing they would be in a position where they would have to deal with issues as and when they happened. This could lead to severe delays in getting people on the way to their destination, as they would be at the mercy of market forces in aircraft availability.
Summary
• Performance testing is how well a product performs.
• Load testing determines whether the product can fulfil varying levels of multiple usage.
• Stress testing determines the limits of a product.
• Usability testing determines whether the product is easy to use.
• Reliability testing ascertains whether the product is reliable.
• Maintainability tests ensure that the product can be serviced and/or overhauled effectively.
• Configuration testing determines whether products or services with multiple configurations work in each of their configurations.
• Security testing checks the security of a product.
• Recovery testing is done to calculate whether the product can recover and what steps are needed for the recovery.
• Interoperability testing is done to check whether a product works with other products.
Testing under normal conditions
Normal conditions are those where a product is used under the conditions it was designed for.
An excellent example of a normal testing scenario is the MOT test. The MOT test is a mandatory test that all vehicles over three years old in the United Kingdom must take.
MOT stands for the Ministry of Transport, a government department that regulates the tests.
The MOT test contains about one hundred and fifty checks which a vehicle must undergo, to assess whether the vehicle is road worthy. These one hundred and fifty checks are the requirements of the MOT test.
Remember earlier it was stated that something is needed to test against, that is a set of requirements.
For the MOT, the requirements are tested against to prove that a vehicle is road worthy.
The United Kingdom the MOT test is divided into the following categories:
• Registration plates and Vehicle Identification Number (VIN)
• Steering
• Horn
• Lights
• Bonnet Catch
• Doors
• Vehicle structure
• Seats
• Brakes
• Windscreen
• Mirrors
• Suspension
• Seat belts
• Exhaust system
• Fuel system
• Tyres and road wheels
In each category, several tests are carried out to see if the vehicle meets the requirements for that category. As an example the Registration Plates & Vehicle Identification Number (VIN) category and the Steering category are shown below:
1. Registration Plates and VIN numbers
• The vehicle must have a registration plate.
• The vehicle’s registration plate must be securely fitted.
• The vehicle’s registration plate must be clearly visible and not be faded, obscured or dirty.
• The vehicle’s registration plates lettering must conform to the permissible lettering and spacing.
• The vehicle must have a VIN (Vehicle Identification Number).
• The vehicle’s VIN must conform to the permissible lettering and styles
• The vehicle’s VIN number must be legible
When the MOT test is carried out, the MOT tester will check the items from the MOT check list. Recording whether the item being tested has met the minimum requirements set (shown on next page).
1. Registration Plate / Vehicle Identification Number inspection.
2. Steering inspection
As you can see the steering test is very comprehensive and the level of functionality that the steering wheel and it’s components display can be extensively tested. The reasoning behind such a large number of tests required to be carried out, is that a failure in the steering and/or it’s components can lead to a life threatening situation.
This severity of a function failing determines the level of risk associated with a product. In the MOT test the level of severity attached to the steering wheel failing is higher than that if the Registration Plate screws were rusting and about to fall out.
If a vehicle passes the MOT test, does this prove that the vehicle is reliable?
Unfortunately the MOT test is not an indicator of the reliability of a vehicle and only assesses the minimum requirements needed to prove it’s road worthiness. Hence a vehicle may pass the MOT on one day and the next day it could break down.
To test the reliability of the vehicle, more comprehensive testing that focuses on reliability will need to be undertaken.
Summary
• Testing under normal conditions can verify whether the product works as it’s designed to.
• Testing under normal conditions requires a set of requirements to test against.
• The extent of testing under normal conditions is based on the severity of a failure of a product function/component.
• Reliability cannot be measured by passing a series of functional tests
Testing under abnormal conditions
In the section functional testing, the alarm clock was tested to see if all its functions worked as expected. This included whether the time could be set, the alarm could be set and whether the alarm actually came on for when it was set.
Would this be all the tests to assess that the functionality of the alarm clock was satisfactory and the alarm clock was therefore deemed ready for sale?
Yes, if the assumption was that the alarm clock was going to be used under normal conditions. Where people would use it as per the instructions supplied. So when the alarm was being set, they would follow the instructions supplied by the manual religiously.
However this is never really the case and many people don’t even read the manuals before trying to use a product. Only when they become really stuck do they decide to read the manual.
It would be foolhardy therefore to assume that the tests involving just following the instructions in the manual are sufficient as the only functional tests required to make sure the product is quality assured.
• What happens if the customer presses the date button at the same time as the time button?
• What if they hold down the date, time and alarm buttons together for 60 seconds?
• What happens if they press the alarm button, time and date button all at once?
These conditions whilst not specified in the normal operating instructions of the alarm clock would be classed as abnormal conditions.
Whilst the normal operation of the alarm clock would be classed as normal conditions.
Good and the bad
The input is also known as data and is used to test the functional aspects of a product and this can be divided into three areas.
The first area which is concerned with normal testing is good data.
So if the manual dictates the setting of the time as the following sequence:
1. Press and hold the TIME button,
2. Press the HOUR button,
3. Press the MINUTE button.
4. Release the TIME button
Adhering to this process of setting the time would be classed as testing using good data.
The next two types of inputs are concerned with abnormal conditions.
These include the use of bad data, such as
1. Press and hold the HOUR button,
2. Press the MINUTE button,
3. Press the TIME button.
Or
1. Press and hold the MINUTE button,
2. Press the TIME button,
3. Press the HOUR button.
As you can see there are several sequences in which the buttons TIME, HOUR and MINUTE can be used. Of which one sequence will correctly set the time. The other sequences can be classed as bad data because they are not in the correct order as specified by the instructions listed in the manual.
Is this important?
Absolutely, bad data could corrupt the alarm clock and cause it to fail.
The alarm clock is expecting data (that is input) to be entered in a specific way. It is not expecting bad data to be entered.
To overcome the issues of bad data the manufacturers of the alarm clock may alter the programming of their alarm clock. To ignore the pressing of buttons in different sequences to the one listed in the manufacturer’s instructions.
So if the MINUTE button is pressed the alarm clock programming ignores this until it is pressed when the TIME button is being pressed and held. At this point the alarm clock program will accept input from the MINUTE button.
The Ugly
The final form of input again associated with abnormal conditions is using ugly data. This is data that is out of the ordinary. Whilst the possibilities of the user of the product applying such data maybe remote, it’s consequences need to be assessed and controlled.
It can never be taken for granted that ugly data will never be introduced to a product.
So there’s three types of data: the Good, Bad and the Ugly.
Ugly data has the possibility to damage and even destroy a product. More importantly it could injure the person using the product.
Examples of ugly data used on the alarm clock could be,
• holding a button down for a prolonged period of time,
• pressing and holding all the buttons together at once,
• switching the alarm clock on/off over and over again.
Who’s to say that using ugly data with the alarm clock doesn’t cause it to electrically short and even electrocute someone?
Testing with bad and ugly data is essential in ensuring that a product is safe to use.
Real Life Ugly Data
A set of journalists based in Sweden were driving a newly released car in the north of Sweden. They decided to see how well the car would cope if an elk (moose) ran across its path and they had to swerve violently to avoid it.
Whilst not being a normal driving procedure, the violent swerving acted as a ugly data input. Where it was something that could happen even though the likelihood of it happening was remote.
They drove the vehicle at normal speed and then turned the steering wheel violently as if to avoid an elk running across their path. The vehicle rolled over onto its side.
The journalists were very surprised that the vehicle rolled over so easily. This was not at all what they expected and clearly demonstrated that the vehicle may not be completely safe to drive.
The vehicle manufacturer initially argued that swerving violently as in the case with the Swedish journalists, was not the normal driving behaviour, therefore their vehicle was safe to drive and didn’t require any modifications.
The vehicle manufacturer quickly realised that this stance could seriously damage their reputation, which in turn could seriously affect the sales of their new vehicle. This left little option other than to admit there was a stability problem and make corrections to reduce the effect of the violent steering movements.
To further highlight the importance of using bad and ugly data, consider the previous section ‘Testing under normal conditions’, where the Ministry of Transport, MOT test was highlighted as a test under normal conditions.
If a vehicle passes all the MOT tests, does that mean that the vehicle is safe to drive?
The MOT test cannot guarantee the safety of the vehicle under test. Simply because the tests done only fulfil the minimum requirement needed to prove that the vehicle is road worthy, which doesn’t imply that it is safe to use.
The Swedish journalists mentioned earlier tested a new vehicle which would have easily have passed the MOT test. However, it transpired that driving the vehicle under abnormal conditions caused it to roll over. Highlighting that this particular vehicle was not safe to drive, irrespective of whether the manoeuvre used was expected during normal day to day driving.
By introducing abnormal conditions which are conditions which would probably not occur regularly in the day to day running of the vehicle, the safety level of the vehicle can be more thoroughly tested.
So whilst in the MOT test the steering can be tested to see if the steering components are in good condition and whether the play between the steering is acceptable. It’s only when the vehicle is driven and the steering tested under abnormal conditions, can a better understanding of the safety of the steering systems be developed.
Driving at speed and turning the steering wheel sharply, will provide a better indication of how safe the steering is to use. Turning the steering wheel from left to right and vice versa in quick succession, over several minutes will also provide a better insight into the safety of the steering in the vehicle. Compared to just physically checking the components or testing the steering by turning it from left to right and vice versa during the vehicle being stationary.
It may be that turning the steering wheel sharply to the left and right causes unacceptable vibrations to travel through the steering column. Or the steering column breaks causing the vehicle steering to fail completely.
If after all the tests under abnormal conditions have been applied and the vehicle is still working as expected. Then it can be said that the vehicle is safe to use.
Global Markets
It’s important to remember that a lot of products marketed today are sold in a multitude of markets globally. Where different conditions need to be considered in the functional testing process.
A product in one market may need to withstand high temperatures and humidity. Whilst in another market, the temperatures to withstand are near freezing or below, making the testing in extreme conditions a must in any testing program.
It is essential to test products for every market in which the product will be used
These extremes would be classed as bad data but more importantly what happens when ugly data is introduced. In this example, ugly data would include high temperatures and rapid changes to low temperatures and vice versa over a prolonged period.
This could cause the functions/components to fail. As an example the lights on the vehicle could be tested in extremes of temperature to see whether they still managed to function.
One particular car caused it’s manufacturer some embarrassment, when parked cars simply started rolling down hill on their own accord when the temperatures were very low.
The cars caused an immense amount of damage, from rolling into houses, other vehicles and of course damage to themselves from the impact suffered.
Whilst the hand braking system had been tested, the manufacturer had failed to consider abnormal conditions where the outside temperature was lower than tested.
When the handbrake was applied, the braking mechanism would brake by using the front brake discs. Which when the car had been running would have become hotter and as a consequence would have expanded slightly.
When the car was parked and the handbrake was engaged the brake pads would engage the front discs. This would stop the car from rolling.
However as the discs cooled down, they began to contract and in extreme colder temperatures the level to which they contracted meant that the grip from the brake pads to the discs, lessened causing the cars to roll.
The extreme coldness of the temperature would have been classed as ugly data. Data which the discs and brake pads were not designed to handle.
As a consequence the manufacturer had to eventually compensate for the damage caused by the cars rolling out of control and rectify the hand brake system design.
When a new design of commercial plane is released there are certain tests that a plane must undergo to ensure that it meets the air safety requirements set out by the Federal Aviation Administration (FAA).
Whilst the standard, good data and it’s bad data counterparts are used to determine the satisfactory level of functionality of the components within the airplane, there is one test where ugly data is introduced that really catches the eye.
This test is called the tail strike test, where the plane takes off at a sharper than normal angle causing the rear of the plane to strike the ground. This causes damage to the rear of the plane and the tail.
The testers then measure the functionality of the plane, as with the good and bad data tests whilst the plane is in the air and then when it lands. If a plane does not perform as expected, that is, it can’t fly or land with the moderate tail strike damage, then the plane model will not receive it’s airworthiness certificate.
This test is not done on all planes, as this would simply not be practical. This is due to the fact that the planes would be damaged, therefore this test is only done to the initial plane or planes released.
It can be difficult to determine how much testing is required. The next section focuses on determining the level of testing required.
Summary
• The abnormal conditions include applying bad and ugly data.
• Testing under abnormal conditions includes using a product under conditions which it was not designed for.
• Testing under abnormal conditions is necessary to ensure that a product can meet different global demands.
How much testing is required?
The hardest part of any testing is to determine how much testing is required. There is no fixed rule to dictate the length of testing needed. Even when the product has been tested there may still be something that’s missed which could cause problems later.
It is impossible to test every product that is made. Not only would there be a limited time to do the testing but the costs involved would be astronomical.
It is impossible to test every product made.
Just as it’s impossible to test every product that rolls off the production line, it also becomes impossible to test a product completely.
It is impossible to test a product completely.
Again this is due to the time available and the costs involved, making it just too impractical to completely test the product.
Consider a chocolate manufacturer who needs to make sure that their chocolates taste great. They have a production line that can produce 100,000 chocolate pieces a day. There are 20 people involved with managing the day to day production of the chocolates.
To completely test every chocolate piece made would mean taking a slice out of each chocolate to taste whether it was satisfactory and analysing the slice chemically to see whether it’s constituents were as they should be.
However, by taking a slice from each chocolate piece manufactured, the chocolate would become unsellable. As the chocolate would be classed as being damaged.
The additional costs involved in employing additional people to taste every chocolate piece manufactured would substantially increase the manufacturing costs. Which in turn would increase the price of the chocolates. This in turn would prove too expensive for the end consumer.
The satisfactory recourse in this example would be to sample a random selection of chocolates, so when a new batch of chocolates is made, ten, twenty or thirty, for example, random pieces are taken to be analysed and tasted.
By doing this the costs of testing are reduced substantially. With little or no disruption in the manufacturing process for the chocolates.
By sampling, testing can still be effective as well as costing less to implement.
A large aircraft manufacturer must view their testing goals differently to that of the chocolate manufacturer. Whilst several factors will determine whether it may be too costly and impractical for the chocolate manufacturer to test all their chocolates. The aircraft manufacturer must test all the planes that they manufacture.
As each plane can cost several million pounds, testing each plane will be paramount to ensure that the customer receives not only value for money but a plane that is safe to fly.
Whilst every plane manufactured will be tested, the amount of testing done per plane will not be complete. Certain tests such as those involved with the tail strike will damage the plane and these tests will only be conducted on the initial batch of planes.
Even with a product with a small margin of error, such as a plane. There is still no effective way for it to be fully tested. As such there have been instances where this has led to a catastrophe.
Level Of Risk
In essence the level of testing done is determined by the level of risk from the product failing. Whilst with the chocolate manufacturer, the level of risk, if their chocolates fail to live up to the standards set, would be a loss of sales and the loss of their image and brand.
The plane manufacturer, the risks involved are much higher as there would be the loss of life, as well as the loss of sales and the loss of image/brand to contend with.
Simply by introducing the element of an increased possibility of the loss of life through the failure of a product, substantially increases the risks involved with that particular product.
The plane manufacturer to counter the risks involved must test all the planes that they produce, however that doesn’t mean that they are fully tested. Only the components that are critical are tested per plane, others are tested at the prototype stage.
It’s akin to baking a cake, initially there is uncertainty of what quantity of ingredients are required. But as soon as this is realised, any subsequent cakes baked don’t require the trial and error measuring to see if the ingredients are in the right quantities.
The higher the risks involved the greater detail of testing done.
The same applies with the plane, once you know how thick the fuselage metal skin needs to be, there is no need to test this further to see whether this thickness can withstand stresses and strains. The only tests required would be tests to seek out any imperfections in the fuselage.
Level Of Testing
The level of testing is of the utmost importance. Testing something more or less than it should be can also be a risk. For the plane manufacturer, spending a disproportionately longer time testing the smoke alarms in the planes toilets than the effectiveness of the door seals, will introduce a higher risk.
Whilst the smoke alarms failing are not going to be life threatening, the same can’t be said for the door seals failing. As such, a failure would create rapid depressurisation and potentially could cause the plane to crash.
Millennium Bug
A good example of the level of risk is the millennium bug problems from the previous millennia. This introduced a frenzy of testing, as more and more organisations became aware of the issues of the millennium bug. Especially how the change to the year 2000 could cause wholesale computer systems failure.
Computer’s running older programs would be unable to recognise that it was the year 2000, instead thinking it was 1900. This could potentially stop computer systems from working, affecting the daily lives of millions. It all boiled down to how computer programs stored a date’s year. Most of the business critical programs being used at the time and even still now, were written in the 1960’s, 1970’s and 1980’s.
These computer programs used two digits to represent the year in the date. So 1990 would be represented by 90, 1991 by 91 and so on up to 1999, being represented by 99. The problem occurred when the year reached 2000, as this would be represented by 00. The computer programs would see the 00 and think it represented 1900, causing a multitude of problems in accuracy of the programs being run.
So why didn’t they think of this when the programs were written in the 1960’s, 1970’s and 1980’s and more importantly why didn’t they test this?
To many at the time when these computer programs were developed, the year 2000 seemed a distant dream. As such little real thought was given to the potential impact of the date year change.
Testing is about thinking about the potential problems and trying to prepare for them.
Testing could have highlighted the risks of using two digits. This could have helped reduce the billions spent on rectifying the computer programs worldwide during the late nineties.
When the computer programs were being tested in the 1960’s, 1970’s and 1980’s, the perceived risk of the millennium bug was low. The millennium was too far away to be considered a serious risk. Compared to the late 1990’s, where the perceived risk from the failure of computer programs during the transition to the new millennia was enormous.
As such an incredible amount of resource and money was spent bringing the older computer programs to an acceptable four digit year accepting standard.
Summary
• Nothing can be fully and completely tested.
• The level of testing done is determined by the risk of something going wrong and the issues that this would create.
• Risk can be variable, something done before a certain time period may have a low risk whilst something afterwards may have a higher risk.
• Testing is reducing the level of risk and not trying to completely eliminate it.
How to document test results
When a key is turned in a door lock, what happens?.
The door unlocks and by pushing or pulling the door, it can be opened. We know that by turning the key we expect a certain outcome to occur, that is the door open.
This outcome is known as the expected result of the action. The action is turning the key.
As you can see above, the task was to unlock the door, which required an action, which was to turn the key. The expected result of this action was that the door would unlock.
What would have happened if another key was used?.
If the wrong key is used then the door will not unlock and as a result this task fails. As its prime purpose is to open the door which has not being accomplished.
Expected Results
In testing, a similar principle is involved where a certain result is expected, or a set of results to be returned for the actions applied.
If these results don’t appear then the test in question is deemed to have failed. Each of these test components is known as a test case. In the above examples, the test cases would have been ‘Test whether the key opens the door’
It must be known what result needs to be expected if the test is run as it should be.
So for example if a car is being tested to see whether it is stable enough if an elk ran in front of it, when evasive action was taken by violently turning the steering wheel.
The expected result would be that the car remained upright and did not rollover. However when actually running the test, if the car rolled over then it would be deemed as failing the test. It would be pointless having a test where the outcome expected was to see if the car rolled over.
As it would serve no purpose to the car manufacturer. Who doesn’t want their car to rollover and then have to compensate in millions, to any injured parties as a result of their vehicle rolling over.
Instead they will definitely make several test cases where they test that their vehicle is stable.
Designing a product or service to fail to deliver what is expected of it, isn’t what any organisation will invest their money in.
Test Cases
By defining test cases, it becomes not only easy to evaluate tests and help even the novice run through several test cases but it makes the testing accountable. Once all the planned test cases have been tested it can be proved whether the product or functions as it should.
For the example of opening a door with a key, the test case would be ‘Test whether the key opens the door’. Instead of using the word ‘Action’ the word ‘Data’ will be used. Also included will be whether the test passed or failed and more importantly when it was carried out.
The latter is very important as the test case may be run several times and a way to clearly differentiate each test is required.
If the key did not open the door as expected then the test case could be written as follows:
Now this may seem a very over complicated way of representing how a door is opened by using a key. However in the real world when a product is being tested, representing the information this way, that is in a format that can be easily understood and interpreted by all those people who would be interested in the testing, is very important.
Test Recording
It is imperative that proper test recording is done in a format that is acceptable.
By recording test results in this manner, if there is any recourse later on from other people involved with the product or service, then proper documented evidence is available.
Take the MOT test, if we conduct this test using the test case method, we can construct a test as shown below:
The above example and the one on the next page, have all assumed the use of data that is normal.
So for example turning the steering wheel in a manner that would be used on a daily basis.
Introducing Bad And Ugly Data
Test cases also allow us to introduce bad and even ugly data. If we run through the above steering wheel test cases but this time substitute the good data with bad and ugly data. It becomes apparent that test cases are an excellent way of running tests
The Test Case below, uses data which is bad data, as this would not be the expected behaviour of the person using the steering wheel on a daily basis.
But what happens if the vehicle is going to be used off road or for rallying?.
Then the violent turning of the steering wheel may be a commonplace in such a situation.
In the next test case the extreme is used that is ugly data.
Here what is least expected in the daily driving habit of a typical car driver is done, that is driving at high speed in reverse and turning the steering wheel abruptly from one side to the other.
Summary
• It is very important to record tests in a manner that is accountable to all those involved with the product.
• By knowing what the outcome should be (expected result) it can be determined whether a test has passed or failed.
• The expected result should be positive and any deviation will be a failure.
• Test cases should include bad and ugly data as well as good data.
Related Questions
What is software testing and why it is important? Software testing allows organisations to check not only that their software works as expected but also that any potential bugs have been found before the software is released.
What is software testing in simple words? Software testing is checking for bugs in coding, vulnerabilities in design and determining if requirements have been met.
