Software Testing

Software Testing
Software testing is an activity aimed at evaluating a trait or capability of a program or system and determining that it meets its required results. Although crucial to software quality and extensively deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software. The difficulty in software testing stems from the complexity of software: we can not completely test a program with moderate complexity.
Testing is more than just debugging. The intention of testing can be quality assurance, verification and validation, or reliability estimation. Testing can be used as a generic metric as well. Correctness testing and reliability testing are two major areas of testing. Software testing is a trade-off between budget, time and quality.

Software Testing; Introduction

Software Testing

Introduction

When Testing should occur:

Testing is sometimes incorrectly thought as an after-the –fact activity, performed after programming is done for a product. Instead, testing should be performed at every development stage of the product. Test data sets must be derived and their correctness and consistency should be monitored through out the development process. If we divide the lifecycle of software development into “requirement analysis”, “design”, “programming”, and “operation and maintenance”, then testing should accompany each of the above phases. If testing is isolated as a single phase in the cycle, errors in the problem statement or design may incur exorbitant costs. Not only must the original error be corrected, but the entire structure built upon it must also be changed. Therefore, testing should be involved through out the SDLC in order to bring a quality product. Testing activities in each phase: Requirement analysis: Determine correctness Generate functional test data Design Determine correctness and consistency Generate structural and functional test data Programming Determine correctness and consistency Generate structural and functional test data Apply test data Refine test data Operation and maintenance Retest When should testing stop “When to stop testing” is one of the most difficult questions to a test engineer.

Some of the common test stop criteria are:

All the high priority bugs are fixed. The rate at which bugs are found is too small. The testing budget is exhausted. The project duration is complete. The risk in the project is under acceptable limit.

Practically, we feel that the decision of stopping testing is based on the level of the risk acceptable to the management. As testing is a never ending process we can never assume that 100% testing has been done, we can only minimize the risk of shipping the product to client with testing done. The risk can be measured by risk analysis but for small duration/ low budget/ low resources project, risk can be deduced by simply:-

Measuring test coverage. Number of test cycles. Number of high priority bugs.

How much to test

Testing is never ending process Every testing has some associated cost Each planned testing should be evaluated for “testing cost” Vs “cost of defect”

Test Strategy

Before starting any test activities, the team lead will have to think a lot and arrive at a strategy. This will describe the approach, which is to be adopted for carrying out test activities including the planning activities. This is a formal document and the very first document regarding the testing area and is prepared at a very early stage in SDLC. This document must provide generic test approach as well as specific details regarding the project. The following areas are addressed in the test strategy document.

Test Plan

The test plan, containing the details of the testing process, has to be prepared during the project planning stage itself. This plan should contain all the details of required resources, the testing approaches to be followed, the testing methodologies, the test cases etc. The test strategy identifies multiple test levels, which are going to be performed for a project. Activities at each level must be planned well in advance and it has to be formally documented. Based on individual plans only, the individual test levels are carried out.

The plans are to be prepared by experienced people only. In all test plans the EVTX (Entry task validation exit) criteria are to be mentioned. Entry means the entry point to that phase. For example for unit testing, the coding must be complete and then only one can start unit testing. Task is the activity that is performed. Validation is the way in which the progress and correctness and compliance are verified for that phase. Exit tells the completion criteria of that phase after validation is done.

EVTX is a modeling technique for developing worldly and atomic level models. It sands for entry, task, validation and exit. It is a task based model where the details of each task are explicitly defined in a specification table against each phase.

Test Plan Format

Before taking up the testing, a test plan has to be prepared. It is preferable if the test plan is prepared during the planning stage itself and revised just before the testing process starts. A test plan format is given:

Test Plan Project Name: Estimated start date for testing: Estimated end date for testing: Actual start date for testing: Actual end date for testing: Estimated effort in person hours: Test set up (including the hardware, software environment), other peripherals required: Test personnel and their responsibility: Type of testing to be carried out: For each type of testing, the test cases are to be specified: Testing tools to be used: Test schedule for each type of test: Test cases to be executed for regression testing:

Software Testing Cycle

Software Testing

Software Testing Cycle

Requirement: Requirement manager is used to link the requirements with the tests to be carried out. Each requirement in the SRS has to be tested at least once. In the SRS, the functional requirements and performance requirements are specified. Functional requirements are generated from use case scenarios. Performance requirements are dependent on the application. Use case: As requirements are gathered as part of informal meetings, the software engineer can create a set of scenarios that identify a thread of usage for the system to be constructed. The scenarios, often called used case provide a description of how the system will be used. Write/ Run test case: Test cases can be defined as sets of input parameters for which the software will be tested. Test case is selected according to the requirements of the client. Then these test cases are executed and the results are compared with the expected results. Test cases have to be designed based on two criteria: reliability and validity. A set of test cases is considered to be reliable if it detects all the errors. A set of test cases are valid if at least one test case reveals the errors.

Test case format

Test Case No. Test CasePurpose Procedure Expected Result Actual Result ID which is given to the Test Case What to test How to test What should happen What actually happened

Test Results:

When a test is run, the actual results are compared with the expected results. The test log contains the pass or fails results of various tests. Every defect found during the testing is logged in a defect log.

Defect Tracking: the defect tracking process is given in the figure.

Quality

The most important requirement of a software product is that it should meet the requirement of the customer. In addition to the user requirement, the management of the development organization may put additional requirements to ensure that the product is developed at a low cost and also to ensue that the software can be maintained at low cost. If the management wants to modify customized software to make it a generic product at a later date, additional requirement may be put such as portability. In general the characteristics of a software product can be divided in to

Operational characteristics. Transition characteristics. Revision characteristics.

Operation characteristics specify the requirements during the operation. Transition characteristics specify the requirements for its usage in other hardware/ operating systems environment. Revision characteristics specify the requirements for making the changes to software easy.

Quality management systems

When software development is sub contracted to an organization, the prime contractor needs to ensure that the work is sub contracted to a reliable organization, which can deliver quality software. So, the prime contractor has to ensure that the subcontractor has a defined quality system. To evaluate the effectiveness of the quality system, external auditing agencies will assess the quality system and certify it. The ISO 9000 standards and capability maturity model integration (CMMI) are two important quality management systems.

The management of every software development organization has to decide which quality system has to be implemented. If the organization has already obtained ISO 9000 or CMMI certification, for each project, the quality plan has to be in place based on these standards.

All the quality have the same underlying philosophy- to develop the software based on well defined processes, with the necessary documentation and ensure that every employee is committed to the quality improvement program. The quality standards should be considered as facilitators to the quality program of the organization rather than hindrance.

The management of every software development organization has to create a culture whereby the team members work within a well defined framework to develop quality software by giving thrust to quality at each and every stage of the development process.

The credit for the success of a project goes to the people who worked on the project and not to the processes. Similarly, the failure of a project is attributed to the people and not the processes. So it is the people who are responsible for success or failure of project.

Quality Standards

Quality is most important thing in a software industry. To deliver a product that is delight of the customer should be the main objective of the project manager, every team member as well as the senior management. To achieve this objective, quality standards come in handy. The management of every organization has to adapt quality standards so the quality is ensured in every product that is delivered. Implementation of quality standards also gives visibility to the organization.

Software engineering standards play an important role in defining the quality processes. There are nearly 250 software engineering standards developed by major standardization and professional bodies on various aspects.

In any standardization, process, measure play an important role. However, it is very difficult, in the field of software engineering, to obtain objective measures, and eliminate subjectivity completely. The quantity denoted by one liter of oil is same throughout the world, but the number of lines of code in a program may be different when measured by different organizations. Hence, the software standards should be treated as guidelines for improving the quality of software.

There are international, national and organizational standards. Most of the organizational standards are derived from national or international standards. For software engineering, ISO 9000 and CMMI are most widely used international standards.

V Model

Software Testing

V Model

Black Box Testing; Introduction

Black Box Testing

Introduction

Black Box Testing is testing without knowledge of the internal workings of the item being tested. For example, when black box testing is applied to software engineering, the tester would only know the "legal" inputs and what the expected outputs should be, but not how the program actually arrives at those outputs. It is because of this that black box testing can be considered testing with respect to the specifications, no other knowledge of the program is necessary. For this reason, the tester and the programmer can be independent of one another, avoiding programmer bias toward his own work. For this testing, test groups are often used, "Test groups are sometimes called professional idiots...people who are good at designing incorrect data." 1 Also, do to the nature of black box testing, the test planning can begin as soon as the specifications are written. The opposite of this would be glass box testing, where test data are derived from direct examination of the code to be tested. For glass box testing, the test cases cannot be determined until the code has actually been written. Both of these testing techniques have advantages and disadvantages, but when combined, they help to ensure thorough testing of the product. Synonyms for black-box include: behavioral, functional, opaque-box, and closed-box.

Advantages & Disadvantages

Black Box Testing

Advantages & Disadvantages

Advantages of Black Box Testing

More effective on larger units of code than glass box testing Tester needs no knowledge of implementation, including specific programming language Tester and programmer are independent of each other tests are done from a user's point of view will help to expose any ambiguities or inconsistencies in the specifications test cases can be designed as soon as the specifications are complete

Disadvantages of Black Box Testing

Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever Without clear and concise specifications, test cases are hard to design There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried May leave many program paths untested Can't be directed toward specific segments of code which may be very complex (and therefore more error prone) Most testing related research has been directed toward glass box testing

Testing Strategies

Black Box Testing

Testing Strategies

Testing Strategies/ Techniques

black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function data outside of the specified input range should be tested to check the robustness of the program boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output the number zero should be tested when numerical data is to be input stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems crash testing should be performed to see what it takes to bring the system down test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing. finite state machine models can be used as a guide to design functional tests According to Beizer 2 the following is a general order by which tests should be designed: Clean tests against requirements. Additional structural tests for branch coverage, as needed. Additional tests for data-flow coverage as needed. Domain tests not covered by the above. Special techniques as appropriate--syntax, loop, state, etc. Any dirty tests not covered by the above.

Functional Testing

Black Box Testing

Functional Testing

In this type of testing, the software is tested for the functional requirements. The tests are written in order to check if the application behaves as expected. Although functional testing is often done toward the end of the development cycle, it can—and should, —be started much earlier. Individual components and processes can be tested early on, even before it's possible to do functional testing on the entire system. Functional testing covers how well the system executes the functions it is supposed to execute—including user commands, data manipulation, searches and business processes, user screens, and integrations. Functional testing covers the obvious surface type of functions, as well as the back-end operations (such as security and how upgrades affect the system).

Stress Testing

Black Box Testing

Stress Testing

The application is tested against heavy load such as complex numerical values, large number of inputs, large number of queries etc. which checks for the stress/load the applications can withstand. Stress testing deals with the quality of the application in the environment. The idea is to create an environment more demanding of the application than the application would experience under normal work loads. This is the hardest and most complex category of testing to accomplish and it requires a joint effort from all teams. A test environment is established with many testing stations. At each station, a script is exercising the system. These scripts are usually based on the regression suite. More and more stations are added, all simultaneous hammering on the system, until the system breaks. The system is repaired and the stress test is repeated until a level of stress is reached that is higher than expected to be present at a customer site. Race conditions and memory leaks are often found under stress testing. A race condition is a conflict between at least two tests. Each test works correctly when done in isolation. When the two tests are run in parallel, one or both of the tests fail. This is usually due to an incorrectly managed lock. A memory leak happens when a test leaves allocated memory behind and does not correctly return the memory to the memory allocation scheme. The test seems to run correctly, but after being exercised several times, available memory is reduced until the system fails.

Load Testing

Black Box Testing

Load Testing

The application is tested against heavy loads or inputs such as testing of web sites in order to find out at what point the web-site/application fails or at what point its performance degrades. Load testing operates at a predefined load level, usually the highest load that the system can accept while still functioning properly. Note that load testing does not aim to break the system by overwhelming it, but instead tries to keep the system constantly humming like a well-oiled machine.In the context of load testing, extreme importance should be given of having large datasets available for testing. Bugs simply do not surface unless you deal with very large entities such thousands of users in repositories such as LDAP/NIS/Active Directory; thousands of mail server mailboxes, multi-gigabyte tables in databases, deep file/directory hierarchies on file systems, etc. Testers obviously need automated tools to generate these large data sets, but fortunately any good scripting language worth its salt will do the job.

Usability Testing

Black Box Testing

Usability Testing

This testing is also called as ‘Testing for User-Friendliness’. This testing is done if User Interface of the application stands an important consideration and needs to be specific for the specific type of user. Usability testing is the process of working with end-users directly and indirectly to assess how the user perceives a software package and how they interact with it. This process will uncover areas of difficulty for users as well as areas of strength. The goal of usability testing should be to limit and remove difficulties for users and to leverage areas of strength for maximum usability. This testing should ideally involve direct user feedback, indirect feedback (observed behavior), and when possible computer supported feedback. Computer supported feedback is often (if not always) left out of this process. Computer supported feedback can be as simple as a timer on a dialog to monitor how long it takes users to use the dialog and counters to determine how often certain conditions occur (ie. error messages, help messages, etc). Often, this involves trivial modifications to existing software, but can result in tremendous return on investment. Ultimately, usability testing should result in changes to the delivered product in line with the discoveries made regarding usability. These changes should be directly related to real-world usability by average users. As much as possible, documentation should be written supporting changes so that in the future, similar situations can be handled with ease.

Smoke Testing

Black Box Testing

Smoke Testing

This type of testing is also called sanity testing and is done in order to check if the application is ready for further major testing and is working properly without failing up to least expected level. A test of new or repaired equipment by turning it on. If it smokes... guess what... it doesn't work! The term also refers to testing the basic functions of software. The term was originally coined in the manufacture of containers and pipes, where smoke was introduced to determine if there were any leaks. A common practice at Microsoft and some other shrink-wrap software companies is the "daily build and smoke test" process. Every file is compiled, linked, and combined into an executable program every day, and the program is then put through a "smoke test," a relatively simple check to see whether the product "smokes" when it runs.

Volume Testing

v

Black Box Testing

Volume Testing

Volume testing is done against the efficiency of the application. Huge amount of data is processed through the application (which is being tested) in order to check the extreme limitations of the system. Volume Testing, as its name implies, is testing that purposely subjects a system (both hardware and software) to a series of tests where the volume of data being processed is the subject of the test. Such systems can be transactions processing systems capturing real time sales or could be database updates and or data retrieval. Volume testing will seek to verify the physical and logical limits to a system's capacity and ascertain whether such limits are acceptable to meet the projected capacity of the organization’s business processing.

Domain Testing

Black Box Testing

Domain Testing

Domain testing is the most frequently described test technique. Some authors write only about domain testing when they write about test design. The basic notion is that you take the huge space of possible tests of an individual variable and subdivide it into subsets that are (in some way) equivalent. Then you test a representative from each subset.

Regression Testing

Black Box Testing

Regression Testing

Regression testing is a style of testing that focuses on retesting after changes are made. In traditional regression testing, we reuse the same tests (the regression tests). In risk-oriented regression testing, we test the same areas as before, but we use different (increasingly complex) tests. Traditional regression tests are often partially automated. These note focus on traditional regression.

Regression testing attempts to mitigate two risks: A change that was intended to fix a bug failed. Some change had a side effect, unfixing an old bug or introducing a new bug.

Regression testing approaches differ in their focus. Common examples include:

Bug regression: We retest a specific bug that has been allegedly fixed.

Old fix regression testing: We retest several old bugs that were fixed, to see if they are back. (This is the classical notion of regression: the program has regressed to a bad state.)

General functional regression: We retest the product broadly, including areas that worked before, to see whether more recent changes have destabilized working code. (This is the typical scope of automated regression testing.)

Conversion or port testing: The program is ported to a new platform and a subset of the regression test suite is run to determine whether the port was successful. (Here, the main changes of interest might be in the new platform, rather than the modified old code.)

Configuration testing: The program is run with a new device or on a new version of the operating system or in conjunction with a new application. This is like port testing except that the underlying code hasn't been changed--only the external components that the software under test must interact with.

Localization testing: The program is modified to present its user interface in a different language and/or following a different set of cultural rules. Localization testing may involve several old tests (some of which have been modified to take into account the new language) along with several new (non-regression) tests.

Smoke testing also known as build verification testing: A relatively small suite of tests is used to qualify a new build. Normally, the tester is asking whether any components are so obviously or badly broken that the build is not worth testing or some components are broken in obvious ways that suggest a corrupt build or some critical fixes that are the primary intent of the new build didn't work. The typical result of a failed smoke test is rejection of the build (testing of the build stops) not just a new set of bug reports.

User Acceptence Testing

Black Box Testing

User Acceptence Testing

In this type of testing, the software is handed over to the user in order to find out if the software meets the user expectations and works as it is expected to. In software development, user acceptance testing (UAT) - also called beta testing, application testing, and end user testing - is a phase of software development in which the software is tested in the "real world" by the intended audience. UAT can be done by in-house testing in which volunteers or paid test subjects use the software or, more typically for widely-distributed software, by making the test version available for downloading and free trial over the Web. The experiences of the early users are forwarded back to the developers who make final changes before releasing the software commercially.

Alpha Testing

In this type of testing, the users are invited at the development center where they use the application and the developers note every particular input or action carried out by the user. Any type of abnormal behavior of the system is noted and rectified by the developers.

Beta Testing

In this type of testing, the software is distributed as a beta version to the users and users test the application at their sites. As the users explore the software, in case if any exception/defect occurs that is reported to the developers. Beta testing comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the company. Report errors.

White Box Testing; Introduction

White Box Testing

Introduction

White box testing strategy deals with the internal logic and structure of the code. White box testing is also called as glass, structural, open box or clear box testing. The tests written based on the white box testing strategy incorporate coverage of the code written, branches, paths, statements and internal logic of the code etc.

In order to implement white box testing, the tester has to deal with the code and hence is needed to possess knowledge of coding and logic i.e. internal working of the code. White box test also needs the tester to look into the code and find out which unit/ statement/ chunk of the code is malfunctioning.

Advantages and Disadvantages

White Box Testing

Advantages and Disadvantages

Advantages of White box testing are: As the knowledge of internal coding structure is prerequisite, it becomes very easy to find out which type of input/ data can help in testing the application effectively.

The other advantage of white box testing is that it helps in optimizing the code. It helps in removing the extra lines of code, which can bring in hidden defects.

Disadvantages of white box testing are:

As knowledge of code and internal structure is a prerequisite, a skilled tester is needed to carry out this type of testing, which increases the cost.

It is nearly impossible to look into every bit of code to find out hidden errors, which may create problems, resulting in failure of the application.

Testing Methods

White Box Testing

Testing Methods

Unit Testing

The developer carries out unit testing in order to check if the particular module or unit of code is working fine. The Unit testing comes at the very basic level as it is carried out as and when the unit of the code is developed or a particular functionality is built. Unit testing deals with testing a unit as a whole. This would test the interaction of many functions but confine the test within one unit. The exact scope of a unit is left to interpretation. Supporting test code, sometimes called scaffolding, may be necessary to support an individual test. This type of testing is driven by the architecture and implementation teams. This focus is also called black-box testing because only the details of the interface are visible to the test. Limits that are global to a unit are tested here. In the construction industry, scaffolding is a temporary, easy to assemble and disassemble, frame placed around a building to facilitate the construction of the building. The construction workers first build the scaffolding and then the building. Later the scaffolding is removed, exposing the completed building. Similarly, in software testing, one particular test may need some supporting software. This software establishes an environment around the test. Only when this environment is established can a correct evaluation of the test take place. The scaffolding software may establish state and values for data structures as well as providing dummy external functions for the test. Different scaffolding software may be needed from one test to another test. Scaffolding software rarely is considered part of the system. Sometimes the scaffolding software becomes larger than the system software being tested. Usually the scaffolding software is not of the same quality as the system software and frequently is quite fragile. A small change in the test may lead to much larger changes in the scaffolding. Internal and unit testing can be automated with the help of coverage tools. A coverage tool analyzes the source code and generates a test that will execute every alternative thread of execution. It is still up to the programmer to combine this test into meaningful cases to validate the result of each thread of execution. Typically, the coverage tool is used in a slightly different way. First the coverage tool is used to augment the source by placing informational prints after each line of code. Then the testing suite is executed generating an audit trail. This audit trail is analyzed and reports the percent of the total system code executed during the test suite. If the coverage is high and the untested source lines are of low impact to the system's overall quality, then no more additional tests are required.

Static and Dynamic Analysis

Static analysis involves going through the code in order to find out any possible defect in the code. Dynamic analysis involves executing the code and analyzing the output.

Statement Coverage

In this type of testing the code is executed in such a manner that every statement of the application is executed at least once. It helps in assuring that all the statements execute without any side effect.

Branch Coverage

No software application can be written in a continuous mode of coding, at some point we need to branch out the code in order to perform a particular functionality. Branch coverage testing helps in validating of all the branches in the code and making sure that no branching leads to abnormal behavior of the application.

Security Testing

Security Testing is carried out in order to find out how well the system can protect itself from unauthorized access, hacking – cracking, any code damage etc. which deals with the code of application. This type of testing needs sophisticated testing techniques.

Mutation Testing

A kind of testing in which, the application is tested for the code that was modified after fixing a particular bug/ defect. It also helps in finding out which code and which strategy of coding can help in developing the functionality effectively.

SDLC

SDLC

Software Development Life Cycle

The software development life cycle (SDLC) is the entire process of formal, logical steps taken to develop a software product. Software development is the process of developing software through successive phases in an orderly way. This process includes not only the actual writing of code but also the preparation of requirements and objectives, the design of what is to be coded, and confirmation that what is developed has met objectives. The software development life cycle (SDLC) is a conceptual model used in project management that describes the stages involved in an information system development project, from an initial feasibility study through maintenance of the completed application.

Typical phases of software development:

Requirement Analysis Software design Development or Coding Testing Maintenance

Requirement Analysis:

The new system requirements are defined. The requirements of a desired software product are extracted. Based the business scenario the SRS (Software Requirement Specification) document is prepared in this phase. The purpose of this document is to specify the functional requirements of the software that will be produced by the SOS project group. The specifications are intended to guide the group through the development process.

Design

Plans are laid out concerning the physical construction, hardware, operating systems, programming, communications, and security issues for the software. Design phase is concerned with making sure the software system will meet the requirements of the product. It should also ensure that the future requirements will also be addressed.

Development (or) Coding:

Programs are developed by the developers with the help of the design. The design is reduced to code by the software engineers.

Testing

Testing is evaluating the software to check for the user requirements. Here the software is evaluated with intent of finding defects.

Maintenance:

Once the new system is up and running for a while, it should be exhaustively evaluated. Maintenance must be kept up rigorously at all times. Users of the system should be kept up-to-date concerning the latest modifications and procedures.