About testocrat

A tester who is in the learning phase and is interested in learning new things

Open Source Performance Testing Tools

Tool 1: Allmon

Description: The main goal of the project is to create a distributed generic system collecting and storing various runtime metrics collections used for continuous system performance, health, quality and availability monitoring purposes. Allmon agents are designed to harvest a range of metrics values coming from many areas of monitored infrastructure (application instrumentation, JMX, HTTP health checks, SNMP). Collected data are base for quantitative and qualitative performance and availability analysis. Allmon collaborates with other analytical tools for OLAP analysis and Data Mining processing.

Requirement: Platform independent

Link: http://code.google.com/p/allmon/downloads/list

Tool 2: Apache JMeter

Description: Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions. Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types. You can use it to make a graphical analysis of performance or to test your server/script/object behavior under heavy concurrent load.

Requirement: Solaris, Linux, Windows (98, NT, 2000). JDK1.4 (or higher)

Link: http://jakarta.apache.org/jmeter/

Tool 3: benerator

Description: benerator is a framework for creating realistic and valid high-volume test data, used for (unit/integration/load) testing and showcase setup. Metadata constraints are imported from systems and/or configuration files. Data can be imported from and exported to files and systems, anonymized or generated from scratch. Domain packages provide reusable generators for creating domain-specific data as names and addresses internationalizable in language and region. It is strongly customizable with plugins and configuration options.

Requirement: Platform Independent

Link: http://databene.org/databene-benerator

Tool 4: CLIF is a Load Injection Framework

Description: CLIF is a modular and flexible distributed load testing platform. It may address any target system that is reachable from a Java program (HTTP, DNS, TCP/IP…) CLIF provides 3 user interfaces (Swing or Eclipse GUI, command line) to deploy, control and monitor a set of distributed load injectors and resource consumption probes (CPU, memory…) An Eclipse wizard helps programming support for new protocols. Load scenarios are defined through XML-editing, using a GUI, or using a capture tool. The scenario execution engine allows the execution of up to millions of virtual users per load injector.

Requirement: Java 1.5 or greater, with enhanced support for Linux, Windows XP, MacOSX/PPC

Link: http://clif.objectweb.org/

Tool 5: ContiPerf

Description: ContiPerf is a lightweight testing utility that enables the user to easily leverage JUnit 4 test cases as performance tests e.g. for continuous performance testing. It is inspired by JUnit 4’s easy test configuration with annotations and by JUnitPerf’s idea of wrapping Unit tests for performance testing, but more powerful and easier to use.

Requirement: Windows, Mac OSX, Linux, Solaris and all other platforms that support Java 5

Link: http://databene.org/contiperf

Tool 6: curl-loader

Description: A C-written web application testing and load generating tool. The goal of the project is to provide a powerful open-source alternative to Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and TLS/SSL protocol stacks, simulating tens of thousand and hundred users/clients each with own IP-address. The tool supports user authentication, login and a range of statistics.

Requirement: Linux

Link: http://curl-loader.sourceforge.net/

Tool 7: D-ITG

Description: D-ITG (Distributed Internet Traffic Generator) is a platform capable to produce traffic at packet level accurately replicating appropriate stochastic processes for both IDT (Inter Departure Time) and PS (Packet Size) random variables.

Requirement: Linux, Windows

Link: http://www.grid.unina.it/software/ITG/

Tool 8: Database Opensource Test Suite

Description: The Database Opensource Test Suite (DOTS) is a set of test cases designed for the purpose of stress-testing database server systems in order to measure database server performance and reliability.

Requirement: Linux, POSIX

Link: http://ltp.sourceforge.net/

Tool 9: DBMonster

Description: DBMonster is an application to generate random data for testing SQL database driven applications under heavy load.

Requirement: OS Independent

Link: http://sourceforge.net/projects/dbmonster/

Tool 10: Deluge

Description: An open-source web site stress test tool. Simulates multiple user types and counts. Includes proxy server for recording playback scripts, and log evaluator for generating result statistics. Note: this tool is no longer under active development although it is still available on Sourceforge.

Requirement: OS independent

Link: http://deluge.sourceforge.net/

Tool 11: Dieseltest

Description: Contains the high-end features common to packages costing $50,000 or more. Dieseltest is a Windows application that simulates hundreds or thousands of users hitting a website. To run a load test, you first create a test script using our script editor. The script contains all of the requests that a real-world user would make of a website. You then load the script and run the test. The system will show you real-time results while the script is running, and produce a report analyzing the results at the conclusion.

Requirement: Windows

Link: http://sourceforge.net/projects/dieseltest/

Tool 12: Faban

Description: Faban is a facility for developing and running benchmarks, developed by Sun. It has two major components, the Faban harness and the Faban driver framework. The Faban harness is a harness to automate running of server benchmarks as well as a container to host benchmarks allowing new benchmarks to be deployed in a rapid manner. Faban provides a web interface to launch & queue runs, and extensive functionality to view, compare and graph run outputs.

Requirement: OS independent; JVM 1.5 or later

Link: http://java.net/projects/faban/

Tool 13: FunkLoad

Description: FunkLoad is a functional and load web tester, written in Python, whose main use cases are functional and regression testing of web projects, performance testing by loading the web application and monitoring your servers, load testing to expose bugs that do not surface in cursory testing, and stress testing to overwhelm the web application resources and test the application recoverability, and writing web agents by scripting any web repetitive task, like checking if a site is alive.

Requirement: OS independent – except for the monitoring which is Linux specific

Link: http://funkload.nuxeo.org/

Tool 14: FWPTT load testing web applications

Description: fwptt is an open source Web application testing program for load testing web applications. It can record normal and AJAX requests. It has been tested on ASP.Net applications, but it should work with JSP, PHP or other.

Requirement: Windows

Link: http://fwptt.sourceforge.net/

Tool 15: Grinder

Description: The Grinder is a Java load-testing framework making it easy to orchestrate the activities of a test script in many processes across many machines, using a graphical console application.

Requirement: OS Independent

Link: http://grinder.sourceforge.net/

Tool 16: GrinderStone

Description: GrinderStone is an Eclipse plug-in for Grinder load testing scripts development including debugging, modularity and pretty logging

Link: http://code.google.com/p/grinderstone/

Tool 17: Hammerhead 2 – Web Testing Tool

Description: Hammerhead 2 is a stress testing tool designed to test out your web server and web site. It can initiate multiple connections from IP aliases and simulated numerous (256+) users at any given time. The rate at which Hammerhead 2 attempts to pound your site is fully configurable, there are numerous other options for trying to create problems with a web site (so you can fix them).

Requirement: Hammerhead has been used with Linux, Solaris and FreeBSD

Link: http://hammerhead.sourceforge.net/

Tool 18: Hammerora

Description: Hammerora is a load generation tool for the Oracle Database and Web Applications. Hammerora includes pre-built schema creation and load tests based on the industry standard TPC-C and TPC-H benchmarks to deploy against the Oracle database with multiple users. Hammerora also converts and replays Oracle trace files and enables Web-tier testing to build bespoke load tests for your entire Oracle application environment.

Requirement: Platform Independent (Binaries for Linux and Windows)

Link: http://hammerora.sourceforge.net/

Tool 19: httperf

Description: Httperf is a tool for measuring web server performance. It provides a flexible facility for generating various HTTP workloads and for measuring server performance. The focus is not on implementing one particular benchmark but on providing a robust, high-performance tool that facilitates the construction of both micro and macro level benchmarks. The three distinguishing characteristics of httperf are its robustness, which includes the ability to generate and sustain server overload, support for the HTTP/1.1 and SSL protocols, and its extensibility.

Requirement: Linux (Debian package available), HP-UX, perhaps other Unix

Link: http://www.hpl.hp.com/research/linux/httperf/

Tool 20: http_load

Description: http_load runs multiple HTTP fetches in parallel, to test the throughput of a Web server. However, unlike most such test clients, it runs in a single process, to avoid bogging the client machine down. It can also be configured to do HTTPS fetches.

Requirement: tbc

Link: http://www.acme.com/software/http_load/

Test 21: Iperf

Description: Iperf was developed by NLANR/DAST as a modern alternative for measuring maximum TCP and UDP bandwidth performance. Iperf allows the tuning of various parameters and UDP characteristics. Iperf reports bandwidth, delay jitter, datagram loss.

Requirement: Platform Independent

Link: http://iperf.sourceforge.net/

Tool 22: IxoraRMS

Description: Monitoring tool with great visualization and customization capabilities. It’s quick to install and suitable for use in performance labs.

Requirement: Windows, Unix

Link: http://www.ixorarms.com/

Tool 23: JChav

Description: JChav is a way to see the change in performance of your web application over time, by running a benchmark test for each build you produce. JChav reads all the JMeter logs from each of your runs (one per build), and produces a set of charts for each test in each run.

Requirement: JMeter

Link: http://jchav.blogspot.com/

Tool 24: JCrawler

Description: Stress-Testing Tool for web-applications. It comes with the crawling/exploratory feature. You can give JCrawler a set of starting URLs and it will begin crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. The load parameters (hits/sec) are configurable.

Requirement: OS Independent

Link: http://jcrawler.sourceforge.net/

Tool 25: loadUI

Description: loadUI is a tool for Load Testing numerous protocols, such as Web Services, REST, AMF, JMS, JDBC as well as Web Sites. Tests can be distributed to any number of runners and be modified in real time. LoadUI is tightly integrated with soapUI. LoadUI uses a highly graphic interface making Load Testing Fun and Fast.

Link: http://www.loadui.org/

Tool 26: Lobo, Continuous Tuning

Description: Lobo is a tool for performance testing and monitoring that allows you to monitor the evolution of performance along the time-line of the project. It was specially designed to be used in agile-iterative and evolutionary approaches.

Requirement: Java

Link: http://www.oncast.com.br/eng/open_source_lobo.php

Tool 27: MessAdmin

Description: MessAdmin is a light-weight and non-intrusive notification system and HttpSession administration for J2EE Web Applications, giving detailed statistics and informations on the application. It installs as a plug-in to any Java EE WebApp, and requires zero-code modification.

Requirement: OS Independant

Link: http://messadmin.sourceforge.net/

Tool 28: mstone

Description: Mstone started as a mail performance measurement system but now can test svn, etc. It can simultaneously test SMTP, POP, IMAP, and some HTML based systems. It measures transaction latency in multiple stages, and graphs the combined results from multiple clients.

Requirement: multiple (perl based)

Link: http://mstone.sourceforge.net/

Tool 29: Multi-Mechanize

Description: Multi-Mechanize is an open source framework for web performance and load testing. It allows you to run simultaneous python scripts to generate load (synthetic transactions) against a web site or web service.

Link: http://code.google.com/p/multi-mechanize/

Tool 30: NTime

Description: The NTime tool is very similar to NUnit tool to perform repeatable tasks that help managers, architects, developers and testers to test an application against its performance.

Requirement: Windows 98 or above, .Net framework 1.1 or 2.0

Link: http://www.codeproject.com/KB/dotnet/NTime.aspx

Tool 31: OpenSTA

Description: A distributed software testing architecture based on CORBA. Using OpenSTA (Open System Testing Architecture) a user can generate realistic heavy loads simulating the activity of hundreds to thousands of virtual users. OpenSTA graphs both virtual user response times and resource utilization information from all Web Servers, Application Servers, Database Servers and Operating Platforms under test, so that precise performance measurements can be gathered during load tests and analysis on these measurements can be performed.

Requirement: Windows 2000, NT4 and XP

Link: http://opensta.org/

Tool 32: OpenWebLoad

Description: OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing near real-time performance measurements of the application under test.

Requirement: Linux, Windows

Link: http://openwebload.sourceforge.net/

Tool 33: Ostinato

Description: Ostinato is an open-source, cross-platform packet/traffic generator and analyzer with a friendly GUI. It aims to be “Wireshark in Reverse” and thus become complementary to Wireshark.

Requirement: Cross-Platform

Link: http://code.google.com/p/ostinato/

Tool 34: p-unit

Description: An open source framework for unit test and performance benchmark, which was initiated by Andrew Zhang, under GPL license. p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time consumption, and generates the result in the form of plain text, image or pdf file.

Requirement: OS Independent

Link: http://p-unit.sourceforge.net/

Tool 35: PandoraFMS

Description: Pandora FMS is a monitoring Open Source software. It watches your systems and applications, and allows you to know the status of any element of those systems. Pandora FMS could detect a network interface down, a defacement in your website, a memory leak in one of your server application, or the movement of any value of the NASDAQ new technology market. If you want, Pandora FMS could send out SMS message when your systems fails… or when Google’s value drop below US$ 500.

Requirement: 32-bit MS Windows (NT/2000/XP), All POSIX (Linux/BSD/UNIX-like OSes), Solaris, HP-UX, IBM AIX

Link: http://pandora.sourceforge.net/

Tool 36: postal

Description: SMTP benchmarking tool. It is threaded, uses very little disk I/O (e-mail body content randomly generate text). It has an SMTP source, SMTP sink and POP server load tester (to pull sent mail)

Requirement: Linux/UNIX; requires C compiler

Link: http://doc.coker.com.au/projects/postal/

Tool 37: Pylot

Description: Pylot is a free open source tool for testing performance and scalability of web services. It runs HTTP load tests, which are useful for capacity planning, benchmarking, analysis, and system tuning. Pylot generates concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests suites are executed and monitored from a GUI.

Requirement: Python 2.5+. required.Tested on Windows XP, Vista, Cygwin, Ubuntu, MacOS

Link: http://www.pylot.org/

Tool 38: Raw Load Tester

Description: This application calls the URL you select as many times as you choose and tells you how long it took the server to respond. It writes some additional runtime details to the PHP log file so you can optionally do more granular analysis afterwards. Although the server processes most of the statistics, all URL requests come from the browser. You can run as many browsers and workstations simultaneously as you want.

Requirement: PHP/JavaScript

Link: http://www.room4me.com/techtools/RawLoadTester/index.html

Tool 39: Seagull

Description: Seagull is a multi-protocol traffic generator test tool. Primary aimed at IMS protocols, Seagull is a powerful traffic generator for functional, load, endurance, stress and performance tests for almost any kind of protocol. Currently supports Diameter, XCAP over HTTP, TCAP (GSM Camel, MAP, Win) protocols.

Requirement: Linux/Unix/Win32-Cygwin

Link: http://gull.sourceforge.net/

Tool 40: Siege

Description: SIEGE is an http regression testing and benchmarking utility. It was designed to let web developers measure the performance of their code under duress, to see how it will stand up to load on the internet. It lets the user hit a webserver with a configurable number of concurrent simulated users. Those users place the webserver “under siege.” SCOUT surveys a webserver and prepares the urls.txt file for a siege. In order to perform regression testing, siege loads URLs from a file and runs through them sequentially or randomly. Scout makes the process of populating that file easier. You should send out the scout, before you lay siege.

Requirement: GNU/Linux, AIX, BSD, HP-UX and Solaris.

Link: http://www.joedog.org/

Tool 41: Sipp

Description: SIPp is a performance testing tool for the SIP protocol. Its main features are basic SIPStone scenarios, TCP/UDP transport, customizable (xml based) scenarios, dynamic adjustement of call-rate and a comprehensive set of real-time statistics. It can also generate media (RTP) traffic for audio and video calls.

Requirement: Linux/Unix/Win32-Cygwin

Link: http://sipp.sourceforge.net/

Tool 42: SLAMD

Description: SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing and performance analysis of network-based applications.

Requirement: Any system with Java 1.4 or higher

Link: http://www.slamd.com/

Tool 43: Soap-Stone

Description: Network benchmark application which can put your network under load and conduct automatic benchmark and recording activities.

Requirement: OS Independent

Link: http://soap-stone.sourceforge.net/

Tool 44: stress_driver

Description: General-purpose stress test tool.

Requirement: Windows NT/2000, Linux

Link: http://sourceforge.net/projects/stress-driver/

Tool 45: TestMaker

Description: TestMaker from PushToTest.com delivers a rich environment for building and running intelligent test agents that test Web-enabled applications for scalability, functionality, and performance. It comes with a friendly graphical user environment, an object-oriented scripting language (Jython) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS, SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write scripts for you, a library of fully-functional sample test agents, and shell scripts to run test agents from the command line and from unit test utilities.

Requirement: Windows, Linux, Solaris, and Macintosh

Link: http://www.pushtotest.com/testmaker-open-source-testing

Tool 46: TPTEST

Description: The purpose with TPTEST is to allow users to measure the speed of their Internet connection in a simple way. TPTEST measures the throughput speed to and from various reference servers on the Internet. The use of TPTEST may help increase the consumer/end user knowledge of how Internet services work.

Requirement: MacOS/Carbon and Win32

Link: http://tptest.sourceforge.net/about.php

Tool 47: Tsung

Description: Tsung is a distributed load testing tool. It is protocol-independent and can currently be used to stress HTTP, SOAP and Jabber servers (SSL is supported). It simulates complex user’s behaviour using an XML description file, reports many measurements in real time (including response times, CPU and memory usage from servers, customized transactions, etc.). HTML reports (with graphics) can be generated during the load. For HTTP, it supports 1.0 and 1.1, has a proxy mode to record sessions, supports GET and POST methods, Cookies, and Basic WWW-authentication. It has already been used to simulate thousands of virtual users.

Requirement: Tested on Linux, but should work on MacOSX and Windows.

Link: http://tsung.erlang-projects.org/

Tool 48: Valgrind

Description: Valgrind is an award-winning suite of tools for debugging and profiling Linux programs. With the tools that come with Valgrind, you can automatically detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable. You can also perform detailed profiling, to speed up and reduce memory use of your programs.

Requirement: Linux

Link: http://valgrind.org/

Tool 49: Web Application Load Simulator

Description: LoadSim is a web application load simulator. It allows you to create simulations and have those simulations run against your webserver.

Requirement: JDK 1.3 or above

Link: http://jobmanager.sourceforge.net/openware_pub/

Tool 50: Web Polygraph

Description: Benchmarking tool for caching proxies, origin server accelerators, L4/7 switches, content filters, and other Web intermediaries.

Requirement: C++ compiler

Link: http://www.web-polygraph.org/

Tool 51: WebLOAD

Description: WebLOAD Open Source is a fully functional, commercial-grade performance testing product based on WebLOAD, Radview’s flagship product that is already deployed at 1,600 sites. Available for free download and use, WebLOAD is a commercial-grade open source project with more than 250 engineering years of product development. Companies that require commercial support, additional productivity features and compatibility with third-party protocols have the option of purchasing WebLOAD Professional directly from RadView.

Requirement: Windows NT/2000/XP

Link: http://www.webload.org/

Source: Open Source Testing

Open Source Functional Testing Tools

Tool 1: Abbot Java GUI Test Framework

Description: The Abbot framework provides automated event generation and validation of Java GUI components, improving upon the very rudimentary functions provided by the java.awt.Robot class (A Better ‘Bot). The framework may be invoked directly from Java code or accessed without programming through the use of scripts. It is suitable for use both by developers for unit tests and QA for functional testing.

Link: http://abbot.sourceforge.net/

Tool 2: Anteater

Description: Anteater is a testing framework designed around Ant, from the Apache Jakarta Project. It provides an easy way to write tests for checking the functionality of a Web application or of an XML Web service.

Requirement:  OS Independent

Link: http://aft.sourceforge.net/

Tool 3: Arbiter

Description: Document based acceptance tester. Similar to FIT in goal. Manages requirements documents in Word or RTF format that are created jointly by customer and developer. Requirements are parsed to extract a glossary and test suite.

Link: http://arbiter.sourceforge.net/

Tool 4Canoo WebTest

Description: Used for functional testing of web pages, WebTest is an open source testing framework built on top of HttpUnit. It allows tests to be defined in XML as Ant targets.

Requirement:  JDK 1.2 and ANT v1.3

Link: http://webtest.canoo.com/webtest/

Tool 5DejaGnu

Description: DejaGnu is a framework for testing applications such as Tcl, C, C++, Java and network applications and cross testing of embedded systems. Its purpose is to provide a single front end for all tests. Think of it as a custom library of Tcl procedures crafted to support writing a test harness.

Requirement:  MacOS, Windows, POSIX

Link: http://www.gnu.org/software/dejagnu/

Tool 6Doit: Simple Web Application Testing

Description: Doit is a scripting tool and language for testing web applications that use forms. Doit can generate random or sequenced form fill-in information, report results (into a database, file, or stdout), filter HTML results, and compare results to previous results, without having to manually use a web browser. It uses a console-based web client tool (like Curl or Wget) to send and receive HTTP requests and responses respectively.

Requirement:  You must have Perl 5 or greater and the appropriate Perl modules (detailed in Doit manual) installed on your system before you can use SPL.

Link: http://doit.sourceforge.net/

Tool 7: Enterprise Web Test

Description: Enterprise Web Test allows Java programmers to write re-usable tests for web applications that, unlike HttpUnit, “drive” the actual web browser on the actual platform they intend to support. Tests can be leveraged for functional, stress, reliability.

Requirement: Microsoft, OS Independent, Linux

Link: http://sourceforge.net/projects/webunitproj/

Tool 8Harness

Description: An open source Java API for creating Java test software

Link: http://sourceforge.net/projects/harness/

Tool 9ITP 

Description: Lightweight, yet powerful web application test harness. Test scripts written in XML. No programming required and no changes required to your web application. Supports sessions/cookies, POST form data. Command line based for integration into other tools. Also useful for regression and smoke testing

Link: http://www.incanica.com/itp.html

Tool 10ivalidator

Description: Regression testing framework written in java but by no means restricted to java testing. Test suites are declared in XML. Especially designed for complex testing scenarios and integration testing.

Requirement:  JDK 1.3

Link: http://www.ivalidator.org/

Tool 11Jameleon 

Description: Jameleon is a plug-in driven automated testing tool that separates applications into features and allows those features to be tied together independently, creating test cases. Test cases can be data-driven and executed against different environments and test case docs are generated from the test cases. The goal is to create an automated testing tool that can be used for the enterprise. A UI that ties a series of features to a test case, generating both the test script and the test case documentation is in the works.

Requirement:  OS Independent, JDK 1.4 or higher.

Link: http://jameleon.sourceforge.net/

Tool 12Jemmy

Description: Jemmy is a tool allowing you to create automated tests for Java GUI applications. Tests are written on Java, using Jemmy as a regular Java library. No recording (yet), no GUI, no XML, no bells and whistles – all the work being done in Jemmy is dedicated to test stability, so it could be used for big, complicated and “dynamic” Java GUI applications.

Requirement: Java 1.2 or above

Link: http://jemmy.netbeans.org/

Tool 13JFunc: JUnit Functional Testing Extension

Description: JFunc is an extension to the JUnit testing framework to make it easier for use with functional tests. Functional testing (also called integration testing) significantly differs from unit testing in a number of respects. Part of this project is dedicated towards putting together code to address these differences; the other part of this project is putting together methodologies for functional testing.

Requirement:  JUnit

Link: http://jfunc.sourceforge.net/

Tool 14jWebUnit

Description: jWebUnit is a Java framework that facilitates creation of acceptance tests for web applications. jWebUnit provides a high-level API for navigating a web application combined with a set of assertions to verify the application’s correctness. This includes navigation via links, form entry and submission, validation of table contents, and other typical business web application features. This code utilizes HttpUnit behind the scenes. The simple navigation methods and ready-to-use assertions allow for more rapid test creation than using only JUnit and HttpUnit.

Link: http://jwebunit.sourceforge.net/

Tool 15Latka

Description: Latka is a functional (end-to-end) testing tool. It is implemented in Java, and uses an XML syntax to define a series of HTTP (or HTTPS) requests and a set of validations used to verify that the request was processed correctly.

Requirement: JDK 1.3 or better

Link: http://jakarta.apache.org/commons/latka/index.html 

Tool 16: Linux Test Project

Description: The Linux Test Project is a collection of tools for testing the Linux kernel and related features. Our goal is to improve the Linux kernel by bringing test automation to the kernel testing effort.

Requirement: Linux

Link: http://ltp.sourceforge.net/

Tool 17LogiTest  

Description: LogiTest is the core application in the LogiTest suite. The LogiTest application provides a simple graphical user interface for creating and playing back tests for testing Internet-based applications.

Requirement: JDK 1.2 or higher 

Link: http://logitest.sourceforge.net/logitest/index.html

Tool 18Marathon

Description: Marathon is a general purpose tool for both running and authoring acceptance tests geared at the Java Platform Version 1.3 or later. Included with marathon is a rich suite of components to help you interact with your application at the User Interface Level (GUI). To aid with the regression testing of existing applications, Marathon comes bundled with a recorder to capture events as you use and interact with your application and these events are then converted into a valid Marathon test which can subsequently be played back.

Requirement: Java 1.3 or later

Link: http://marathonman.sourceforge.net/

Tool 19MaxQ 

Description: MaxQ is a free web functional testing tool. It includes an HTTP proxy that records your test script, and a command line utility that can be used to playback tests. The paradigm of MaxQ is similar to commercial web testing tools like Astra QuickTest or Empirix e-Test. These products are quite expensive. MaxQ hopes to provide the essential features: HTTP test recording, scripting, and playback without the huge cost.

Requirement: Java 1.2 or later

Link: http://maxq.tigris.org/ 

Tool 20: Pounder 

Description: Pounder is a utility for testing Java GUIs. It allows developers to dynamically load components, record scripts, and then use those scripts in JUnit. It supports custom components, drag and drop, and the examination of test runs in source.

Link: http://pounder.sourceforge.net/ 

Tool 21QAT (Quality Assurance Tests)

Description: QAT was developed to ease the issues encountered by having to perform Quality Assurance tests across a variety of hardware and software combinations. The QAT tool can be divided into two main sections, the Agent, responsible for actually running each test or group of tests, and the Harness, which is responsible for test selection, management, result and agent co-ordination.

Requirement: Java 2

Link: http://qat.sourceforge.net/ 

Tool 22QMTest

Description: CodeSourcery’s QMTest provides a cost-effective general purpose testing solution that allows an organization to implement a robust, easy-to-use testing program tailored to its needs. QMTest’s extensible architecture allows it to handle a wide range of application domains: everything from compilers to graphical user interfaces to web-based applications.

Requirement: QMTest works with most varieties of UNIX, including GNU/Linux, and with Microsoft Windows.

Link: http://www.codesourcery.com/qmtest 

Tool 23: Scalable Test Platform

Description: STP is a system for automating the QA testing process for the Linux Kernel, as well as automating benchmarking and regression testing on diverse hardware systems.

Requirement: Linux

Link: http://www.osdl.org/lab_activities/kernel_testing/stp/ 

Tool 24: Selenium

Description: Testing tool for browser-based testing of web applications. It can be used both for functional, compatability (it has extensive cross-browser support) and regression testing.

Requirement: Windows, Linux or Mac

Link: http://selenium.thoughtworks.com/index.html 

Tool 25Solex 

Description: This project is a set of Eclipse plugins providing non regression and stress tests of Web application servers. Test scripts are recorded from internet browser thanks to a built in web proxy.

Requirement: Eclipse 2.1 or above 

Link: http://solex.sourceforge.net/ 

Tool 26Tclwebtest

Description: Tclwebtest is a tool for writing automated tests on web applications in tcl. It implements some basic html parsing functionality to provide comfortable commands for operations on the html elements (most importantly forms) of the result pages.

Link: http://tclwebtest.sourceforge.net/

Tool 27TestCaseMaker / Runner

Description: Test Case Document driven functional test tool for web applications. Maker creates a test case documents, and Runner executes the test case document itself.

Requirement: J2SDK 1.4.x

Link: http://www5f.biglobe.ne.jp/~webtest/testcasetool/index.html

Tool 28Toster – The Object-oriented Software Testing Environment

Description: Toster is a system for sharing a set of tools that allow you to implement methods for object-oriented testing. Any method based on UML diagrams and on the software source code can easily be implemented as a TOSTER module. The environment itself makes a number of mechanisms available, such as information transfer from UML diagrams, mapping this information to source code, introducing modifications to the source code, launching the tested application, or presenting the results.

Link: http://toster.sourceforge.net/

Tool 29: Watir – Web Application Testing in Ruby

Description: Watir (Web Application Testing in Ruby) is a functional testing tool for web applications. It supports tests executed at the web browser layer by driving a web browser and interacting with objects on a web page. It uses the Ruby scripting language.

Link: http://wtr.rubyforge.org/

Tool 30Web Form Flooder

Description: Web Form Flooder is a Java console utility that will analyze a Web page, complete any forms present on the page with reasonable data, and submit the data. The utility will also crawl links within the site in order to identify and flood additional forms that may be present.

Link: http://formflood.sourceforge.net/

Tool 31: WebInject 

Description: WebInject is a free tool for automated testing of web applications and services. It can be used to test any individual system component with an HTTP interface, and as a test harness to create a suite of automated functional and regression tests.

Requirement: Windows, OS Independent, Linux

Link: http://www.webinject.org/

Tool 32: WebTst

Description: WebTst is a Web development test infrastructure. It aims to simplify testing by implementing a capture engine: a Web proxy which records a tester’s actions using a real browser, and then replays them during testing. It comes with support for digital certificates, and a number of simple tests, such as cookie setting, pattern matching, response status, and many others. It features an extensible plug-in system.

Requirement: POSIX, Linux

Link: http://webtst.assisrosa.com/

Tool 33: X11::GUITest

Description: X11::GUITest is a Perl package intended to facilitate the testing of GUI applications by means of user emulation. It can be used to test and interact with GUI applications which have been built (Xlib, GTK, etc.) upon the X toolkit.

Requirement: Linux, FreeBSD

Link: http://sourceforge.net/projects/x11guitest/

Tool 34: XML Test Suite

Description: XmlTestSuite provides a powerful way to test web applications. Writing tests requires only knowledge of HTML and XML. We want XmlTestSuite to be adopted by testers, business analysts, and web developers who don’t have a java background.

Requirement: Windows 95/98/2000, Windows NT/2000, Linux, SunOS/Solaris

Link: http://xmltestsuite.sourceforge.net/

Tool 35QAliber

Description: Test automation framework, for building fast, re-usable and reliable system tests under windows OS. It includes a tool for developing automation in .NET, and a GUI tool to compose automation with no coding skills.

Link: http://qaliber.net

Microsoft Team Foundation Server

Team Foundation Server (commonly abbreviated to TFS) is a Microsoft product offering source control, data collection, reporting, and project tracking, and is intended for collaborative software development projects. It is available either as stand-alone software or as the server side back end platform for Visual Studio Team System (VSTS). TFS was launched in the year 2005.

The following major features are included in Team Foundation Server:

  • Version control, for managing source code and other deliverables that require versioning.
  • Work item tracking, for keeping track of such things as defects, requirements, tasks, and scenarios.
  • Project management functions, which allow the shaping of a team project based on a user-specifiable software process, and which enable planning and tracking using Microsoft Excel and Microsoft Project.
  • Team build, for enabling a common process for building executable products.
  • Data collection and reporting, which aid in the assessment of a team project’s state, based on information gleaned from Team Foundation Server tools.
  • The Team Project Portal, which provides a central point of communication for a team project packaged as a Microsoft Windows SharePoint Services site.
  • Team Foundation Shared Services, which provide a number of common infrastructure services that invisible to end users but that are important to toolsmiths and extenders.

Metrics Used In Testing

The Product Quality Measures

1. Customer satisfaction index

This index is surveyed before product delivery and after product delivery (and on-going on a periodic basis, using standard questionnaires).The following are analyzed:

  • Number of system enhancement requests per year
  • Number of maintenance fix requests per year
  • User friendliness: call volume to customer service hotline
  • User friendliness: training time per new user
  • Number of product recalls or fix releases (software vendors)
  • Number of production re-runs (in-house information systems groups)

2. Delivered defect quantities

They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.

3. Responsiveness (turnaround time) to users

  • Turnaround time for defect fixes, by level of severity
  • Time for minor vs. major enhancements; actual vs. planned elapsed time

 4. Product volatility

  • Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)

5. Defect ratios

  • Defects found after product delivery per function point.
  • Defects found after product delivery per LOC
  • Pre-delivery defects: annual post-delivery defects
  • Defects per function point of the system modifications

 6. Defect removal efficiency

  • Number of post-release defects (found by clients in field operation), categorized by level of severity
  • Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects
  • All defects include defects found internally plus externally (by customers) in the first year after product delivery

7. Complexity of delivered product

  • McCabe’s cyclomatic complexity counts across the system
  • Halstead’s measure
  • Card’s design complexity measures
  • Predicted defects and maintenance costs, based on complexity measures

8. Test coverage

  • Breadth of functional coverage
  • Percentage of paths, branches or conditions that were actually tested
  • Percentage by criticality level: perceived level of risk of paths
  • The ratio of the number of detected faults to the number of predicted faults.

9. Cost of defects

  • Business losses per defect that occurs during operation
  • Business interruption costs; costs of work-arounds
  • Lost sales and lost goodwill
  • Litigation costs resulting from defects
  • Annual maintenance cost (per function point)
  • Annual operating cost (per function point)
  • Measurable damage to your boss’s career

10. Costs of quality activities

  • Costs of reviews, inspections and preventive measures
  • Costs of test planning and preparation
  • Costs of test execution, defect tracking, version and change control
  • Costs of diagnostics, debugging and fixing
  • Costs of tools and tool support
  • Costs of test case library maintenance
  • Costs of testing & QA education associated with the product
  • Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)

11. Re-work

  • Re-work effort (hours, as a percentage of the original coding hours)
  • Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
  • Re-worked software components (as a percentage of the total delivered components)

12. Reliability

  • Availability (percentage of time a system is available, versus the time the system is needed to be available)
  • Mean time between failure (MTBF).
  • Man time to repair (MTTR)
  • Reliability ratio (MTBF / MTTR)
  • Number of product recalls or fix releases
  • Number of production re-runs as a ratio of production runs

Metrics for Evaluating Application System Testing:

Metric = Formula

Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)

Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).

Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria

Defects per size = Defects detected / system size

Test cost (in %) = Cost of testing / total cost *100

Cost to locate defect = Cost of testing / the number of defects located

Achieving Budget = Actual cost of testing / Budgeted cost of testing

Defects detected in testing = Defects detected in testing / total system defects

Defects detected in production = Defects detected in production/system size

Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100

Effectiveness of testing to business = Loss due to problems / total resources processed by the system.

System complaints = Number of third party complaints / number of transactions processed

Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10

Source Code Analysis = Number of source code statements changed / total number of tests.

Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual Effort for Design and Documentation

Test Execution Productivity = No of Test cycles executed / Actual Effort for testing