SOFTWARE RELIABILITY ENGINEERING Reliability Estimation Tools SRE TOOLKIT _____________________________________________________ Introduction The reliability estimation tools described in this guide are particularly useful during system test and field trial This version of the SRE TOOLKIT contains the standard release of the reliability estimation tool EST, the graphics support tool PLOT and a number of prototype tools used in conjunction with exercises in course SN9110 (Software Reliabil- ity Engineering - Application) provided through Kelly Education and Training Center. This guide is a reference guide for using the standard tools EST and PLOT. Programmer Notes on using the prototype tools are provided in Appendix B. As such, they are not intended as a training or tutorial guide on Software Reliability Engineering and should be used along with training from course SN9110. The reliability estimation tool EST described in this guide is particularly useful during system test and field trial phases of software product development. During these phases, failure events are encountered and the underlying faults that cause the failures are being removed. This results in "reliability growth" during product test or tri- al. The tools implement techniques discussed in reference [MIO] to estimate the current level of software product reliability and to predict the remaining time to attain a specified reliability objective. The tool EST discussed in this guide can be used to "fit" one of two reliability models described in reference [MIO] to failure data. In turn, EST uses the "fitted" model to estimate several useful reli- ability measures such as present failure intensity, remaining time to reach a specified failure inten- sity objective and a calendar date when a failure intensity objective will be met. EST produces its Page 1. Iss. 1D1 Reliability Estimation Tools SRE TOOLKIT output as a tabular report and a file of "plot" commands. The tool PLOT in turn takes the file of plot commands and produces a set of plots on a graphics media. Graphic media currently supported set of plots on a graphics medium. Graphic media currently supported include postscript printers accessible through UNIX8r9 systems or graphics moni- tors on PC's running under MS-DOS8r9. There are two versions of the tools, one version runs under the UNIX There are two versions of the tools: one version runs under the UNIX operating environment while the other version runs under the MS-DOS operating environment. Both versions of the tools have been carefully engineered so that the format of input and output data, plots, and general use of the tools is the same under either UNIX or MS-DOS. This provides considerable flexibility to the user. A user can distribute work between a large shared UNIX system to take advantage of its support facilities and a small PC workstation to do quick turn-around "what if" analysis of collected data. The user can use terminal emulator programs (such as "ctrm") to up-load and down-load input and output files between a UNIX system and a PC works- tation to take advantage of particular facilities available on each system. A user can distribute work between a large shared UNIX system taking advantage of its support facilities and a small PC work-station obtaining a quick turn-around "what if" analysis of collected data by using the screen-graphics capabilities of the MS-DOS version. Then, the user might use a terminal emulator pro- gram (such as "ctrm") to up-load input data files to a UNIX system and take advantage of documenter and printer facilities to produce an output report including graphical output from the tools. This guide is a reference guide for using the tools and is not intended as a training or tutorial guide on Software Reliability Engineering. This guide should be used in conjunction with training from This guide should be used with training from course SN9110 (Software Reliability Engineering - Practi- cal Applications) Engineering - Application) pro- vided through Kelly Education and Training Center _____________________________________________________ Organization of this Guide The remainder of this guide is divided into two parts. The remainder of this guide is divided into three parts. The first part contains information on required hardware and operating environments for running the tools, instructions on installing the tools, instructions on getting started and tips on using the tools. At the end of the first part is information on what is available in the way of training, project support, tool support and refer- ences. The second part (in the Appendix) is a set of manu- al pages providing a detail The second part is a set of manual pages in Appendix A providing a de- tail reference on using the tools themselves. The manual pages provide examples of inputs to and resulting outputs from the tools. The manual pages also provide pointers back into reference [MIO] for further information on Software Reliability itself, on the input data that is needed by the tools and on interpreting output of the tools. on Software Reliability Engineering itself, on the input data that is needed by the tools and on interpreting output of the tools. The third part is a set of Programmer Notes in Ap- pendix B on running a set of prototype tools developed for SN9110. There are also UNIX and MS- DOS versions these tools. Included with the Pro- grammer Notes are a set of Manual Pages for using these tools. Again, the manual pages provide exam- ples of inputs to and resulting outputs from the tools. _____________________________________________________ Hardware and Software Requirements The UNIX version of the tools runs under any ver- sion of UNIX System V and on any version of hardware processor that supports UNIX System V. Care was exercised to use a restricted set of UNIX system library calls to maintain as much portabili- ty as possible between systems. Iss. 4D4 Page 4. SOFTWARE RELIABILITY ENGINEERING The MS-DOS version should work with MS-DOS release 3.3 or greater running on any AT&T compatible PC. To use graphics, the PC should be equipped with a CGA, EGA, VGA or Hercules compatible graphics board. If extensive plotting is to be done or large (greater than 150 failure events) failure data sets will be analyzed, floating point hardware will sig- nificantly reduce processing time (from minutes to seconds). For running the tools on a PC, this would mean investing in a numeric coprocessor (sometimes referred to as a math coprocessor) chip. this would mean investing in a numeric co-processor (sometimes referred to as a math co-processor) chip. These chips usually have a model designation of 8xx87 or 8x87 depending on the type of processor in your PC. _____________________________________________________ Installation UNIX Version - The UNIX version of the tools is distributed using UNIX electronic mail (email) fa- cilities. To obtain a copy, contact Michael R. Lyu giving your name and email address. You will receive email back confirming the receipt of your request and indicating when and how the tools will be sent to you. The tools will be sent to you via the UNIX uuto(1) command (see reference [ATT]). command (see reference [ATTa]). You will receive email indicating the tools have arrived on your system. At that time, you should execute the UNIX uupick(1) command to retrieve them. First change directories to whichever directory you want the tools directory SRE_TOOLS installed into, then execute uupick. At the prompt from system !whuxr: directory SRE_TOOLS ? from system !mtfmi: directory SRE_TOOLS ? type "m ." followed by a carriage return. At this point the SRE_TOOLS directory will be installed in the current directory you are in. Then, change directory to the SRE_TOOLS directory and read the file READ.ME which provides further information on Iss. 6D6 Page 6. SOFTWARE RELIABILITY ENGINEERING installing the tools. Complete installation requires approximately 5 to 15 minutes depending on the processing speed and processing load levels of your system. MS-DOS Version - The MS-DOS version is distributed either on 5-1/4 inch or 3-1/2 inch floppies. Each floppy contains "executable" program files The MS- DOS version is distributed on a 5-1/4 inch, 360 Kbyte double-sided, double-density floppy. The floppy contains "executable" program files est.exe and plot.exe and test data files tst.fp, tst.ft, tst.pc, tst_stg.fp, tst_stg.ft and tst_stg.pc. The floppy diskette can be inserted in the appropriate drive, the drive selected by typing a: or b:. If your PC has a hard disk, we recommend copying the program and data files into a directory on your hard disk and running the tools from your hard disk. Otherwise, you can run the tools directly from the floppy diskette (of course, making a backup copy of the diskette first). Alternatively, with the UNIX version of the delivered tools, there is a directory named dos that contains copies of the "executable" program and data files. These files can be up-loaded to your PC using the file- transfer capabilities of your favorite terminal emulator package (such as ctrm). ___________________________________________________ __ Page 7. Iss. 7D7 Reliability Estimation Tools SRE TOOLKIT _____________________________________________________ Getting Started, Using the Manual Pages You might want to first review the manual pages in the appendix. You might want to first review the manual pages in Appendix A. Manual page EST(1) describes the est program. First, quickly browse the "Description" section of the manual page and then read the "Example" section to follow the exe- cution of a particular example. After reading the manual page, you may want to proceed to the next section of this guide to execute the example in your UNIX or MS-DOS environment. If you want to learn more about the input data files for est, then you'll want to read the .FP(5) manual page that describes the contents of the "failure parameter" file and .FT(5) manual page that describes the con- tents of the "failure time" file. Again, browse the "Description" section of the manual page, then read the "Example" section. Refer back to the "Description" section read the "Example" section. Refer to the "Description" section whenever you need more detail in following the example. In gen- eral, you don't need to be familiar with the PLOT(1) manual page. However, if you reach a point where you want to tailor some of the plots, then you can read this manual page to see how to change the plot commands in the associated .pc files pro- duced by est. The same applies to getting started with the proto- type tools. First, review the Programmer notes to understand the caveats in running the tools. Then browse the "Description" section of the manual page and concentrate on the "Example" section of the manual page. _____________________________________________________ Getting Started, Running the Examples UNIX Version - After installing the UNIX version of the tools, change directories to the directory your tools were installed in and then to the testdir 9110exer subdirectory under this directory. There are two sets of project data provided in this directory. The first is associated with project tst. This data is in files tst.fp and tst.ft. First run the est program against this project data by typing est tst. Note the tabular output report produced. Manual page EST(1) in the appendix can be used to interpret the contents of this report. Note the tabular output report produced. Manual page EST(1) in Appendix A can be used to interpret the contents of this report. The program creates the plot commands in file tst.pc. Now, you can type plot tst to generate the plots. In the UNIX version, a file tst.po containing pic(1) and troff(1) commands is produced (see the UNIX User Reference Manual for more information on pic and troff). You can now run your favorite command for formatting "troff" text You can now run your favor- ite command for formatting troff text files819 and routing to output to postscript printers or other printers with graphics capabilities with the tst.po file. Don't forget to either first run the UNIX pic(1) command against the file or to include the appropriate option on the command line of the troff text formatter to preprocess the file using the "pic(1)" command. using the pic(1) command. The __________ 1. Examples of such commands are mmx(1), mmt(1), xroff(1). An example invocation of such a command with the tst data would be mmt -p tst.po where the -p option on the command line indicates the file should first be processed by "pic(1)". indicates the file should first be processed by "pic(1)." indicates the file should first be processed by pic(1). Check with your UNIX system administrator to find out what commands are available on your UNIX system. Reference [GE] provides further information on document formating commands Reference [GE] provides further information on document formatting commands under UNIX. Page 9. Iss. 9D9 Reliability Estimation Tools SRE TOOLKIT second set of project data with project name tst_stg is the same as the first except "staged delivery" information has been added to the failure data (see the .FT(5) manual page in the appendix for a description of staged delivery). manual page in Appendix A for a description of staged delivery). You may now want to run est and plot programs with this project data and compare the resulting tabular report and plots with the tst project data. MS-DOS Version - If you have created a directory on your hard disk with the program and data files included with the distribution diskette, then you should first "change directories" to this direc- tory. As with the UNIX version, there are two sets of project data provided. The first is associated with project tst. This data is in files tst.fp and tst.ft. First run the est program against this project data by typing est tst. Note the tabular output report produced. Manual page EST(1) in the appendix can be used to interpret the contents of this report. in Appendix A can be used to inter- pret the contents of this report. In the MS-DOS version of the data files, the genplt parameter has been set in the failure parameter file tst.fp file so no plot commands are produced (this was done because PC's not having a "math coprocessor" board will require a long time to run). not having a "math co-processor" chip will require a long time to run). Instead, the "plot commands" correspond- ing to project tst has already been created and supplied as file tst.pc with the distribution diskette. You may now want to run plot tst to pro- duce the plots directly on your video monitor. The second set of project data with project name tst_stg is the same as the first except "staged delivery" information has been added to the failure data (see the .FT(5) manual page in the appendix for a description of staged delivery). manual page in Appendix A for a description of staged delivery). You may now want to run est and plot programs with this project data and compare the resulting tabular report and plots with the tst project data. Again, genplt has been set so no plot commands are generated. Instead, tst_stg.pc file has been provided with your distribution diskette. Iss. 10D10 Page 10. SOFTWARE RELIABILITY ENGINEERING _____________________________________________________ When running under MS-DOS, you can get hard-copy of screen output by using the MS-DOS mode mode and graphics graphics commands and a locally connected dot-matrix printer (see you MS-DOS User's Guide). To do this you generally execute the mode mode com- mand to define the characteristics of your printer and then the graphics graphics command to load a memory-resident program. To print a copy of the display currently appearing on your terminal's screen, you would depress the "Prt Sc" or "Print Screen" key (the name of this key is dependent on exactly what type of keyboard you have). The program plot(1) plot(1) that is provided with this tool set is a useful tool in itself for pro- ducing displays. The novice user need not get into learning about the plot commands that the plot(1) plot(1) program uses in generating plots. The ex- pert user can begin writing analysis programs that can generate .pc .pc files to create their own graphs. Or, better yet, you can change .pc .pc files created by the est(1) est(1) program to add additional lines, points, labels on particular graphs and so on. One approach to using the tools is to do the heavy CPU est(1) est(1) runs on a UNIX system with processing horse-power. Then download Iss. 12D12 Page 12. SOFTWARE RELIABILITY ENGINEERING the resulting .pc .pc files to a PC, edit the .pc .pc files to do "touch-ups" and customize graphs and run plot(1) plot(1) on the PC to immediately see the effects of changes to the .pc .pc files. Finally, the .pc .pc files can then be up-loaded to a UNIX system to runoff final reports with the graphical output on a laser printer. For those who may be familiar with the RELTOOLS tool set and the reltab reltab and relplt relplt programs, this tool set provides everything in the way of features of these tools plus more. The structure of the input files for est(1) est(1) pro- gram is quite similar to the input files of reltab reltab and relplt relplt programs. The one notable exception is the failure time file. For the reltab/relplt reltab/relplt programs, the corresponding file is referred to as a failure interval file. The times in the failure interval file are "times between failures" rather than actual "failure times." The format of the failure interval and failure time files are different. The failure parameter files for both the est(1) est(1) and reltab/relplt reltab/relplt file are almost identical (there are a few differences in some parameter names). _____________________________________________________ SOFTWARE RELIABILITY ENGINEERING Caveats Version 3.7 is a "Beta Trial" version of the tools that is being made available on a "friendly user" basis. As such, the basic functionality of the tools has been extensively tested and results com- pared with the predecessor software RELTAB/RELPLT. With this version of the tool, the heuristic algo- rithm (in EST) that determines scale values for the X and Y axes has not been fully implemented (this algorithm selects a scale so the X and Y scale values printed contain only a few significant di- gits). Also, some of the less frequently used PLOT PARAMETER options for EST (such as charht, clopt, conlvl, dshopt, grdopt, xwinb, xwine, ywinb, ywine) have not been fully tested. Users of this "Beta Trial" version of the tools are asked to communicate any problems via email to "whuxr!wwe" or (w.w.everett@att.com). _____________________________________________________ References [ATT] "UNIX System V Release 3.0 User Reference Manual", to order, "UNIX System V Release 3.0 User Reference Manual," to order, [ATTa] UNIX System V Release 3.0 User Reference Manual, to order, call AT&T Customer's In- formation Center 1-800-432-6600 and order Select Code 307-184. [ATTb] Reliability by Design, Chapter 8, to order, call AT&T Customer's Information Center 1- 800-432-6600. [MIO] Musa, J. D., A. Iannino and K. Okumoto, Software Reliability - Measurement, Predic- tion, Application Software Reliability - Measurement, Prediction, Application, McGraw-Hill, 1987, ISBN 0-07-044093-X. [MA] Musa, J and A. F. Ackerman, Quantifying Software Validation: When to Stop Testing?, IEEE SOFTWARE, May 1989, pg. 19-27. [GE] Gehani, N., Document Formatting on the UNIX System, Silicon Press, 1986, ISBN 0- 9615336-0-9. _____________________________________________________