CASRE
Several software reliability tools are currently available for
users to apply one or more of the known software reliability models
to a development effort. Popular tools include Statistical Modeling
and Estimation of Reliability Functions for Software (SMERFS),
and Software Reliability Modeling Programs (SRMP). In addition
to allowing users to make reliability estimates, these tools also
allow users to determine the applicability of a particular model
to a set of failure data.
A major issue in modeling software reliability, however, lies in
the ease-of-use of currently available tools. Nearly all current
tools have command-line interfaces, and do not take advantage of
the high-resolution displays that would allow the construction of
menu-driven or direct-manipulation user interfaces. Command-line
interfaces can make it more difficult for users to remember the
specific steps required to accomplish a task, while a menu-driven
interface can be set up to guide users through the necessary steps.
Finally, most currently available tools display their outputs in
tabular form. Although some tools provide a graphical display of
results, in most cases this is of limited utility in that the
display resolution is low since character-based graphics are used
in painting the screen. In addition, the variety of results that
may be graphically displayed is usually limited to interfailure
times or failure intensities. In measuring software reliability,
it is useful to see high-resolution displays of these quantities,
as well as cumulative number of errors, the reliability growth
curve, and the results of statistical methods used to determine
whether the model being executed is appropriate for the current
project.
CASRE is implemented as a software reliability modeling tool that
addresses the ease-of-use issue as well as other issues. CASRE is
an extension of the public-domain tool SMERFS, and is intended to
execute both in a DOS Windows environment and a UNIX X-windows
environment. The command interface is menu driven; users are
guided through the selecting of a set of failure data and executing
a model by selectively enabling pull-down menu options. Modeling
results are also presented in a graphical manner. After one or
more models have been executed, the predicted failure intensities
or interfailure times are drawn in a graphical display window.
Users can manipulate this window's controls to display the results
in a variety of ways, including cumulative number of failures and
the reliability growth curve. Users may also display the results
in a tabular fashion if they wish.
In addition, CASRE is facilitated with a useful functionality.
Namely, results from different models can be combined in various
ways to yield reliability estimates whose predictive quality is
better than the individual models themselves. CASRE is able to
increase prediction accuracy by combining the results of several
models in a linear fashion. Moreover, CASRE allows users to define
their own combinations and record them as part of the tool's
configuration. Weights for the components of the combination may
be static or dynamic, and may be based on statistical techniques
used to determine the applicability of a model to a set of failure
data. Once combination models have been defined, the steps
required to execute them are no different than executing a simple
model.
CASRE Context
There are six major functional areas of CASRE:
1. File Operations ("File" menu)
2. Editing Operations ("Edit" menu)
3. Transformation and Smoothing Operations("Filter" menu)
4. Model Selection and Application("Model" menu)
5. Program Configuration ("Setup" or "Settings" menu)
6. Help System ("Help" menu)
One additional functional area is implemented to set the
controls associated with a graphics display window in which
failure data and the results of modeling activities are
displayed. Contents of the graphics display window can also be
shown in tabular form in another text window.
1. File Operations
There are six operations that can be performed by selecting
items from the File pull-down menu. These are: "Open", "Save",
"Save As", "Setup Printer", "Print", and "Exit". The defini-
tion and usage of these operations follow the usual conventions
in modern user-interface designs.
The File menu appears underneath the File button on the main
menu bar when the File button is selected and clicked. As long
as the File button remains selected, users can navigate through
the File menu. Menu item selection is accomplished by pointing
to an item in the menu and then releasing the mouse button.
2. Editing Operations
Users will have available to them four editing operations which
allow them to modify the currently displayed work space. These
are:
1. Undo - allows users to undo the most recent CASRE operation.
2. Convert Type - allows users to convert failure data in the
form of interfailure times to failure counts and test interval
lengths, and vice versa.
3. External Editor - allows users to invoke a preferred text edi-
tor, word processor, or other application from within CASRE.
The application is selected from a user-configurable menu.
4. Escape to DOS - allows users to temporarily escape to DOS and
execute DOS commands. Users may then re-enter CASRE by typing
"exit" at the DOS prompt.
3. Transformation and Smoothing Operations
Filtering operations allow users to perform global modifications to
the failure data displayed in the work space. Since failure data
is frequently noisy, and may be noisy to the point that it is dif-
ficult or impossible to estimate model parameters, users may wish
to apply a smoothing operation to the data to remove some of the
noise. Users may also wish to apply other transformations which
will allow them change the shape and position of the failure data.
For instance, one of the transformations can change exponential
data to a straight line.
There are three types of filtering operations; these are transfor-
mation, smoothing, and data subsetting. Multiple transformation
and smoothing filters may be applied to the data in a pipeline
fashion. There is also a "Undo All Filters" facility allowing
users to remove all of the filters that have been applied to the
data. The filtering capabilities are as follows:
o Affine Transformations
- Scaling and Offset - (X(i) * A) + B
- Power - X(i) ^ A
- Logarithmic - ln(A * X(i) + B)
- Exponential - exp(A * X(i) + B)
o Smoothing (to use "Hann Window")
o Subset Data (to select severity)
o Remove All Filters
4. Model Selection and Application
CASRE allows users to select and apply existing software reliabil-
ity models to the data displayed in the work space. These models
include Farr Survey
1. Brooks and Motley Model (BM)
2. Geometric Model (GM)
3. Goel-Okumoto (GO)
4. Jelinski-Moranda (JM)
5. Littlewood Model (LM)
6. Littlewood-Verrall (LV)
7. Musa-Okumoto (MO)
8. Generalized Poisson Model (PM)
9. Schneidewind Model (SM)
10. Yamada Delayed S-Shape Model (YM)
Users are also allowed to define combinations of existing models,
edit specifications of user-defined models, and remove user-defined
models from the menu of available models. Users may also specify
the parameter estimation method, the confidence bounds that will be
reported for model parameters, and the amount of time into the
future for which reliability predictions will be made. The Model
menu items are:
1. Model Selection - allows users to select and apply one or more
software reliability models to the failure data displayed in
the work space.
2. Model Definition - allows users to define combination models
to supplement models already provided with the tool. User-
defined models remain available during the current and subse-
quent sessions.
3. Model Editing/Model Removal - allows users to change or remove
descriptions of combinations that were previously created
using the "Model Definition" capability. Only user-defined
combinations of models can be changed or removed.
4. Parameter Estimation - allows users to select the method of
parameter estimation that will be used. The choices are max-
imum likelihood (default) and least squares.
5. Predictions - allows users to specify an interval of time over
which predictions about future reliability behavior will be
made.
5. Program Configuration
Two operations, "Add External Configuration" and "Remove External
Configuration," are available from this manual. The former allows
users to add the name of an application to the "External Editor"
submenu from which external applications can be invoked. The later
allows users to remove the name of an application from the "Exter-
nal Editor" submenu from which applications can be invoked.
6. Help System
The help system provides context-sensitive on-line assistance to
users by allowing them to search for and read descriptions of the
major CASRE functional areas.
7. Displays
The graphics display window is invoked when the "Open" button under
main window's "File" menu operation is selected. The graphics
display window plots the failure data displayed in the work space
and the results of applying models to that data. A separate menu
bar is associated with this window, allowing users to control the
contents and appearance of the display or send the contents of the
window to an output device (disk file or printer). The menu items
on the main menu bar are:
1. Plot - There are 6 plot operations ("Save", "Save As", "Redraw
Data", "Redraw Results", "Draw from File", "Printer
Setup/Print") that can be selected for file manipulation and
drawing.
2. Model Result Selection - Multiple sets of model results can be
displayed in the graphics display window. This capability
allows users to specify the models whose results should be
plotted in that window.
3. Display Type - Modeling results and evaluations of models can
be displayed in a variety of ways. The following types of
model results can be plotted in the graphics window:
o Interfailure times
o Number of failures per test interval
o Test interval lengths
o Cumulative number of failures
o Reliability growth function
In addition to estimating reliability functions, users may wish
to determine the applicability of the model(s) being run to the
current set of failure data. The following types of model
evaluation results can be drawn in the graphics window:
o Goodness-of-fit tests - Chi-Square or Kolmogorov-Smirnov
o Current value(s) of the prequential likelihood
o Prediction noisiness
o u-plot
o y-plot
o Scatter-plot of u's
o Model rankings
4. Scaling - allows users to scale the x and y axes of the graph-
ics display window and to shift its origin.
5. Table - allows users to view the modeling results in the
graphics display window in a tabular form. This table shows
the detailed data on which plotted model results, evaluations,
and rankings are based.
CASRE On-Screen Appearances
Figures 1-8 in the following show a series of screen dumps for the
described CASRE tool. It can be seen that the application of
models to failure data is a straightforward process. Users are
also given a considerable amount of choice in the models to be
applied. This combination of simple operation and variety in the
available models makes it easy for users to identify an appropriate
model for a particular development effort or investigate a family
of models.
o Screen 1 - opening a failure data file
The screen is shown in Figure 2 in the Appendix. To choose a set
of failure data on which a reliability model will be run, users
select the "File" menu with the mouse. After selecting the "Open"
option in the File menu, a dialogue box for selecting a file
appears on the screen. The current directory appears in the edit-
able text window at the top of the dialogue box. The failure his-
tory files in that directory are listed in the scrolling text win-
dow. Users select a file by highlighting its name (scrolling the
file name window if necessary) and then pressing the "Open" button.
To change the current directory, users enter the name of the new
directory in the "Current Directory" window and presses the "Change
Directory" button. Pressing the "Cancel" button removes the dialo-
gue box from the screen.
o Screen 2 - initial failure data display
The screen is shown in Figure 3 in the Appendix. After opening a
failure history file from the "File" menu, the contents of the file
are displayed in tabular and graphic forms. The tabular represen-
tation resembles a spreadsheet, and users can perform similar types
of operations (e.g. selecting a range of data, deleting one or more
rows of data). All of the fields can be changed by users except
for the "Interval Number" field (or "Error Number" field if the
data is interfailure times). In this example, the selected data
set is in the form of test interval lengths and number of failures
per test interval. Users can scroll up and down through this tabu-
lar representation and resize it as per the MOTIF conventions.
The large graphics window displays the same data as the worksheet.
If the failure data set is interfailure times, the initial graphi-
cal display is interfailure times. If, as in this example, the
failure data set is test interval lengths and failure counts, the
initial graphical display is the number of failures per test inter-
val. The display type can be changed by selecting one of the items
from the "Display Type" menu associated with the graphics window.
Users can move forward and backward through the data set by press-
ing the right arrow or left arrow buttons at the bottom of the
graphics window. Finally, the iconified window at the lower left
corner of the screen lists the summary statistics for the data. To
open this window, users just clicks on the icon.
o Screen 3 - selecting failure data range
The screen is shown in Figure 4 in the Appendix. Users will fre-
quently use only a portion of the data set to estimate the current
reliability of the software. This is because testing methods may
change during the testing effort, or different portions of the data
set may represent failures in different portions of the software.
To use only a subset of the selected data set, users may simply
"click and drag" on the tabular representation of the data set to
highlight a specific range of observations. Users may also select
previously-defined data ranges. To do this, users choose the
"Select Range" option of the Edit menu. This brings up a dialogue
box containing a scrolling text window in which the names of
previously-defined data ranges and the points they represent are
listed. To select a particular range, users highlight the name of
the range in the scrolling text window and presses the "OK" button.
Pressing the "Cancel" button removes the dialogue box and the Edit
menu from the screen.
Once a range has been selected, all future modeling operations will
be only for that range. The selected data range is highlighted in
the tabular representation. The graphics display will change to
include only the highlighted data range. All other observations
will be removed from the graphics display.
o Screen 4 - data filtering
The screen is shown in Figure 5 in the Appendix. After selecting a
data range, users may wish to transform the file or smooth the
data. Software failure data is frequently very noisy; smoothing
the data or otherwise transforming it may improve the modeling
results. To do this, users select one of the options in the
"Filter" menu. There are five affine transformations which users
may apply to the data, and six types of smoothing. Transformations
and smoothing operations may be pipelined - for example, users
could select the "ln(A * X(i) + B)" transformation followed by the
B-spline smoothing operation. The number of filters that may be
pipelined is limited only by the amount of available memory. The
tabular representation of the failure is changed to reflect the
filter, as is the graphical display of the data. The type of
filter applied to the data is listed at the right hand edge of the
graphics display window.
Once a series of filters has been applied to the data, users may
remove the effect of the most recent filter by selecting the "Undo"
option of the Filter menu. To remove the effect of the entire
series of filters, users select the "Undo All Filters" option of
the Filter menu.
o Screen 5 - applying software reliability models
The screen is shown in Figure 6 in the Appendix. After users have
opened a file, selected a data range, and done any smoothing or
other transformation of the data, a software reliability model can
be run on the data. In the Model menu, users have the choice of 10
individual models or a set of models which combine the results of
two or more of the individual models. Users may also choose the
method of parameter estimation (maximum likelihood or least
squares), the confidence bounds that will be calculated for the
selected model, and the interval of time over which predictions of
future failure behavior will be made.
o Screen 6 - prioritization of model selection criteria
The screen is shown in Figure 7 in the Appendix. There are many
models from which to choose in this tool. Users may not know which
model is most appropriate for the data set being analyzed. Using
CASRE, users can request, "display the results of the individual
model which best meets the four prioritized criteria of accuracy
(based on prequential likelihood), biasedness, trend, and noisiness
of prediction." To do this, a user first selects the "Individual"
option of the Model menu. A submenu then appears, on which 10
individual models are listed, as well as a "Choose Best" option.
The user selects the "Choose Best" option, which results in a
"Selection Criteria" dialogue box being displayed. The user moves
the four sliders in this dialogue box back and forth to establish
the relative priorities of the four criteria. Numerical values of
the priorities are displayed in the text boxes on the right side of
the dialogue box. Once the priorities have been established, the
user presses the "OK" button. CASRE then proceeds to run all of the
individual models against the data set, first warning the user that
this is a time-consuming operation and allowing cancellation of the
operation. If the user continues, CASRE provides the opportunity
for cancellation at any time if the user decides that the operation
is taking too much time.
o Screen 7 - display of model results
The screen is shown in Figure 8 in the Appendix. Once a model has
been run on the failure data, the results are graphically
displayed. Actual and predicted data points are shown, as are con-
fidence bounds. The model is identified in the window's title bar;
the percent confidence bounds are given at the right side of the
graphics window. This concludes one round of software reliability
estimation with CASRE.
o Screen 8 - determination of model bias
The screen in which one of the result of model evaluation can be
displayed is shown in Figure 9. Statistical methods can be applied
to determine the applicability of the model to the failure data set
on which it was executed. To display the evaluation results, a
user selects the "Evaluations" pull-down menu in the graphics
display window's main menu bar. Several model evaluation methods,
including u-plots and y-plots, are available to users. In this
example, the user have chosen the "U-Plot" menu item. The u-plot,
which indicates biases in the model, is displayed on screen. CASRE
also indicates whether the model has an optimistic bias (predic-
tions of time to the next failure tend to be greater than observed
inter-failure times) or a pessimistic bias. To return to the
display shown in the figure, the user may select the "Display Type"
pull-down menu and choose the desired type of reliability-related
display.