Docsity
Log inSign up
Docsity
Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity

Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan

Guidelines and tips
Guidelines and tips
Sell on Docsity
Sell on Docsity
Log inSign up

Prepare for your exams

Study with the several resources on Docsity

Find documents

Prepare for your exams with the study notes shared by other students like you on Docsity

Search Store documents

The best documents sold by students who completed their studies

Search through all study resources
Docsity AINEW

Summarize your documents, ask them questions, convert them into quizzes and concept maps

Explore questions

Clear up your doubts by reading the answers to questions asked by your fellow students

Earn points to download

Earn points by helping other students or get them with a premium plan

Share documents
download point icon20 Points

For each uploaded document

Answer questions
download point icon5 Points

For each given answer (max 1 per day)

All the ways to get free points
Get points immediately

Choose a premium plan with all the points you need

Study Opportunities

Choose your next study program

Get in touch with the best universities in the world. Search through thousands of universities and official partners

Community

Ask the community

Ask the community for help and clear up your study doubts

University Rankings

Discover the best universities in your country according to Docsity users

Free resources

Our save-the-student-ebooks!

Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors

From our blog

Exams and Study
Go to the blog

Analyzing the Predictive Accuracy of Software Reliability Models and Recalibrating to Improve Upon Predictions | CS 777, Study notes of Computer Science

Pace UniversityComputer Science
Prof. Orlena C. Gotel

Material Type: Notes; Professor: Gotel; Class: Software Reliability and Quality Assurance; Subject: Computer Science; University: Pace University-New York; Term: Unknown 2000;

Typology: Study notes

Pre 2010

Uploaded on 08/09/2009

koofers-user-r9g
koofers-user-r9g 🇺🇸

5

(1)

10 documents

1 / 34

Toggle sidebar

This page cannot be seen from the preview

Don't miss anything!

bg1
Analysing the Predictive
Accuracy of Software Reliability
Models & Recalibrating to
Improve Upon Predictions
NOTE: This material is drawn from Chapter 4 of Lyu & supplemented
by the tutorial material of Prof Bev Littlewood & Pete Mellor (of CSR,
City University, London) – with the kind permission of Pete Mellor
Slides to accompany CS777 discussion
pf3
pf4
pf5
pf8
pf9
pfa
pfd
pfe
pff
pf12
pf13
pf14
pf15
pf16
pf17
pf18
pf19
pf1a
pf1b
pf1c
pf1d
pf1e
pf1f
pf20
pf21
pf22

Partial preview of the text

Download Analyzing the Predictive Accuracy of Software Reliability Models and Recalibrating to Improve Upon Predictions | CS 777 and more Study notes Computer Science in PDF only on Docsity!

Analysing the Predictive

Accuracy of Software Reliability

Models & Recalibrating to

Improve Upon Predictions

NOTE: This material is drawn from Chapter 4 of Lyu & supplemented by the tutorial material of Prof Bev Littlewood & Pete Mellor (of CSR, City University, London) – with the kind permission of Pete Mellor Slides to accompany CS777 discussion

SYS1 – Plotting Time Between

Individual Failures

ti inter-failure time, in seconds Failure number, i

SYS1 – Plotting Cumulative Failure

Over Time

Cumulative number of failures Total elapsed time, in seconds How many failures have been seen Total execution time so far

Is the Truth Out There?

Crude Way to Judge Whether

Median Plots are Reasonable

Are 50:50 Odds Good Enough?

SYS1 - 20 Step-ahead Predictions

30 40 50 60 70 80 90 100 120 130 140 median 1000 2000 3000

JM

LV

LV

(as in previous 1-step ahead prediction) i Failure number, i Estimated median time to next failure, in seconds

Methods to Analyse Predictive

Accuracy (1 Step-ahead)

  • At stage i, we have prediction of the distribution of time to next failure, F i (t)
  • We want this to be close to the unknown true distribution, F i (t)
  • Observe what actually happens, t i
  • Repeat for many i in a time period
  • Sequence of (F i (t), t i ) tells us about accuracy of predictions ^ ^ T i

The u-plot

• For each prediction, calculate:

u i = F i (t i )

• u-plot is sample distribution function of the

u

i

s

• Tells us about what happens “on average”

^

Prediction Observation

SYS1 – u-plots for 3 Models

Jelinski-Moranda (JM) Littlewood (LM) Littlewood-Verall (LV) KS distance 1-step ahead predictions, 100 plots Below line of unit slope – pessimistic predictions Above line of unit slope – optimistic predictions

True Probability Density Function

  • Probability distribution of T i over values it may assume
  • f(t)>=0 for all t; ∫f(t)dt= t true i f (^) ( t )

Prequential Likelihood

• To compare several competing sets of

predictions on the same data source

• Select the one which has given the

globally most accurate predictions

• Detect consistent bias & inappropriate

noise in a prediction system

Prequential Likelihood Function

• Problem: estimating true cdf F

i

(t) of T

i

, on

the basis of the observed t

1

, t

2

, ...t

i-

• Apply prediction system A to a sequence

i=m through i=n

• After some time from each prediction you

observe actual t

i

• The prequential likelihood function for

these predictions is

PL = f ˆ

i

( t

i

)

i = m

i = n

∏