Subject: Reliability with redundancy

From: [email-masked] (Nick Pine)
Date: 20 Sep 1995 08:40:26 -0400

There has been some discussion here lately about electrical network
reliability. With increasing deregulation, some electrical utilities have
made some economic decisions to save money at the expense of decreased
network reliability. For instance, some utilities deliberately undersize
new transformers in power-handling capability, with secondary fuses rated
for, say 10 times the sum of the primary loads. So, if all the customers
attached to a transformer, from 1-5 houses, where I live, were to turn on
all of their lights, electric stoves, etc, at once, the secondary voltage
would go down and the transformer might actually burn up before the
safety fuse on the pole blew. The fuses are there for fault safety,
not sustained load.

Utilities recognize that sustained maximum loads like this rarely happen,
perhaps only once in 10 years, and it is less expensive to suffer an outage
in this rare event (especially if people don't want to pay much for
reliability, as reflected in PUC policies) than to spend money providing
a more conservatively-designed, "worst-case" system, like an aircraft system.
So there is a small but non-zero failure rate built into this utility policy.
Of course additional failures result from acts of God.

Since this is SCI.energy, I'd like to talk for a moment about how to
calculate the reliability of redundant systems, eg 2 vs. 1, if either
can do the job, and we know when one breaks, and we fix it. Having
2 subsystems (I would say "2 redundant subsystems," but that would be
redundant :-) can greatly increase the reliability of an overall system,
roughly squaring the MTTF. MTTF means "Mean Time To Fail," and in most
systems, it is very close to the MTBF, the Mean Time Between Failures, which
is a little bit longer, because it includes the repair time of a system.
If a system works for a long time, then it fails, then it is fixed in a
relatively short time, and then it works for a long time again, the MTTF
is the average amount of time that it stays working, and the MTBF is the
average time between failures, including the average repair time, MTTR. 

If a single system with no redundancy has a 10,000 hour MTBF, it fails about
once every 10,000 hours, ie the average failure rate is 1/10,000 hours, or
about once a year. If fixing it takes an average of 8 hours (counting the
time for the repairperson to get some parts and drive to the system, etc.)
then we'd expect it to be down for about 8 hours a year. This corresponds
to an Availability, or probability that the system is working, of

A = MTBF/(MTTF+MTTR) = 10,000/(10,000+8) = 0.992 or an Unavailability of

U = 1-A = 0.0008,

which means that the system is not working about 0.08 % of the time. In
a year, we would expect the overall system to be down, ie unavailable, for
0.0008 x 366 days x 24 hours/day = 7 hours (which is a bit less than
8 hours because there are only 8766 hours in an average year.) Most
people are used to better electrical service than this, and much better
telephone service... US Telephone companies calculate and budget and try
to deliver user unavailabilities measured in seconds per year. They try to
design telephone networks so that an average telephone is out of order
less than 10 seconds a year. That may increase too, under deregulation.

What happens if we have 2 electrical systems? A power line and a solar
electric system, or a power line and a standby generator, or a power line
and a UPS, or a standby generator and a wind generator with some batteries
and an inverter. Or 2 telephone systems, or 2 cars, or 2 of any sort of
systems, (eg 2 heads, better than one?) If the systems are relatively
independent, and either can do the job by itself, the the reliability and
availability of the combined system goes up, the cost probably goes up, and
the unavailability can go down, which is good. For this to work well,
we have to know when one system is broken (an "alarm"), and we have to
fix it fairly quickly when it breaks. For this discussion, we might
say that a solar electric system is temporarily "broken," ie unavailable
when the batteries are discharged, or more permanently broken, in the
usual sense, when something goes wrong with the inverter. 

Suppose we have 2 electrical power systems, each with an MTTF of 10,000
hours and an MTTR of 8 hours, and they are independent, and we can easily
and quickly switch from one to the other, and we know when one is broken
and start fixing it right away, the overall system can be in one of 3
conditions: 2 systems are working, only 1 is working, and 0 are working,
ie both broken. This leads to the following simple Markov model and state
diagram for the overall system:

 ---------    2L    ---------    L     ---------
|     2   |------->|    1    |------->|    0    |
| (both   |        | (one    |        | (none   |
| working)|<-------| working)|<-------| working)|
 ---------    M    ---------     2M    ---------

At a particular time, the system can be only in 1 of 3 states, numbered 2
1 and 0. Each state has an associated probability, P(2), P(1) and P(0).
In states 2 and 1, we have electrical power. In state 0, we don't.

The unavailability of the system is P(0). We want to minimize that.
The model has state transition rates based on the failure rate L=1/MTTF
and repair rate M=1/MTTR of a subsystem. Since both subsystems are working
in state 2, and either can fail, the transition rate from state 2 to
state 1 is twice the rate of the transition rate from state 1 to 0. The
repair rate from state 0 to 1 is twice the repair rate from state 1 to 2,
assuming 2 repairpersons are at work, when both subsystems are broken. 

The unavailability is easy to calculate: since the overall system can only
be in 1 state at a time, and there are only 3 states, the probabilities
have to add up to 1:

P(2) + P(1) + P(0) = 1                  (1), and from the transition rates, 

P(1) = 2L/M P(2)                        (2), and

P(0) = L/(2M) P(1)                      (3), so

M/(2L) P(1) + P(1) + P(0) = 1           (4), from (1) and (2), and

M/(2L) 2M/L P(0) + 2M/L P(0) + P(0) = 1 (5), from (4) and (3), so

P(0) = 1/(M^2/L^2-2M/L+1).

We could do this calculation exactly, but the first term in the denominator,
M^2/L^2, is over a million, and the second term is 2,500 and the third is 1,
which is how these calculations usually turn out, so we can approximate
P(0) = 1/(M^2/L^2) = (L/M)^2 = (MTTR/MTTF)^2. In our example, this is
0.00000064. So the expected number of seconds a year of electrical outage,
when  neither  system is working, is

0.00000064 x 8766 hr/yr x 60 min/hr x 60 sec/min = 20 seconds.

Much better than 8 hours. Almost as good as the telephone company :-)

Nick

---

From: [email-masked] (Nick Pine)
Date: 22 Sep 1995 05:17:36 -0400

Jim Fitzpatrick <[email-masked]> wrote:
>[email-masked] (Nick Pine) wrote:
>
>> A = MTBF/(MTTF+MTTR)
>
>no it isn't, A=MTTF/(MTTF+MTTR), since MTBF=(MTTF+MTTR)

Oops. Yes. Thanks for correcting this, Jim. Gee, somebody actually reads
these postings :-) Of course, the numbers are almost the same. BTW, there
was also a wrong sign in the original, and I've added a bit more explanation:

>>U = 1-A = 0.0008, which is approximately the MTTR/MTTF.

>>P(0) = 1/(M^2/L^2+2M/L+1).
                   ^
                   | this was a - , in error.

>>P(0) = 1/(M^2/L^2) = (L/M)^2 = (MTTR/MTTF)^2. 

Sometimes I leave errors in postings to see if people read them closely.
Few do. More often I don't see them myself until later.

>>Most people are used to better electrical service than this
>
>how do you know? I have no idea how reliable my home electrical suppy
>is since I spend a large proportion of my year working 25 miles away.
>I have no idea how reliable my work electrical supply is because I
>spend a large proportion of my time at hone asleep! Unless you are in
>one place 24 hours a day 365 days a year with electrical appliance on
>(or, you have statistics supplied by the utilities....) you can't know.

I know when the power fails at home because I walk in and see a few
clocks blinking, the microwave oven, etc. I can tell how long the power
was off by how slow the clocks with hands are running. I know when
the power fails during the night in an office, because the net is down
the next morning, the system I left running all night rebooted, etc.
I have a odd friend who lives near a river, with such miserable service
that he has a battery system with inverters to keep his 11 PCs running.
(His hobby is weather prediction.) People make power line monitors to
detect even small outages. Dranetz (?)

>> and much better telephone service
>
>ditto

Yes, that's harder to tell, unless you use them a lot. 

>> the expected number of seconds a year of electrical outage,
>> when  neither  system is working, is .... 20 seconds
>
>this is fine as it stands, but it neglects one of the most important
>and troublesome aspects of reliability modelling - common cause
>(or common mode) failures. No matter how different your two systems
>may appear to be, they are not independent. At best, you could
>describe them as 'partially diverse'. There will be commom elements in
>both, such as contact breakers, fuses, even the wiring itself.

That's a deliberate neglection.

I think of this as a matter of degree. And some systems are easier and
cheaper to make redundant than others, and not uncoincidentally, more
reliable when made redundant that others. I'm thinking of power supplies
with diode-ORed outputs, or some other kind of paralleled outputs. Power
supplies are sometimes the most unreliable parts in a computer system,
and sometimes the items that can be most easily and reliably paralled. If
the switchover circutry were no more than two diodes, it would probably 
be a lot more reliable than the rest of a switching supply. 

I like to think about the marginal return on an investment in reliability:
where would you spend the first dollar, in making a system more reliable?
Where would you spend the next one? How hard is the switchover? How
fine-grained should the redundancy be? It's an interesting game to play
with a block diagram... And how does the cost of making a system more
reliable compare to the cost of a failure? I like to use numbers for this.

It seems to me that systems that handle significant power, eg AC inverters
(vs say, computers), often have costs that are proportional to their power-
handling capability, eg in their semiconductors and heat sinks and
transformers (vs. their cheap chip controllers), and easy switchover schemes
(compared to computers.) To me, that means that a redundant power supply
system which keeps working if N-1 out of N supplies work, may not cost a
lot more than one single larger supply, and it might be a lot more reliable.
It is easier to show this with numbers than words.

>The presence of common elements means that both systems could fail for the
>same reason. Take the example of two diesel generators, one running
>and one standby. This appears to be a redundant system, until you
>realise that they both require diesel oil, and this oil will very
>likely come from one source.

The Middle East?

>Dirty diesel oil will cause both generators to fail for the same reason,
>and at the same time (and this is a real example...). So, you cannot assume
>that all failures are independent.

I agree. And I'm not assuming that, but as a matter of degree, if things
can be arranged so that there are very few common elements compared to the
more complicated subsystems, with a much lower failure rate for the
common elements, then one is ahead of the game, however you would like
to describe the game in words. (Don't oil filters help?)

>Common cause failures are notoriously difficult to model. There is a
>whole branch of reliability modelling devoted to it.

I've seen some of that. Complicated, sometimes. 

>A simple way of modelling them is to say that a proportion of all failures
>in a system will be due to common causes. In a single system, of course, they
>wouldn't be recognised as such. Typical figures for this proportion
>are 20% for redundant systems, 6% for partially diverse systems, and
>2% for full diversity.

Interesting typical figures.

>In order to determine the unavailability due to common causes, this
>simple model allows us merely to take this proportion of the unavailability
>of a single system. In your example, the unavailability of a single system
>is 8E-4. The proportion of this that is due to common causes is 20%
>(for a redundant system)

It is?

>or an unavailability of 8E-4 x 0.2 = 1.6E-4, which equates to 1.4 hours
>per year. If we could argue partial diversity, then the systems would be
>unavailable for 25 minutes per year.

I'm thinking more about full diversity, with an attempt at clever design,
and picking the right things to diversify. 

>Of course, if the failures are coincident (cf the diesel generators)
>then the downtime is more likely to be the repair time, i.e. back to
>8 hours.

One would hope to avoid that in the design, no?

>Much more realistic, don't you think?

No.

>So, throwing lots of money at the problem in order to provide backup systems
>does not have the dramatic effect you suggested...

Aha, a conclusion :-)

Nick

---

From: [email-masked] (Nick Pine)
Date: 23 Sep 1995 07:21:19 -0400

>Do I win a prize?  :-)

You have my gratitude for finding and fixing the mttf/bf error,
but not for finding the minus sign...

>You seem to have identified the need for a cost-benefit analysis.

I'm thinking more about design, ie synthesis, than analysis. "Marginal
economic availability synthesis"? Seems like it could be done: take each
piece of a block diagram and ask questions like these:

"Suppose this block were doubly-redundant, or (N-1)/N redundant, or...?"
"What would be the new cost of the components that do that job?"
"Would they each do the whole job or share the job or be hot standbys?"
"How does the system performance degrade if one fails?"
"How would the alarm (if any) and switchover mechanisms work?"
"What would they cost, and how reliable would they be?"
"What would the unavailability improvement of the overall system be
 with this particular change?"

One might systematically go over a whole block diagram this way, with
some computer assistance, and explode each block down to the lowest level
and do it again and again. Not easy, and there are local optima. It would
probably be more complicated than a FMECA.

>I rather suspect that, in some instances at least, the benefit would be
>found to outweigh the cost. However, since (going back to the original
>scenario) it is the customer who is inconvenienced by a power outage,
>and 'captive' customers don't count for a bean in the eyes of a
>virtual monopoly, the money isn't spent.

OTOH, regulated utility monopolies sometimes have an public arrangement
where they earn a percentage of what they manage to spend on equipment, etc.
So if they decided to go bananas about reliability, we might have gold-
plated wires everywhere, at the public's expense. But in the US, we
seem to be going in the other direction.

>>if things can be arranged so that there are very few common elements 
>>compared to the more complicated subsystems, with a much lower failure 
>>rate for the common elements, then one is ahead of the game
>
>why not just build your original system with components that have a
>'much lower failure rate'?

That sounds like a good idea. What do they cost? What is the most
cost-effective arrangement to achieve a certain level of unavailability?
 
>Look up the 'Beta Factor' model.

Thanks. I may do that, if somebody pays me to do reliability work again :-) 

>You can never get full diversity. No matter how different your systems
>may be, you'll still have wiring, switches, fuses...etc., in both. You
>can almost get there - for example, buy different types of switches
>from different manufacturers (in different countries, or on a
>different planet, if you like). But in the end, they're still
>switches, and so not fully diverse.

I have a nutty friend who lives in New York City. He just bought about 5 kW
of solar PVs and an inverter. He's planning to fill up his house with
batteries and a whole other set of 110V outlets, wiring, fuse boxes, etc.
(I think he still heats his house with oil :-) Now assuming the outlets
and wiring, etc. are in different places, that seems pretty diverse to me.

Of course, the battery explosions may not be good for the his grid
system wiring, and if his house catches on fire, one or both electrical
systems may stop working, and if a black cloud covers New York for a month,
everyone in the city will use lights more during the day, and that will put
some sort of strain on the grid system, as well as his solar system, and
if a tornado tears his house into little bits, or someone nukes New York,
both electrical systems may stop working, but will he care? Where do you
draw the line? How do you define the universe, in the probabilistic sense,
for common causes?

Nick