Iformatio ad Computatio 170, 184 206 (2001) doi:10.1006/ico.2001.3054, available olie at http://www.idealibrary.com o Probabilistic Quorum Systems Dahlia Malkhi School of Computer Sciece ad Egieerig, The Hebrew Uiversity of Jerusalem, Israel E-mail: dalia@cs.huji.ac.il Michael K. Reiter ad Avishai Wool Bell Labs, Lucet Techologies, Murray Hill, New Jersey E-mail: reiter@research.bell-labs.com, yash@acm.org ad Rebecca N. Wright AT&T Labs Research, Florham Park, New Jersey E-mail: rwright@research.att.com Received March 22, 1999; accepted Jue 6, 2000 We iitiate the study of probabilistic quorum systems, a techique for providig cosistecy of replicated data with high levels of assurace despite the failure of data servers. We show that this techique offers effective load reductio o servers ad high availability. We explore probabilistic quorum systems both for services tolerat of beig server failures ad for services tolerat of arbitrary (Byzatie) oes. We also prove bouds o the server load that ca be achieved with these techiques. C 2001 Academic Press 1. INTRODUCTION Quorums are tools for icreasig the availability ad efficiecy of replicated data. A quorum system is a set of subsets of servers, every two of which itersect. Ituitively, the itersectio property guaratees that if a write operatio is performed at oe quorum, ad later a read operatio is performed at aother quorum, the there is some server that observes both operatios ad therefore is able to provide the up-to-date value to the reader. Thus, system-wide cosistecy ca be maitaied while allowig ay quorum to act o behalf of the etire system. Compared with performig every operatio at every server, usig quorums reduces the load o servers ad icreases service availability despite server crashes. Quorum systems are traditioally assessed by three measures of quality: load [NW98], fault tolerace [BG86], ad failure probability (see [BG87, PW95]). The load of a quorum system is a measure of its efficiecy: it is the rate at which the busiest server is accessed. The fault tolerace of a system is the maximum umber of server failures for which there is still guarateed to be a quorum cotaiig o faulty servers. The failure probability is the probability that every quorum cotais a faulty server, assumig that servers fail idepedetly with a fixed probability. (Load, fault tolerace, ad failure probability are defied precisely i Sectio 2.) The fault tolerace of ay quorum system is bouded by half of the umber of servers. Moreover, the failure probability typically icreases to 1 whe the idividual failure probability of servers exceeds 1/2. Also, there is a trade-off betwee low load ad good fault tolerace, ad i fact it is impossible to achieve optimality i both of them simultaeously. To circumvet these limitatios, we relax the itersectio property of a quorum system to allow quorums chose accordig to a specific access strategy to fail to itersect with some small probability ε. Accordigly, we call these ε-itersectig quorum systems ad, though a abuse of termiology, cotiue to refer to the chose sets of servers as quorums. We heceforth refer to systems that satisfy 184 0890-5401/01 $35.00 Copyright C 2001 by Academic Press All rights of reproductio i ay form reserved.
PROBABILISTIC QUORUM SYSTEMS 185 the origial defiitio of quorums as strict quorum systems. We the exted the defiitio of the three quality measures load, fault tolerace, ad failure probability to address the probabilistic ature of our set systems. By these measures, probabilistic quorum systems show dramatic improvemet over strict quorum systems: Allowig eve a small probability ε of oitersectio yields a clear advatage i the fault tolerace ad failure probability of the system, while the load remais uchaged or improves. We study probabilistic quorum systems further, i a model where servers may exhibit arbitrary (Byzatie) failures. Malkhi ad Reiter [MR98a] adapted strict quorum systems to the task of maskig Byzatie failures to improve the efficiecy of Byzatie fault-tolerat data replicatio. They itroduced b-dissemiatio quorum systems, i which ay two quorums itersect i at least b + 1 servers, to mask the arbitrary failure of b servers for self-verifyig data ad b-maskig quorum systems, i which two quorums itersect i at least 2b + 1 servers, to mask b arbitrary server failures for arbitrary data. I this paper, we also explore relaxig the itersectio properties of these quorums to achieve (b,ε)-dissemiatio ad (b,ε)-maskig systems. Agai, we show that these systems offer substatial improvemets over their strict couterparts i the measures described above. 1.1. Applicatios Due to their relaxed itersectio properties, our probabilistic quorums are most suitable for use whe the cosistecy of replicated data may be relaxed to achieve greater availability of that data. Below we describe several examples of applicatios where this trade-off is justified. The first applicatio arose i the cotext of a electroic votig system desiged by AT&T Labs for the coutry of Costa Rica. I this system, each voter is give a uique voter idetifier whe he or she registers to vote. O electio day, the voter presets this voter ID to ay oe of over 1000 votig statios spread across the coutry i order to cast his or her vote. To prevet a voter ID from beig used multiple times i oe electio, it is ecessary to lock each voter ID coutry-wide whe it is preseted at ay votig statio. I order to preserve the itegrity of the electio, it is oly ecessary to prevet large-scale repeat votig. Therefore, it suffices for each repeated use of the same voter ID to be detected with high probability, so umerous repeat attempts will be detected with virtual certaity. We thus adopted a protocol amog votig statios that uses probabilistic quorums for this purpose. Moreover, by usig our dissemiatio or maskig quorum costructios i the lockig protocol, repeat usage of a voter ID ca be preveted eve if some umber of votig statios do ot follow the lockig protocol (e.g., some statios have bee altered by bribed electio officials). At the same time, the use of our probabilistic quorum costructios esures that the electio progresses eve i the presece of beig failures of sigificat umbers of votig statios. A prototype implemetatio of the Costa Rica electroic votig system was built over the Phalax system [MR98b]. This implemetatio made use of various probabilistic quorum systems, icludig maskig systems, for lockig voter IDs. The secod applicatio is maitaiig the locatio of a mobile device, such as a cellular telephoe. The locatio of a mobile device ca be recorded i a variable that is replicated at several locatio stores. This variable is updated (e.g., by the device itself) usig a quorum-based protocol amog the locatio stores whe the device moves from cell to cell (cf. [HL99]). The ability of callers to access this iformatio, eve at the risk of it beig stale, is the primary requiremet for this applicatio. A caller that receives stale locatio iformatio ca be forwarded by the stale cell to a more recet cell for that device, but the caller ca make o progress if it receives o iformatio about the device s curret or recet whereabouts due to locatio store failures. Fially, we ote that a system built with probabilistic quorum systems ca be stregtheed by a properly desiged diffusio mechaism, which propagates updates to replicated data lazily, i.e., outside the critical path of cliet operatios. Diffusio methods (also kow as epidemic or gossip protocols) have bee studied for both beig failure eviromets [DGH + 87, AES97] ad Byzatie eviromets [MMR99]. Coupled with a diffusio mechaism, the probability of icosistecy usig probabilistic quorum costructios ca be drive further toward zero whe updates are sufficietly dispersed i time, makig probabilistic quorum costructios useful i a wider variety of settigs.
186 MALKHI ET AL. 1.2. Related Work Strict quorum systems have bee extesively studied ad measured (cf., [Gif79, Tho79, Mae85, GB85, Her86, BG87, ET89, CAA90, AE91, NW98, PW97, PW95]). Byzatie quorum systems were itroduced i [MR98a] ad further studied i [MRW00, Baz97]. A prelimiary versio of the work i Sectios 2 4 was preseted i [MRW97]. Because of the possibility of icosistecy admitted by probabilistic quorum systems, they are most attractive for systems i which some level of icosistecy ca be tolerated, ad i particular, where the efficiecy ad availability gaied outweigh the cost of hadlig such icosistecies. Other approaches to relaxig cosistecy i such applicatio domais have bee proposed. For example, Hayde ad Birma implemeted pbcast [HB95], a probabilistic broadcast with relaxed reliability guaratees for buildig fault-tolerat distributed applicatios. Malkhi et al. [MMR97] proposed a probabilistic secure broadcast with a relaxed cosistecy property for securely replicatig services i a very large etwork. I the domai of replicated database systems, several approaches that relax the strict serializability guaratee have bee proposed. The goal of these efforts has bee to decrease the cotetio betwee user trasactios ad hece to icrease the cocurrecy ad decrease the abort rate. Krishakumar ad Berstei [KB94] suggest N-bouded-igorace, a relaxed cosistecy coditio that permits N + 1 coflictig trasactios to be performed cocurretly. Pu ad Leff [PL91] propose epsilo-serializability, aother relaxed cosistecy coditio, that allows query-trasactios (cotaiig read operatios oly) to overlap update-trasactios arbitrarily ad further allow cotet-depedet cocurretly i updatetrasactios, based o the sematics of their operatios. Wog ad Agrawal [WA92] also make use of the sematics of the data items ad furthermore take ito accout the state of each trasactio while it is executig. Aother settig i which the use of replicated variables to give probably correct results has proved to be useful is the efficiet simulatio of a PRAM o a asychroous system [KPRR92, AR92]. Specifically, Kedem et al. [KPRR92] use replicated variables i a way that a correct copy ca be reliably idetified ad probably exists. They the use these variables to create a global couter that processors use to determie whether they are roughly sychroized with other processors ad behave appropriately if they are ot. Auma ad Rabi [AR92] exhibit a clock costructio i a asychroous system with multiple processors that use a shared memory to create a object that correctly behaves as a clock with high probability. They use the clock to esure that processors stay sychroized throughout the computatio. I both cases, the protocols to read ad write the replicated variables are somewhat complex due to the eed to detect or mask icorrect copies. Ulike these previous works, which are tailored to specific applicatio requiremets, i our work we strive for a geeral techique for replicatig data with a high degree of simplicity, efficiecy, ad fault tolerace. Cosequetly, our techiques are very differet from those used i these previous works. 1.3. Our Results We begi by defiig ad explorig the limits of ε-itersectig quorum systems. I particular, we show a lower boud o the load of ε-itersectig quorum systems that is withi a small costat factor of the boud for strict quorum systems. Thus, ε-itersectig quorum systems caot yield substatial improvemets o the load i geeral. I cotrast, we show that ε-itersectig quorum systems ca yield substatial improvemets o the load whe high fault tolerace is also eeded. For ay ε, we provide a simple costructio of a ε-itersectig quorum system that demostrates optimal load ad fault tolerace O(1/ ) ad (), respectively, for a system of servers thereby circumvetig a trade-off betwee optimal load ad fault tolerace iheret i strict quorum systems. I additio, our costructio has failure probability better tha that of ay strict quorum system. For a eviromet i which servers may experiece Byzatie failures but servers store oly selfverifyig data, i.e., data that servers ca suppress but ot udetectably alter (such as digitally siged data), we ivestigate (b,ε)-dissemiatio quorum systems. We demostrate a dramatic improvemet i both the load ad the fault tolerace i this settig: strict b-dissemiatio quorums systems ca be costructed for b 1 arbitrarily faulty servers, ad the load of a dissemiatio quorum system 3 with such resiliece is at least 2. Usig essetially the same costructio as we use to demostrate 3 ε-itersectig systems, we demostrate a (b,ε)-dissemiatio quorum system resiliet to the arbitrary
PROBABILISTIC QUORUM SYSTEMS 187 failure of ay costat fractio of the servers ad with outstadig failure probability, whose load is O(1/ ). For large, this costructio provides a cosiderable advatage over strict dissemiatio quorum system costructios. Fially we defie ad explore (b,ε)-maskig quorum systems, which ca mask b arbitrary server failures for arbitrary forms of data. Usig techiques that diverge from the previous, we derive ad prove correct a geeral (b,ε)-maskig quorum system costructio that ca mask ay b < 2 Byzatie failures with a arbitrarily small ε. Our costructio beats the ( b/) lower boud o the load of ay strict maskig system [MRW00]. For istace, we demostrate a system that ca mask up to b = Byzatie failures with a load of oly O( 0.3 ). Moreover, we show lower bouds o the load of ay (b,ε)-maskig quorum system that demostrate that our costructio is asymptotically load-optimal whe b = ω( ). 1 Note that this also demostrates that our lower bouds are tight i this case. We also show that our costructio offers excellet failure probability. The cotributios of this paper ca be summarized as follows: We iitiate the study of probabilistic quorum systems ad provide formal defiitios for them. We exted the traditioal defiitios of three measures of quality load, fault tolerace, ad failure probability to address probabilistic costructios. We defie three types of probabilistic quorum systems: ε-itersectig quorum systems that tolerate beig failures oly; (b,ε)-dissemiatio quorum systems tolerat of b arbitrary server failures with self-verifyig data; ad (b,ε)-maskig quorum systems that tolerate b arbitrary server failures with arbitrary data. We preset protocols for usig ay quorum system meetig our defiitios to implemet a replicated variable whose sematics approximate that of a multi-reader, sigle-writer safe variable [Lam86]. We provide practical costructios for each class of quorum system. Our costructios have outstadig behavior i all measures: they have higher fault tolerace tha strict oes; they achieve better failure probability, ad i particular, ca achieve vaishigly small failure probability eve whe the idividual compoet failure probability is more tha 1/2, thus beatig the failure probability of ay strict quorum system; ad they maitai these properties simultaeously with optimal load. For Byzatie eviromets, our probabilistic dissemiatio ad maskig costructios ca also beat the geeral lower boud o the load of their strict couterparts. We show lower bouds o the load of each type of probabilistic quorum system that demostrate the load-optimality of our costructios. The rest of this paper is structured as follows. I Sectio 2, we preset the defiitios of strict quorum systems ad of the various traditioal quality measures. Sectio 3 defies ε-itersectig quorum systems ad exteds the traditioal quality measures to the probabilistic settig, proves a lower boud o the load of ay ε-itersectig quorum system, ad presets a costructio exhibitig very good load, fault tolerace, ad failure probability. Sectio 4 itroduces (b,ε)-dissemiatio quorum systems ad provides a costructio tolerat of the Byzatie failure of ay costat fractio of the servers. I Sectio 5, we defie (b,ε)-maskig quorum systems, preset a lower boud o the load of ay such system, ad preset a probabilistic maskig quorum system costructio. We demostrate the advatages of our techiques for particular system sizes i Sectio 6 ad coclude i Sectio 7. 2. STRICT QUORUM SYSTEMS I this sectio, we give a brief review of strict quorum systems. We assume a uiverse U of servers, U =, ad a distict set of cliets. Servers that obey their specificatios are correct. A (Byzatie) faulty server, however, may deviate from its specificatio arbitrarily. Whe workig with Byzatie failures, we assume that up to b servers may exhibit Byzatie failures. At times we restrict our attetio to crash failures oly, where a server fails by simply haltig; we will be explicit whe we do so. Throughout this paper we assume that cliets behave accordig to their specificatios. DEFINITION 2.1. A set system Q over a uiverse U is a set of subsets of U. 1 ω is the little-oh aalog of, amely f () = ω(g()) if f ()/g() as.
188 MALKHI ET AL. DEFINITION 2.2. A (strict) quorum system Q over a uiverse U is a set system over U such that for every Q, Q Q, Q Q. Each Q Q is called a quorum. Ituitively, because every two quorums itersect, whe a cliet reads the replicated data, it is sure to receive the last writte value from some server, amely the oe that is both i its read quorum ad i the last write quorum. I a typical access protocol i which values are writte with timestamps, the reader ca detect the most up-to-date value as the oe with the highest associated timestamp. Remark. I the domai of replicated database systems, it is commo to differetiate betwee the collectio of read-quorums R ad the collectio of write-quorums W [BHG87]. The itersectio requiremet is the R W ad W W for all R R ad W, W W. For simplicity we shall ot make the distictio betwee the two types of quorums. Traditioally, three measures were defied to assess the quality of quorum systems: the load, the fault tolerace, ad the failure probability of the system. 2.1. Load Cliets pick quorums to access i accordace with some access strategy, which defies the likelihood that a quorum is chose for ay give access. DEFINITION 2.3. A access strategy (or simply a strategy) w for a set system Q specifies a probability distributio o the elemets of Q. That is, w : Q [0, 1] satisfies Q Q w(q) = 1. The load of a quorum system, defied by Naor ad Wool [NW98], captures the probability of accessig the busiest server. Load is a measure of efficiecy. All other thigs beig equal, systems with lower load ca process more requests tha those with higher load. DEFINITION 2.4. Let w be a strategy for a set system Q ={Q 1,...,Q m } over a uiverse U. For a server u U, the load iduced by w o u is l w (u) = Q i u w(q i). The load iduced by a strategy w o Q is L w (Q) = max u U {l w (u)}. The load of Q is L(Q) = mi w {L w (Q)}, where the miimum is take over all strategies. It is kow that for ay quorum system Q over servers, L(Q) max{ 1 } where c(q) is the size of the smallest quorum i Q [NW98]. I particular, this implies that for ay quorum system Q, L(Q) 1/. We ote that load is a best case defiitio. The load of the quorum system will be achieved oly if a optimal access strategy is used ad oly i the case that o failures occur. A stregth of this defiitio is that load is a property of a quorum system ad ot of the protocol usig it. A compariso of the defiitio of load to other seemigly plausible defiitios is give i [NW98]. Fidig a live quorum i case of failures is a active research topic, e.g., [PW96, Baz96, Baz99]. Although this topic is outside the scope of our paper, we ote that it would be straightforward to apply the techiques of those papers to our costructios. c(q), c(q) 2.2. Fault Tolerace Fault tolerace ad failure probability capture the resiliece of the system to crash failures. The fault tolerace of a quorum system Q is the size of the smallest set of servers that itersects all quorums i Q. DEFINITION 2.5. For a set system Q ={Q 1,...,Q m },defie S ={S : S Q i for all 1 i m}. The the fault tolerace of Q is A(Q) = mi S S S. Thus, a quorum system Q is resiliet to the failure of ay set of A(Q) 1 or fewer servers. I particular, the failure of at least A(Q) servers is ecessary to disable every quorum i the system, ad some particular set of A(Q) failures ca i fact disable them all. Moreover, the itersectio property implies that the failure of ay full quorum i Q will disable all quorums (i.e., A(Q) c(q)), ad so by the aforemetioed lower boud o load, A(Q) L(Q). Therefore, there is a trade-off betwee load ad fault tolerace i strict quorum systems, ad i particular,
PROBABILISTIC QUORUM SYSTEMS 189 TABLE I Bouds o the Load ad Resiliece of Differet Quorum System Types L(Q) Strict b-dissemiatig b-maskig 1 b + 1 2b + 1 b N/A 1 3 1 4 it follows that ay strict quorum system with optimal load of (1/ ) has fault tolerace of (oly) O( ). 2.3. Failure Probability The failure probability of a quorum system is the probability that the system is disabled whe idividual servers crash idepedetly with a fixed probability. DEFINITION 2.6. The failure probability F p (Q) ofq is the probability that every Q Q cotais at least oe crashed server, uder the assumptio that each server i U crashes idepedetly with probability p. A strict quorum system Q has a good failure probability if lim F p (Q) = 0 whe p < 1 2 [PW95]. Whe p 1 2 the F p(q) p 1 2 for strict quorum systems, ad typically F p(q) 1 whe p > 1 2. 2.4. Byzatie Systems As discussed i Sectio 1, quorum systems are geerally isufficiet to guaratee cosistecy i case of Byzatie server failures. Malkhi ad Reiter exteded quorum systems to hadle Byzatie failures [MR98a]: a b-dissemiatio quorum system icreases quorum overlap to b + 1 servers, which suffices to mask faulty server behavior for self-verifyig data; a b-maskig quorum system further icreases quorum overlap to 2b + 1 servers, maskig faulty server behavior for ay type of data. 2 DEFINITION 2.7. Let Q be a set system. The Q is a b-dissemiatio quorum system if A(Q) > b ad Q Q b + 1 for every Q, Q Q. Q is a b-maskig quorum system if A(Q) > b ad Q Q 2b + 1 for every Q, Q Q. For example, if a b-maskig quorum system is used, the whe a cliet performs a read operatio at some quorum Q, the value writte i the last precedig write operatio, say to Q, is retured by at least b + 1 correct servers, amely servers i the set (Q Q )\B where B is the set of faulty servers. Ay other retured value is either a old value, which ca be detected by its earlier timestamp, or a made-up value retured oly by servers i B. So, if the cliet discards ay values that were retured by b or fewer servers, ad the chooses from the remaiig values the oe with the most recet timestamp, the the cliet is guarateed to obtai the correct value [MR98a]. There are kow lower bouds o the load of strict, b-dissemiatio, ad b-maskig quorum systems ad upper bouds o the attaiable resiliece for each system type [NW98, MR98a, MRW00]. I Table 1 we summarize these bouds cocisely. 3. ε-intersecting QUORUM SYSTEMS I this sectio, we itroduce probabilistic quorum systems ad their properties. We first formally defie ε-itersectig quorum systems ad show a protocol for usig them. Next, we exted the defiitio of the three quality measures to accout for the probabilistic ature of our systems. We the prove a lower boud o the load of ε-itersectig quorum systems, which shows that their relaxed cosistecy 2 The origial defiitio of [MR98a] allows more geeral failure cofiguratios tha we do here. The simplified defiitio preseted here suffices for our purposes.
190 MALKHI ET AL. caot yield substatial improvemets o the load i geeral. Fially, we show that ε-itersectig quorums are ot subject to the load fault tolerace trade-off, by demostratig a costructio over a uiverse of servers that has a load of (1/ ) ad fault tolerace of (), for which ε vaishes as grows. We show that our costructio has exceptioally good failure probability for essetially limitless compoet failure probabilities, for appropriate system sizes. The failure probability of our costructio is provably better tha that of ay strict quorum system. 3.1. Defiitios ad Usage of ε-itersectig Quorum Systems We begi by defiig ε-itersectig quorum systems. Q is a ε-itersectig quorum system if the total access probability of pairs of itersectig quorums is at least 1 ε. Formally, we have the followig. DEFINITION 3.1. Let Q be a set system, let w be a access strategy for Q, ad let 0 <ε<1begive. The tuple Q,w is a ε-itersectig quorum system if P(Q Q ) 1 ε, where the probability is take with respect to the strategy w. Abusig termiology slightly, we still call elemets of Q quorums. To demostrate the utility of this defiitio, we ow show a simple protocol which borrows from the protocols of Gifford [Gif79] ad Thomas [Tho79] for accessig replicated data by a sigle writer ad multiple readers, but with the distictio that it uses a ε-itersectig quorum system. Each server stores a copy of the replicated variable x ad a associated timestamp value t that will be updated by cliets. Write ad read operatios proceed as follows: Write: For a cliet to write the value v to x, it 1. chooses a quorum Q accordig to the strategy w, 2. chooses a timestamp t greater tha ay timestamp it has chose i the past, ad 3. updates x ad the associated timestamp at each server i Q to v ad t, respectively. Read. For a cliet to read x,it 1. chooses a quorum Q accordig to the strategy w, 2. queries each server i Q to obtai a set of value timestamp pairs V ={ v u, t u } u Q, 3. chooses the pair v, t i V with the highest timestamp, ad 4. chooses v as the result of the read operatio. We are aware of o stadard defiitio of variable sematics that ca be used to prove correctess of the above protocol, due to the possibility (albeit small) that a read quorum does ot itersect the most recet write quorum. The followig theorem evertheless clarifies its utility, showig that the protocol approximates a multi-reader, sigle-writer, safe variable [Lam86]. Safe variables are but oe example of useful shared data abstractios implemeted usig probabilistic quorums; other replicated data objects ca be costructed either usig probabilistic quorum systems directly (e.g., locks [MR98b]) or usig variables for buildig blocks (e.g., atomic variables, borrowig from the techiques of [Lam86, IS92]). THEOREM 3.2. Cosider a multi-reader, sigle-writer variable replicated usig the above access protocol with a ε-itersectig quorum system. If a read operatio is ot cocurret with ay write operatio ad oly crash failures occur, the with probability at least 1 ε the read returs the value writte by the last precedig write operatio. Proof. Cosider the last write operatio prior to the read operatio. Sice there is oly oe writer, it follows by the specificatio of the write protocol that it has the highest timestamp of ay write operatio that precedes the read. Moreover, with probability at least 1 ε, the quorum Q picked i this write operatio ad the quorum Q picked i the curret read operatio satisfy Q Q. So, with probability at least 1 ε, this value timestamp pair appears i V ad thus the correct value will be retured by the read. Remark. The defiitio of a ε-itersectig quorum system cotais a access strategy, which is chose to achieve the desired boud ε o oitersectio betwee two quorums chose accordig to the strategy. Other access strategies o the same set system may fail to achieve the same itersectio
PROBABILISTIC QUORUM SYSTEMS 191 guaratee, as ca be trivially demostrated by a strategy that chooses each of two oitersectig quorums with probability 1/2. Thus, for a ε-itersectig quorum system to obtai the advertised probability of itersectio whe used i a protocol, the specified access strategy must be eforced. 3.2. Measures of Quality We ow tur to adaptig the various measures of quorum systems defied i Sectio 2 to probabilistic quorum systems. The defiitio of load carries over immediately. DEFINITION 3.3. Let (Q,w) beaε-itersectig quorum system. The the load of Q,w is L( Q,w ) = L w (Q). However, the defiitios of fault tolerace ad failure probability, as formulated for strict quorum systems, are usatisfactory i a probabilistic settig. To demostrate this, we show how to covert ay ε-itersectig quorum system Q,w ito a ew system Q,w which has essetially the same cosistecy guaratee 1 ε but with a artificially iflated fault tolerace. The set system Q is created by simply addig every possible sigleto set as a quorum: Q = Q {{u 1 },...,{u }}. For ay γ ε, the strategy w is defied by w (Q) = (1 γ )w(q) for all Q Q, ad w ({u i }) = γ/ for all the sigletos. Sice the sigleto quorums {u i } are used with such low probability, it is easy to see that Q,w is ε -itersectig, with ε ε. However, the oly way to disable all the quorums of Q is to have all the servers crash, so A(Q ) =. Likewise, the failure probability of Q is ureasoably good: accordig to Defiitio 2.6, F p (Q ) = p. The problem with aively usig Defiitios 2.5 ad 2.6 is that they allow the fault tolerace to be derived from quorums that itersect few other quorums ad are hardly ever used by the strategy. Ay reasoable defiitio of fault tolerace for probabilistic quorum systems should require that the fault tolerace be derived from those quorums that itersect other quorums with high probability. To make this ituitio precise, we eed the followig techical defiitio ad lemma, leadig to Defiitios 3.7 ad 3.8. From here o, all probabilities ad expectatios are take with respect to the strategy w, uless explicitly deoted otherwise. DEFINITION 3.4. Let Q,w be a ε-itersectig quorum system, ad let 0 δ 1 be give. The set of δ-high quality quorums of Q,w is where Q Q is chose accordig to w. R ={Q Q : P(Q Q ) 1 δ}, The followig lemma shows that i a ε-itersectig quorum system, most of the weight lies o the δ-high quality quorums. LEMMA 3.5. P(Q R) 1 ε δ. Proof. From Defiitio 3.1, ε P(Q Q = ) = Q Q w(q) Q :Q Q = w(q ) Q R w(q) Q :Q Q = For ay fixed Q R, Q :Q Q = w(q ) = P(Q Q = ) >δby Defiitio 3.4. Thus, ε δ w(q) = 1 P(Q R). Q R w(q ). Cosequetly, by choosig δ so that both δ ad ε/δ are small, the δ-high quality quorums are high quality i two respects: they itersect other chose quorums with high probability (by defiitio) ad they costitute the quorums that are selected with high probability (by Lemma 3.5). A reasoable choice of δ to reder both δ ad ε/δ small whe ε is small is δ = ε. Heceforth, we adopt this covetio ad refer to the ε-high quality quorums as simply the high quality quorums:
192 MALKHI ET AL. DEFINITION 3.6. Let Q,w be a ε-itersectig quorum system. The the high quality quorums of Q,w are the ε-high quality quorums of Q,w. We are ow prepared to state our defiitios for fault tolerace ad failure probability. The differece betwee these defiitios ad Defiitios 2.5 ad 2.6 is that here we cosider the system to be disabled if all the high quality quorums are hit. DEFINITION 3.7. Let Q,w be a ε-itersectig quorum system. Let R be the set of high quality quorums of Q,w, ad let S ={S : S Q for all Q R}. The the fault tolerace A( Q,w ) is mi S S S. DEFINITION 3.8. Let Q,w be a ε-itersectig quorum system, ad let R be the set of high quality quorums of Q,w. The failure probability F p ( Q,w ) is the probability that every Q R cotais at least oe crashed server, uder the assumptio that each server i U crashes idepedetly with probability p. These defiitios are cosistet with Defiitios 2.5 ad 2.6 for strict quorum systems: I ay strict quorum system Q, all the quorums are high quality quorums by defiitio, irrespective of the access strategy used. Hece, A( Q,w ) = A(Q) for all strategies w. For a probabilistic quorum system Q,w, A( Q,w ) A(Q) ad F p ( Q,w ) F p (Q), which stads to reaso sice A( Q,w ) ad F p ( Q,w ) deped o w. The reader ca verify that ulike the strict A(Q) ad F p (Q), the probabilistic measures A( Q,w ) ad F p ( Q,w ) caot be artificially iflated by addig hidde servers ad quorums. 3.3. A Lower Boud o the Load I this sectio we state ad prove a lower boud o the load of ε-itersectig quorum systems. This lower boud is close to the lower boud for strict quorum systems ad thus idicates that we should ot look to ε-itersectig quorums as a techique to circumvet the lower boud for strict oes. THEOREM 3.9. Let Q, w be a ε-itersectig quorum system, ad let the radom variable Q be the size of a quorum chose accordig to w. The L( Q,w ) max { E[ Q ], (1 } ε) 2. E[ Q ] Theorem 3.9 is similar to the bouds show i [NW98] for strict quorum systems. The mai differeces are that here we have a specific strategy w so we ca work with the expected quorum size (rather tha the miimal quorum size) ad that we eed to accout for the small probability of quorums ot itersectig each other. We prove Theorem 3.9 via the followig two lemmas. LEMMA 3.10. Let Q be a set system ad let w be a strategy for Q. The L w (Q) E[ Q ]. Proof. By summig the total load iduced by w o all the elemets of U we obtai L w (Q) l w (u) = w(q) = w(q) Q =E[ Q ]. u U u U Q u Q Q LEMMA 3.11. Let Q,w ad Q be as i Theorem 3.9. The L w (Q) (1 ε) 2 /E[ Q ]. Proof. Let R be the high quality system associated with Q,w (as i Defiitio 3.4 with δ = ε). Defie a restricted strategy w r over Q by { w(q w r (Q )/P(Q R), if Q R, ) = 0, otherwise.
The expected chose quorum size with respect to w r obeys E wr [ Q ] = w r (Q ) Q = Q Q Q R ad hece by Lemma 3.5, PROBABILISTIC QUORUM SYSTEMS 193 w(q ) P(Q R) Q 1 P(Q R) w(q ) Q = Q Q E[ Q ] P(Q R), E[ Q ] (1 ε)e wr [ Q ]. (1) Now fix some ˆQ R with ˆQ E wr [ Q ]. (Such a set must exist by the defiitio of E wr.) Summig the load iduced by w o the elemets of ˆQ we have ˆQ L w (Q) l w (u) = w(q) = w(q) Q ˆQ u ˆQ u ˆQ Q u Q Q w(q) = P(Q ˆQ ) 1 ε (2) Q:Q ˆQ by defiitio sice ˆQ R. Now by (2), (1), ad the defiitio of ˆQ, weget L w (Q) 1 ε ˆQ 1 ε E wr [ Q ] (1 ε) 2. E[ Q ] Theorem 3.9 follows directly from Lemmas 3.10 ad 3.11. COROLLARY 3.12. L( Q,w ) (1 ε). Proof. Immediate from Theorem 3.9. 3.4. A ε-itersectig Quorum System Costructio We ow demostrate a ε-itersectig quorum system Q,w with O(1/ ) load ad () fault tolerace that meets ay required ε for sufficietly large. The costructio is very simple: Give a uiverse of servers, the quorums are all the sets of size l ; the strategy chooses a quorum uiformly at radom. The costat l is chose to make ε sufficietly small. Ituitively, it is easy to see that this should work the expected, ad most probable, size of the itersectio of two such quorums is l 2,so by makig l sufficietly large, it should be possible to reduce the probability ε that the itersectio of two quorums is empty to ay desired level. This is similar to the well-kow birthday paradox (see [CLR89]): Give two quorums, the probability that ay give elemet i oe quorum is also i the secod quorum is quite small ( l ), but the probability that some elemet appears i both quorums is quite high (at least 1 e l2, as we prove below). DEFINITION 3.13. Let U be a uiverse of size. The R(, q) is the system Q,w defied by Q ={Q U : Q =q} with the uiform strategy w(q) = 1 for all quorums Q Q. Q We cosider R(,l ) ad show that the probability of choosig at radom two quorums that do ot itersect ca be made sufficietly small by appropriate choice of l. We use the followig combiatorial fact. PROPOSITION 3.14. For itegers, c, ad i, ( c c i )/( c ) ( c )i ( c i )c i. LEMMA 3.15. Let Q ad Q be quorums of size l each chose uiformly at radom. The P(Q Q = ) < e l2.
194 MALKHI ET AL. Proof. P(Q Q = ) = ( l l ( l ) ( l ) ) l e l l = e l2, where the first iequality follows from Propositio 3.14. It is immediate from Lemma 3.15 that R(,l )isaε-itersectig quorum system: THEOREM 3.16. R(,l ) is a (e l2 )-itersectig quorum system. Quality Measures. Sice every elemet is i ( 1 l 1 ) quorums, the load L(R(,l l )) is = O(1/ ). R(,l ) is a symmetrical costructio with a uiform access strategy, ad hece all of its members are high quality quorums. Because oly l servers eed be available i order for some (high quality) quorum to be available, the fault tolerace A(R(,l )) = l + 1 = (). The failure probability of R(,l ) is also exceptioally good. Let p deote the idepedet failure probability of servers. For the system to fail, at least l + 1 servers must fail. Usig Cheroff s boud, this probability is at most F p (R(,l )) = P(#fail > l ) e 2(1 l p) 2 = e () for all p 1 l. Peleg ad Wool showed that the failure probability of ay strict quorum system whose fault tolerace is f is at most e ( f ) [PW95]. Furthermore, they showed that for p > 1, the failure 2 probability is at least p. Therefore, if p 1 l, the failure probabaility of R(,l ) is asymptotically optimal, ad if 1 2 p 1 l, this probability is provably better tha that of ay strict quorum system. Sectio 6 provides cocrete examples of R(,l ) for various values of ad l, compared i all three measures agaist cocrete examples of strict quorum systems. 4. (b,ε)-dissemination QUORUM SYSTEMS To achieve cosistecy whe servers ca fail arbitrarily, it is ot sufficiet that two quorums have a oempty itersectio. This is because two quorums may itersect i a set cotaiig faulty servers oly, which may deviate arbitrarily ad udetectably from their assiged protocol. Malkhi ad Reiter [MR98a] defied (strict) dissemiatio quorum systems that ca be used to costruct Byzatie faulttolerat replicated services that store self-verifyig data. Data are self-verifyig if servers caot alter data udetectably, e.g., because cliets digitally sig it. For such data a faulty server is costraied to retur some value that was previously writte or be detected as faulty. I this case, it is sufficiet to require that the itersectio of every two quorums cotais at least oe ofaulty server, sice this guaratees a correct, up-to-date value will be preset ad ca be recogized. That is, the itersectio of every two quorums should be of size at least b +1, where b is the maximum umber of Byzatie faults. Here we modify dissemiatio quorum systems to a probabilistic settig. To achieve probable cosistecy i a Byzatie eviromet, it is ot sufficiet that two quorums should have a probably oempty itersectio, sice agai two quorums may itersect i a set cosistig of faulty servers. Istead we use the followig defiitio. DEFINITION 4.1. Let Q be a quorum system, let w be a access strategy for Q, ad let 0 <ε<1 ad a iteger b > 0 be give. The Q,w isa(b,ε)-dissemiatio quorum system if A( Q,w ) > b ad P(Q Q B) 1 ε for all B U such that B =b. (b,ε)-dissemiatio quorum systems ca be used to implemet Byzatie fault-tolerat services for the same types of data that strict dissemiatio quorum systems ca, usig the same access protocol [MR98a]. Specifically, the read operatio becomes:
PROBABILISTIC QUORUM SYSTEMS 195 Read. For a cliet to read x,it 1. chooses a quorum Q accordig to the strategy w, 2. queries each server i Q to obtai a set of value timestamp pairs V ={ v u, t u } u Q, 3. computes the set V cosistig of elemets from V that are verifiable, 4. chooses the pair v, t i V with the highest timestamp, ad 5. chooses v as the result of the read operatio. The timestamps are assumed to be icluded as part of the self-verifyig data. If follows that this protocol approximates a multi-reader, sigle-writer safe variable whe used with verifiable data i a Byzatie eviromet. THEOREM 4.2. Cosider a multi-reader, sigle-writer variable over verifiable data replicated usig the above access protocol with a (b,ε)-dissemiatio quorum system. If a read operatio is ot cocurret with ay write operatio ad at most b Byzatie failures occur, the with probability at least 1 ε the read returs the value writte by the last precedig write operatio. Proof. As i the proof of Theorem 3.2, the last write operatio prior to the read opearatio has the highest timestamp of ay write operatio that precedes the read. Moreover, with probability at least 1 ε, the quorum Q picked i this write operatio ad the quorum Q picked i the curret read operatio satisfy Q Q B where B is the set of actually faulty servers. So, with probability at least 1 ε, this value timestamp pair appears i V. Further, by the verifiability of the data, oly values from correct servers appear i V. It follows that the correct value will be retured by the read. Note that sice ay (b,ε)-dissemiatio quorum system is also a ε-itersectig quorum system, the lower boud of Theorem 3.9 applies. Noetheless, we show that relaxig quorum itersectio i a Byzatie eviromet ca provide dramatic improvemets i both load ad availability over strict dissemiatio quorum systems. Specifically, we show that our ε-itersectig quorum costructio R(,l ) provides the followig i this eviromet. First, it breaks the lower boud o the load of strict dissemiatio quorum systems of ( b/) ad achieves a load of O(1/ ). Secod, its resiliece level b ca be icreased to ay costat fractio of the system, thus breakig the 1 3 upper boud o the resiliece of strict dissemiatio quorum systems, while retaiig asymptotically optimal load. Third, it maitais a outstadig failure probability (for crash failures) for sufficietly large uiverses eve for p > 1/2, thus beatig the best failure probability of ay strict quorum system. For coveiece of the expositio, we first preset a costructio whose resiliece is b = ad later 3 modify it for arbitrarily large b. 4.1. A (b,ε)-dissemiatio Quorum System for b = 3 I this sectio, we show that R(,l ), as defied i Sectio 3.4, ca be used as a (b,ε)-dissemiatio quorum system for a Byzatie threshold b =, the resiliece boud for strict dissemiatio quorum 3 systems [MR98a]. R(,l ) exhibits much better load ad fault tolerace (to crash failures) tha strict quorum costructios for this value of b. Although the boud ε is differet tha for the case with o Byzatie server failures, we are still able to show that for a appropriate choice of the parameter l, this costructio esures itersectio with ay desired probability for sufficietly large uiverse. LEMMA 4.3. Let U be a uiverse of size, let B be a subset of U of size b where b =, ad let Q 3 ad Q be quorums of size l each chose uiformly at radom. The P((Q Q ) B) 2e l2 /6. Proof. P(Q Q B) = P( Q Q = Q Q B ) l = P(( Q Q =i) ( Q Q B =i)) (3) i=0
196 MALKHI ET AL. l ( l )( l i l i ( ) i=0 l i=0 ) ( ) 1 i (4) 3 l ( )( ) i ( ) l i ( ) l l l 1 i (5) i i 3 l 6 i=0 (l 2 ) i i! e (l i) 2 i 3 i + l i= l 6 +1 3 i (6) l 6 i=0 ( l 2 ) i 3 i! e l2 ( 5 6) 2 + l i= l 6 +1 3 i (7) e l2 ( 5 6) 2 e l2 3 + 3 l 6 (8) 1 2e l2 6. (9) Let c = l. The (4) holds because P( Q Q B =i) :( Q Q =i)) = 1 2 ( 3 i )( i )( c c )( c i c i )/ 2 ( c )( c i c i ) = ( 3 )!( i)!/( 3 i)!! ( 1 3 )i ; (5) is by Propositio 3.14; (6) is because for the first part of the sum: ( c i )( c )i (c i /i!)(c i / i ) = (l 2 ) i /i! ad 1 + x e x, for the secod: ( c i )(c/)i (( c)/ ( i)) c i 1; (7) holds sice e (c i)2 /( i) e (c (c/6))2 / = e l2 (5/6) 2 for i c ; (8) is because 6 i 0 (l2 /3) i /i! = e l2 /3 ; ad (9) is because e < 3 ad l. Thus we have proved the followig result: THEOREM 4.4. R(,l ), where l< 2 3, is a ( 3, /6 )-dissemiatio quorum system. 2e l2 Quality Measures. Load, fault tolerace, ad failure probability do ot deped o b or ε. (Recall that fault tolerace ad failure probability relate to crash failures, while b is the umber of Byzatie failures tolerated.) Hece, we have as before that the load L(R(,l )) is l, the fault tolerace A(R(,l )) is l + 1, ad the failure probability F p (R(,l )) is at most e 2γ 2, where γ = 1 l p, for p < 1 l. 4.2. A (b,ε)-dissemiatio Quorum System for b = α Surprisigly, the same techique ca be used to overcome ay fractio α of Byzatie failures. I this case, the parameter l eeded to achieve a particular value of ε depeds o the fractio α of servers that may simultaeously fail. Sice our costructio works, with appropriate choice of parameters, for b = α for ay costat fractio α of the servers, it is sigificatly more versatile tha costructios of strict dissemiatio quorum systems, where a upper boud of b = 1 limits the resiliece. We 3 preset the result here for 1 3 <α<1, as the case 0 <α 1 was already covered i the previous sectio. 3 (A similar result holds for 0 <α<1, but yields a more complicated ε.) Let 1 3 <α<1 ad let ε α = 2 ((1 α)/2) 1 α αl2. A argumet similar to Lemma 4.3 shows the followig. LEMMA 4.5. Let U be a uiverse of servers, let B be a subset of U of size b where b = α for some 1 3 <α<1, ad let Q ad Q be quorums of size l each chose uiformly at radom. The P((Q Q ) B) ε α. THEOREM 4.6. R(,l ), where l< (1 α), is a (α,ε α )-dissemiatio quorum system. Remarks. Sice we assume that α servers may fail, we must have l >α, or equivaletly, l< (1 α). This limits the achievable itersectio guaratee ε α of R(,l ), for ay particular system size ad Byzatie threshold α.
PROBABILISTIC QUORUM SYSTEMS 197 Note that Q ad w do ot directly deped o α. Hece, eve if the fractio of Byzatie faults that may occur is ot kow, it is possible to use this costructio, but the itersectio parameter ε that is achieved will also be ukow. Furthermore, the costructio has the desirable graceful degradatio property that actual itersectio probability will be better if fewer Byzatie faults actually occur. 5. (b,ε)-masking QUORUM SYSTEMS Whe Byzatie faults occur with data that are ot self-verifyig, it is ecessary that correct servers be able to out-vote icorrect oes. Accordigly, a strict b-maskig quorum system is defied to be oe i which ay two quorums itersect i at least 2b + 1 elemets [MR98a]. As a result, whe a cliet performs a read operatio at some quorum Q, the value writte i the last precedig write operatio, say to Q, is retured by at least b + 1 correct servers, amely servers i the set (Q Q )\B where B is the set of faulty servers. Ay other retured value is either a old value, which ca be detected by its earlier timestamp, or a made-up value retured oly by servers i B. So, if the cliet discards ay values that were retured by b or fewer servers, ad the chooses from the remaiig values the oe with the most recet timestamp, the the cliet is guarateed to obtai the correct value [MR98a]. To formulate a probabilistic versio of maskig quorum systems, a atural place to start is the defiitio of a ε-itersectig quorum system. Mimickig that approach for maskig quorum systems, we would require that ay two selected quorums itersect i at least 2b + 1 elemets with high probability. Oe advatage of such a defiitio is that there is o eed to chage the cliet access protocol: simply adoptig the read ad write protocols from [MR98a] would esure that cliets receive correct aswers with high probability. However, this defiitio does ot yield the performace beefits that the probabilistic approach did for regular quorum systems. I particular, it is ot difficult to verify that the load for ay such system with b = () would be costat, which is poor. The trouble with this defiitio is that it is stroger tha ecessary. If Q ad Q are the quorums used i a read ad a previous write operatio, respectively, ad B is the set of faulty servers, the the defiitio requires Q Q \B to be so large that it is impossible for Q B to be of equal cardiality. For the correct aswer to be probably detectable to a readig cliet, the set Q Q \B eed oly be of a size sufficietly large that it is improbable that Q B is of the same size or larger. To weake this requiremet, our defiitio of a (b,ε)-maskig quorum system employs a threshold value k that is expected to be betwee Q B ad Q Q \B. Thus a readig cliet that requires at least k occureces of a value i order to accept it as the outcome of the read operatio will get the right value with high probability. DEFINITION 5.1. Let Q be a set system over a uiverse U of size, let w be a access strategy for Q, ad let 0 <ε<1 ad itegers 1 k ad b > 0 be give. The tuple Q,w,k is a (b,ε)-maskig quorum system if A( Q,w ) > b ad P( Q B < k Q Q \B k) = Q,Q Q Q B <k Q Q \B k ω(q)ω(q ) 1 ε, for all B U such that B =b, where the probability is take with respect to w. We modify the access protocol as follows. Write operatios are as before, but read operatios ow require a value that passes the threshold k: Read. For a cliet to read x,it 1. chooses a quorum Q accordig to the strategy w, 2. queries each server i Q to obtai a set of value timestamp pairs V ={ v u, t u } u Q, 3. computes the set V ={ v, t : C Q[ C k u C [v u = v t u = t]]}, 4. returs the pair v, t i V with the highest timestamp, or if V is empty. THEOREM 5.2. Cosider a multi-reader, sigle-writer variable replicated usig the above access protocol with a (b,ε)-maskig quorum system. If a read operatio is ot cocurret with ay write
198 MALKHI ET AL. operatio ad at most b Byzatie failures occur, the with probability at least 1 ε the read returs the value writte by the last precedig write operatio. Proof. As i the proof of Theorem 3.2, the last write operatio prior to the read operatio has the highest timestamp of ay write operatio that precedes the read. Moreover, with probability at least 1 ε, the quorum Q picked i this write operatio ad the quorum Q picked i the curret read operatio satisfy Q B < k (Q Q )\B k where B is the set of actually faulty servers. So, with probability at least 1 ε, this value timestamp pair appears i V ad thus the correct value will be retured by the read. Note that whe a icorrect value is retured, it ca either be a old or ull value (if (Q Q )\B < k) or a value chose by the faulty servers (if Q B k). We defie load, fault tolerace, ad failure probability of (b,ε)-maskig quorum systems i the stadard maer: DEFINITION 5.3. Let Q,w,k bea(b,ε)-maskig quorum system. The The load of Q,w,k is L( Q,w,k ) = L w (Q). The fault tolerace of Q,w,k is A( Q,w,k ) = A( Q,w ). The failure probability of Q,w,k is F p ( Q,w,k ) = F p ( Q,w ). 5.1. Lower Bouds o the Load I additio to the geeral lower boud o load give i Theorem 3.9, which a fortiori holds i the case of (b,ε)-maskig quorum systems, we preset here a lower boud that depeds o the umber of servers, o the threshold b of tolerated Byzatie faults, ad o the error probability ε. This demostrates a relatioship betwee the umber of faulty servers a system tolerates ad the load it may achieve. Our mai result i this sectio is the lower boud of Theorem 5.5. To prove this, we show that the expected quorum size must exceed b (up to a factor close to 1) i order to satisfy the itersectio requiremet ad the use half of Theorem 3.9. LEMMA 5.4. Let Q,w,k bea(b,ε)-maskig quorum system. The P( Q > b) 1 2ε 1 ε. Proof. Fix some ˆQ Q with ˆQ b. (If o such ˆQ exists, the we are doe.) The there exists some set B ˆQ of size b such that ˆQ B ˆQ.ByDefiitio 5.1, 1 ε P( Q B ˆQ < k (Q Q )\B ˆQ k) P( Q B ˆQ < k) P( Q ˆQ < k). Sice ˆQ was chose arbitrarily subject to the restrictio that ˆQ b, we have the followig boud o the coditioal probability: P( Q ˆQ < k ˆQ b) 1 ε. (10) Now usig (10) ad Defiitio 5.1, for ay B with B =b we have ε P( Q B k Q Q \B < k) P( Q Q < k) P( Q Q < k Q b) = P( Q Q < k Q b) P( Q b) (1 ε) P( Q b). Thus P( Q b) ε/(1 ε) ad P( Q > b) (1 2ε)/(1 ε).