Distrib. Comput. (1998) 11: 03 13 c Spriger-Verlag 1998 Byzatie quorum systems Dahlia Malkhi, Michael Reiter AT&T Labs Research, Florham Park, NJ 0793-0971, USA (e-mail: {dalia,reiter}@research.att.com) Received: October 1996 / Accepted Jue 1998 Summary. Quorum systems are well-kow tools for esurig the cosistecy ad availability of replicated data despite the beig failure of data repositories. I this paper we cosider the arbitrary (Byzatie) failure of data repositories ad preset the first study of quorum system requiremets ad costructios that esure data availability ad cosistecy despite these failures. We also cosider the load associated with our quorum systems, i.e., the miimal access probability of the busiest server. For services subject to arbitrary failures, we demostrate quorum systems over servers with a load of O( 1 ), thus meetig the lower boud o load for beigly fault-tolerat quorum systems. We explore several variatios of our quorum systems ad exted our costructios to cope with arbitrary cliet failures. Key words: Quorum systems Byzatie failures Replicatio Fault tolerace 1 Itroductio A well kow way to ehace the availability ad efficiecy of replicated data is by usig quorums. A quorum system for a uiverse of data servers is a collectio of subsets of servers, each pair of which itersect. Ituitively, each quorum ca operate o behalf of the system, thus icreasig its availability ad performace, while the itersectio property guaratees that operatios doe o distict quorums preserve cosistecy. I this paper we cosider the arbitrary (Byzatie) failure of cliets ad servers, ad iitiate the study of quorum systems i this model. Ituitively, a quorum system tolerat of Byzatie failures is a collectio of subsets of servers, each pair of which itersect i a set cotaiig sufficietly may correct servers to guaratee cosistecy of the replicated data as see by cliets. We provide the followig cotributios. 1. We defie the class of maskig quorum systems, with which data ca be cosistetly replicated i a way that is resiliet to the arbitrary failure of data repositories. We show ecessary ad sufficiet coditios for the existece of maskig quorum systems uder differet failure assumptios, ad preset several example costructios of such systems.. We explore two variatios of maskig quorum systems. The first, called dissemiatio quorum systems, is suited for services that receive ad distribute self-verifyig iformatio from correct cliets (e.g., digitally siged values) that faulty servers ca fail to redistribute but caot udetectably alter. The secod variatio, called opaque maskig quorum systems, is similar to regular maskig quorums i that it makes o assumptio of self-verifyig data, but it differs i that cliets do ot eed to kow the failure scearios for which the service was desiged. This somewhat simplifies the protocol by which cliets access the replicated data ad, i the case that failures are maliciously iduced, reveals less iformatio to cliets that could guide a attack attemptig to compromise the system. 3. We explore the load of each type of quorum system, where the load of a quorum system is the miimal access probability of the busiest server, miimizig over all strategies for pickig quorums. We preset a maskig quorum system with the property that its load over a total of servers is O( 1 ), thereby meetig the lower boud for the load of beigly-fault-tolerat quorum systems. For opaque maskig quorum systems, we prove a lower boud of 1 o the load, ad preset a costructio that meets this lower boud ad proves it tight. 4. For services that use maskig quorums (opaque or ot), we show how to deal with faulty cliets i additio to faulty servers. The primary challege raised by cliet failures is that there is o guaratee that cliets will update quorums accordig to ay specified protocol. Thus, a faulty cliet could leave the replicated data i a icosistet ad irrecoverable state. We develop a update protocol, by which cliets update the replicated data, that prevets cliets from leavig the data i a icosistet state. The protocol has the desirable property that it ivolves oly the quorum at which a access is attempted, while providig system-wide cosistecy properties. Our quorum systems, if used i cojuctio with appropriate protocols ad sychroizatio mechaisms, ca be used to implemet a wide rage of data sematics. I this
04 paper, however, we choose to demostrate a variable supportig read ad write operatios with relatively weak sematics, i order to maitai focus o our quorum costructios. These sematics imply a safe variable [0], which a set of correct cliets ca use to build other abstractios, e.g., atomic, multi-writer multi-reader registers [18, 0, 1], cocurret timestamp systems [10, 17], l-exclusio [, 9], ad atomic sapshot sca [1, 5]. Our quorum systems ca be used for buildig other protocols i additio to shared read/write register emulatio. For example, i a ogoig effort [5], we use Byzatie quorum systems i costructig a large-scale, survivable service supportig persistet data abstractios such as cosesus objects [4], locks ad files. I additio, i Sect. 6, we demostrate how maskig quorum systems ca be used to guaratee cosistecy ad completio of updates, eve those executed by faulty cliets. The rest of this paper is structured as follows. We begi i Sect. with a descriptio of related work. I Sect. 3 we preset our system model ad defiitios. We preset quorum systems for the replicatio of arbitrary data subject to arbitrary server failures i Sect. 4, ad i Sect. 5 we preset two variatios of these systems. We the detail a access protocol for replicated services that tolerate faulty cliets i additio to faulty servers i Sect. 6. We coclude i Sect. 7. Related work Our work was iflueced by the substatial body of literature o quorum systems for beig failures ad applicatios that make use of them, e.g., [4, 7, 11 13, 15,, 31, 37]. I particular, our grid costructio of Sect. 4 was iflueced by grid-like costructios for beig failures (e.g., [7]), ad we borrow our defiitio of load from [31]. Quorum systems have bee previously employed i the implemetatio of security mechaisms. Naor ad Wool [3] described methods to costruct a access-cotrol service usig quorums. Their costructios use cryptographic techiques to esure that out-of-date (but correct) servers caot grat access to uauthorized users. Agrawal ad El Abbadi [3] ad Mukkamala [30] cosidered the cofidetiality of replicated data despite the disclosure of the cotets of a threshold of the (otherwise correct) repositories. Their costructios used quorums with icreased itersectio, combied with Rabi s dispersal scheme [33], to ehace the cofidetiality ad availability of the data despite some servers crashig or their cotets beig observed. Our work differs from all of the above by cosiderig arbitrarily faulty servers, ad accommodatig failure scearios beyod a simple threshold of servers. Herlihy ad Tygar [16] applied quorums with icreased itersectio to the problem of protectig the cofidetiality ad itegrity of replicated data agaist a threshold of arbitrarily faulty servers. I their costructios, replicated data is stored ecrypted uder a key that is shared amog the servers usig a threshold secret-sharig scheme [35], ad each cliet accesses a threshold umber of servers to recostruct the key prior to performig (ecrypted) reads ad writes. This costructio exhibits oe approach to make replicated data self-verifyig via ecryptio, ad thus the quorum system they develop is a special case of our dissemiatio quorum systems, i.e., for a threshold of faulty servers. Sice the iitial coferece publicatio of this work [3], several works that build upo its cotributios have appeared. A subsequet paper [7] is devoted to costructios of maskig quorum systems for the special case of a threshold of faulty servers. Bazzi [6] explored a variatio of our quorum systems for sychroous systems. Probabilistic costructios for dissemiatio ad maskig quorum systems are explored i [8] ad [9], respectively. A practical effort for buildig a large-scale survivable data repository usig Byzatie quorums is described i [4], ad the costructio of a survivable cosesus object i this cotext is described i [5]. 3 Prelimiaries 3.1 System model We assume a uiverse U of servers, U =, ad a arbitrary umber of cliets that are distict from the servers. A quorum system Q U is a o-empty set of subsets of U, every pair of which itersect. Each Q Q is called a quorum. Servers (ad cliets) that obey their specificatios are correct. Afaulty server, however, may deviate from its specificatio arbitrarily. A fail-proe system B U is a oempty set of subsets of U, oe of which is cotaied i aother, such that some B B cotais all the faulty servers. The fail-proe system represets a assumptio characterizig the failure scearios that ca occur, ad could express typical assumptios that up to a threshold of servers fail (e.g., the sets B 1,...,B k could be all sets of f servers), but it also geeralizes to allow less uiform assumptios. For example, servers i physical proximity to each other or i the same admiistrative domai may exhibit correlated probabilities of beig captured, or servers with idetical hardware ad software platforms may have correlated probabilities of electroic peetratio. By exploitig such correlatios (i.e., kowledge of the collectio B ), we ca desig quorum systems that more effectively mask faulty servers. I the remaider of this sectio, ad throughout Sects. 4 ad 5, we assume that cliets behave correctly. I Sect. 6 we will relax this assumptio (ad will be explicit whe we do so). We assume that ay two processes (cliets or servers) ca commuicate over a poit-to-poit chael. If both edpoits of the chael are correct, the this chael is both autheticated ad reliable. That is, a correct process receives a message from aother correct process if ad oly if the other correct process set it. However, we do ot assume kow bouds o message trasmissio times; i.e., commuicatio is asychroous. 3. Access protocol We cosider a problem i which the cliets perform read ad write operatios o a variable x that is replicated at each server i the uiverse U. A copy of the variable x is
05 stored at each server, alog with a timestamp value t. Timestamps are assiged by a cliet to each replica of the variable whe the cliet writes the replica. Our protocols require that differet cliets choose differet timestamps, ad thus each cliet c chooses its timestamps from some globally-kow set T c that does ot itersect T c for ay other cliet c. The timestamps i T c ca be formed, e.g., as itegers appeded with the ame of c i the low-order bits. The read ad write operatios are implemeted as follows. Write. For a cliet c to write the value v, it queries servers to obtai a set of timestamps A = {<t u >} u Q for some quorum Q; chooses a timestamp t T c greater tha the highest timestamp value i A ad greater tha ay timestamp it has chose i the past; ad seds the update <v, t> to servers util it has received a ackowledgemet for this update from every server i some quorum Q. Read. For a cliet to read x, it queries servers to obtai a set of value/timestamp pairs A = {<v u,t u >} u Q for some quorum Q. The cliet the applies a determiistic fuctio Result() to A to obtai the result Result(A) of the read operatio. I the case of a write operatio, each server updates its local variable ad timestamp to the received values <v, t> oly if t is greater tha the timestamp curretly associated with the variable. I ay case, it returs a ackowledgemet to the cliet. Two poits about this descriptio deserve further discussio. First, the ature of the quorums Q ad the fuctio Result() are itetioally left uspecified; further clarificatio of these are the poit of this paper. Secod, read ad write operatios eed to exchage messages with a full quorum of servers. For example, the read operatio requires a cliet to obtai a set A cotaiig value/timestamp pairs from every server i some quorum Q. This requiremet stems from our lack of sychroy assumptios o the etwork: i geeral, the oly way that a cliet ca kow that it has accessed every correct server i a quorum is to access every server i the quorum. Our framework guaratees the availability of a quorum at ay momet, ad thus by attemptig the operatio at multiple quorums, a cliet ca evetually make progress. I some cases, the cliet ca achieve progress by icremetally accessig servers util it obtais resposes from a quorum of them. I Sects. 4 ad 5, we will argue the correctess of the above protocol istatiated with quorums ad a Result() fuctio that we will defie accordig to the followig sematics; a more formal treatmet of these cocepts ca be foud i [0]. We say that a read operatio begis whe the cliet iitiates the operatio ad eds whe the cliet determies the read result; a operatio to write value v with timestamp t begis whe the cliet iitiates it ad eds whe all correct servers i some quorum have received the update <v, t>. A operatio op 1 precedes a operatio op if op 1 eds before op begis (i real time). If op 1 does ot precede op ad op does ot precede op 1, the they are called cocurret. Give a set of operatios, a serializatio of those operatios is a total orderig o them that exteds the precedece orderig amog them. The, for the above protocol to be correct, we require that ay read that is cocurret with o writes returs the last value writte i some serializatio of the precedig writes. This will immediately imply safe variable sematics [0]. 3.3 Load A measure of the iheret performace of a quorum system is its load [31], defied as follows: Give a quorum system Q,aaccess strategy w is a probability distributio o the elemets of Q ; i.e., Q Q w(q) =1.w(Q) is the probability that quorum Q will be chose whe the service is accessed. Load is the defied as follows: Defiitio 3.1 Let a strategy w be give for a quorum system Q = {Q 1,...,Q m } over a uiverse U. For a elemet u U, the load iduced by w o u is l w (u) = Q w(q i u i). The load iduced by a strategy w o a quorum system Q is L w (Q ) = max {l w(u)}. u U The system load (or just load) o a quorum system Q is L(Q ) = mi {L w(q )}, w where the miimum is take over all strategies. We reiterate that the load is a best case defiitio. The load of the quorum system will be achieved oly if a optimal access strategy is used, ad oly i the case that o failures occur. A stregth of this defiitio is that load is a property of a quorum system, ad ot of the protocol usig it. A compariso of the defiitio of load to other seemigly plausible defiitios is give i [31]. 4 Maskig quorum systems I this sectio we itroduce maskig quorum systems, which ca be used to mask the arbitrarily faulty behavior of data repositories. To motivate our defiitio, suppose that the replicated variable x is writte with quorum Q 1, ad that subsequetly x is read usig quorum Q.IfB is the set of arbitrarily faulty servers, the the followig is obtaied by readig from Q : the correct value for x is obtaied from each server i (Q 1 Q ) \ B (see Fig. 1); out-of-date values are obtaied from Q \ (Q 1 B); ad arbitrary values are obtaied from Q B. I order for the cliet to obtai the correct value, the cliet must be able to idetify the most up-to-date value/timestamp pair as oe retured by a set of servers that could ot all be faulty. This yields requiremet M-Cosistecy below. I additio, sice commuicatio is asychroous ad thus accurate failure detectio is ot possible, i order for a cliet to kow it completes a operatio with all the correct servers of some quorum, it must be able to obtai resposes from a full quorum. Therefore, for availability we require that there be o set of faulty servers that itersects all quorums. Defiitio 4.1 A quorum system Q is a maskig quorum system for a fail-proe system B if the followig properties are satisfied.
06 B Q Q 1 Fig. 1. Readig from a maskig quorum Q M-Cosistecy: Q 1,Q Q B 1,B B : Q ) \ B 1 B M-Availability: B B Q Q : B Q = (Q 1 For example, i the case that at most f servers ca fail, M-Cosistecy guaratees that every pair of quorums itersect i at least f + 1 elemets, ad thus i f + 1 correct oes. If a read operatio accepts oly a value retured by at least f + 1 servers, the ay accepted value was retured by at least oe correct server. More geerally, the maskig quorum system requiremets eable a cliet to obtai the correct aswer from the service despite the Byzatie failure of ay fail-proe set. The write operatio is implemeted as described i Sect. 3. To obtai the correct value of x from a read operatio, the cliet reads a set of value/timestamp pairs from a quorum Q, discards values that are retured from ay B B or subsets thereof, ad chooses amog the remaiig values the oe with the highest timestamp. This guaratees correctess of the retured value/timestamp pair, which was received from some set B + Q of servers, where B + is ot cotaied i ay B B ad therefore must cotai at least oe correct server. Furthermore, it is easy to see that if the most recet write has completed i quorum Q, the all of the servers i Q Q \ B will retur this most up-to-date value, ad sice by defiitio Q Q \ B is ot cotaied i ay B B, this value will be retured by the read operatio. The read operatio is thus as follows: Read. For a cliet to read a variable x, it queries servers to obtai a set of value/timestamp pairs A = {<v u,t u >} u Q for some quorum Q. The cliet computes the set A = {<v, t> : B + Q [ B B [B + B] u B + [v u = v t u = t] ]}. The cliet the chooses the pair <v, t> i A with the highest timestamp, ad chooses v as the result of the read operatio; if A is empty, the cliet returs (a ull value, which idicates that the read failed). Lemma 4. A read operatio that is cocurret with o write operatios returs the value writte by the last precedig write operatio i some serializatio of all precedig write operatios. Proof. Let W deote the set of write operatios precedig the read. The read operatio will retur the value writte i the write operatio i W with the highest timestamp, sice, by the costructio of maskig quorum systems, this value/timestamp pair will appear i A ad will have the highest timestamp i A (ay pair with a higher timestamp will be retured oly by servers i some B B ). So, it suffices to argue that there is a serializatio of the writes i W i which this write operatio appears last, or i other words, that this write operatio precedes o other write operatio i W. This is immediate, however, as if it did precede aother write operatio i W, that write operatio would have a higher timestamp. This lemma implies that the protocol above implemets a multi-writer multi-reader safe variable [0]. A failure value ( ) may be retured whe some write overlaps a read operatio. From safe variables multi-writer multi-reader atomic variables ca be built usig well-kow costructios [18, 0, 1]. A ecessary ad sufficiet coditio for the existece of a maskig quorum system (ad a costructio for oe, if it exists) for ay give fail-proe system B is give i the followig theorem: Theorem 4.3 Let B be a fail-proe system for a uiverse U. The there exists a maskig quorum system for B iff Q = {U \ B : B B } is a maskig quorum system for B. Proof. Obviously, if Q is a maskig quorum system for B, the oe exists. To show the coverse, assume that Q is ot a maskig quorum. Sice M-Availability holds i Q by costructio, there exist Q 1,Q Q ad B,B B, such that (Q 1 Q )\B B. Let B 1 = U \Q 1 ad B = U \Q. By the costructio of Q, we kow that B 1,B B.By M-Availability, ay maskig quorum system for B must cotai quorums Q 1 Q 1, Q Q. However, for ay such Q 1,Q, it is the case that (Q 1 Q )\B (Q 1 Q )\B B, violatig M-Cosistecy. Therefore, there does ot exist a maskig quorum system for B uder the assumptio that Q is ot a maskig quorum system for B. Corollary 4.4 Let B be a fail-proe system for a uiverse U. The there exists a maskig quorum system for B iff for all B 1,B,B 3,B 4 B, U B 1 B B 3 B 4. I particular, suppose that B = {B U : B = f}. The, there exists a maskig quorum system for B iff >4f. Proof. By Theorem 4.3, there is a maskig quorum for B iff Q = {U \ B : B B } is a maskig quorum for B. By costructio, Q is a maskig quorum iff M-Cosistecy holds for Q, i.e., iff for all B 1,B,B 3,B 4 B : ((U \ B 1 ) (U \ B )) \ B 3 B 4 U \ (B 1 B ) B 3 B 4 U B 1 B B 3 B 4. The existece criterio for maskig quorum systems idetified by Theorem 4.3 characterizes all possible maskig systems for the fail-proe system B. I particular, the system Q i Theorem 4.3 is domiated (i the sese of [1])
07 by ay other maskig quorum system Q for B, i that for every Q Q there must exist Q Q such that Q Q. While this provides a characterizatio of maskig quorum systems for ay fail-proe system B, it does ot help i costructig oes to meet ay specific requiremets. Garcia- Molia ad Barbara [1] preset techiques for eumeratig a certai class of (o-byzatie) quorum systems. Their methods are ot directly applicable for eumeratig maskig quorum systems, ad we leave as a ope research topic the questio of efficietly mechaizig maskig quorum geeratio. A separate paper [7] provides costructios that are optimal i load ad various availability measures for ay threshold failure assumptio up to the maximum of /4. The followig theorem was proved i [31] for beigfailure quorum systems, ad holds a fortiori for maskig quorums (as a result of M-Cosistecy). Let c(q ) deote the size of the smallest quorum of Q. Theorem 4.5 [31] If Q is a quorum system over a uiverse of elemets, the L(Q ) max{ c(q 1, c(q ) ) }, ad thus, L(Q ) 1. Below we give several examples of maskig quorum systems ad describe their properties. Example 4.6 (f-maskig) Suppose that B = {B U : B = f}, >4f. Note that this correspods to the usual threshold assumptio that up to f servers may fail. The, the quorum system Q = {Q U : Q = +f+1 } is a maskig quorum system for B. M-Cosistecy is satisfied because ay Q 1,Q Q will itersect i at least f +1 elemets. M-Availability holds because +f+1 f. A strategy that assigs equal probability to each quorum iduces a load of 1 +f+1 o the system. By Theorem 4.5, this load is i fact the load of the system. The followig example is iterestig sice its load decreases as a fuctio of, ad sice it demostrates a method for esurig system-wide cosistecy i the face of Byzatie failures while requirig the ivolvemet of fewer tha a majority of the correct servers. These advatages are dramatic whe is sufficietly large, e.g., hudreds of servers. Example 4.7 (Grid quorums) Suppose that the uiverse of servers is of size = k for some iteger k ad that B = {B U : B = f}, 3f +1. Arrage the uiverse ito a grid, as show i Fig.. Deote the rows ad colums of the grid by R i ad C i, respectively, where 1 i. The, the quorum system { Q = C j R i : I,{j} {1... } }, I =f +1 i I is a maskig quorum system for B. M-Cosistecy holds sice every pair of quorums itersect i at least f + 1 elemets (the colum of oe quorum itersects the f + 1 rows of the other), ad M-Availability holds sice for ay choice of f faulty elemets i the grid, f +1 full rows ad a colum remai available. A strategy that assigs equal probability to each quorum iduces a load of (f+) (f+1), ad agai by Theorem 4.5, this is the load of the system. Note that by choosig B = { } (i.e., f =0)ithe example above, the resultig costructio has a load of O( 1 ), which asymptotically meets the bouds give i Theorem 4.5. I geeral, however, this costructio yields a load of O( f ), which is ot optimal: Malkhi et al. [7] show a lower boud of f+1 o the load of ay maskig quorum system for B = {B U : B = f}, ad provide a costructio whose load matches that boud. k k Fig.. Grid costructio, k k =, f = 1 (oe quorum shaded) Example 4.8 (Partitio) Suppose that B = {B 1,...,B m }, m>4, is a partitio of U where B i /= for all i, 1 i m. This choice of B could arise, for example, i a wide area etwork composed of multiple local clusters, each cosistig of some B i, ad expresses the assumptio that at ay time, at most oe cluster is faulty. The, ay collectio of oempty sets ˆB i B i,1 i m, ca be thought of as super-elemets i a uiverse of size m, with a threshold assumptio f = 1 (see Fig. 3). Therefore, the followig is a maskig quorum system for B : { } Q = ˆB i : I {1,...,m}, I = m+3 i I M-Cosistecy is satisfied because the itersectio of ay two quorums cotais elemets from at least three sets i B. M-Availability holds sice there is o B B that itersects all quorums. A strategy that assigs equal probability to each o the system regardless of the size of each ˆB i, ad agai Theorem 4.5 implies that this is the load of the system. If m = k for some k, the a more efficiet costructio ca be achieved by formig the grid costructio from Example 4.7 o the super elemets { ˆB i }, achievig a load of quorum iduces a load of 1 m m+3 4 m 3 m. 5 Variatios 5.1 Dissemiatio quorum systems As a special case of services that ca employ quorums i a Byzatie eviromet, we ow cosider applicatios i which the service is a repository for self-verifyig iformatio, i.e., iformatio that oly cliets ca create ad to which cliets ca detect ay attempted modificatio by a faulty server. A atural example is a database of public key
08 U { }} { ˆB 1 ˆB ˆB3 ˆB4 ˆB5 } {{ } } {{ } } {{ } }{{} } {{ } B 1 B B 3 B 4 B 5 Fig. 3. Partitio {B 1,B,B 3,B 4,B 5 }, ˆB i s shaded certificates as foud i may public key distributio systems (e.g., [8, 19, 36]). I its simplest form, a public key certificate is a structure cotaiig a ame for a user ad a public key, ad represets the assertio that the idicated public key ca be used to autheticate messages from the idicated user. This structure is digitally siged (e.g., [34]) by a certificatio authority so that ayoe with the public key of this authority ca verify this assertio ad, providig it trusts the authority, use the idicated public key to autheticate the idicated user. Due to this sigature, it is ot possible for a faulty server to udetectably modify a certificate it stores. However, a faulty server ca udetectably suppress a chage from propagatig to cliets, simply by igorig a update from a certificatio authority. This could have the effect, e.g., of suppressig the revocatio of a key that has bee compromised. As ca be expected, the use of digital sigatures to verify data decreases the cost of accessig replicated data. To support such a service, we employ a dissemiatio quorum system, which has weaker requiremets tha maskig quorums, but which evertheless esures that i applicatios like those above, self-verifyig writes will be propagated to all subsequet read operatios despite the arbitrary failure of some servers. To achieve this, it suffices for the itersectio of every two quorums to ot be cotaied i ay set of potetially faulty servers (so that a writte value ca propagate to a read). This leads to requiremet D-Cosistecy below. Ad, supposig that operatios are required to cotiue i the face of failures, the due to the lack of accurate failure detectio, there should be quorums that a faulty set caot disable; this yields requiremet D-Availability below. Defiitio 5.1 A quorum system Q is a dissemiatio quorum system for a fail-proe system B if the followig properties are satisfied. D-Cosistecy: Q 1,Q Q B B : Q 1 Q B D-Availability: B B Q Q : B Q = A dissemiatio quorum system will suffice for propagatig self-verifyig iformatio as i the applicatio described above. The write operatio is implemeted as described i Sect. 3, ad the read operatio becomes: Read. For a cliet to read a variable x, it queries servers to obtai a set of value/timestamp pairs A = {<v u,t u >} u Q for some quorum Q. The cliet the discards those pairs that are ot verifiable (e.g., usig a appropriate digital sigature verificatio algorithm) ad chooses from the remaiig pairs the pair <v, t> with the largest timestamp. v is the result of the read operatio. It is importat to ote that timestamps must be icluded as part of the self-verifyig iformatio, so they caot be udetectably altered by faulty servers. I the case of the applicatio described above, existig stadards for public key certificates (e.g., [8]) already require a real-time timestamp i the certificate. The followig lemma states correctess of the above protocol usig dissemiatio quorum systems. The proof is almost idetical to that for maskig quorum systems. Lemma 5. A read operatio that is cocurret with o write operatios returs the value writte by the last precedig write operatio i some serializatio of all precedig write operatios. Due to the assumptio of self-verifyig data, we ca also prove i this case the followig property. Lemma 5.3 A read operatio that is cocurret with oe or more write operatios returs either the value writte by the last precedig write operatio i some serializatio of all precedig write operatios, or ay of the values beig writte i the cocurret write operatios. The above lemmata imply that the protocol above implemets a regular variable [0]. Theorems aalogous to the oes give for maskig quorum systems above are easily derived for dissemiatio quorums. Below, we list these results without proof. Theorem 5.4 Let B be a fail-proe system for a uiverse U. The there exists a dissemiatio quorum system for B iff Q = {U \ B : B B } is a dissemiatio quorum system for B. Corollary 5.5 Let B be a fail-proe system for a uiverse U. The there exists a dissemiatio quorum system for B iff for all B 1,B,B 3 B, U B 1 B B 3. I particular, suppose that B = {B U : B = f}. The, there exists a dissemiatio quorum system for B iff >3f. Below, we provide several example costructios of dissemiatio quorum systems. Example 5.6 (f-dissemiatio) Suppose that B = {B U : B = f}, >3f. Note that this correspods to the usual threshold assumptio that up to f servers may fail. The, the quorum system Q = {Q U : Q = +f+1 } is a dissemiatio quorum system for B with load 1 +f+1. Example 5.7 (Grid) Let the uiverse be arraged i a grid as i Example 4.8 above, ad let B = {B U : B = f}, f +1. The, the quorum system
09 { Q = C j R i : I,{j} {1... } }, I = f +1 i I B Q 1 is a dissemiatio quorum system for B. The load of this system is (f+) (f+1). Example 5.8 (Partitio) Suppose that B = {B 1,...,B m }, m > 3, is a partitio of U as i Fig. 3. For ay collectio of oempty sets ˆB i B i, 1 i m, the f- dissemiatio costructio of Example 5.6 o the superelemets ˆB i B i (as i Example 4.8) yields a dissemiatio quorum system with a load of 1 m m+.ifm = k for some k, the Grid costructio of Example 5.7 achieves a load of 3 m m. 5. Opaque maskig quorum systems Maskig quorums impose a requiremet that cliets kow the fail-proe system B, while there may be reasos that cliets should ot be required to kow this. First, it somewhat complicates the cliet s read protocol, i particular, whe o cocise descriptio of B exists. Secod, by revealig the failure scearios for which the system was desiged, the system also reveals the failure scearios to which it is vulerable, which could be exploited by a attacker to guide a active attack agaist the system. By ot revealig the failproe system to cliets, ad ideed givig each cliet oly a small fractio of the possible quorums, the system ca somewhat obscure (though perhaps ot secure i ay formal sese) the failure scearios to which it is vulerable, especially i the absece of cliet collusio. I this sectio we describe oe way to modify the maskig quorum defiitio of Sect. 4 to be opaque, i.e., to elimiate the eed for cliets to kow B. I the absece of the cliet kowig B, the oly method of which we are aware for the cliet to reduce a set of replies from servers to a sigle reply from the service is via votig, i.e., choosig the reply that occurs most ofte. I order for this reply to be the correct oe, however, we must stregthe the requiremets o our quorum systems. Specifically, suppose that the variable x is writte with quorum Q 1, ad that subsequetly x is read with quorum Q.IfB is the set of arbitrarily faulty servers, the (Q 1 Q ) \ B is the set of correct servers that possess the latest value for x (see Fig. 4). I order for the cliet to obtai this value by vote, this set must be larger tha the set of faulty servers that are allowed to respod, i.e., Q B. Moreover, sice these faulty servers ca team up with the out-of-date but correct servers i a effort to suppress the write operatio, the umber of correct, up-to-date servers that reply must be o less tha the umber of faulty or out-of-date servers that ca reply, i.e., (Q B) (Q \Q 1 ). Fially, to effectively mask failures by ay B B i a asychroous eviromet, we add the availability requiremet (O-Availability). Defiitio 5.9 A quorum system Q is a opaque maskig quorum system for a fail-proe system B if the followig properties are satisfied. O-Cosistecy1: Q 1,Q Q B B : (Q 1 Q ) \ B (Q B) (Q \ Q 1 ) Q O1: O: + > Fig. 4. O-Cosistecy1 ad O-Cosistecy O-Cosistecy: Q 1,Q Q B B : (Q 1 Q ) \ B > Q B O-Availability: B B Q Q : B Q = Note that O-Cosistecy1 admits the possibility of equality i size betwee (Q 1 Q ) \ B ad (Q B) (Q \ Q 1 ). Equality is sufficiet sice, i the case that the faulty servers team up with the correct but out-of-date servers i Q, the value retured from (Q 1 Q ) \ B will have a higher timestamp tha that retured by (Q B) (Q \ Q 1 ). Therefore, i the case of a tie, a reader ca choose the value with the higher timestamp. It is iterestig to ote that a strog iequality i O-Cosistecy1 would permit a correct implemetatio of a sigle-reader siger-writer safe variable that does ot use timestamps (by takig the majority value i a read operatio). It is ot difficult to verify that a opaque maskig quorum system eables a cliet to obtai the correct aswer from the service. The write operatio is implemeted as described i Sect. 3, ad the read operatio becomes: Read. For a cliet to read a variable x, it queries servers to obtai a set of value/timestamp pairs A = {<v u,t u >} u Q for some quorum Q. The cliet chooses the pair <v, t> that appears most ofte i A, ad if there are multiple such pairs, the oe with the highest timestamp. The value v is the result of the read operatio. Opaque maskig quorum systems, combied with the access protocol described previously, provide the same sematics as regular maskig quorum systems. The proof is almost idetical to that for regular maskig quorums. Lemma 5.10 A read operatio that is cocurret with o write operatios returs the value writte by the last precedig write operatio i some serializatio of all precedig write operatios. Below we give several examples of opaque maskig quorum systems (or just opaque quorum systems ) ad describe their properties. Example 5.11 (f-opaque) Suppose that B = {B U : B = f} where 5f ad f > 0. The, the quorum
10 system Q = {Q U : Q = +f 3 } is a opaque quorum system for B, whose load is 1 +f 3. The ext theorem proves a resiliece boud for opaque quorum systems. Theorem 5.1 Suppose that B = {B U : B = f}. There exists a opaque quorum system for B iff 5f. Proof. That 5f is sufficiet is already demostrated i Example 5.11 above. Now suppose that Q is a opaque quorum system for B. Fix ay Q 1 Q such that Q 1 f (Q 1 exists by O-Availability); ote that Q 1 > f by O-Cosistecy. Choose B 1 Q 1, B 1 = f, ad some Q Q such that Q U\B 1 (Q exists by O-Availability). The Q 1 Q f. By O-Cosistecy, Q 1 Q f, ad therefore there is some B B such that B Q 1 Q. The 3f Q Q 1 B = (Q Q 1 ) \ B (Q 1 \ Q ) (Q 1 B ) (1) = Q 1 \ Q + B B 1 + B =f Where (1) holds by O-Cosistecy1. Therefore, we have 5f. Example 5.13 (Partitio) Suppose that B = {B 1,...,B 3k }, k>1, is a partitio of U where B i /= for all i, 1 i 3k. Choose ay collectio of sets ˆB i B i,1 i 3k, such that ˆB i = c for a fixed costat c>0. The, the f-opaque costructio of Example 5.11 o the super-elemets { ˆB i } (as i Example 4.8), with uiverse size 3k ad a threshold assumptio f = 1, yields a opaque quorum system with load k+1 3k. Ulike the case for regular maskig quorum systems, a ope problem is to fid a techique for testig whether, give a fail-proe system B, there exists a opaque quorum system for B (other tha a exhaustive search of all subsets of U ). I the costructios i Examples 5.11 ad 5.13, the resultig quorum systems exhibited loads that at best were costat as a fuctio of. I the case of maskig quorum systems, we were able to exhibit quorum systems whose load decreased as a fuctio of, amely the grid quorums. A atural questio is whether there exists a opaque quorum system for ay fail-proe system B that has load that decreases as a fuctio of. I this sectio, we aswer this questio i the egative: we show a lower boud of 1 o the load for ay opaque quorum system costructio, regardless of the fail-proe system. Theorem 5.14 The load of ay opaque quorum system is at least 1. Proof. O-Cosistecy1 implies that for ay Q 1,Q Q, Q 1 Q Q 1 \ Q, ad thus Q 1 Q Q1. Let w be ay strategy for the quorum system Q, ad fix ay Q 1 Q. The, the total load iduced by w o the elemets of Q 1 is: l w (u) = w(q i ) u Q 1 u Q 1 Q i u = w(q i ) Q i u Q 1 Q i Q i Q 1 w(q i) = Q 1 Therefore, there must be some server i Q 1 that suffers a load at least 1. We ow preset a geeric costructio of a opaque quorum system for B = { } ad icreasigly large uiverse sizes, that has a load that teds to 1 as grows. We give this costructio primarily to show that i at least some cases the lower boud of 1 is tight; due to the requiremet that B = { }, this costructio is ot of practical use for copig with Byzatie failures. Example 5.15 Suppose that the uiverse of servers is U = {u 1,...,u } where = l for some l>, ad that B = { }. Cosider the Hadamard matrix H(l), costructed recursively as follows: [ ] 1 1 H(1) = H(k) = 1 1 [ H(k 1) H(k 1) H(k 1) H(k 1) ],k H(l) has the property that H(l)H(l) T = I, where I is the idetity matrix. Usig well-kow iductive argumets [14, Ch. 14], it ca be show that (i) the first row ad colum cosist etirely of 1 s, (ii) the i-th row ad i-th colum, for each i, has 1 s i positios (ad similarly for 1 s), ad (iii) ay two rows (ad ay two colums) i, j have idetical elemets i positios, i.e., 1 s i 4 commo positios ad 1 s i 4 commo positios. We treat the rows of H(l) as idicators of subsets of U. That is, let Q i = {u j : H(l)[i, j] =1} be the set defied by the i-th row, 1 i. Note that Q 1 = ad that u 1 is ot icluded i ay Q i. We claim that the system Q = {Q,..., Q } is a opaque quorum system for B. Usig properties (i) (iii) above, we have that Q i = for each i ; that each u i, i, is i exactly of the sets Q,...,Q ; ad that for ay i, j, if i/= j the Q i Q j = 4. From these, the O-Cosistecy1 ad O- Cosistecy requiremets ca be quickly verified, ad a load of 1 ca be achieved, e.g., with a strategy that assigs equal probability to each quorum. 6 Faulty cliets So far, we have bee cocered with providig a cosistet service to a set of correct cliets. I this sectio, we exted our treatmet to address faulty cliets i additio to faulty servers. Sice updates may ow be geerated by faulty cliets, we ca make o assumptio of self-verifyig
11 1. If a server receives <update, Q, v, t> from a cliet c, ift T c, ad if the server has ot previously received from c a message <update, Q, v, t > where either t = t ad v /= v or t >t, the the server seds <echo, Q, v, t> to each member of Q.. If a server receives idetical echo messages <echo, Q, v, t> from every server i Q, the it seds <ready, Q, v, t> to each member of Q. 3. If a server receives idetical ready messages <ready, Q, v, t> from a set B + of servers, such that B + B for all B B, the it seds <ready, Q, v, t> to every member of Q if it has ot doe so already. 4. If a server receives idetical ready messages <ready, Q, v, t> from a set Q of servers, such that for some B B, Q = Q \ B, the (i) if t is greater tha the timestamp it curretly holds, the it updates its variable ad timestamp to v ad t, respectively, ad (ii) regardless of whether it updates the variable ad timestamp, it seds a ackowledgmet message to c where T c t. Fig. 5. A update protocol data, ad thus use maskig quorum systems (Sect. 4) to implemet the service. We focus o esurig the cosistecy of the data stored at the replicated service as see by correct cliets oly. A difficulty i hadlig faulty cliets is that a faulty writer might sed differet updates to differet servers ad may fail to cotact a full quorum. We therefore modify the write protocol to prevet cliets from leavig the service i a icosistet state, ad to guaratee that updates propagate to (at least) a full quorum. We maitai availability of the service despite the possibly malicious behavior by ay umber of cliets, so that a correct cliet ca always complete a write operatio with as little as oe available quorum. The treatmet here provides a sigle-writer multi-reader safe variable sematics (igorig reads by faulty cliets). Sice the iitial coferece publicatio of this work [3], sigle-writer objects with stroger sematics i the case of faulty cliets have bee costructed usig Byzatie quorums ad have bee used to solve the distributed cosesus problem [4]. Other work has exteded the treatmet here to provide multi-writer variables [5]. A alterative ad geeral correctess coditio for shared objects accessed by faulty cliets has bee developed i [6], which our protocol here also satisfies. For brevity, however, here we cotiue i the framework of the previous sectios. The write protocol performed by a cliet is chaged i that a writer computes the timestamp locally, without cosultig the servers, ad i that it deotes the quorum it attempts to access i the update request. We replace the write operatio of Sect. 3 by the followig: Write. For a cliet c to write the value v, it chooses a timestamp t T c greater tha ay value it has chose before, ad the performs the followig two steps: (i) it chooses a quorum Q ad seds a update message <update,q,v,t> to each server i Q, ad (ii) if after some timeout period, it has ot received a ackowledgemet from every server i Q, it repeats (i) (ad (ii)). Every server that receives a update message from a cliet egages i a update protocol to guaratee uiqueess of the value associated with a timestamp ad its propagatio to a full quorum. The protocol is preseted i Fig. 5. I order to argue correctess for this protocol, we have to adapt the defiitio of operatio precedece ad operatio duratio to allow for the behavior of a faulty cliet. The reaso is that it is uclear how to defie whe a operatio by a faulty cliet begis or eds, as the cliet ca behave outside the specificatio of ay protocol. We make use of the followig termiology: Defiitio 6.1 We say that a server delivers a update <v, t> whe it receives <ready,q,v,t> from each server i the set Q = Q \ B for some fail-proe set B (step 4 of the update protocol i Fig. 5). We ow say that a write operatio that writes v with timestamp t begis whe the first correct server receives <update,q,v,t>, ad eds whe all correct servers i some quorum have delivered the update. Note that by this defiitio, a write operatio by a faulty cliet could last arbitrarily log, ad could overlap other writes by the same cliet. Nevertheless, carryig over the remaider of the precedece defiitio, we have that the write protocol together with the update protocol i Fig. 5 implemet a sigle-writer multireader safe variable: Lemma 6. A correct process read operatio that is cocurret with o write operatios returs the value writte by the last precedig write operatio i some serializatio of all precedig write operatios. To prove this lemma, we eed the followig properties of our protocol: Lemma 6.3 A correct server delivers <v, t> oly if some correct server previously received <update,q,v,t>. Proof. To deliver <v, t>, a correct server must receive a ready message from some correct server. Moreover, the first <ready, Q, v, t> message from a correct server is set oly after it receives <echo,q,v,t> from each member of Q. Sice, a correct member seds <echo, Q, v, t> oly if it first receives <update,q,v,t>, this proves the lemma. Lemma 6.4 (Agreemet) If a correct server delivers <v, t> ad a correct server delivers <v,t>, the v = v. Proof. As argued i the previous lemma, for a correct server to deliver <v, t>, <echo, Q, v, t> must have bee set by all servers i Q. Similarly, <echo, Q, v, t> must have bee set by all servers i Q. Sice every two quorums itersect i (at least) oe correct server, ad sice ay correct server seds <echo,, ˆv, t> for at most oe value ˆv, v must be idetical to v. Proof of Lemma 6.. Let W deote the set of write operatios precedig the read. Note that by Lemma 6.4, ay value/timestamp pair i W is well defied, i.e., the same value correspods to ay timestamp at all correct servers that deliver it. By defiitio, every write i W was delivered to a full quorum, ad by assumptio ad Lemma 6.3, o correct server has delivered ay write outside W. Therefore, by the costructio of maskig quorum systems, the
1 read operatio will retur the value writte i the write operatio i W with the highest timestamp. So, it suffices to argue that there is a serializatio of the writes i W i which this write operatio appears last, or i other words, that this write operatio precedes o other write operatio i W. This results, however, from the fact that there is a sigle writer ad that servers echo a update request oly if its timestamp is higher tha the oe they have i store, ad so ay later write operatio has a higher timestamp. I additio, we argue liveess ad completeess of our protocol as follows: Lemma 6.5 (Propagatio) If a correct server delivers <v, t>, the evetually there exists a quorum Q Q such that every correct server i Q delivers <v, t>. To prove this lemma, we make use of the followig fact: Lemma 6.6 If Q is a maskig quorum system over a uiverse U with respect to a fail-proe system B, the Q Q B 1,B,B 3 B, Q B 1 B B 3. Proof. Assume otherwise for a cotradictio, i.e., that there is a Q Q ad B 1,B,B 3 B such that Q B 1 B B 3. By M-Availability, there exists Q Q, Q B 1 =. The, Q Q B B 3 ad thus (Q Q )\B B 3, cotradictig M-Cosistecy. Proof of Lemma 6.5. Accordig to the protocol, the correct server that delivered <v, t> received a message <ready, Q, v, t> from each server i Q = Q \ B for some Q Q ad B B. Sice, for some B B, (at least) all the members i Q \ B are correct, every correct member of Q receives <ready, Q, v, t> from each of the members of B + = Q \B. Sice, B B, Q \B B (by Lemma 6.6), the ready messages from B + cause each correct member of Q to sed such a ready message. Cosequetly, <v, t> is delivered by all of the correct members of Q. Lemma 6.7 (Validity) If a correct cliet c seds <update,q, v, t> to every server i Q ad all servers i Q are correct, the evetually a correct server delivers <v, t>. Proof. Sice both the cliet ad all of the members of Q are correct, <update, Q, v, t> will be received ad echoed by every member i Q. Cosequetly, all the servers i Q will sed <ready, Q, v, t> messages to the members of Q, ad will evetually deliver <v, t>. 7 Coclusios The literature cotais a abudace of protocols that use quorums for accessig replicated data. This approach is appealig for costructig replicated services as it allows for icreasig the availability ad efficiecy of the service while maitaiig its cosistecy. Our work exteds this successful approach to eviromets where both the servers ad the cliets of a service may deviate from their prescribed behavior i arbitrary ways. We itroduced a ew class of quorum systems, amely maskig quorum systems, ad devised protocols that use these quorums to ehace the availability of systems proe to Byzatie failures. We also explored two variatios of our quorum systems, amely dissemiatio ad opaque maskig quorums, ad for all of these classes of quorums we provided various costructios ad aalyzed the load they impose o the system. Our work leaves a umber of itriguig ope challeges ad directios for future work. Oe is to characterize the average performace of our quorum costructios ad their load i less-tha-ideal scearios, e.g., whe failures occur. Also, i this work we described oly quorum systems that are uiform, i the sese that ay quorum is possible for both read ad write operatios. I practice it may be beeficial to employ quorum systems with distiguished read quorums ad write quorums, with cosistecy requiremets imposed oly betwee pairs cosistig of at least oe write quorum. Although this does ot seem to improve our lower bouds o the overall load that ca be achieved, it may allow greater flexibility i tradig betwee the availability of reads ad writes. Ackowledgmets. We are grateful to Adrew Odlyzko for suggestig the use of Hadamard matrices to costruct opaque maskig quorum systems with a asymptotic load of 1. We also thak Yehuda Afek ad Michael Merritt for helpful discussios, ad Vassos Hadzilacos ad Rebecca Wright for may helpful commets o earlier versios of this paper. A isightful commet by Rida Bazzi led to a substatial improvemet over a previous versio of this paper. Refereces 1. Afek A, Attiya H, Dolev D, Gafi E, Merritt M, Shavit N: Atomic sapshots of shared memory. Joural of the ACM 40(4):873 890. Afek Y, Dolev D, Gafi E, Merritt M, Shavit N: A bouded first-i first-eabled-solutio to the l-exclusio problem. I: Proceedigs of the 4th Iteratioal Workshop o Distributed Algorithms, LNCS 486, Spriger 1990 3. Agrawal D ad El Abbadi A: Itegratig security with fault-tolerat distributed databases. Comput J 33(1):71 78 4. Agrawal D, El Abbadi A: A efficiet ad fault-tolerat solutio for distributed mutual exclusio. ACM Trasactios o Computer Systems 9(1):1 0 (1991) 5. Aderso JH: Composite registers. Distrib Comp 6(3):141 154 (1993) 6. Bazzi RA: Sychroous Byzatie quorum systems. I: Proceedigs of the 16th ACM Symposium o Priciples of Distributed Computig, pp 59 66 (1997) 7. Cheug SY, Ammar MH, Ahamad M: The grid protocol: A high performace scheme for maitaiig replicated data. I: Proceedigs of the 6th IEEE Iteratioal Coferece o Data Egieerig, pp 438 445, 1990 8. Iteratioal Telegraph ad Telephoe Cosultative Committee (CCITT): The Directory Autheticatio Framework, Recommedatio X.509, 1988 9. Dolev D, Gafi E, Shavit N: Toward a o-atomic era: l-exclusio as a test case. I: Proceedigs of the 0th ACM Symposium o Theory of Computig, pp 78 9, May 1988 10. Dolev D, Shavit N: Bouded cocurret time-stamp systems are costructible. SIAM Joural of Computig, to appear. Also i: Proceedigs of the 1st ACM Symposium o the Theory of Computig, pp 454 466, 1989 11. El Abbadi A, Toueg S: Maitaiig availability i partitioed replicated databases. ACM Trasactios o Database Systems 14():64 90 (1989) 1. Garcia-Molia H, Barbara D; How to assig votes i a distributed system. Joural of the ACM 3(4):841 860 (1985)