How to Select a Replication Protocol According to Scalability, Availability and Communication Overhead

Transcription

1 Ho to Select a Replicatio Protocol Accordig to Scalability, Availability ad Commuicatio Overhead R. Jiméez-Peris, M. Patiño-Martíez Λ School of Computer Sciece Techical Uiversity of Madrid (UPM) Madrid, Spai frjimeez, mpatiog@fi.upm.es G. Alo Departmet of Computer Sciece Siss Federal Istitute of Techology (ETHZ) Zürich, Sitzerlad alo@if.ethz.ch B. Kemme School of Computer Sciece McGill Uiversity Motreal, Caada kemme@cs.mcgill.ca Abstract Data replicatio is playig a icreasigly importat role i the desig of parallel iformatio systems. I particular, the idespread use of cluster architectures i high-performace computig has created may opportuities for applyig data replicatio techiques i e areas. For istace, as part of ork related to cluster computig i bioiformatics, e have bee cofroted ith the problem of havig to chose a optimal replicatio strategy i terms of scalability, availability, ad commuicatio overhead. Thus, e have evaluated several represetative replicatio protocols i order to better uderstad their behavior i practice. The results obtaied are surprisig i that they challege may of the assumptios behid existig protocols. Our evaluatio idicates that the covetioal read-oe/rite-all approach is the best choice for a large rage of applicatios requirig data replicatio. We believe this is a importat result for aybody developig code for computig clusters as the read-oe/rite-all strategy is much simpler to implemet ad more flexible tha quorum-based approaches. I this paper e sho that, i additio, it is al the best choice usig a umber of other selectio criteria. Keyords: data replicatio, quorums, trasactios. Itroductio Data replicatio is playig a icreasigly importat role i the desig of distributed iformatio systems. I particular, the idespread use of cluster architectures i highperformace computig [] has created may opportuities for replicated databases. I applicatios that ca be easily parallelized (eb servers, data miig, computatioal geomics) it is coceivable to use a series of replicated databases as the meas to share up-to-date iformatio amog all the sites i a cluster. The database takes care of complex tasks such as idexig, recoverability, or cocurrecy cotrol thereby makig the applicatio easier to develop ad maitai. Λ This ork has bee partially fuded by the Spaish Research Coucil (CICYT), cotract umber TIC--C-. Data replicatio, hoever, must be used carefully he performace is cocered. The most relevat strategy for cluster computig is eager, update everyhere replicatio (e.g., full cosistecy ad updates ca be geerated ayhere i the system). Ufortuately, this strategy ca be tremedously iefficiet []. There are ays to get aroud the iefficiecies, at least for systems up to a give size [, ], but there are still may aspects of the problem that have ot bee studied i detail. More specifically, oe could thik of usig quorum based strategies to achieve cosistecy across the cluster hile still miimizig the overall cost i terms of performace pealties, commuicatio overhead, ad overall availability. As part of ork related to cluster computig i bioiformatics [], e have bee cofroted ith exactly this problem. We have a umber of large-scale parallel computatios that eed to share data dyamically. The best lutio is clearly to use a replicated database (for a umber of reas that are beyod the scope of this paper but that iclude the volume of data, the eed to idex it ad the ability to perform complex queries). Our ituitio told us that e should be able to improve the system by usig quorums but it as ot clear hich oe ould provide the best results. Although there are several studies of the availability ad load distributio of quorum systems [,,,, ] e could ot fid i the literature ay idicatio of hat type of quorum could be most suitable for our purposes. Thus, e evaluated several protocols i order to better uderstad the problem. Ulike previous ork, our aalysis is based o realistic eviromets. For istace, comparis amog quorum properties have bee typically made o a asymptotic basis. Hoever, asymptotic aalysis for eager data replicatio is ot very useful sice this kid of replicatio does ot scale beyod a fe tes of sites. For the aalysis to be meaigful, it is ecessary to take ito accout costat ad multiplicative factors that are usually discarded i a asymptotic aalysis. Additioally, e take ito accout commo database optimizatio strategies that substatially chage the behavior of the protocols. These optimizatios itroduce asymmetries i the processig of trasactios. Such asymmetries have ot bee take ito accout by previous studies, hich rogly assume that trasactios require the same computig reurces at all

2 sites here they are executed. The study of the ifluece of this asymmetry is oe of the mai cotributios of this paper. The results of our aalyses are surprisig i that they challege the otio that quorums improve performace, availability, or commuicatio overhead. The results idicate that the covetioal read-oe/rite-all approach is clearly the best choice for a large rage of applicatios requirig data replicatio. This is a importat result sice the read-oe/rite-all strategy is much simpler to implemet ad much more flexible tha quorum based approaches. Thus, scietists ad applicatio developers usig replicated databases i clusters ca take advatage of a simpler desig hile still havig the guaratee that they are gettig the best possible behavior out of the replicatio protocol. I hat follos e aalyze a umber of replicatio protocols from the poit of vie of scalability (Sectio ), availability (Sectio ), ad commuicatio overhead (Sectio ). For obvious reas, e do ot aalyze all existig protocols but a represetative subset. We cosider realistic scearios ad study the effect of typical optimizatio strategies used i databases. For each protocol, e idicate the applicatio for hich they are best suited ad the compare all of them from the poit of vie of cluster computig (Sectio ). System Model A replicated database cosists of a group of sites = fn ;N ;:::;N g hich commuicate by exchagig messages. Sites are fail-stop, ad site failures ca be detected. We cosider a crash-recovery model here sites ca recover ad re-joi the system after sychroizig their state ith the oe of the ruig replicas. The database is fully replicated, ad thus, each site cotais a copy of the database. We assume that all sites are homogeeous ad each site is able to execute t trasactios per secod (tps). Cliets iteract ith the database by issuig trasactios. Trasactios are executed atomically, i.e., a trasactio either commits or aborts. Trasactios are partially ordered sets of read (r) ad rite () operatios. Trasactios are executed atomically, i.e., a trasactio either commits are aborts at all participatig sites. If a trasactio cotais rite operatios, a -phase-commit protocol at the ed of the trasactio is executed amog all sites. For replicated databases, the correctess criterio is oecopy-serializability []. I this criterio, each copy must appear as a sigle logical copy ad the executio of cocurret trasactios must be equivalet to a serial executio over all the physical copies. A cliet submits a trasactio to oe of the sites i the system, ad this site coordiates its actios ith the rest of the system. A trasactio is called local at the site it is submitted to, ad remote at the other sites. We assume that all sites receive the same amout of local trasactios. We cosider i the study to kids of trasactios: queries, hich cotai oly read operatios, ad update trasactios, hich cotai both read ad rite operatios. Replicatio Protocols Several protocols have bee proposed for database replicatio. These protocols maily differ i the umber of sites cotacted to perform a read or rite operatio. I this sectio e describe the four protocols e ill aalyze.. Read-oe Write-all Uder the read-oe rite-all (ROWA) algorithm [], a trasactio just reads a copy of the data (the local oe to miimize the commuicatio cost), hile rite operatios must be performed at all sites. Read-oly trasactios (queries) ca be executed locally, hile update trasactios are executed at all sites i the system. Writes ca be propagated immediately or they ca be deferred util the trasactio commits. A aive versio of ROWA ould require all replicas to be available [] to perform a rite operatio. A variatio of this techique ko as read-oe rite-all-available (ROWAA) [] improves the availability of the database. From o o e ill cosider the later protocol.. Quorums Quorums [, ] ere origially proposed to cope ith commuicatio failures, esurig that i case of partitios at most oly oe partitio ill ru, thereby prevetig icosistecies. A additioal claimed advatage of quorums has bee that the cost of ruig a trasactio is reduced ith respect to ROWAA as rites are oly performed i a quorum istead of i all the sites. A quorum system is defied as a set of subsets of sites, or quorums, ith pair-ise o-empty itersectios. The oempty itersectio property is crucial i that it allos ay quorum to take decisios o behalf of the hole system, ad still guaratee overall cosistecy. I particular, read (rq) ad rite (q) quorums must be such that read ad rite operatios or to rite operatios o the same data item overlap. That is, ay read quorum must overlap ith ay rite quorum, ad rite quorums must overlap amog them. We are aare that may differet variatios of these protocols have bee proposed i the literature. Hoever, our goal ca be accomplished by examiig a small umber of caoical protocols. Thus, for simplicity ad reas of space, i hat follos e ill cosider oly these protocols ad igore ay existig variatios of them. For the purposes of our study, existig variatios have similar characteristics to the protocols e study (i.e., they oly itroduce small costat factors). I the cases ere there are sigificat chages i behavior, this is briefly discussed i the correspodig sectio.

3 Majority I the majority quorum (al ko as quorum cosesus) algorithm [] read ad rite quorums must fulfill the folloig costrais: q > ad rq + q >, beig the umber of sites. The miimum quorums satisfyig these costraits are: q = +ad rq + q = + ad therefore, q = b c +ad rq = d e. Thus, a rite quorum ca be formed by ay majority, ad read quorums by half of the system sites, if is eve, or by a majority if is odd. Tree I [] a e kid of quorum protocol is proposed to reduce the size of rite quorums hile keepig a sigleto read quorum (i a o-failure sceario) as i the ROWAA protocol. This is achieved by imposig a logical tree structure o the sites. This tree must be a complete tree of a odd degree. Write operatios are performed o a quorum formed by the root, a majority of its childre, a majority of the childre of each of these childre, ad forth. Hece, to cocurret rites have at least oe elemet i commo at each level of the tree. For istace, i a tree of degree ad three levels (Fig. ), a rite operatio could access the set of copies f,,,,,, g. Figure : Tree of degree ad height Reads are performed o the root. If the root is ot available, the read should be performed o a majority of its childre. For each uavailable site eeded for the majority, a read o a majority of its childre should be performed, ad forth. For istace, i the tree of Fig. a read could access fg. If is uavailable, it could access f, g ad if,, ad are uavailable, it could access f,, g. Sice rite operatios access a majority of sites at each level, read ad rite operatios ill overlap at least i oe site. Grid A differet kid of quorum, grid quorum, is proposed i []. This quorum assumes that the sites are orgaized i a grid of r ros ad c colums. A read quorum cosists of accessig a elemet of each colum of the grid. A rite quorum requires applyig the updates i oe colum ad lockig a elemet from each of the remaiig colums (a read quorum). Give the grid i Fig., examples of read quorums are: f,,,g, f,,,g, or f,,,g. Ex- Figure : Grid amples of rite quorums are: f,,,,,g, f,,,,,g, f,,,,,g. Read operatios overlap ith rites, sice read quorums cotai oe elemet from each colum. That is, they cotai oe elemet from the colum used for a rite operatio. To rite operatios al overlap. Sice a rite quorum locks oe elemet i each colum, o other rite ca be cocurretly performed i ay colum. Scalability. Scalability ith symmetric load I a first approximatio, e ill assume that all operatios have the same cost, regardless of hether they are local or remote operatios. We ill refer to this type of systems as symmetric. Let L be the trasactioal load arrivig at the etire system. Let L = L be the load created by updates, ith beig the proportio of update operatios i the load. Let L r =( ) Lbe the load created by read operatios. Assume that rite operatios are performed i q sites (rite quorum) ad read operatios i rq sites (read quorum). Let P = q be the probability for a site to participate i a rite quorum. Let P r = rq be the probability for a site to participate i a read quorum. With this, the load, t, at a site is give by the expressio: t = P L + P r L r () hich i terms of L ca e reritte as: t = L ( P +( ) P r ) () The scalability of a system is give by the total processig capacity of the etire system divided by the processig capacity of oe site. I our case the total load is L ad the load at each site is t. With this, e ca study the scalability i terms of the scale out factor, = L t : = L t = P +( ) P r (). Scalability ith asymmetries i the system Expressio assumes a symmetric system. Hoever, distributed databases are rarely symmetric. The most commo

4 asymmetry is due to a load optimizatio strategy used to miimize redudat ork. A local operatio is executed by parsig the correspodig SQL statemet ad executig it i its etirety. The effects of the executio might be very small compared ith the amout of data perused. For istace, it might be ecessary to sca through a log table (i.e., read all tuples) to update just oe tuple i that table. Doig this at all sites is quite iefficiet. Istead, the local site ca sed to all other sites the key to the tuple that eeds to be updated alog ith the e value. Remote sites oly eed to apply the update ithout havig to read ay data. This strategy is used, for istace, i Postgres-R []. I theory, this strategy ca be applied to both read ad rite SQL statemets. I practice, hoever, it is of limited use for read operatios. Read operatios are deoted as such because they do ot itroduce e iformatio i the database but this does ot mea that they do ot itroduce e data. A joi, for istace, results i a temporary e relatio (could al be permaet if materialized vies are used). The creatio of such temporary structures is govered by relatioal algebra, hich operates o relatios, ot o tuples (tuples are accessed by maipulatig the relatio accordig to predicates that select the desired tuples). It is easy to rerite a update SQL statemet that it updates oly the tuples correspodig to a set of keys. It is much more difficult, ad probably quite iefficiet, to do the same for read SQL statemets other tha selectio. Hece, i here e ill distiguish oly betee local ad remote rites. Read operatios ill be cosidered symmetric. With this, the probability for a site to participate i a rite operatio ca be divided ito to parts: P = P O + P R here P O is the probability of beig the origiator of a rite trasactio ad P R is the probability of participatig i a remote trasactio (i.e., a trasactio origiated by other sites). Moreover, e assume that the cost of performig a update locally is hile the cost of performig a remote update is give by a variable factor o ( <o», he o =the system is symmetric for rite operatios). With this: = L t = P O + o P R +( ) P r. Quorum probabilities The probabilities of beig i read or rite quorums for each differet protocol are summarized i the folloig table: () P r P O P R ROWAA Majority ( Tree() blog c ) Tree(d) ο d+ log d+ d d Grid p p ο : We assume that the origiator of a trasactio submits it to a quorum of sites to hich it belogs. I the case of P r, the probabilities are give by the size of the read quorum divided by the umber of sites i the system. I the case of P R, it is the size of a rite quorum mius oe divided by the umber of sites sice, sice by defiitio a site i a remote quorum caot be the origiator. For P O,itisthe probability of beig the origiator of a rite quorum, that is, the size of the quorum divided by the umber of sites of the system. For simplicity e assume that the umber of sites are as follos: a eve umber for Majority, a perfect square for Grid, ad the size of a complete d-ary tree for Tree. This simplificatio cosiders optimal quorum sizes for all protocols. Except for the tree quorum, the calculatios are straightforard. The tree quorum is a special case i that it does ot provide ay obvious mechaism to automatically distribute the load. All other protocols esure that all sites do about the same amout of ork ithout ay special arragemet. For Tree quorum, if e assume there is a sigle tree structure for the database, the the scalability is. Obviously, if the etire load goes through the root, the system ill oly scale as much as the root. If e assume all sites have the same capacity, the the scale out factor is simply for all values of,, ad o. To avoid this limitatio, e ill assume that the database ca be divided i partitios, each oe assiged to oe site. We ill further assume that each site/partitio is assiged a differet tree here that site acts as the root of the tree of the partitio. The traffic ill be divided amog the partitios that trasactios are executed oly ithi oe partitio (otherise, serializability caot be guarateed). A additioal shortcomig of the tree quorum is that the quorum size gros he failures occur. For the sake of simplicity, i our scalability aalysis e ill oly cosider the quorum size durig ormal operatio (i.e., ithout failures). We provide a geeral expressio for trees of degree d (ro Tree(d) i the table) ad relve the expressio for terary trees (ro Tree() i the table). For aalysis purposes, e ill ork ith a d-ary tree (ith d = m ;m >, hece d is odd ad larger tha ) of height h ad a total umber of sites of. Thus, blog mc h = blog d c +ad = d i (assumig the root i= is at level, its childre at level,...). Read quorums are uitary ad thus: P tree r P =

5 A rite quorum is a tree of degree d+, superimposed o the origial tree of degree d ad ith the same height. The size of tree correspodig to the rite quorum is give by: For d =: X blog c i= For d>: i = +blog c ο log log ο log ο : X blog d c i= ψ! i d + = d+ blog (d d c +) d ο (d +) log d+ d d. Protocol compari: scalability Give the quorum probabilities, expressios ad ca be calculated for each of the protocols. The results are summarized i table. Fig. shos the scale out factor for the differet protocols i a symmetric system as a fuctio of ad. Fig. shos the same type of graphs for asymmetric systems ith o =:. I a symmetric system, the protocols greatly deped o, i.e., they are very sesitive to the proportio of rite operatios i the overall load. Majority quorum has a much smaller depedecy but at the cost of very limited scalability. This is easy to explai give ho Majority quorum distribute the load: each operatio, hether read or rite, requires half the sites. Thus, the system behaves as it ould have tice the capacity of a sigle ode. Tree ad Grid quorums al deped o, but both have the advatage of beig directly proportioal to. For a square grid, the depedecy is o : hile for a -ary tree the depedecy is o :. Thus, i priciple, Tree ad Grid quorums should scale better tha ROWAA ad Majority. Fig., hoever, gives a much more precise picture. For high update rates all protocols exhibit very poor scalability. As the proportio of read operatios icreases, ROWAA ad Tree quorum are a much better choice, ith Tree quorum beig slightly better. Thus, Majority ad Grid quorums seem to be a advatage oly i applicatios ith a large proportio of rite operatios: more tha % he compared ith ROWAA ad more tha % he compared ith Tree quorum. The compari betee the protocols is, ufortuately, ot that straightforard. As poited out above, except the tree, all protocols have a built-i load distributio mechaism. If all the odes receive the same umber of trasactios the load ill al be evely distributed across the system. This is very easy to implemet (e.g., by allocatig trasactios to sites i a roud-robi fashio). I Tree quorum this is ot the case. The assumptio e have made is that access to the data partitios is evely distributed, hich is a differet thig all together. The slightest deviatio from a eve distributio i data access ill have a immediate impact o the scalability because of the bottleeck effect caused by the root of the data partitio more frequetly accessed. Sice all databases have hot-spots, Tree quorum is rather difficult to use i practice as it ould require very phisticated dyamic data partitios. Additioally, ad like ith Grid quorum, the possible cofiguratios are severely restricted. Oe caot add a arbitrary umber of odes to a system sice is dictated by the data structure used (e.g., e caot alays costruct a grid or a tree for ay value of ). The rage of viable values for is very restricted i Tree quorums (e.g., = ; ; ; ::: for terary trees). For Grid quorums, optimal grids are oly obtaied ith square grids ad, hece, must be a perfect square ( = ; ; ; :::). Ay deviatio from a square grid leads to orse scalability (plus orse availability ad larger commuicatio overhead). Fig. demostrates that scalability quickly degrades as the proportio of rite operatios i the system icreases. A ay to miimize this depedecy is to use asymmetric systems. As Fig. shos, itroducig asymmetries for rite operatios does ot chage the ature of the scalability for each protocol but it does help to miimize the degradatio caused by. Iterestigly, reducig the cost of remote rites makes Majority ad Grid less attractive as the regios here they are better tha ROWAA ad Tree quorum are eve smaller tha before ( must be higher). Thus, the results for scalability ca be summarized as follos: High scalability ca oly be achieved ith a lo rate of update operatios regardless of the protocol used. For most values of, ROWAA ad Tree offer sigificatly better scalability tha Majority ad Grid. If the cost of remote rites decreases, Majority ad Grid are oly iterestig i terms of scalability for rite oly applicatios ( ο ). Availability I this sectio e aalyze ad compare the availability of the differet protocols. We assume that failures are idepedet ad that the probability of a site beig up is p. We ill assume that p > : sice it has bee sho that for p<: the best optio is ot to use replicatio []. The overall availability of the system ill be referred to as av ad e ill distiguish betee the availability for read operatios av R ad the availability for rite operatios av W. Thus: av = av W +( ) avr ()

6 ( fl ) (o) (o) ( fl ) ROWAA ο + ( ) ο + o ( ) o Majority + ο ο + (+ (o )) + ( o) Tree() + ( blog ο : c ) + o ( blog ο : c ) o Tree(d) Grid + d+ d ( d+ (+) p blog d c ) ο (d ) log d (d+) ο p + d+ + o d+ d ( d+ ( o) ( p )+ p blog ο logd d+ d c ) ο o d+ d p ( o) Table : Scalability of differet quorums ,,,,,,,,, ,,,,,,,,, (a) ROWAA (b) Majority ,,,,,,,,, ,,,,,,,,, (c) Tree() (d) Grid Figure : Scale-out for differet values of ad ROWAA Usig the ROWAA protocol the system is available for both rite ad read as log as oe site is available. That is, it tolerates up to failures. The probability that all sites fail is: ( p). Therefore, the availability ROWAA is give by: av = av R + avw = ( p) Majority I a majority quorum protocol, the system ca progress as log as there is a majority of available sites. For simplicity i the otatio, e assume a odd umber of sites, = k +(that is, both rite ad read quorums require k + sites ad e do ot eed to distiguish betee the to types of quorums). From here, the availability is give by: av = P robability(k + copies up)+p robability(k + copies up) k+ Xψ + :::+ P robability( copies up) = i= k + i! p k+i ( p) k+ i

7 ,,,,,,,,,,,,,,,,,, (a) ROWAA (b) Majority ,,,,,,,,, ,,,,,,,,, (c) Tree() (d) Grid Figure : Scale-out for differet values of ad for o =: Tree We assume a tree of degree d, here d is odd ad d >, i.e., d+ builds a majority of childre. The read availability, av rh, of a tree of height h ad degree d is characterized by the folloig recurrece relatios []: av rh+ = P robability(root is up)+p robability(root is do) P robability(a majority of subtrees is read available) That is: av rh+ = p +( p) X d+ i= ψ d+ d + i! av r = p d+ avr +i h ( av rh ) d+ i The correspodig recurrece relatios for rite quorums are: av h+ = P robability(root is up) P robability(a majority of subtrees is rite available) That is: av h+ = p X d+ i= ψ d+ d + i! d+ av +i h ( av h ) d+ i Grid av = p The availability of a grid does ot oly deped o the umber of elemets i the grid, but al o the grid cofiguratio, that is, the umber of ros ad colums. Folloig [] the rite availability of a grid of size r c is: av W =( ( p) r ) c ( p r ( p) r ) c Similarly, the read availability of a grid of size r c is: av R =( ( p) r ) c If, as before, e assume a square grid (r = c = p ), the overall availability usig equatio is give by: p p p p p ) ) av =( ( p) ( p ( p)

8 ,E+,E-,E-,E-,E-,E-,E-,E-,E-,E-,E-,E-,E-,E- ROWAA Majority Grid Tree Figure : Uavailability for all protocols (p =:). Protocol compari: availability For compari purposes, e ill ork ith the uavailability of the system. The uavailability is calculated as av ad represeted i logarithmic scale. Thus, a uavailability of i correspods to a availability of i ies (e.g., to ies is.). Fig. compares the differet protocols i terms of availability. To make it a fair compari, e ill oly cosider optimal cofiguratios for each protocol. For Majority quorum e cosider oly cofiguratios ith a odd umber of sites; for Tree quorum e cosider complete terary trees of heights to (i.e., ith,, ad sites); for Grid quorum e cosider cofiguratios correspodig to square grids. As Fig. shos, ROWAA has, by far, the best availability. Each additioal site added to the system icreases its availability by oe ie. I compari, Majority quorum requires additioal sites to icrease the availability by oe ie. I both protocols, the availability is idepedet of ad directly proportioal to. For Grid quorum, the results are quite differet. Grids of size,, ad are perfect squares ad the availability slightly icreases as gros. The icrease for square grids is, oetheless, much smaller tha for ROWAA or Majority quorum. Tree quorum shos the orst availability of all protocols. I fact, the availability remais close to : ad does ot improve as more sites are added to the system. We are aare that there are alterative versios of the tree ad grid protocols that improve the overall availability (e.g., [] shos a variatio of grid ith a improved availability). Hoever, all quorum based protocols have a upper boud i terms of availability. This upper boud is defied by the probability of a quorum beig available. Assumig a quorum size of qs: av» ( p) qs () The effects of his upper boud have bee previously studied [] ad the results are cosistet ith our aalyses. I practice, this upper boud implies that, of all protocols based o quorums (majority, grid, ad tree), majority is the most available quorum for p>:. Tree ad grid are orse tha majority as they have smaller quorums. I geeral, the availability of Tree quorum is orse tha that of the Grid quorum, but the potetial availability of Tree quorum is better tha the oe for Grid quorum. This peculiarity has to do ith the asymmetries i the tree protocol []. These results idicate that the best optio from the availability poit of vie is ROWAA. It could be argued, hoever, tha ROWAA does ot tolerate etork partitios hile the other protocols do. Hoever, ROWAA ca be made to tolerate etork partitios fairly easily ithout affectig its behavior by usig a primary compoet approach here oly the primary compoet is alloed to progress. That is, a partitio is oly alloed to progress if it cotais at least a quorum. I fact usig a primary-compoet approach the primary compoet ca be defied by usig ay quorum system. With this i mid, the results for availability ca be summarized as follos: ROWAA ad Majority quorums are the oly protocols that provide a relevat icrease i availability as more sites are added to the system. ROWAA offers liear icreases (i a logarithmic scale) i availability as more sites are added, hile Majority quorum requires additioal sites to match the icrease i availability reached he oe site is added to ROWAA. The availability of Tree ad Grid quorums decreases as the proportio of rites i the load icreases (i.e., as icreases). The availability of ROWAA ad Majority quorum is idepedet of. Tree ad Grid quorums are highly sesitive to the cofiguratio ad provide very bad availabilities for certai values of. This produces a o-mootoic availability, ad thus, the additio of e sites ca metimes result i a reduced availability. Commuicatio overhead Replicatio requires the participatig sites to coordiate their activities by exchagig messages. I practice, this ca have a sigificat impact o the overall behavior of the protocol. O the oe side, CPU cycles are lost i dealig ith the messages (flatteig, sedig, receivig, ad uflatteig) []. O the other side, the etork badidth might be exceeded, resultig i additioal delays. The message overhead affects ot oly the scalability but al the availability []. I this sectio e study the message overhead of the differet protocols.. Message cost To get a accurate picture of the commuicatio overhead, oe eeds to take ito cosideratio the trasactioal cotext i hich data replicatio occurs. Thus, e follo a slightly differet model from the oe used i previous studies here issues like PC ad system asymmetries have ot bee take ito accout. For the purposes of our study,

9 e cosider oly the best possible implemetatio (the oe ith the least umber of messages). We ill assume that all rite operatios are executed locally o a shado copy. Thus, updates are set to the replicas oly at the ed of the trasactio i oe sigle message. If a trasactio cotais rite operatios, the participatig sites have to agree o the outcome of the trasactio usig a PC protocol. Sedig the rite operatios ca be combied ith the vote request message of the PC protocol. The participatig sites must respod ith a vote message. I the last phase the origiator seds a commit or abort message to all sites i the rite quorum. I cotrast to rite operatios, read operatios caot be delayed util the ed of the trasactios or executed o a local shado copy. Hece, for each read operatio of a trasactio, the origiator of the trasactio must sed a read request to each member of the read quorum ad each participatig site must retur a reply message cotaiig the read value ad its versio. For ROWAA ad Tree quorum, e ill assume the read is performed locally ad o messages are eeded for read operatios. I additio, to calculate the message overhead per operatio, e have to take ito cosideratio the umber of rite operatios per trasactio. Sice the message overhead caused by rite operatios is costat per trasactio, the message overhead per idividual rite operatio decreases ith icreasig umber of rite operatios per trasactio. If trasactios oly have o average oe rite operatio, the each rite operatio i the system causes three message rouds. If trasactios have o average te rite operatios, the the overhead per rite operatio is oly a teth. Note, that the umber of rite operatios per trasactio is ot determied by. A small idicates that there are geerally fe rite operatios i the system. For istace, the orkload ca have may queries (read-oly trasactios) ad fe update trasactios. Each of these fe update trasactios, hoever, ca have may rite operatios. With this, assumig that a trasactio cotais o average o rite operatios, the umber of message per operatio is msg = (q ) o +( ) (rq ) for poit-to-poit messages. If a multicast primitive is available, the umber of messages exchaged chages slightly. For all updates, oe message is eeded for the vote request, q messages are eeded to get the vote message of each participat, ad oe more message is required to commit or abort the trasactio. For each read operatio, oe eeds a message to request the read, ad rq messages to get the resposes from the participats. With this, the average umber of message per operatio becomes q + msg = +( ) rq o. Protocol compari: message overhead Table shos the average umber of messages per operatio eeded for each of the protocols, usig poit-to-poit commuicatio ad he multicast is available. As i the rest of the paper, e provide formulas for the optimal quorum sizes of each protocol. I read itesive eviromets ROWAA ad the Tree protocol behave the best, hile Majority has the orst behavior. Eve a multicast eviromet does ot help much. The grid protocol behaves slightly better tha Majority but ot by a large margi. For high update rates, ROWAA has the orst behavior i poit-to-poit etorks. Hoever, oce multicast is available, the overhead caused by ROWAA sigificatly decreases ad there are o sigificat differeces betee the differet protocols. We ca summarize as follos: The message overhead is proportioal to for all protocols, although the depedecy is much smaller for Tree ad Grid quorums tha for ROWAA ad Majority quorum. The message overhead is sigificatly reduced if multicast facilities are available. For read itesive operatios, the best optio is ROWAA idepedetly of hether multicast facilities are available or ot. The orst optio is Majority quorum. For rite itesive applicatios Tree ad Grid quorums create the least message overhead. Hoever, if multicast facilities are available, there is o sigificat differece betee the protocols. Discussio ad Coclusios With the results preseted far, e ca o give a quite comprehesive evaluatio of the protocols from a practical poit of vie. Note, hoever, that e have a very cocrete applicatio ad hardare cofiguratio. This cofiguratio, cluster computig, is becomig icreasigly pervasive ad, therefore, e believe the folloig summary is relevat for ftare ad applicatio desigers orkig o clusters. Scalability: ROWAA ad Tree quorum are the correct choices for read itesive eviromets. For very rite itesive eviromets (close to rite oly applicatios), Grid ad Majority quorum offer better scalability. Availability: ROWAA provides the best possible availability: the availability icreases liearly ith each additioal replica added to the system. Moreover, the availability does ot deped o the type of orkload (read itesive or rite itesive). Majority al provides reaable availability. Grid ad Tree quorums are poor choices from the poit of vie of availability. Message overhead: May clusters are built oadays i a star cofiguratio ith a sitch at the ceter. As a result, multicast is the ormal mode of operatio. With this i mid, ROWAA ad the Tree quorum have the best behavior i terms of message overhead. Majority quorum has the orst overhead.

10 protocol poit to poit (a) multicast (b) ROWAA o ( ) majority o +( ) ( ) grid o ( p )+ ( ) p ( ) tree() o ( +blogc ) Table : Commuicatio overhead o ( +) o + +( ) + o p +( ) p o +blogc Cofiguratio: ROWAA ad Majority do ot impose ay restrictio i the umber of odes ad require very little i terms of structurig the system. Grid ad the Tree quorum, are extremely restrictive. Grid quorum is ot as limitig but deviatios from square grids results i sigificatly orse behavior. Tree quorum ca be used i very fe cofiguratio ad requires a sigificat effort to structure the system. Realistic loads: Typical orkloads are either extremely read itesive or extremely rite itesive. I most cases, there is a tedecy to have more reads tha rites (ith a / or eve / ratio i most cases). Thus, most applicatios greatly beefit from local reads. This makes ROWAA ad Tree quorum better choices for systems that must support a ide rage of loads. Load balacig: A key aspect i the performace of a cluster is load balacig. I this ROWAA ad Majority quorum excel. Grid quorum is al adequate but it is very depedet o the cofiguratio chose (particularly if it is ot a square). Tree quorum is almost ot a optio from this poit of vie. Although it ould be theoretically possible to balace the load usig Tree quorum, i practice it almost impossible ad ould require to chage the system cofiguratio every time the load chages. The coclusio e dra from these results is that ROWAA is the best choice for a ide rage of applicatios over clusters. It offers good scalability (ithi the limitatios of replicatio protocols), very good availability, ad give the etorks used i most clusters, a reduced commuicatio overhead. It al has the sigificat advatage of beig very simple to implemet. For very peculiar loads ad cofiguratios, it is possible that me variatio of quorum does better tha ROWAA. The aalyses provided i the paper clearly idetify these situatios ad ca serve as a guide to system desigers for the fe cases i hich ROWAA is ot adequate. Refereces [] D. Agraal ad A. E. Abbadi. The Tree Quorum Protocol: A Efficiet Approach for Maagig Replicated Data. I Proc. Of the th VLDB Cof., Brisbae, Australia,. [] M. Ahamad ad M. H. Ammar. Performace Characterizatio of Quorum-Cosesus Algorithms for Replicated Data. IEEE TSE, ():, Apr.. [] G. Alo, W. Bausch, C. Pautas, M. Hallett, ad A. Kah. Depedable Computig i Virtual Laboratories: Loggig ad Recovery of Log Lived Computatios. I IEEE It. Cof. i Data Egieerig,. [] T. Ader, Y. Breitbart, H. F. Korth, ad A. Wool. Replicatio, Cosistecy, ad Practicality: Are These Mutually Exclusive? I ACM SIGMOD Coferece,. [] P. A. Berstei, V. Hadzilacos, ad N. Goodma. Cocurrecy Cotrol ad Recovery i Database Systems. Addi Wesley, Readig, MA,. [] R. Buyaa(Ed.). High Performace Cluster Computig: Architectures ad Systems. Pretice-Hall,. [] S. Y. Cheug, M. Ahamad, ad M. H. Ammar. The grid protocol: a high performace scheme for maitaiig replicated data. I Proc. of ICDE, pages,. [] D. K. Gifford. Weighted Votig for Replicated Data. th ACM Symp. o Operatig Systems, pages,. [] J. Gray, P. Hellad, P. O Neil, ad D. Shasha. The Dagers of Replicatio ad a Solutio. I Proc. of the SIGMOD, pages, Motreal,. [] R. Guerraoui, P. Felber, B. Garbiato, ad K. R. Mazoui. System support for object groups. I ACM OOPSLA, Oct.. [] B. Kemme ad G. Alo. Do t be lazy, be cosistet: Postgres-R, A e ay to implemet Database Replicatio. I Proc. of VLDB,. [] M. Naor ad A. Wool. The Load, Capacity, ad Availability of Quorum Systems. SIAM Joural of Computig, ():, Apr.. [] M. Nicola ad M. Jarke. Performace Modelig of Distributed ad Replicated Databases. IEEE Tras. o Koledge ad Data Egieerig, ():, July. [] M. Patiño Martíez, R. Jiméez Peris, B. Kemme, ad G. Alo. Scalable Replicatio i Database Clusters. I I Proc. of It. Cof. o Distributed Computig, DISC, LNCS-. Toledo, Spai, pages,. [] D. Peleg ad A. Wool. The Availability of Quorum Systems. Iformatio ad Computatio, ():,. [] D. Saha, S. Ragaraja, ad S. K. Tripathi. A aalysis of the average message overhead i replica cotrol protocols. IEEE Tras. o Paral. ad Dist. Syst., (),. [] O. Theel ad H. Pagia. Optimal Replica Cotrol Protocols Exhibit Symmetric Operatio Availabilities. I Proc. of Symp. o Fault-Tolerat Computig (FTCS),. [] R. H. Thomas. A Majority Cosesus Approach to Cocurrecy Cotrol for Multiple Copy Databases. ACM Trasactios o Database Systems, ():, Jue.