Abstract
Automatic verification of threshold-based fault-tolerant distributed algorithms (FTDA) is challenging: FTDAs have multiple parameters that are restricted by arithmetic conditions, the number of processes and faults is parameterized, and the algorithm code is parameterized due to conditions counting the number of received messages. Recently, we introduced a technique that first applies data and counter abstraction and then runs bounded model checking (BMC). Given an FTDA, our technique computes an upper bound on the diameter of the system. This makes BMC complete for reachability properties: it always finds a counterexample, if there is an actual error. To verify state-of-the-art FTDAs, further improvement is needed. In contrast to encoding bounded executions of a counter system over an abstract finite domain in SAT, in this paper, we encode bounded executions over integer counters in SMT. In addition, we introduce a new form of reduction that exploits acceleration and the structure of the FTDAs. This aggressively prunes the execution space to be explored by the solver. In this way, we verified safety of seven FTDAs that were out of reach before.
Keywords: Parameterized verification, Bounded model checking, Completeness, Partial orders in distributed systems, Reduction, Fault-tolerant distributed algorithms, Byzantine faults
Introduction
Replication is a classic approach to make computing systems more reliable. In order to avoid a single point of failure, one uses multiple processes in a distributed system. Then, if some of these processes fail (e.g., by crashing or deviating from their expected behavior) the distributed system as a whole should stay operational. For this purpose one uses fault-tolerant distributed algorithms (FTDAs). These algorithms have been extensively studied in distributed computing theory [1, 50], and found application in safety critical systems (automotive or aeronautic industry). With the recent advent of data centers and cloud computing we observe growing interest in fault-tolerant distributed algorithms, and their correctness, also for more mainstream computer science applications [19, 20, 31, 47, 52, 54, 60].
We consider automatic verification techniques specifically for threshold-based fault-tolerant distributed algorithms. In these algorithms, processes collect messages from their peers, and check whether the number of received messages reaches a threshold, e.g., a threshold may ensure that acknowledgments from a majority of processes have been received. Waiting for majorities, or more generally waiting for quorums, is a key pattern of many fault-tolerant algorithms, e.g., consensus, replicated state machine, and atomic commit. In [34] we introduced an efficient encoding of these algorithms, which we used in [33] for abstraction-based parameterized model checking of safety and liveness of several case study algorithms, which are parameterized in the number of processes n and the fraction of faults t, e.g., . In [41] we were able to verify reachability properties of more involved algorithms by applying bounded model checking. We showed how to make bounded model checking complete in the parameterized case. In particular, we considered counter systems where we record for each local state, how many processes are in this state. We have one counter per local state , denoted by . A process step from local state to local state is modeled by decrementing and incrementing . When processes perform the same step one after the other, we allow the processes to do the accelerated step that instantaneously changes the two affected counters by . The number is called acceleration factor, which can vary in a single run.
As we focus on threshold-based FTDAs, we consider counter systems defined by threshold automata. Here, transitions are guarded by threshold guards that compare a shared integer variable to a linear combination of parameters, e.g., or , where x is a shared variable and n and t are parameters.
Completeness of the method [41] with respect to reachability is shown by proving a bound on the diameter of the accelerated system. Inspired by Lamport’s view of distributed computation as partial order on events [43], our method uses a reduction similar to Lipton’s [48]. Instead of pruning executions that are “similar” to ones explored before as in partial order reduction [28, 53, 59], we use the partial order to show (offline) that every run has a similar run of bounded length. Interestingly, the bound is independent of the parameters. In [41], we introduced the following automated method, which combines this idea with data abstraction [33]:
Apply a parametric data abstraction to the process code to get a finite state process description, and construct the threshold automaton (TA) [33, 36].
Compute the diameter bound, based on the control flow of the TA.
Construct a system with abstract counters, i.e., a counter abstraction [33, 55].
Perform SAT-based bounded model checking [6, 16] up to the diameter bound, to check whether bad states are reached in the counter abstraction.
If a counterexample is found, check its feasibility and refine, if needed [13, 33].
Figure 1 gives on top a diagram [40] that shows the technique based on counter abstraction. While this allowed us to automatically verify several FTDAs not verified before, there remained two bottlenecks for scalability to larger and more complex protocols: First, counter abstraction can lead to spurious counterexamples. As counters range over a finite abstract domain, the semantics of abstract increment and decrements on the counters introduce non-determinism. For instance, the value of a counter can remain unchanged after applying an increment. Intuitively, processes or messages can be “added” or “lost”, which results in that, e.g., in the abstract model the number of messages sent is smaller than the number of processes that have sent a message, which obviously is spurious behavior. Second, counter abstraction works well in practice only for processes with a few dozens of local states. It has been observed in [4] that counter abstraction does not scale to hundreds of local states. We had similar experience with counter abstraction in our experiments in [41]. We conjecture that this is partly due to the many different interleavings, which result in a large search space.
Fig. 1.
Tool chain with counter abstraction [27, 33, 41] on top, and with SMT-based bounded model checking on bottom
To address these bottlenecks, we make two crucial contributions in this paper:
To eliminate one of the two sources of spurious counterexamples, namely, the non-determinism added by abstract counters, we do bounded model checking using SMT solvers with linear integer arithmetic on the accelerated system, instead of SAT-based bounded model checking on the counter abstraction.
We reduce the search space dramatically: we introduce the notion of an execution schema that is defined as a sequence of local rules of the TA. By assigning to each rule of a schema an acceleration factor (possibly 0, which models that no process executes the rule), one obtains a run of the counter system. Hence, due to parameterization, each schema represents infinitely many runs. We show how to construct a set of schemas whose set of reachable states coincides with the set of reachable states of the accelerated counter system.
The resulting method is depicted at the bottom of Fig. 1. Our construction can be seen as an aggressive form of reduction, where each run has a similar run generated by a schema from the set. To show this, we capture the guards that are locked and unlocked in a context. Our key insight is that a bounded number of transitions changes the context in each run. For example, of all transitions increasing a variable x, at most one makes true, and at most one makes false (the parameters n and t are fixed in a run, and shared variables can only be increased). We fix those transitions that change the context, and apply the ideas of reduction to the subexecutions between these transitions.
Our experiments show that SMT solvers and schemas outperform SAT solvers and counter abstraction in parameterized verification of threshold-based FTDAs. We verified safety of FTDAs [10, 18, 29, 51, 56, 57] that have not been automatically verified before. In addition we achieved dramatic speedup and reduced memory footprint for FTDAs [9, 12, 58] which previously were verified in [41].
In this article we focus on parameterized reachability properties. Recently, we extended this approach to safety and liveness, for which we used the reachability technique of this article as a black box [37].
Our approach at a glance
For modeling threshold-based FTDAs, we use threshold automata that were introduced in [38, 41] and are discussed in more detail in [40]. We use Fig. 2 to describe our contributions in this section. The figure presents a threshold automaton TA over two shared variables x and y and parameters n, t, and f, which is inspired by the distributed asynchronous broadcast protocol from [9]. There, correct processes concurrently follow the control flow of TA, and f processes are Byzantine faulty. As is typical for FTDAs, the parameters must satisfy a resilience condition, e.g., , that is, less than a third of the processes are faulty. The circles depict the local states , two of them are the initial states and . The edges depict the rules labeled with guarded commands , where is one of the threshold guards “”, “”, and “”, and an action increases the shared variables (x and y) by one, or zero (as in rule ).
Fig. 2.
An example threshold automaton with threshold guards “”, “”, and “”
We associate with every local state a non-negative counter that represents the number of processes in . Together with the values of x, y, n, t, and f, the values of the counters constitute a configuration of the system. In the initial configuration there are processes in initial states, i.e., , and the other counters and the shared variables x and y are zero.
The rules define the transitions of the counter system. For example, according to the rule , if in the current configuration the guard holds true and , then five processes can instantaneously move out of the local state to the local state , and increment x as prescribed by the action of (since the evaluation of the guard cannot change from true to false). This results in increasing x and by five, and decreasing the counter by five. When, as in our example, rule is conceptually executed by 5 processes, we denote this transition by , where 5 is the acceleration factor. A sequence of transitions forms a schedule, e.g., .
In this paper, we address a parameterized reachability problem, e.g., can at least one correct process reach the local state , when correct processes start in the local state ? Or, in terms of counter systems, is a configuration with reachable from an initial configuration with ? As discussed in [41], acceleration does not affect reachability, and precise treatment of the resilience condition and threshold guards is crucial for solving this problem.
Schemas
When applied to a configuration, a schedule generates a path, that is, an alternating sequence of configurations and transitions. As initially x and y are zero, threshold guards , , and evaluate to false. As rules may increase variables, these guards may eventually become true. In our example we do not consider guards like that are initially true and become false, although we formally treat them in our technique. In fact, initially only is unlocked. Because increases x, it may unlock . Thus becomes unlocked. Rule increases y and thus repeated application of (by different processes) first unlocks and then . We introduce a notion of a context that is the set of threshold guards that evaluate to true in a configuration. For our example we observe that each path goes through the following sequence of contexts , , , and . In fact, the sequence of contexts in a path is always monotonic, as the shared variables can only be increased.
The conjunction of the guards in the context implies the guards of the rules ; we call these rules unlocked in the context. This motivates our definition of a schema: a sequence of contexts and rules. We give an example of a schema below, where inside the curly brackets we give the contexts, and fixed sequences of rules in between. (We discuss the underlined rules below.)
| 2.1 |
Given a schema, we can generate a schedule by attaching to each rule an acceleration factor, which can possibly be 0. For instance, if we attach non-zero factors to the underlined rules in S, and a zero factor to the other rules, we generate the following schedule (we omit the transitions with 0 factors here).
| 2.2 |
It can easily be checked that is generated by schema S, because the sequence of the underlined rules in S matches the sequence of rules appearing in .
In this paper, we show that the schedules generated by a few schemas —one per each monotonic sequence of contexts —span the set of all reachable configurations. To this end, we apply reduction and acceleration to relate arbitrary schedules to their representatives, which are generated by schemas.
Reduction and acceleration
In this section we show what we mean by a schedule being “related” to its representative. Consider, e.g., the following schedule from the initial state with , , , and :
Observe that after , variable , and thus is true. Hence transition changes the context from to . Similarly and change the context. Context changing transitions are marked with curly brackets. Between them we have the subschedules ( is empty) marked with square brackets.
To show that this schedule is captured by the schema (2.1), we apply partial order arguments —that is, a mover analysis [48] —regarding distributed computations: As the guards and evaluate to true in , and precedes in the control flow of the TA, all transitions can be moved to the left in . Similarly, can be moved to the left in . The resulting schedule is applicable and leads to the same configuration as the original one. Further, we can accelerate the adjacent transitions with the same rule, e.g., the sequence can be transformed into . Thus, we transform subschedules into , and arrive at the schedule from Eq. (2.2), which we call the representative schedule of . As the representative schedule is generated from the schema in (2.1), we say that the schema captures schedule . (It also captures .) Importantly for reachability checking, if and are applied to the same configuration, they end in the same configuration. These arguments are formalized in Sects. 5, 6 and 7.
Encoding a schema in SMT
One of the key insights in this paper is that reachability checking via schemas can be encoded efficiently as SMT queries in linear integer arithmetic. In more detail, finite paths of counter systems can be expressed with inequalities over counters such as and , shared variables such as x and y, parameters such as n, t, and f, and acceleration factors. Also, threshold guards and resilience conditions are expressions in linear integer arithmetic.
We give an example of reachability checking with SMT using the simple schema which is contained in the schema S in Eq. (2.1). To obtain a complete encoding for S, one can similarly encode the other simple schemas and combine them.
To this end, we have to express constraints on three configurations , , and . For the initial configuration , we introduce integer variables: for local state counters, and for shared variables, and n, t, and f for parameters. As is written in Eq. (2.3), the configuration should satisfy the initial constraints, and its context should be empty (that is, all guards evaluate to false):
| 2.3 |
The configuration is reached from by applying a transition with the rule and an acceleration factor , and the configuration is reached from by applying a transition with the rule and an acceleration factor . Applying transition with the rule to just means to increase both and x by and decrease by . Hence, we introduce four fresh variables per transition and add the arithmetic operations. According to the schema, configuration has the context . The following equations express these constraints1:
| 2.4 |
| 2.5 |
Finally, we express the reachability question for all paths generated by the simple schema . Whether there is a configuration with reachable from an initial configuration with and can then be encoded as:
| 2.6 |
Note that we check only against zero, as the local state is never updated by the rule . It is easy to see that conjunction of Eqs. (2.3)–(2.6) does not have a solution, and thus all paths generated by the schema do not reach a configuration with . By writing down constraints for the other three simple schemas in Eq. (2.1), we can check reachability for the paths generated by the whole schema as well. As discussed in Sect. 2.1, our results also imply reachability on all paths whose representatives are generated by the schema. More details on the SMT encoding can be found in Sect. 9.
Parameterized counter systems
We recall the framework of [41] to the extent necessary, and extend it with the notion of a context in Sect. 3.2. A threshold automaton describes a process in a concurrent system, and is a tuple
defined below.
The finite set contains the local states, and is the set of initial states. The finite set contains the shared variables that range over the natural numbers . The finite set is a set of parameter variables that range over , and the resilience condition is a formula over parameter variables in linear integer arithmetic, e.g., . The set of admissible parameters is
.
A key ingredient of threshold automata are threshold guards (or, just guards):
Definition 3.1
A threshold guard is an inequality of one of the following two forms:
- (R)
, or
- (F)
,
where is a shared variable, are integer coefficients, and are parameters. We denote the set of all guards of the form (R) by , and the set of all guards of the form (F) by .
A rule defines a conditional transition between local states that may update the shared variables. Formally, a rule is a tuple : the local states and are from . (Intuitively, they capture from which local state to which a process moves.) A rule is only executed if the conditions and evaluate to true. Condition is a conjunction of guards from , and is a conjunction of guards from (cf. Definition 3.1). We denote the set of guards used in by , and is the set of guards used in .
Rules may increase shared variables using an update vector that is added to the vector of shared variables. As , global variables can only be increased or left unchanged. As will be later formalized in Proposition 3.1, guards from can only change from false to true (rise), and guards from can change from true to false (fall). Finally, is the finite set of rules. We use the dot notation to refer to components of rules, e.g., or .
Example 3.1
In Fig. 2, the rule that describes a transition from to , can formally be written as . Its intuitive meaning is as follows. If the guard evaluates to true, a process can move from the local state to the local state , and the global variable x is incremented, while y remains unchanged. We formalize the semantics as counter systems in Sect. 3.1.
Definition 3.2
Given a threshold automaton , we define the precedence relation
: for a pair of rules , it holds that if and only if . We denote by the transitive closure of . Further, we say that , if , or .
Assumption 3.3
We limit ourselves to threshold automata relevant for FTDAs, i.e., those where for all rules that satisfy . Such automata were called canonical in [41].
Remark 3.1
We use threshold automata to model fault-tolerant distributed algorithms that count messages from distinct senders. These algorithms are based on an “idealistic” reliable communication assumption (no message loss); these assumptions are typically expected to be ensured by “lower level bookkeeping code”, e.g., communication protocols. As a result, the algorithms we consider here do not gain from sending the same message (that is, increasing a variable) inside a loop, so that we can focus on threshold automata that do not increase shared variables in loops.
Example 3.2
In the threshold automaton from Fig. 3 we have that . Thus, we have that . In our case this implies that by definition. Similarly we can conclude that .
Fig. 3.
A threshold automaton with local states and rules . The rules drawn with solid arrows constitute a single equivalence class, while all other rules are singleton equivalence classes
Looplets The relation defines equivalence classes of rules. An equivalence class corresponds to a loop or to a single rule that is not part of a loop. Hence, we use the term looplet for one such equivalence class. For a given set of rules let be the set of equivalence classes defined by . We denote by the equivalence class of rule r. For two classes and from we write
iff there are two rules and in satisfying and and and . As the relation
is a strict partial order, there are linear extensions of
. Below, we fix an arbitrary of these linear extensions to sort transitions in a schedule: We denote by
a linear extension of
.
Example 3.3
Consider Fig. 3. The threshold automaton has five looplets: , , , , and . From , it follows that
, and from , it follows that
. We can pick two linear extensions of
, denoted by
and
. We have
, and
. In this paper we always fix one linear extension.
Remark 3.2
It may seem natural to collapse such loops into singleton local states. In our case studies, e.g, [29], non-trivial loops are used to express non-deterministic choice due to failure detectors [12], as shown in Fig. 4. Importantly, some local states inside the loops appear in the specifications. Thus, one would have to use arguments from distributed computing to characterize when collapsing states is sound. In this paper, we present a technique that deals with the loops without need for additional modelling arguments.
Fig. 4.
A typical structure found in threshold automata that model fault-tolerant algorithms with a failure detector [12]. The gray circles depict those local states, where the failure detector reports a crash. The local states and differ only in the output of the failure detector. As the failure detector output changes non-deterministically, the threshold automaton contains loops of size two
Counter systems
We use a function to capture the number of processes for each combination of parameters. As we use SMT, we assume that can be expressed in linear integer arithmetic. For instance, if only correct processes are explictly modeled we typically have , and the respective SMT expression is . Given , a threshold automaton , and admissible parameter values , we define a counter system as a transition system . It consists of the set of configurations , which contain evaluations of the counters and variables, the set of initial configurations , and the transition relation :
Configurations and A configuration consists of a vector of counter values (for simplicity we use the convention that ) a vector of shared variable values , and a vector of parameter values . The set is the set of all configurations. The set of initial configurations contains the configurations that satisfy , , and . This means that in every initial configuration all global variables have zero values, and all modeled processes are located only in the initial local states.
Example 3.4
Consider the threshold automaton from Fig. 2 with the initial states and . Let us consider a system of five processes, one of them being Byzantine faulty. Thus, , , and we explicitely model correct processes. One of the initial configurations is , where , , and . In other words, there is one process in , three processes in , and global variables are initially . Note that .
Transition relation A transition is a pair of a rule of the and a non-negative integer called the acceleration factor, or just factor for short. (As already discussed in Sect. 2.1, we will use the zero factors when generating schedules from schemas.) For a transition we refer by to , and by to , etc. We say a transition t is unlocked in configuration if , for . Note that here we use a notation that a configuration satisfies a formula, which is considered true if and only if the formula becomes true when all free variables of the formulas are evaluated as in the configuration.
We say that transition t is applicable (or enabled) in configuration , if it is unlocked, and . (As all counters are non-negative, a transition with the zero is always applicable to all configurations provided that the guards are unlocked.) We say that is the result of applying the enabled transition t to , and write , if
and
- if then
- ,
- , and
- it holds that
if then
The transition relation of the counter system is defined as follows: iff there is a rule and a factor such that for . Updates do not decrease the values of shared variables, and thus the following proposition was introduced in [41]:
Proposition 3.1
[41] For all configurations , all rules r, and all transitions t applicable to , the following holds:
1. If then 3. If then
2. If then 4. If then
Schedules and paths A schedule is a (finite) sequence of transitions. For a schedule and an index , by we denote the ith transition of , and by we denote the prefix of . A schedule is applicable to configuration , if there is a sequence of configurations with for . If there is a , then a schedule is accelerated.
By we denote the concatenation of two schedules and . A sequence of alternating configurations and transitions is called a (finite) path, if transition is enabled in and , for . For a configuration and a schedule applicable to , by we denote the path with and , for .
Contexts and slices
The evaluation of the guards in the sets and in a configuration solely defines whether certain transitions are unlocked (but not necessarily enabled). From Proposition 3.1, one can see that after a transition has been applied, more guards from may get unlocked and more guards from may get locked. In other words, more guards from may evaluate to true and more guards from may evaluate to false. To capture this intuition, we define:
Definition 3.4
A context is a pair with and . We denote by .
For two contexts and , we define that if and only if . Then, a sequence of contexts is monotonically increasing, if , for . Further, a monotonically increasing sequence of contexts is maximal, if and and , for . We obtain:
Proposition 3.2
Every maximal monotonically increasing sequence of contexts is of length . There are at most such sequences.
Example 3.5
For the example in Fig. 2, we have , and . Thus, there are maximal monotonically increasing sequences of contexts. Two of them are and . All of them have length .
To every configuration , we attach the context consisting of all guards in that evaluate to true in , and all guards in that evaluate to false in :
Definition 3.5
Given a threshold automaton, we define its configuration context as a function that for each configuration gives a context with and .
The following monotonicity result is a direct consequence of Proposition 3.1.
Proposition 3.3
If a transition t is enabled in a configuration , then either , or .
Definition 3.6
A schedule is steady for a configuration , if for every prefix of , the context does not change, i.e., .
Proposition 3.4
A schedule is steady for a configuration if and only if .
In the following definition, we associate a sequence of contexts with a path:
Definition 3.7
Given a configuration and a schedule applicable to , we say that is consistent with a sequence of contexts , if there exist indices , with , such that for every k, , and every i with , it holds that .
Every path is consistent with a uniquely defined maximal monotonically increasing sequence of contexts. (Some of the indices in Definition 3.7 may be equal.) In Sect. 4, we use such sequences of contexts to construct a schema recognizing many paths that are consistent with the same sequence of contexts.
A context defines which rules of the are unlocked. A schedule that is steady for a configuration visits only one context, and thus we can statically remove ’s rules that are locked in the context:
Definition 3.8
Given a threshold automaton and a context , we define the slice of with context as a threshold automaton , where a rule belongs to if and only if and .
In other words, contains those and only those rules r with guards that evaluate to true in all configurations with . These are exactly the guards from . When , then all guards from evaluate to true, and then must also be true. As contains those guards from that evaluate to false in , then all other guards from must evaluate to true, and then must be true too. Figure 5 shows an example of a slice.
Fig. 5.
The slice of the in Fig. 2 that is constructed for the context
Model checking problem: parameterized reachability
Given a threshold automaton , a state property B is a Boolean combination of formulas that have the form , for some . The parameterized reachability problem is to decide whether there are parameter values , an initial configuration , with , and a schedule , such that is applicable to , and property B holds in the final state: .
Main result: a complete set of schemas
To address parameterized reachability, we introduce schemas, i.e., alternating sequences of contexts and rule sequences. A schema serves as a pattern for a set of paths, and is used to efficiently encode parameterized reachability in SMT. As parameters give rise to infinitely many initial states, a schema captures an infinite set of paths. We show how to construct a finite set of schemas with the following property: for each schedule and each configuration there is a representative schedule such that: (1) applying to results in , and (2) is generated by a schema from .
Definition 4.1
A schema is a sequence of alternating contexts and rule sequences. We often write for a schema. A schema with two contexts is called simple.
Given two schemas and with , we define their composition to be the schema that is obtained by concatenation of the two sequences: .
Definition 4.2
Given a configuration and a schedule applicable to , we say that is generated by a simple schema , if the following hold:
For there is a monotonically increasing sequence of indices , i.e., , and there are factors , so that schedule .
The first and the last states match the contexts: and .
In general, we say that is generated by a schema S, if for simple schemas and such that each is generated by the simple schema , for and .
Remark 4.1
Definition 4.2 allows schemas to generate paths that have transitions with zero acceleration factors. Applying a transition with a zero factor to a configuration results in the same configuration , which corresponds to a stuttering step. This does not affect reachability. In the following, we will apply Definition 4.2 to representative paths that may have transitions with zero factors.
Example 4.1
Let us go back to the example of a schema S and a schedule introduced in Eqs. (2.1) and (2.2) in Sect. 2.1. It is easy to see that schema S can be decomposed into four simple schemas , e.g., and . Consider an initial state with , , , , , and for . To ensure that is generated by schema S, one has to check Definition 4.2 for schemas and schedules , , , and , respectively. For instance, is generated by . Indeed, take the sequence of indices 1 and 2 and the sequence of acceleration factors 1 and 1. The path ends in the configuration that differs from in that , , and . The contexts and match the contexts of schema , as required by Definition 4.2.
Similarly, is generated by schema . To see that, compare the contexts and use the index sequence 1, 2, 4, and acceleration factors 1.
The language of a schema S —denoted with —is the set of all paths generated by S. For a set of configurations and a set of schemas , we define the set to contain all configurations reachable from C via the paths generated by the schemas from , i.e., . We say that a set of schemas is complete, if for every set of configurations it is the case that the set of all states reachable from C via the paths generated by the schemas from , is exactly the set of all possible states reachable from C. Formally, .
In [41], a quantity has been introduced that depends on the number of conditions in a TA. It has been shown that for every configuration and every schedule applicable to , there is a schedule of length at most that is also applicable to and results in [41, Thm. 8]. Hence, by enumerating all sequences of rules of length up to d, one can construct a complete set of schemas:
Theorem 4.1
For a threshold automaton, there is a complete schema set of cardinality .
Although the set is finite, enumerating all its elements is impractical. We show that there is a complete set of schemas whose cardinality solely depends on the number of guards that syntactically occur in the TA. These numbers and are in practice much smaller than the number of rules :
Theorem 4.2
For all threshold automata, there exists a complete schema set of cardinality at most . In this set, the length of each schema does not exceed .
In the following sections we prove the ingredients of the following argument for the theorem: construct the set Z of all maximal monotonically increasing sequences of contexts. From Proposition 3.2, we know that there are at most maximal monotonically increasing sequences of contexts. Therefore, . Then, for each sequence , we do the following:
We show that for each configuration and each schedule applicable to and consistent with the sequence z, there is a schedule that has a specific structure, and is also applicable to . We call the representative of . We introduce and formally define this specific structure of representative schedules in Sects. 5, 6 and 7. We prove existence and properties of the representative schedule in Theorem 7.1 (Sect. 7). Before that we consider special cases: when all rules of a schedule belong to the same looplet (Theorem 5.1, Sect. 5), and when a schedule is steady (Theorem 6.1, Sect. 6).
Next we construct a schema (for the sequence z) and show that it generates all paths of all schedules found in (1). The length of the schema is at most . This is shown in Theorem 7.2 (Sect. 7).
Theorem 4.2 follows from the above theorems, which we prove in the following.
Remark 4.2
Let us stress the difference between [41] and this work. From [41], it follows that in order to check correctness of a it is sufficient to check only the schedules of bounded length . The bound does not depend on the parameters, and can be computed for each . The proofs in [41] demonstrate that every schedule longer than can be transformed into an “equivalent” representative schedule, whose length is bounded by . Consequently, one can treat every schedule of length up to as its own representative schedule. Similar reasoning does not apply to the schemas constructed in this paper: (i) we construct a complete set of schemas, whose cardinality is substantially smaller than , and (ii) the schemas constructed in this paper can be twice as long as the schemas in .
As discussed in Remark 3.2, the looplets in our case studies are typically either singleton looplets or looplets of size two. In fact, most of our benchmarks have singleton looplets only, and thus their threshold automata can be reduced to directed acyclic graphs. The theoretical constructs of Sect. 5.2 are presented for the more general case of looplets of any size. For most of the benchmarks —the ones not using failure detectors —we need only the simple construction laid out in Sect. 5.1.
Case I: one context and one looplet
We show that for each schedule that uses only the rules from a fixed looplet and does not change its context, there exists a representative schedule of bounded length that reaches the same final state. The goal is to construct a single schema per looplet. The technical challenge is that this single schema must generate representative schedules for all possible schedules, where, intuitively, processes may move arbitrarily between all local states in the looplet. As a consequence, the rules that appear in the representative schedules can differ from the rules that appear in the arbitrary schedules visiting a looplet.
We fix a threshold automaton, a context , a configuration with , a looplet c, and a schedule applicable to and using only rules from c. We then construct the representative schedule and the schema .
The technical details of the construction of for the case when is given in Sect. 5.1, and for the case when in Sect. 5.2. We show in Sect. 5.3 that these constructions give us a schedule that has the desired properties: it reaches the same final state as the given schedule , and its length does not exceed .
Note that in [41], the length of the representative schedule was bounded by |c|. However, all representative schedules of a looplet in this section can be generated by a single looplet schema.
Singleton looplet
Let us consider the case of the looplet c containing only one transition, that is, . There is a trivial representative schedule of a single transition:
Lemma 5.1
Given a threshold automaton, a configuration , and a schedule , ..., applicable to , one of the two schedules is also applicable to and results in : schedule , or schedule (r, 0).
Proof
We distinguish two cases:
Case Then, , and for . Consequently, the schedule (r, 0) is applicable to , and it results in .
Case We prove by induction on the length of a prefix of , that the following constraints hold for all k:
| 5.1 |
| 5.2 |
| 5.3 |
Base case . As schedule is applicable to , its first transition is enabled in . Thus, by the definition of an enabled transition, the rule r is unlocked, i.e., for all , it holds . By the definition, once the transition is applied, it holds that and . Thus, Constraints (5.1)–(5.3) are satisfied for .
Inductive step . As schedule is applicable to , its prefix is applicable to . Hence, transition is applicable to .
By the definition of an enabled transition, for all , it holds
By applying the Eq. (5.2) for of the inductive hypothesis, we obtain that for all , it holds that . By combining this constraint with the constraint (5.3) for , we arrive at the constraint (5.3) for k.
By applying , we get that and . By applying (5.1) and (5.2) for to these equations, we arrive at the Eqs. (5.1) and (5.2) for k.
Based on (5.1) and (5.3) for all values of k, and in particular , we can now show applicability. From Eq. (5.1), we immediately obtain that . From constraint (5.3), we obtain that for all . These are the required conditions for the transition to be applicable to the configuration .
Consequently, when c has a single rule r, for configuration and a schedule , Lemma 5.1 allows us to take the singleton schedule (r, f) as and to take the singleton schema as . The factor f is either or zero.
Non-singleton looplet
Next we focus on non-singleton looplets. Thus, we assume that . Our construction is based on two directed trees, whose undirected versions are spanning trees, sharing the same root. In order to find a representative of a steady schedule which leads from to , we determine for each local state how many processes have to move in or out of the state, and then we move them along the edges of the trees. First, we give the definitions of such trees, and then we show how to use them to construct the representative schedules and the schema.
Spanning out-trees and in-trees
We construct the underlying graph of looplet c, that is, a directed graph , whose vertices consist of local states that appear as components or of the rules from c, and the edges are the rules of c. More precisely, we construct a directed graph , whose edges from are labeled by function with the rules of c as follows:
Lemma 5.2
Given a threshold automaton and a non-singleton looplet , graph is non-empty and strongly connected.
Proof
As, and thus , graph is non-empty. To prove that is strongly connected, we consider a pair of rules . By the definition of a looplet, it holds that and . Thus, there is a path in from to , and there is a path in from to . As and correspond to some edges in , there is a cycle that contains the vertices , , , and . Thus, graph is strongly connected.
As is non-empty and strongly connected, we can fix an arbitrary node —called a hub —and construct two directed trees, whose undirected versions are spanning trees of the undirected version of . These are two subgraphs of : a directed tree , whose edges are pointing away from h (out-tree); a directed tree , whose edges are pointing to h (in-tree). For every node , it holds that and .
Further, we fix a topological order on the edges of tree . More precisely, is such a partial order on that for each pair of adjacent edges , it holds that . In the same way, we fix a topological order on the edges of tree .
Example 5.1
Consider again the threshold automaton from Example 3.3 and Fig. 3. We construct trees and for looplet , shown in Fig. 6.
Fig. 6.
The underlying graph of the looplet of the threshold automaton from Example 3.3 and Fig. 3 (left), together with trees (middle) and (right)
Note that , and Fix as a hub. We can fix a linear order such that , and a linear order such that .
Note that for the chosen hub and this specific example, and are uniquely defined, while an out-tree can be different from from our Fig. 6 (the rules constitute a different tree from the same hub). Because out-tree is not a chain, several linear orders different from can be chosen, e.g., .
Representatives of non-singleton looplets
Using these trees, we show how to construct a representative of a schedule applicable to with . For a configuration and a schedule applicable to , consider the trees and . We construct two sequences: the sequence of all edges of following the order , i.e., if , then ; the sequence of all edges of following the order . Further, we define the sequence of rules with for , and the sequence of rules with for . Using configurations and , we define:
If , then processes should leave the local state towards the hub, and they do it exclusively using the edge . If , then processes should reach the state from the hub, and they do it exclusively using the edge . The negative values of and do not play any role in our construction, and thus, we use and .
The main idea of the representative construction is as follows. First, we fire the sequence of rules to collect sufficiently many processes in the hub. Then, we fire the sequence of rules to distribute the required number of processes from the hub. As a result, for each location in the graph, the processes are transferred from to the other locations, if , and additional processes arrive at , if . Using and , we define the acceleration factors for each rule as follows:
Finally, we construct the schedule as follows:
| 5.4 |
Example 5.2
Consider the TA shown in Fig. 7. Let c be the four-element looplet that contains the rules , , , and , and be the schedule , that uses the rules of the looplet c. Consider a configuration with , and . The final configuration has the following properties: and . By comparing and , we notice that one process should move from to , and one from to . We will now show how this is achieved by our construction.
Fig. 7.
Construction of the representative of a schedule using the rules in the four-element looplet, following Example 5.2
For constructing the representative schedule , we first define trees and . If we chose to be the hub, we get that , and thus the order is . Therefore, we obtain , and . By calculating for every , we see that and are positive. Consequently, two processes go to the hub: one from and one from . The coefficients give us acceleration factors for all rules.
Similarly, we obtain , and the order must be . Thus, , , and . Here only has a positive value, and hence, two processes should move from hub to the local state . To achieve this, the acceleration factor of every rule , , must be .
Therefore, by Eq. (5.4), the representative schedule is
Choosing another hub gives us another representative. For each hub, the representative is not longer than , and leads to when applied to .
In the following, we fix a threshold automaton , a context , and a non-singleton looplet c of the slice . We also fix a configuration of and a schedule that is contained in c and is applicable to . Our goal is to prove Lemma 5.8, which states that is indeed applicable to and ends in . To this end, we first prove auxiliary Lemmas 5.3–5.7.
Lemma 5.3
For every , it holds that , where .
Proof
Recall that by the definition of a configuration, every counter is non-negative. If , then , which is bound from above by . Otherwise, , and we trivially have and .
Lemma 5.4
Schedule is applicable to configuration .
Proof
We denote by the schedule , for . Then .
All rules are from , and thus are unlocked. Hence, it is sufficient to show that the values of the locations from the set are large enough to enable each transition for . To this end, we prove by induction that , for and .
Base case . For , we want to show that . As is the first element of the sequence , which respects the order , we conclude that . From Lemma 5.3, it follows that .
Inductive step k assume that for all , schedule is applicable to and show that with .
To this end, we construct the set of edges that precede the edge in the topological order , that is, . We show that the following equation holds:
| 5.5 |
Indeed, if one picks an edge , the edge adds to the counter . As the sequence is topologically sorted, it follows that . Moreover, as the tree is oriented towards the root, is the only edge leaving the local state . Thus, no edge with decrements the counter .
From Eq. (5.5) and Lemma 5.3, we conclude that is not less than , which equals to . This proves the inductive step.
Therefore, we have shown that is applicable to .
The following lemma is easy to prove by induction on the length of a schedule. The base case for a single transition follows from the definition of a counter system.
Lemma 5.5
Let and be two configurations and be a schedule applicable to such that . Then it holds that .
Further, we show that the required number of processes is reaching (or leaving) the hub, when the transitions derived from the trees and are executed:
Lemma 5.6
The following equality holds:
Proof
Recall that is a tree directed towards h, and the undirected version of is a spanning tree of graph C. Hence, for each local state , there is exactly one edge with . Thus, the following equation holds:
| 5.6 |
Similarly, is a tree directed outwards h, and the undirected version of is a spanning tree of graph C. Hence, for each local state , there is exactly one edge with . Thus, the following equation holds:
| 5.7 |
By combining (5.6) and (5.7), we obtain the following:
| 5.8 |
As the initial schedule is applicable to , and , by Lemma 5.5, . As all rules in are from and thus change only the counters of local states in , for each local state , its respective counter does not change, that is, . Hence, . From this and Eq. (5.8), the statement of the lemma follows.
Lemma 5.7
If denotes the schedule , the following equation holds:
Proof
We prove the lemma by case distinction:
Case We show that . Indeed, let P be the indices of edges coming into h, i.e., . As all edges in are oriented towards h, it holds that equals to . By unfolding the definition of , we obtain that . We observe that by Lemma 5.6, this sum equals to . This proves the first case.
Case We show that . Indeed, fix a node and construct two sets: the set of incoming edges and the singleton set of outgoing edges . By summing up the effect of all transitions in , we obtain . By unfolding the definition of , we obtain , which can be rewritten as , which, in turn, equals to . This proves the second case.
Now we are in a position to prove that schedule is applicable to configuration and results in configuration :
Lemma 5.8
The schedule has the following properties: (a) is applicable to , and (b) results in when applied to .
Proof
Denote with the prefix of the schedule . For each , denote with the prefix of that has length of . Note that .
Proving applicability of to We notice that all rules in are from and thus are unlocked, and that is applicable to by Lemma 5.4. Hence, we only have to check that the values of counters from are large enough, so that transitions can fire.
We prove that each schedule is applicable to , for . We do so by induction on the distance from the root h in the tree .
Base case root node h. Denote with the set . Let be the indices of all edges in , and be the maximum among them.
From Lemma 5.7, . Thus, every transition with , is applicable to . Also, .
Inductive step assume that for a node and an edge outgoing from node , schedule is applicable to configuration . Show that for each edge outgoing from node the following hold: (i) schedule is also applicable to ; and (ii) .
(i) As the sequence is topologically sorted, for each edge outgoing from node , it holds that .
From Lemma 5.7, we have that . Because the transition adds to , we have . Let S be the set of all immediate successors of , i.e., . From the definition of , it follows that . Thus, the transition for edge outgoing from node , can be executed.
(ii) Let be the indices of all edges outgoing from , and be the maximum among them. From (i), it follows that
which equals to .
This proves that the schedule is applicable to .
Proving that results in From the induction above, we conclude that for each , it holds that . Edges in the trees and change only local states from . We conclude that for all , it holds that . As the rules in non-singleton looplets do not change shared variables, . Therefore, .
Representatives for one context and one looplet
We now summarize results from Sects. 5.1 and 5.2, giving the representative of a schedule in the case when uses only the rules from one looplet, and does not change its context. If the given looplet consists of a single rule, the construction is given in Sect. 5.1, and otherwise in Sect. 5.2. We show that these constructions indeed give us a schedule of bounded length, that reaches the same state as .
In the following, given a threshold automaton and a looplet c, we will say that a schedule is contained in c, if for .
Theorem 5.1
Fix a threshold automaton, and a context , and a looplet c in the slice . Let be a configuration and be a steady schedule contained in c and applicable to . There exists a representative schedule with the following properties:
schedule is applicable to , and ,
the rule of each transition t in belongs to c, that is, ,
schedule is not longer than .
Proof
If , then we use a single accelerated transition or the empty schedule as representative, as described in Lemma 5.1.
If , we construct the representative as in Sect. 5.2, so that by Lemma 5.8 property (a) follows. For every edge , the rule belongs to c, and thus satisfies property (b). As and , we conclude that , and thus property c) is also satisfied. From this and Lemma 5.8, we conclude that is the required representative schedule.
Theorem 5.1 gives us a way to construct schemas that generate all representatives of the schedules contained in a looplet:
Theorem 5.2
Fix a threshold automaton , a context , and a looplet c in the slice . There exists a schema with the following properties:
Fix an arbitrary configuration and a steady schedule that is contained in c and is applicable to . Let be the representative schedule of , from Theorem 5.1. Then, is generated by . Moreover, the length of is at most .
Proof
Note that can be constructed in two different ways depending on the looplet c.
If , then by Lemma 5.1 we have that for a rule and a factor . In this case we construct to be
It is easy to see that is generated by , as well as that the length of is exactly 1, that is less than .
If , then we use the trees and to construct the schema as follows:
| 5.9 |
Since for an arbitrary configuration and a schedule , we use the same sequence of edges in Eqs. (5.4) and (5.9) to construct and , the schema generates all paths of the representative schedules, and its length is at most .
Case II: one context and multiple looplets
In this section, we show that for each steady schedule, there exists a representative steady schedule of bounded length that reaches the same final state.
Theorem 6.1
Fix a threshold automaton and a context . For every configuration with and every steady schedule applicable to , there exists a steady schedule with the following properties:
is applicable to , and ,
To construct a representative schedule, we fix a context of a TA, a configuration with , and a steady schedule applicable to . The key notion in our construction is a projection of a schedule on a set of looplets:
Definition 6.1
Let , for , be a schedule, and let C be a set of looplets. Given an increasing sequence of indices , where , i.e., , for , a schedule is a projection of on C, if each index belongs to if and only if .
In fact, each schedule has a unique projection on a set C. In the following, we write to denote the projection of on a set .
Provided that are all looplets of the slice ordered with respect to
, we construct the following sequences of projections on each looplet (note that is the empty schedule): .
Having defined , we construct the representative simply as a concatenation of the representatives of each looplet:
Example 6.1
Consider the TA shown in Fig. 8. It has three looplets, namely , , , and the rules are depicted as solid, dotted, and dashed, respectively. These looplets are ordered such that
.
Fig. 8.
Threshold automaton and configurations used in Example 6.1
Let be the configuration represented in Fig. 8 left, i.e. and . Let be the schedule . Note that is applicable to and that is the configuration from Fig. 8 right, i.e. , and . We construct the representative schedule .
Projection of on the looplets , , and , gives us the following schedules:
Recall that
In order to construct this schedule, we firstly construct the required configurations. Note that . Then , and this is the configuration from Fig. 8 lower left, i.e. , and . Configuration is represented on Fig. 8 lower right, i.e. and all other counters are zero.
Section 5 deals with the construction of representatives of schedules that contain rules from only one looplet. Recall that construction of corresponds to the one from Example 5.2. Thus, we know that
As is a singleton looplet, we use the result of Sect. 5.1. Thus,
Using the result from Sect. 5.2 we obtain that
and finaly we have the representative for that is
Lemma 6.1
(Looplet sorting) Given a threshold automaton, a context , a configuration , a steady schedule applicable to , and a sequence of all looplets in the slice with the property
for , the following holds:
Schedule is applicable to the configuration .
Schedule is applicable to the configuration .
Schedule , when applied to , results in configuration .
Proof
In the following, we show Points 1–3 one-by-one.
We need extra notation. For a local state we denote by the -dimensional vector, where the th component is 1, and all the other components are 0. Given a schedule , we introduce a vector to keep counter difference and a vector to keep difference on shared variables as follows:
Proof of (1) Assume by contradiction that schedule is not applicable to configuration . Thus, there is a schedule and a transition that constitute a prefix of , with the following property: is applicable to , whereas is not applicable to . Let and .
There are three cases of why may be not applicable to :
(i) There is not enough processes to move: . As is applicable to , there is a transition t of with and as well as . From this, by definition of
, it follows that
. This contradicts the lemma’s assumption on the order
.
(ii) The condition is not satisfied, that is, . Then, there is a guard with .
Since is applicable to , there is a prefix of , for a schedule and a transition t that unlocks in , that is, and . Thus, transition t changes the context: . This contradicts the assumption that schedule is steady.
(iii) The condition is not satisfied: . Then, there is a guard with .
Let be the longest prefix of satisfying . Note that is also a prefix of . As and no transition decrements the shared variables, we conclude that . From this and from the fact that , it follows that . Thus transition is not applicable to . This contradicts the assumption that is applicable to .
From (i), (ii), and (iii), we conclude that (1) holds.
Proof of (2) We show that is applicable to .
To this end, we fix an arbitrary prefix of , a transition t, and a suffix , that constitute , that is, . We show that if schedule is applicable to , then so is .
Let us assume that is applicable to , and let denote the resulting state . We consider two cases:
. This case holds trivially, as equals to , which is applicable to by assumption.
. In order to prove that is applicable to , we show that counters and shared variables are large enough, so that transition t is applicable to :
(i) We start by showing that . We distinguish between different cases on source and target states of transition t.
- (i.A)
We will show by contradiction that there is no rule with . Let’s assume it exists. Then, on one hand, as , by definition of
, it follows that
. On the other hand, as and are all classes of the rules used in , it holds that . By the lemma’s assumption,
, and thus,
. We arrive at a contradiction.- (i.B)
- Let’s consider the case of a rule with . Assume by contradiction that t is not applicable to , that is, . On one hand, transition t is not applicable to . Then by the definition of , it holds that . By observing that , we derive the following inequality:
On the other hand, schedule is applicable to configuration . Thus, . By observing that and , we arrive at:6.1
By subtracting (6.2) from (6.1), and by commutativity of vector addition, we arrive at . Thus, there is a transition in and a rule such that . We again arrived at the contradictory Case (i.A). Hence, transition t must be applicable to configuration .6.2 - (i.C)
Otherwise, neither nor belong to the set of local states affected by the rules from , i.e., is empty. Then, schedule does not change the counter , and . As t is applicable to , that is, , we conclude that .
(ii) We now show that . Assume by contradiction that . There are two cases to consider.
- If .
By definition, the shared variables are never decremented in a non-singleton looplet. As is a prefix of , schedule includes all transitions of . Thus, . From this and , it follows that . This contradicts applicability of to .
- If .
Then, there is a guard with . On one hand, is applicable to . On the other hand, is applicable to . We notice that . As shared variables are never decreased, it follows that . Thus, . This contradicts the assumption on that schedule is steady.
Having proved that, we conclude that transition t is applicable to configuration . Thus, by induction is applicable to . We conclude that Point 2 of the theorem holds.
Proof of (3) By the commutativity property of vector addition,
Thus, , and Point (3) follows.
We have thus shown all three points of Lemma 6.1.
Proof
(of Theorem 6.1) By iteratively applying Lemma 6.1, we prove by induction that schedule is applicable to and results in . From Theorem 5.1, we conclude that each schedule can be replaced by its representative . Thus, is applicable to and results in . By Proposition 3.4, schedule is steady, since .
Finally, we show that for a given context, there is a schema that generates all paths of such representative schedules.
Theorem 6.2
Fix a threshold automaton and a context . Let be the sorted sequence of all looplets of the slice , i.e.,
. Schema has two properties: (a) For a configuration with and a steady schedule applicable to , of the representative is generated by ; and (b) the length of is at most .
Proof
Fix a configuration with and a steady schedule applicable to . As is a sorted sequence of the looplet representatives, all paths of are generated by , which is not longer than .
Proving the main result
Using the results from Sects. 5 and 6, for each configuration and each schedule (without restrictions) we construct a representative schedule.
Theorem 7.1
Given a threshold automaton, a configuration , and a schedule applicable to , there exists a schedule with the following properties:
is applicable to , and ,
.
Proof
Given a threshold automaton, fix a configuration and a schedule applicable to . Let be the maximal monotonically increasing sequence of contexts such that is consistent with the sequence by Definition 3.7. From Proposition 3.2, the length of the sequence is . Thus, there are at most K transitions in that change their context, i.e., for , it holds for ’s respective state in . Therefore, we can divide into steady schedules separated by the transitions :
Now, the main idea is to replace the steady schedules with their representatives from Theorem 6.1. That is, using and , we construct the schedules (by convention, is the empty schedule):
Finally, the representative schedule is constructed as follows:
From Theorem 6.1, it follows that is applicable to and it results in . Moreover, the representative of a steady schedule is not longer than , which together with K transitions gives us the bound . As we have that , this gives us the required bound.
Further, given a maximal monotonically increasing sequence z of contexts, we construct a schema that generates all paths of the schedules consistent with z:
Theorem 7.2
For a threshold automaton and a monotonically increasing sequence z of contexts, there exists a schema that generates all paths of the representative schedules that are consistent with z, and the length of does not exceed .
Proof
Given a threshold automaton, let be the sequence of all rules from , and let be a monotonically increasing sequence of contexts. By the construction in Theorem 7.1, each representative schedule consists of the representatives of steady schedules terminated with transitions that change the context. Then, for each context , for , we compose and . This composition generates the representative of a steady schedule and the transition changing the context from to . Consequently, we construct the as follows:
By inductively applying Theorem 6.2, we prove that generates all paths of schedules that are consistent with the sequence z. We get the needed bound on the length of by using an argument similar to Theorem 7.1 and by noting that for every context, instead of one rule that is changing it, we add extra rules.
Complete set of schemas and optimizations
Our proofs show that the set of schemas is easily computed from the TA: the threshold guards are syntactic parts of the TA, and enable us to directly construct increasing sequences of contexts. To find a slice of the TA for a given context, we filter the rules with unlocked guards, i.e., check whether the context contains the guard. To produce the simple schema of a looplet, we compute a spanning tree over the slice. To construct simple schemas, we do a topological sort over the looplets. For example, it takes just 30 s to compute the schemas in our longest experiment that runs for 4 h. In our tool we have implemented the following optimizations that lead to simpler and fewer SMT queries.
Entailment optimization We say that a guard entails a guard , if for all combinations of parameters and shared variables , it holds that . For instance, in our example, entails . If entails , then we can omit all monotonically increasing sequences that contain a context with and . If the number of schemas before applying this optimization is m! and there are k entailments, then the number of schemas reduces from m! to . A similar optimization is introduced for the guards from .
Control flow optimization Based on the proof of Lemma 6.1, we introduce the following optimization for TAs that are directed acyclic graphs (possibly with self loops). We say that a rule may unlock a guard , if there is a and satisfying: (the rule is unlocked); (the guard is locked); (the guard is now unlocked).
In our example from Fig. 2, the rule may unlock the guard .
Let be a guard, be the rules that use , and be the rules that may unlock . If
, for and , then we exclude some sequences of contexts as follows (we call
forward-unlockable). Let be the guards of . Guard cannot be unlocked before , and thus we can omit all sequences of contexts, where appears in the contexts before . Moreover, as are the only guards of the rules unlocking , we omit the sequences with different combinations of contexts involving and the guards from . Finally, as the rules appear after the rules in the order
, the rules appear after the rules in a rule sequence of every schema. Thus, we omit the combinations of the contexts involving and .
Hence, we add all forward-unlockable guards to the initial context (we still check the guards of the rules in the SMT encoding in Sect. 9). If the number of schemas before applying this optimization is m! and there are k forward-unlocking guards, then the number of schemas reduces from m! to . A similar optimization is introduced for the guards from .
Checking a schema with SMT
We decompose a schema into a sequence of simple schemas, and encode the simple schemas. Given a simple schema , which contains m rules, we construct an SMT formula such that every model of the formula represents a path from —the language of paths generated by schema S —and for every path in there is a corresponding model of the formula. Thus, we need to model a path of configurations and m transitions (whose acceleration factors may be 0).
To represent a configuration , for , we introduce two vectors of SMT variables: Given the set of local states and the set of shared variables , a vector to represent the process counters, a vector to represent the shared variables. We call the pair the layer i, for .
Based on this we encode schemas, for which the sequence of rules is fixed. We exploit this in two ways: First, we encode for each layer i the constraints of rule . Second, as this constraint may update only two counters —the processes move from and move to according to the rule —we do not need counter variables per layer, but only encode the two counters per layer that have actually changed. As is a common technique in bounded model checking, the counters that are not changed are “reused” from previous layers in our encoding. By doing so, we encode the schema rules with integer variables, 2m equations, and inequalities in linear integer arithmetic that represent threshold guards that evaluate to true (at most the number of threshold guards times m of these inequalities).
In the following, we use the notation to denote the set . In order to reuse the variables from the previous layers, we introduce a function that for a layer and a local state , gives the largest number of the layer, where the counter is updated:
Having defined layers, we encode: the effect of rules on counters and shared variables (in formulas M and U below), the effect of rules on the configuration (T), restrictions imposed by contexts (C), and, finally, the reachability question.
To represent m transitions, for each transition , we introduce a non-negative variable for the acceleration factor, and define two formulas: formula to express the update of the counter of local state , and formula to represent the update of the shared variable :
The formula collects all constraints by the rule :
For a formula , we denote by the formula, where each variable is substituted with . Then, given a context , a formula adds the constraints of the context on the layer i:
Finally, the formula captures all the constraints of the schema , and thus, its models correspond to the paths of schedules that are generated by S.
Let I(0) be the formula over the variables of layer i that captures the initial states of the threshold automaton, and B(i) be a state property over the variables of layer i. Then, parameterized reachability for the schema S is encoded with the following formula in linear integer arithmetic:
Experiments
We have extended our tool ByMC (Byzantine Model Checker [2]) with the technique discussed in this paper. All of our benchmark algorithms were originally published in pseudo-code, and we model them in a parametric extension of Promela, which was discussed in [27, 34].
Benchmarks
We revisited several asynchronous FTDAs that were evaluated in [33, 41]. In addition to these classic FTDAs, we considered asynchronous (Byzantine) consensus algorithms, namely, BOSCO [57], C1CS [10], and CF1S [18], that are designed to work despite partial failure of the distributed system. In contrast to the conference version of this paper [39], we used a new version of the benchmarks from [37] that have been slightly updated for liveness properties. Hence, for some benchmarks, the running times of our tool may vary from [39]. The benchmarks, their source code in parametric of Promela, and the code of the threshold automata are freely available [30].
Implementation
ByMC supports several tool chains (shown in Fig. 1, p. 3), the first using counter abstraction (that is, process counters over an abstract domain), and the second using counter systems with counters over integers:
Data and counter abstractions In this chain, the message counters are first mapped to parametric intervals, e.g., counters range over the abstract domain , . By doing so, we obtain a finite (data) abstraction of each process, and thus we can represent the system as a counter system: We maintain one counter per local state of a process, as well as the counters for the sent messages. Then, in the counter abstraction step, every process counter is mapped to the set of parametric intervals . As the abstractions may produce spurious counterexamples, we run them in an abstraction-refinement loop that incrementally prunes spurious transitions and unfair executions. More details on the data and counter abstractions and refinement can be found in [33]. In our experiments, we use two kinds of model checkers as backend:
BDD The counter abstraction is checked with nuXmv [11] using Binary Decision Diagrams (BDDs). For safety properties, the tool executes the command check_invar. In our experiments, we used the timeout of 3 days, as there was at least one benchmark that needed a bit more than a day to complete.
BMC The counter abstraction is checked with nuXmv using bounded model checking [6]. To ensure completeness (at the level of counter abstraction), we explore the computations of the length up to the diameter bounds that were obtained in [41]. To efficiently eliminate shallow spurious counterexamples, we first run the bounded model checker in the incremental mode up to length of 30. This is done by issuing the nuXmv command check_ltlspec_sbmc_inc, which uses the built-in SAT solver MiniSAT. Then, we run a single-shot SAT problem by issuing the nuXmv command gen_ltlspec_sbmc and checking the generated formula with the SAT solver lingeling [5]. In our experiments, we set the timeout to 1 day.
Reachability for threshold automata In this tool chain, to obtain a threshold automaton, our tool first applies data abstraction over the domain to the Promela code, which abstracts the message counters that keep the number of messages received by every process, while the message counters for the sent messages are kept as integers. More details can be found in [40]. Having constructed a threshold automaton, we compare two verification approaches:
Bounded model checking with SMT The approach of this article. BYMC enumerates the schemas (as explained in Sect. 4), encodes them in SMT (as explained in Sect. 9) and checks every schema with the SMT solver Z3 [17].
FAST Acceleration of counter automata In this chain, our tool constructs a threshold automaton and checks the reachability properties with the existing tool FAST [3]. For comparison with our tool, we run FAST with the MONA plugin that produced the best results in our experiments.
The challenge in the verification of FTDAs is the immense non-determinism caused by interleavings, asynchronous message passing, and faults. In our modeling, all these are reflected in non-deterministic choices in the Promela code. To obtain threshold automata, as required for our technique, our tool constructs a parametric interval data abstraction [33] that adds to non-determinism.
Comparing to [39], in this paper, we have introduced an optimization to schema checking that dramatically reduced the running times for some of the benchmarks. In this optimization, we group schemas in a prefix tree, whose nodes are contexts and edges are simple schemas. In each node of the prefix tree, our tool checks, whether there are configurations that are reachable from the initial configurations by following the schemas in the prefix. If there are no such reachable configurations, we can safely prune the whole suffix and thus prove many schemas to be unsatisfiable at once.
Evaluation
Table 1 summarizes the features of threshold automata that are automatically constructed by ByMC from parametric Promela. The number of local states varies from 7 (FRB and STRB) to hundreds (C1CS and CBC). Our threshold automata are obtained by applying interval abstraction to Promela code, which keeps track of the number of messages received by each process. Thus, the number is proportional to the number of control states and , where is the domain of parametric intervals (discussed above) and k is the number of message types. Sometimes, one can manually construct a more efficient threshold automaton that models the same fault-tolerant distributed algorithm and preserves the same safety properties. For instance, Fig. 2 shows a manual abstraction of ABA that has only 5 local states, in contrast to 61 local states in the automatic abstraction (cf. Table 1). We leave open the question of whether one can automatically construct a minimal threshold automaton with respect to given specifications.
Table 1.
The benchmarks used in our experiments. Some benchmarks, e.g., ABA, require us to consider several cases on the parameters, which are mentioned in the column “Case”. The meaning of the other columns is as follows: is the number of local states in TA, is the number of rules in TA, and is the number of (R)- and (F)-guards respectively. Finally, is the number of enumerated schemas, and Bound is the theoretical upper bound on , as given in Theorem 4.2
|
Table 2 summarizes our experiments conducted with the techniques introduced in Sect. 10.2: BDD, BMC, PARA, and FAST. On large problems, our new technique works significantly better than BDD- and SAT-based model checking. BDD-based model checking works very well on top of counter abstraction. Importantly, our new technique does not use abstraction refinement. In comparison to our earlier experiments [39], we verified safety of a larger set of benchmarks with nuXmv. We believe that this is due to the improvements in nuXmv and, probably, slight modifications of the benchmarks from [37].
Table 2.
Summary of our experiments on AMD Opteron®6272, 32 cores, 192 GB. The symbols are: “
” for timeout (72 h. for BDD and 24 h. otherwise); “
” for memory overrun of 32 GB; “
” for BDD nodes overrun; “
” for timeout in the refinement loop (72 h. for BDD and 24 h. otherwise); “
” for spurious counterexamples due to counter abstraction
|
NBAC and NBACC are challenging as the model checker produces many spurious counterexamples, which are an artifact of counter abstraction losing or adding processes. When using SAT-based model checking, the individual calls to nuXmv are fast, but the abstraction-refinement loop times out, due to a large number of refinements (about 500). BDD-based model checking times out when looking for a counterexample. Our new technique, preserves the number of proceses, and thus, there are no spurious counterexamples of this kind. In comparison to the general-purpose acceleration tool FAST, our tool uses less memory and is faster on the benchmarks where FAST is successful.
As predicted by the distributed algorithms literature, our tool finds counterexamples, when we relax the resilience condition. In contrast to counter abstraction, our new technique gives us concrete values of the parameters and shows how many processes move at each step of the counterexample.
Our new method uses integer counters and thus does not introduce spurious behavior due to counter abstraction, but still has spurious behavior due to data abstraction on complex FTDAs such as BOSCO, C1CS, and NBAC. In these cases, we manually refine the interval domain by adding new symbolic interval borders, see [33]. We believe that these intervals can be obtained directly from threshold automata, and no refinement is necessary. We leave this question to future work.
Sets of schemas and time to check a single schema
On one hand, Theorem 4.2 gives us a theoretical bound on the number of schemas to be explored. On the other hand, optimizations discussed in Sect. 8 introduce many ways of reducing the number of schemas. Two columns in Table 1 compare the theoretical bound and the practical number of schemas: the column “Theoretical bound” shows the bound of , while the column shows the actual number of schemas. (For reachability, we are merging the schemas with the prefix tree, and thus the actual number of explored schemas is even smaller.) As one can see, the theoretical bound is quite pessimistic, and is only useful to show completeness of the set of schemas. The much smaller numbers for the fault-tolerant distributed algorithms are due to a natural order on guards, e.g., as becomes true earlier than under the resilience condition . The drastic reduction in the case of CBC is due to the control flow optimization discussed in Sect. 8 and the fact that basically all guards are forward-unlocking.
When doing experiments, we noticed that the only kinds of guards that cannot be treated by our optimizations and blow up the number of schemas are the guards that use independent shared variables. For instance, consider the guards and that are counting the number of 0’s and 1’s sent by the correct processes. Even though they are mutually exlusive under the resilience condition , our tool has to explore all possible orderings of these guards. We are not aware of a reduction that would prevent our method from exploding in the number of schemas for this example.
Since the schemas can be checked independently, one can check them in parallel. Figure 9 shows a distribution of schemas along with the time needed to check an individual schema. There are only a few divergent schemas that required more than 7 s to get checked, while the large portion of schemas require 1–3 s. Hence, a parallel implementation of the tool should verify the algorithms significantly faster. We leave such a parallel extension for future work.
Fig. 9.
The times required to check individual schemas and the distribution of schemas over these times (the value 0 refers to the running times of less than a second). The benchmarks containing the schemas that are verified in (a) sec. and (b) sec. are: (a) C1CS, CBC, CF1S, and (b) CBC and CF1S
Discussions and related work
We introduced a method to efficiently check reachability properties of FTDAs in a parameterized way. If as for BOSCO, even the simplest interesting case with leads to a system size that is out of range of explicit state model checking. Hence, FTDAs force us to develop parameterized verification methods.
The problem we consider is concerned with parameterized model checking, for which many interesting results exist [14, 15, 21–23, 35]; cf. [7] for a survey. However, the FTDAs considered by us run under the different assumptions.
From a methodological viewpoint, our approach combines techniques from several areas including compact programs [49], counter abstraction [4, 55], completeness thresholds for bounded model checking [6, 16, 42], partial order reduction [8, 28, 53, 59], and Lipton’s movers [48]. Regarding counter automata, our result entails flattability [46] of every counter system of threshold automata: a complete set of schemas immediately gives us a flat counter automaton. Hence, the acceleration-based semi-algorithms [3, 46] should in principle terminate on the systems of TAs, though it did not always happen in our experiments. Similar to our SMT queries based on schemas, the inductive data flow graphs iDFG introduced in [24] are a succinct representations of schedules (they call them traces) for systems where the number of processes (or threads) is fixed. The work presented in [25] then considers parameterized verification. Further, our execution schemas are inspired by a general notion of semi-linear path schemas SLPS [45, 46]. We construct a small complete set of schemas and thus a provably small SLPS. Besides, we distinguish counter systems and counter abstraction: the former counts processes as integers, while the latter uses counters over a finite abstract domain, e.g., [55].
Many distributed algorithms can be represented with I/O Automata [50] or TLA+ [44]. In these frameworks, correctness is typically shown with a proof assistant, while model checking is used as a debugger on small instances. Parameterized model checking is not a concern there, except one notable result [32].
The results presented in this article can be used to check reachability properties of FTDAs. We can thus establish safety of FTDAs. However, for fault-tolerant distributed algorithms liveness is as important as safety: The seminal impossibility result by Fischer, Lynch, and Paterson [26] states that a fault-tolerant consensus algorithm cannot ensure both safety and liveness in asynchronous systems. In recent work [37] we also considered liveness verification, or more precisely, verification of temporal logic specification with the and temporal operators. In [37], we use the results of this article as a black box and show that combinations of schemas can be used to generate counterexamples to liveness properties, and that we can verify both safety and liveness by complete SMT-based bounded model checking.
Acknowledgements
Open access funding provided by Austrian Science Fund (FWF). We are grateful to Azadeh Farzan for valuable discussions during her stay in Vienna and to the anonymous reviewers for their insightful comments regarding partial order reduction, and for suggestions that helped us in improving the presentation of the paper.
Footnotes
Our model requires all variables to be non-negative integers. Although these constraints (e.g., ) have to be encoded in the SMT queries, we omit these constraints here for a more concise presentation.
Supported by: the Austrian Science Fund (FWF) through the National Research Network RiSE (S11403 and S11405), project PRAVDA (P27722), and Doctoral College LogiCS (W1255-N23); and by the Vienna Science and Technology Fund (WWTF) through project APALACHE (ICT15-103). This is an extended version of the paper “SMT and POR beat Counter Abstraction: Parameterized Model Checking of Threshold-Based Distributed Algorithms” that appeared in CAV (Part I), volume 9206 of LNCS, pages 85–102, 2015.
Contributor Information
Igor Konnov, Email: konnov@forsyte.at, http://forsyte.at/konnov.
Marijana Lazić, Email: lazic@forsyte.at, http://forsyte.at/lazic.
Helmut Veith, Email: veith@forsyte.at, http://forsyte.at/veith.
Josef Widder, Phone: +43 (1) 58801-18263, Email: widder@forsyte.at, http://forsyte.at/widder.
References
- 1.Attiya H, Welch J. Distributed computing. 2. New York: Wiley; 2004. [Google Scholar]
- 2.ByMC: Byzantine model checker (2013). http://forsyte.tuwien.ac.at/software/bymc/. Accessed Dec 2016
- 3.Bardin S, Finkel A, Leroux J, Petrucci L. Fast: acceleration from theory to practice. STTT. 2008;10(5):401–424. doi: 10.1007/s10009-008-0064-3. [DOI] [Google Scholar]
- 4.Basler G, Mazzucchi M, Wahl T, Kroening D (2009) Symbolic counter abstraction for concurrent software. In: CAV. LNCS, vol 5643, pp 64–78
- 5.Biere A (2013) Lingeling, Plingeling and Treengeling entering the SAT competition 2013. In: Proceedings of SAT competition 2013; Solver and p. 51
- 6.Biere A, Cimatti A, Clarke EM, Zhu Y (1999) Symbolic model checking without BDDs. In: TACAS. LNCS, vol 1579, pp 193–207
- 7.Bloem R, Jacobs S, Khalimov A, Konnov I, Rubin S, Veith H, Widder J. Decidability of parameterized verification, synthesis lectures on distributed computing theory. San Rafael: Morgan & Claypool; 2015. [Google Scholar]
- 8.Bokor P, Kinder J, Serafini M, Suri N (2011) Efficient model checking of fault-tolerant distributed protocols. In: DSN, pp 73–84
- 9.Bracha G, Toueg S. Asynchronous consensus and broadcast protocols. J ACM. 1985;32(4):824–840. doi: 10.1145/4221.214134. [DOI] [Google Scholar]
- 10.Brasileiro FV, Greve F, Mostéfaoui A, Raynal M (2001) Consensus in one communication step. In: PaCT. LNCS, vol 2127, pp 42–50
- 11.Cavada R, Cimatti A, Dorigatti M, Griggio A, Mariotti A, Micheli A, Mover S, Roveri M, Tonetta S (2014) The nuXmv symbolic model checker. In: CAV. LNCS, vol 8559, pp 334–342
- 12.Chandra TD, Toueg S. Unreliable failure detectors for reliable distributed systems. J ACM. 1996;43(2):225–267. doi: 10.1145/226643.226647. [DOI] [Google Scholar]
- 13.Clarke E, Grumberg O, Jha S, Lu Y, Veith H. Counterexample-guided abstraction refinement for symbolic model checking. J ACM. 2003;50(5):752–794. doi: 10.1145/876638.876643. [DOI] [Google Scholar]
- 14.Clarke E, Talupur M, Touili T, Veith H (2004) Verification by network decomposition. In: CONCUR 2004, vol 3170, pp 276–291
- 15.Clarke E, Talupur M, Veith H (2008) Proving Ptolemy right: the environment abstraction framework for model checking concurrent systems. In: TACAS’08/ETAPS’08. Springer, Berlin, pp 33–47
- 16.Clarke EM, Kroening D, Ouaknine J, Strichman O (2004) Completeness and complexity of bounded model checking. In: VMCAI. LNCS, vol 2937, pp 85–96
- 17.De Moura L, Bjørner N (2008) Z3: an efficient SMT solver. In: Tools and algorithms for the construction and analysis of systems. LNCS, vol 1579, pp 337–340
- 18.Dobre D, Suri N (2006) One-step consensus with zero-degradation. In: DSN, pp 137–146
- 19.Drăgoi C, Henzinger TA, Zufferey D (2016) PSync: a partially synchronous language for fault-tolerant distributed algorithms. In: POPL, pp 400–415
- 20.Drăgoi C, Henzinger TA, Veith H, Widder J, Zufferey D (2014) A logic-based framework for verifying consensus algorithms. In: VMCAI. LNCS, vol 8318, pp 161–181
- 21.Emerson E, Namjoshi K (1995) Reasoning about rings. In: POPL, pp 85–94
- 22.Emerson EA, Kahlon V (2003) Model checking guarded protocols. In: LICS. IEEE, pp 361–370
- 23.Esparza J, Ganty P, Majumdar R (2013) Parameterized verification of asynchronous shared-memory systems. In: CAV, pp 124–140
- 24.Farzan A, Kincaid Z, Podelski A (2013) Inductive data flow graphs. In: POPL, pp 129–142
- 25.Farzan A, Kincaid Z, Podelski A (2015) Proof spaces for unbounded parallelism. In: POPL, pp 407–420
- 26.Fischer MJ, Lynch NA, Paterson MS. Impossibility of distributed consensus with one faulty process. J ACM. 1985;32(2):374–382. doi: 10.1145/3149.214121. [DOI] [Google Scholar]
- 27.Gmeiner A, Konnov I, Schmid U, Veith H, Widder J (2014) Tutorial on parameterized model checking of fault-tolerant distributed algorithms. In: SFM. LNCS, vol 8483. Springer, Berlin, pp 122–171
- 28.Godefroid P (1990) Using partial orders to improve automatic verification methods. In: CAV. LNCS, vol 531, pp 176–185
- 29.Guerraoui R. Non-blocking atomic commit in asynchronous distributed systems with failure detectors. Distrib Comput. 2002;15(1):17–25. doi: 10.1007/s446-002-8027-4. [DOI] [Google Scholar]
- 30.https://github.com/konnov/fault-tolerant-benchmarks/tree/master/fmsd17
- 31.Hawblitzel C, Howell J, Kapritsos M, Lorch JR, Parno B, Roberts ML, Setty STV, Zill B (2015) Ironfleet: proving practical distributed systems correct. In: SOSP, pp 1–17
- 32.Jensen H, Lynch N (1998) A proof of Burns n-process mutual exclusion algorithm using abstraction. In: Steffen B (ed) TACAS. LNCS, vol 1384. Springer, Berlin, pp 409–423
- 33.John A, Konnov I, Schmid U, Veith H, Widder J (2013) Parameterized model checking of fault-tolerant distributed algorithms by abstraction. In: FMCAD, pp 201–209
- 34.John A, Konnov I, Schmid U, Veith H, Widder J (2013) Towards modeling and model checking fault-tolerant distributed algorithms. In: SPIN. LNCS, vol 7976, pp 209–226
- 35.Kaiser A, Kroening D, Wahl T (2012) Efficient coverability analysis by proof minimization. In: CONCUR, pp 500–515
- 36.Kesten Y, Pnueli A. Control and data abstraction: the cornerstones of practical formal verification. STTT. 2000;2:328–342. doi: 10.1007/s100090050040. [DOI] [Google Scholar]
- 37.Konnov I, Lazić M, Veith H, Widder J (2017) A short counterexample property for safety and liveness verification of fault-tolerant distributed algorithms. In: POPL, pp 719–734
- 38.Konnov I, Veith H, Widder J (2014) On the completeness of bounded model checking for threshold-based distributed algorithms: reachability. In: CONCUR. LNCS, vol 8704, pp 125–140
- 39.Konnov I, Veith H, Widder J (2015) SMT and POR beat counter abstraction: parameterized model checking of threshold-based distributed algorithms. In: CAV (Part I). LNCS, vol 9206, pp 85–102
- 40.Konnov I, Veith H, Widder J (2016) What you always wanted to know about model checking of fault-tolerant distributed algorithms. In: PSI 2015, revised selected papers. LNCS, vol 9609. Springer, pp 6–21
- 41.Konnov I, Veith H, Widder J. On the completeness of bounded model checking for threshold-based distributed algorithms: reachability. Inf Comput. 2017;252:95–109. doi: 10.1016/j.ic.2016.03.006. [DOI] [Google Scholar]
- 42.Kroening D, Strichman O (2003) Efficient computation of recurrence diameters. In: VMCAI. LNCS, vol 2575, pp 298–309
- 43.Lamport L. Time, clocks, and the ordering of events in a distributed system. Commun ACM. 1978;21(7):558–565. doi: 10.1145/359545.359563. [DOI] [Google Scholar]
- 44.Lamport L. Specifying systems: the TLA+ language and tools for hardware and software engineers. Boston: Addison-Wesley Longman Publishing Co. Inc; 2002. [Google Scholar]
- 45.Leroux J, Sutre G (2004) On flatness for 2-dimensional vector addition systems with states. In: CONCUR 2004-concurrency theory. Springer, pp 402–416
- 46.Leroux J, Sutre G (2005) Flat counter automata almost everywhere! In: ATVA. LNCS, vol 3707, pp 489–503
- 47.Lesani M, Bell CJ, Chlipala A (2016) Chapar: certified causally consistent distributed key-value stores. In: POPL, pp 357–370
- 48.Lipton RJ. Reduction: a method of proving properties of parallel programs. Commun ACM. 1975;18(12):717–721. doi: 10.1145/361227.361234. [DOI] [Google Scholar]
- 49.Lubachevsky BD. An approach to automating the verification of compact parallel coordination programs. I. Acta Inform. 1984;21(2):125–169. doi: 10.1007/BF00289237. [DOI] [Google Scholar]
- 50.Lynch N. Distributed algorithms. Burlington: Morgan Kaufman; 1996. [Google Scholar]
- 51.Mostéfaoui A, Mourgaya E, Parvédy PR, Raynal M (2003) Evaluating the condition-based approach to solve consensus. In: DSN, pp 541–550
- 52.Padon O, McMillan KL, Panda A, Sagiv M, Shoham S (2016) Ivy: safety verification by interactive generalization. In: PLDI, pp 614–630
- 53.Peled D (1993) All from one, one for all: on model checking using representatives. In: CAV. LNCS, vol 697, pp 409–423
- 54.Peluso S, Turcu A, Palmieri R, Losa G, Ravindran B (2016) Making fast consensus generally faster. In: DSN, pp 156–167
- 55.Pnueli A, Xu J, Zuck L (2002) Liveness with (0,1,$\infty $)-counter abstraction. In: CAV. LNCS, vol 2404, pp 93–111
- 56.Raynal M (1997) A case study of agreement problems in distributed systems: non-blocking atomic commitment. In: HASE, pp 209–214
- 57.Song YJ, van Renesse R (2008) Bosco: one-step Byzantine asynchronous consensus. In: DISC. LNCS, vol 5218, pp 438–450
- 58.Srikanth T, Toueg S. Simulating authenticated broadcasts to derive simple fault-tolerant algorithms. Distrib Comput. 1987;2:80–94. doi: 10.1007/BF01667080. [DOI] [Google Scholar]
- 59.Valmari A (1991) Stubborn sets for reduced state space generation. In: Advances in Petri Nets 1990. LNCS, vol 483. Springer, pp 491–515
- 60.Wilcox JR, Woos D, Panchekha P, Tatlock Z, Wang X, Ernst MD, Anderson TE (2015) Verdi: a framework for implementing and formally verifying distributed systems. In: PLDI, pp 357–368









