Microservice Disaster Crash Recovery: A Weak Global Referential Integrity Management

Abstract

Microservices which use polyglot persistence (using multiple data storage techniques) cannot be recovered in a consistent state from backups taken independently. As a consequence, references across microservice boundaries may break after disaster recovery. In this paper, we give a weak global consistency definition for microservice architectures and present a recovery protocol which takes advantage of cached referenced data to reduce the amnesia interval for the recovered microservice, i.e., the time interval after the most recent backup, during which state changes may have been lost.

Introduction

Microservices are small autonomous services, deployed independently, that implement a single, generally limited, business functionality [6, 14, 21, 23]. Microservices may need to store data. Different data storage pattern exist for microservices [21]. In the Database per Service pattern, defined in [19]: each microservice stores its persistent data in a private database. Each microservice has full control of a private database, persistent data being accessible to other services only via an API [24]. The invocation of a service API will result in transactions which only involve its database.

Relationships between related entities of an application based on a microservice architecture are represented by links: the state of a microservice can include links to other entities found on other microservice APIs [18]. Following the hypermedia design principle of the REST architectural style, these links can be expressed with Uniform Resource Identifiers (URIs) which globally address the referenced entities.

Since microservices are autonomous, not only do they use the most appropriate database technology for persistent storage of their state, but they also operate following an independent lifecycle, when their database is periodically backed up. For an entire microservice architecture, in practice, it is not very feasible to take an atomic snapshot of the state of all microservices. Thus, in case of one microservice crashes, which then needs to be recovered from its backup, the overall state of the microservice architecture may become inconsistent after recovery [18]. After recovery, such inconsistency may manifest itself as broken links between different microservices.

This paper presents a solution to ensure that the links between different entities managed by different microservices remain valid and intact even in the case of a database crash. The solution assumes that microservices referring to entities managed by other microservices will not only store the corresponding link, but also conserve a cached representation of the most recent known values. We present a recovery protocol when the crashed microservice can merge its own possibly stale backup with the possibly more recent cached representations obtained from other microservices. Thus, we revisit the definition of weak referential integrity across distributed microservice architectures.

Background and Related Work

Database Consistency, Durability, Backup and Disaster Crash Recovery

A database has a state, which is a value for each of its elements. The state of a database is consistent if it satisfies all the constraints [22]. Among constraints that ensure database consistency, referential integrity [8] is a database constraint that ensures that references between data are indeed valid and intact [4]. In a relational database, the referential integrity constraint states that a tuple/row in one relation referring, using a foreign key, to another relation, must refer to an existing tuple/row in that relation [11]. When a reference is defined, i.e. a value is assigned to a foreign key, the validity of the reference is checked, i.e. the referenced tuple should exist. In case of deletion, depending on the foreign key definition, the deletion of a tuple is forbidden if there are dependent foreign-key records, or the deletion of a record may cause the deletion of corresponding foreign-key records, or the corresponding foreign keys are set to null. Referential integrity is really broader and encompasses databases in general and not only relational ones [4].

Durability means that once a transaction, i.e. a set of update operations on the data, is committed, it cannot be abrogated. In the centralized databases systems, checkpoint and log are normally used to recover the state of the database in case of a system failure (e.g. the contents of main memory disappear due to a power loss and the content of a broken disk becoming illegible) [22]. Checkpoint is the point of synchronization between the database and transaction log file when all buffers are force-written to secondary storage [7]. For this kind of failure, the database can be reconstructed only if:

• the log has been stored on another disk, separately from the failure one(s),

• the log has been kept after a checkpoint, and

• the log provides information to redo changes performed by transactions before the failure, and after the latest checkpoint.

To protect the database against media failures an up-to-date backup of the database, i.e. a copy of the database separate from the database itself, is used [22]. A backup of the database and its log can be periodically copied onto offline storage media [7]. In case of database corruption or device failure, the database can be restored from one of the backup copies, typically the most recent one [3]. In this case, the recovery is carried out using the backup and the log – see [15], for more details.

A database has a Disaster Crash when the main memory and the log, or a part of the log, are lost. Therefore, to recover the database, an old, maybe obsolete, backup of the database is used. Data which was not part of the backup will be lost. In case of a disaster crash, the system cannot guarantee the durability property. However, in a centralized database, recovery from a backup provides a database which has a consistent state.

Microservices as a Federated Multidatabase

Each database of a microservice can be seen as a centralized database. Seen across an entire microservice architecture, the microservice databases represent a distributed database system. A multidatabase is a distributed database system in which each site maintains complete autonomy. Federated multidatabase is a hybrid between distributed and centralized databases. It is a distributed system for global users and a centralized one for local users [7]. According to the definitions above, stateful microservice architectures can therefore be seen as a federated multidatabase.

A microservice database can store either a snapshot of the current state of the data, containing the most recent value of data, or an event log, i.e. the current state of the data can be rebuilt by replaying the log entries, which record the changes to the microservice state in the database transaction log. Let’s consider an example of a microservice managing orders. Using the snapshot architecture, the current state of an order can be stored in a row of a relational table Order. When using the event sourcing (log) [17], the application persists each order as a sequence of events e.g., listing the creation of the order, its update with customer details and the addition of each line item.

Each microservice ensures the durability and the consistency of its database, like in centralized databases. In the microservice context, each microservice manages its own database and stores independent backup of its own database, in order to permit disaster recovery from backup. However, while managing consistent backup is simple in a centralized database, maintaining consistent backups with distributed persistence in a federated multidatabase is challenging, as shown in the survey of [13]. So a model providing global consistent backup is necessary for microservices.

Microservice architecture deals with breaking foreign key relationships [16]. Each microservice can refer to other microservices data through loosely coupled references (i.e., URLs or links), which can be dereferenced using the API provided by the microservice managing the referenced data. Microservices are independent and the managing reference integrity between them is challenging. As for the World Wide Web [9, 12], there is no guarantee that a link retrieved from a microservice points to a valid URL [18]. In the following section, we propose a model providing global reference integrity for microservices.

Microservice Disaster Recovery

In [18], the authors have addressed the problem of backing up an entire microservice architecture and recovering it in case of a disaster crash affecting one microservice. They defined the BAC theorem, inspired from the CAP Theorem [5], which states that when backing up an entire microservice architecture, it is not possible to have both availability and consistency.

Let us consider the microservice architecture defined in [18], where each microservice manages its own database and can refer to other microservices data through loosely coupled references. Each microservice does independent backup of its own database for the purpose of allowing disaster recovery from backup.

Figure 1 presents an example of two microservices with their independent backup, data of the microservice Order referring data of microservice Customer. Database of each microservice is represented in gray and data in black. Each database contains three entities. Entities C/i () correspond to customers, described by a name, and are managed by microservice Customer. Entities O/i () correspond to orders and are managed by microservice Order. Each order O/i refers to a customer C/i. Backups of the database are represented in blue. The backup of microservice Customer only contains a copy of customers C/1 and C/2. The backup and the database of microservice Order are, on the contrary, synchronized.

Fig. 1.

An example of microservice architecture with independent backup (Color figure online)

As explained in [18], in case of disaster crash, independent backup may lead to broken link (see Fig. 2): no more customer C/3 exists after Customer recovery, then O/3 has a broken link.

Fig. 2.

The link from the Order microservice to entity C/3 is broken after the recovery of Customer microservice from an old backup

A solution to avoid broken link is to synchronize the backup of all microservices, leading to limited autonomy of microservices and loss of data. In Fig. 3, both order and customer C/3 and O/3 are lost after the recovery.

Fig. 3.

Synchronized backup of an entire microservice architecture

Please note that broken link can also appear when a referenced data is deleted, e.g. when a customer is deleted in the local database of microservice Customer. In this case, the referential integrity is not respected.

As aforementioned, several approaches indicate that microservice architecture implies some challenging problems of data integrity and consistency management [2, 18], as well as the difficulty of managing consistent backups due to distributed persistence [13]. However, as far as we know, no approach proposes a solution to such problems. In the following, we present a solution that can bypass such referential integrity violation and broken links.

Our Solution: A Weak Referential Integrity Management

In this work, we focus on referential integrity. We present a solution to help the user in the recovery of the system referential integrity in case of a disaster crash. We define the global consistency as a time-dependent property. We propose a new global consistency definition, called the weak global referential consistency. Our solution provides information about the global state in case of a disaster crash that the users can pinpoint exactly the location, and time interval, of missing data which needs to be manually repaired.

In the following, we first present the context and assumptions (Subsect. 3.1), without taking disaster crash into consideration. Then, we introduce our definition of global consistency (Subsect. 3.2). Based on this definition, we show the method of recovery from a disaster crash affecting one microservice. All symbols used in this article can be found in Table 1 above.

Table 1.

Table of symbols

Symbol	Description
	Microservice
D	Database of microservice
e	Entity of a database D
	Uniform Resource Identifier of an entity e
	Date of the last update of an entity e in D
	Dependency counter associated with entity e
	Epoch identity, k being a timestamp
	ith timestamp related to epoch
B	Backup of the local database of microservice
	Amnesia interval of an entity e

Context and Assumptions

In this article, microservice follows the pattern called Database per Service (defined in [19]), where each microservice has full control of a private database, persistent data being accessible to other services only via an API. Each microservice also use an event-driven architecture, such as the one defined in [20], consuming and publishing a sequence of state-changing events.

The following are our assumptions:

• Microservices are part of the same application.

• All microservices of an application trust each other.

• Each microservice has a database D storing a set of entities.

• Each entity can be either a RESTful API resource, a relational tuple, a key-value record, a document or graph database item.

• Each entity e has a Uniform Resource Identifier, , that identifies the entity.

• The state of each entity e is read, updated and deleted using standard HTTP protocol primitives (GET, PUT and DELETE). In addition, we introduce two additional operations: getReference, deleteReference.

• Each microservice ensures the consistency and the durability of its own database D.

Taking into consideration the following ; handling the references between the different microservices and ensuring that the system is reliable when no failure occurs:

• Each microservice has its own clock. The clock of different microservices are not necessarily synchronized.

• An entity , managed by a microservice , can refer to another entity e managed by a microservice .

• The reference from microservice to an entity e, managed by a microservice , is the couple with the timestamp marking the date of the last update of entity e in D as it is known by the microservice , i.e. exactly when queries the microservice , using the clock of microservice .

There are 2 cases as far as reference storage is concerned:

1. the minimalist case consists in just storing the reference and the most recent modified timestamp, i.e. couple ;

2. the eager/self-contained backup case consists in storing a copy of the referenced entity state, that can be cached by . When microservice stores a copy of the referenced entity in its cache, this former copy is considered as detached, identical to detached entity in object-relational mapping using JPA specification [10]. Detached means that the copy is not managed by , microservice being responsible for keeping its cache up-to-date. Cached representation is only a representation of the original entity state, thus it may only contain a projection. For our solution, we assume that it is possible to reconstruct the original entity state from its cached representation.

Global Consistency

In case of no disaster crash, the global consistency can be defined as follows:

Definition 1

The global consistency

A global state is consistent if:

• (local database consistency) each local database is locally consistent in the traditional sense of a database, i.e. all its integrity constraints are satisfied.

and

• (global referential integrity) the timestamp value associated with each reference is less than or equal to the timestamp value of the corresponding referenced entity.

  Formally: for each couple associated with an entity referencing another entity e of , , with the most recent update timestamp of e in .

Case of Snapshot Data Storage Pattern. When using the snapshot data storage pattern, each local database contains the current state of its microservice. In order to guarantee the referential integrity in the microservice architecture, a microservice , cannot delete an entity e, if there is an entity managed by another microservice that refers to the entity e. We suggest a referential integrity mechanism based on dependency counters, as follows:

• Each entity e managed by a microservice is associated with a dependency counter . This counter indicates how many other entities managed by other microservices refer to entity e. It is initially set at 0.

• When a microservice wants to create the entity that refers an entity e managed by , it sends a getReference message to microservice . The corresponding dependency counter is incremented. Then, sends the couple back to , with , the date of the most recent update of entity e.

• When microservice receives the information about e, it creates the entity .

• When microservice deletes an entity that refers e, it sends a message deleteReference to microservice , indicating that the reference to e does not exist any more. is therefore decremented.

• Microservice cannot delete an entity e if its dependency counter is . It retains the most recent value of entity e with its most recent update time, , and flags the entity by indicating that e must be deleted when its dependency counter reaches the value of 0.

According to Definition 1, a reliable microservice system using the referential integrity mechanism based on dependency counters, will always be globally consistent.

Case of Event Sourcing Data Storage Pattern. When choosing event sourcing as data storage pattern [20], each local database contains an event log, which records all changes of the microservice state. Thus, it is possible to rebuild the current state of the data by replaying the event log. In this case, we propose the following referential integrity mechanism:

• When a microservice wants to create the entity that refers an entity e managed by , it sends a getReference message to microservice . When receives the information about e, it creates the entity and a creation event, associated with the corresponding reference , is stored in the log of .

• When an entity e of microservice must be deleted, instead of deleting it, the microservice flags it by , and a deletion event, associated with the related timestamp, is stored in its log, representing the most recent valid value of entity e.

Thus, it is easy to prove that global consistency state can be obtained from the logs. For each couple of a referenced entity e, timestamp must appear in the event log of microservice . Moreover the most recent record associated with entity e, corresponding to an update or deletion of e, in the event log of , has a timestamp , with , any timestamp appearing in any reference couple stored in the event log of any other microservice referencing e.

Fault Tolerant Management of Microservice Referential Integrity

As explained in Sect. 2.3, disaster crash can occur in microservice architectures. In the following, we consider disaster crash affecting only one microservice.

To protect the local database from media-failure, each microservice stores an up-to-date backup of its database, i.e. a copy of the database separate from the database itself. Each microservice individually manages the backup of its database. The way in which microservices independently manage their backup is out of the scope of this paper.

A disaster crash of a microservice means that its local database and its log are lost and we have to recover the database from a past backup. The backup provides a consistent state of the local database. However, as the database has been recovered from a past backup, data could have been lost. In order to provide a state of the local database as close as possible to the one of the database before the failure, data cached by other microservices can be used. When a microservice refers to an entity managed by another microservice , it can store a detached copy of the referenced entity. Therefore, these detached copies can be used to update the state of the database obtained after recovery from the backup.

In the following, we present the concepts used to manage disaster crash, our recovery protocol, how to optimize it and we define the Weak Global Referential Integrity.

Backup and Recovery, Amnesia Interval and New Epoch. To manage disaster crash, our assumptions are:

• Each entity of the local database of microservice is associated with an epoch identity . An epoch is a new period after a disaster crash recovery. A new epoch begins at the first access of an entity after recovery. Therefore, a timestamp associated with an entity e represents the ith timestamp related to epoch . when no crash has occurred, otherwise. The value of k always increases, being associated with time.

• When a backup B of the local database of microservice is done, operation BCK, the backup is associated with clock epoch identity , and with a creation timestamp , such that: all entities e, stored in backup B, have an updated timestamp . Epoch associated with backup and epoch associated with the local database are such that: .

• As long as there is no disaster crash, the local database and the backup are associated with the same epoch identity.

• In case of disaster crash of microservice , when the local database is locally recovered from an past obsolete backup created at time , it is known that local database has an amnesia interval starting from . This amnesia interval is associated with all entities saved in the backup and lasts until such entities are accessed again (see Definition 2).

• Each entity of the recovered database is associated with a timestamp related to epoch of the backup. This timestamp remains as long as no updates have been carried out. A new epoch begins at the first reading or written access of an entity, k containing the current date. A written operation, PUT, overwrites whatever value was recovered. However, epoch should also be updated after a reading operation, GET. Any other microservice reading from the state of the recovered entity will establish a causal dependency, which would be in conflict with further more recent recovered values from the previous epoch (see [1] for more details).

Definition 2

Amnesia Interval

An amnesia interval of microservice is a time interval indicating that a disaster crash has occurred for the local database of . This interval is associated with each entity managed by . An amnesia interval of an entity e means that:

• Epoch is the epoch associated with the backup used for the database recovery.

• Timestamp corresponds to the time of most recent known update of e. It is either the timestamp associated with the backup used for recovery, or the timestamp of a cached copy of e stored in a microservice referring entity e.

• Timestamp corresponds to the first reading or written operation on e from another microservice , after ().

Weak Global Referential Integrity. After a crash recovery, data can be lost, so we define a weak global referential integrity of the microservice architecture. Weak means that either the global referential integrity has been checked, verifying Definition 1, or an amnesia is discovered; data has been lost as well as the interval of time when the data was lost. This makes it possible to focus on the manual data recovery and reconstruction effort within the amnesia interval.

Definition 3

Weak global consistency

After a disaster crash recovery of a microservice , the system checks a weak global consistency iff:

• (local database consistency) each local database is locally consistent in the traditional sense of a database, i.e. all its integrity constraints are satisfied.

and

• (weak global referential integrity) the timestamp value associated with each reference is either less than or equal to the timestamp value of the corresponding referenced entity or included in an amnesia interval. Formally: for each couple associated with an entity of referencing another entity e of :

  • either , with the most recent update timestamp of e in , epochs and being comparable ();
  • or , with the amnesia interval associated with the referenced entity e, after a disaster crash of that manages e.

Consider a scenario of two microservices and , referencing an entity managed by (see Fig. 4), but without storing any cache of the referenced entity. In figures, only timestamps of one entity of microservice are considered, timestamps and epoch identities being only represented by numbers. In Fig. 4, at time an entity is created by microservice ; operation PUT. A backup B is made, storing entities of microservice created before ; operation BCK. Microservice refers the entity of created at time ; operation GET. An update of the entity is carried out by the microservice at time ; operation PUT. When disaster crash appears to (see red flash), must recover using the backup, update of time is lost, therefore it has amnesia that begins from . An update is done to the entity, then a new epoch 2 begins and timestamp is associated to the updated value. The amnesia interval is then updated to . If does another GET to refresh the referenced value, the up-to-date timestamp is sent by .

Fig. 4.

Example of a scenario with 2 microservices, without cached data. PUT represents a state change of the referenced entity. BCK indicates when a backup snapshot is taken. LR shows when the microservice is locally recovered from the backup. (Color figure online)

Recovery Protocol. When a disaster crash occurs to a microservice , informs all other services of its recovery. Moreover, when microservices stored copies of the entities they refer to, in their cache, the amnesia interval associated with each recovered entity of can be reduced using cached replicas. In order to do so, the steps are:

• After the recovery of , an event indicating that there is amnesia is sent, or broadcast, to other microservices.

• When a microservice receives an amnesia event from microservice , managing an entity e it refers to; if it has stored a replica of e in its cache, then sends the replica of the entity it refers to, to .

• When microservice receives replies carrying information from about its entity e, it compares the value of e, associated with timestamp , with the value, associated with timestamp , it stored, if epochs k and are comparable. Then, it retains the more up-to-date value and shrinks the amnesia interval associated with e if necessary.

• Once a read operation or an update operation is done on e, a new epoch begins, and the first timestamp associated with this new epoch represents the end of the amnesia interval.

• The beginning of the amnesia interval can still be shifted if more up-to-date values are received from belated replies from other cached replicas.

In Fig. 5, stores an up-to-date value of the referenced entity, after the backup of ; operation GET. After the recovery from the past backup of time , sends an event about its amnesia, associated with interval . After receiving this amnesia event, sends its up-to-date value, associated with , to ; event reply. stores this up-to-date value, associated with timestamp , and updates the amnesia interval to . After a update is done to the entity, a new epoch 2 begins and timestamp is associated to the updated value ; operation PUT. The amnesia interval is then updated to .

Fig. 5.

Recovery scenario using cached data more recent than the backup.

Availability vs Consistency. After a disaster crash of : either is immediately available after its local recovery, or it expects information sent by other microservices that refer its entities before the disaster crash, to provide a more recent database snapshot than the past used backup, updating the value stored in the backup with the copy stored in the cache of the other microservices.

If we are uncertain that all microservices will answer the amnesia event or if ignores or partially knows which microservices refer to (case 1), can wait for a defined timeout.

If we assume that all microservices are available and will answer to the amnesia event (case 2), waits until all microservices have sent their reply to the amnesia event.

After the timeout (case 1) or the reception of all responses (case 2), the recovery is ended and is available.

When dependency counters are used (see Sect. 3.2) and if we are sure that the identity of all microservices that refer to is known after the disaster crash: an optimization of the recovery process can be used. In this case, an amnesia event is sent only to all microservices referring to , instead of broadcast, and waits until all the aforementioned microservices reply. To do so, the address of each microservice referring to should be stored by , when the dependency counter is updated. Each referencing microservice can either send the value it stores in its cache, or a message indicating that it is no longer concerned by the amnesia, because it currently does not refer to any entity of .

The choice between the aforementioned steps depends on the focus on availability ( is available as soon as possible after its disaster crash, with a large amnesia interval) or on consistency (we prefer to wait in order to provide a more recent snapshot than the one used for the recovery before making available).

Conclusions and Future Work

In this paper, we have focused on preserving referential integrity within microservice architecture during disaster recovery. We have introduced a definition of weak global referential consistency and a recovery protocol taking advantage of replicas found in microservice caches. These are merged with local backup to reduce the amnesia interval of the recovered microservice. The approach has been validated under several assumptions: direct references to simple entities, single crashes and no concurrent recovery of more than one failed microservice.

In this paper we focused on reliability aspects, whereas as part of future work we plan to assess the performance implications of our approach in depth. We will also address more complex relationships between microservices, e.g., transitive or circular dependencies, which may span across multiple microservices. While microservice architecture is known for its ability to isolate failures, which should not cascade across multiple microservices, it remains an open question how to apply our approach to perform the concurrent recovery of multiple microservices which may have failed independently over an overlapping period of time.