Thoughts on High Availability

High Availability (HA) is a hot topic nowadays, but it is not a new one. Of course, it became more popular with the advent of the Internet, but this term is much older.

Consider, for instance, the software that took the astronauts of the Apollo 11 to the Moon: even with the limitations of that ancient hardware, HA played a key role in the design of the system.

Every company providing a service in the cloud strives to ensure the highest possible value for this measure but, what is HA and why is it so important?

A service is considered “available” when it is working properly. A service having high-availability is a service that is working properly most of the time. More formally, HA is a quality of a system that guarantees operational performance throughout time.

HA is usually implemented using a combination of Software and Hardware-based facilities like clustering, RAID, load balancers, and replicated servers. The keyword here would be “redundancy”.

No so long ago, we made the commitment of providing HA facilities to one of the key components of one of our products, Symphonica Service Activator. Interesting enough is the fact that we decided to develop our solution ourselves, even after exploring some options like Linux HA.

So, why did we decide to develop our own solution? The main reason is just simplicity. We have identified some key features that must be implemented in order for the system to be more robust and on a bigger scale.

This post is not about a particular implementation, this post is about general ideas and concepts that arise when thinking about the topic of High Availability.

Designing the high availability protocol

Defining the protocol is not only the first but also the most important concept. In order to design a good protocol it is important to understand the problem that needs to be solved:

• Am I trying to increase throughput?
• Am I trying to be fault-tolerant?

Even though this post is about High Availability – which is closely related to fault-tolerance – clustering is also used for increasing throughput. It would be a sin to avoid mentioning that. The idea is simple: If you have more computer power, then you can handle more work.

In this case, we are trying to be fault-tolerant, so the protocol should consider the following things:

• There shall be (at least) a single active server
• There shall be some backup servers ready to take over when the main server experiences failure

Note: The terms “master” and “active” are used interchangeably in this text. The same applies for “backup” and “fallback”.

How do we know when the main service is malfunctioning?

This problem is usually approached using one of the following strategies:

1. Wait for the service to tell us it is malfunctioning
2. Repeatedly poll the service for its internal state

The first approach is reactive, which means we don’t have to do anything until we hear the alarm. It also has a big disadvantage: The service might crash, and we would never know because there is no one to call us.

The second approach is more pro-active and it may require more effort from our side, but it is more accurate. Of course, the service could lie to us and report everything as ok, but that can be mitigated with more robust health-check algorithms.

The best approach is to use a combination of both. It is ok to ask the service how it is doing, but the service should also trigger a trap when something bad happens. There is also an additional approach which might be used in tandem these two and it is called “The Heartbeat”: Not as (potentially) expensive as running a health-check, but good enough to check if a service is still running.

It is worth mentioning that a more elaborate strategy might allow detecting a problem before it happens by monitoring the service’s behavior over time. Any analysis of this kind would require strong knowledge of the service’s internals or its specification and is out of the scope of this post. It is an interesting topic to discuss in the future, though.

The communication layer

The communication layer consist of two concepts or entities:

1. The messages being transferred between the nodes in the cluster.
2. The communication layer per se.

Nodes will be talking to each other, so it makes sense to define a language that everyone understands. The language defined should support at least the following expressions:

• Who is the master?
• I am the master

There might be a third expression which could help to recover from errors more quickly:

• I am not the master anymore!

We will expand on each expression in the following sections but – at this point – I think they are pretty self-explanatory.

The second concept, the actual communication layer, refers to the strategy we choose to exchange messages in the cluster. Options include P2P protocols, messages queues, a proprietary network library, etc. Regardless of the weapon of choice, there is one condition that must be met without exceptions: Broadcasting.

The communication layer must support message broadcasting. When a node sends a message, the message should be sent to all the nodes in the cluster. This requirement can be flexibilized at some extent but not always.

Of course, there is always the option of using a centralized registry, but that would mean having a single point of failure, which is not an option.

How does a node enter the cluster?

Any new node that gets added to the cluster needs to know what role it should take: Will I be the active node? Should I wait for the main service to malfunction?

It makes sense for a node to start in passive mode and test the cluster for an active node. If there is an active node, then there is nothing to do. If there is no active node, then the new one must step up and take over the cluster.

How does the node know whether there is an active node or not?

The node broadcasts the message “Who is the master?” and waits for a response of the kind “I am the master”. If there is no response, then it means there is no master.

How does a node detect the master is malfunctioning?

Even when the master can step down and notify the nodes, there may be a case where the node is unable to issue that notification. Possible causes are a sudden crash or a deadlock.

This makes it necessary to perform a proactive check on the node in order to take action when a problem is detected. Therefore, a node should continuously poll the master node for its status.

This doesn’t sound very optimal at first but it can be optimized easily using the broadcasting nature of the communication layer. At some point in time, a node would broadcast the message “Who is the master?”, which will cause the master to broadcast a reply with the expression “I am the master”. Given that this message will be received by all nodes in the cluster, all the other nodes could re-schedule the next poll to a later point in time.

How does a service take over the cluster?

A node may attempt to become master only in the following conditions:

1. It is starting and detects that there is no master.
2. It detects that the master is down – either because of a communication timeout or because the master stepped down explicitly.

Given the strategy that we’re discussing here, in order for a node to become the master, it should be enough to broadcast the message “I am the master”.

Other nodes note the message and act accordingly, as discussed in the previous section.

What about collisions?

There is a chance that multiple nodes send the message “I am the master” at the same time, or at least before receiving in from the other.

The safest thing to do in this case is to step down. All nodes should give up their briefly acquired role and become fallback with one condition: They should wait a small period of time before trying to become master again. This time cannot be the same for all nodes. Otherwise, the collision may never be solved.

Using a random but bounded period of time for each node ensures that the probability of having a collision decreases dramatically on each iteration.

How does a node step down?

A node may step down in one of the following conditions:

• The node is stopping/shutting down
• The node detected that it is malfunctioning
• The node detected a collision by received a message of type “I am the master” which didn’t come from him

Given that, it is not desirable to let the nodes waiting until a communication timeout is reached. The node should send a message of type “I am not the master anymore!” so the other nodes know that there is something happening.

How do I integrate a service with an HA facility?

This may depend on the way the HA system is implemented. Some options include:

• A shared library used directly by the service that needs HA  This approach is very useful because the only thing the service has to do is to call an API function to initialize the HA facility. 
• A host process using shared memory or another kind of IPC mechanism  This method is more robust than the shared library approach because it is not affected by program crashes or hangs. The host program takes care of monitoring the child process and anything related to checking the health of the system Any domain-related activity is done by the service itself, without affecting the host 
• External, totally decoupled services  This approach is usually more complex than the ones mentioned above. The general idea is to have a service that monitors the status of the entire system and continuously communicates with the other nodes in the cluster. Generally, shutting down a node means shutting down the whole computer. 

Wrapping up

A good HA system needs to implement the following actions:

• Define a communication layer that allows message broadcasting so all nodes can be notified of events a the same time
• Define a set of actions that each node should take in certain situations  Query for the master when starting up Query for the master at certain intervals Notify the nodes in the cluster when a node becomes active Notify the nodes in the cluster when the active node steps down due to malfunction or any other reason 
• Define a method for checking the health of the master node in a non-intrusive way