Asterisk ss7 redundant

Overall


The cluster functionality has been part of chan_ss7 since version 0.8.

The overall goal of setting up a redundant SS7 cluster is to eliminate single point of failure.

This means that even in the presense of any single component failing, incoming calls will still be processed (and outgoing succeed, when we get to do that as well).

The main mechanisms to achieve this are

  • One node detects the unavailability of another node, and hardware-blocks its circuits. This causes the remote SS7 signalling point to avoid routing calls to the failed node.
  • MTP3 changeover in case of signalling link failures allows re-routing signalling through another signalling link or through another node in case the local signalling link fails.

Capacity considerations


I found this capacity calculation on http://www.ss7box.com:

  • Each narrow band link is 64kb/s, or 8000 octets/sec
  • Use 40% of bandwidth (typical SS7 engineering rule), 3200 octets/sec
  • Each POTS call uses 5 messages for a total of 160 octets (1 X 80 and 4 X 20)
  • 3200/160 = 20 calls per second, 72,000 calls/hour

That site expresses concerns that this is a very high capacity, and that a single Asterisk box will be able to handle only a fraction of this capacity, thus wasting signalling link capacity (claimed expensive in that page).

However, I looked at some statistics from our voicemail system. The average call duration varies quite a bit, it is between 11.5 seconds/call and 23 seconds/call depending on the customer.

With 20 calls/second that amounts to 230 and 460 concurrent calls. However, because of the redundancy demand, with two links on a cluster, if one link goes down the other link needs to handle the signalling load of the entire cluster. Thus a single signalling link on a cluster will support only 115 or 230 voice circuits. That amounts to one or two quad-span E1 cards, which Asterisk is able to handle (tested with 120 simultaneous on dual Xeon 3.2GHz, and that was fine with very little system load).

Thus from this calculation, we do not expect any problems with wasting signalling link capacity by running the signalling directly on the Asterisk host.

If more that 20 calls/second are needed, we should be able to handle that easily by deploying additional two-node Asterisk clusters.

Cluster interconnect


Any two nodes in the cluster need a pair of interconnects. Two independent interconnects are required to be able to distinguish interconnect failure and node failure. Only in the latter case must the surviving node hardware block the circuits belonging to the dead node. A split-brain symptom could lead to each node blocking the other, thinking it dead.

The simplest seems to be to use two NICs, one using the LAN switch and the other with a cross cable or another switch if more than two nodes are in the cluster. Another option is to use NIC + and another interface that somehow provides IP, e.g. a USB ethernet gadget.

Nodes send signalling information to each other, this provides load sharing as well as redundancy. To keep the detection time of a failing node short, nodes send frequent keepalive packets on both links in case no signnaling information is ready to be sent.

Failover scenarios


The goal of the redundant chan_ss7 cluster is to avoid single point of failure. Thus, there is no attempt to handle multiple failures without downtime! Keeping this in mind, the possible failover scenarios are fairly simple:

  1. Loss of keepalive on both interconnects. In this case, it must be assumed that the other node is completely gone (total hardware or software failure). A surviving node issues an MTP3 emergency changeover on the now dead link, and sends ISUP circuit group hardware block messages on the circuits belonging to the dead node. Until the other node comes back again, the surviving node must respond appropriately to all ISUP messages related to the dead circuits, ie. send RLC on REL messages, and send BLOCKED on IAM.
  2. MTP3 link down request. This could come from the peer end of the link set, or from an Asterisk console command issued by an operator. In this case a normal MTP3 changeover is initiated at the affected node. This includes retrieving the retransmission buffer from the MTP2 level and sending it to the other node in the cluster. The ISUP circuits remain in full operation, with ISUP traffic being routed over the link of the other node and the interconnect.
Todo:
  1. Loss of LAN access to databases, backend system etc. In this case, the affected node should send a hardware circuit group blocking message to the peer signalling point. Assuming the other node retains connectivity, it will continue to carry traffic. As opposed to the previous scenarios (which are fully detected and handled inside chan_ss7), this scenario is probably better handled outside asterisk, with eg. a cluster framework like heartbeat. This external cluster monitoring would detect the loss of network connectivity and would then issue an appropriate blocking management command to chan_ss7.



Overall


The cluster functionality has been part of chan_ss7 since version 0.8.

The overall goal of setting up a redundant SS7 cluster is to eliminate single point of failure.

This means that even in the presense of any single component failing, incoming calls will still be processed (and outgoing succeed, when we get to do that as well).

The main mechanisms to achieve this are

  • One node detects the unavailability of another node, and hardware-blocks its circuits. This causes the remote SS7 signalling point to avoid routing calls to the failed node.
  • MTP3 changeover in case of signalling link failures allows re-routing signalling through another signalling link or through another node in case the local signalling link fails.

Capacity considerations


I found this capacity calculation on http://www.ss7box.com:

  • Each narrow band link is 64kb/s, or 8000 octets/sec
  • Use 40% of bandwidth (typical SS7 engineering rule), 3200 octets/sec
  • Each POTS call uses 5 messages for a total of 160 octets (1 X 80 and 4 X 20)
  • 3200/160 = 20 calls per second, 72,000 calls/hour

That site expresses concerns that this is a very high capacity, and that a single Asterisk box will be able to handle only a fraction of this capacity, thus wasting signalling link capacity (claimed expensive in that page).

However, I looked at some statistics from our voicemail system. The average call duration varies quite a bit, it is between 11.5 seconds/call and 23 seconds/call depending on the customer.

With 20 calls/second that amounts to 230 and 460 concurrent calls. However, because of the redundancy demand, with two links on a cluster, if one link goes down the other link needs to handle the signalling load of the entire cluster. Thus a single signalling link on a cluster will support only 115 or 230 voice circuits. That amounts to one or two quad-span E1 cards, which Asterisk is able to handle (tested with 120 simultaneous on dual Xeon 3.2GHz, and that was fine with very little system load).

Thus from this calculation, we do not expect any problems with wasting signalling link capacity by running the signalling directly on the Asterisk host.

If more that 20 calls/second are needed, we should be able to handle that easily by deploying additional two-node Asterisk clusters.

Cluster interconnect


Any two nodes in the cluster need a pair of interconnects. Two independent interconnects are required to be able to distinguish interconnect failure and node failure. Only in the latter case must the surviving node hardware block the circuits belonging to the dead node. A split-brain symptom could lead to each node blocking the other, thinking it dead.

The simplest seems to be to use two NICs, one using the LAN switch and the other with a cross cable or another switch if more than two nodes are in the cluster. Another option is to use NIC + and another interface that somehow provides IP, e.g. a USB ethernet gadget.

Nodes send signalling information to each other, this provides load sharing as well as redundancy. To keep the detection time of a failing node short, nodes send frequent keepalive packets on both links in case no signnaling information is ready to be sent.

Failover scenarios


The goal of the redundant chan_ss7 cluster is to avoid single point of failure. Thus, there is no attempt to handle multiple failures without downtime! Keeping this in mind, the possible failover scenarios are fairly simple:

  1. Loss of keepalive on both interconnects. In this case, it must be assumed that the other node is completely gone (total hardware or software failure). A surviving node issues an MTP3 emergency changeover on the now dead link, and sends ISUP circuit group hardware block messages on the circuits belonging to the dead node. Until the other node comes back again, the surviving node must respond appropriately to all ISUP messages related to the dead circuits, ie. send RLC on REL messages, and send BLOCKED on IAM.
  2. MTP3 link down request. This could come from the peer end of the link set, or from an Asterisk console command issued by an operator. In this case a normal MTP3 changeover is initiated at the affected node. This includes retrieving the retransmission buffer from the MTP2 level and sending it to the other node in the cluster. The ISUP circuits remain in full operation, with ISUP traffic being routed over the link of the other node and the interconnect.
Todo:
  1. Loss of LAN access to databases, backend system etc. In this case, the affected node should send a hardware circuit group blocking message to the peer signalling point. Assuming the other node retains connectivity, it will continue to carry traffic. As opposed to the previous scenarios (which are fully detected and handled inside chan_ss7), this scenario is probably better handled outside asterisk, with eg. a cluster framework like heartbeat. This external cluster monitoring would detect the loss of network connectivity and would then issue an appropriate blocking management command to chan_ss7.



Created by: knielsen, Last modification: Thu 17 of Dec, 2009 (13:02 UTC) by cervajs
Please update this page with new information, just login and click on the "Edit" or "Discussion" tab. Get a free login here: Register Thanks! - Find us on Google+