Developing Carrier Class Solutions

What does the development of Carrier Class/Grade products involve?

“In telecommunications, a “carrier grade” or “carrier class” refers to a system, or a hardware or software component that is extremely reliable, well tested and proven in its capabilities. Carrier grade systems are tested and engineered to meet or exceed “five nines” high availability standards, and provide very fast fault recovery through redundancy (normally less than 50 milliseconds)”. [1] There is almost no information regarding carrier class products on the Internet. The concept is not widely known.
Carrier class/grade is a term that is widely used in telecommunications. Mostly, it is used for hardware to ensure that the network is always healthy, even in the event of an earthquake. But at Intraway we build software that maximizes hardware capabilities, so it is important that we know what a carrier class product is.
Most of the time, the software we build every day will be part of the core business of a telecommunications company that provides phone, data and video support for millions of people or businesses. Our products are mission critical, so they need to be carefully designed, built and tested at every step to ensure that they fulfill their purpose.
There are some important attributes that our software must meet in order to qualify as carrier grade:

• High Availability
• Performance
• Reliability
• Security
• Scalability
• Support

High Availability

High availability is a characteristic of a system, which aims to ensure an agreed level of operational performance, usually uptime, for a higher than normal period.

CSPs do not take holidays, they must attend clients’ needs 24x7x365. Our systems must always be on! This means that our products must be up to the challenge. From architecture design to development implementation, high availability must be taken into account. Zero downtime is the objective that we must achieve.
One of the goals of high availability is to eliminate single points of failure in your infrastructure. A single point of failure is a component of your technology stack that would cause a service interruption if it became unavailable. As such, any component that is a requisite for the proper functionality of your application that does not have redundancy is considered to be a single point of failure. As Intraway software is the core of our clients’ systems, we must never be the point of failure.
Let’s see how High Availability is calculated.
Uptime Calculation
Availability is usually expressed as a percentage of uptime in a given year.

How do we ensure high availability in our solutions?
The goal is to eliminate single points of failure. Every single piece of software or hardware can be a point of failure, so we must pay attention to the whole.
Let see how to address the different components of our solution:

• Environment: If all of your servers are located in the same geographical area, an environmental disaster like an earthquake or flooding could take your whole system down. Having redundant servers in different datacenters and geographical areas will increase reliability.

• Hardware: We must ask our clients for redundant solutions, where all the servers on which our solution will be installed must be replicated, backed up and able to handle power outages. Hard disks and network interfaces must be resilient to hardware failures. Please consider always asking for RAID solutions and duplicated network interfaces.

• Data: Data loss or inconsistency can be caused by several factors. Hardware failure is the number one cause and can be addressed with RAID solutions and backups according to the severity of the data you are handling. Nevertheless, an ill-judged choice of database or even a faulty implementation can cause problems too. If you are handling transactions, please be sure that you use a database that can handle ACID operations. Data inconsistency can be avoided by using a RDBMS databases that helps us to define data models with consistency.

• Software: Here is where we really need to pay attention and dedicate time in order to deliver solutions that are up to the task. In order to build software that can handle high availability we have to plan it from the proof of concept phase. The architecture phase is key here, as it is in that phase that the product architect and software architect take the time to think handle high availability in the service is required. As we are moving from monolithic architecture to services, new challenges will appear. It is very important to identify which of our services are critical paths and which are not. This means that not all our services are important to the solution.

All services that are in the critical path must be at least replicated in different hardware. Take note that scalability will also be achieved with this approach. On the top of every business service is the Api Gateway. Our Api Gateway solution handles load balancing and failover. If a service is down, it will redirect requests to the next one that is available. Also, it implements the Circuit Breaker [2] pattern that helps us (if correctly implemented) to maintain high availability even if some services are down.

What about the Api Gateway? All request must go through this component making it the biggest point of failure of our solution.

Performance

CSPs handle millions of subscribers and hundreds of millions of devices. Not to mention that every day technology is expanding further, making that number even bigger.
Intraway develops products that manage the complexity of that infrastructure, and therefore must be able to process millions of transactions as well as support these growing needs.
Let´s see how we do it at Intraway:
Design & Architecture Phase

• Design with growth in mind: Every service must be prepared to outgrow even the biggest T1 that we know. A bad design will be difficult to overcome and will cost us a lot of money.
• Choose technology wisely: Have you ever tried to write web pages with C++? Is it possible? YES. But is it advisable? NO. Programming languages and frameworks match use cases and in each case we must ask ourselves if there is a better alternative. This is a list of some of the questions that we have to ask ourselves:  Do we need a transactional database? What about a nosql? Do we need to guarantee ACID? Do we need to cache data? With model of cache are we going to choose? Are the programming language and framework ready to scale? Do they manage threading natively?
  Does the ecosystem have all the tools that we need? Where do we have to pay close attention to performance? Where are the potential bottlenecks? 
• Be able to scale (horizontally): Vertical scaling has limits (hardware limits), scalation these days is about starting more new instances of one or more services. Amazon allows us to start new servers just by interacting with the API, giving us the possibility to scale when we have peaks and decrease instances when the load falls. This is not free, so the design of the products must be carefully thought out. If we have a bottleneck in the database, and the database cannot be sharded, it doesn’t matter how many services we put online, nothing will change.

Development Phase

• Take the hardware into account when developing: Many times one of our requirements is not only a functional specification, it’s also a hardware limitation. It is very important to understand where the software is going to run before you start coding it.
• Maximize code review on the critical features: It is advisable that critical use cases go through a code review process. Poor implementation on the critical path could affect the performance of the whole application.
• Minimize IOPS and network access: IOPS or network access requires interaction with a physical element, i.e.latency (more info). CPU process is infinitely faster than physical access. Of course it is impossible to execute without the use of a disk or network, it does make sense. Nonetheless we should be very careful when using it on process that are meant to be performant.
• Avoid big transactions: Performance and transactions generally pull in opposite directions. A transactional system will always have performance penalties related to its nature. Transactions cannot be disposed of. But if we want to achieve performance in a transactional system, we must keep our transactions to a minimum.
• Third-party services: If your service relies on a third-party service, take into account that you are not in control of slowdowns, crashes, etc. It is impossible to avoid dependency with a third party, so we have to properly design the interaction. Timeouts, fallbacks and retries must be handled in order to isolate your service from third-party issues.

Testing Phase
As every piece of software that is developed must be part of a testing phase, we cannot forget about performance testing. Every product or application must know its limits and it must be continuously validated.

Performance testing must be planned and executed as part of our testing phase.

There are several types of performance testing:

• Load testing: A load test is usually conducted to understand the behaviour of the system under a specific expected load.
• Stress testing: Stress testing is normally used to understand the upper limits of capacity within the system.
• Soak testing: Soak testing, also known as endurance testing, is usually done to determine if the system can sustain the continuous expected load.
• Spike testing: Spike testing is done by suddenly increasing or decreasing the load generated by a very large number of users, and observing the behaviour of the system.

Each of them have to be designed and planned accordingly. We use Jmeter and SoapUI to build and execute these kinds of tests.
Implementation Phase

• Base software optimization: Systems must be properly tuned. While default configurations make it easy to get new components up and running, they’re not always appropriate for your services in a live production environment.
• Load balancing distribution: Poor load distribution can cause slow response times by incorrectly assigning new transactions to bogged-down servers instead of others with cycles to spare. In our architecture, the Api Gateway is the component that will handle load balancing. It is crucial that the configuration is validated and tested.
• Optimize databases: An optimized database allows for the highest levels of security and performance, while an unoptimized database can destroy a production application.

Reliability

Reliability refers to the ability of a system or component to perform its required functions under stated conditions and for a specified period of time. Furthermore, if the system has the ability to deliver the expected service despite the occurrence of faults, we can say that the system is ‘Fault Tolerant’.
Reliability is the consequence of a job well done and following standards and good practices that help us to discard proven errors or faulty implementations.

Software reliability is hard to achieve, because the complexity tends to be high. The complexity of software is inversely related to reliability.

In order to improve reliability we work according to the following concepts:

• State of the art software development process: Our development process (from design to implementation) has to be state of the art.
• Rapid Error detection and fixing: Once we are in a production environment, if an error occurs, we must be able to detect it fast, analyze the cause and correct it.

Let’s review some tips for building reliable applications:

•   Test, test, test: Testing is a must with carrier class products. In each iteration a testing subphase should be present to detect bugs that may lead to failures. Skipping the testing phase only leads to a buggy application. 
• Frameworks and third-party libraries: Did you know that at least 50% of our applications’ code is inside the framework or third-party libraries that we use? If not, you are doing it wrong. Choosing a framework is a key decision that will affect the end-of-life of the product. Check the evolution and roadmap that the framework has what kind of community it has. For our solutions we will use the Spring framework project [3].
• Always validate input: In every service, in every public operation, we must validate the input that we receive. We are moving toward API Rest services, so validation is about validating the parameters what we receive on the http operation.
• Exception handling: Exception handling is very important in every programming language, but in Java it is a way of life. An operation must finish as expected if we want to rely on it. If a fatal error is not documented, it can provoke a possible failure. Reliable code is code that always behaves as expected and is documented. If the documentation says that it won’t return exceptions, it is mandatory that it do so.
• Service Concept: Services help to achieve reliability by separating your business into autonomous, self-aware, fault-tolerant parts (called services). As each service is designed to achieve a specific goal and does not know the others, we can say they are totally independent from each other.
• Circuit Breaker pattern: Service implementation involves remote calls and high level interdependency between services. New solutions bring new problems. How do we handle a remote call that hangs without a response or is unavailable for some time? CircuitBreaker [2] design patterns come to the rescue.
• Observable malfunction: Developing a system is a very complex art, hence there will always be bugs. If all our validations fail and an error appears we must detect it fast and be able to identify what is happening. If an error is not observable, it can become a failure very fast. Reliable systems can have errors, we are humans. But they must be able to detect and recover very quickly.The way to detect and identify errors is through the use of logs.
• Alert system: Proactive Monitoring is also a must for a reliable system. Being able to understand the whole state of a system with one click is what we want in carrier gradle products. Sentinel is our solution for this issue.

Security

There is no such thing as a closed system anymore. Every system or product that we develop will at some point be exposed to the Internet. Many companies are migrating to the cloud and CSPs will eventually do so as well.
Intraway has started deploying solutions into the cloud, and as we start to expose our solution to internet, security jumps to the top of our priorities. A system that can be hacked or broken because of a lack of security isn’t reliable.

The next image shows the solution that we use to deal with security:

Every endpoint that we publish must always be routed through our Api Gateway. Using the API Gateway Pattern helps us to hide private operations and acts as a reverse proxy. The API Gateway supports https protocol and certificates.
Authentication and authorization will be handled by the SSO component. The API Gateway makes a token validation before redirecting the request to the required service.
Using the SOA approach helps us to isolate functionality, but also helps us to isolate security. Even if one service is broken, only that service is compromised and not the whole system. Service isolation is a built-in security schema inside the architecture.

Scalability

As our clients gain more customers, devices, capabilities and business functionalities, our products have to fulfill the rise in demand. All our products must be scaleable and improve performance when hardware is added. Scalability is not only about managing requests, it’s also the possibility of adding new functionalities while still being able to meet the requirements.
There is a book called The Art of Scalability [4] that does a good job of explaining the issues related to scaling a system. A summary can be found here.
The book basically talks about three axes of scaling:

• X-Axis: Scaling by cloning
• Y-Axis: Scaling by function decomposition
• Z-Axis: Scaling by splitting similar things

The Y-axis (functional decomposition) is handled by our SOA Architecture. But what about the X-axis(cloning) and Z-axis(data partitioning)?
Horizontal Scale
Depending on our clients’ needs, the first thing to do is to detect which services need to be scaled.If the service that requires scaling only handles requests that do not increase data load, x-axis scaling applies. This consists of running multiple instances of an application behind a load balancer (Api Gateway). If there are N copies, then each copy handles 1/N of the load. Every copy must run on a different hardware.

When we have to deal with incremental data storage that needs to be sharded for example, we need to have a different approach. Z-axis splits are commonly used to scale databases. Data is partitioned across a set of servers based on an attribute of each record (shard key). The primary server or router sends each request to the appropriate partition, where it is indexed and stored.
Z-axis scaling helps us to improve cache utilization and reduces memory usage and I/O traffic.

Monitoring
Spring Cloud gives us some useful features for monitoring production environments, giving us the ability to make smart decisions. One killer feature is called Hystrix Dashboard.

Support

It doesn’t matter how good the product that we have developed is, it will fail. When that happens a carrier class product must have excellent support and our response time must be immediate. It is not only how fast we attend the phone or reply in the ticketing system, it is also the logging and debugging system that we have installed into our implementation. Understanding what happens is the key factor to resolving the problem: if we don’t know what is happening, God help us.
A few months ago we started to develop a new solution for proactive support called Sentinel. If you don’t have it yet, please contact us for more information.

Conclusion

Developing Products for CSPs is not an easy task. There are lots of constraints that do not exist in other industries. The difference between a “three-nine” software and a “five-nine” software is huge, and walking that path requires expertise and a solid knowledge of software development and industry insights.
At Intraway we have been doing the Job for more than a decade now. All Our team is focused on building not only a good product, but a carrier class product that fits telecom industry requirements.

References

Carrier Grade”

“CircuitBraker”

Spring Framework

The Art of Scalability