Traditionally telecommunication service provider networks have been complex with multiple protocols and multiple communication endpoints in their environments. Service providers require exhaustive testing before adding new features to their networks because they must meet regulatory requirements and their own key performance indicators, including customer satisfaction as measured by net promoter score.
Telcos began their digital transformation journeys with virtualization as defined by ETSI Network Functions Virtualization (NFV). Some of their goals with NFV were more agility, faster delivery of new services, and providing more value added services to their customers. NFV helped the industry realize software based network infrastructure and, more recently, the value of horizontal cloud platforms for scalability, flexibility and reduced vendor lock-in. OpenStack played a strong role in building those telco clouds.
The industry has moved faster toward the adoption of containers and microservices architecture. ETSI has defined 5G as a service based architecture that uses microservices and cloud native technologies. Kubernetes has become the de facto standard for container orchestration and next generation distributed platform, bringing increased scalability, portability and flexibility to IT workloads and promising the same benefits also for 5G.
One of the new promising 5G features is network slicing. With 5G telco service providers finally are taking the opportunity to slice the radio and use it more efficiently to monetize the frequencies they rent from governments. However, adding network slicing along with private 5G and having multiple vendors for each of their older generation mobile networks will make an even more diverse and complex ecosystem. To manage this new environment without dramatically increasing staff, requires service providers to adopt new operational methods. Operation teams should handle the new challenges differently and they should engage more closely with the engineering teams.
GitOps could be an answer to this challenge using advanced automation with control loops, which is a kind of an extension of the CI/CD approach. GitOps is a new operational model for network function deployments and operations that is being widely adopted by the Kubernetes community in IT. GitOps users define infrastructure as code with procedural state definitions and store all in a version controlled system like a git repository as a singlesource of truth.
GitOps has two elements, Git and Kubernetes, in that context, however this practice is not limited to these technologies and can be easily extended to OpenStack and physical infrastructures. If we look at these two technologies, what capabilities are there?
Git
Git is free and open source software and is a de facto standard for files-based change tracking with version control that is ideal for handling everything from small to very large projects with speed and efficiency. As a change tracking system, it can be used as the single source of truth for configuring infrastructure, security, application and kubernetes elements and be integrated into the overall life cycle with these features/concepts:
- History of changes. Shows your network function software version and configuration history with configuration ownership at any point in time.
- Life cycle. Logs changes for your infrastructure tied to releases and allows you to have release configurations for each environment, from test to pre-production and pushing into the production environment.
- Peer review. Provides an approval mechanism before committing changes into production. Review gates in the process to roll out the configuration will lead to a more stable telco network.
- Changes could be with opening tickets known as pull requests, so all changes are self-documented, providing a built-in change request for all of your all infrastructure. The system exposed APIs also provide integration points with existing ticketing systems
- Eliminate hidden configuration. With all visible to team members, employee turnover will not cause a loss of knowledge regarding why a configuration was implemented. A git integrated process helps prevent undocumented configurations.
- Version rollback. In case anything goes wrong with a newer configuration version of a network function, you have the option to simply return to the previous known-good version that git has already tracked, shortening recovery time.
- Cover other equipment. Adding automation logic and configuration to git for external equipment (e.g., load balancer, physical network equipment) as part of the overall source of truth, makes it reusable with repeatable outcomes.
- Improve security and compliance. Security and compliance teams can use git for auditing. Changes will be transparent and collaboration will increase between teams.
Kubernetes
One of the great patterns of Kubernetes is having declarative architecture, i.e., where you are configuring towards the desired state and let the platform bring itself to that state. This makes the platform easy to scale and manage via automation. Compared to previous platform operating models, the deployment of network functions on Kubernetes is faster and brings the accelerated adoption of new features and updates via the same declarative models. As a platform Kubernetes abstracts numerous functions required for efficient operations, such as configuration handling, logging and metrics collection, container IP address management, and services discovery, which all work toward a version controlled (potentially git-based) configuration management capability as an enabler.
Git + Kubernetes
Kubernetes is set up to easily synchronize with state definitions coming from a git source. These two technologies today pose a best fit for innovation in telco clouds, coordinating and providing a framework for the hundreds of moving pieces. GitOps should also include CI/CD elements to address telco self-organizing and self-healing requirements around component changes brought in by cloud native network functions (CNFs).
Telco GitOps and CI/CD
GitOps and CI/CD suggest that all infrastructure and software configuration should be stored in a source control system like git that can be considered a single source of truth by controllers like ArgoCD or the multi-cluster management tools synchronizing configuration to kubernetes clusters. Tekton is a modular, kubernetes native open source framework that provides CI/CD pipeline functions. For CI/CD in telco networks additional efforts are required. Automated tests specific to telco – synthetic calls, checks from customer experience management reports, deep packet inspection tools, network performance reports, which includes key performance indicators – should be part of the CI/CD processes. When a new configuration gets pushed by a pull request indicating a change in the source of truth system, the desired state changes for the targeted environment, and a tekton based ci/cd pipeline is triggered to execute preliminary validations, preparations or even integrations via Red HatⓇAnsibleⓇAutomation Platform. The desired state intrinsically should carry checks and validations, higher level simulated traffic tests, and external component checks to be evaluated on the spot. If all these checks are successful, changes in one stage of the configuration repository can be pushed to the next (e.g., into production).
Figure 1. Open5GS example
In Figure 1 we demonstrated an Open5GS CNF setup with multiple components forming a working simulation. All configuration and software deployment tasks were performed under Red Hat Advanced Cluster Management for Kubernetes control and coordination. It has the capability to deploy clusters and apply initial configuration (e.g. enablement of Stream Control Transmission Protocol (SCTP) and network attachment definitions for user plane functions, which needs multus interfaces).
Practically every change in a parameter, kubernetes deployment or configuration in the git source of truth will trigger a rollout in the target Red Hat OpenShift cluster based on placement labels (i.e., central or edge), apply configurations based on desired outcomes (i.e., SCTP settings), and initiate the right Tekton pipeline to run related checks on the gNB and UE elements, execute the download tests prescribed and, if all checks out, flag the package in the git repo as ready for the next stage (pre-production or staging, etc.). At this point Red Hat Advanced Cluster Management picks up the new configuration and deploys all relevant elements from the git source of truth to the newly prescribed pre-production cluster.
Service providers in general can reach the agility they need for operational responsiveness with such cloud native tools, not only adopting kubernetes as container orchestration, but also focusing more on the practices and tooling that contribute to improving their agility.
Peer review: S. Molnar
Peer review: W. Caban
Peer review: S. Kriger
