APISIX Empowers Operations and Traffic Control in Human-Machine Interaction

Overview

About CVTE and Seewo

CVTE is a global leader in LCD (Liquid Crystal Display) technology, specializing in LCD mainboards, intelligent interactive panels, medical devices, and related Intelligent hardware. Established in 2005, it holds a 31% market share in the LCD TV mainboard category. This expertise has culminated in the establishment of several prominent industry brands, including Seewo, a provider of digital educational tools and services, and MAXHUB, a sophisticated smart collaborative platform, which have empowered education collaboration and corporate communication.

With over 6,500 employees, CVTE's Seewo interactive flat panels are in 1 million classrooms worldwide. With expertise in display drivers, signal processing, power management, human-computer interaction, application development, and system integration, the company is dedicated to enhancing communication experiences in consumer and commercial electronics through ongoing technological innovation.

Challenges

• Rapid business expansion has frequently triggered reloading in Seewo's two-layer gateway architecture, impacting the system's performance and stability and leading to a suboptimal user experience.
• Seewo's gateway has encountered difficulties in efficiently coordinating traffic control, resulting in diminished operational efficiency and limited traffic control capabilities.
• Seewo's gateway should manage many domains with various personalized configuration requirements, requiring meticulous management and customized control.

Results

• APISIX provides a more flexible and seamless configuration update method, allowing Seewo to easily update routes and certificates as needed, ultimately enhancing Seewo's operational efficiency and system stability.
• After implementing APISIX, Seewo has strengthened its circuit-breaking and rate-limiting capabilities, which have improved traffic control and further solidified its core business processes.
• APISIX gateway's robust scalability and exceptional plugin performance have empowered Seewo to develop custom plugins, extending its gateway functionality to meet the ever-growing business demands.

Background

Seewo's Gateway has evolved through several iterations, showcasing a journey marked by constant technological innovation and the team's persistent dedication. With each release, we've strived to enhance and optimize the previous version, delivering users a network experience that's not only more efficient and stable but also comes packed with richer features.

• First-generation gateway: Seewo's initial gateway was constructed using OpenResty and NGINX with a static configuration. However, it relied on Secure Copy (SCP) for deployments, necessitating operational assistance to ensure smooth releases.

• Second-generation gateway: The second-generation system introduced improvements by integrating the upsync module with OpenResty and incorporating Consul for service discovery. This version empowered developers to independently release updates, though operational support remained essential for scaling.

• Third-generation gateway: Seewo's third-generation system was developed on Kubernetes (K8s). While this adaptation addressed previous deployment and scalability challenges, it introduced new complexities. Recognizing that some applications still resided on the host machine, the gateway architecture incorporated Ingress NGINX on K8s as the second layer, while maintaining OpenResty as the first-layer gateway. However, this approach, while enhancing scalability, led to disruptions during routing changes, affecting applications with persistent connections.

The swift growth of Seewo's business has raised the bar for overall stability. Adopting this dual-layer gateway structure comes with an issue – any adjustments, whether it is the reloading of the first-layer NGINX or modifications in the routing of the second-layer gateway, can lead to the disruption of long-lasting connections. This could pose a substantial problem, particularly in scenarios where maintaining uninterrupted connections is critical. For instance, imagine a situation where Seewo's software is in the middle of retrieving a teacher's teaching status, and suddenly, the connection gets cut off. This interruption not only disrupts the status retrieval process but also has a direct impact on the quality of the teaching experience.

Drawbacks Before Using APISIX

• Time-consuming Reloading in Dual-Layer Architecture: In a dual-layer gateway structure, making even minor adjustments, such as adding domains, configuration modifications, or implementing special rules, requires a time-consuming NGINX reloading. This process can be cumbersome and impacts operational efficiency for Seewo.

• Unmanageable Traffic Control and Switching: Addressing complex traffic control issues in Seewo's architecture, serving over 10 million users, demands a holistic perspective. Seewo has faced a significant undertaking in coordinating components for traffic control, considering the expansive user base. In client-side issues, Seewo has encountered a risk of disrupting server-side operations and cascading failures. The lack of robust traffic control at the gateway level even worsened the impact on Seewo's backend systems. Besides, persistent legacy challenges in the interaction between NGINX and Ingress gateway have posed obstacles for Seewo's future traffic switching, necessitating prompt resolution for seamless infrastructure operation.

• Domain Overload and Customization: Seewo's operations involved managing over 700 domains, each accompanied by a myriad of customized configurations, including redirects, denylists, and allowlists. Adapting these configurations to APISIX plugins introduced significant complexity.

• Complicated Dual-Layer DNS: Seewo has faced a hurdle with its dual-layer DNS architecture, posing inconveniences in terms of rollback procedures. In the meantime, it has lacked the optimization needed for efficient internal network calls within the infrastructure.

In response to these challenges and to improve operational efficiency, Seewo made the strategic decision to adopt the APISIX gateway solution.

Solutions When Migrating to APISIX

Optimizing APISIX Route Generation and Architecture

In the early stages of the migration process, Seewo strategically crafted APISIX routes, a pivotal move aimed at enhancing the system's functionality. This decision entailed streamlining the existing architecture by eliminating an additional layer of specific functions like "rewrite" and "set-header." These functions were seamlessly integrated into the second-layer Ingress, contributing to a more efficient and simplified overall structure. Notably, Seewo seamlessly incorporated APISIX plugins into the NGINX configuration, further emphasizing the versatility and seamless integration capabilities of APISIX.

Validating Route Forwarding and Assessing Performance

After generating the routes, Seewo turned its attention to the task of validating the entire forwarding process, recognizing the pivotal role of APISIX. Employing the goreplay tool for recording and playback, Seewo ensured the precision of route forwarding—a testament to APISIX's integral function in maintaining accuracy. Rigorous automated testing was then applied to scrutinize the functionality of APISIX plugins, guaranteeing their seamless operation. With confirmed functionality, Seewo strategically shifted focus to evaluating how well APISIX aligned with its internal performance requirements. To facilitate this assessment, the elastic-apm plugin was introduced, and specific plugins influencing QPS underwent optimization, highlighting APISIX's versatile capabilities in enhancing both performance and functionality throughout the entire process.

Streamlining Traffic Switching for Seamless Rollbacks

The gateway migration encountered its most significant hurdle during traffic switching, directly impacting production quality. Despite leveraging goreplay for traffic recording and playback, the imperative for a reliable rollback solution took center stage. In the face of anomalies, such as forwarding issues or APISIX crashes resulting from traffic switching, Seewo necessitated a swift and dependable rollback mechanism.

To tackle this challenge, Seewo strategically initiated the transition of public network traffic, underscoring the pivotal role of APISIX. This strategic maneuver empowered Seewo to efficiently modify source addresses during the transition, ensuring a prompt traffic rollback in exceptional situations. The entire process of traffic switching under extraordinary circumstances could be seamlessly executed within seconds, thereby showcasing APISIX's instrumental function in facilitating a quick and uninterrupted transition.

Overcoming SSL and Route Matching Challenges with APISIX Deployment

When Seewo first implemented APISIX, it encountered challenges related to SSL handshake. For example, in older systems or OpenSSL libraries, the ssl_ciphers do not intersect with the server's default values, resulting in failed SSL handshakes. However, the APISIX team actively addressed these issues. While navigating compatibility hurdles in Seewo's old system and upgrading versions, Seewo closely collaborated with the APISIX team to find effective solutions, recommending checking the SSL handshake intersection before deployment for a seamless migration.

Meanwhile, the upgrade to version 2.15 LTS of APISIX introduced some new challenges, particularly in route matching. Despite facing issues, the APISIX team responded swiftly and resolved them in their master branch. It is essential to emphasize that, notwithstanding these challenges, APISIX has been proactively providing robust support. This collaboration goes beyond the technical realm, representing a shared journey in driving positive business development.

Achievements After Using APISIX

Optimized Operational Efficiency

Following the seamless integration of APISIX into its operational framework, Seewo has achieved a remarkable milestone by completely eradicating the challenges associated with reloads. This accomplishment has significantly empowered Seewo, allowing it to execute updates to routes and certificates with remarkable ease, resulting in a streamlined and more efficient workflow for its team of developers.

Elevated Traffic Management Proficiency

Thanks to APISIX's remarkable extensibility and exceptional plugin performance, the integration of APISIX into Seewo's infrastructure has significantly elevated its circuit-breaking and rate-limiting capabilities. This strategic implementation not only enhances Seewo's ability to effectively manage and control traffic but also catalyzes substantial reinforcement in its core business processes. The inherent strengths of APISIX contribute to the overall resilience of Seewo's operations, ensuring a smoother and more reliable service delivery. Leveraging APISIX's robust extensibility, Seewo proactively engages in plugin development to further amplify its capabilities and address evolving needs.

Tailored Plugin Development for Gateway Enhancement

Seewo has harnessed the capabilities of APISIX, which boasts extensibility and dependable plugin performance, to engage in custom plugin development. One noteworthy example is its seamless integration of unified authentication features into APISIX, effectively eliminating the requirement for separate authentication systems when onboarding new services. This streamlined approach has significantly expedited Seewo's product iteration process, allowing the company to adapt more swiftly to changing requirements and evolving market demands, thereby fostering a more agile and responsive operational framework.

Efficient Architecture for Cost Reduction

Seewo's strides in cost savings and operational efficiency were propelled by the elimination of the redundant NGINX layer. In their earlier dual-layer gateway structure, developers faced opacity with the initial NGINX layer. However, the amalgamation of the two gateway layers into one not only eradicated redundancy but also granted developers a transparent view of architectural configurations, encompassing routes, plugins, and more. This harmonious integration not only streamlined troubleshooting and issue resolution but also spotlighted the instrumental role of APISIX in fostering clarity and efficiency within Seewo's infrastructure.

Summary

Since implementing APISIX, Seewo has significantly elevated operational efficiency and traffic control capabilities. Concurrently, bolstering gateway functionalities has also expedited its product iteration process, resulting in heightened overall work efficiency and reduced costs. Throughout this journey, Seewo has actively contributed to the community, submitting a total of 8 pull requests to address and enhance various issues related to ecosystem plugins. These contributions include refining support for custom URIs in the batch_request feature and implementing request body validation for the hmac-auth plugin. Looking ahead, Seewo aims to fully leverage and explore the features of APISIX in our ongoing practices, anticipating the introduction of more collaborative functionalities in the future.