Outgrow 2.5 - A Transformation in Scalability and Efficiency
In the ever-evolving landscape of technology, scalability, cost-efficiency, and architectural optimisation are paramount. The "Outgrow 2.5" project, which I led during my tenure at Waycool, represents a significant leap forward in addressing these challenges. This transformative initiative sought to redesign and redefine the entire infrastructure and architecture of Outgrow, a platform that was struggling with issues of scalability, server costs, and monolithic architecture.
The Initial Challenge:
Outgrow 2.0, the existing infrastructure, operated on a monolith architecture. The server costs were soaring, reaching approximately 4 lakh rupees per month, which encompassed all facets, including frontend, backend, and AI/ML servers. The scalability was limited, with Outgrow 2.0 supporting 20 crops. The plan was to scale this to a staggering 140 crops. Integration of the acquired Gramworkx IoT platform was another crucial challenge. Most of the services shared the same virtual machines (VMs) in Azure, leading to a single point of failure.
The Outgrow 2.5 Transformation:
The journey of transforming Outgrow 2.0 into Outgrow 2.5 commenced in August 2022 and concluded in January 2023. The primary focus of this transformation was on agility, scalability, and reliability. The architecture and infrastructure were entirely re-envisioned to address these key considerations.
Key Components of the Outgrow 2.5 Transformation:
- Microservices Architecture: Every feature was scrutinized, and the decision was made to transition to a microservices-based architecture, providing greater flexibility and scalability.
- Kubernetes Setup: A comprehensive Kubernetes setup was established within a Virtual Private Network (VPN) in Azure. This involved assembling a dedicated DevOps team and creating an automated pipeline.
- Automated Pipeline with GitHub Actions: An automated pipeline was implemented using GitHub Actions, tightly integrated with the Azure Kubernetes Service (AKS) cluster. This pipeline encompassed unit testing, integration testing, and enforced standardized coding practices.
- Dockerization: Upon successful pipeline testing, Docker images were built and deployed to Docker Hub. These images were subsequently pulled for deployment within the Kubernetes cluster.
- Logging and Monitoring: The architecture featured an extensive logging and monitoring setup for both infrastructure and applications. Prometheus, Grafana, Loki, and Elasticsearch were utilized for this purpose.
- Third-Party Services Integration: Outgrow 2.5 incorporated numerous third-party services, including MQTT broker systems for IoT, Exotel for IVR, Jira and Zendesk for issue tracking, Msg91 for SMS services, Kong for API gateway, Firebase for FCM and crash logs, MySQL for database management, Azure Blob Storage for data storage, the Elastic Stack for log management, and Metabase for analytics.
The Impact:
Outgrow 2.5 wasn't just an architectural transformation; it was a financial and operational revolution. By reducing server costs to 1.5-2 lakhs per month, the project significantly improved the scalability and efficiency of the platform.
The Outgrow 2.5 Landscape:
Outgrow 2.5 boasted an architecture comprising 17 microservices, over 200 APIs, and four distinct environments (development, quality assurance, staging, and production). This marked a substantial improvement over Outgrow 2.0, which had only two environments.
Emphasizing Security:
Security was a top priority. The new setup included robust security monitoring across the entire infrastructure.
A Future-Ready Solution:
Outgrow 2.5 is not just a solution for the present; it is poised to embrace the future. It's now easier than ever to incorporate third-party services, including drone or satellite imagery, into the platform.