Cut Cloud Costs for Real-Time ML on Mobile Apps by ~50% with NimbleEdge

Increasing adoption of AI and machine learning

In recent years, there has been a rapid surge in the adoption of Artificial Intelligence (AI) and Machine Learning (ML) in enterprises, with organizations implementing these technologies to enhance user experience as well as optimize operations. IBM estimates in their Global AI Adoption Index Report 2023 that ~35% of enterprises have already deployed AI in at least one business function, while another ~40% are actively exploring adoption. Adoption has been especially prominent in digital-first verticals such as E-commerce, Fintech, and Gaming, where customers’ expectations of mobile app experiences have increased steadily. For instance, customers of companies in these verticals have come to anticipate features such as personalized product recommendations and search optimization in mobile apps, which rely heavily on ML. 

Lately, several enterprises have also started exploring real-time ML to provide users even more personalized experiences by generating relevant and session-aware predictions. Industry leaders such as Instacart, Doordash and Pinterest have deployed such systems at scale in recent times, and reaped significant benefits. For instance, Pinterest achieved an ~11% increase in key user engagement metrics by incorporating real-time user-product interactions in their personalized home page feed generation system. Such transformative outcomes from early adopters indicate the vast potential of real-time ML systems.

‍

Spiraling cloud costs 

Even though ML has the potential to drive tremendous upside for enterprises, the cloud storage and compute requirements for deploying at scale are often enormous. This in turn leads to correspondingly large cloud bills, which can dampen ROI. Real-time ML exacerbates this issue, as its requirement for cloud compute resources is significantly larger than batch prediction systems. The cost of vast computational resources required for training and deploying intricate real-time ML models, and for real-time data processing can often be prohibitively expensive.

Hence, enterprises face a challenging problem: How can organizations deliver cutting-edge, hyper-personalized experiences to users without facing exorbitant cloud bills? 

NimbleEdge provides an elegant solution by deploying ML workloads on mobile edge devices.

Typical cloud-based real-time ML ecosystem

‍

A cloud-based real-time ML ecosystem consists of the following major components:

1. Stream processing: Event streaming platforms, such as Kafka, collect real-time events and publish those to relevant topics (e.g. products viewed/search queries in an e-commerce browsing session). These streams are passed through a stream processing engine, such as Flink, which extracts relevant real-time features for the ML model from this data‍

2. Feature Stores: These databases store a) real-time features (e.g. product views, session duration) as well as b) batch features (e.g. user demographics, historical purchase data), and require rapid computational capabilities to minimize latency

3. Inference platform: Online inference platforms, usually hosted as microservices, serve ML models which return relevant predictions based on near real-time inputs from the aforementioned feature stores

The components mentioned above are the key drivers of cloud costs in real-time ML. Our comprehensive assessment of cloud pipelines across various customers indicates that over 40% of cloud expenses in most real-time ML use-cases are driven by the need to manipulate real-time user engagement data, such as click-streams, product views and additions to cart. Contribution from real-time user engagement data may often be even higher, as exemplified by Alibaba’s recommender systems on edge, where >90% of the features are based on real-time user-product interactions.  

‍

How NimbleEdge helps cut cloud costs of real-time ML 

NimbleEdge offers an effective solution to reduce cloud costs of real-time ML in mobile apps by leveraging users’ mobile phones as edge devices for processing and storage:

1. Stream processing: Real-time, session-based features, such as product-user interactions serve as the basis for about half of all features in most real-time ML systems. NimbleEdge SDK enables processing of this data on users’ mobile devices, thereby reducing the volume of data handled by cloud-based processors such as Spark, Flink, and batch ETLs. As a result, using edge devices for this processing reduces the cost of stream processing by 40-50%
   ‍(Source: NimbleEdge customer conversations, proprietary research)

2. Feature Stores: NimbleEdge provides an Edge Feature Store that manages both a) on-device real-time streaming features (e.g. product user interactions) and b) global batch features from the cloud (e.g. time of day, user demographics), which together serve as the basis for the ML prediction engine. This leads to a significant reduction in the volume of read and write requests sent to cost intensive, cloud-based feature stores (e.g. Amazon SageMaker Feature Store). As a result, NimbleEdge helps customers reduce cost of feature stores for real-time ML by over 60%.
   (Source: NimbleEdge customer conversations, proprietary research)

3. Inference platform: Devices themselves, powered by the NimbleEdge SDK, act as endpoints for delivering predictions, eliminating the need for additional cloud -based prediction and backend services that constitute inference platforms. This helps customers of NimbleEdge reduce the cloud cost of ML prediction and backend orchestration services by 90-100%.
   (Source: NimbleEdge customer conversations, proprietary research)

‍

A Comparison: Cloud vs NimbleEdge

Cumulatively, using edge devices for storage and processing across stream processing, feature stores, and inference platform results in ~50% reduction in overall cloud costs associated with real-time ML, as demonstrated in the illustrative schematic above (based on NimbleEdge’s work with a major e-commerce client).

If you are also struggling with the exorbitant cost of real-time cloud-based ML, please get in touch with us at sales@nimbleedgehq.ai. We would be delighted to assist you in your journey to deploying real-time ML using edge. In case you have any thoughts or queries regarding this article, please also feel free to reach out to us at the email address above.