Serverless Computing and Grid for Apps Explained
“Serverless computing is a new computing model for which server capacity and management decisions are completely hidden from the developer or the operator.” That’s what I found on Wikipedia. The word “serverless” is misleading; it doesn’t mean that servers aren’t required to run code, but that the developer can focus on his code and the infrastructure will take care of the rest.
The Evolution of Computing
Unlike what happens in traditional computing models, serverless is designed to support applications that strongly depend on third-party services. A serverless architecture is defined by stateless compute resources, such as ephemeral containers or event-triggered functions.
A function is a relatively small bit of code associated with some configuration information, such as its name, description, and resource requirements. The code must be written in a stateless style assuming there is no affinity with underlying compute infrastructure, and any persistent state should be stored in an object store or an external DB service.
It is very common, for example, that a specific function is triggered directly by an event that is generated from an infrastructure component, such a storage platform, a messaging system, or an IoT device.
Because of its characteristics, the function is very simple and execution time is usually limited to a few seconds. In fact, most service providers charge for this type of service by the millisecond, inciting developers to focus on code optimization rather the components and services necessary to run complex applications in containers or virtual machines.
Because of this, serverless computing platforms can be highly scalable, able to run thousands of functions concurrently. A characteristic provided by the back-end infrastructure itself.
Introducing Grid for Apps
OpenIO has been developing a scale-out object storage platform for years: it is called OpenIO SDS. Recently, it has adopted a serverless counterpart called Grid for Apps.
The two can be configured as separate components, with OpenIO SDS working as an object store and Grid for Apps as a general-purpose serverless computing framework. But when they work together, you get the best of both worlds: very low $/GB, high scalability, and data persistency provided by the object store. This is enhanced by an integrated computing framework that can process information directly on the storage hardware itself, speeding up many workflows while improving the value of the data stored there. This is the perfect match, with the most stateful of storage systems (thanks to the concept of object, made of data and metadata, and advanced data protection mechanisms) complemented by a powerful stateless computing framework for fast, seamless data processing.
Grid for Apps can run simple, single-purpose functions that are triggered by events from the infrastructure and external services. This simplifies infrastructure management while improving its overall efficiency.
There are an endless number of use cases for SDS and Grid for Apps, limited only by your imagination.
The mechanism is very simple. Each event that occurs at the storage layer (PUT, GET, UPDATE, DELETE, etc.) can be intercepted by Grid for Apps, and the function associated with that particular event is triggered to perform specific tasks. These tasks can be as simple as raising an alarm, or more complex, such as image recognition, pattern matching, video transcoding, or full text indexing. The output of the function could be an enhanced object or enriched metadata, as well as messages or information passed on to other systems, even updating databases.
A few examples will show why this solution is so valuable.
- Data validation: the storage system can check if data coming from sensors or remote devices is valid, in a specific range, or the data can be filtered, leaving only the parts that are necessary for subsequent tasks.
- Metadata enrichment: data stored in the system can be automatically combined with metadata by examining the content and other information included in the original files.
- Pattern matching: each file ingested by the system can be searched for specific patterns, and alarms can be raised or other actions taken if the pattern matches certain rules (an antivirus is a classic example of this use case).
- Video transcoding: storing a video can automatically trigger a transcoding process that creates additional files for different formats and/or bitrates. And the opposite is also true: when a video is read, the system can resize the video in real time for the device accessing it.
- Real time analytics: trigger a specific analytics task if a file contains logs, for example.
Advanced indexing and searching: index files when imported.
- Image recognition: recognize content, or just faces, and perform actions accordingly. You could save pictures and protect privacy by blurring faces, for example.
- You name it: any other operation that simplifies complex workflows and that can be triggered by an event is feasible. Sometimes it takes just a few lines of code and some open-source tools, while, in other cases, you may need a full-fledged application (i.e. Apache Spark, Elastic). But anything can run on top of an SDS cluster.
Serverless computing, and, more specifically, Grid for Apps, is a new programming approach to complex IT problems. Scalability, performance, or any other infrastructure issue that is not involved with the solution itself is no longer visible, simplifying and improving the overall development process. By slicing a problem into small pieces and solving it with lightweight functions triggered by events, programming logic is simplified and code is much easier to maintain.
From a business point of view, an end-to-end solution like SDS plus Grid for Apps allows for faster development, while getting the best from the infrastructure and increasing the quality and value of data stored in it.
Grid for Apps is the icing on the cake for an SDS cluster: it has the potential to integrate applications directly with data, without any layer in the middle, simplifying the stack and improving overall efficiency. It is a unique and cost-effective solution, because of the light weight and flexibly of SDS, which leaves a lot of compute resources available for Grid for Apps and its applications. This enables the creation of next-generation advanced data services tailored for the needs of each user.