FunDa: scalable serverless data analytics and in situ query processing

Abstract

The pay-what-you-use model of serverless Cloud computing (or serverless, for short) offers significant benefits to the users. This computing paradigm is ideal for short running ephemeral tasks, however, it is not suitable for stateful long running tasks, such as complex data analytics and query processing. We propose FunDa, an on-premises serverless data analytics framework, which extends our previously proposed system for unified data analytics and in situ SQL query processing called DaskDB. Unlike existing serverless solutions, which struggle with stateful and long running data analytics tasks, FunDa overcomes their limitations. Our ongoing research focuses on developing a robust architecture for FunDa, enabling true serverless in on-premises environments, while being able to operate on a public Cloud, such as AWS Cloud. We have evaluated our system on several benchmarks with different scale factors. Our experimental results in both on-premises and AWS Cloud settings demonstrate FunDa’s ability to support automatic scaling, low-latency execution of data analytics workloads, and more flexibility to serverless users.

Introduction

Prior to the advent of Cloud computing, organizations were tied to traditional infrastructure to meet their computing demands. Such computing infrastructure involved hosting physical servers either on-premises or at hosting facilities, which required substantial investments and ongoing maintenance. Cloud computing changed the landscape of enterprising computing, offering a more flexible and cost-efficient alternative by abstracting much of the physical infrastructure, and disencumbering organizations from hardware management responsibilities. Serverless Cloud computing or serverless, a new Cloud computing paradigm, has taken this one step further. It introduces a pay-as-you-go model where developers can deploy code without worrying about the underlying servers, leading to automatic scaling based on demand. The Function-as-a-Service (FaaS), a core component of serverless, allows developers to execute small, event-driven functions with no need to provision resources, offering near-instant scalability and resource management. However, while serverless is ideal for short-running tasks, it struggles with the challenges of data analytics and resource-heavy tasks, such as long-running queries or machine learning workloads.

Although serverless offers a useful means for on-demand execution of short-running jobs, it is not designed for general purpose data analytics. Existing serverless platforms, such as AWS Lambda, Google Cloud Functions, and Azure Functions, were conceptualized to be frameworks to support elastic execution of short-running ephemeral jobs. These frameworks impose significant constraints that include a limit on the execution time, limited size (e.g. maximum 10GB RAM), a lack of persistent local storage, and inter-instance communication facilities [1]. Moreover, commercial serverless offerings usually charge a premium price for FaaS based services. On these platforms, users are charged based on the number of requests for functions and their execution duration, where the duration price of functions can be 3.5$\times$ to 5.6$\times$ higher than that of conventional VMs [2]. Hence, for long running data analytics and query processing workloads, serverless may incur unexpectedly high expenses to the users. Moreover, inter-function communication is poorly supported in most serverless environments, hindering complex workflows and data-sharing between functions. These constraints drive the need for more flexible, cost-effective solutions capable of handling both short and long-running tasks with minimal overhead.

Existing serverless platforms cannot be directly used for data analytics due to the aforementioned issues. Recently, researchers have proposed a few experimental prototypes, such as Flint [3] and MetaQ [4]. However, these systems utilize AWS Lambda FaaS functions and therefore inherit the limitations of AWS Lambda. In this paper, we present FunDa, a new approach to serverless computing, designed to overcome these challenges by enabling users to execute both short-running and resource-heavy analytical workloads. FunDa can harness idle or underutilized on-premises hardware, such as university lab machines, and integrates them as workers in the serverless framework, while maintaining their availability for regular use. The system is versatile and can run on on-premises setups or on AWS containers, providing a flexible and scalable solution for a wide range of computing environments, and brings back advantages of running on-premises such as safety of the data when kept local and full ownership and control over the hardware, software and data, while also having the advantages of serverless.

An important objective of FunDa is to offer a serverless platform for unified data analytics and in situ analytical query processing. FunDa leverages the distributed data analytics system DaskDB [5], and an open-source container-native serverless framework Fn project [6]. In today’s data-driven world, large volumes of data are stored in raw files and querying this data typically requires an ETL (Extract, Transform, Load) process into a relational database management system, which can hinder time-to-insight for data scientists and analysts. To address this, DaskDB was previously developed by us to enable in situ SQL query processing, allowing the use of Python functions, libraries, and user-defined functions (UDFs) within queries. This approach enables fast querying of raw data without the need for complex, time-consuming preparation steps. DaskDB is scalable and can operate across multiple nodes in a distributed manner and it also supports geospatial analytics [7]. By extending DaskDB, FunDa takes advantage of DaskDB’s capabilities as a unified data analytics and query processing system.

With its event-driven architecture, FunDa aims to be a customizable, developer-friendly, and modular platform for machine learning and data analytics. FunDa empowers developers to create diverse serverless functions tailored to different workloads, from simple tasks to complex, resource-heavy operations, making it a robust tool for modern computational demands.

We demonstrated the viability of a serverless data analysis prototype with a preliminary system [8], henceforth retroactively referred to as Fun${\text{Da}}_{\mathit{prelim}}$. Through our preliminary experimental evaluation involving Fun${\text{Da}}_{\mathit{prelim}}$ with TPC-H benchmark and DaskDB’s UDF benchmark, we identified cold start latency as a significant overhead. This latency is due to the container launch time and preprocessing time in the cold start initialization phase. The cold start problem has been also identified by other researchers as a major issue with current serverless frameworks [9]. In this work, we present a full-fledged system, henceforth referred to as FunDa, which addresses the cold start latency. Its system architecture is designed to minimize overheads while ensuring an efficient and seamless user interaction with the system. In particular, we focus on maintaining FunDa’s true serverless nature by optimizing communication between components and refining the life-cycle management of functions. By concentrating on these aspects, we aim to ensure that FunDa is a viable solution for both light and resource-heavy computational tasks.

In this paper, we conduct extensive experimental evaluation with FunDa using five different benchmarks: TPC-H benchmark, TPC-DS benchmark, DaskDB’s UDF benchmark, a spatial query workload and a benchmark based on the NYC Taxi & Limousine Commission dataset [10] that was used by Apache Flint. We evaluate our system on both a private Cloud (local cluster) and AWS Cloud setups. Our experimental results show that FunDa significantly reduces cold start latency compared to our early prototype. We also perform a cost-efficiency analysis of Funda against Flint, PySpark and Spark, and show that FunDa incurs significantly less cost. Overall, the results demonstrate FunDa’s effectiveness as a serverless system for data analytics with automatic scaling, low-latency execution, and cost-efficiency.

The rest of the paper is organized as follows. We explain the research gaps and motivation in Section "Research gaps and motivation". In Section "Our approach: system organization", we present our approach and system architecture. Section "Interfaces for job submission and system administration" details the developed interface mechanisms. Cold start mitigation is discussed in Section "Mitigation of cold start issues". The experimental evaluation is presented in Section "Experimental evaluation", followed by a discussion in Section "Discussion". Related work is reviewed in Section "Related work", and the paper is concluded in Section "Conclusion and future work".

Research gaps and motivation

In this section, we outline some of the open issues and research gaps with current serverless frameworks, particularly in the context of the requirements of advanced data analytics applications.

Research gaps and open issues

Serverless platforms have gained popularity due to their scalability and cost-efficiency. To fully utilize the potential of serverless platforms, a lot of research efforts have been invested into resolving a few open issues that people are facing in real life practices. In this section, we outline these topics and discuss the reasons behind them.

Cold start issues

One of the most persistent challenges with serverless applications is the cold start issue. This latency occurs when the number of serverless instances are scaled down to zero during inactivity. They must be initialized upon invocation, resulting in a delay, as the necessary runtime environment and dependencies are loaded [11]. Cold starts can have a particularly detrimental impact on applications that require real-time processing, particularly for applications such as machine learning (ML) workloads, where models and datasets must be loaded into memory before execution can begin [9, 12].

The developers of several serverless frameworks acknowledge the cold start issue. The authors of PyWren [13] mentions the problem but states that it does not implement any methods to mitigate it. Starling [14] employs a warming-up technique by keeping functions running in anticipation of incoming traffic, which, while potentially reducing cold start latency, introduces additional costs due to idle resource consumption. Pixels-Turbo [15] combines a warming-up technique with smaller, more lightweight functions to minimize startup delays.

Despite the optimizations, all of the aforementioned frameworks and other similar ones operate within the constraints of Cloud providers, particularly AWS, where cold start latencies are influenced by factors beyond their control, such as infrastructure startup times. This limitation differs significantly from on-premises deployments, where both hardware and software configurations can be managed to reduce cold start overhead.

Limited support for complex analytics

Current serverless solutions, including PyWren [13] and Flint [16], primarily cater to basic MapReduce patterns or simple data analytics tasks. They face limitations when handling complex workloads [17]. Moreover, these systems do not support a unified approach for data analytics and in situ analytical query processing on heterogeneous data sources. These operations often necessitate fine-grained control over data, models, and computational resources, which existing serverless frameworks do not adequately provide. Furthermore, the assumption that all input data is stored in Cloud environments like Amazon S3 limits the ability to perform in situ analytics on distributed or on-premises datasets [18].

Vendor lock-in and lack of portability

The architecture of most serverless frameworks, particularly those utilizing AWS Lambda, fosters a form of vendor lock-in. Users are often constrained to the functionalities and performance characteristics of the Cloud provider they choose first. This makes migration between different Cloud providers or transitioning to private Clouds a complex, time and resource consuming process [19]. The abstraction of function execution and resource management by Cloud providers further limits flexibility for organizations that may wish to deploy these systems on-premises [20].

Over-reliance on Cloud providers

Many serverless data processing frameworks, such as Starling and Lambada, are heavily reliant on AWS Lambda. This ties them to specific commercial cloud providers. Dependency on a specific provider without ease of migration can lead to challenges regarding privacy, compliance, and cost predictability. Especially for organizations that manage sensitive data or adhere to strict data governance policies [21]. The relinquishment of control over the execution environment and hardware infrastructure is a significant drawback for organizations seeking more autonomy in their data processing capabilities [22].

Support for stateful long running tasks

Most serverless platforms, particularly those offered by major Cloud providers like AWS, Google Cloud, and Azure, impose strict execution time limits. For example, AWS Lambda functions have a maximum runtime of 15 minutes, Google Cloud Functions allow up to 9 minutes (depending on the region), and Azure Functions in Consumption plans are limited to just 5 minutes, with longer durations available only in premium tiers.

Additionally, serverless platforms are inherently stateless, meaning they do not retain memory or computational state between invocations. This makes them unsuitable for long-running workloads, such as data analytics, where preserving computational state is crucial. Some systems, like Flint [3], attempt to overcome this limitation by transferring intermediate results to newly spawned function instances to allow long running computations. However, this approach introduces overhead due to data transfers, reinitialization, and the setup of new instances. In contrast, a system that allows computations to persist within the same instance-without arbitrary execution limits-would provide better performance and efficiency.

Cost of serverless platforms: a case study

The financial costs associated with executing long-running and complex analytic tasks on serverless platforms represent a significant constraint for users. To better understand the cost differences across various platforms, we conducted a case study that emulated real-world data analytics scenarios. In this study, we compared the costs of executing tasks on five leading serverless platforms: Amazon AWS Lambda, Azure Functions, Oracle Cloud Infrastructure (OCI) Functions, Google Cloud Functions, and IBM Cloud Functions.

The execution time model, widely adopted in the industry, describes the computing resources consumed by executed tasks. Resources are measured in units of GB-second or CPU-second. To apply this model, we began by estimating the resources required for a typical data analytics task. Consider a task that requires 10 CPU cores, each with 1.5GB of memory, running for 100 seconds per invocation. Assuming that a function is invoked 1 million times per month, we calculated the total computational resources required monthly. Using the pricing models of the selected serverless platforms, we derived a comparison of their financial costs, as illustrated in Fig. 1.

Fig. 1.

Monthly costs (estimated) by serverless platforms for an analytic task

Under our task assumptions, Google Cloud Functions and IBM Cloud Functions emerged as the most cost-effective options. However, this cost comparison is sensitive to changes in task parameters. For instance, if the function is invoked at a much higher frequency, such as significantly more than 1 million times per month, Google Cloud Functions may become more expensive than AWS Lambda due to its higher cost per million requests. This highlights the importance of tailoring platform selection to specific workload characteristics to optimize cost-efficiency.

Motivation

FunDa aims to address these aforementioned limitations by offering a flexible, serverless in situ analytics platform that supports both SQL-based query processing and complex analytical workloads, especially workloads involving statistical and machine learning models. By leveraging the open-source Fn project, FunDa can be deployed in on-premises or hybrid cloud environments. Thereby, granting users greater control over their execution environments and hardware infrastructure. Additionally, FunDa facilitates seamless integration with existing Python UDFs. This aims at alleviating the burden of codebase migration while supporting scalable workload handling. Without reliance on proprietary Cloud-based scaling solutions. Finally, FunDa addresses the cold start latency problem by incorporating several strategies.

Our approach: preliminaries

FunDa leverages several powerful and flexible software tools and frameworks to build a distributed processing system that is highly scalable and suitable for data-intensive tasks. In this section, we describe the software used in the making and deployment FunDa.

DaskDB

DaskDB [5, 7] is an in situ distributed query processing and data analytics system for both regular and spatial data, built by extending Python’s Dask [23] and Dask-Geopandas [24] framework. It enables the user to perform SQL-based querying of distributed data, alongside the execution of other Python code as User Defined Functions (UDFs) to the SQL queries.

This allows users to perform complex analytics, machine learning tasks, and other operations on large datasets. DaskDB is designed to run in a distributed configuration, enabling seamless processing of data across multiple nodes or machines.

DaskDB supports various file formats, including CSV, shapefiles (shp), parquets, and others, allowing users to query and analyze structured, unstructured, and spatial data. By combining SQL interface and UDF functionality with Dask’s scalability, DaskDB offers a flexible and powerful solution for both regular as well as spatial big data analytics. It serves as a foundational tool for FunDa.

DaskDB follows a distributed architecture consisting of a Scheduler, multiple Workers and a Client. The Client receives and SQL query from the user, parses it and generates a physical plan which constitutes of operators similar to that of Dask. The physical plan is then converted to an equivalent Python code having the actual Dask operators. The generated code is then submitted to the Scheduler for execution. The Scheduler divides and distributes the task across the different Workers for processing, gathers the intermediate results and then finally combines them to return the result back to the Client, which is in-turn returned back to the user.

Fn project

The Fn Project [6] is an open-source, serverless computing platform that enables developers to build and run functions as a service (FaaS) across a wide range of environments. It allows for the deployment, management, and execution of event-driven applications, where functions are triggered by specific events or requests. It serves as one of the building blocks within the FunDa architecture.

Docker

For the distribution and deployment of FunDa, we utilize Docker [25] containers, a technology that packages applications and their dependencies into isolated, lightweight units known as containers. This containerization approach guarantees consistent performance across various environments by encapsulating everything the application requires to function, independent of the underlying infrastructure.

Container orchestration involves the automated management, scaling, and deployment of containerized applications across a cluster of machines, ensuring efficient and seamless operation at scale.

Our approach: system organization

In this section, we describe the system architecture of FunDa. The system is composed of several key components, including a user interface for interaction as detailed in Section "Interfaces for job submission and system administration", as well as the Service Coordinator, Driver, DaskDB Scheduler, and DaskDB Worker. The overall architecture of FunDa is illustrated in Fig. 2. We first provide a detailed explanation of each individual component, followed by an overview of the flow of execution when a job is submitted to FunDa.

Fig. 2.

System architecture of FunDa

Service coordinator

The service coordinator handles jobs submitted through various channels, including HTTP POST requests, the Web interface, or an Fn function. The Service Coordinator is responsible for managing incoming jobs by appending them to a job queue.

Jobs are organized in a FIFO (First In, First Out) manner, ensuring that the first submitted job is the first to be executed. When sufficient resources become available, the next job in the queue is dequeued, resources are provisioned, and the job execution is initiated. After provisioning the necessary resources and executing the job, the system moves to the next job in the queue, repeating the process.

Driver

The Driver is a component capable of creating, starting, stopping and removing containers on all listed machines. It carries out a series of operations on Docker containers for the workers and/or scheduler components of FunDa, as required by the implemented logic, to allow jobs to run in a fully serverless and optimal manner. It keeps track of the job status of the submitted jobs, as well as their progress information. The Driver adeptly orchestrates the deployment, scaling, and management of these containers across multiple nodes, ensuring provisioning resources and freeing them completely as required. It is designed to have a comprehensive API allowing it to be modular and to execute the commands passed to it, the details of which are in Section "System administration".

Container orchestration with Docker Swarm

Docker Swarm [26] is used for orchestrating containers across the cluster. It manages container deployment by defining services, which are sets of identical containers distributed over multiple nodes. Swarm uses a declarative model where the desired state of the system is specified, and it automatically schedules and distributes containers to achieve that state. Swarm also provides built-in scaling capabilities. For example, if a service needs to handle increased load, a simple command like Docker service scale $<$service_name$>$=$<$replica_count$>$ will instruct Swarm to launch additional container replicas across available nodes. Conversely, if demand decreases, Swarm can reduce the number of replicas. It monitors container health through built-in health checks and can automatically restart failed containers or reschedule them on different nodes if necessary. Additionally, Docker Swarm supports rolling updates and rollbacks, allowing seamless updates of services without downtime. This orchestration tool abstracts the complexity of managing individual containers, ensuring high availability, fault tolerance, and efficient resource usage by dynamically adjusting container placement based on the current cluster state and resource availability.

DaskDB Scheduler

The Scheduler container is launched by the Driver via the “Init Scheduler” operation or manually using Docker commands. It remains active on the main machine beyond the duration of any individual job. When a job is submitted by the Driver, the Scheduler coordinates the distribution of its sub-tasks across the Worker containers. It collects partial results from these Workers to compute the final output. The Scheduler achieves this by constructing a task graph, where each node represents a Python function, and the edges between nodes represent the data being transferred between operations. Once the task graph is generated, the sub-tasks are executed in parallel by the Worker containers.

DaskDB Worker

During the start of execution of each job, the Driver launches and initiates the Worker containers on the available nodes. The Driver controls the entire lifecycle of these Worker containers, which are created from the base image daskdb_worker on the machines listed on a Driver’s list of available machines. The Driver can start, stop, or even remove these containers from the available worker machines, as required.

Once a Worker container is instantiated from the image, it can be started at any time by the Driver. The Scheduler automatically recognizes the Worker and adds it to the list of available workers. The Worker is then assigned sub-tasks by the Scheduler, utilizing the machine’s resources (e.g., CPU cores, memory) to perform computations. The Worker can store intermediate results and manage them throughout the task execution. Upon completing its sub-task, it returns the results to the Scheduler.

After the job is completed, the Worker reaches the end of its serverless lifecycle, and its execution is stopped. This ensures that all resources-CPU, memory, and any other machine resources-are fully freed, leaving no lingering processes. When a new job is submitted, the Driver initiates a new cycle, running Worker containers on selected worker machines to execute the task.

The FunDa system maintains a list of all available machines that can be used as workers, which includes details such as IP addresses, usernames, and passwords. This list enables the Scheduler and Driver to have full control over each machine, allowing it to deploy Worker containers when needed and release resources when tasks are completed. The list can contain a large number of nodes (potential workers), such as machines at a university computer science lab. However, if a job only requires a small number of workers, say three, the Driver will only launch containers on those three machines, and frees them after execution completes.

It is important to note that machines used as Workers for FunDa can still perform other tasks, as only a portion of their resources is utilized by the system. This model enables us to leverage idle resources from a range of environments, such as university labs, AWS instances, or any server with the necessary minimum requirements, which can be added to FunDa’s pool of available workers. These machines can be dynamically allocated and freed, optimizing resource usage without locking down infrastructure.

Flow of execution of a job

As previously mentioned, Fig. 2 outlines the system architecture of FunDa, which leverages DaskDB for distributed processing. Here is a step-by-step breakdown of how the system functions:

1. User interaction:

   • The user interacts with the system either by submitting a job through a serverless function (A) or by utilizing a Web interface (B) or using HTTP post requests (C).
   • The job could be anything from querying data to executing an analytical task.
2. Service Coordinator:

   • The Service Coordinator takes charge of managing user-submitted jobs. It places these jobs into its queue (e.g., Job 1, Job 2, etc.), making them ready for processing by the backend system.
3. Driver (Orchestrating task execution):

   • Once a job is pulled from the queue, the Driver orchestrates the interaction with the underlying DaskDB system. It uses its built-in commands to control the Scheduler and Worker containers.
   • Driver commands (a to f):

     • (a) Init Scheduler: Initializes the scheduler component in DaskDB, which handles task distribution among workers.
     • (b) Create Worker: Creates the worker nodes that will execute the tasks in parallel.
     • (c) Start Worker: Initiates the worker nodes to be ready for task execution.
     • (d) Execute: Executes the job/task across the DaskDB system.
     • (e) Stop Worker: After completion, the driver stops the workers to free up resources.
     • (f) Delete Worker: Removes or shuts down the worker nodes once they are no longer required.
4. DaskDB (Processing Engine): The DaskDB [5] system, consisting of a scheduler and multiple worker nodes, is responsible for processing the actual tasks submitted by the user. It distributes tasks across multiple workers, optimizing performance through parallel execution.

   • The Scheduler receive jobs from the Driver and start by parsing them, extracting relevant queries using the (SQL) parser. The metadata and query are then sent to the query planner, which generates a logical plan, followed by a physical execution plan. This plan is passed on to the Dask planner, which translates it into Dask code, enabling it to be processed and executed across all worker nodes by the underlying Dask framework.
   • The Workers execute the distributed tasks handed to them by the DaskDB client/scheduler. These are dynamically created, started, stopped, and deleted as per the system’s workload needs.
5. Job completion notification:

   • Once the job is either successfully executed or fails, the system sends a notification back to the Service Coordinator to relay the status to the user.
6. Final response:

   • The Service Coordinator sends back the result (or an error) to the user, completing the job life-cycle.

This modular design ensures the efficient and seamless serverless execution of jobs (tasks/queries) across multiple nodes, while providing users with flexible interaction options to submit jobs and monitor the system’s performance. The upcoming section will outline the available system interfaces for user interaction.

Interfaces for job submission and system administration

In this section, we describe the different interfaces available to the end users to submit jobs to the FunDa system and the interfaces available to a system administrator.

Job submission

The new interaction mechanism for FunDa offers users three primary interfaces for submitting jobs: via a serverless function, through a web interface, or using direct HTTP submission. This component streamlines the user experience and enhances flexibility for various use cases.

Job submission via serverless function

Users can create serverless functions as needed using the Fn project framework, which supports multiple programming languages (e.g., Python, JavaScript, Java, Go). This allows users to develop serverless functions that combine DaskDB code with other operations in a chosen language, leveraging the flexibility of the framework to integrate with other systems or pipelines.

DaskDB code can be embedded directly within the serverless function’s main logic, and it can be combined with other frameworks or external libraries to perform more complex workflows. For instance, the serverless function might ingest data, pre-process it, execute DaskDB queries, and then pass the results to additional systems for further processing or visualization.

These serverless functions abstract the need for managing server infrastructure and significantly reduce operational overhead, as the necessary resources are dynamically allocated only when the function is triggered.

For example, a user could create a serverless function called Kmeans_calculator to compute K-means clustering on a large dataset. The function may perform the following actions:

1. Retrieve data from a storage system.

2. Execute K-means algorithm using DaskDB.

3. Pass the clustering results to another system for further processing or analysis.

4. Send the processed results to a third system for storage or display on a dashboard.

The entire process can be encapsulated within a single serverless function, which can be triggered through the terminal using a simple curl command. This allows the user to reuse the function as needed and integrate it into different pipelines or applications.

Submission via web interface

In this mode, users interact with FunDa through a graphical Web interface, enabling intuitive job submission and monitoring without requiring technical knowledge of command-line tools or APIs.

As shown is Fig. 3, visiting http://$<$ip_address$>$:8788/ will open a Web interface corresponding to a “landing page” protected with a password and equipped with a session manager to grant access only to logged-in users. After logging in, a webpage is displayed (Fig. 4) that contains a file input field and a submit button, allowing users to upload code files designed for DaskDB execution on the cluster.

Fig. 3.

Web interface for login

Fig. 4.

Submitting a task through the Web interface

The submitted files are passed for execution, and the execution results are directly displayed on the web browser. In case of errors, the error messages are also shown. If the output of the execution is a file, it can be saved to any specified location as defined in the code.

Monitoring tasks, such as monitoring the status of the Scheduler, Workers, job and network can be done through Dask’s interface

ocated at http://$<$ip_address$>$:8787/

Submission via HTTP requests

Users have the option to submit jobs directly through HTTP requests using terminal-based tools such as curl or any other HTTP post method (e.g. Python requests)

At http://$<$ip_address$>$:8788/fnexec an API endpoint is exposed by the FunDa system. Through this, users can post job requests. This method provides a lightweight and flexible way to interact with the system, making it ideal for users who prefer terminal-based operations or wish to integrate FunDa into other tools or scripts.

To submit a job, users can issue an HTTP POST request to the exposed endpoint with the relevant payload, such as a DaskDB code file, in the body of the request. The results of the execution (or errors) will be returned as a response to the request.

System administration

A range of API commands are available to a system administrator of FunDa. For instance, to configure and operate the DaskDB System, workers need to be created in the nodes. The Driver can be used by the following command:

curl -X “POST” $<$url of daskdb_init trigger$>$-H “Content-Type: application/json” -d ’{“cmd”:“create $<$ip_of_scheduler${>}^{\prime \prime }{\}}^{\prime }$

Among the key operations, the start workers operation starts a specific number of worker instances. The command to do so is:

curl -X “POST”$<$url of daskdb_init trigger$>$ -H “Content-Type: application/json” -d ’{“cmd”:“start $<$Pnumber_of_workers$>$”}’

The Stop Workers operation frees up system resources by terminating the workers, which results in the shutdown of their Docker containers on the worker nodes. To stop worker instances, the following command can be executed:

curl -X “POST”$<$url of daskdb_init trigger$>$ -H “Content-Type: application/json” -d ’{“cmd”:“stop $<$number_of_workers$>$”}’

If there’s a need to remove a worker instance completely, the command would be:

curl -X “POST” $<$url of daskdb_init trigger$>$ -H “Content-Type: application/json” -d ’{“cmd”:“delete $<$number_of_workers$>$”}’

This structured approach ensures effective management and orchestration of the system.

Mitigation of cold start issues

Cold start latency mitigation mechanisms are essential for improving the performance of serverless systems like FunDa. In this section we discuss our cold start mitigation strategies.

Cold start mitigation approaches

Cold start latency is a significant challenge in serverless computing. The latency introduced by cold starts is influenced by factors such as the size of the function’s deployment package, the complexity of the runtime environment, and the overhead associated with resource allocation [27].

Mitigation strategies like provisioned concurrency, which pre-warms a specified number of function instances to handle incoming requests without delay, have shown promise in addressing this issue [28]. However, these solutions often come at the expense of increased costs and memory consumption.

Other techniques, such as warm-up mechanisms, periodically invoke functions to keep them active, reducing cold start latency. However, these methods also introduce additional resource usage, making them less efficient in environments with sporadic workloads [29]. Additionally, cold start mitigation is less effective in data-intensive applications, where the time required to load complex dependencies-such as ML models and large datasets-remains a significant bottleneck [12].

FunDa’s approach to cold start mitigation

In contrast to many existing serverless frameworks that are tightly bound to Cloud providers, FunDa’s decoupled architecture allows for greater flexibility in mitigating cold starts in on-premises and Cloud environments. By empowering users to have more control over their own infrastructure, FunDa provides a more adaptable solution that is tailored to specific workloads and infrastructure constraints.

Next, we describe the specific optimizations implemented in the FunDa system, focusing on how these strategies were used to improve upon our earlier preliminary system Fun${\text{Da}}_{\mathit{prelim}}$, effectively reducing startup times and improving overall performance.

Application-based optimization

Fun${\text{Da}}_{\mathit{prelim}}$ containers are deployed using Docker [25], with two primary images: one for the DaskDB Scheduler (initially sized at 6.9 GB) and another for DaskDB Workers (initially sized at 6.1 GB). These large container sizes contributed to high cold start latencies due to the time needed to pull and initialize the images. To address this, FunDa aims to optimize the system by creating smaller, more efficient Docker images, reducing the number of installed packages, and ultimately enhancing performance and decreasing initialization time.

Initially, the Fun${\text{Da}}_{\mathit{prelim}}$ prototype used a standard Ubuntu OS image, which resulted in large and inefficient containers. This image included many unnecessary packages and libraries, increasing both the size and complexity of the containers. To reduce the container size, we investigated alternative operating systems that could provide a more minimal and optimized base for the FunDa containers.

Choice of Base image

We first considered Alpine Linux, which is known for its minimal footprint, but it proved incompatible with the DaskDB setup due to its lack of the apt package manager and other essential Debian-based utilities. While Alpine would have provided the smallest image size, its incompatibilities with DaskDB’s environment outweighed this benefit.

Next, we experimented with Minideb, a minimal Debian-based image. However, after installing Python and its dependencies, the Minideb image became bulky and unstable, negating its size advantages. In this case, the need for stability and compatibility with the DaskDB environment led us to discard Minideb despite its minimalism.

Finally, after thorough testing, we opted for a Debian 12 (Bookworm) image, specifically the slim variant, which provided the optimal balance of minimalism, stability, and compatibility. Debian 12 is a stable, well-maintained operating system with broad support for required libraries and tools. The slim variant is designed to be a minimal base, reducing unnecessary overhead compared to a standard Debian installation while ensuring compatibility with critical system components and dependencies.

Package curation

Alongside the choice of base image, we focused on carefully curating the installed packages to ensure a minimal and efficient container. By handpicking only the essential packages and their latest stable versions, we avoided the inclusion of unnecessary dependencies, further reducing the container size. The careful selection of packages ensured that the system remained lightweight, with no extraneous libraries, while maintaining functionality and performance.

Python version selection

The choice of Python 3.12 was an additional performance optimization, as it offers a performance boost of up to 20% over Python 3.9, which was previously used by DaskDB. Initially, we also considered Python 3.13b1 (a beta release), but it was discarded due to its instability and lack of support for critical libraries like NumPy and Pandas at the time of testing. Despite its promising performance improvements, driven by its included Just-in-Time (JIT) compiler, the trade-offs in reliability and compatibility made it unsuitable. Python 3.12, being the latest stable release at the time, provide both superior performance and ensured full compatibility with DaskDB’s codebase.

By selecting the appropriate Debian 12-based image and curating only the necessary packages, we successfully reduced the size of the DaskDB Scheduler and Worker images. The reduction in size, as shown in Fig. 5, led to faster container pulls, quicker initialization times, and a reduction in cold start latency. This application-based optimization highlights the importance of selecting an appropriate base image and minimizing installed packages in serverless environments like FunDa.

Fig. 5.

Size comparison of the original i.e. Fun${\text{Da}}_{\mathit{prelim}}$ Docker images vs. the new (current) FunDa Docker images, lower is better

Checkpoint-based optimization

Initially, we experimented with CRIU (Checkpoint/Restore In Userspace) [30] to checkpoint the DaskDB Scheduler container and restore it when needed. However, due to the relatively large size of the Scheduler container (approximately 1.8 GB), restoring from disk resulted in considerable startup delays. On our test machine, it takes approximately 6 seconds to load the scheduler container, as it requires reading both the container image and the checkpoint data from disk.

The nature of the checkpoint system, which saves and restores data from the hard disk, proved suboptimal for the large Scheduler container. The time required to bring back these saved states negated much of the potential advantage of using checkpoints for cold start mitigation.

As a result, we opted for an always-running Scheduler container, eliminating the need to restore it from checkpoints. Additionally, we applied an application-based optimization to preload essential libraries and components, ensuring that the environment is ready for immediate execution. By prebaking or preloading the necessary libraries, code, and dependencies, such as the JVM for Calcite, the Python interpreter, few Python libraries, and DaskDB’s internal libraries, we significantly reduced the cold start latency.

With this approach, when code arrives for execution, the system does not need to wait for the initialization of these components. The Scheduler container is always running with all essential elements preloaded and ready to execute user queries as soon as they are received, thereby mitigating cold start delays effectively.

Caching-based optimization

The statefulness of FunDa and its internal components such as DaskDB allow for different caching techniques, which help eliminate redundant computations, thus improving performance.

Dask caching The FunDa system is built on top of DaskDB, which leverages the Dask framework [23]. By enabling and configuring Dask’s built-in caching functionality, we were able to take advantage of its opportunistic caching mechanism. This mechanism stores intermediate tasks that exhibit the following characteristics:

• Expensive to compute

• Frequently used

• Cheap to store

Dask automatically identifies such tasks and caches their intermediate results, reducing the need to recompute them on subsequent invocations. This caching feature significantly benefits workloads where specific computations are reused multiple times.

Metadata caching During our analysis of FunDa’s workload, we identified that one of the most computationally expensive tasks was the calculation of the number of rows in large data files. To address this, we implemented persistent storage that caches metadata from previously processed data files. This optimization allows the system to avoid redundant computations on datasets that have already been processed.

For users running analytics on the same dataset multiple times, this caching strategy results in a substantial reduction in latency for subsequent runs. After the first run, where the dataset’s metadata is stored, future invocations can retrieve this cached metadata, thus bypassing the need for expensive re-computation, and significantly speeding up the process.

Summary and impact of the optimizations

Table 1 summarizes the various optimizations implemented to reduce startup time, preprocessing time, and overall execution time in FunDa. Each optimization approach and its respective effect on the system is detailed below.

Table 1.

Summary of Cold start mitigation strategies and their impact on FunDa

Optimization approach	Effect on FunDa
Smaller, optimized Docker images (Base image and package curation)	Reduces image size, making container pulls and initialization faster.
Always-running Scheduler container	Eliminates scheduler startup delay, making task scheduling immediate.
Preloading of critical libraries and components	Avoids repeated initialization overhead for essential libraries (e.g., JVM, Python).
Metadata caching of table details (e.g., row count)	Prevents re-computation of metadata for frequently accessed datasets.
Dask caching (Dask framework)	Stores parts of previous computations to avoid redundant processing.

In research projects, careful package curation, base image selection, and software version management are often overlooked. However, these optimizations collectively reduce system size and enhance performance. Caching techniques further minimize redundant computations, such as calculating or retrieving dataset details (e.g., number of rows) or performing repetitive calculations. In repeated analytics involving the same dataset or similar queries, FunDa’s caching mechanisms provide significant speed improvements compared to stateless serverless systems like Flint or PyWren that need to recompute everything.

Experimental evaluation

To the best of our knowledge, we are the first to propose an open-source distributed architecture based serverless platform for both traditional and geo-spatial based data analytics and in-situ query processing. Most of the existing serverless platforms are based on commercial solutions such as AWS Lambda. The open-source serverless architectures (developed over OpenFaaS or OpenWhisk) are either not distributed or do not support data analytics and query processing. The closest system that we found that does something similar to our approach is Flint [3], a prototype Spark execution engine that supports serverless data analytics. It is based on AWS Lambda, which is not open source, and comes with all the limitations of Spark. We began our evaluation with a series of experiments on FunDa, comparing its performance under various scenarios and benchmarks against our earlier prototype, Fun${\text{Da}}_{\mathit{prelim}}$ [8]. Building on these initial experiments, we subsequently extend our analysis by benchmarking FunDa against Flint, PySpark, and Spark on AWS. In these experiments, we calculate the overall cost, both computational and financial, per processed data unit to provide a meaningful and comprehensive comparison.

Experimental setup

We ran the experiments on a local Cloud (a cluster of local machines) and on a small cluster of AWS t2.micro instance whose configurations are as follows and also shown in Table 2:

1. Local Cloud (LC): It consists of a 4-node cluster each running Ubuntu 20.04 LTS on an Intel(R) Core(TM) i5-2400 CPU and having 8GB RAM.

2. AWS Cloud1: It consists of a cluster of 3 EC2 (Elastic Compute Cloud) instances, each running Ubuntu 22.04 LTS on a single virtual CPU and 1GB-2GB of RAM. The purpose of this choice of virtual environment is to illustrate that FunDa can operate on a resource-constrained setup on a publicly available commercial Cloud. More details are provided in Section 7.2.5.

We measured the launch time of the worker nodes, pre-processing time (which includes reading the relevant metadata files after a job (query/task) is issued), and the execution time. Every time a job was submitted all the workers were started and was then stopped once execution was over. However, the scheduler was kept up and running for the entire duration of the experiments. For all the queries, datasets were stored and read from a network-mounted file system (NFS).

Table 2.

Experimental setup

	CPU	OS	RAM size
Local Cloud	Intel(R) Core(TM) i5-2400 having 4 cores	Ubuntu 20.04 LTS	8GB
AWS Cloud	vCPU on i386 Intel Xeon	–	1GB

Datasets

For the purpose of evaluation, we used FunDa to execute queries and data analytics tasks from various workloads. For traditional queries, we executed Q1, Q3 and Q10 from TPC-H [31] and Q3, Q15 and Q42 from TPC-DS [32] benchmark. All these queries were executed for Scale Factors (SF) 1, 5 and 10, where Scale Factor 1 (SF1) corresponds to about 1GB of data and is used to assess the system’s baseline performance. We also executed two well-known data analytics tasks, which are Linear Regression and K-Means clustering as UDFs to a SQL query, to create a customized UDF benchmark, as shown in Table 3. We also created a Geospatial benchmark and evaluated three custom spatial queries from the GeoNB dataset [33] (which provides geographic data for the province of New Brunswick in Canada), as shown in Table 4. In order to compare with other frameworks, we replicated some of the experiments conducted by Flint [3] using a smaller data size and tested FunDa on both local cloud and AWS. We executed Q1, Q2 and Q4 from NYC Taxi & Limousine Commission [10] with 21.5GB data size, and also compared the cost efficiency of FunDa with the other frameworks. For clarity, we will refer to Q1, Q2, and Q4 as NT1, NT2, and NT4, respectively.

Table 3.

UDF benchmark (queries with custom UDF)

Tasks	Query
LR	select myLinearFit(l$\_$discount, l$\_$tax) from orders, lineitem where l$\_$orderkey = o$\_$orderkey and o$\_$orderstatus = ‘F’
K-Means	select myKMeans(l$\_$discount, l$\_$tax) from orders, lineitem where l$\_$orderkey = o$\_$orderkey and o$\_$orderstatus = ‘F’

Table 4.

Geospatial benchmark (spatial queries on GeoNB dataset)

Tasks	Query
Spatial Analysis (S1)	select ST_Boundary(ppam_geom) from property_assessment_map where Descript = ’RESIDENCE & LOT’;
Spatial Join (S2)	select BuildingID from buildings, flood_risk_areas where ST_Contains(frar_geom, buil_geom);
Spatial Join with non-spatial filter (S3)	select ADDR_DESC from address_new_brunswick, flood_risk_areas where ST_Intersects(frar_geom, adnb_geom) and STREET = ’ROUTE 105’;

Experimental results

In this section we evaluate FunDa on various combinations of datasets/benchmarks and cloud instances as follows, which are also summarized in Table 5:

1. TPC-H on a Local Cloud (with comparison against FunDa$_{\mathit{prelim}}^{}$).

2. UDF on Local Cloud (with comparison against FunDa$_{\mathit{prelim}}^{}$).

3. TPC-DS on Local Cloud.

4. Geospatial on Local Cloud.

5. TPC-DS on AWS Cloud.

6. NYC Taxi & Limousine Commission on Local Cloud

7. NYC Taxi & Limousine Commission on AWS Cloud

Table 5.

Experimental setup

	Queries/tasks	Scale factor	Environment
TPC-H Benchmark	Q1, Q3 and Q10	1, 5 and 10	Local Cloud
TPC-DS Benchmark	Q3, Q15 and Q42	1, 5 and 10	Local Cloud, AWS Cloud
UDF Benchmark	LR, KMeans	–	Local Cloud
Geospatial Benchmark	S1, S2 and S3	–	Local Cloud
NYC Taxi & Limousine Commission	NT1, NT2 and NT4	21.5GB	Local Cloud, AWS Cloud

TPC-H on Local Cloud: Funda$_{\mathit{prelim}}^{}$ vs Funda

In this section, we evaluate FunDa on TPC-H benchmark for Q1, Q3 and Q10 on SF 1, 5 and 10. We also compare its performance with FunDa$_{\mathit{prelim}}^{}$ to show the performance improvement achieved by it over the preliminary prototype. The results are shown in Fig. 6. From the figure, we see that for all scale factors, queries in FunDa takes around 50% less time to execute compared to FunDa$_{\mathit{prelim}}^{}$. FunDa takes much less time to launch all the worker containers, owing to its reduced size. It also takes negligible amount of time for preprocessing due to metadata caching but consumes almost similar time in the execution stage, which is expected, as they are executed on the same platform. As the scale factor is increased, the launch, pre-processing and execution times increase proportionately for Fun${\text{Da}}_{\mathit{prelim}}$, whereas the launch and pre-processing time stay almost identical across all runs. This showed that the dataset itself had no effect on the launch and pre-processing times, except for the execution time, which is an expected behavior.

Fig. 6.

TPC-H benchmark on Local Cloud

To give an idea of the performance improvement, We take for example the query Q3 executed on the highest scale factor i.e. SF10 (Fig. 6c), Fun${\text{Da}}_{\mathit{prelim}}$ requires approximately 14 seconds for launch, 126 seconds for preprocessing, and 147 seconds for execution, totaling 261.53 seconds. In contrast, FunDa takes only around 9 seconds for launch, a significantly reduced 0.01 seconds for preprocessing, and the same 147 seconds for execution, with a total execution time of 155.76 seconds. This significant reduction in both launch and preprocessing time in FunDa highlights its optimization and efficiency improvements over Fun${\text{Da}}_{\mathit{prelim}}$. FunDa gave a reduction of the overall query execution time by almost half, which is an improvement seen consistently across all other queries and scale factors.

UDF on Local Cloud: Funda$_{\mathit{prelim}}^{}$ vs Funda

Similar to the TPC-H evaluation, we evaluate FunDa against Fun${\text{Da}}_{\mathit{prelim}}$ on the custom UDF benchmark on SF 1, 5 and 10 with LR and K-Means tasks. The results are shown in Fig. 7. The performance improvement of FunDa over Fun${\text{Da}}_{\mathit{prelim}}$ is similar to that of TPC-H evaluation. Here too it is observed that overall FunDa takes much less time to execute a job compared to Fun${\text{Da}}_{\mathit{prelim}}$. Similar to the previous scenario, an increase in SF does not affect the launch and pre-processing times, except for the execution times. Due to reduced Docker image sizes, FunDa launches the workers faster compared to Fun${\text{Da}}_{\mathit{prelim}}$, has negligible pre-processing time while the execution time stays the same. For example, for SF 10 on K-Means (Fig. 7c), Fun${\text{Da}}_{\mathit{prelim}}$ takes 8 seconds, 24 seconds and 23 seconds for launching, pre-processing and execution respectively, whereas FunDa takes only 5 seconds, 0.01 seconds and 23 seconds for launching, pre-processing and execution respectively. The total execution times for this task on Fun${\text{Da}}_{\mathit{prelim}}$ and FunDa are 269.58 and 141.09 seconds respectively, showing a performance increase of around 47%.

Fig. 7.

UDF benchmark on Local Cloud

TPC-DS on Local Cloud

TPC-DS is an industry-standard benchmark for general-purpose decision support systems. Evaluating the performance of the FunDa system with TPC-DS provides significant evidence of the value of a serverless architecture in the context of Data Warehousing.

The results in Fig. 8 display the launch, preprocessing, and execution times for three TPC-DS benchmark queries (Q3, Q15, and Q42) across three scale factors (1, 5, and 10). As the scale factor increases, execution time becomes the dominant component, particularly for larger queries like Q42. The preprocessing time remains very minimal and becomes negligible at higher scale factors, while launch time remains relatively stable, adding only a small portion to the overall time. The launch time typically ranges from 2 to 10 seconds similar to previous TPC-H and UDF tests. This indicates that as the dataset size grows, the system manages to efficiently handle the workload, with most of the time spent on actual query execution and less on launch and preprocessing overhead.

Fig. 8.

TPC-DS benchmark on Local Cloud

Geospatial on Local Cloud

We evaluate the performance of spatial queries over FunDa using the GeoNB dataset, as shown in Table 4. To show that FunDa can support a broad spectrum of spatial queries, we use three different types of queries for the purpose of evaluation. They are Spatial Analysis (S1), Spatial Join (S2) and a Spatial Join with non-spatial filter (S3). The results for running these jobs are shown in Fig. 9. From the results, we see that the time taken to launch all the workers are between 2-4 seconds, the pre-processing time is between 0.04-0.08 seconds, while the query execution time varies between 3-10 seconds. The preprocessing time is negligible compared to the launch and execution times, due to which it is not visible in the plot.

Fig. 9.

Geospatial benchmark on Local Cloud

TPC-DS on AWS Cloud

We conducted three TPC-DS benchmark queries (Q3, Q15, Q42) at scale factor 1 (1GB data) on AWS to demonstrate FunDa’s compatibility with public cloud services. We used three EC2 instances, each with 1 vCPU: a T2.small instance as the FunDa driver and DaskDB scheduler (2GB memory, 16GB storage), and two T2.micro instances as DaskDB worker nodes (1GB memory, 8GB storage). Datasets were stored in AWS S3 for shared access.

All experiments ran successfully on AWS Cloud, delivering expected processing and launch times (Fig. 10). However, preprocessing time was longer due to metadata retrieval from S3. We plan to explore several optimization techniques to improve AWS S3 access performance in future work.

Fig. 10.

TPC-DS benchmark on AWS Cloud

Despite the constrained performance under basic AWS setups with low system resources, FunDa successfully delivered results in a serverless configuration, demonstrating its compatibility and scalability on a commercial Cloud platform.

NYC taxi dataset: FunDa vs other frameworks

We evaluated three queries, NT1, NT2 and NT4, from the benchmark of NYC Taxi & Limousine Commission [10], which is the same benchmark that was used by Flint [3]. We used a data size that was 10% of the original (21.5 GB instead of 215 GB). In Table 6, we compared our configurations of FunDa test environments with those used by Flint [3].

Table 6.

Evaluation environment comparison between FunDa and other frameworks

	FunDa		Flint	PySpark	Scala Spark
Platform	Local Cloud	AWS EC2	AWS Lambda	AWS EC2	AWS EC2
Instances	PC x 4	T2.small, T2.micro	Lambda	m4.2xlarge	m4.2xlarge
CPU Cores	16	4	80	88	88
RAM	32GB	4GB	3GB x 80	32GB x 11	32GB x 11

It can be seen that FunDa was deployed with limited resources both on our local Cloud and AWS. The run time results of FunDa from local Cloud are displayed in Fig. 11, corresponding run time results from AWS can be seen in Fig. 12. We observe that on both environments, only a small portion of the total time consumption was taken by launch time while the preprocessing time is negligible. It also showed that FunDa performed relatively better on local cloud because of richer computing resources, such as CPU cores and size of RAM, comparing to the corresponding test environment configuration on AWS.

Fig. 11.

Funda run times for NYC Taxi & Limousine queries on Local Cloud

Fig. 12.

Funda run times for NYC Taxi & Limousine queries on AWS

In addition to the run time presented in Figs. 11 and 12, we evaluated FunDa’s cost-efficiency by measuring the computational resources and financial cost required for delivering a set amount of query performance. To quantify this, we have derived a metric based on the execution time model, a widely adopted industry standard for measuring FaaS resource usage that was outlined in Section "Cost of serverless platforms: a case study". In our metric, we define cost-efficiency (E) as a function of the combined computational resources and financial cost (R) that were utilized during the execution time (T) relative to the datase size (D). Hence, the cost-efficiency can be expressed as Eq. 1.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill E& =\frac{R.T}{D}\hfill \end{array}\end{array}$$ 1

To determine the computational resources and financial cost (R), we considered several factors: the number of CPUs ($N_{\mathit{cpu}}$), the allocated memory size ($S_{\mathit{mem}}$) and the price of the used resources per second ($P_{\mathit{service}}$). And to integrate both computational and financial cost factors, we introduced weighting coefficients for CPU, RAM, and financial cost, denoted as $\alpha$, $\beta$, and $\gamma$, respectively as shown in Eq. 2.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R=\alpha ·N_{\mathit{cpu}}+\beta ·S_{\mathit{mem}}+\gamma ·P_{\mathit{service}}\end{array}\end{array}$$ 2

We assign the values $\alpha =0.1$, $\beta =0.05$, and $\gamma =10000$. These values are chosen based on common hardware configurations, where 2 GB of RAM is typically allocated per CPU core, leading us to weight CPU cores two times higher than RAM capacity. Additionally, since the GB-second cost of running AWS Lambda instances is very small ($0.00001667 per GB-second), we multiply it by 10000 to obtain meaningful results. This approach ensures that the final resource cost ($R$) is influenced by all three factors in a balanced and equitable manner. Eq. 3 shows the used weights :

$$\begin{array}{c}\hfill R=0.1·N_{\mathit{cpu}}+0.05·S_{\mathit{mem}}+10000·P_{\mathit{service}}\end{array}$$ 3

The financial cost per second is represented by the variable $P_{\mathit{service}}$. There are three ways to determine it: (1) retrieving the actual cost from execution logs, (2) using published pricing data, or (3) calculating it based on the provider’s pricing model, which for the case of AWS Lambda is achieved by multiplying the base price per second ($P_{\mathit{lambda}}$) by the allocated memory size ($S_{\mathit{mem}}$), resulting in :

$$\begin{array}{c}\hfill P_{\mathit{service}}=P_{\mathit{lambda}}·S_{\mathit{mem}}\end{array}$$ 4

Similarly, for AWS EC2 instances, $P_{\mathit{service}}$ is calculated by multiplying the cost per second of an instance ($P_{ec2}$) by the number of instances used ($N_{\mathit{instances}}$), giving:

$$\begin{array}{c}\hfill P_{\mathit{service}}=P_{ec2}·N_{\mathit{instances}}\end{array}$$ 5

From the Eqs. 1 and 2, we derive the cost-efficiency metric ($E$) (Eq. 6). This metric quantifies the ratio of combined computational and financial resources (CPU, RAM, and monetary costs) to the volume of data processed, serving as a holistic measure of a FaaS service’s efficiency in balancing infrastructure expenditure and workload throughput.

$$\begin{array}{c}\hfill E=\frac{(\alpha ·N_{\mathit{cpu}}+\beta ·S_{\mathit{mem}}+\gamma ·P_{\mathit{service}})·T_{\mathit{runtime}}}{D}\end{array}$$ 6

We utilize the metric defined in the previous equations to perform a comprehensive cost-efficiency comparison between FunDa, Flint [3], PySpark, and Spark. In this analysis, we calculate the cost-efficiency ($E$) of FunDa’s query execution results and compared them with those of the other platforms obtained from Flint’s paper.

Figure 13 shows that FunDa demonstrate superior cost-efficiency, even while running on the limited resources provided by AWS. The cost per unit of data processed by FunDa with the Query NT1 is 20.17 while Pyspark’s cost was 63.64 and Flint costed 44.89 resources per unit of data processed. This highlights FunDa’s ability to adapt to the configurations of Cloud platforms, achieving low overall costs.

Fig. 13.

Cost-efficiency comparison of PySpark, Spark, Flint, and FunDa (lower is better)

This cost calculation approach that we developed provides a normalized way to compare the systems’ cost in terms of resources required per unit of data processed. Furthermore, researchers can replicate this methodology to compare their systems with those included in this study without requiring identical configurations. By following the same weighting factors and applying the formula in Eq. 4, they can obtain results that are directly comparable to all the mentioned systems.

Summary of the experimental evaluation

A summary of all the experiments conducted, along with their figure numbers, is shown in Table 7. Overall, it is observed that FunDa outperforms its preliminary counterpart Fun${\text{Da}}_{\mathit{prelim}}$ in terms of performance. Unlike Fun${\text{Da}}_{\mathit{prelim}}$, the launch and preprocessing time of FunDa are not dependent on the dataset size. The launch time of the workers remains consistent regardless of the workload, and the preprocessing time is very minimal, significantly improving performance over our previous effort. This also indicates that as the dataset size grows, the system efficiently handles the workload, with most of the time spent on actual query execution and less on launch and preprocessing overhead. Therefore, FunDa proves to be a much more scalable, optimized and thus efficient solution, especially for larger datasets as well as limited resources, as demonstrated by the evaluation results. Additionally, experiments on the NYC Taxi dataset show that FunDa outperforms Flint, Spark, and PySpark in terms of overall resource usage and financial cost per processed data quantity, providing a solid open-source serverless solution.

Table 7.

Summary of experiments

Dataset	Environment	Figure
TPC-H	Local Cloud	6
UDF	Local Cloud	7
TPC-DS	Local Cloud	8
GeoNB	Local Cloud	9
TPC-DS	AWS Cloud	10
NYC Taxi & Limousine Commission	Local Cloud	11
NYC Taxi & Limousine Commission	AWS Cloud	12

Discussion

Throughout this work, our primary focus was on building a strong architectural foundation and ensuring the stability of the FunDa framework. Our design aligns with the principles of serverless computing, offering resource savings and increased efficiency by maintaining an architecture that optimizes performance and resource utilization. The FunDa system is faster, more modular, and has fully decoupled components, making it capable of seamless integration with various systems.

Moreover, key software optimizations, such as selecting optimal operating systems, up-to-date libraries, and optimized containers, further enhance FunDa’s performance. These improvements result in smaller container sizes, faster startup times, and more efficient network transfers, all of which contribute to overall cost reduction. By minimizing the memory and storage footprint through lightweight Docker containers, FunDa effectively reduces the overhead associated with traditional VM-based systems, where resources often remain idle during low-demand periods.

Cost analysis

FunDa on a local Cloud

Unlike traditional serverless solutions that rely on AWS or other Cloud providers, where costs are dictated by provider pricing, FunDa mainly operates on-premises, leveraging existing hardware in organizations, universities, and research labs. This means that idle or underutilized machines can be seamlessly integrated into a distributed processing network, enabling efficient handling of large-scale tasks.

By re-purposing available resources, organizations and educational institutions can build their own private Cloud at no additional cost, facilitating large-scale data processing and workload management without relying on expensive cloud services.

FunDa on a private Cloud

The cost-efficiency of running FunDa on AWS was demonstrated through the cost analysis and comparisons in Section "NYC taxi dataset: FunDa vs other frameworks", where FunDa is evaluated against other platforms. It is worth highlighting that many serverless frameworks rely on AWS Lambda, which, despite its widespread adoption, incurs higher costs compared to traditional AWS EC2 instances, especially in high-utilization scenarios.

Unlike solutions that depend on Lambda instances, FunDa leverages standard AWS EC2 Cloud virtual machines to create a serverless platform. This approach is particularly advantageous in sustained high-utilization scenarios, where EC2 instances are approximately 50% cheaper than Lambda. This cost efficiency aligns with FunDa’s design as an affordable and flexible framework that utilizes local or Cloud infrastructure to enable serverless functionalities.

A practical example of this cost-saving potential can be seen in a university setting. By deploying FunDa on an EC2 instance, the university can allow students to execute queries/functions in a serverless manner while maintaining continuous high utilization across multiple users. This setup can result in cost savings of up to 50% compared to using Lambda. Furthermore, the system is designed for scalability: when additional computational resources are needed, more EC2 instances can be seamlessly integrated as worker nodes. This ensures that the framework remains scalable, efficient, and cost-effective, even as demand fluctuates.

Compatibility

FunDa was tested on a range of hardware, including desktop machines and laptops of various specifications. It is designed to operate on almost any commodity hardware, making it capable of creating a local Cloud environment with minimal requirements. Additionally, FunDa can be easily installed and configured to run on cloud platforms like AWS, as demonstrated in our paper. It can also be deployed on other Cloud providers that offer sufficient machine instances and storage capacity. In contrast, many of the other systems we encountered were designed specifically for AWS, limiting their ability to function in other environments.

At its core, FunDa is built around DaskDB, but its architecture is flexible enough to integrate other distributed data processing frameworks that provide similar functionality. For example, FunDa could be adapted to work with systems like Spark or Ray, provided the necessary modifications are made. The modularity of the system allows for easy component interchange, enabling migration to different software solutions when needed. In contrast, many of the other systems we encountered were not designed with flexibility in mind, often being tightly integrated with specific platforms or frameworks, and having monolithic designs or closed sources. This lack of modularity limits their ability to incorporate alternative solutions or migrate to new technologies, further reducing their adaptability and scope of use.

Overcoming Cloud provider limitations

FunDa operates on local hardware without imposed time limits or resource restrictions, making it well-suited for stateful and long-running tasks. This capability addresses the constraints of most serverless solutions, which impose execution time limits (as discussed in Section 2.5). Consequently, FunDa is particularly advantageous for complex data analytics tasks that require sustained computation beyond the limitations of traditional serverless platforms.

Another limitation is that AWS Lambda enforces a memory cap of 3GB per function (10GB in some regions), limiting its ability to handle large tasks that cannot be efficiently decomposed into smaller chunks. FunDa overcomes this by running on on-premises clusters, where each machine can have 16 to 32GB of RAM, enabling the processing of larger workloads. Additionally, AWS Lambda restricts low-level access to networking and underlying infrastructure, whereas FunDa provides full control over its hosting environment, allowing for greater flexibility and customization.

Cost-efficiency of serverless query services

In the era of serverless Cloud services, users’ primary concern has shifted from execution performance to overall cost-efficiency. This change is largely due to the fact that serverless services relieve users from the burdens of system operations and resource management. Additionally, the presence of varying service-level agreements (SLAs) guarantees corresponding performance levels [34]. This shift in user priorities is particularly pronounced in data analytics tasks, where queries tend to take longer and are thus less sensitive to minor changes in runtime but more significantly influenced by financial costs. In such cases, it becomes more relevant for the query engine to accommodate different types of workloads and optimize query execution to reduce financial costs.

In FunDa, several optimizations have been implemented in the Service Coordinator and Driver to mitigate overhead from factors, such as service startup, loading, and preprocessing. Furthermore, at the core of query execution, DaskDB enables in situ query processing with high scalability and performance [35]. These characteristics of FunDa contribute to its overall cost-efficiency, as demonstrated in our tests with the NYC Taxi & Limousine Commission dataset [10].

Related work

In this section we describe previous work that is related to our proposed system.

Scalable data analytics

Hadoop [36], which is based on MapReduce paradigm, is a popular system for distributed data analytics. Hive [37] builds on Hadoop to provide analytical query execution functionalities. Due to the inefficiencies with the Hadoop framework, Apache Spark [38] framework was proposed. It supports both analytical query processing, and statistical and machine learning in a distributed execution environment. However, there are some limitations with Spark. First, data analytics functions are usually written as UDFs, which are restricted to use in Spark APIs and data types. This is inconvenient for users who are not experienced in Spark. Secondly, already existing Python UDFs cannot be directly imported to Spark, but it has to be re-written again using their own data types. This is not ideal for a situation when a user already has thousands of lines of existing UDF code that needs to be used as is. DaskDB [5] was introduced to address the issues with Spark. For a given SQL query, DaskDB emits Python code comprised of Dask operators, which are then submitted to Dask Scheduler for execution. Since DaskDB emits Python code, existing Python UDFs can be integrated with it easily without much effort.

Serverless cloud data processing

The topic of serverless cloud computing has been an active area of research in recent years. However, only a few projects have explored how to develop serverless data analytics and/or query processing. PyWren [13] is one of the earliest systems to leverage serverless architecture with stateless functions as the abstraction for data analytics. It exposes a Python based map API on top of AWS Lambda. PyWren supports three computation patterns, namely, Map, Map + monolithic Reduce, and MapReduce with a remote storage (e.g. S3) for data shuffling. Unlike PyWren, Flint [3] targets general purpose data analytics with Apache Spark and utilizes AWS Lambda to enable serverless data analysis. It requires all the input data to reside in an S3 bucket. The output is also written to another S3 bucket. Flint utilizes most of the existing Spark components, while providing a serveless implementation of the backend-scheduler.

Starling [14] and Lambada [39] are both SQL query execution frameworks using AWS Lambda functions. Starling uses several techniques to support serverless query processing. It uses Amazon S3 to shuffle data. Its coordinator generates C++ source code for an input query, compiles it into a executable that is packaged, compressed and uploaded to AWS Lambda. It supports optimization of a query for cost or latency by adjusting the number of function invocations at each task stage. Lambada also employs several solutions to address the challenges of serverless distributed query execution. For fast batch-start of many serverless workers it uses a tree-based invocation. Besides, it proposes a purely serverless exchange operator, which communicates through external storage based on S3.

Pixels-Turbo [15] is a distributed query engine that offers a hybrid cloud approach where elasticity can be provided to a system instantiated on long-running VMs by leveraging AWS Lambda function. The long-running VMs handle stable workloads, whereas workload spikes and unpredictable workload changes are gracefully handled by AWS Lambda. MetaQ [4] proposed the idea of ephemeral per-query engines (EPQE) on serverless architectures using AWS Lambda, where given a query, a query engine is dynamically instantiated. All of the above-mentioned systems leveraged AWS Lambda functions.

The goal of FunDa is to offer a serverless in situ data analytics framework over DaskDB that can be deployed both in a commercial cloud and on-premises private cloud. Unlike serverless query processing systems Starling, Lambada and Pixels-Turbo, our system aims to support both query processing and analytical workloads involving statistical/ML analysis. Moreover, these systems use custom-built execution engines. To that end, FunDa shares a similar vision as the system recently proposed by Wawrzoniak et al. [40]: to support an existing unmodified data analytics engine over a serverless platform. However their approach, like previous approaches, is based on AWS Lambda serverless framework. We subscribe to the idea [15] that using an existing commercial serverless platform such as AWS Lambda, comes at the expense of relinquishing the control over the hardware and execution environment to the cloud service provider. FunDa is based on the serverless framework Fn, which can be deployed on-premises and potentially adopted by commercial cloud providers.

Conclusion and future work

We have presented FunDa, our serverless data analytics framework for unified data analytics and in situ query processing. FunDa adopts a serverless computing system architecture that is aimed at optimizing performance and minimizing resource utilization, while offering greater autonomy and flexibility. We particularly focused on addressing the cold start issues and have incorporated a number of optimizations within FunDa that significantly reduces the cold start latency over our previous preliminary system Fun${\text{Da}}_{\mathit{prelim}}$. We have conducted a systematic experimental evaluation of FunDa with several benchmarks in both on-premises and AWS Cloud setups, which demonstrate that it is a viable option for users operating in serverless environments requiring more control over execution environments.

Furthermore, FunDa provides an open-source experimental platform for advancing serverless data analytics research-a field still in its nascent stages. Beyond ongoing experimentation with system components, operating systems, and tooling, we outline three key directions for future work:

• High-performance storage integration: A key priority is integrating high-performance storage solutions such as Apache Crail. Given FunDa’s focus on distributed, large-scale data processing, optimizing the storage layer is essential. Incorporating Crail (or similar low-latency storage systems) could address I/O bottlenecks inherent in traditional storage methods, enhancing FunDa’s performance for data-intensive workloads such as distributed machine learning pipelines or real-time stream processing.

• Python Just-In-Time (JIT) compilation: Future work will explore leveraging Python’s new JIT compiler (introduced in version 3.13), which dynamically compiles Python bytecode into optimized machine code at runtime. Unlike traditional interpretation, this approach enables context-specific optimizations tailored to the execution environment. Integrating this capability into FunDa could accelerate data transformation, model training, and real-time analytics tasks-particularly those involving computationally intensive loops or mathematical operations. Preliminary benchmarks suggest potential execution time improvements of 5-10% for such workloads.

• Optimized private Cloud execution: Finally, we aim to enhance FunDa’s efficiency in private and hybrid Cloud environments. For deployments on platforms like AWS, we propose exploring S3-specific optimizations such as data partitioning (organizing data into smaller, logically grouped segments to reduce access latency) and adaptive caching mechanisms (minimizing redundant data retrieval). These strategies could significantly improve performance for latency-sensitive workflows while maintaining cost-effectiveness.

Author contributions

All the co-authors contributed in this paper.

Funding

This research was supported in part by NSERC Discovery Grant RGPIN-2016-03787 (Ray) and NBIF Grant (Ray). Elyes was supported by a Globalink Research Internship during his stay at the University of New Brunswick.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

Not applicable.

Competing interests

The authors declare no competing interests.