How CSIRO’s Serverless Beacon Searches Genomic Data in 1.52 Seconds—Without Duplicating It

\n

A Serverless Query Innovation That Finds Genomic Data Without Moving It

If you need just 10,000 base pairs from a 1.1 GB genome file, why copy the entire file and load it into a central database?

Traditional genomic data processing usually works like this: VCF (Variant Call Format) files held by different institutions or research centers are moved to a central location, loaded into a database, and indexed. Researchers can then search by combining criteria such as a specific variant, disease, location, or age.

The problem is that genomic data is anything but small. Copying, transforming, and loading files takes time and money. And bringing sensitive personal genomic data together in one place also raises concerns about privacy and data governance.

CSIRO’s Serverless Beacon (sBeacon) offers a different answer:

Instead of moving data to the search system, the search system goes to where the data is and reads only what it needs.

Read Only the Required Region, Not the Entire File

sBeacon keeps the original VCF files in Amazon S3. It does not move all the genomic data into a separate database. Instead, when a user submits a query specifying variant locations and criteria, the system first narrows down the datasets relevant to that query.

For example, consider the following question:

Among samples from people living in Australia who have a particular disease, is variant A present at a specific location on chromosome 1?

sBeacon does not immediately read every VCF file. It first uses Athena and DynamoDB to identify candidate datasets that match metadata criteria such as disease, age, and location. Only then does it read the relevant chromosome and coordinate range from those datasets’ VCF files.

The key is the HTTP byte-range request. Rather than downloading the entire 1.1 GB file into the Lambda execution environment, this method fetches only the bytes in S3 associated with the requested genomic coordinates.

In other words, if a search targets a 10,000-base-pair region, the system reads only the data needed for that range, as far as possible. Compared with moving entire large files, this can significantly reduce network traffic, temporary storage needs, and function execution time.

Why Serverless Works for Genomic Searches

In this architecture, Serverless is more than a way to reduce server operations. It is an architectural choice that avoids loading massive datasets into always-on clusters, performing computation and accessing files only when a query arrives.

In sBeacon, AWS Lambda executes each query. Lambda selects candidate datasets based on metadata, uses bcftools and htslib to look up only the required regions of the VCF files, and aggregates the results. When no one is querying, there is no need to run a dedicated computing cluster for genomic searches.

This approach is especially useful for:

• Research projects that do not have a consistently high volume of queries
• Environments where multiple institutions need to retain their own copies of the original data
• Cases where replicating entire large files to a central repository is difficult
• Workloads that need to quickly check a specific variant or a limited genomic region

In an evaluation published by CSIRO, researchers used chromosome 1 data from the 1000 Genomes Project: 2,504 samples and a 1.1 GB compressed VCF file. Queries covering 10,000 base pairs had an average latency of 1.52 seconds, with estimated computing costs of about $0.00013 per query.

These figures, of course, should not be taken as guaranteed performance for every genomic query. Results will vary depending on the number of datasets, the quality of file indexes, request concurrency, the regional placement of S3 and Lambda, and the scope of the query. Even so, the evaluation makes one thing clear:

Large genomic files do not have to be moved to a central database in order to be searched.

Design for Data Access, Not Data Movement

The real value of sBeacon lies less in Lambda itself than in minimizing data movement. The original genomic files stay in S3, while only the metadata needed for search is managed separately. Information such as disease names or phenotypes—which may be recorded differently across institutions—can be semantically normalized through Ontoserver.

As a result, researchers can explore multiple datasets through the standardized GA4GH Beacon API, while data providers avoid unnecessarily duplicating their original files.

This does not mean sBeacon replaces every kind of genomic analysis. Data warehouses or batch analytics platforms may be better suited to complex cohort analyses, large-scale statistical computations, and repeated multidimensional aggregations. sBeacon excels at discovery across large genomic datasets and at answering targeted variant queries.

Ultimately, sBeacon is less a technology for “turning genomes into a serverless database” than a serverless data access layer that reads only what it needs, right where the genomic data resides.

With Serverless, the Data Stays as It Is—and You Search Only What You Need

sBeacon’s performance isn’t determined by AWS Lambda alone. The key is a two-step design: first select only the datasets likely to contain what you’re looking for, then read only the genomic regions you need within those datasets.

Large-scale genomic data is typically stored in VCF files, each of which can range from hundreds of megabytes to several gigabytes. Downloading an entire file for every query—or loading everything into a centralized database—can quickly degrade search performance and drive up costs. sBeacon sidesteps this problem with a Serverless architecture.

First, narrow down the candidates with metadata

When a user asks a question such as, “Is variant A on chromosome 1 found in people living in Australia who have a particular disease?”, the system doesn’t immediately read every VCF file.

Instead, it uses Athena and DynamoDB to search phenotype metadata such as disease, age, location, and dataset attributes. The goal is simple:

Keep only the candidate datasets worth checking against the actual genomic files.

For example, even if there are 100 genomic datasets, if only 3 match the criteria, the variant search that follows runs against just those 3 files. By significantly narrowing the search before accessing large files, this approach avoids unnecessary S3 requests and reduces Lambda execution time.

Read only the bytes you need—not the entire VCF

Once the candidate datasets have been selected, the Variant Querying Module uses bcftools and htslib to query specific chromosome positions in the relevant VCF files. VCF indexes and HTTP byte-range requests play a crucial role here.

A typical file download brings the entire 1.1 GB VCF into the application’s execution environment. By contrast, sBeacon uses the index to determine where the desired chromosome and position are located in the file, then requests only the required byte range from S3.

In other words, instead of transferring a multi-gigabyte file to search a 10,000-base-pair region, it fetches only a small portion of the file that contains that region.

This design delivers several benefits:

• Reduces the volume of data transferred to Lambda
• Cuts function execution time and temporary storage requirements
• Eliminates the need for a centralized copy of the original VCF files
• Keeps query scope limited even as datasets grow
• Enables a usage-based cost structure suited to occasional research queries

The heart of Serverless lies in how data is accessed—not just how code runs

If you think of Serverless simply as “Lambda functions without servers to manage,” it’s easy to miss what makes sBeacon powerful. What matters here is not the compute environment, but the data access strategy:

1. Search metadata to select the relevant files
2. Use indexes to locate the data within those files
3. Read only the required bytes remotely from object storage
4. Have Lambda aggregate the results into the standard Beacon format

With this approach, the data stays in its original form in S3, and compute runs only when a query arrives. In other words, instead of fully migrating large genomic files into a database, sBeacon acts as a Serverless access layer that makes files searchable as data sources.

Of course, this approach isn’t suited to every type of analysis. If you need to repeatedly run complex statistical analyses across thousands of datasets or perform large-scale cohort aggregation, a dedicated analytics platform may be more efficient. But for checking whether a specific variant is present and quickly finding datasets that match a set of criteria, this two-step design offers a compelling balance of cost and performance.

Serverless Performance: Querying a 1.1 GB File in 1.52 Seconds

How long would it take to download and search a compressed VCF file weighing 1.1 GB every time? Factor in network transfer, decompression, and file scanning, and a short API response time may seem unlikely. Yet in CSIRO’s public sBeacon evaluation, queries across a 10,000-base-pair region of a 1.1 GB VCF containing 2,504 samples took an average of 1.52 seconds. The estimated compute cost was about $0.00013 per query.

Even more striking, latency remained largely steady as the number of matching variants grew from 4 to 400. The key is not “how many results are there?” but “where in the file do we need to read?”

Read only the bytes you need—not the whole file

sBeacon’s Serverless architecture doesn’t copy the entire VCF file to AWS Lambda. Instead, bcftools and htslib use the VCF index to locate the desired chromosome and coordinate range, then send an HTTP byte-range request to Amazon S3.

For example, if the query targets a specific 10,000 bp region on chromosome 1, the system fetches only the portion of the file containing that region—not the entire 1.1 GB. This approach reduces several costs at once:

• Network traffic required to download large files
• Lambda execution time and temporary storage usage
• CPU costs from scanning the entire file
• Operational overhead from replicating the source data into a separate database

In other words, the 1.1 GB file size matters less than how narrowly and precisely the target region can be identified.

Why latency didn’t surge as the number of matches grew

It’s easy to assume that processing time will increase as variant matches grow from 4 to 400. But in this evaluation, response times did not deteriorate significantly. That’s because the system doesn’t search the entire VCF, finding results one by one.

The process works roughly as follows:

1. Athena and DynamoDB first narrow down candidate datasets using criteria such as disease, age, and location.
2. Lambda runs coordinate-based variant queries only against those candidate datasets.
3. Using the VCF index, it reads only the required file segments from S3.
4. It checks for variants within those small segments and returns the results in Beacon format.

So when the number of results increases within the same coordinate range, the main performance costs are less about the number of results and more about index lookup, partial reads from S3, function initialization, and result aggregation. Even when the result count differs between 4 and 400, latency may not change much if the file region that needs to be read is about the same.

Why these numbers shouldn’t be overgeneralized

An average of 1.52 seconds and a cost of $0.00013 per query are impressive, but they are not guaranteed figures for every genomic query. These results were measured using chromosome 1 data from the 1000 Genomes Project, with 2,504 samples and a 10,000 bp query range.

Actual performance may vary depending on:

• VCF file size and index quality
• Query range and the number of datasets queried simultaneously
• Lambda cold starts and concurrency settings
• The regions where S3 and the execution environment are located
• Athena metadata query time
• The size of the results that need to be returned

Still, this evaluation points to an important opportunity for Serverless approaches to handling large-scale scientific data. You don’t have to move massive files into a database in their entirety: combining precise metadata filtering with coordinate-based partial reads can help achieve both fast response times and low costs.

The Costs and Boundaries Behind Serverless’s Strengths

Not having to operate servers yourself does not make bottlenecks and risks disappear. sBeacon’s serverless architecture reduces the cost of always-on clusters by running only when needed. But the picture changes when concurrent queries surge or repeated queries are used to infer sensitive genomic information.

The key point is not that “there are no servers,” but that what you need to operate shifts from servers to service limits, access policies, and data request patterns.

Concurrency Is Not Another Name for Automatic Scaling

sBeacon combines managed services such as Lambda, S3, Athena, and DynamoDB. Each component can scale elastically, but that does not mean the system as a whole can scale without limit.

When many users search multiple datasets at once, bottlenecks can arise:

• Lambda concurrency limits: If many queries arrive at once, functions may be delayed or throttled.
• Increased S3 requests: Even byte-range requests that read only part of a VCF can cause latency and costs to rise if request volumes surge.
• Athena query concurrency: If the metadata search stage that finds candidate datasets based on phenotype criteria slows down, subsequent VCF lookups will be delayed as well.
• DynamoDB throughput: A surge in dataset metadata and index lookups can lead to read-throughput constraints and throttling.
• Result aggregation overhead: The process of combining results from multiple institutions or many datasets in Lambda can also increase response times.

In other words, a serverless architecture eliminates the need to plan server capacity, but replaces it with a need to design around service quotas and request flows. Operations teams need to consider Lambda concurrency limits, retry policies, backoff, caching, request queuing, and per-user rate limits together.

Partial Reads Are Not Always the Cheapest Option

One of sBeacon’s key advantages is that it reads only the required chromosomal region instead of downloading an entire VCF file. But byte-range requests are not the most cost-effective option in every situation.

Partial reads work well when users occasionally search very narrow regions. But they can become less efficient when:

• Broad genomic regions are queried repeatedly
• A very large number of small requests are made for different coordinates
• VCF indexes are inaccurate or poorly optimized
• S3 and Lambda are in different regions, generating data transfer costs
• Popular queries against the same dataset are repeated

In these cases, you need to consider not just the cost per request, but also the total number of requests, data transferred, function execution time, and index access patterns. It may make more sense to cache frequently queried regions or convert the data into a separate structure for analysis.

Rather than viewing sBeacon as a replacement for large-scale analytics engines, it is more accurate to see it as an access layer optimized for dataset discovery and limited variant lookups.

With Sensitive Genomic Data, Even “Whether It Exists” Is Information

The most important boundary is not performance, but privacy. A Beacon API often returns a limited response to the question, “Does this variant exist?” But a response that is just a Boolean value cannot automatically be assumed to be safe.

By combining multiple queries, an attacker may be able to infer:

• Whether someone with a particular rare variant is included in a dataset
• Whether a disease group or regional cohort is represented
• Whether a particular individual is included by making small changes to query criteria
• Whether a particular variant is associated with a disease in a small population

The risk of re-identification grows when rare diseases, rare variants, and small cohorts intersect. The fact that the original VCF remains in S3 and is not replicated to a central database does not, by itself, resolve privacy risks.

Safeguards Needed in Production

To deploy sBeacon in real research settings or federated environments across institutions, clear controls are needed at both the API front end and the data access layer.

| Control | Why it is needed | |---|---| | User- and institution-level authentication | To prevent unauthorized users from running sensitive variant queries | | Granular access controls | To separate access by dataset, disease, region, and research purpose | | Query rate limits | To mitigate automated repeated queries and guessing attacks | | Minimum cohort thresholds | To reduce re-identification risk by restricting responses for groups that are too small | | Response-level controls | To distinguish between presence/absence, counts, and detailed results based on authorization | | Audit logs | To track who queried what, under which conditions, and how often | | Anomaly detection | To identify attack patterns such as searching for rare variants or repeatedly changing query criteria | | Data location policies | To comply with data sovereignty and transfer requirements across regions, institutions, and countries |

The important thing is not to treat security as a feature that can be added later. For genomic data, the query interface itself is a security boundary. API design, response policies, log analysis, and access approval procedures must therefore be built into the architecture.

The Real Operational Challenges of Serverless

sBeacon shows how serverless can be a powerful way to handle large-scale scientific data. Narrowing down targets with metadata and then reading only the required VCF regions—without moving the original data—is an effective way to reduce costs and data transfer.

But success does not come from deploying Lambda. You need to be able to answer questions such as:

• If concurrent queries increase by 100 times over normal levels, which layer will slow down first?
• How will you detect repeated queries designed to infer whether a specific individual is present?
• Which queries should be cached or rate-limited as request costs accumulate?
• When complex analysis becomes necessary, how will you separate the serverless lookup layer from the analytics platform?

Ultimately, sBeacon’s value does not lie in “getting rid of servers.” It lies in minimizing data movement while making performance limits and privacy risks visible in a manageable form. Serverless is not a technology that eliminates operations; it shifts the focus of operations to more deliberate design and governance.

The Next Stage of Platforms That Don’t Move Data—Enabled by Serverless

The significance of sBeacon is not simply that it “made the entire genomic analysis workflow Serverless.” The more important shift is that it demonstrates a concrete approach to data access: keep the original data in each institution’s own storage, and connect only through standard APIs and the computations needed.

Traditional data platforms have typically brought data together in one place for analysis. VCF files held by individual institutions are copied to a central database or data warehouse, indexed, and then queried. This approach can be efficient for repeated analysis, but with large-scale genomic data, the costs of replication, time spent moving data, and privacy-governance challenges can escalate quickly.

sBeacon reverses that sequence.

Instead of moving data to the analysis environment, send the query to where the data lives.

Keep the Originals; Connect Only the Search Layer

In sBeacon, the original VCF files remain in Amazon S3. The system does not fully convert them into separate database tables. Instead, it manages metadata—such as each dataset’s disease area, age range, region, and research purpose—separately, and first identifies only the candidate datasets that match the user’s criteria.

AWS Lambda then reads only the relevant genomic coordinate ranges from the required VCF files. By using HTTP byte-range requests and VCF indexes, it avoids the need to download entire files that may be several gigabytes in size.

This structure redefines the role of a data platform:

| Traditional centralized model | sBeacon-style access model | |---|---| | Replicate data into a central analytics system | Keep original data in each institution’s storage | | Load entire files or large tables | Index only the metadata needed | | The analytics environment is at the center of the data | Standard APIs are at the center of data access | | Analyze data after moving it | Filter by criteria, then read only the required ranges remotely | | A central operations team manages infrastructure and data | Connect institutions while preserving each one’s data ownership |

This is more than a change in how data is stored. It is an architectural shift that separates data ownership from the location of computation.

Serverless Becomes More Powerful in Federated Data Environments

Genomic data is especially difficult to centralize. Hospitals, research institutes, government agencies, and pharmaceutical companies all have different data-access policies and legal responsibilities. The moment data is brought together in one place, the challenge extends beyond technology to contracts, regulation, and privacy.

sBeacon addresses this challenge through the GA4GH Beacon standard API. Each institution retains its data while using a standardized query interface to provide external systems with limited information about “what data exists under these criteria.”

Here, Serverless is not merely a way to reduce the number of servers to operate. It is an execution model that makes federated architectures like this practical:

• Computing resources can be invoked only when a query arrives.
• Workloads can scale flexibly to match each institution’s data volume and query traffic.
• Standard APIs can be provided without operating always-on clusters.
• Metadata search, authorization checks, file-range reads, and result aggregation can be handled separately.
• Discovery across institutions can be supported without replicating data.

In other words, Serverless functions not as a platform that directly stores massive datasets, but as a lightweight execution layer that connects distributed data.

Closer to a “Data Access Layer” Than an “Analytics Platform”

That said, sBeacon should not be interpreted as the answer to every analytics problem. Complex cohort analyses, large-scale statistical modeling, iterative machine-learning training, and multidimensional aggregation may be better suited to a separate data warehouse or distributed analytics engine.

What sBeacon does well is clear:

• Find datasets that contain a specific variant
• Explore samples by disease, region, or age criteria
• Improve the discoverability of data across institutions
• Quickly inspect only the genomic coordinates needed
• Provide standardized queries without moving data

This structure is therefore more accurately viewed not as a replacement for a central data lake, but as an intelligent data access layer positioned in front of or above it. Users can first use an interface such as sBeacon to confirm that data exists and is accessible, then move to a deeper analytics environment only if they have the necessary permissions.

A Pattern That Can Be Applied to Other Industries

This pattern is not limited to genomic data. The core idea is to keep large source datasets in Object Storage and combine metadata filtering with partial reads.

For example, it can be applied to:

• Services that retrieve only image tiles from specific locations and capture dates in satellite imagery
• Systems that search medical-image repositories for data matching specific conditions and examination criteria
• Platforms that extract only records for particular time periods and event types from security logs
• Research infrastructure that searches scientific experimental data by instrument, conditions, and measurement range
• Services that provide unified search across document repositories distributed among multiple companies or institutions

In these environments, the goal is not to “put all the data into one platform.” It is to make data accessible wherever it resides, through consistent policies, standard interfaces, and computation limited to the minimum necessary scope.

sBeacon points in exactly this direction. The next stage of Serverless goes beyond eliminating servers. It is about evolving into a data platform that can safely find and connect only what is needed, when it is needed—without moving data unnecessarily.