What Is Edge AI SLAFlow? Exploring the Latest Runtime Technology for High Quality and Low Latency

\n

Edge AI: What Should You Give Up with 10 Seconds Left?

A field camera has detected a subtle sign of equipment trouble. The AI has already analyzed the image. Now it needs to search relevant maintenance records and manuals, then generate an answer explaining the cause. Finally, it must check the answer’s format and reasoning—and may need to revise it one more time.

The problem is, there’s no time.

There are just 10 seconds until the request’s deadline. What should the Edge AI system cut back?

• Reduce the number of supporting search results from eight to three?
• Limit the generated answer to 256 tokens instead of 768?
• Skip the final regeneration after validation?
• Or preserve quality and miss the deadline?

This isn’t simply a question of model performance. In real-world services, an answer that’s good enough and arrives on time may be more valuable than the “best” answer.

The challenge is even sharper in Edge AI environments. Inference runs on site to avoid the round-trip latency of sending data to the cloud, but CPU, memory, and accelerator resources are limited. When camera analysis, knowledge retrieval, LLM generation, and result validation happen in sequence, small delays at each step can quickly add up.

For example, broadening the search may give the answer a stronger foundation—but it also takes longer to retrieve results. Increasing the token budget may allow for a more detailed explanation, but it also increases generation time. Running one more refinement after validation could reduce errors, but that extra revision could cause the entire system to miss its SLA.

This is exactly where SLAFlow comes in. It doesn’t prescribe “always generate a long answer” or “always use the smallest configuration.” Instead, for every request, it considers the time remaining, current CPU and memory conditions, queue length, input uncertainty, and latency profiles for each stage. Then it selects the configuration with the highest expected quality from among the options that can finish before the deadline.

The key is that SLAFlow reassesses the decision during execution. After generating and validating an initial answer, the system recalculates whether another refinement is really worth it:

Can the revision be completed in the time remaining?
Is it likely to improve quality meaningfully?

If the answer to either question is no, the AI decisively skips the final revision. Instead, it returns the most recent result that has already passed validation.

At first glance, this may look like “sacrificing quality.” But delivering a validated result within the time available is far more practical than missing the deadline and delivering nothing. Edge AI’s competitive edge doesn’t come from running more inference at all costs. It comes from knowing exactly what to carry through to completion—and when to stop—within the time available.

Edge AI’s Three Dials: Retrieval Volume, Tokens, and Refinement Passes

Increasing the number of retrieved sources from 3 to 8, raising the generation token budget from 256 to 768, and refining an answer two additional times can improve quality. But with the richest configuration, p95 service time reaches 370 seconds.

So, is doing more inference always the better choice for an Edge AI service? SLAFlow tackles this question with three adjustable dials.

| Dial | What it controls | Evaluation range | Expected benefit | Cost | |---|---|---:|---|---| | Retrieval volume | Retrieval depth (k) | 3, 5, 8 | More supporting evidence | Increased retrieval latency | | Generation volume | Token budget (T) | 256, 512, 768 | Longer explanations and more complex responses | Increased LLM generation time | | Refinement passes | Refinement depth (R_{\max}) | 0, 1, 2 | Error correction and improved formatting | Increased validation and regeneration time |

Retrieval Volume: Does More Evidence Always Mean Greater Accuracy?

Retrieval depth (k) is the maximum number of evidence items retrieved before generating a response. For example, with (k=3), the system uses only the top three pieces of evidence; with (k=8), it can incorporate up to eight documents or knowledge items into the response.

More search results give the model richer context to draw on. In a medical support service, it can consult more relevant cases and guidelines; in an industrial diagnostic service, it can compare a larger number of past incident records.

But increasing retrieval volume comes with two costs:

• Retrieval itself takes longer.
• The model has to read and synthesize more evidence, increasing the burden on subsequent generation.

In Edge AI environments in particular, local storage, limited memory, and low-power CPUs or NPUs can become bottlenecks. So the principle that “more evidence is better” holds only when there is enough time before the deadline.

Generation Tokens: A Longer Answer Isn’t Necessarily a Better One

Generation token budget (T) is the maximum number of tokens allowed for the initial response. SLAFlow evaluates budgets of 256, 512, and 768 tokens.

A larger token budget lets the model include more detailed explanations of its evidence, explore more conditional branches, and produce a more complete structure. But in generative models, token count is generally directly linked to increased latency.

In the experiment, raising the token budget from 256 to 768 increased p95 generation latency as follows:

| Generation token budget | p95 generation latency | |---:|---:| | 256 | 74.7 seconds | | 768 | 223.2 seconds |

In other words, tripling the maximum token count doesn’t simply make the answer longer. It consumes a significant share of the overall SLA headroom for an Edge AI service.

For tasks where immediacy matters—such as short field alerts, equipment fault notifications, or camera event summaries—256 tokens may be the more reasonable choice. By contrast, a larger budget may be worthwhile when tasks such as drafting a lengthy report or supporting a complex decision can tolerate delays of tens of seconds or more.

Refinement Passes: Edits Improve Quality, but the Returns Diminish

Refinement depth (R_{\max}) is the maximum number of times a response can be revised after validation. With a value of 0, the system generates only a draft; with a value of 1 or 2, it revises the response once or twice based on validation results.

The process can be understood as follows:

1. Generate an initial response.
2. Check its format, logic, and whether any required information is missing.
3. If there are problems or room for improvement, revise the response.
4. If enough time remains, validate it again and repeat if needed.

Refinement is useful, but it cannot be applied indiscriminately to every request. In the paper’s experiments, one refinement pass had a p95 latency of 102.3 seconds. What’s more, the quality gains from refinement diminished with each pass.

| Stage | Mean sample-wise F1 | |---|---:| | Perception only | 0.600 | | After initial generation | 0.719 | | After one refinement pass | 0.733 | | After two refinement passes | 0.753 |

Initial generation delivers meaningful improvements in many cases, but subsequent refinement can offer limited gains relative to its additional cost. In fact, initial generation improved 35% of cases, while the first refinement improved only 12%, and the second just 10%.

The Three Dials Don’t Move Independently

SLAFlow’s key insight is that retrieval volume, token budget, and refinement passes are not optimized independently. They share the budget for a single workflow.

The smallest configuration is:

[ (k,T,R_{\max})=(3,256,0) ]

This means retrieving three pieces of evidence, generating up to 256 tokens, and doing no refinement. It achieved a Macro-F1 of 0.726, with a p95 service time of about 80 seconds.

At the other extreme, the largest configuration is:

[ (k,T,R_{\max})=(8,768,2) ]

It retrieves eight pieces of evidence, generates a response of up to 768 tokens, and revises it up to twice if needed. It delivered the highest quality, but its p95 service time reached 370 seconds.

The middle configuration, ((5,512,1)), achieved a Macro-F1 of 0.792 and a p95 service time of 226 seconds. A good Edge AI policy, then, isn’t about always choosing the most powerful setup. It’s about finding the combination that delivers the highest quality within each request’s deadline.

More inference can potentially produce better results.
But when there’s no time left, more inference may mean an SLA violation—not a better answer.

That’s why SLAFlow checks the time remaining, system load, input uncertainty, and estimated cost of each stage for every request. Then it decides again: “Should I search further, increase the token budget, or revise this answer one more time?” This is the essence of Edge AI runtime orchestration—going beyond simply choosing a model.

Runtime Replanning for Edge AI: Reconsidering the Next Step with the Time Left

There’s no guarantee that an execution plan will remain valid all the way through. A generation step may take longer than expected, or CPU utilization and queue lengths may spike as other workloads compete for resources. For Edge AI services running with limited resources, even a few seconds of error can result in an SLA violation.

SLAFlow doesn’t stop at choosing settings just once, when a request begins. After initial generation and validation are complete, it asks again: “Is another refinement step worth doing now?”

Reassessing the situation based on the remaining deadline

At the point of reassessment, SLAFlow takes the following information into account:

• The current time and the request’s remaining deadline
• CPU and memory usage over the last 500 ms
• Queue length and recent execution delays at each stage
• The estimated execution cost of the next refinement step
• The expected sample-wise F1 improvement from refinement
• Pre-built performance profiles for each platform, model, and knowledge base

The principle is simple. Even if there seemed to be enough time at the outset, a longer-than-expected generation step may mean the plan needs to be scaled back. Conversely, if generation finishes faster than expected and resources are available, SLAFlow can consider additional refinement to improve quality.

“There’s an error, so generate again” isn’t enough

In a typical workflow, a problem found during validation might trigger a new generation attempt. But in an Edge AI environment, that isn’t always the right choice. If regeneration pushes the request past its deadline, trying to produce a better answer may mean failing to deliver any result on time.

Before performing refinement, SLAFlow checks that both of the following conditions are met:

1. The estimated refinement time fits within the remaining deadline
2. The expected quality improvement exceeds a minimum threshold

In the paper, refinement proceeds only when the expected sample-wise F1 improvement is at least 0.005. In other words, if the quality gain is negligible, SLAFlow won’t keep regenerating at the cost of CPU time and the remaining deadline.

The decision can be expressed as follows:

[ \text{Estimated Refinement Cost} \leq

\text{Remaining Deadline}

\text{Safety Margin} ]

The quality-gain condition must also be met before the next step is executed. That’s what makes SLAFlow runtime orchestration that responds to how execution is unfolding—not merely a token-limit policy.

A design that adapts to the gap between estimates and reality

Suppose the initial plan calls for five retrievals, 512 tokens of generation, and one refinement. At first, there may appear to be enough time for one round of revision. But if generation takes longer than expected and CPU load is high, the estimated cost of refinement may exceed the time remaining.

In that case, SLAFlow skips refinement and returns the current result. Conversely, if generation finishes quickly and validation indicates room for improvement, SLAFlow can add a refinement step while time remains.

| Situation during execution | SLAFlow’s decision | |---|---| | Generation takes longer than expected | Scale back or stop refinement | | CPU or memory load increases | Switch to a more conservative configuration | | Enough time remains and there is strong potential for improvement | Proceed with refinement | | Expected quality gain is low | Return the current result | | Refinement fails or times out | Return the previously validated result |

last_valid matters more than the latest output

Runtime replanning also accounts for failure safety. SLAFlow keeps the most recently generated result, candidate, separate from the latest result that passed validation, last_valid.

That way, if refinement fails or times out, SLAFlow can still return an earlier valid result. This is more reliable than blindly returning only the latest output—a particularly important design choice for Edge AI services operating with limited memory and fluctuating workloads.

Ultimately, what sets SLAFlow apart isn’t “choosing the best plan at the start.” It’s reconsidering what to do next as circumstances change during execution, based on the time remaining and the expected quality gain.

Quality Improved, but the Deadline Was Missed: The Paradox in the Edge AI Numbers

SLAFlow’s results are clearly impressive. Under a Nominal workload and tight SLA conditions, SLAFlow achieved a Deadline-Weighted Quality (DWQ) score of 0.668 and an SLA violation rate of 2.7%. Static-Small, a fixed, lightweight configuration, achieved a 0% violation rate, but its DWQ was only 0.610.

Looking at these conditions alone, the conclusion seems simple: SLAFlow, which adjusts the amount of retrieval, the number of generated tokens, and the number of refinement passes for each request, strikes the best balance between quality and latency.

But the story changes under a Bursty workload, where traffic surges suddenly.

| Policy | DWQ | SLA violation rate | |---|---:|---:| | SLAFlow | 0.604 | 11.4% | | Static-Small | 0.601 | 1.5% |

SLAFlow’s DWQ was 0.003 higher than Static-Small’s. But in exchange for that small quality gain, its SLA violation rate rose from 1.5% to 11.4%—about 7.6 times higher.

These numbers raise a crucial question for Edge AI operations:

Is it a success to gain 0.003 in quality if more than one in ten requests misses its deadline?

More important than average quality is whether the result arrives on time

DWQ treats the quality of a request that misses its deadline as zero. In other words, it measures not just the accuracy of the output, but also whether the user or system receives the result within the time it needs.

Even so, the high violation rate under a bursty workload is tied to a structural dilemma in adaptive policies. SLAFlow selects the highest-quality configuration it can within the available constraints. For example, it might retrieve more evidence, generate a longer response, or perform one additional refinement pass.

The problem is that when the load surges, several variables change quickly:

• The queue grows in a short period of time.
• Competition for CPU and memory increases.
• The latency of each stage exceeds what profiling predicted.
• Delays in earlier requests eat into the time remaining before later requests’ deadlines.

As a result, a high-quality configuration that “looked feasible at the time” can miss its deadline during actual execution. Generative workflows are especially vulnerable because token generation and refinement can take a long time, so even a small prediction error can lead to a large difference in total service time.

When a fixed, lightweight configuration is better

Static-Small is a conservative configuration, with a retrieval depth of 3, a generation token budget of 256, and no refinement passes. It does not aim for the highest possible quality, but its execution time is relatively predictable and its resource usage is limited.

When the load surges, that simplicity becomes an advantage. Each request uses fewer resources and finishes quickly, helping prevent the queue from growing in a cascading fashion. As a result, even with slightly lower quality, Static-Small kept the SLA violation rate at 1.5%.

This does not mean SLAFlow has failed. Rather, it shows why the objective function must be made explicit when designing Edge AI policies.

• For services where answer quality matters, such as customer support or field-assistance tools, it may be reasonable to accept a small number of missed deadlines in exchange for a higher DWQ.
• For services where response timing itself is tied to safety, such as industrial control, safety monitoring, robotics, and autonomous driving, the violation-rate ceiling must be enforced strictly, even at the cost of some quality.

Design around a “violation-rate budget,” not “maximum quality”

In real-world product environments, constraints like the following are often more practical than a policy that simply maximizes quality:

[ \text{maximize quality} \quad \text{subject to} \quad \text{SLA violation rate} \leq \epsilon ]

Here, (\epsilon) is the violation rate the service can tolerate. For example, if a safety-monitoring system requires a violation rate of 1% or lower, SLAFlow must first determine whether it can stay within that limit before selecting a high-quality configuration.

So under a bursty workload, what is needed may not be more aggressive quality optimization, but more conservative control, such as:

• Immediately reduce retrieval depth and token budgets when the queue exceeds a certain level.
• Disable refinement by default during periods of congestion.
• Increase the safety margin as latency prediction errors grow.
• Clearly switch between high-quality and stability modes based on system load.
• Fall back to a safe mode, such as Static-Small, if the recent SLA violation rate exceeds a threshold.

The most interesting result from SLAFlow is not that “adaptive policies are always better than fixed policies.” Rather, it shows that adaptivity itself must operate within safety constraints. In Edge AI, the best decision is not the one that performs the most inference, but the one that repeatedly delivers the quality needed within the time available.

Edge AI’s Next Step: Optimizing the Entire Service, Not Just the Model

The real question SLAFlow raises doesn’t end with “Which model should we use?” The more practical question is:

“Given the time left to serve this request, how much of each step should we perform?”

That question shifts the unit of Edge AI optimization from a single model to the entire service workflow. Choosing between a small model and a large one is no longer enough. The system must also decide how many documents to retrieve, how many tokens to generate, and whether to allow another round of refinement after validation.

Imagine, for example, a clinical support service that must return a result within 240 seconds. The system analyzes an image, retrieves relevant evidence, has an LLM generate a result, then validates its format and reasoning—and may produce a revised answer if needed. But running both generation and refinement could push the service past its deadline.

SLAFlow does not force every request to use the same quality settings.

• If there is ample time and the input is highly uncertain, it expands the retrieval scope and token budget.
• If the queue is long or CPU utilization is high, it chooses a smaller configuration.
• Even if the initial result is valid, it skips regeneration when refinement is unlikely to improve quality enough or time is running short.
• If the final refinement fails, it preserves the previously validated last_valid result instead of blindly returning the latest output.

In other words, SLAFlow is not “a system that performs as much computation as possible to produce the highest-quality answer.” It is closer to a system that selects the best—and safest possible—result within a fixed time limit.

Service-Level Control Beyond Model Selection

Traditional Edge AI optimization has generally focused on choices such as:

• Choosing between a lightweight model and a large model
• Allocating CPU, GPU, and NPU resources
• Offloading inference between the edge and the cloud
• Adjusting batch sizes or request order

SLAFlow, by contrast, adjusts multiple steps within a single request at the same time.

| Control target | SLAFlow’s question | |---|---| | Retrieval | Should it retrieve just 3 evidence documents, or expand the search to 8? | | Generation | Should it give a concise answer in 256 tokens, or allow up to 768? | | Refinement | Is it worth generating one or two more revised answers after validation? | | Deadline | Is there enough time left to actually improve quality? |

This approach is particularly well suited to generative Edge AI services. Unlike image classification, which can finish in a single inference, RAG-based assistants and field diagnostic systems string together retrieval, generation, validation, and revision. If even one step slows down, the entire service can miss its SLA.

SLAFlow stands out because it makes decisions based not on the average latency of individual steps, but on the time remaining for the request as a whole.

Improving Quality and Meeting a Deadline Are Not the Same Thing

This approach matters because quality does not always improve linearly. In the paper’s experiments, the configuration that maximized retrieval depth, generation tokens, and refinement rounds could deliver the highest quality—but increased p95 service time to 370 seconds.

By contrast, the minimal configuration ((k=3, T=256, R=0)) was much faster, with a p95 of about 80 seconds, but delivered limited quality.

Refinement, in particular, is not an operation where more is always better. The initial generation can produce a substantial quality improvement, but the gains taper off with a second round of revision. Yet each refinement takes considerable time. When time is short, attempting another generation for a modest quality gain can push the service past its deadline—potentially causing its overall service quality to be scored as zero.

That is why SLAFlow checks two things before running refinement:

1. Can the next refinement finish within the remaining time, including the safety margin?
2. Is the expected sample-wise F1 improvement at least the minimum threshold of 0.005?

This is far more practical to operate than the simple rule “regenerate because the result is invalid.” In Edge AI environments, resources are limited and workloads fluctuate, so the attempt to produce an even better result can itself lead to service failure.

But Will It Deliver the Same Results on a Real NPU?

There is an important limitation to acknowledge. SLAFlow shows clear promise, but its current evaluation was conducted on a CPU powered by an Intel Core i5-4460. It did not use a GPU or a real embedded NPU, and the prototype was evaluated with a single FIFO worker and a batch size of 1.

That leaves several questions unanswered:

• Will profile-based predictions remain accurate on real Edge AI hardware such as Jetson, Hailo, Qualcomm Dragonwing, or Raspberry Pi?
• How will NPU compiler optimizations, memory transfers, thermal throttling, and interference from concurrent workloads affect latency profiles?
• How often will profiles need to be rebuilt when the model, runtime, or knowledge base changes?
• Will the same policy work reliably in environments where multiple edge nodes and the cloud operate together?
• In fields with strict limits on violation rates, such as safety monitoring or industrial control, can the policy be redesigned to prioritize meeting deadlines over improving quality?

Under bursty workloads in particular, SLAFlow achieved only a marginally higher DWQ than Static-Small, while its SLA violation rate was higher. This leaves a product-policy question on the table: how many deadline violations are acceptable in exchange for a modest gain in quality?

Ultimately, SLAFlow’s core value does not lie in declaring victory for a particular model or a particular piece of hardware. It shows that as Edge AI advances, the important capability is not running the largest model, but intelligently scaling the entire service up or down at every moment, within tight limits on time, power, and memory.