Why Does Modern Edge AI Require Rethinking Both Algorithms and Systems?

\n

Edge AI: The Moment AI Sends Data to the Cloud, It’s Already Too Late

The moment an autonomous vehicle needs to recognize a pedestrian at a crosswalk, the moment a camera detects an intruder, and the moment a navigation system must identify a sudden accident or traffic jam are all happening now.

Sending data to a cloud server and waiting to receive the analysis results inevitably creates delays. If network conditions are poor, the difference can be hundreds of milliseconds—and in safety-critical environments, even that brief moment is far from insignificant. From the user’s perspective, delayed responses to voice commands, interrupted camera recognition, and stuttering AR displays all quickly translate into a decline in service quality.

Edge AI emerged to solve this problem. Instead of running AI models in a distant cloud, it executes them inside devices such as smartphones, vehicles, cameras, sensors, and kiosks—or on nearby edge servers. In other words, data is analyzed and acted upon immediately where it is generated.

How Edge AI Changes the Distance to Decision-Making

The workflow of traditional cloud AI is relatively simple:

Data collection → Transmission to the cloud → Server-side analysis → Receiving the results → Executing an action

By contrast, Edge AI drastically reduces the amount of data movement:

Data collection → On-device analysis → Immediate decision-making and execution

This difference means more than simply being “a little faster.” As the location where AI makes decisions moves closer to where data is generated, the very design of real-time services begins to change.

• Autonomous driving and advanced driver-assistance systems can analyze camera and sensor data on-site to quickly recognize vehicles, pedestrians, lanes, and traffic signals.
• Intelligent security cameras can detect intrusions or unusual behavior and transmit only the necessary events, reducing response times.
• Navigation and traffic-control systems can immediately analyze road conditions and vehicle movements, then reflect them in traffic-signal control or route guidance.
• Smartphones and wearable devices can perform speech recognition, translation, photo enhancement, and health-signal analysis faster and more naturally.

More Than Speed: Privacy and Cost

Cloud transmission brings not only latency, but also privacy and operating-cost challenges. When sensitive data such as video, audio, location information, and biometric signals is continuously sent to external servers, security risks and regulatory burdens grow.

Edge AI processes raw data inside the device whenever possible. For example, even when a camera detects a person, it can transmit only the necessary result—such as “a suspected intrusion event has occurred”—instead of sending all video footage to a server. This reduces the possibility of data exposure while also lowering network bandwidth usage.

In environments where countless IoT devices and cameras transmit data every second, cloud inference costs can rise rapidly. Increasing the proportion of local inference reduces communication traffic and the burden on servers, improving overall operational efficiency.

Why Running AI at the Edge Is Difficult

Of course, running AI inside a device is not easy. Unlike cloud data centers, edge devices operate under constrained conditions.

• Limited CPU, GPU, and NPU performance
• Small memory capacity and limited memory bandwidth
• Battery and heat-management challenges
• The possibility of network disconnection
• Service requirements that demand real-time responses

That is why Edge AI’s competitiveness is not determined by model accuracy alone. Models must operate with fewer computations while maintaining the same level of accuracy, minimize memory movement, and efficiently leverage heterogeneous processors such as CPUs, GPUs, and NPUs.

This is where recent developments are gaining attention. Algorithmic optimization techniques such as quantization and pruning make AI models lighter, while system optimization accounts for the device hardware and runtime environment. The key is no longer a larger model, but a model that makes decisions as quickly and reliably as possible within a given power and time budget.

The era of AI waiting for data in the cloud is coming to an end. Edge AI—understanding and responding to data immediately at the place where it is born—is establishing itself as a core technology of next-generation computing, transforming safety, privacy, cost, and user experience all at once.

One Small Model Is Not Enough: Co-Designing Edge AI Algorithms and Systems

What if cutting a model’s parameters in half barely reduced battery consumption or processing latency? The problem may not simply be that the model is not lightweight enough. In many cases, the real bottleneck lies outside the AI model itself—in how the model and hardware exchange data and schedule computations.

The latest Edge AI research goes beyond simple model compression. The key is co-design: treating neural network architecture, quantization methods, memory access, workload distribution across CPUs, GPUs, and NPUs, and runtime scheduling as one integrated problem and designing them together.

Why a Smaller Model Does Not Necessarily Run Faster

Model compression is the starting point of Edge AI, but it does not solve every performance problem on its own. On edge devices, the following factors can cost more than the computations themselves:

• The time required to load weights and intermediate data from memory
• The process of copying tensors between the CPU, GPU, and NPU
• Delays caused by operations unsupported by hardware accelerators
• Runtime overhead from repeatedly executing small operations
• Performance degradation caused by battery and thermal constraints

For example, even if the number of model parameters is reduced, latency and power consumption may not decrease significantly if intermediate results—tensors—continue moving back and forth through memory. This is especially true in environments that process high-resolution data, such as real-time camera video, where the amount of data movement can matter more than the number of computations.

In Edge AI, the important question is not “How small is the model?” but “How little data movement and power does this model require to run on this device?”

What Does Co-Design Optimize Together?

Co-design is not a process in which AI algorithm teams and system or hardware teams optimize separately and connect their results afterward. It is an approach that designs the model and execution environment together from the beginning, taking the device’s characteristics and service requirements into account.

Redesigning Model Architecture and Computational Precision

The model itself is the first element to consider. Compression techniques such as pruning, low-rank decomposition, and knowledge distillation can remove unnecessary parameters. Quantization can also be applied to replace 32-bit floating-point operations with 8-bit integer operations.

However, the key is not simply to reduce the number of bits. After determining which operation formats and tensor sizes the target device’s NPU or DSP can handle most efficiently, the model must be designed in a form that the accelerator can process effectively. If too many operations have limited hardware support, the workload may ultimately fall back to the CPU, slowing down the entire pipeline.

Dividing Roles Among the CPU, GPU, and NPU

Smartphones and intelligent cameras do not rely on just one processor. They typically combine a CPU suited to general-purpose tasks, a GPU optimized for parallel computation, an NPU specialized in AI inference, and a DSP designed for signal processing.

In a co-designed system, each operation is assigned to the device best suited for it. For example, image preprocessing can be handled by the DSP or GPU, convolution and matrix operations can run on the NPU, and final control logic can be managed by the CPU.

However, blindly distributing workloads across multiple devices is not always the answer. Excessive data copying between devices can actually reduce performance. An Edge AI runtime must therefore evaluate all of the following together:

• Which operations should be assigned to which processor
• Whether the cost of moving data outweighs the benefits of computation acceleration
• Whether processing can be completed within the real-time response deadline
• The current battery level, thermal conditions, and status of other running tasks

Execution Plans That Minimize Memory Movement

In edge environments, memory access is just as important as computation—and sometimes even more so. When model weights and intermediate tensors repeatedly move between memory and cache, power consumption rises and response times suffer.

That is why efficient Edge AI systems use techniques such as tensor reuse between layers, cache-friendly data layouts, elimination of unnecessary intermediate results, and operator fusion. By combining multiple operations into a single execution unit and avoiding writes of intermediate data to external memory, systems can reduce both data movement and the number of runtime calls.

The Amount of Computation Should Adapt to the Input

Not every input requires the same level of computation. Processing simple and complex scenes with a model of identical depth wastes both battery life and processing time.

One way to address this is dynamic inference. By placing early-exit points within the model, inputs that can be classified with high confidence can receive an answer quickly. In more difficult situations—such as nighttime roads, crowded intersections, or pedestrians whose view is partially obstructed—the system can execute a deeper computational path.

This approach is particularly effective in services such as navigation, traffic control, and smart cameras, where input difficulty constantly changes. It allows the system to operate at low power during normal conditions and use additional resources only when they are truly needed.

In the End, the Real Advantage Lies in ‘Field Performance,’ Not ‘Model Performance’

The success of Edge AI is not determined by benchmark accuracy alone. Real-world products must satisfy all of the following conditions at once:

• Operate for long periods on a limited battery
• Make rapid decisions without a network connection
• Avoid sharp performance degradation caused by heat
• Run reliably across device-specific hardware variations
• Process personal information without transmitting it externally

For this reason, the future of Edge AI will not focus solely on building “smaller models.” The real difference will come from co-designing the model’s computational methods, hardware accelerator utilization, memory architecture, execution order, and power budget as one connected system.

A small model is only the starting point. To build AI that runs quickly, for long periods, and reliably at the edge, the model and the system must be optimized together—as if they were a single product.

Four Technical Weapons for Making Edge AI Models Lightweight

Should every input be processed through the heaviest possible path? Applying the same amount of computation to a bright, clear image and to a complex road scene filled with vehicles, pedestrians, and traffic lights can place an unnecessary burden on edge devices.

The core of Edge AI is not simply using smaller models. It lies in changing the model and its execution strategy so that only the necessary computations are performed when they are needed. Four representative technologies make this possible.

1. Pruning: Removing Unimportant Connections

Pruning is a technique that removes low-impact weights, channels, or layers from a model. Large neural networks may contain connections that contribute very little to the actual inference results.

• Unstructured pruning: Removes individual weights with low importance.
• Structured pruning: Removes hardware-friendly units such as channels, filters, heads, and layers.

Structured pruning is particularly advantageous in edge environments. It reduces not only the number of parameters but also the actual computational workload and the number of memory accesses. For example, removing infrequently used channels from a camera-based object detection model directly reduces the size of the tensors that an NPU or GPU must process.

However, aggressively cutting down a model can reduce its accuracy. That is why fine-tuning after pruning is essential for recovering lost performance.

2. Quantization: Replacing 32-Bit Computation with Smaller Numbers

Quantization represents a model’s weights and activations using lower precision—such as 8-bit integers instead of 32-bit floating-point values, or in some cases 4-bit or even lower precision.

This change makes a bigger difference than you might expect.

• The model file becomes smaller.
• Memory bandwidth usage decreases.
• Inference becomes faster on NPUs and DSPs optimized for integer operations.
• Reduced data movement lowers power consumption.

For example, converting an FP32 model into an INT8 model can theoretically reduce the storage space required for weights to approximately one quarter. In Edge AI, the bottleneck is often not the computation itself but the cost of reading and writing data in memory. As a result, quantization can directly affect both performance and battery efficiency.

Quantization methods are broadly divided into two types:

• Post-Training Quantization (PTQ): Converts a trained model after training is complete. It is fast and easy to apply.
• Quantization-Aware Training (QAT): Reflects the low-precision environment during training to minimize accuracy loss.

For tasks sensitive to accuracy, such as traffic detection and facial recognition, QAT may be more suitable.

3. Lightweight Architecture Design: Building for the Edge from the Start

Reducing a heavy server-side model is not the only answer. Another approach is to design the network architecture from the beginning with mobile and embedded environments in mind.

Representative approaches include:

• Depthwise Separable Convolution: Separates standard convolution operations to significantly reduce computational demands.
• Bottleneck structures: Lower computational costs by processing only important features within a narrow channel space.
• Efficiency-focused Transformer variants: Instead of processing an entire long input uniformly, they focus on the tokens or regions that matter.
• Neural Architecture Search (NAS): Automatically finds architectures suited to the target device’s latency, power, and memory constraints.

The key to this approach is not simply reducing the FLOPs figure. The goal is to choose computation patterns that run quickly on the actual device. Some operations may appear lightweight in theory but run slowly in practice because they are unsupported by a particular NPU or require extensive memory access. Edge AI models must therefore be designed with the characteristics of the hardware in mind.

4. Dynamic Inference: Using Only the Computation Easy Inputs Need

Dynamic inference is a technique that does not apply the same computational path to every input. The model assesses the difficulty or circumstances of an input and performs only as much computation as necessary.

Representative methods include:

• Early Exit: If an intermediate layer already produces a result with high confidence, inference ends without executing the later layers.
• Dynamic layer and channel selection: For simple inputs, some layers or channels are skipped.
• Adaptive resolution processing: Unimportant scenes are analyzed at a lower resolution, while high-resolution processing is triggered when an abnormal situation is detected.
• Region-based computation: Instead of focusing on the entire screen, computation is concentrated on areas containing changes, such as people, vehicles, or products.

For example, when a traffic camera captures an empty road, it can quickly determine whether any vehicles are present using a simple path. In contrast, when a complex scene simultaneously includes pedestrians, bicycles, vehicles, and changing traffic signals, the full model can be executed to make a more precise judgment.

These methods are effective for reducing average latency and energy consumption. However, caution is necessary for safety-related services. If dynamic inference exits too early, it may miss rare but dangerous situations. Confidence thresholds and exception-handling rules must therefore be designed conservatively.

The competitive advantage of Edge AI lies not in “always running the largest model,” but in choosing the most efficient computation for each input situation.

Pruning, quantization, lightweight architectures, and dynamic inference are not substitutes for one another. In practice, all four should be applied together while also considering the execution characteristics of CPUs, GPUs, and NPUs. Only then can fast and reliable local AI inference be implemented within limited power and memory budgets.

Edge AI’s Real Bottleneck: Data Movement, Not Computation

Even when a device is equipped with an AI accelerator or NPU, it may still fail to perform as quickly as expected—or generate excessive heat and drain the battery during inference. The cause may not be insufficient computational performance from the accelerator. The real bottleneck may lie in repeatedly moving data between memory and the processor while the model is performing calculations.

This issue is particularly important in Edge AI environments. Smartphones, cameras, automotive computing devices, and IoT sensors often cannot afford server-class memory bandwidth or power budgets. Therefore, no matter how fast the computing units are, the accelerator will be forced to wait if the data it needs is not supplied in time.

Memory Access Can Cost More Than Computation

During AI inference, weights, input data, and activation tensors—the intermediate results—move back and forth between memory and the computing units. For example, when an object detection model processes a camera frame, the following steps are repeated:

1. Camera image data is loaded into memory.
2. The NPU or GPU reads the inputs and weights.
3. The intermediate results from each layer are stored in memory.
4. The next layer reads those results again and performs its calculations.
5. The final results are passed to the CPU or application.

If data cannot remain in the on-chip cache and has to travel frequently to external memory, latency and energy consumption can rise rapidly. In particular, bandwidth-intensive workloads such as high-resolution video, multiple camera streams, and voice data can make the cost of data movement exceed the cost of computation.

Edge AI efficiency is not determined solely by “how many computations can be processed per second.”
The key is how close the required data can be kept—and how little it has to be moved.

Why Data Movement Causes Heat and Latency

In many cases, reading and rewriting large amounts of data from DRAM requires more energy than performing additions or multiplications in the processor. As models grow larger—and as the number of intermediate tensors increases—this cost grows as well.

This problem appears in several ways:

• Increased inference latency: The compute units are ready, but remain idle while waiting for data from memory.
• Higher battery consumption: Repeated memory access and data transfers increase power usage.
• Rising temperatures: Continuous use of memory and buses generates heat, which can eventually lead to performance throttling.
• Dropped frames: In real-time video analysis, processing time fails to keep up with the incoming frame rate.
• Lower accelerator utilization: Even with a high-performance NPU, actual performance may fall far short of its specifications.

In other words, simply adding a “faster chip” is not enough for Edge AI. The model, memory architecture, runtime, and data flow must be viewed as a single system.

Design Strategies for Reducing Data Movement in Edge AI

Solving the data movement problem requires algorithmic and system-level optimization to be applied together. This is precisely why co-design approaches have attracted attention in recent years.

1. Tensor Reuse and Memory Buffer Optimization

Creating new memory space for every layer and continuously copying intermediate results wastes bandwidth. Instead, systems should be designed so that the output buffer from a previous operation can be reused by the next operation, while eliminating unnecessary tensor copies.

A runtime can analyze the model graph and perform the following:

• Reuse the memory space of intermediate tensors whose lifetimes have ended
• Eliminate unnecessary data copies between consecutive operations
• Apply tensor layouts that fit efficiently in the cache
• Reduce the number of memory allocations and deallocations

These may appear to be minor optimizations, but their cumulative impact can be significant in real-time Edge AI services.

2. Reducing Intermediate Storage Through Operator Fusion

A typical model executes multiple operations in sequence, such as convolution, normalization, and activation functions. If the result of each step is stored in external memory, data movement is repeated over and over.

Operator fusion combines multiple operations into a single kernel or execution unit. For example, instead of storing the result of a convolution in memory and then reading it again to apply an activation function, the next operation can be performed immediately while the result is still in a register or cache.

This approach can deliver the following benefits:

• Fewer memory reads and writes
• Lower inference latency
• Reduced pressure on memory bandwidth
• Lower power consumption and heat generation

3. Reducing the Amount of Data Transferred Through Quantization

32-bit floating-point data offers high precision, but requires substantial storage space and transfer volume. Converting it to 8-bit integers or a lower precision can significantly reduce the size of model weights and activation tensors.

For example, quantizing 32-bit data to 8-bit data can theoretically reduce the memory capacity required to move the same amount of information to one-quarter. This does more than simply reduce the size of the model file. It can also improve memory bandwidth, cache efficiency, and power consumption.

However, accuracy loss, hardware support, and the sensitivity of individual operations must all be evaluated together. In applications such as object detection or safety-critical systems, a mixed-precision strategy—adjusting precision layer by layer rather than applying a single setting to the entire model—may be more effective.

4. Minimizing Unnecessary Round Trips Among the CPU, GPU, and NPU

Task placement is also important on devices where heterogeneous hardware coexists. A particular operation may run faster on the NPU, but if the tasks before and after it run on the CPU, the data may have to travel through memory and across multiple processors.

For example, the following structure can cause performance loss:

• Preprocessing on the CPU
• Core inference on the NPU
• Post-processing on the GPU
• Final result handling on the CPU again

At each stage, data format conversions, buffer copies, and synchronization between devices may occur. Therefore, the runtime must schedule tasks while accounting not only for computation speed but also for the cost of moving data.

The goal is not to choose the fastest processor. It is to choose the execution plan that minimizes data movement across the entire pipeline.

Performance Metrics Must Change Too

When evaluating Edge AI models, focusing only on accuracy and FLOPS has limitations. Even a model with low FLOPS can be slow on an actual device if its memory access is inefficient. Conversely, a model with somewhat more computation may be faster and more power-efficient if it has an effective data reuse structure.

In practice, it is useful to examine the following metrics together:

• Inference latency on the actual device
• Energy consumption per frame
• Memory usage and memory bandwidth utilization
• NPU, GPU, and CPU utilization
• Heat generation and throttling during extended operation
• The number of data copies and transfers between devices

Ultimately, the next source of competitiveness in Edge AI will not come from simply pushing more computation through the system. It will come from placing model weights and intermediate results where they are needed, exactly when they are needed, and moving them as little as possible. Fast inference does not come from a powerful accelerator alone. It comes from a system designed around the data flow as well.

Edge AI: Beyond Smaller Models, Toward Broader AI

The goal of Edge AI is not simply to reduce the size of model files. Its true purpose is to execute AI capabilities instantly on the device itself—capabilities that previously required a connection to the cloud.

Imagine an environment where cameras distinguish between people and vehicles in real time, in-vehicle navigation systems assess dangerous situations without communication delays, and kiosks provide personalized guidance without sending personal information outside the device. This is the vision of Edge AI made possible by co-designing models and systems.

Why Model Compression Alone Is Not Enough

A smaller model is not always a faster model. Even when the number of model parameters and computational operations is reduced, actual response times may not improve as expected if the device performs excessive memory transfers or if workloads are inefficiently distributed across the NPU, GPU, and CPU.

For example, real-time video analysis repeatedly goes through the following process:

• The camera captures high-resolution frames.
• Preprocessed video data passes through memory and moves to the AI accelerator.
• An object detection or classification model performs inference.
• The results lead to on-screen displays, alerts, or control commands.

In this process, bottlenecks may arise not from the AI computation itself, but from data copying, memory bandwidth, and transitions between hardware components. A co-designed architecture therefore goes beyond compressing the model—it optimizes both the path data takes and the order in which computations are executed.

How Edge AI Co-Design Changes the Way AI Runs

Co-design does not treat AI algorithms and hardware or runtimes as separate elements. From the model design stage onward, it considers the target device’s NPU performance, memory capacity, power budget, and real-time response requirements together.

The key shift is from asking, “Which model is the most accurate?” to asking, “How should this model run on this device to achieve the highest efficiency?”

| Design Perspective | Conventional Approach | Co-Design Approach | |---|---|---| | Model development | Focused on accuracy and benchmarks | Considers accuracy, latency, power, and memory simultaneously | | Hardware utilization | Ports a trained model afterward | Designs the model and computation graph around the characteristics of the CPU, GPU, and NPU | | Performance optimization | Focuses on reducing model size | Optimizes workload placement, memory access, and data movement | | Service operation | Relies on cloud inference | Uses a distributed structure combining local inference with cloud management |

With this approach, some computations can be concentrated on the NPU, while control logic and lightweight post-processing can be assigned to the CPU. Highly parallel tasks, such as video preprocessing, can be handled by the GPU or DSP. Memory buffers can also be reused so that results do not need to be copied again.

In other words, Edge AI performance is determined not by model size alone, but by how organically the model, compiler, runtime, accelerator, and memory architecture work together.

The Value It Creates Across Industries

These changes expand the scope of AI adoption across many industries. They are especially valuable in fields where latency, connectivity, and privacy are critical.

• Transportation and Mobility: Roadside cameras and in-vehicle systems can identify pedestrians, traffic signals, and obstacles on-site. By reducing network round-trip time, they can deliver faster warnings and control responses.
• Retail and Shopping: Kiosks and digital signage can analyze customer reactions and product conditions locally. Since original video or voice data does not need to be sent to external servers, privacy concerns are also reduced.
• Entertainment: Smart TVs, set-top boxes, and gaming devices can provide instant voice recognition and personalized recommendations. They can also minimize the decline in perceived performance caused by poor internet connectivity.
• Smart Cameras and Security: Instead of uploading every video stream to the cloud, systems detect only relevant events and transmit them. This approach reduces both network and storage costs.

The important point is that Edge AI does not mean completely replacing the cloud. A practical division of responsibilities is for devices to handle real-time decisions, while the cloud manages large-scale training, long-term analysis, and model deployment. Co-design provides the foundation for making this distributed structure operate more efficiently.

Developers’ Roles Expand from ‘Model Deployment’ to ‘System Design’

The change is equally clear for developers. In the past, the focus was often on selecting a highly accurate model and connecting it through an API. Now, developers must also design for the conditions under which the model will actually run on the target device.

The following questions become especially important:

• Which levels of precision, such as INT8 or FP16, does the target device’s NPU support most efficiently?
• Could data transfer and preprocessing introduce more latency than model inference itself?
• Can the system run continuously when battery life, heat generation, and memory usage are taken into account?
• Will core functions remain available even if the network connection is lost?
• How will model versions and performance degradation on deployed devices be monitored?

As a result, future Edge AI development will require the ability to treat model training, quantization, compilation, runtime optimization, and remote updates as one continuous process. Just as important as the ability to create more accurate models will be the ability to build AI systems that operate reliably in the real world.

Ultimately, a co-designed architecture does not create “small AI.” It creates broader AI—AI that can make more decisions within limited power and memory, protect more sensitive data locally, and be deployed across a wider range of industries and real-world environments.