Engineers implement always-on AI vision at the edge by moving the decision closer to the pixel. Instead of streaming every frame to a separate processor, the system senses continuously at low power, filters out redundant data, runs inference only on what matters, and sends or acts on a small result. Intelligent vision sensors with in-pixel computing take this furthest, because they reduce data movement at the source rather than after the fact.
That one idea explains most design choices in this field. The sections below cover why continuous vision is hard, which architectures teams use, how the pipeline fits together, and how to judge a solution before you commit to it.
Why Is Continuous Vision So Hard On Power, Bandwidth, And Latency?
A conventional camera pipeline captures every frame, moves it off the sensor, and hands it to a processor. That works well when power is plentiful and the scene is worth recording. It works poorly when a device must watch all day and act in milliseconds.
Conventional CMOS image sensors read out full frames continuously, even when nothing in the scene has changed. Readout rather than compute can become the dominant system cost at high frame rates and resolutions, which is the problem intelligent vision sensors are built to address. Every hop adds delay and draws power: pixel array, sensor interface, bus, external memory, host processor.
Bandwidth shows the problem clearlyAlways-on vision pushes on all three limits at once: a small power budget, a thin communication link, and a tight deadline between seeing and acting.
What Is Always-On AI Vision?
Always-on AI vision is a system that watches continuously and interprets what it sees locally, so that significant power, bandwidth, and computation are spent only when something relevant happens.
Typical real-world uses include:
- Occupancy and presence sensing in buildings, where lights and HVAC respond to people.
- Gesture, hand, and eye tracking in AR, VR, and wearables, under tight power and thermal budgets.
- Obstacle detection and navigation for robots and drones.
- Inspection and anomaly detection on production lines.
- Traffic, pedestrian, and flood monitoring in cities.
- Perception on unmanned platforms with size, weight, and power (SWaP) limits and little or no network.
Edge AI use cases by industry covers many more, and the recurring capabilities are the same: presence, counting, movement, inspection, and anomaly detection.
What Are The Main Engineering Challenges?
Power Budgets
If every frame triggers downstream processing, the whole chain stays awake. Battery-powered and SWaP-limited products cannot afford that. Power is also where architecture matters most, since less redundant data movement means less energy spent on transmission and reprocessing.
Data Movement
Moving pixels costs more than most teams expect. Research such as this MDPI study on embedded AI vision systems, supports the general point that data movement is a central bottleneck.
Latency
Each hop between sensor and decision adds time. For a robot arm, a conveyor, or a gesture interface, milliseconds can decide whether a part is missed or an interface feels broken.
Privacy
Local processing helps, but it does not make a system private or compliant on its own. Retention, access, notice, and purpose limitation still matter. The UK ICO’s data minimisation guidance is a useful reference. Sending counts or events instead of images is one practical way to apply it.
Reliability
The system must perform predictably across shifts, lighting changes, vibration, and temperature. Aggressive filtering can also discard information that downstream algorithms need, and dense, fast-changing scenes can generate so many events that bandwidth savings shrink. Plan for both.
What Are The Common Architectures?
Most always-on designs fall into three families. They differ mainly in where the first decision is made and how much data travels to reach it.
CMOS Sensor Plus Processor
The sensor outputs full frames at a fixed rate and a CPU, GPU, or NPU interprets them. It is familiar, flexible, and the right choice when image quality, resolution, or extensive downstream image signal processing matters most. It is also the most expensive way to stay awake all day.
Low-Power Trigger Systems
A lightweight detector watches for change and wakes the heavier pipeline only when needed. Event-based sensors are one version of this idea. They report pixel changes asynchronously, which keeps bandwidth low. The limit is that reporting only change can miss meaningful information when nothing moves.
Intelligent Vision Sensors
Here the sensor itself does part of the interpretation. An intelligent vision sensor embeds computation in the pixel array or sensing substrate, so it extracts task-relevant information at capture instead of streaming full raw frames.
How Do Engineers Build The Always-On Vision Pipeline?
Whatever the hardware, the pipeline follows the same five stages. The aim at each stage is to hand the next one less data than it received.
Sensing
Choose the sensor for the task, not for image beauty. If a human will never look at the output, resolution, frame rate, and readout mode can be tuned for the algorithm. Programmable readout is valuable here, because the best settings change with the scene.
Filtering
Remove what does not change the answer: static background, unchanged pixels, regions outside the area of interest. This is where the largest savings come from, and where designs most often go wrong, so keep a way to switch back to full frames while you validate.
Inference
Run the detection, tracking, or classification model on the reduced data. Check that your model still performs when its input is events or features instead of full frames. Do not assume it will.
Communication
Send decisions, counts, or events, not video. This cuts link cost, storage, and the amount of personal imagery that leaves the device.
Action
Close the loop: switch the light, stop the conveyor, steer the robot, update the display. The latency budget is measured from photons to this step, so test the whole path, not each block alone.
How Do In-Sensor And In-Pixel Computing Reduce Data Movement?
Traditional image sensor
Full frames out, interpretation on a separate processor.
Edge processing
Interpretation is off-sensor but physically near the camera.
In-sensor processing
Early operations such as change detection happen inside the sensor package.
In-pixel computing
Processing sits at individual pixels or small pixel groups, closest to capture.
Intelligent vision sensing
Pixel array, logic, and I/O are unified in a typically programmable unit that can be reconfigured after deployment.
The advantage of in-pixel computing is that each pixel can decide whether anything interesting is happening and process data in place, in parallel, so only relevant data leaves the chip. These designs typically rely on stacked backside-illuminated (BSI) CMOS, where photodiode and logic wafers are bonded together.
There are trade-offs. Fixed-function designs are simpler and cheaper, while programmable ones add control logic. Stacked in-pixel designs need advanced wafer bonding, which affects yield and cost. In-sensor processing is also not the right answer everywhere: image-quality-driven pipelines, systems on stable power with ample compute, and early products still defining their requirements are often better served by a standard sensor and processor reference design.
How Does OCULI Support Always-On Vision?
OCULI is a fabless semiconductor company in Rochester, NY, building programmable vision sensors that process machine vision at the edge.
OCULI SPU™
The SPU, short for Sensing and Processing Unit, integrates sensing, processing, and memory on one chip. Its four blocks are a pixel array of IntelliPixel units, a read-out control block, an I/O block for the host, and an ISP stage. It is offered as a standalone chip or as a complete lens module.
IntelliPixel®
Each IntelliPixel combines a sensor, front-end, ALU, and local memory in a patented digital pixel, processing, and memory architecture. Pixels compute in parallel and only relevant data leaves the sensor.
Software-Defined Vision Sensor™
The chip can be reconfigured in software, without new hardware, to move between full-frame video and a reduced stream of actionable information. The SPU can output full frames, events, smart events, or actionable information. This lets one device serve several products, which reduces how often OEMs must design or qualify new sensor variants.
Which Industries Use Always-On Edge Vision?
Smart Buildings
Occupancy-based lighting and HVAC control benefit from counting people, not just detecting motion. A PNNL simulation report found 17.8% average energy savings with advanced occupancy sensors versus 5.9% for common ones. Presence data also avoids collecting raw images.
Smart Cities
Traffic flow, pedestrian counts, parking, and infrastructure analytics need devices that run for long periods with little cloud dependency.
Robotics And Industrial Systems
Robots and production lines need short sensor-to-decision loops: obstacle detection, defect detection, assembly verification, and pick-and-place guidance.
Wearables And Smart Devices
AR, VR, and wearable devices need gesture, eye, and face tracking within a small battery and thermal budget. Demos show eye tracking at about 2% and face detection at about 0.9% of full-frame bandwidth.
Defense And Unmanned Systems
Drones, ground robots, and marine vehicles have strict SWaP limits and often operate without a network. Local interpretation, with no round trip to a server, is a requirement here rather than a preference.
How Should You Evaluate An Always-On Vision Solution?
Datasheets are not enough. Test on your own scenes, lighting, and decision deadlines. Use the table below as a starting checklist.
| Criterion | What to ask |
| Power | What is the draw while idle, and what is it during active inference? Does it fit your battery or SWaP budget over a full duty cycle? |
| Accuracy | Does your model hold up on the reduced output? Compare against a full-frame baseline on the same scenes, including hard cases. |
| Latency | What is the total time from capture to action, measured end to end, against your deadline? |
| Bandwidth | How much data leaves the sensor, and how does it behave in crowded, fast-changing scenes? |
| Integration | What interfaces, SDK, and host processing does it need? How much engineering effort to bring up? |
| Reliability | Does it perform consistently across shifts, temperature, vibration, and lighting? How predictable is response time? |
| Programmability | Can you change output modes and vision parameters after deployment, without new hardware? |
Start With The Data You Do Not Need To Move
Always-on AI vision at the edge works when the system is designed around one question: what is the least data that still lets us make the right decision? Sensing, filtering, inference, communication, and action each answer it a little differently, and the earlier you filter, the more every later stage saves.
Engineering teams building always-on vision products can evaluate OCULI against their own application.
Test the SPU with a hardware kit or a software simulator, measure power, latency, and accuracy on your scenes, and decide with your own data.
Apply for an evaluation program or talk to the OCULI team.
Frequently Asked Questions
What is the difference between edge AI and sensor-level processing?
Edge AI runs inference near the data source, but it often still works from full frames handed over by the sensor. Sensor-level processing changes what the sensor sends in the first place. The two are complementary, and many systems use both.
Do intelligent vision sensors replace industrial cameras?
Not necessarily. Many deployments combine a conventional full-frame path with sensor-level intelligence. Conventional sensors remain preferable when image quality, resolution, or extensive downstream image processing matter more than power or latency.
Is an intelligent vision sensor the same as an event-based sensor?
No. Event-based sensors report pixel changes. Intelligent vision sensors can also include feature extraction and inference logic, and some architectures combine event-driven readout with in-pixel processing.
Does processing on the device make vision data private?
It helps, because less imagery leaves the device. It does not guarantee compliance by itself. Retention, access, notice, and purpose limitation still apply, and sending counts or events instead of images supports data minimisation.
When is in-sensor processing not the right choice?
It is a weaker fit for image-quality-driven pipelines, for systems on stable power with ample compute and relaxed latency, and for early products whose requirements are still changing. A standard sensor plus processor design is more flexible in those cases.
How do I test an AI vision sensor before building around it?
Run it on your own scenes and compare power, latency, bandwidth, and accuracy against a full-frame baseline. Software simulators let a team start on a laptop before hardware arrives, and a proof of concept can validate results in your operating environment.
What can software-defined vision do that a fixed sensor cannot?
A software-defined vision sensor can be reconfigured after manufacture for different tasks, outputs, or power profiles, so one hardware design can serve several products and markets.
