Compute-in-Memory and Sensing-Storage-Computing Fusion: A Technical Breakthrough in Brain-Inspired Vision

1. Introduction: When Visual Perception Hits the "Memory Wall"

From face unlock on smartphones to environmental perception in autonomous driving, and to millisecond-level defect detection in industrial quality inspection, machine vision has long left the lab and become the "eyes" of intelligent systems. Yet when we try to squeeze vision capabilities into a thumbnail-sized edge chip, an architectural design born in 1945 becomes an invisible wall.

That is the "memory wall" problem of the von Neumann architecture. It is not a defect in any specific component, but the inevitable price of the fundamental design that separates computation from storage under modern data-intensive workloads. Brain-inspired vision is precisely one of the scenarios with the highest data density and the most stringent real-time requirements.

This article, starting from the underlying contradiction of the "memory wall," will sort out how ferroelectric photoelectric sensor arrays achieve sensing-storage-computing integration, how brain-inspired vision performs weighted summation while sensing an image, and the real potential and engineering challenges of this technical route in edge AI.

2. The "Memory Wall" Problem in the von Neumann Architecture

2.1 The Original Sin: Separation of Computation and Storage

The core idea of the von Neumann architecture is "stored program": instructions and data are stored together in memory, and the CPU fetches instructions and data, computes, and writes back on demand. This design gave computers generality, but also planted a structural hazard — every operation requires data to be shuttled back and forth between the processor and memory.

In early computing tasks, this cost was negligible. But when the task becomes "performing per-pixel convolution on a 1920×1080 image," the problem balloons: a single 3×3 convolution requires each output pixel to read 9 input pixels and 9 weights, and the weights are reused across the entire image. The data movement far exceeds the actual computation; the processor spends most of its time "waiting for data," not "computing."

2.2 Dual Bottlenecks: Bandwidth and Power

The "memory wall" manifests as two bottlenecks:

• Bandwidth bottleneck: Processor compute growth follows Moore's law, while DRAM bandwidth growth has lagged far behind. The gap is the so-called "memory wall" (Memory Wall). No matter how powerful the compute, without data to feed it, the processor idles.
• Power bottleneck: In advanced process nodes, a 32-bit floating-point multiply-accumulate operation costs on the order of picojoules, while an off-chip DRAM access can cost hundreds of times that. The famous conclusion that "data movement energy far exceeds compute energy" is a direct consequence of the separation of storage and compute.

Typical energy comparison (order of magnitude):
  32-bit multiply operation      ≈ 3.1 pJ
  32-bit SRAM read               ≈ 5 pJ
  32-bit DRAM read               ≈ 640 pJ   ← movement cost is 200× computation

2.3 Why Vision Tasks Hit the Wall First

Vision tasks inherently have three characteristics that all trigger the pain points of the memory wall:

1. Large data volume: A single high-res frame has millions of pixels; video streams are a continuous data flood.
2. Compute-intensive but structurally regular: Convolutions, pooling, etc. are highly repetitive and theoretically suited for parallel acceleration, but they are limited by data supply.
3. Real-time demands: Autonomous driving perception latency is measured in milliseconds; data cannot afford to wait in queues on the bus.

So a natural question arises: Can we compute where the data lives, instead of moving the data to the compute unit? This is the starting point of Compute-in-Memory (CIM).

The following diagram compares the data flow paths of traditional vs. in-memory compute architectures, showing the difference between "moving" and "computing in place":

flowchart TB
  subgraph T["Traditional von Neumann Architecture"]
    T1["Sensor acquires image"] --> T2["Data written to memory"]
    T2 --> T3["Processor reads data"]
    T3 --> T4["Computation, then write back to memory"]
    T4 --> T2
  end
  subgraph C["Compute-in-Memory Architecture"]
    C1["Sensor acquires image"] --> C2["Data goes directly into compute array"]
    C2 --> C3["Multiply-accumulate performed in place"]
    C3 --> C4["Output result"]
  end
  T4 -.->|"Data moves repeatedly, bandwidth and power bottleneck"| C1

3. Ferroelectric Photoelectric Sensor Array: The Sensing-Storage-Computing Integration Approach

3.1 From "Compute-in-Memory" to "Sensing-Storage-Computing Integration"

Compute-in-memory solves the separation of computing and storage, but a vision system has a third stage — "sensing." The traditional chain is: photoelectric sensor captures light → converts to digital → transfers to processor → stores → computes. Every cross-stage data transfer consumes bandwidth and power.

In-Sensor Computing goes a step further: the sensor element itself possesses sensing, storage, and computing capabilities, performing preliminary information processing at the very first site of light-to-electric conversion. This effectively merges part of the "eye" and part of the "brain" into the same physical device.

The chain evolution from "sensing" to "computing" and "storage" can be summarized:

flowchart LR
  subgraph P1["Traditional separate chain"]
    A1["Photoelectric sensing"] --> A2["Analog-to-digital conversion"] --> A3["Transfer to processor"] --> A4["Storage"] --> A5["Computation"]
  end
  subgraph P2["In-Sensor Computing chain"]
    B1["Photoelectric sensing + weight storage + analog multiply-accumulate"] --> B2["ADC quantization"] --> B3["Output result"]
  end
  P1 -.->|"Stages merged, data movement eliminated"| P2

3.2 Why Ferroelectric Materials Are the Key

Ferroelectric materials (such as HfO₂-based ferroelectric thin films, P(VDF-TrFE), etc.) hold a central place in this scheme due to their unique physical properties:

• Non-volatile multistate storage: Ferroelectric polarization has multiple distinguishable stable states and persists after power-off, making it naturally suitable for multivalued storage of synaptic weights.
• Tunable photoresponse: The transport of photo-generated carriers in ferroelectric films is modulated by the polarization state, which means the "stored weight" can directly influence the "sensed signal," achieving physical coupling of sensing and computation.
• Low power and CMOS compatibility: HfO₂-based ferroelectric materials are compatible with standard CMOS processes, providing a process foundation for large-scale array integration.

3.3 Array-Level Architecture: Each Pixel Is a Compute Node

In a ferroelectric photoelectric sensor array, each pixel is no longer just a "light intensity recorder"; it is a programmable computing node:

Single pixel unit structure:
  Light input → [Ferroelectric photoelectric layer] → Photo-generated current
                ↑
           [Ferroelectric polarization state = stored weight]
                ↓
        Output current = light intensity × weight (analog multiply-accumulate)

The entire array is organized as a crossbar array: row lines apply read voltages, column lines collect currents. According to Kirchhoff's current law, the output currents of all pixels on the same column naturally sum — the weighted summation, the core operation of neural networks, is "for free" physically performed.

The physical structure of the crossbar array is shown below; row lines apply read voltage and column lines aggregate currents, with weighted summation naturally occurring on the column lines:

flowchart TB
  V1["Row line V1"] --> R1["Pixel cell W11"]
  V1 --> R2["Pixel cell W12"]
  V2["Row line V2"] --> R3["Pixel cell W21"]
  V2 --> R4["Pixel cell W22"]
  R1 --> COL["Column line current accumulation I = Σ(Vi × Wij)"]
  R3 --> COL
  R2 --> COL2["Column line current accumulation"]
  R4 --> COL2
  COL --> OUT["Analog weighted summation output"]
  COL2 --> OUT

3.4 Comparison with Traditional Approaches

Dimension	Traditional CMOS image sensor + digital processor	Ferroelectric photoelectric in-sensor array
Data movement	Full-frame transfer to processor each frame	In-array local preliminary computation
Weight storage	Off-chip DRAM / on-chip SRAM	Ferroelectric polarization state, non-volatile
Multiply-accumulate	Sequential digital circuits	Parallel analog physical operations
Power	High (dominated by data movement)	Significantly reduced
Precision	High (digital determinism)	Affected by device non-idealities

This table highlights the key trade-off: in-sensor computing trades analog-domain physical parallelism for energy efficiency, but must also face the inherent accuracy and consistency problems of analog computing — which will be discussed in the engineering challenges later.

4. Brain-Inspired Vision: Performing Weighted Summation While Sensing an Image

4.1 Inspiration from Biological Vision

The human retina is not a "camera." Behind the hundreds of millions of photoreceptors, there are multiple layers of local neural processing: bipolar cells, horizontal cells, amacrine cells, ganglion cells... Before light signals reach the brain, edge enhancement, contrast adaptation, motion detection, and many other preprocessing tasks have already been performed.

This means that in biological vision systems, "sensing" and "computing" are fused from the very beginning. The retina itself is a highly parallel analog computing chip, consuming only about 20 milliwatts, yet it can process visual information equivalent to billions of bits per second in real time.

4.2 Physical Implementation Path of Sensing-Computing Integration

The hardware implementation of brain-inspired vision centers on mapping the neural network's "weighted summation" onto physical behavior of devices. With a ferroelectric photoelectric array, the workflow is:

flowchart LR
  A["Incident light image"] --> B["Ferroelectric photoelectric pixel array"]
  B --> C["Photo-generated current = light intensity × polarization weight"]
  C --> D["Column line current accumulation (Kirchhoff's law)"]
  D --> E["Analog weighted summation result"]
  E --> F["ADC quantization / threshold decision"]
  F --> G["Output feature map or classification result"]

Throughout this process, the image is never fully "read out and then fed back in"; the first convolution or fully connected operation is completed inside the array. For edge devices, this saves not only power but also latency.

4.3 Natural Compatibility with Spiking Neural Networks (SNNs)

Another important branch of brain-inspired vision is event-driven spiking neural networks. Dynamic Vision Sensors (DVS) output events only when pixel brightness changes, which is inherently sparse; the multistate properties of ferroelectric devices can simulate synaptic plasticity.

Combining the two can build a "sensing-computing-storage" trinity pulsed vision system: the event stream directly drives weight updates and membrane potential accumulation in the array, eliminating clock-synchronized full-frame processing. This asynchronous, sparse, event-driven working mode is precisely the source of the low-power advantage of brain-inspired computing.

The event-driven pulsed vision processing flow can be expressed as:

flowchart LR
  E1["DVS pixel brightness change"] --> E2["Output sparse event stream"]
  E2 --> E3["Ferroelectric synaptic array weight update"]
  E3 --> E4["Membrane potential accumulation"]
  E4 --> E5{"Threshold exceeded?"}
  E5 -->|"Yes"| E6["Emit spike"]
  E5 -->|"No"| E2
  E6 --> E7["Output recognition result"]

4.4 An Intuitive Analogy

If a traditional vision system is like a pipeline that "takes a photo, develops the film, then sends it for analysis," brain-inspired vision is more like "the eye makes a preliminary judgment the moment it looks." The former moves the film at every step; the latter has the judgment capability embedded right in the photoreceptor cells.

5. Potential of Compute-in-Memory Chips in Edge AI

5.1 Core Requirements of Edge AI

Edge AI devices operate under harsh constraints: power is typically in the milliwatt-to-watt range, compute is limited, and they often run on batteries or energy harvesting for long durations. Meanwhile, applications are getting heavier — face recognition in smart locks, health monitoring in wearables, real-time obstacle avoidance in drones.

Under these constraints, energy efficiency (TOPS/W) becomes more critical than absolute compute. This is exactly the main battlefield of compute-in-memory.

5.2 Typical Application Scenarios

• Smart security and access control: Face detection and recognition can perform preliminary feature extraction at the sensor, uploading only feature vectors — reducing power and preserving privacy.
• Industrial visual inspection: High-speed camera images on production lines can be screened for defects at the sensor, sharply reducing back-end compute demand.
• Autonomous driving perception: Multiple camera streams are preprocessed and pre-screened near the sensor, relieving the bandwidth pressure on the vehicle compute platform.
• Wearable and implantable medical devices: Ultra-low-power sensing-computing arrays suit long-term continuous monitoring, such as retinal prostheses and smart contact lenses.

5.3 Quantifying the Energy Efficiency Advantage

The energy efficiency advantage of compute-in-memory comes from two "subtractions":

1. Subtracting weight movement: Weights are stored inside the compute array, no need to read them from memory each operation.
2. Subtracting intermediate result movement: Analog multiply-accumulate is done in the analog domain; analog-to-digital conversion happens only when necessary, so intermediate feature maps don't repeatedly enter and leave memory.

Under ideal conditions, these two subtractions can bring one to two orders of magnitude energy efficiency improvement. Of course, behind the phrase "ideal conditions" lies a series of engineering obstacles that must be overcome.

6. Engineering Challenges: From Device to System

6.1 Device-Level Non-Idealities

The beautiful picture of analog computing must first face the physical reality of devices:

• Device-to-Device Variation: The ferroelectric domain switching behavior differs between pixels, causing the same weight to behave inconsistently across different units.
• Cycle-to-Cycle Variation: Writing the same weight repeatedly into the same device also yields random fluctuations in conductance.
• Retention and Fatigue: Ferroelectric polarization has a depolarization risk over time, and repeated switching can bring fatigue failure.
• Linearity and Symmetry: Ideal weight updates should be linear and symmetric, but actual conductance update curves are often nonlinear and asymmetric, directly affecting training convergence.

6.2 Array and Circuit Level Challenges

• Parasitic Resistance and IR Drop: In large crossbar arrays, parasitic resistance on row and column lines causes the actual voltage at remote cells to deviate from design values, introducing computation errors.
• Leakage and Crosstalk: Unselected cells are not perfectly isolated; leakage current pollutes the column current summation, requiring selector devices or compensation algorithms.
• ADC/DAC Overhead: The advantage of analog computing can be offset by the power and area of high-precision analog-to-digital conversion, so there is a trade-off between precision and energy efficiency.
• Dark Current and Non-Uniform Illumination: Dark current and response non-uniformity introduced by photoelectric sensing stack onto an already fragile analog signal.

6.3 Algorithm and System Level Co-Design

Hardware non-idealities cannot be completely eliminated, so compensation must happen on the algorithm side:

• Hardware-aware Training: Model device noise and non-linearity into the forward pass during training, so the network learns to "tolerate" hardware imperfections.
• Error Compensation and Calibration: Use on-chip calibration circuits or software post-processing to correct systematic deviations.
• Hardware/Software Co-Design: From network structure (quantization bit-width, sparsity) to array mapping strategies, everything must be jointly optimized with the hardware characteristics.

6.4 Ecosystem and Industrialization Barriers

Beyond technology, there are more practical barriers:

• Process yield and consistency: Going from a single lab device to a million-cell array, yield and consistency are prerequisites for mass production.
• Missing design toolchain: Traditional EDA tools don't yet support analog in-memory arrays well; there is no standardized modeling or simulation flow.
• No unified benchmark standards: Different teams report energy efficiency and accuracy metrics with different methodologies, making horizontal comparison difficult.
• Long application validation cycles: From prototype to certification in harsh scenarios like automotive or medical, it often takes years.

The engineering challenges from device to system can be summarized as a layer-by-layer barrier chain:

flowchart TD
  L1["Device layer: variation / fluctuation / fatigue / nonlinearity"] --> L2["Array layer: IR drop / leakage crosstalk / ADC overhead"]
  L2 --> L3["Algorithm layer: hardware-aware training / error calibration / HW-SW co-design"]
  L3 --> L4["Industry layer: yield / toolchain / benchmark standards / certification cycles"]
  L4 --> GOAL["Mass-producible edge brain-inspired vision chip"]

7. Summary and Outlook

Looking back at this technical line, the logic is quite clear:

The "memory wall" of the von Neumann architecture is essentially the data movement cost caused by separating computation and storage; compute-in-memory solves this by computing where the data lives; in-sensor computing further folds "sensing" into the same physical medium, so an image completes preliminary weighted summation the moment it is produced.

The ferroelectric photoelectric sensor array is the concrete landing point of this idea at the material and device level. It uses ferroelectric polarization states to store weights, photoelectric response to carry input, and Kirchhoff's law to perform summation — handing the core operation of neural networks to physical laws themselves.

But we must also soberly see that, from a high-performance lab device to a chip that runs steadily for years in an edge device, there are multiple barriers: device non-idealities, array parasitic effects, algorithm co-design, process yield, and toolchain ecosystem. On this road, materials scientists, device engineers, circuit designers, and algorithm researchers must work closely together.

The ultimate vision of brain-inspired vision is to make the machine's eye like the biological eye — low power, low latency, and "thinking while seeing." Compute-in-memory and sensing-storage-computing fusion are among the most promising paths toward this vision. It may not be the only answer, but it may be the one closest to the true meaning of "bio-inspired."