Physical Twinning for Joint Encoding-Decoding Optimization in Computational Optics

By Abdul Ahad NazakatReviewed by Louis CastelAug 19 2025

Computational optics represents a shift in approach where optical hardware and computational algorithms are designed to work together, enabling imaging capabilities that surpass those of traditional optical systems alone. Unlike conventional imaging, which relies mainly on physical optics, computational optics intentionally captures encoded measurements that are later processed through algorithms to reconstruct the desired information.¹ Traditional optical systems are typically developed in separate stages: first optimizing the optical components, then designing independent image processing algorithms. This segmented process often results in fundamental limitations, such as reduced performance and limited adaptability.

Physical twinning offers a different approach by creating digital twins of optical systems, allowing for the joint optimization of optical hardware (encoder) and computational algorithms (decoder) within an end-to-end differentiable framework. This method incorporates differentiable physics models, simultaneous optimization of both physical and computational elements, and task-specific design tailored to specific applications.

Image Credit: U.P.SD/Shutterstock.com

Why Joint Optimization Matters in Computational Imaging

The traditional separation between optics design and algorithm development creates limitations. Design approaches focused on general imaging metrics rather than task-specific performance lead to suboptimal solutions, while sequential optimization cannot recover information lost during the optical encoding stage. Independent optimization of components results in locally optimal solutions rather than achieving global optima across the entire system.

Joint optimization through physical twinning offers advantages across multiple domains. In low-light imaging applications, the optical encoder can be optimized to maximize light collection while the decoder reconstructs high-quality images from noisy measurements. This approach proves valuable in astronomical imaging and surveillance applications where photon efficiency is critical.

For compressed sensing applications requiring high-speed imaging or bandwidth-limited transmission, joint optimization enables optimal compression ratios while maintaining reconstruction quality. Hyperspectral imaging systems benefit significantly from this approach. Research has shown that performance improves when coded apertures and reconstruction networks are jointly optimized within CASSI (Coded Aperture Snapshot Spectral Imaging) systems, compared to existing methods that treat these components separately. ²

Research demonstrates that end-to-end optimization achieves superior image quality compared to separately optimized systems. Studies in extended depth of field imaging show that co-optimized phase masks and neural networks produce fewer artifacts compared to traditional approaches.^{3, 4} Additionally, joint optimization enables more efficient use of photons and computational resources, as demonstrated by lensless imaging systems that achieve high-quality imaging with minimal hardware complexity.

How Physical Twinning Works: Conceptual Overview

The foundation of physical twinning is built on creating differentiable models of optical phenomena that accurately simulate wave optics effects, such as diffraction, interference, and coherence. These models must also account for interactions with optical elements, including phase modulation, amplitude modulation, and polarization effects.

Take, for example, a basic physical twinning system that uses a coded aperture paired with a neural network decoder. In this setup, the optical encoder features a coded aperture made up of transparent and opaque regions that selectively block portions of incoming light. Rather than relying on traditional design methods, the aperture pattern is treated as a learnable parameter within the optimization process.

A differentiable model simulates how light from the scene interacts with this coded aperture. The measured intensity is modeled as the convolution of the scene with the point spread function defined by the aperture pattern, along with added noise. These encoded measurements are then processed by a neural network decoder to reconstruct the desired output.

During joint optimization, gradients flow through the entire pipeline. The forward pass runs from the scene through the coded aperture, to the sensor, through the neural network, and finally to the reconstructed output. A loss function compares this reconstruction with the ground truth, and the backward pass propagates gradients to update both the aperture pattern and the network weights simultaneously. This allows both the optical and computational components to be refined together in a fully integrated design loop.

The core advancement lies in making the optical simulation differentiable. This involves using smooth approximations for physical constraints, employing efficient Fast Fourier Transform-based models for light propagation and gradient calculation, and incorporating hardware constraints to ensure the resulting designs are practical.⁵

Key Applications and Case Studies

Microscopy Applications

Computational microscopy has embraced physical twinning for quantitative phase imaging, with systems jointly optimizing illumination patterns and reconstruction algorithms to achieve superior phase reconstruction compared to traditional methods.⁶ Researchers have demonstrated lensless microscopy systems where illumination patterns and reconstruction networks are co-optimized, achieving high-resolution phase imaging without traditional objective lenses.⁷

Joint optimization also enables novel fluorescence imaging modalities with improved signal-to-noise ratios and reduced photobleaching through optimized excitation patterns and computational reconstruction.

Hyperspectral Cameras

Traditional hyperspectral imaging requires scanning, limiting temporal resolution. Physical twinning enables snapshot systems that capture full spectral information in a single exposure. The HyperReconNet system demonstrates joint optimization of coded apertures and reconstruction networks for compressive hyperspectral imaging, achieving high-quality spectral reconstruction with significantly reduced acquisition time.²

Recent advances show real-time hyperspectral imaging using trained metasurface encoders, where the metasurface pattern and reconstruction algorithm are jointly optimized for specific spectral bands.⁸

Lensless Imaging Systems

Lensless systems replace traditional lenses with coded apertures, dramatically reducing size and weight while maintaining imaging capability. Research demonstrates lensless cameras that achieve performance comparable to traditional systems while being orders of magnitude smaller and lighter.⁹ These systems combine novel optical encoders such as random phase masks with deep learning decoders to achieve specialized imaging capabilities like extended depth of field or motion deblurring.

Notable research includes Stanford's Deep Optics project, which pioneered joint optimization of diffractive optical elements and neural networks, and MIT's comprehensive frameworks for end-to-end optimization of imaging systems including both refractive and diffractive elements.

Technological Challenges and Considerations

Physical twinning faces several significant technological challenges. Fabrication constraints present major hurdles, as manufacturing limitations including minimum feature size, surface roughness, and limited material choices must be incorporated into optimization algorithms. Design-for-manufacturability requires ensuring that optimal designs can actually be manufactured with available technologies.¹⁰

The simulation-to-reality gap represents another critical challenge. Success hinges on the accuracy of differentiable physics models. However, simplified models can overlook critical physical effects, while full wave optics simulations remain computationally intensive. Small modeling errors can lead to significant performance degradation in real systems.

Real systems require precise alignment and calibration, with manufacturing tolerances, environmental factors, and system integration all affecting performance. Systems must maintain performance across varying temperature conditions, mechanical vibrations, and aging effects while remaining resilient to various noise sources including shot noise, read noise, and dark current.

Hybrid analog-digital system trade-offs involve balancing optical encoding complexity against computational decoding requirements. Applications requiring real-time processing must consider computational complexity and potential need for specialized processors, while power consumption becomes crucial for portable and embedded systems.

Future Potential and Industry Outlook

Current research directions include advanced optimization techniques such as multi-objective optimization balancing performance, manufacturability, and robustness. Novel optical elements including metasurfaces, liquid crystal devices, and quantum optical elements offer unprecedented control possibilities.

Physical twinning could revolutionize edge AI devices by reducing computational requirements through optical preprocessing, enabling new form factors through lensless and ultra-compact imaging systems, and improving energy efficiency by performing computation optically.

Autonomous systems benefit from advanced vision capabilities that combine optical encoding with computational processing to enhance range and resolution. Applications include specialized vision systems for vehicles, multi-spectral sensing for reliable operation in all weather conditions, and compact integration to minimize size and weight. In robotics, enhanced perception enables novel depth sensing techniques and task-specific optical encoders that improve recognition accuracy.

In IoT applications, ultra-low-power vision sensors are made possible through event-driven imaging, compressed sensing to minimize data transmission, and energy harvesting systems to reduce power demands. Medical devices gain from these advancements as well, with implantable and wearable systems offering continuous monitoring, extreme miniaturization, and simplified electronics to improve biocompatibility.

Market opportunities span consumer electronics, industrial inspection, scientific instrumentation, and security applications. Research institutions and startups actively commercialize physical twinning technologies through strong patent portfolios, prototype systems demonstrating commercial viability, and industry partnerships for technology development.

The long-term vision extends to fully integrated systems seamlessly combining optical, electronic, and computational components, adaptive systems that continuously optimize based on changing conditions, and quantum-enhanced systems leveraging quantum optical effects with classical computational techniques.

Conclusion

Physical twinning for joint encoding-decoding optimization marks a major evolution in computational optics, shifting from traditional sequential design methods to a more integrated, system-wide optimization process. By enabling end-to-end differentiable optimization, this approach opens the door to capabilities and performance levels that were previously out of reach. The field has already shown strong results across a wide range of applications, and ongoing research continues to tackle existing challenges and explore new paths toward commercial integration in both consumer and industrial products.

Learn more about integrated photonics by reading on

References

1. Sitzmann, V., Diamond, S., Peng, Y., Dun, X., Boyd, S., Heidrich, W., Heide, F., & Wetzstein, G. (2018). End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics, 37(4), 1-13. https://doi.org/10.1145/3197517.3201333
2. Wang, L., Zhang, T., Fu, Y., & Huang, H. (2019). HyperReconNet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging. IEEE Transactions on Image Processing, 28(5), 2257-2270. https://doi.org/10.1109/TIP.2018.2884076
3. Akpinar, U., Sahin, E., Meem, M., Menon, R., & Gotchev, A. (2021). Learning wavefront coding for extended depth of field imaging. IEEE Transactions on Image Processing, 30, 3307-3320. https://doi.org/10.1109/TIP.2021.3060166
4. Elmalem, S., Giryes, R., & Marom, E. (2018). Learned phase coded aperture for the benefit of depth of field extension. Optics Express, 26(12), 15316-15331. https://doi.org/10.1364/OE.26.015316
5. Ho, C. J., Belhe, Y., Rotenberg, S., Ramamoorthi, R., Li, T. M., & Antipa, N. (2024). A differentiable wave optics model for end-to-end imaging system optimization. SPIE Proceedings, 12.3001139. https://doi.org/10.1117/12.3001139
6. Ray, C., Crombez, S., Exbrayat-Héritier, C., Ruggiero, F., & Ducros, N. (2024). Hyperspectral computational SPIM for quantitative multicolor imaging. SPIE Proceedings, 12.3022171. https://doi.org/10.1117/12.3022171
7. Xiong, Z., Engle, I., Garan, J., Melzer, J. E., & McLeod, E. (2018). Optimized computational imaging methods for small-target sensing in lens-free holographic microscopy. SPIE Proceedings, 12.2288570. https://doi.org/10.1117/12.2288570
8. Makarenko, M., Burguete-Lopez, A., Wang, Q., Getman, F., Giancola, S., Ghanem, B., & Fratalocchi, A. (2022). Real-time hyperspectral imaging in hardware via trained metasurface encoders. Computer Vision and Pattern Recognition, 12730-12740. Available at https://openaccess.thecvf.com/content/CVPR2022/html/Makarenko_Real-Time_Hyperspectral_Imaging_in_Hardware_via_Trained_Metasurface_Encoders_CVPR_2022_paper.html
9. Horisaki, R., Okamoto, Y., & Tanida, J. (2020). Deeply coded aperture for lensless imaging. Optics Letters, 45(11), 3131-3134. https://doi.org/10.1364/OL.390810
10. Li, Y., Chen, R., Gao, W., & Yu, C. (2022). Physics-aware differentiable discrete codesign for diffractive optical neural networks. Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, Article 78. https://doi.org/10.1145/3508352.3549378

Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.