Researchers have developed a deep reinforcement learning framework to optimally design a silicon-based photonic crystal fiber optical modulator with enhanced modulation performance and ultra-low insertion loss.
Study: AI-enhanced inverse design of photonic crystal fiber optical modulator using deep reinforcement learning technique. ThirdUnit/Shutterstock.com
Challenges in Photonic Inverse Design
Inverse design in photonics requires structural parameters that produce desired optical functions. These are essential for advanced modulators in telecommunications and integrated circuits.
Traditional methods, first-principles modeling, FDTD simulations, and optimization algorithms often need extensive human effort and computational resources and rely on iterative trial-and-error.
Deep learning has renewed automation efforts but is limited by the dependence on large, costly datasets, ambiguity in mapping optical responses to structures, and limited optimization beyond initial designs. Reinforcement learning (RL), particularly Deep RL (DRL), overcomes these issues using adaptive, goal-driven exploration through interaction with simulations, reducing data needs.
Despite its success in complex tasks, DRL’s application to photonic inverse design, especially optical modulators, remains limited.
This work introduces a Deep Q-Network (DQN) RL framework to optimize a silicon-based photonic crystal fiber modulator with an integrated VO2 phase-change layer, leveraging VO2’s reversible refractive index contrast. The goal is to minimize insertion loss and enhance modulation performance via geometry optimization.
DQN-RL Framework and PCF-OM Setup
The optical modulator under consideration is a D-shaped photonic crystal fiber (PCF) with a square lattice of air holes embedded in silicon. The polished, flat surface of the D-shaped fiber facilitates the integration of the VO2 layer near the fiber core, enabling efficient light-matter interaction.
A thin silica (SiO2) dielectric spacer separates the VO2 layer and the silicon substrate. Key geometrical parameters, such as the air-hole diameter, pitch between holes, vertical distance of the polished surface from the holes, and the thicknesses of the VO2 and SiO2 layers, define the discrete design space explored by the RL agent.
The VO2 phase-change material transitions from an insulating monoclinic state with low optical absorption to a conducting rutile state with strong plasmonic-like absorption, controlled by external stimuli. This transition results in distinct optical properties that enable modulation of the guided optical mode.
A Deep Q-Network reinforcement learning framework is employed in which an agent iteratively selects design parameters to interact with a full 3D FDTD simulation environment that evaluates the optical insertion loss (IL) for the proposed geometry. The agent is rewarded based on the IL, thereby guiding exploration toward minimum-IL designs. This framework eliminates the need for pre-collected training datasets by learning directly from simulation interactions.
The design space consists of discrete values for five key parameters, resulting in 243 (3^5) possible configurations. Simulations focus on the TM-polarized fundamental mode in the mid-infrared wavelength range around 5 μm, where silicon transparency and VO2 refractive indices are well characterized.
Want to save this article for later? Click here.
Comparisons are conducted against traditional optimization methods, including particle swarm optimization (PSO), random search, grid search, and Bayesian optimization, under identical computational budgets and resources, to benchmark the convergence speed and output quality of the DQN-RL method.
Fabrication tolerance analyses are also performed by varying geometrical parameters ±5% around the optimal design values to evaluate the robustness of the optimized modulator.
Optimization Performance and Analysis
The DQN-RL agent rapidly and stably converges to an optimal photonic crystal fiber modulator design, minimizing insertion loss within approximately 12 iterations (~3 minutes), outperforming PSO, random search, grid search, and Bayesian optimization.
The optimized device achieves an ultra-low insertion loss of 0.935 dB/mm in a compact 100 μm length, addressing key challenges in integrated photonics. Optical analysis shows strong TM mode confinement in the VO2 insulating phase (ON state) and significant attenuation due to VO2’s conducting phase (OFF state), confirming efficient switching and miniaturization viability.
The modulator exhibits an extinction ratio over 280 dB/mm and modulation depth of 99.9%, with broadband mid-infrared operation from 3 to 7 μm, suitable for sensing and communications. Fabrication tolerance tests reveal robust performance under ±5% parameter variations, maintaining insertion loss below 1 dB/mm, indicating manufacturability resilience.
Compared to state-of-the-art modulators, the DQN-RL design achieves lower loss and shorter length, excelling in discrete, constrained design spaces without gradient reliance. This study highlights reinforcement learning’s power to autonomously generate high-performance photonic designs by coupling with electromagnetic simulations, overcoming traditional methods’ data and tuning limitations.
Summary and Future Outlook
This study introduces a deep reinforcement learning (DRL) framework for inverse design of photonic crystal fiber optical modulators incorporating VO2 phase-change materials. By integrating a Deep Q-Network with 3D FDTD simulations, the method enables autonomous, dataset-free, gradient-free optimization.
These findings demonstrate the effectiveness of reinforcement learning in managing discrete, nonlinear, and high-dimensional photonic design problems. The proposed framework offers a scalable, efficient pathway for automating inverse design of advanced optical components, with potential applications beyond modulators.
Combining reinforcement learning with rigorous electromagnetic modeling presents a powerful solution to longstanding challenges, accelerating innovation in photonic device engineering.
Journal Reference
Dawood N.Y.M., Younis B.M., et al. (2026). AI-enhanced inverse design of photonic crystal fiber optical modulator using deep reinforcement learning technique. Scientific Reports 16, 19551. DOI: 10.1038/s41598-026-52039-zz, https://www.nature.com/articles/s41598-026-52039-z