The exponential growth of data traffic in the information age has imposed a strong demand for significantly increasing the capacity of optical fiber communication. Conventional single-mode fiber (SMF) systems based on wavelength division multiplexing (WDM) are approaching their theoretical capacity limit. Space division multiplexing (SDM) technology enhances transmission capacity by developing the spatial dimension of optical fibers, making it a highly promising solution. Among various SDM fibers, few-mode multi-core fiber (FM-MCF) shows significant potential due to its high scalability in the number of spatial channels and supported modes. However, the practical deployment of high-capacity FM-MCF systems faces multiple key challenges: the adverse effects of inter-core crosstalk (ICXT) and inter-mode crosstalk (IMXT), and the limitations imposed by optical fiber nonlinear effects under high power levels. ICXT severely compromises signal integrity between adjacent cores, leading to demodulation errors at the receiver, while IMXT within a single core causes signal distortion, forcing the system to adopt expensive compensation techniques. In addition, nonlinear effects, such as self-phase modulation and four-wave mixing, are more pronounced in SDM fibers, causing signal degradation and limiting achievable power. To mitigate these nonlinear impairments, it is crucial to increase the effective mode area ( A_\mathrmeff ) and reduce the nonlinear coefficient ( \gamma ). Therefore, the primary objective of this study is to design an FM-MCF structure that can simultaneously achieve: (a) suppression of ICXT and IMXT through core isolation technology and enhanced effective refractive index difference ( \Delta n_\mathrmeff ), and (b) increase of A_\mathrmeff and reduction of \gamma to minimize nonlinear coefficient. This dual goal will be achieved by strategically optimizing the core radius and the refractive index distribution of the core material during the optical fiber design stage. Achieving a large A_\mathrmeff requires adjusting the core size, while lowering \gamma involves tailoring the core material’s nonlinear refractive index. The design of such a novel FM-MCF structure that can effectively suppress crosstalk and nonlinearity is crucial for breaking through current bottlenecks and fully releasing the high-power and high-capacity potential of SDM technology.

The finite element method (FEM) was employed to simulate and analyze the optical characteristics of heterogeneous few-mode multi-core fibers assisted by trench and air-hole arrays with central nanopores. The transmission characteristics were modeled and calculated based on wave optics and power coupling theory. The influence mechanisms of key parameters were systematically investigated: the effect of bending radius on ICXT, the suppression patterns of three types of microstructures (trench width, air-hole size and distribution, core spacing) on ICXT, the mechanism of central nanopore size regulating IMXT through effective refractive index difference ( \Delta n_\mathrmeff ), and the synergistic effects of operating wavelength and nanopore parameters on effective mode area ( A_\mathrmeff ) and nonlinear coefficient ( \gamma ). Through comprehensive comparison of simulated data (ICXT, IMXT, \Delta n_\mathrmeff , A_\mathrmeff , \gamma ) under different structures and working conditions, based on COMSOL Multiphysics® simulations, the optimized optical fiber design scheme achieving low crosstalk and low nonlinearity was determined and verified.

Through comprehensive simulation and numerical analysis verification, the optimized structure of heterogeneous few-mode multi-core fiber assisted by trench and air-hole arrays successfully achieved the coordinated optimization of low crosstalk and low nonlinearity. Critical design rules were established: inter-core crosstalk (ICXT ) < −45 dB when bending radius > 50 mm and transmission distance was 100 km (Fig. 4). ICXT was minimized by optimizing microstructure parameters: trench width of 3 μm (Fig.6b), air-hole radius of 2.5 μm (Fig.6c), and core spacing Λ=38 μm(Fig.6a). The central nanopore radius of 0.4 μm resulted in an effective refractive index difference \Delta n_\mathrmeff > 2.1 × 10 ⁻³ between adjacent LP modes (Fig.7), significantly enhancing the suppression capability for inter-mode crosstalk (IMXT). In the 1530 nm~1630 nm band, effective mode area A_\mathrmeff > 124 μm² (Fig.9) and nonlinear coefficient \gamma < 0.90 W⁻¹·km⁻¹ (Fig.10) were achieved. Experimental data confirmed that ICXT remained below −45 dB and \Delta n_\mathrmeff > 2.0×10⁻³ in the C+L band. The A_\mathrmeff under this fiber structure was significantly improved compared to traditional homogeneous multi-core fibers (MCF). This heterogeneous design exhibited dual core advantages: large mode field area reducing nonlinear effects, and high \Delta n_\mathrmeff ensuring mode stability. Ultimately, by determining appropriate structural parameters, stable transmission with low crosstalk and high capacity in the SDM system was achieved.

This study successfully designs a heterogeneous few-mode multi-core fiber with low crosstalk and low nonlinearity, which enables stable transmission of three LP modes (LP₀₁, LP₁₁, LP₂₁) across the C+L band. Through structural optimization, this optical fiber achieves key performance indicators required for conventional installation. Specifically, ICXT is below −345 dB at a bending radius of 50 mm and a transmission distance of 100 km, ensuring reliable spatial channel isolation and reducing interference between cores. By reasonably introducing intra-core nanopores and parameter optimization, the effective refractive index difference ( \Delta n_\mathrmeff ) between modes exceeds 2.1×10⁻³, effectively suppressing IMXT and enabling stable mode transmission. Meanwhile, the fiber has a large effective mode area (A_eff > 124 μm² for all modes) and a low nonlinear coefficient (γ< 0.90 W⁻¹·km⁻¹), significantly mitigating nonlinear effects such as four-wave mixing during long-distance transmission. Most importantly, the fiber’s relative core multiplicity factor (RCMF) reaches 26, with a capacity 26 times that of conventional single-mode fiber (SMF). These comprehensive performance characteristics—including good crosstalk suppression, excellent nonlinear anti-interference ability, and high spatial mode density—can meet the high bandwidth and long-distance transmission requirements of SDM systems. This provides an advanced and feasible solution for the next-generation optical networks that combine high capacity and stable transmission.