Science

400. Dispersion Splitting of Phonon Polaritons in van der Waals Heterostructures

R
Raimundas Juodvalkis

Introduction

Phonon polaritons are hybrid quasiparticles arising from the strong coupling between electromagnetic waves and optical phonons in polar dielectric media. In the infrared and terahertz regimes, they offer a compelling route to confine light to deeply subwavelength volumes while retaining comparatively low dissipation. Among the broad class of phonon-polaritonic materials, van der Waals crystals have emerged as especially versatile because their anisotropic lattice structure enables natural hyperbolicity, strong in-plane anisotropy, and atomically sharp interfaces that can be integrated into layered heterostructures.

A particularly important material in this context is the biaxial van der Waals crystal α-phase molybdenum trioxide, α-MoO3. In its Type-I Reststrahlen band (RB-I), α-MoO3 supports hyperbolic phonon polaritons with anomalous dispersion. Unlike conventional isotropic polaritons, these modes can propagate with highly directional wavefronts and exhibit negative or strongly anisotropic group velocities depending on crystallographic orientation. Because the polaritons in α-MoO3 are long-lived and low-loss, they are attractive for nanophotonic waveguiding, sensing, and on-chip infrared signal processing.

Despite these advantages, dispersion engineering in the RB-I of α-MoO3 has remained relatively underdeveloped. Most prior work has focused on single-slab propagation, natural anisotropy, edge reflections, and geometric confinement. A more powerful route is to exploit interlayer coupling between adjacent polaritonic slabs. When two polaritonic eigenmodes are brought into close proximity, their electromagnetic fields overlap and hybridize, producing mode splitting analogous to coupled oscillators in quantum mechanics or supermodes in integrated photonics. In the polaritonic case, this hybridization can alter both the momentum and symmetry of the modes, creating distinct branches with different confinement and propagation characteristics.

This article examines the dispersion splitting of phonon polaritons in van der Waals heterostructures formed by two α-MoO3 slabs separated by a thin hexagonal boron nitride (hBN) spacer. We focus on the physical origin of the splitting, its experimental observation using scattering-type scanning near-field optical microscopy (s-SNOM), and the possibility of dynamically tuning the split branches by incorporating graphene into the heterostructure. The resulting platform enables active and mode-selective manipulation of hyperbolic phonon-polariton dispersion in the mid-infrared.

Hyperbolic phonon polaritons in α-MoO3

The optical response of α-MoO3 is strongly anisotropic because its crystal axes have different infrared-active phonon resonances. In the Type-I Reststrahlen band, one principal component of the permittivity tensor is negative while the other two remain positive. This condition yields a hyperbolic dispersion relation in which the isofrequency surface is open rather than closed. As a result, polaritons propagate with wavevectors that can become much larger than the free-space wavenumber, enabling extreme confinement.

In a single α-MoO3 slab, the polariton dispersion is governed by the slab thickness, crystal orientation, and boundary conditions at the interfaces. The fields are concentrated near the surfaces and decay evanescently into the surrounding media. The propagation direction is not isotropic; instead, it is determined by the in-plane anisotropy of the dielectric tensor. This leads to directional energy flow and characteristic interference patterns that can be directly imaged with near-field microscopy.

The unusual feature of the RB-I in α-MoO3 is that the polaritons are not merely confined surface waves but hyperbolic guided modes whose momentum can be tuned by thickness and environment. This makes α-MoO3 an excellent candidate for dispersion engineering through vertical stacking. If two slabs are placed close enough for their evanescent fields to overlap, the interaction can no longer be treated as independent propagation in separate layers. Instead, the system supports collective eigenmodes spanning both slabs.

Mechanism of dispersion splitting

The splitting of polaritonic dispersion in a double-slab structure is fundamentally an interlayer coupling effect. Each α-MoO3 slab supports its own set of phonon-polaritonic modes. When the slabs are separated by a thin dielectric spacer, the near fields penetrate into the spacer and overlap with the fields of the neighboring slab. This overlap produces hybridization, lifting the degeneracy that would exist if the slabs were isolated.

The resulting supermodes are analogous to bonding and antibonding states. One branch corresponds to an even symmetry of the electric field across the spacer, while the other corresponds to an odd symmetry. These symmetry classes lead to different boundary conditions and therefore different effective momenta. Typically, the symmetric mode has stronger field confinement in the spacer and thus a larger in-plane wavevector, whereas the antisymmetric mode can exhibit reduced confinement and a smaller wavevector. The exact ordering depends on frequency, thickness, and dielectric environment.

In a coupled-mode description, the dispersion splitting can be understood as the eigenvalue separation induced by the coupling coefficient between the two slabs. As the spacer thickness decreases, the coupling increases exponentially because the evanescent tail of the polaritonic field decays across the gap. Consequently, the momentum splitting becomes larger for thinner spacers. If the spacer is thick enough, the modes decouple and the dispersion branches collapse back toward the single-slab dispersion.

This mechanism is especially significant in the RB-I of α-MoO3 because the mode lifetime is long enough for coherent hybridization to be experimentally resolved. In more lossy polaritonic systems, the splitting would be obscured by damping. Here, however, the low intrinsic loss allows the emergence of distinct branches with measurable near-field contrast.

Heterostructure design with hBN spacer

Hexagonal boron nitride is an ideal spacer material for constructing α-MoO3 heterostructures. It is chemically stable, atomically flat, and supports its own phonon-polaritonic response in the mid-infrared, although within a different spectral window. For the present application, hBN serves primarily as a dielectric spacer that controls the interlayer separation and mediates the coupling between the two α-MoO3 slabs.

The use of hBN offers several advantages. First, its van der Waals nature allows clean stacking without dangling bonds or significant interfacial contamination. Second, its thickness can be precisely controlled at the few-nanometer scale, which is critical because the coupling strength depends sensitively on spacing. Third, hBN has a relatively high dielectric constant compared with air, which modifies the effective mode profile and can further influence the splitting.

In a typical heterostructure geometry, two α-MoO3 flakes are aligned with their crystallographic axes chosen to maximize or tailor the interaction. The hBN spacer is inserted between them, and the entire stack is placed on a suitable substrate such as SiO2/Si or another infrared-transparent platform. The thicknesses of the α-MoO3 layers and the spacer define the hybrid mode spectrum. Because α-MoO3 is biaxial, the orientation of the in-plane crystal axes relative to the propagation direction adds another degree of freedom, allowing the split branches to be engineered not only by vertical coupling but also by lateral anisotropy.

Experimental observation by s-SNOM

The dispersion splitting of the coupled polaritons can be probed experimentally using scattering-type scanning near-field optical microscopy. In s-SNOM, a sharp metallic tip illuminated by infrared light acts as a nanoscale launcher and detector of evanescent fields. The tip locally converts incident radiation into high-momentum polaritons and simultaneously scatters the returning near fields into the far field, enabling spatial mapping of the local optical response far below the diffraction limit.

In the α-MoO3/hBN/α-MoO3 heterostructure, s-SNOM reveals interference fringes and standing-wave patterns associated with the hybrid polaritonic modes. By scanning across the sample and analyzing the near-field amplitude and phase, one can extract the polariton wavelength and infer the in-plane momentum. The key experimental signature of mode splitting is the observation of two distinct propagation constants under the same frequency conditions, corresponding to the two hybrid branches.

Near-field images show that the hybrid modes can possess different symmetry profiles across the stack. Depending on the excitation geometry and detection channel, one branch may dominate the response because the tip couples more efficiently to one symmetry class than the other. This mode selectivity is important because it demonstrates that the splitting is not merely a theoretical artifact but a physically accessible feature of the heterostructure.

The experimental results confirm that the interlayer interaction between α-MoO3 slabs is strong enough to alter the polariton dispersion in the RB-I. The measured splitting is consistent with the expected dependence on spacer thickness and dielectric environment, validating the coupled-mode picture. Importantly, the observation establishes that phonon-polariton hybridization can be realized in a regime previously considered difficult to tune.

Theoretical framework for hybrid dispersion

A useful way to model the system is to treat each α-MoO3 slab as a polaritonic waveguide supporting discrete guided modes. The coupling between the slabs can then be described by a 2 × 2 eigenvalue problem in which the uncoupled mode dispersions are perturbed by an off-diagonal interaction term. The eigenfrequencies or wavevectors of the supermodes are obtained by diagonalizing the coupled system.

In the limit of weak coupling, the dispersion splitting is approximately proportional to the overlap integral of the electromagnetic fields in the spacer region. Because the fields decay exponentially away from each slab, the coupling scales roughly as exp(-kz d), where kz is the out-of-plane decay constant and d is the spacer thickness. This exponential dependence explains why even nanometer-scale changes in hBN thickness can produce sizable shifts in the polariton momentum.

The biaxial nature of α-MoO3 adds a further layer of complexity. The in-plane dispersion is not circular but elliptical or hyperbolic depending on the frequency and propagation angle. Therefore, the coupling-induced splitting can vary with direction, leading to anisotropic supermode surfaces in momentum space. This makes the heterostructure a platform for directional dispersion control, where the branch separation depends on both vertical and lateral anisotropy.

From a broader perspective, the coupled α-MoO3 system represents a photonic analog of band splitting in solid-state physics. The two branches can be viewed as hybridized polaritonic bands whose separation is controlled by structural parameters rather than by electronic interactions. This analogy suggests that concepts such as avoided crossing, symmetry-protected modes, and band engineering can be transplanted into phonon-polaritonic nanophotonics.

Graphene-enabled active tuning

A major advantage of van der Waals heterostructures is the possibility of incorporating graphene as an active element. Graphene supports tunable plasmonic response in the infrared, with its conductivity strongly dependent on Fermi energy. By integrating graphene into the α-MoO3/hBN/α-MoO3 stack, one can modify the electromagnetic boundary conditions and thereby tune the polaritonic dispersion dynamically.

The physical mechanism is twofold. First, graphene changes the effective dielectric environment experienced by the polaritons. Second, if the graphene plasmon resonance is spectrally close enough, it can hybridize with the phonon-polaritonic modes, producing additional anticrossings or branch reshaping. By electrostatically gating graphene, the Fermi energy can be adjusted in situ, enabling active control of the mode splitting.

This tunability is especially valuable for mode-selective engineering. Depending on the graphene doping level, one branch may be shifted more strongly than the other, or one symmetry class may couple preferentially to the graphene plasmon. As a result, the dispersion splitting can be enhanced, suppressed, or made frequency selective. Such control is difficult to achieve in conventional dielectric structures and highlights the flexibility of van der Waals integration.

The graphene element also introduces a route to reconfigurable polaritonic circuits. By patterning gates or using local electrostatic control, one could envision spatially varying the splitting across a device, thereby creating polaritonic waveguides, switches, or resonant cavities based on hybridized phonon-polariton bands.

Implications for dispersion engineering

The observation of dispersion splitting in α-MoO3 heterostructures has broad implications for nanophotonics. First, it demonstrates that hyperbolic phonon polaritons are not fixed by material anisotropy alone; their dispersion can be engineered through interlayer coupling. Second, it provides a route to creating multiple polaritonic channels within a single spectral band, which could be useful for multiplexing and directional routing. Third, the symmetry dependence of the split branches opens opportunities for selective excitation and detection.

Because the splitting is controlled by spacer thickness and dielectric composition, it can be designed with nanometer precision. This is particularly important for integrated devices, where compactness and reproducibility are essential. The heterostructure approach also enables combination with other two-dimensional materials, including semiconductors, metals, and moiré systems, expanding the design space beyond simple two-layer stacks.

More generally, the work suggests that phonon-polariton mode splitting should be expected in a wide range of van der Waals systems whenever evanescent fields overlap across a thin barrier. Similar physics could arise in other biaxial or hyperbolic crystals, as well as in multilayer stacks with alternating polaritonic and dielectric layers. The concept therefore extends beyond α-MoO3 and may become a general strategy for tailoring mid-infrared dispersion.

Conclusion

Dispersion splitting of phonon polaritons in α-MoO3 heterostructures represents a significant advance in the control of hyperbolic light-matter states. In the Type-I Reststrahlen band, α-MoO3 supports low-loss hyperbolic phonon polaritons with strong anisotropy and long propagation lengths. When two α-MoO3 slabs are brought into close proximity and separated by an hBN spacer, their polaritonic eigenmodes hybridize, producing two distinct branches with different momenta and field symmetry.

This splitting has been experimentally observed using s-SNOM, which directly resolves the near-field interference and reveals the coexistence of hybrid supermodes. The results confirm that interlayer coupling can be used to engineer polaritonic dispersion in a regime previously dominated by single-slab behavior. Moreover, the incorporation of graphene provides a path toward active tuning through electrostatic control of the Fermi energy, enabling dynamic and mode-selective manipulation of the branches.

The broader significance of this work lies in its demonstration that van der Waals heterostructures can function as programmable polaritonic media. By combining anisotropic phonon-polaritonic crystals, dielectric spacers, and tunable electronic layers, one can design infrared optical states with tailored momentum, symmetry, and confinement. This opens a pathway toward reconfigurable nanophotonic components based on hyperbolic phonon polaritons, including switches, filters, interferometers, and compact waveguides. Dispersion splitting in α-MoO3 is therefore not only a fundamental physical phenomenon but also a foundational tool for next-generation polaritonic engineering.