We present a theoretical study of the optical properties of TE- and TM- modes in a four-layer structure composed of a magneto-optical yttrium iron garnet guiding layer on a dielectric non-magnetic substrate covered by a planar nanocomposite guiding multilayer. We examine important issues for TM-modes, calculate and compare in details the dispersion spectra and the energy flux distributions across the structure for TE- and TM modes of different orders and show new features concerning the splitting, switching and filtration possibilities of the fundamental modes of the orthogonal polarizations. The presented theoretical approach may be utilized for designing of different magneto-optical devices with preselected optical properties.