The system is detailed below. In a traditional coronograph (i.e. one not employing a vortex phase mask), the mask in focal plane FP1 is a zero light-transmitting block of very small angular extent. Because the image of a star that the system is pointed at is formed in plane FP1, its light is filtered out of the final image by this mask. The Lyot stop in plane PP2 then blocks the light from the star that is diffracted by the mask. The resulting intensity collected in plane FP3 is largely contributed to by any point source near the star, e.g. an exoplanet.
What is not clear to me is why replacing the block in FP1 by a vortex phase mask improves the performance of the coronograph. Mathematical arguments are presented, but I find an intuitive explanation lacking.