A linear transformation P on a vector space V is called a projection if P2 = P.
Let V be a Hilbert space. An orthogonal projection P is a projection such that ⟨Px, y⟩ = ⟨x, Py⟩
Note: Annoyingly, orthogonal
projections are not orthogonal matrices. They're Hermitian matrices. The
terminology comes from the fact that the image and kernel are
orthogonal as subspaces.
Let P be an orthogonal projection. Then ker ⊕ Im ≅ V
Let x ∈ V and write x = Px + (x−Px) ∈ ker + Im .
And if x ∈ ker ∩ Im , we would have x = Px = 0.
Using Cauchy-Schwarz, $\langle Px, Px \rangle = \langle P^{2} x, x \rangle = \langle Px, x\rangle\leq \sqrt{\langle Px, Px \rangle \langle x,x\rangle}$
So $\| Px \| = \sqrt{\langle Px, Px \rangle} \leq \| x \| $.