Show that $tr(\sqrt{\sqrt A B \sqrt A})\leq 1$ , where both $A$ and $B$ are positive semidefinite with $tr(A)=tr(B)=1.$

Show that
\[ tr(\sqrt{\sqrt A B \sqrt A})\leq 1,\]where both $A$ and $B$ are positive semidefinite with $tr(A)=tr(B)=1.$

Answer

First we show that 
\[tr (\sqrt{\sqrt{A} B \sqrt{A}})=tr (\sqrt{AB}). \]
Note that $\sqrt{A} B \sqrt{A}$ and $AB$ are similar matrices, since $AB = \sqrt{A} (\sqrt{A} B \sqrt{A}) \sqrt{A}^{-1}$. Hence
\[\sqrt{AB}=\sqrt{A} \sqrt{\sqrt{A} B \sqrt{A}} \sqrt{A}^{-1}.\]
 By similarity $\sqrt{AB}$ has the same eigenvalues as $\sqrt{\sqrt{A} B \sqrt{A}}$, and hence the same trace. 

Thus it is enough to shwo that $tr (\sqrt{AB})\leq 1$.

For a positive semidefinite matrix $C$, $|C|$ is defined as $|C|=(C^*C)^{1/2}.$  On the other hand by the polar decomposition there is a unitary matrix $U$ such that $C=U|C|$ and consequently $U^*C=|C|.$ Hence for $C=B^{1/2}A^{1/2} $ we have $$(A^{1/2}BA^{1/2})^{1/2}=|B^{1/2}A^{1/2}|.$$ Thus $$U^*B^{1/2}A^{1/2}=|B^{1/2}A^{1/2}|$$ for a unitary matrix $U.$ Let $\{e_n\}_{n=1}^d$ be an orthonormal basis and $f_n=Ue_n.$ Then $\{f_n\}_{n=1}^d$ is an orthonormal basis and

$$ {\rm tr}\, |B^{1/2}A^{1/2}|={\rm tr}\, [U^*B^{1/2}A^{1/2}]=\sum_{n=1}^d \langle U^*B^{1/2}A^{1/2}e_n,e_n\rangle \\ =\sum_{n=1}^d \langle A^{1/2}e_n,B^{1/2}Ue_n\rangle = \sum_{n=1}^d \langle A^{1/2}e_n,B^{1/2}f_n\rangle.$$

By ppplying the Cauchy-Schwarz inequality twice we get

$${\rm tr}\, |B^{1/2}A^{1/2}|\le \sum_{n=1}^d \| A^{1/2}e_n\|\|B^{1/2}f_n\| \\ \le\left (\sum_{n=1}^d\| A^{1/2}e_n\|^2\right )^{1/2} \left (\sum_{n=1}^d\| B^{1/2}f_n\|^2\right )^{1/2} \\ =({\rm tr}\,A)^{1/2}({\rm tr}\,B)^{1/2}= 1.$$

The answer is accepted.