Optimisation Problem

I didn't check the previous work you referenced in the math stackexchange answer, so I assume everything is correct up to the point of your questions.

Question 1. Could you please explain what the person means by "vanishing determinant condition".

In their answer they have stablished that if the $3x3$ matrix (in terms of $\rho, s, \lambda$) is invertible, this would lead to an inadmissible answer $(\rho = 0$ and hence $h=0)$. 

Therefore, the matrix must not be invertible. This is equivalent to say that its determinat is zero. This is what they mean by vanishing determinant condition.

Question 2. how does he derive that $h=2\sqrt{(2r+R)(r+2R)}$  

The determinant condition is $\frac{\pi^3}{3}\rho \sin(\theta) \cos^2(\theta) (\rho-3s) =0$.

This can only happen if or $\cos(\theta) =0$ or $\rho = 3s$ or $\rho \sin(\theta) =0$ .

$\cos(\theta) =0$ would lead to $h=0$, which has already been ruled out as an impossible condition, so we ignore this possibility.

The last condition (first case) is the one that implies that $h=2\sqrt{(2r+R)(r+2R)}$.

To see this, we go back to the change of variables equations and replace $\rho$ by $3s.

$h = 3s \cos(\theta)$
$ r = \frac{s+3s\sin(\theta)}{2} = \frac{s(1+3\sin(\theta))}{2}$
$ R = \frac{s-3s\sin(\theta)}{2} = \frac{s(1-3\sin(\theta))}{2}$

Notice that since $h>0$, we can say that $\cos(\theta) = \sqrt{1-\sin^2(\theta)}$. Also, notice the sign complementarity relations between $r$ and $R$ in terms of signs. This is what motivates them to compute

$(2r+R)(2R+r) = \frac{9s^2}{4}\left(1-\sin^2(\theta)\right)$, so that $2\sqrt{(2r+R)(r+2R)} = 3s\sqrt{1-\sin^2(\theta)} = 3s\cos(\theta) = h$.

Question 3. Explain the second case because I don't understand why he says "return to the cartesian system"

The second case is when $\rho \sin(\theta) = 0$. We discard the possibility of $\rho =0$ as it would lead to $h=0$.

Going back to the change of variables equations and replacing $ \sin(\theta) = 0$ we get that

$r = \frac{s}{2} = R$.

With this equation stablished, we can go back to the partial derivatives equation at the very beggining of the solution (these original equations are equivalent to the second set of equations in the new variables):

$\Lambda_r = \pi h + \pi h \lambda r +2\pi r = \pi(h \lambda r + h + 2r)= 0$
$\Lambda_R = \pi h + \pi h \lambda r +2\pi r = \pi(h \lambda r + h + 2r)= 0$
$\Lambda_{h} = 2\pi r +\pi \lambda r^2 = \pi r(2 \lambda r) = 0$
$\Lambda_{\lambda}= \pi h r^2 - V_0 = 0$