How to determine the stability of an ODE

OK, I'll give $2 worth of an answer! :-)

First, you're wrong in that "if the derivative is zero, the derivative of the derivative is also zero".
For example, consider $f(x) = x^2$. At $x=0$ you have $f'(x) = 0$, but $f''(x) = 2$ .

Second, you have to distinguish the derivative $y'$ of the function $y:t\mapsto y(t)$, with respect to $t$,
from the derivative $f'(y)$ of the function $f(y)$, with respect to $y$.
(If you consider the derivative of $f(y)$ with respect to $t$, you get $\frac d{dt} f(y) = f'(y)\cdot y'$!)

Now, consider the example $y' = f(y)$ with $f(y) = c\,y$, where $c$ is any real constant.
We know that the solution to this ODE is $y(t) = A\,e^{c\,t}$ with any arbitrary $A=y(t=0)\in\R$.
You do have $y' = f(y)=0$ at $y=y^*=0$, but $f'(y^*) = c$ which can be positive or negative.

We know that for $c<0$, the solution $y(t)=A\,e^{c\,t}$ tends to zero as $t\to\infty$, for any starting value $A=y(t=0)$.
In that case, $y^*=0$ is stable: if $y$ is slightly different from $y^*=0$, it will tend to $y=0$.
On the other hand, if $c = f'(y^*=0) > 0$, then the equilibrium point $y*=0$ is unstable:
$y(t)$ will tend to infinity, as $t\to\infty$, whenever it (or $A = y(t=0)$) is even a little bit different from $y^*=0$.

A satisfying explanation of the general case would require a more elaborate answer which I can't give here. (The margin is too small, as would say Fermat.) But it's not that far from the preceding simple example because for studying stability, you actually linearize the system (~Taylor series) because you're interested in what happens in a small neighborhood of the critical point. I think you can find the missing details in Wikipedia or (re-)ask the question here with more incentive for a more elaborate answer. :-)