Questions about using matrices for finding best straight line by linear regression

I'll try to answer both your questions at once as they are getting at the same core point. The key observation, which you've made in the second question is "the measured points will not be exactly on the straight line". That is correct they will not be exactly on the straight line. In fact, linear regression in general consists of an overdetermined system where there are more equations than unknowns. In your simple example $n = 3$ but there are only $p = 2$ observations. From basic algebra we know that overdetermined systems almost always have no solutions. This is exactly the fact you stated when you said the points will not be on the straight line.

With that in mind, we need to ask the question "what are we actually doing when we are constructing a least squares line?". The answer is partially in the name. Let $\hat{M}$ be our coefficient estimates, then we have an estimate of $Y$ given by $\hat{Y} = X\hat{M}$. In least squares, we seek to minimize the sum of squared residuals. In matrix form, we can write this as:

$$\min_M (Y - XM)^T (Y - XM) = \min_M (Y - \hat{Y})^T (Y - \hat{Y}) = \min_M \sum_{i=1}^n (y_i - \hat{y}_i)^2 $$

So in least squares, we are trying to find a coefficient vector $M$ that minimizes the above expression. Now to understand why the solution to this problem is given by $\hat{M} = (X^T X)^{-1} X^T Y$, we minimize the above expression using standard calculus techniques of taking the gradient and setting it equal to zero. I won't go into those details here but they are commonly found, e.g. in this wikipedia article.

The process taking $Y = XM$ and solving for $M$ by multiplying both sides by $X^T$ and rearraging is just something to be considered for intuition. The more rigorous way to solve the expression is via minimization is described above. Some "reasons" to consider on why we multiply by $X^T$ and not some other matrix is that the quadratic form $X^T X$ is at least positive semi-definite and in many cases, it is positive definite. If $X^T X$ is positive definite, then it is guaranteed to have an inverse, allowing us to even rearranging the expression using $(X^T X)^{-1}$. Another point to be made is that we can write:

$$Y = XM \implies Y - XM = 0 \implies X^T(Y - XM) = 0$$

The meaning of the last equation on the right is that the residual $Y - XM$ is orthogonal to the column space of $X$. In some sense, this means that the difference between your estimate and the data is not contained in the data $X$. 

These sorts of intuitive points are not exactly straightforward to make precise. In summary, when you are finding the best straight line for some data, you are actually solving a minimization problem. It happens that the "intuitive" approach of multiplying by the transpose and rearranging gives a solution that corresponds exactly to the solution of the minimization problem. There are deep reasons for this correspondence, but it goes beyond the scope of this response.