Applying a 3x3 matrix to a set of 4 coordinates

I am guessing that you are applying a linear transformation to map the 4 corners of a square. The linear transformation is a $3 \times 3$ matrix, say $A$. Coordinates of the square, $P_1, P_2, P_3, P_4$ are points in $\mathbb{R}^3$. In order to find the transformation of the square, you need to find transformation of the corners. Indeed you need to compute
\[PAP_i,   i=1,2,3,4\]
which are the transformations of the corners of the square under the map $A$. Note that the above matrix multiplication makes sense, and we are applying a $3 \times 3$ matrix, by a vector in $\mathbb{R}^3$.