I understand there are three questions at play here.
1. Why are they referencing the group $S_8$?
2. What is the subgroup they are referencing here?
3. How does this subgroup relates to the given problem.
1. In principle, because we have 4 cards and we care about their orientation (facing up or down), we can think there are 8 objects.
$A, \underline{A}, B, \underline{B}, C, \underline{C}, D, \underline{D}$.
All the possible ways to rearrange these 8 objects are the group $S_8$.
We are interested in what happens when we rearrange these objects following a particular set of rules.
2. What is the subgroup they are referencing here?
Naturally, we can only consider permutations that leave the natural front and back of the same card next to each other. For example, we can interpret the configuration $A, \underline{A}, B, \underline{B}$ as the card $A$ facing up, followed necessarily by its face down counterpart $\underline{A}$, followed next by the card $B$ facing up and then followed necessarily by its face down counterpart $\underline{B}$.
Other configurations that make sense are:
$A,\underline{A}, \underline{B},B$
$\underline{A}, A, B, \underline{B}$,
$\underline{A}, A, B, \underline{B}$.
Configurations that would be impossible for the cards would be:
$A,\underline{B}, \underline{A},B$
$A,\underline{B}, B, \underline{A}$
and so forth. Notice then that we don't need to specify a complete pair $\underline{A}, A$ or $A, \underline{A}$ but just the leading card: either $\underline{A}$ or $A$.
Within this set of allowed configurations, we consider the subgroup generated by three operations $(\kappa, \tau, \phi)$ that take an allowed configuration and return an allowed configuration. This is clearly a subset of all the possible permutations $(S_8)$ and because we care about the generated subgroup, it will be itself a group (and hence, a particular subgroup of all the possible permutations in $S_8$). This generated subgroup is called $H$.
3. The subgroup $H$ referred to at the beggining of page $385$ corresponds to the group generated by
$\kappa, \tau$ and $\phi$. Notice that operation (ii) can be thought of as the application of $\kappa$ twice in a row. That operation (iv) can be thought of as the application of $\kappa$ three times in a row. That operation (iii) and (v) are the application of $\tau$ once, and that operation (vi) is the application of $\phi$ once.
The indiscriminate application of operations (ii) to (vi) is nothing but the continuous application of the permutations $\kappa, \tau$ and $\phi$. This is why it is necessary to consider the whole subgroup generated by them.
Final comment: Operations (i) and (vii) are addressed in proposition 3 and just before proposition 3, respectively.
This was a detailed answer, so a tip would be greatly appreciated.
Edit:
I'm asking for the specific permutations that would work for this trick/question.
The specific permutations referenced on the top of page 385 are listed below:
$\kappa(A, \underline{A}, B, \underline{B}, C, \underline{C}, D, \underline{D}) = (C, \underline{C}, D, \underline{D},A, \underline{A}, B, \underline{B})$
$\tau(A, \underline{A}, B, \underline{B}, C, \underline{C}, D, \underline{D}) = (\underline{B}, B, \underline{A}, A, C, \underline{C}, D, \underline{D})$
$\phi(A, \underline{A}, B, \underline{B}, C, \underline{C}, D, \underline{D}) = (\underline{D}, D, \underline{C}, C, \underline{B}, B, \underline{A}, A)$
Clearly $\kappa, \tau, \phi \in S_8$ and $H = <\kappa, \tau, \phi >$ is the subgroup generated by these three elements.
It is proven in the paper that $C_0$ and $C_1$ are set invariants under the action of $H$: $H(C_0) = C_0$, $H(C_1)=C_1$.
The first step in the trick is to land the arrangement of cards into the intersection of $C_0$ and $C_1$. Then, under the action of $H$, the resulting arrangement is still in $C_0 \cap C_1$.
The final proposition proves that under the first step, the final result is guaranteed.
Markgvanzalk
