One technique - there could be many others

String [ ] students = {"studentX", "studentY", "studentW", "studentP", "studentQ"}

since there are 5 students we will make an array of length 5 to keep track of each:

int [ ] quizzesTaken = {0,          0,           0,            0,       0}

Each time a student takes a test, the number in the quizzesTaken array will increase by 1.
For example, if studentY takes a test, since studentY is at students[1], then quizzesTaken[1] should increase:

                    quizzesTaken[1] = quizzesTaken[1] + 1;

Now, get a random number from 0 to 4 inclusive so we can pick a student to take the test:

                    int randomInt = (int)(Math.random()*5);

Suppose randomInt had a value of 3.  This would mean that we are considering students[3] or studentP to take a quiz. 
We want to see if studentP has already taken two quizzes...i.e.,  if (quizzesTaken[3] ==2)

Specifically, we give the student at randomInt ( students[randomInt] ) a quiz only if they have not already had two.

OK, suppose students[randomInt] has had two already (quizzesTaken[randomInt] !=2)
  in this case we would want to get another randomInt to try another student...and if that one had two we would try another!

So, rather than an if statement, we really want a while

    int randomInt = (int)(Math.random()*5);
    while (quizzesTaken[randomInt] == 2){
       randomInt = (int)(Math.random()*5);
    }

Only after we have decided on someone who has not had two quizzes do we choose their spot in quizzesTaken[] to increase:
    quizzesTaken[randomInt] = quizzesTaken[randomInt] + 1;

Thus, a student can never be picked to take more than 2 quizzes