Okay, don't panic. It's simpler than it looks. Looking at genNextRect, it takes the width and height of a rectangle as global parameters, gla2 and glb2, and sets them to the next smallest golden rectangle. Note, genNextRect assumes the gla2 and glb are already a golden rectangle. This is done by taking one of the dimensions and dividing by the golden ratio, which will result in a smaller length. The flip flag determine which dimension to shorten. It is the side with the longest length that is shortened. You will need to keep track of flip such that, when genNextRect is called, that the longest length of the rectangle is shortened. Alternatively, you could remove flip entirely and change the condition in genNextRect to select the longer edge. You may want to check that your rectangle's aspect ratio matches (or is close enough to) the golden ratio after calling genNextRect.
After you have called genNextRect you will need to draw it as demonstrated in the glBegin/glEnd block in display. The assignment requires you continue drawing rectangles until the rectangles are too small to display. So, all that's left to complete the assignment is finding the first golden rectangle in the screen and a loop that calls genNextRect and draws the new rectangle until the rectangle is too small to display.
thank you very much. I will do this with your help