Well, I've thought about it, and now I've finally figured out what was wrong with it.
Consider this function:
function RotateX(firstvertex,lastvertex,rotate, centerX, centerY, centerZ)
for i = firstvertex, lastvertex do
vertex3dY[i] = vertex3dY[i] - centerY
vertex3dZ[i] = vertex3dZ[i] - centerZ
vertex3dY[i] = (vertex3dY[i] * math.cos(rotate)) - (vertex3dZ[i] * math.sin(rotate))
vertex3dZ[i] = (old3dy * math.sin(rotate)) + (vertex3dZ[i] * math.cos(rotate))
vertex3dY[i] = vertex3dY[i] + centerY
vertex3dZ[i] = vertex3dZ[i] + centerZ
end
end
This rotates all the vertices in the object around a point defined by coordinates centerX, centerY, and centerZ.
These two lines are where I found the problem:
vertex3dY[i] = (vertex3dY[i] * math.cos(rotate)) - (vertex3dZ[i] * math.sin(rotate))
vertex3dZ[i] = (vertex3dY[i] * math.sin(rotate)) + (vertex3dZ[i] * math.cos(rotate))
Can you spot what it is?
It took me ages to figure out... but basically, notice how the two results of these two lines are actually referenced by each other.
For example, before these lines are run, vertex3dY[ i] will have a particular value. After the first line has been run, vertex3dY[ i] will have a new value. Then, in the second line, we refer to vertex3dY[ i] in making our calculation. But it has a new value now!
This was leading to a not-terribly-gradual decay in the size of the cube, leading to some interesting dimension effects... but certainly it did not look right.
So, I have made the following change:
old3dy = vertex3dY[i]
vertex3dY[i] = (vertex3dY[i] * math.cos(rotate)) - (vertex3dZ[i] * math.sin(rotate))
vertex3dZ[i] = (old3dy * math.sin(rotate)) + (vertex3dZ[i] * math.cos(rotate))
This has fixed the rotation about the X axis - I've had a cube spinning at light speed for 5 minutes now without getting any smaller or flatter. I now go to implement the same fix on the Y axis and Z axis rotation functions. :>