Prove that (0,1)x(0,1) is an open set of R^2

The order of the quantifiers is backwards there (which might be part of the problem). For all (x,y) in (0,1) x (0,1), you want to find an epsilon > 0 with the property that the disc with radius epsilon and center (x,y) is a subset of (0,1)x(0,1). (As you have ordered it, the same single epsilon has to work for *all* (x,y). And it's not possible to find an epsilon that does this, since that rectangle contains points arbitrarily close to the origin, for example.) To save space I'll write G for the set (0,1)x(0,1).

It should be clear that whatever (x,y) in G that we pick, the largest "epsilon" that has any chance of working is the distance from (x,y) to the boundary of the rectangle. This distance is easy to evaluate for particular points (e.g. when (x,y) = (1/3, 1), it is clearly 1/3) but we want a formula that works for all points. But with some work, you should be able to convince yourself (with a picture or otherwise) that the distance from (x,y) to the boundary is the smallest of the numbers x, y, 1-x, 1-y. To prove that G is open, we just need to prove that this choice of epsilon works.

So, fix (x,y) in G, and let

epsilon = min(x,y,1-x,1-y).

Note that since 0 < x < 1 and 0 < y < 1, we have 0 < 1-x < 1 and 0 < 1 - y < 1, so each of the numbers 1, x, 1-x, 1-y is positive, and so epsilon, the minimum of a finite set of positive numbers, is positive. We will now show that the open disc with center (x,y) and radius epsilon is a subset of G.