What does this notation mean

For any number c, define:

fc (x) = min { (x - c) ^ 2 , (x - c - 2) ^ 2 }

What does this equation mean?

It means the minimum of the two given inputs. So min {4, 5} = 4.

min {(x - c)^2, (x - c - 2)^2} depends on the values for x and c. Let's assume x=5, and c=1, just for fun.

min {(5-1)^2, (5-1-2)^2} = min{16, 4} = 4.

Hope that helped.