Find the following limit (involves non-integratable integrals)

I'm not sure exactly how that equation is written (looks like a muddled mess to me, sorry!); however! Try integration by parts.
http://en.wikipedia.org/wiki/Integration_by_parts

Essentially, you split the integral into two sections: one that is easily integrated, the other is more easily derived. For example, (2x)ln(x) could be split to where f(x) = ln(x) and g(x) = 2x, and then substituted into the generic formula. I hope that helps!

It does not exist. Taking lines closer and closer to the line x=y, you need to get closer and closer to (1, 2) before the expression gets near 0. (Of course, the expression is undefined when b - a - 1 = 0, though even calling it 0 on this line isn't enough.)

More rigorously, the functions e^(x^2) and e^(y^10) are continuous and non-zero everywhere, so we may choose a and b close enough to 1 and 2 that on (1, a), 1/2 e^(1^2) <= e^(x^2) <= 3/2 e^(1^2) and on (2, b), 1/2 e^(2^10) <= e^(y^10) <= 3/2 e^(2^10). Integrating these inequalities, for each a and b there is some constant c_(a, b) such that for some m and M, 0 < m < c_(a, b) < M and

∫ e^(x^2) dx (from x=a to 1) * ∫ e^(y^10) dy (from y=b to 2)
= c_(a, b) * (1-a) * (2-b)

The original expression is then
c_(a, b) * (1-a)(2-b) / (b - a - 1)
= c_(a, b) * (1-a)(2-b) / ((b-2) + (1-a))
= c_(a, b) * xy / (x - y)

where x = 1-a, y = 2-b. The limit as (x, y) -> (0, 0) of xy / (x-y) behaves as I described above, and the constant factor doesn't change this behavior in an essential way.