Perform a rotation of axes on 3x^2 -2√(3xy) + y^2 + 2x +2√(3y) = 0

I'm not too sure about this problem.

Assuming you mean 3x^2 - 2√3 (xy) + y^2 + 2x + 2√3 (y) = 0:

Presumably, you want to rotate the axes so that the quadratic no longer has a cross product 'xy' term.

This is achieved as follows:
Look at the degree 2 terms: 3x^2 - 2√3 (xy) + y^2.

Choose t so that tan(2t) = -2√3 / (3 - 1) = -√3
==> 2t = -π/3 + π (so that the angle is positive)
==> t = π/3.

So, we set x = x' cos(π/3) - y' sin(π/3) and y = x' sin(π/3) + y' cos(π/3)
==> x = (x' - y'√3)/2 and y = (x'√3 + y')/2.

Link/Reference:
http://en.wikipedia.org/wiki/Rotation_of…
--------------------------------------…
Substituting this into our given quadratic yields
3 [(x' - y'√3)/2]^2 - 2√3 [(x' - y'√3)/2] [(x'√3 + y')/2] + [(x'√3 + y')/2]^2 + 2[(x' - y'√3)/2]
+ 2√3 [(x'√3 + y')/2] = 0

This simplifies to 4x' + 4(y')^2 = 0
==> x = -(y')^2, a parabola.

I hope this helps!