LOL I remember first year special relativity. Dr Barnham would so often start off with "Kirk and Spock are approaching each other at a relativistic velocity..." and bang, more equations for the next hour to describe what's happening.
Yes, observation doesn't need math in itself, but then you need to try and explain WHAT you're seeing - math again.
Sorry, but if playing around with algebra and stuff doesn't come easily to you by now, it probably never will. You could get better at it with hard work but really, it might be better to aim at something you CAN do. People either tend to find math something their brain naturally gets on with or they don't. I'm definitely one of the first kind, but even so, there are better people than me, and when in the second year of university we did second order partial differential equations in three dimensions, that was when I started to feel in over my head. But a mathematically-orientated degree made it reasonably easy to train as an accountant, and that's been fun in its own way.
Still, there's a lot you can do as an amateur. Amateur astronomers can often do things the professionals can't. One particular field they do well in is discovering comets and supernovae. The professionals are using big telescopes that only image a tiny part of the sky at a time, and they need to book time to look at something specific. Their work has to be planned. But a new comet or supernova could pop up anywhere and that's where an amateur with a small telescope can score one over them!