Is physics considered the hardest field of science

Is that true?
by definition of hard I mean the most difficult to do well in.
I want to become a scientist and I would just like to know what other scientists think.
Thank You.

It depends really. Physics of course is the most mathematically-orientated science and therefore you will need a good grasp of mathematics if you want to do well in. Chemistry is considered the hardest by some people, and biology by others. It all depends on who you are, some will find some sciences easier than others.
Personally, I find physics fine to learn but hard in exams. Biology I find is more memorisation. Chemistry is regarded as most difficult by people in my class. I like chemistry, even though I do see how it is difficult I am prepared to work because I enjoy it.

difficulty is decided by what you do easily. if complex math is difficult for you but you are still good at putting puzzles together bio might be more your style, if the opposite is true then go with simple math. the scale works a little like this

biology<--->chemistry<---> physics<--->math

math is the most set in rules and biology is the most fluid. each sector goes to the one to its right to solve problems it has no solution to.

Some things in physics are just hard to get your head around, like time dilation and quantum mechanics that don't obey "common sense".

Not sure what you mean "do well". Get a Nobel prize ? Become a millionaire ? Get lots of papers cited and the acclaim of your peers ?
Just go with what you love.