Would it be considered beastiality if a human and a humanoid alien had sex

Hmm, well as bestiality is defined as sexual relations between humans and animals, the sexual act with the alien would not be classed as bestiality because it is not an animal. Outside this strict boundary of the definition, I would call it bestiality as the alien is humanoid as you state and would likely be classed as an animal if it lived on earth !
The sentience of the animal species in question does not determine whether or not the act is classed as bestiality.