Rudel joined the Luftwaffe in 1936 and began his military career as an air reconnaissance pilot. He flew on long-range reconnaissance missions over Poland. During 1940, and served as a regimental adjutant for the 43rd Aviators Training Regiment, based at Vienna.
In early 1941, he underwent training as a Stuka pilot and was posted to 1 Staffel Sturzkampfgeschwader 2 (StG 2), which was moved to occupied Poland in preparation for Operation Barbarossa in June 1941. On 21 September 1941, Rudel took part in an attack on the Soviet battleship Marat of the Baltic Fleet. Marat was sunk at her moorings on 23 September 1941 after being hit by one 1,000-kilogram (2,200 lb) bomb near the forward superstructure. Her sinking is commonly credited to Rudel alone, but Rudel dropped only one of the two bombs that sank her.

From May 1941 to January 1942, Rudel flew 500 missions. In February 1943, he flew his 1,000th combat mission, earning him the status of national hero. He then participated in the experiments with using the
Ju 87-G in the anti-tank role. Rudel participated in the Battle of Kursk and on 12 July 1943 he claimed 12 Soviet tanks in one day. In October 1943, he was credited with the destruction of his 100th tank and was awarded the Knight's Cross of the Iron Cross with Oak Leaves and Swords.

Rudel was appointed Gruppenkommandeur of III. Gruppe on 22 February 1944. On 20 March, Rudel performed a forced landing behind Soviet lines and he and his gunner escaped to the German lines. The men attempted to swim across the Dniester River but Hentschel (his gunner) drowned in the attempt. Rudel was promoted to Oberstleutnant on 1 September 1944, and appointed leader of SG 2 on 1 October 1944. On 22 December 1944, Rudel completed his 2,400th combat mission, and the next day, he reported his 463rd tank destroyed. On 29 December 1944, Rudel was promoted to Oberst, and was awarded the Knight's Cross of the Iron Cross with Golden Oak Leaves, Swords, and Diamonds, the only person to receive this decoration.

On 8 February 1945, Rudel was badly wounded in the right foot, and landed inside German lines as his radio operator shouted flight instructions. Rudel's leg was amputated below the knee but he returned to flying on 25 March 1945 and claimed 26 more tanks destroyed by the end of the war.

Rudel was shot down 30 times during his 2,530 missions. He destroyed one battleship, one cruiser, one destroyer, 70 landing craft, 800 vehicles, 150 gun positions, 519 tanks and nine aircraft. He was the most highly decorated German serviceman of WWII.