Well, one of the worse scars of my youth has always been Project X on Amiga. It's probably been the shmup I played most, since I found it impressive at the time it hit the shelves. It was fairly popular among my friends as well, BUT! With just 1cc, of course, 80% of my friends never managed to finish the FIRST level.
I, as an hardcore player, managed to finish the second level just once, then got after the insane bonus level that scrolled so fast that you had to follow the voice "go up" "go down" because you couldn't possibly see where you were going, then I saw the first screen of the third level and died.
the power-up buildup was so long and hard that once you reached the maximum (plasma x3, lateralx3, satellite x2) there were hopes to go ahead. Once you died and started back from plasmax1 and lateralx1, you were fXKX'd up. That's unfairly hard, alright.
Agony too, I found it fairly hard, just because you had just 1 kind of shot with 3 different widths, but its strength never changed - but monsters became harder and harder
On new gen, I'd say Ikaruga and Darius G are fairly unrelaxing.