A shoot em up video game
Firstly, you're writing a video game. A video game. Games are supposed to be fun. Writing something that is actually fun is a very hard thing to do but should be your primary focus. You should be playing your shmup while you write it and if you find yourself getting bored - you need to address the fun factor. The exception to this would of course be Easy/Novice/Beginner modes that are designed to allow Joe Bloggs down the street to at least reach Stage 3 of your shmup before he has to really think too hard. Those modes can be unintense - but your default (i.e Regular Difficulty) should not be boring.
Secondly - although this is of equal importance... You are writing a shoot em up. In the french video - history of shmups, one of Caves developers made the best point ever that really stuck with me. A shoot 'em up at its core is about shooting enemies that explode in a satisfying way. This doesn't mean that you need to tart your game up to epic proportions so that every enemy explosion gushes with sparkles like Aphrodite mid coitus (!) - but there should be an element of satisfaction and responsiveness when your enemies blow up - even if that's just a satisfying crunch sound and a simple explosion ring.
That's the mindset you probably want to be in to write a good shmup. Fun and explosions.
Related Video:
Part1 : http://www.youtube.com/watch?v=3AHhkq7p3Qw
Part2 : http://www.youtube.com/watch?v=Fv7fCr6QFis
Part3 : http://www.youtube.com/watch?v=B-qFAtkCcIk
Scoring Systems
Scoring systems enhance the experience of a shmup on a different level that most casual players wont get. Using Street Fighter as an analogy - most players will start throwing a few fireballs and learning a couple of 2-3 hit combos. As they play the game more, they'll start learning about reversals and counters doing more damage. Towards the later stage of learning their character, they gain an understanding of how to extract the most from every situation. They never really stop learning and competing.
A modern shmup can be very competetive. It's you versus the machine on whatever difficulty you can manage. Take Mushihimesama Futari Black Label on Original Mode for instance. After some considerable amount of practise on this game and getting over my issues with using bombs in the later stages (Damn you 1UP Dragon!) I managed to 1CC that game and at the time place around 230th on the leaderboards. In spite of Original Modes very simply scoring mechanics - I know looking back on my run that there's more I could do to improve my score. There are points that I missed that would allow me to extract more points with a little bit of patience and better positioning. A good shmup should always offer the player the ability to increase their performance and work towards counter stopping the game. Whether or not counter stopping the game is possible is an entirely different discussion and there's an argument for not allowing a counter stop. However the thing that keeps us coming back and replaying games that we've "beaten" with a 1CC is the knowledge that there are always more points to gain.
BUT...
There are times when your scoring system can just get in the way of a fun game. Remember - fun is your primary objective. If your game feels like a chore to play, then you'll only get the niche within the niche playing it. You don't want that. So now on to the meat of this post, which is gimmicky scoring systems and mechanics that can make a shmup boring to play.. What do I mean by boring? Anything that takes the action out of the game, leaves the player with sections where they're just holding a button and not doing much else.
These are a bunch of things I've played with and either refactored, moved to an alternate game mode or flat out removed:
Bullet Cancelling
Bullet Cancelling can be a wonderful thing.
There are a bunch of shmups out there that use bullet cancelling to good effect. See DDP:DFK. However this game hinges around the fact that you must earn your bullet cancelling through taking risks as you weave through fields of bullets. When you're in bullet cancelling mode, there are mechanics in place to add a mini game of shot switching to keep your chain going, push back lasers and reap points from bullets. When Bullet Cancelling doesn't work is when you give your player too much power to deflect bullets.
Lets say you've got a hyper system, upon activation your ship becomes invulnerable and you zoom around the screen collecting bullets with your shield. That's fine - but you need to take balance in to consideration here. Are you allowing the player to accumulate another hyper during this phase and keep it going? If your ship is invulnerable - that's going to lead to some serious boredom after you get your first hyper as there's no risk. Crimson Clover is a perfect example of how to make your player feel satisfyingly overpowered and godlike by making them earn that lovely break mode and adding an additional mechanic within the break mode that allows them to extend this further - giving them something to work for in that mode too.
Be careful with bullet cancelling and protecting your player... make sure there's always something for them to be mindful of.
Slow Motion
Some games do this well (Mushihimesama), other games dont (Sine Mora).
One thing that generally hasn't worked (in my personal opinion) is an ability to slow down bullets by holding a button. My preferred method of implementing slow motion is to engineer a wait system (The game pauses for a frame, ala Mushi) and directly tie that in with the number of sprites on screen while enforcing limitations on how much things can slow down.
I had to take out the original slow motion features when I started writing Chronoblast because they just didn't sit well with the game. Given that the game is called CHRONO-Blast, you'd think that'd be it's unique selling point. Turns out it sucked! So no slowmo there.
I've seen a lot of indie games that implement slow motion systems that just allow you to weave through bullets without much issue. That's not really fun! If you're going to incorporate slow motion in to your game, then you need to make this a calculated decision for the player. If they fill a meter - activate and enter a slowmo mode - what are the mechanics for reaping points from that? Is there a timer that upon expiry cancels the bullets in to points? If so, then the player has to decide when and where to use their ability and work out when it is simply wasteful to do so. I've experimented with fields around the player in hyper mode that slow down bullets, or just having all the bullets go really slowly. Whatever I did with slow motion, beyond a wait mechanic for incredibly dense sections or when there were a lot of powerups to colllect - everything just seemed so wrong. So I've pretty much decided that slow motion is rubbish unless it's done properly, like Espgaluda 2.
Also if you're going to use a weighted wait system where you calculate a value based on the amount of shit on screen - add a nice juicy weight for your players bomb or hyper activation. You'll thank me later when you play for score.
Special Abilities in general!
We're not talking about risk/reward mechanics here. We're talking about things like slow motion, invincibility on a meter. Keeping them in moderation.
For example, Espgaluda 2 has both of these things but they're restricted to your bomb meter and slow motion is dictated by the number of gems you've harvested. For this game it works well. But if you were to increase the availability of these powers and allow them to be used 2 or 3 times as much as they are now - the game would be massively different and probably not as solid as it currently is.
Try not to design your shmup with your special ability as the core focus of the game and instead use it to enhance the core gameplay. The core focus should still be shooting things for points with your special abilities coming in to play to deal with whatever remains on screen after the enemy has been shot at. In the example above with slowmo, if it's painfully obvious that your ability MUST be used in that one spot with the million bullets on screen, everybody is going to do it and you won't get a lot of variance in scores. If you must experiment with fancy mechanics, throw in an arrange mode to showcase these cool little tricks. Make sure these abilities dont impact your main game. God knows I've had a bunch of mechanics in there ranging from Crimson Clover style missiles, to bullet cancelling options that move like Akai Kitana/Deathsmiles etc... all completely terrible and too confusing. Hold shot all the way.

Turning the player ship around
http://www.youtube.com/watch?v=cj1LB56zaD8
After playing Deathsmiles, part of me went "Hey, wouldn't it be cool if you could spin your ship around and make enemies come from both top and bottom of the screen?! It was a cool little idea and for a while it worked well, but it was an extra button to push that just got in the way of core gameplay.... It's not big and it's not clever. Plus Uridium probably did it better. There's a video of this in something I put together for a job interview. It was taken out because it sucked.
Insane varying bullet modes
Red and yellow and green and blue... games too confusing its true...
Playing Espgaluda 2 or Muchi Muchi Pork Arrange, having fun and getting a vague handle on its mechanics does not mean that you should go off and start leading your player through 20billion different modes of play whenever they activate hyper. Don't have them shoot bullets to turn them green and then have those turn in to points by collecting them before the time runs out or any of that nonsense. Like turning ships around, I found this to be neither big nor clever. I ditched this system in favour of simply turning bullets red and making them go faster when the rank meter was full, or the player was in hyper. This lead to a much better style of gameplay where the player becomes aware that danger is imminent and must try to shoot as many enemies as possible in order to fill the rest of their hyper meter so they can panic bomb. There's a bonus for staying in this mode as long as possible during the stage, but there's also an opposing bonus for being in hyper. Simple risk/reward and meter management - highly effective. Don't know what I was thinking with all that "green bullet" nonsense in the video Sined posted.
Chaining
"Wait what?! Chaining isn't a gimmick. How dare you belittle the Angry Bee Boss?!" I hear you say.
You're quite right, it's not. But as I mentioned before there's times where a mechanic can get in the way of fun. It will also add to your development time considerably and is not for the faint hearted. If you're able to implement a chaining system to good effect - go for it. They're glorious things, however after extensive testing of my own game it started to become apparent that chaining simply wasn't the way to go in the face of fun as it started to detract from the games other scoring mechanics (shedding gems from large enemies etc)... so that's why it's listed here.
Sure, your first couple of stages are doable with a bit of calculated use of hyper (to slow down the combo timer) while the bullet count is relatively low - but when you start getting in to the later stages it can become painfully obvious when you start dropping in tougher less threatening enemies just for the player to buffer their chain on. Like.. painfully obvious to the point where you might aswell put your player on rails because there's no room for deviation through risk of losing a chain.
I'm not saying that chaining is bad, in fact if I'm honest - chaining is one of the better mechanics you can find in a shmup. But if you're going for Daioujou levels of precision with your chains, you need to be an amazing shmup player... because you'll have to make sure those chains are doable on every single difficulty through the fields of bullets that you create. In short, if you've never actually reached the second loop of Dodonpachi Daioujou - throwing that many bullets at your player and then expecting them to carry a chain through it is a real gamble... Plus I already have access to every version of DDP I could ask for and they're all better than anything I could write, coming from a professional company... This brings me to:
BUTTONS
A is fire, B is bomb, X is Hyper, Y is Banana Mode, RT is Rewind, LT is slow motion, RB is slow mode, LB is reverse direction, L3 is mega bomb, R3 is switch shot, Left Stick aims your gun, Right stick aims your crosshair and whats a dpad?
A,B,C is all you need! Possibly a D if your game has a switch mode. Rapid fire on C, Bomb/Hyper on B and Tap A for shot, Hold A for laser. Don't reinvent the wheel and don't try to convince yourself that tapping A requires more skill than holding down the auto fire button. You can have tapping of shot to allow the player to get their own natural rythm going - I know that I tap shot in games from time to time when I don't need to, just because it feels cool and helps me stay in the zone. But never at the expense of losing that rapid fire button.
This is not to say that you have to stick firmly to a 3/4 button layout. Heck, go crazy and have all the buttons in use - but one thing is for certain MAKE SURE THERE ARE ENOUGH BUTTONS ON A STANDARD ARCADE STICK. This means you probably shouldn't use L3/R3 or their equivilents since we don't have those on a standard 6-8 button arcade stick.
Above all, make sure that your buttons are laid out in a way that is intuitive. Copy other games control schemes - not so much because they're how it SHOULD be done, but it's certainly what people are used to. If someone can hit the ground running with years of muscle memory aiding them to just pick up your game - they're more likely to keep playing it. If they have to scratch their heads and go "ummm, what button did that thi... Ah shit I'm dead" - less likely.
Don't write for a skill level higher than your own
...without playtesters who love the genre to death and are better than you.
There will always be someone better at the game than you. Any game I've contributed to and released, despite the weeks of playtesting it myself and being rather good at it - there's always a player who has a natural flare for taking the top spot. So be prepared for and in fact *embrace* comments like "Touhou is harder" or "Game was too easy". If you released your game and somoene says "This game is fun but it could be harder" - make a sequel, or "Black Label" style patch and take pride in the fact that someone paid you the highest compliment. Your game was fun.
I'd be interested in hearing from the rest of you what mechanics you've tried over the years that you stripped out and replaced, or indeed any games you've seen that have gone down in flames for overly complicated systems and mechanics.
DEVELOPER BLINDNESS
You WILL get it. You will always get it. You simply can't avoid it. This is that magical thing that happens when you play your game constantly, with about 30-40 compiles a day with just you sitting there and playing it. So you wanted to do a Yagawa and test the players raw dodging abilities on stage 1? You've had hours upon hours of playing the same segment over and over again while you were just refining your players shot pattern. Your brain wasn't even looking at how difficult that section was - it became muscle memory. Your new players are not going to have this muscle memory.
Every so often, you should go off and play another game or two - give yourself a few weeks break. If you can't stop working on your game - take a backup and work on a different branch. Try making some things easier and play that for a week. Go back to your old version and see how it compares. If you find yourself saying "Fuck!" every few minutes with every miss - you suffered from developer blindness. Trust me, I've done it - it's not pretty. Chronoblast has gotten a mixed reaction of praise and negativity due to it being absolutely nonsensically brutal. Some people find it refreshing and don't mind the difficulty - however seasoned players are having none of it. I'm a seasoned player and on reflection, I'm having none of it - hence working on a patch for the bloody thing. Difficulty might pull in some casual players who have "never seen anything like it" - but long term replayability and scoring is going to be reserved for the exceptionally elite players (and they're in the minority) if your opening stages utterly destroy your players.
So in summary, my list of evils:
* Too much bullet cancelling
* Slow Motion
* Rotating Ship
* Forced special abilities
* Crazy bullet modes
* Chaining at the expense of fun
* Difficulty levels that you can't verify as completable without cheats
* Unnecessary buttons
* Copying Ikaruga