I'm probably one of the older players here but without giving anything away some of these games were beaten upwards of 15 years ago so some memories may be slightly fuzzy.
All difficulties and quality rankings are based on a 1-10 scale and the entire scale was used, with 5 being average. Games that have a quality rating of 8 and above I generally hold in really high regard as top tier games. I certainly am not a super player like some of you guys who can 2-All CAVE games and what not. I used DOJ Black Label as a base line for the hardest game I can beat (a 10 on the 1-10 scale) and blind 1CC's as the easiest (a 1 on the 1-10 scale). I also tried to take execution into account as well. A game with heavy memorization and strict death punishment but easy execution like R-Type 1 will rank lower than something like Giga Wing which also requires consistent reflexes on top of route precision. Generally speaking, if you frequent this forum, you probably know how to practice these games.
I used generally well known, well regarded, and modestly difficult games like like Mushi Futari Black Label and Deathsmiles as the baseline for an average "5" difficulty.
I used save states to practice for most games if they were available in MAME or other emulators. For example, R-Type took me three days to 1CC with save state practice. At my "kuso" level of skill I generally expect most games will take four days of 2 hours+ per day of save state practice to memorize. That is about a "5" in difficulty. Though various factors may also affect the final difficulty ratings (R-Type basically requiring a one life clear bumps it up a couple points). "1's" were generally blind 1CC's. DDP DOJ, the only "10" in my ratings so far, took me 60 hours with focused practice. All final runs were done clean without save states. Much like learning an instrument, I firmly believe in trying to use the most efficient practice possible until you are ready to perform the real thing. Not all games have save state practice available like X-Box 360 only ports and that may affect difficulty ratings as well. I also generally used auto fire when available. I understand there will be some disagreements about this, as I was initially was hesitant as well. However, over time auto fire has become the Japanese arcade standard to the point where it is generally expected, and therefore, I feel it is generally acceptable to use. I have notated the games where auto fire was used.
I hope this list is useful to you. There is the Japanese rankings available here but I always wished I had a resource like this starting out.
I encourage you to make your own list using the criteria that makes sense to you. I'd especially like to hear from some of you who are really good at these games.
