13th Annual Top 25 Pre-Discussion Thread

[CWM]

I don't really get it, to be honest. Even in genres I'm very familiar with, I'd have trouble finding 25 games worthy of being on a "Top" list, and you want to add even more because 25 isn't enough? That there is crazy talk.

Counterquestion: why not? Chances are you're going to see more varied lists when people are allowed to vote for more games which makes for a more interesting read. There's nothing to lose.

Honestly, that line was a bit of a joke, it doesn't bother me at all if someone has 50 or 100 games they want to list. That's actually one of the reasons I prefer a fixed-point voting system - it lets every individual decide how many games they'd like to have on their list and how large a "piece" of their vote should go to each one.

Currently, though, each list is required to have 25 entries, which results in some amount of padding from people who'd prefer to have shorter lists. That's not really a problem in itself, but it becomes one when you observe how votes are weighted - popularity (as in, the percentage of lists a game appears on) is weighted twice as much as the average score. The result is that the top25 is full of games which are good and a lot people have played, rather than games which are less played, but rated higher by people who have played them. Hence why they consistently have nearly all of Cave's releases and only token entries from other developers or non-bullet-hell subgenres.

I don't really get it, to be honest. Even in genres I'm very familiar with, I'd have trouble finding 25 games worthy of being on a "Top" list, and you want to add even more because 25 isn't enough? That there is crazy talk.

Still, plenty of users manage it every year. I like to think of it less as an absolute "best of the best" and more a way of giving credit to many games that have proven enjoyable in some way. By now I've seen enough comments about leaving games off and long-time voters switching out entries to think it's at least worth a trial. I don't know how many will want to make one, but even if it's only 1 out of 4 that still makes 20-odd HMs total.

See my response to Perikles above, I wasn't really trying to argue against allowing longer lists, though it seems like a lot of people read it like that, so I probably expressed myself poorly. I do think, however, that requiring people to list a minimum of 25 games skews results with the way final scores are calculated.

In short: Allow lists of arbitrary length (though keep the 0.25 weight maximum per game).

The current system seems like an attempt of sorts to merge multiple top25 lists into one, but in practice it's just a weighting system with arbitrary integer scores, which I think is unnecessarily complicated.

It isn't arbitrary, users can still assign weight however they want and IMO it's simpler to count games rather than needing to have everybody counting point totals. It's not really a 'typical' system, but that's part of what gives the whole event its charm.

But in order to know how much of my vote goes to a given game, I need to actually add all of them together and divide, and this is difficult to tell at a glance. Points are also not comparable between submissions - me giving 5 to Battle Garegga and you giving 5 to Battle Garegga have different outcomes depending on the rest of our lists.

For the record, I'm not invested in trying to change a system that has worked for you guys for many years now; this is more of an abstract discussion for me, as I'm professionally interested in customer rating systems, and it's interesting to me in how a given process affects the end result.

[Nifty]

That's not really a problem in itself, but it becomes one when you observe how votes are weighted - popularity (as in, the percentage of lists a game appears on) is weighted twice as much as the average score. The result is that the top25 is full of games which are good and a lot people have played, rather than games which are less played, but rated higher by people who have played them. Hence why they consistently have nearly all of Cave's releases and only token entries from other developers or non-bullet-hell subgenres.

If you take a look at the total weight from last year's full results (or 'pure' score, to compare with your idea) you'll find it doesn't greatly change the general composition, particularly near the top, and does roughly jack squat about Cave overall. The more votes something gets, the harder it is to maintain a much higher or lower average score per vote, which no reasonably-designed system is going to change. Without going into minute numerical detail, what I like about the classic system is how well-balanced it is. Voters can make a standard 25-to-1 list, or take advantage of the weight system without blowing a game way out of its depth. There's no piddling around with tiny increments barely worth mentioning, every single game vote counts - hence why it works when there are 25 of them to keep a nice even spread.

I do think, however, that requiring people to list a minimum of 25 games skews results with the way final scores are calculated.

In short: Allow lists of arbitrary length (though keep the 0.25 weight maximum per game).

Aside from everything else mentioned (minimum expectation of genre value, ability to change weighting for games of less interest, etc) the experience requirements for voting (i.e. the basis for voting for a T25) aren't going to change. As a result, I think it's only fair if ballots are to be considered relative to the expected size. Lists of 10? Pfft, that's tiny. Lists of 20? Might as well go for 25 anyway. Even then if I were to bring it back, I'd be bringing back the vote weight penalty with it, which unfortunately I'm not really set up to include in the calculations at the moment. That and the HM's would be a lot of change to consider at once. If enough folks really want it I can revisit it next time, but for now it would make the process a lot smoother to have this be the year of more!.

On that note, 25 games also acts as a solid maximum, as opposed to allowing lists to extend infinitely off into the sunset. Having optional "bonus" extras sounds a lot better than trying to set some arbitrary upper bound, should the static length limitation be lifted.

But in order to know how much of my vote goes to a given game, I need to actually add all of them together and divide, and this is difficult to tell at a glance. Points are also not comparable between submissions - me giving 5 to Battle Garegga and you giving 5 to Battle Garegga have different outcomes depending on the rest of our lists.

I know, it's great

[rjosal]

I doubt DBCS will make it to the top 5 considering even CC:WI just barely cracked the top 10 and that was probably far more popular than DBCS. Wouldn't be surprised to see it and Eschatos in the top 25 though.

In the past, DBACEX has only been hampered by how few people have had the chance to play it; when it does appear on ballots, it scores very highly (in last year's poll, its share of points awarded per ballot it appeared on was higher than Futari's). With a wide playerbase, it's going to be nearly unstoppable.

CC:WI had more hype, but even before WI, its average points-per-ballot was nowhere even close to what the games at the top of the list were doing, and WI was worse than the doujin version.

Is there a way to mitigate this? Maybe if each voter also submits a list of games played, you could somehow weight votes better? I always wonder when reading the top 25 results how much popularity plays a role. Maybe a better idea is to determine a "popularity" value for each game which is used in the final ranking formula to weight games. Say DBACEX popularity is 1.36 and DDP is 25.2, and you multiply the final scores of the games by the reciprocal of the popularity. There has to be some way.

EDIT: it sounds like the popularity mentioned above, that is the percentage of lists the game appears on, is something of an approximation. Which is good, but I still wonder, can you tell that the reason a game is not on many peoples lists is due to it being not so great, or due to fewer people playing it?

[trap15]

That sounds like a pretty poor way to rank something. If I decide "yes I will vote 25% of my score to X-PLAN" (the worst STG ever made), nobody else is probably going to vote for it, and the popularity will be very low, so it'd end up getting boosted way up.

[Obscura]

EDIT: it sounds like the popularity mentioned above, that is the percentage of lists the game appears on, is something of an approximation. Which is good, but I still wonder, can you tell that the reason a game is not on many peoples lists is due to it being not so great, or due to fewer people playing it?

In the past, I've advocated for people listing and ranking every single shmup they've ever played, then using a Condorcet method on the results, throwing out games that appear on fewer than 5% of ballots. Everyone says this would be too much work, though.

EDIT: If people were too lazy to list every game they've ever played (which, let's face it, would be quite a chore), but someone with web programming experience was willing to expend some programming effort, there's a three-step process that would be feasible:

1. Nominations -- everyone makes a list of "worthy" games that they'd consider putting on a top 25 list.
2. A web-app is made that has a list of all nominated games next to checkboxes. Each voter checks each one they've played, and each "checked" game goes into a list on a separate pane of the web-app; each voter then sorts that list from favorite to least favorite, and submits that list as their ballot.
3. Run Condorcet method on the resulting ballots, throwing out games that appear on less than 5% of ballots.

Not only does this method only give people the burden of listing games that have a resonable shot at top 25 instead of every game they've ever played, but it also means that people who have played very few shmups and wouldn't otherwise be qualified to vote can vote in this system and have their vote counted equally without messing up the data.

[CWM]

Generally speaking, factoring popularity into a rating system is a well-studied problem in the field of Bayesian statistics. The basic idea there is to treat ratings as samples from a distribution of an unknown mean, and attempt to correct for larger deviations in smaller samples. Let's say we're rating things on a 1 to 5 scale, our average would then be 3. Then items with small amounts of votes would have a rating skewed towards 3, in recognition of the fact that a small amount of votes does not give sufficient information to determine the "true" rating. On the other hand, items with lots of votes would have a rating close to the actual average of the votes, reflecting the low probability of the result being skewed some way.

Incidentally, that's another way to do a vote, anyone votes for as many games as they want and gives each a rating of 1 to 5. You tally up votes, tweak the parameters for the Bayesian average, and enjoy butthurt over the results. Another website I post on did this recently, results were very interesting, some relatively unknown (but good, as it turned out) games had placed very high.

[chum]

There used to be something along the lines of "top 10 for the less experienced".

Could just bring that back.

BTW, I don't think having put minimal time into tons of different games is any better than having deep knowledge of a few games, but if people really can't think of more than 10 games they want to list, they most likely just lack the experience to vote and should abstain.

So instead of changing anything, bringing back the alternate list seems better (if there is demand for it)

[Obscura]

My reason for suggesting a different system has very little to do with expanding voting (that's just a bonus effect). My main reason for suggesting a different system is to help kill the "games that are widely available on recent platforms win the vote for free" bias.

In other words, I'm suggesting a system to knock Futari off of the top spot

[Chaos Phoenixma]

Saw the comment about World Ignition being worse than the doujin version and wondering why you think that.


Both versions look fine to me at the moment. I haven't played World Ignition for much time yet, but I've been playing Boost for the most part when I do outside of a credit fed run of Unlimited to see how well it runs for me(pretty much perfectly, though I was surprised by the apparent lack of slowdown). My best for Boost so far is making it to Stage 3 cause I easily die to stuff due to the fast bullet speed.

[royalfan84]

I honestly like the idea of the HM list. I love shmups; but am still only 3+ years into acquiring and playing them regularly...and both years I've done a top25 I have had to leave many games off. I think the HM list will offset some of that and possibly lead to a "slightly" more varied result. It's not a huge deal either way to me. I do like being able to list more games personally.

I think this Top25 list you do is very neat idea and glad that it annually exists....the top25 is what has lead me into several of my favorite games and is the pinnacle of where to find what is worth playing in the genre if you are noobie or are just looking to try something new.

I have to say I am more interested in how the "label" or "mode" versions in games will be counted this year. To me; either all should be counted and seperated or none. It's my biggest issue with the results....

[Xyga]

There used to be something along the lines of "top 10 for the less experienced".

Could just bring that back.

BTW, I don't think having put minimal time into tons of different games is any better than having deep knowledge of a few games, but if people really can't think of more than 10 games they want to list, they most likely just lack the experience to vote and should abstain.

So instead of changing anything, bringing back the alternate list seems better (if there is demand for it)

That's wrong.

I've been playing games since the 80's, including tons of shmups, then later in the early 2000's the genre really began to settle as one of my favourites ever and since then I have played I don't remember how many and for how many hours.

It's just that I can't in my make a difference between, say, #18 and #17, or #20 and #21 over 25 entries. Can't look for such very small and subtle differences in my personal judgement and preferences that would make enough sense for that.

Over 10 entries I think I could, because that's rather narrow and would require me to think harder why I would place #6 over #7.
EDIT: 15 at most maybe. And that represents quite a bit of thinking already.

Then a HM list following, no problem with that.

[chum]

You can award the same amount of points to however many games you like.

[Obscura]

Saw the comment about World Ignition being worse than the doujin version and wondering why you think that.


Both versions look fine to me at the moment. I haven't played World Ignition for much time yet, but I've been playing Boost for the most part when I do outside of a credit fed run of Unlimited to see how well it runs for me(pretty much perfectly, though I was surprised by the apparent lack of slowdown). My best for Boost so far is making it to Stage 3 cause I easily die to stuff due to the fast bullet speed.

I've only spent real time with Original in both doujin and WI, so I can't speak for Boost.

As far as Original goes, the biggest thing is that WI is simply too hard to be fun; hit the second half of stage 3, and the game pretty much just LOLs at you. But there are other downgrades; lock-multiplier being more important for meter building screwed up the flow and leads to a bunch of dumb "oh, you locked onto one too few enemies, now your whole route is invalid!" nonsense, extra lives being based on stars instead of raw score is a total WTF, the changed bombing system is pointless, and a lot of the new boss patterns don't match the boss sprites at all (why does the stage 2 boss fire shots from out of his legs, where he doesn't have any visible guns or turrets?)

[Nifty]

I can't into asleep so here are more thoughts

It's better not to get too caught up in the idea of completely ranking everything on a ballot, games can easily be grouped or tiered according to preference, even at the top. I suppose the 'default' 25-to-1 method sort of creates that impression, but if that ain't your thang, it needn't have to be that way.

Regarding smaller ballots (again, not happening this time) there'd need to be at least some minimum number of games. 10 is not nearly enough to satisfy the almighty T25 imo, it would need to be at least 15. In turn, that would only leave a difference of 10 between the minimum and maximum, so again, if someone had 20 they may as well either keep going or pare it back down. A single "shortlist" size of 15 could be an option, at a stretch... but if someone then wanted to make 15 + a HM's list, well, that'd just make 25 again Really all shorter lists usually offer is less variety and more vulnerability to populism, when visibility only really matters in the full results, wherein games only stand to lose ground via exclusion moreso than others gain it.

tl;dr like more things, it's fun

[DJ Incompetent]

While the hard-limit of 25 games has its upsides, it can make for some tough decisions for those who struggle to include all their favourites. Thusly, I am hereby proposing
Personalised Honourable Mentions Lists!

Is this for people who enter an elevator and have difficulty not hitting every button?

[LSU]

Honestly, being 46 years old now and having religiously played shmups as my favourite type of game ever since I was about 15 years old, it's really difficult each year to vote for only 25 all-time best games. There are SOoooooo many great shooters from the history of this awesome genre... literally hundreds of games to choose from... I'd welcome any attempt to widen the voting, but the HM list thing seems like something just a little bit tacked on. Couldn't we figure out something a little more precise? Like increasing the vote entries to - say - 30, 40, or perhaps even 50 games..? The latter number might be overkill for some, but honestly, in my list the 50th entry would still be something really great and essential to play. Does IMDB limit its top movies ever to 25? No, it doesn't. I know it's an older medium, but 25 doesn't have to be a limit. Just a thought.

But, I'll join in again this year whatever route we go, of course. Looking forward to it as always.

[copy-paster]

In other words, I'm suggesting a system to knock Futari off of the top spot

That and Garegga are truly nemesis here, so the #1 rank in this year are predictable.

[Bananamatic]

lock-multiplier being more important for meter building screwed up the flow and leads to a bunch of dumb "oh, you locked onto one too few enemies, now your whole route is invalid!" nonsense

I don't think you have a route fam

[Illyrian]

lock-multiplier being more important for meter building screwed up the flow and leads to a bunch of dumb "oh, you locked onto one too few enemies, now your whole route is invalid!" nonsense

I don't think you have a route fam

Are you even able to post anything that isn't a 4chan meme? Every single post you make is a shit post.

Do you sit at your computer and imagine everyone who reads your posts bursting into gales of laughter at how totally hilarious and original you are?

It is literally pathetic.

[Bananamatic]

just doing my part in maintaining the general quality of discussion

[Nameschonvergeben]

Really all shorter lists usually offer is less variety and more vulnerability to populism

I don't understand this, if someone only really likes 10 shmups and you force her to put 25, she's just gonna be like "okay I guess I have heard of these 15 other popular games before and seen 10 minutes of gameplay, so I guess they must be pretty good" and add them, which actually makes the long list more vulnerable to populism, not a short one.
I also don't see why variety should be forced in a popularity poll? Isn't that kinda missing the target? (or maybe I am misunderstanding something?)

[Kollision]

with the incredible number of shooters out there, my feeling is that a list of 25 games is doable

if someone can only think of 10 shooters to vote on, he/she clearly isn't ready to vote yet

[Nameschonvergeben]

how about you have to post a respectable score in the game you want to vote for then? If you don't have one you clearly aren't ready to vote yet. Why is one way of playing shmups elitism and the other(more casual) is the way to prove how much you like the genre?

[royalfan84]

You'd have to change or do away with the current weighted formula if you chose to allow less than 25 votes. It's a top 25 list...if you don't personally believe that at least 25 of the games you've played deserve a vote...then I don't know what to tell you...you get to assign points to your votes so you could always just give the games you are forced to put on the list to meet the requirements minimal points- without it affecting your overall input much at all.

[Kollision]

how about you have to post a respectable score in the game you want to vote for then?

that could be a valid point in another type of {elitist} list, but then it would prevent people from voting on the pure awesomeness that the first Mushi is in order to bring down that Futari hack

anyway, I absolutely love and think that DOJ will never leave my list even though I never posted any score on the corresponding thread

[Xyga]

with the incredible number of shooters out there, my feeling is that a list of 25 games is doable

if someone can only think of 10 shooters to vote on, he/she clearly isn't ready to vote yet

Nah you're another one who doesn't get it, to me it's the opposite; I have way too many games in mind, and 25# is too many steps to make enough sense between one step to the immediate next or previous one.
Restricting it to a lower number of entries makes much more sense than getting it even bigger. Or do people want one of those 'top 50', 'top 100' or whatever shites we can read on many websites ? (50 best kind of movies, 100 best video games, 200 best sex toys, blah blah blah)
Ranking ever greater numbers of whatever doesn't make sense, it's more listing than ranking.
That's how it is for me, fewer is better, I can rank 10 or 15 games weighing each title's value carefully, but 25 would begin to lose it's coherence, and even more would just be a list with no real rank value.
Anyway I'm the minority apparently so I'll probably abstain this year again.

[chum]

and again what are you going on about.

you can ju st give the same score to all your games so they have the same weight if you can't decide that you prefer one to another.

if you have "way too many games in mind" there is literally no downside to being able to name more of them. just what are you saying...?

[Bananamatic]

why can't we discuss gameplay like this

[Nameschonvergeben]

You'd have to change or do away with the current weighted formula if you chose to allow less than 25 votes. It's a top 25 list...if you don't personally believe that at least 25 of the games you've played deserve a vote...then I don't know what to tell you...you get to assign points to your votes so you could always just give the games you are forced to put on the list to meet the requirements minimal points- without it affecting your overall input much at all.

Can someone explain to me why you can't use the same kind of weighting for a list of 10 as you could for one of 25? I'm a bit slow sometimes

that could be a valid point in another type of {elitist} list, but then it would prevent people from voting on the pure awesomeness that the first Mushi is in order to bring down that Futari hack

anyway, I absolutely love and think that DOJ will never leave my list even though I never posted any score on the corresponding thread

Ah yes, of course we can't make rules that would not allow YOU to vote for your favourite games

why can't we discuss gameplay like this

What's your favourite Touhou game?

[gimebe]

Voting systems used to compile popularity rankings are fun and interesting. Unlike shmups.

I don't see why lesser lists with low-priority picks cut out couldn't be included. It could be treated as a list of 25 with blank votes for the low-priority slots - an example under the current system:

[25] Battle Garegga
[24] Battle Garegga
[23] Battle Garegga
[22] Battle Garegga
[21] Battle Garegga
[20] Battle Garegga
[19] Battle Garegga
[18] Battle Garegga
[17] Battle Garegga
[16] Mushihimesama Futari
[1] This game is not Garegga or Futari
(repeat as necessary)

Trimming the bottom end of one's list would simply turn the less worthy games from [1] (drop in the bucket) to [0] (nothing), which doesn't seem like that big of a difference. Or you could just make a very top-heavy ranking of 25 to state enormous favorites as illustrated above. I don't know the purpose of this specific weighting/ranking system and the specific number 25, but it seems like a working system. A change in numbering in either direction doesn't sound like a big deal since the user has a lot of freedom to weigh and rank as they like under this system already.