Score validation and WolfMAME

[Dave_K.]

Split from ITALY v.s. SHMUPS.COM thread:

WolfMAME for replay file read your moves not video as in avi files, if you use save state with VMWare replay is impossible to playback correctly. If you are VMWare master or know who is master, please use it and made extra replays for WolfMAME. I will wait for you after 100mln years in hibernation.

It took about 15 minutes.

http://www.system11.org/temp/forumposts/xev1.inp
http://www.system11.org/temp/forumposts/xev2.inp

Notes:
1) I suck on keyboard.
2) There is a pause in ship movement where the 2 INPs start to differ, this is me letting go of the controls - obviously the best time to do the save state would be between levels, but I wanted to show you can do it mid game.
3) The input is not delayed - note deliberate bombing of different ground targets to prove it wasn't input lagged.
4) You can verify this is a matching replay up to a point by comparing the files in a hex editor, or passing them through 'strings' on a unix machine which will extract things that look like keycodes and comparing that.

Now just stop it with the whole 'proof' thing ok? I wonder how many other people know this is possible and incredibly easy. No, I'm not going to tell anyone exactly how - it's obvious to anyone half technical. I wonder if any MARP replays used it? You can never know.

BF, you do realize this is a slap in the face to a alot more people than just the italians right? And if this has been such a simple thing anyone half technical could figure out, why hasn't anyone told the fine people at MARP? Or do they already know this and its like some big conspiracy thing they are keeping underwraps?

I think score validation is a real topic of discussion, and not just in the context of the ITALY vs SHMUPS.COM thread, but for internet ranking in general. Yes there will always be cheaters. And as shown above, we can never be 100% sure of any particular validation method. So how do we go about validating scores that may seem outlandish or surprising and still remain unbiased? What happens when a newbie shows up and beats a score of a veteran on the board here?

Now before the trolls come in and start spouting off "its just a game", and "chill out", the discussion is not about personal vanity. Its about trying to level a competitive playing field so we can rank performance. Otherwise why bother even having a highscore section?

[ArrogantBastard]

It's a shame that people have to cheat via MAME. It makes people like me who have put in considerable amount of time into certain games to attain such a score, look bad or looked down; because of some retarded dipshit's inability to invest his / her time into a game to get a legit score.

I'd afford PCBs if I had the means of money to prove that my scores are legit if anyone tries to accuse me of cheating.

But yeah, that's one major slap in the face for those who use WolfMAME as a tool of validating the INP files. Haha...

[Thunder Force]

Well, you're right that someone should at least inform the MARP group of the possibilities for tampering, if they do indeed believe their WolfMAME method to be tamper-proof. Under normal circumstances it might not be such a big deal, but I just checked their site and noticed their community has built up a competitive structure including cash bounties for replay achievements.

[Icarus]

MAME should never become the be-all-and-end-all of record validation any way, due to the fact that MAME itself is subject to little bugs and differences to the original hardware.

Makes you wonder, doesn't it?

[zakk]

Slap in the face? what?

Look, the point here is that using any file that originates on the CLIENT machine to validate ANYTHING is bound to fail if the person wants to put the effort into it. It's why DRM schemes consistently fail given enough time. Also why cracking PC games was so common.

All you can hope to do is raise the bar high enough that the people who want to cheat have to put too much effort into it. Maybe wolfmame does that for 90% of the people out there, who knows. And i know how he did it, and it's really not that hard. There are even other ways to cheat that are undetectable.

His 'revelation' changes nothing. The reality is this has always been possible. Viewing wolfmame (or any other version of mame) input files as some absolute proof of score has always been misguided. You cannot prove someone's score is legitimate over the internet, it's not possible.

[Rastan78]

MAME should never become the be-all-and-end-all of record validation any way, due to the fact that MAME itself is subject to little bugs and differences to the original hardware.

Very true. A lot of ports also have scoring potential differences to the PCB. Take a look at Arcadia and you see that they only allow scores played on the PCB and verified by certain arcade operators. If a player broke a record in MAME, the PCB score would still be considered official, not the other way around.

Since we don't have such a system in place and our level of competition is not nearly as hard core, we just have to rely on trust and reputation. When new forum members show up with great high scores, no reputation and zero knowledge of a game's system (Nobunaga ) their scores will probably get disregarded.

Most of us play for personal satisfaction which doesn't come as a result of fabricating fake scores. Plus we all have our favorite way to play. I wouldn't want to shelf my PCBs and switch to MAME (not a big fan of it) only to justify myself to other people.

As far as MARP and Twin Galaxies, who knows how legit their scores really are? My suggestion would be to take issues straight to the source.

[Dave_K.]

MAME should never become the be-all-and-end-all of record validation any way, due to the fact that MAME itself is subject to little bugs and differences to the original hardware.

Makes you wonder, doesn't it?

I would never consider Mame as the be-all-end-all of record validation (I don't think the italians think this either). But its the closest we can get to
"internet" ranking outside of blind trust. I'd be willing to validate anyone's scores on my PCB's if they come to a bay area shmupmeet, as that would be end-all verification.

Slap in the face? what?

Sorry, maybe "wakeup call" is a more appropriate phrase. I just think its strange that if this was such an obvious hack, why people haven't discussed this on MARP, which has been using WolfMAME for how long now?

Most of us play for personal satisfaction which doesn't come as a result of fabricating fake scores. Plus we all have our favorite way to play. I wouldn't want to shelf my PCBs and switch to MAME (not a big fan of it) only to justify myself to other people.

Agreed. But I'd still like to see people using Mame to easily share their skill/knowlegde of the games through INP replays.

[zakk]

Slap in the face? what?

Sorry, maybe "wakeup call" is a more appropriate phrase. I just think its strange that if this was such an anobvious hack, why people haven't discussed this on MARP, which has been using WolfMAME for how long now?

Often 'obvious' hacks such as this don't get discussed because the community that's concerned with hacks is focusing in a completely different area.
Wolfmame was made to combat a particular approach to cheating. That is, manipulating mame itself via save states/slowdown/etc and direct .inp hacking. This has also usually been the focus of cheating type discussions. I'm not really sure there's much to discuss about the method bloodf used; it's pretty hard to stop. Although I can postulate there may be a few methods to detect it.

I also have another scheme in my head that would be near impossible to detect; but it would require writing actual proof of concept code that's way more effort than I ever care to put forth. Luckily it's probably way too much effort for anyone else to put forth too. Although I suspect there may be commercial programs that implement something like it; I just don't know that particular software space well enough to know.

Here's the thing: people who would want to cheat via these methods are probably rare compared to the ones who do so via the methods wolfmame shut down. More people cheated with those methods because they were insanely easy, they were built right into mame! (except for direct .inp manipulation, but that's a whole different breed of person). The problem is I think many parts of the community have put wolfmame and related programs a bit up on a pedestal and that's bad. It solves some specific problems, that's it.

What does this mean for MARP or whatever? Unknown, it's up to each individual community to decide what their acceptable risk is and how much they care. I think they benefit from being small enough niche communities that it's not going to be a huge problem.

[system11]

BF, you do realize this is a slap in the face to a alot more people than just the italians right? And if this has been such a simple thing anyone half technical could figure out, why hasn't anyone told the fine people at MARP? Or do they already know this and its like some big conspiracy thing they are keeping underwraps?

Sorry Dave, never saw a house of cards I wasn't temped to pull down

I suspect clever people know it's possible - hell, if you were REALLY dedicated to cheating, from the format of the INP file it doesn't seem like it would be very hard work to program your score run. Laborious yes, difficult - perhaps not so much. It's not something I'd invest time in, but it does open the door to multiple playthroughs, then glueing your best sections together with some crafted transition segments. I'm not anything close to even 50% sure of that, just a cursory glance and the fact that it DOES have to be a parsed format for the file. I'd be surprised if anyone /is/ doing what I did, really - you'd have to want badly to cheat, I can't see why anyone would have that level of drive to do so, unless faced with the aforementioned house of cards, and boisterous defender prancing along the top.

Personally, I consider any score under emulation to be null and void. Emulation isn't identical on all peoples machines, therefore the playing field isn't level to begin with - some people may be penalised more than others by their particular PC configuration. Not to say they're all meaningless, not at all, but it would be insane to base any kind of material reward on it.

If I really cared about genuine scores (I don't), I'd only be involved with competitions where the original hardware is used, with judges present. Logistically impossible for the collective Western world, unless someone wants to give out a whole load of free plane tickets.

If you base INP files as being useful and evidence from a level playing field, you've already walked down the wrong path, basically - the field isn't level until MAME comes shipped in a tamperproof box, and issues codes like on Cave PCBs.

[zlk]

The main reason why I like MARP is not for competition, but to watch other people play games and learn techniques. I think the people at MARP know that if someone wants to cheat bad enough, they will figure out a way.

I still trust an inp much more than someone just saying their score or producing a screen shot. It takes no effort to say you scored a certain score. It takes 10 minutes to fake a screenshot. It is a lot harder to fake a high quality replay, although it has been done before. Even with a fake replay, people can watch it to see if it "makes sense" playwise.

[BBH]

Personally, I consider any score under emulation to be null and void. Emulation isn't identical on all peoples machines, therefore the playing field isn't level to begin with - some people may be penalised more than others by their particular PC configuration.

If people are using the same emulator, the emulation IS identical between those machines. If they're on different computers the speed might be different if their CPU speed isn't enough to maintain 100% speed on a game, but if the speed dips below 100% it's logged by WolfMAME anyway.

No, WolfMAME isn't completely tamper-proof, I don't think anyone claimed it was perfect. If someone wants to be a prick and cheat, they'll probably still find a way. INP replay files may not be the be-alland end-all of proving legitimate play, but they've served a great purpose at MARP - to allow people to see how other people play games. There's a lot of things I never would have picked up on certain games if I hadn't been able to watch recordings. And for people that know how to use MAME, replay files are a lot more convenient to upload/download then it would recording massive .avi files or whatever. And even if you don't think .inp files are 100% legitimate for whatever reason, you still have to admit it does a lot more to prove a score than a single screenshot.

[Thunder Force]

It may be possible to make that version of Mame more secure by having it regularly "phone home" from the user's machine (e.g. every 5 seconds) to a central MARP authentication server of some kind which is not easily spoofed (e.g. via encryption keys), and imbedd the additional authentication information into a signed replay file. It would be a more secure way to authenticate replays, possibly preventing VMWare save state misuse. If the MARP community feels strongly about validation security, they may want to investigate such avenues.

[Dave_K.]

Thanks guys, all excellent responses.

[system11]

If people are using the same emulator, the emulation IS identical between those machines. If they're on different computers the speed might be different if their CPU speed isn't enough to maintain 100% speed on a game, but if the speed dips below 100% it's logged by WolfMAME anyway.

Wrong.

Display modes change how things look and how fast they draw depending on system. Some USB controllers and peripherals are faster than others. Some drivers are better than others, some motherboards are better than others, some PCs have more latency basically. When I made those INP files, wolfmame told me I was getting 60fps. Certainly didn't look like it to me, stuttering all over the place. Even a user with a poor quality LCD screen can be at a disadvantage to one with a CRT. Added to that, even games in MAME that are allegedly perfect, frequently feel different to the arcade boards, games running at 59.x HZ on a 60 HZ display are already slightly wrong in look. So actually it's inconsistent /and/ innaccurate.

[silvery wings]

Wrong.

Display modes change how things look and how fast they draw depending on system. Some USB controllers and peripherals are faster than others. Some drivers are better than others, some motherboards are better than others, some PCs have more latency basically. When I made those INP files, wolfmame told me I was getting 60fps. Certainly didn't look like it to me, stuttering all over the place. Even a user with a poor quality LCD screen can be at a disadvantage to one with a CRT. Added to that, even games in MAME that are allegedly perfect, frequently feel different to the arcade boards, games running at 59.x HZ on a 60 HZ display are already slightly wrong in look. So actually it's inconsistent /and/ innaccurate.

Sorry, but this makes no sense. An inp file is simply a recording of input based on timing built into the emulator. If you record a playthrough of a game on one machine that is running the game at 100% speed, and then play it back on a weaker machine that plays it at 50%, the replay will be identical in nature, only played back with every other frame dropped. The key here is the timing is based in the emulator, which means its not hardware specific.

As for the discussion i think alot of people forget one key issue, trust. As was said before, the easy ways to cheat have been removed from wolfmame, which removes alot of the temptation for new people. The rest of the people are in it to have friendly competition with eachother, to learn from eachother, and at least on some level to trust eachother. Stuff like this isn't discussed on the marp board because the only people who would discuss it are those who wish to cheat.

There is no way to verify any score without being there in person, even a video of someone playing a pcb can be doctored. That shouldn't stop people from having friendly competition, much like the high score boards here at shmups.

[system11]

Sorry, but this makes no sense. An inp file is simply a recording of input based on timing built into the emulator. If you record a playthrough of a game on one machine that is running the game at 100% speed, and then play it back on a weaker machine that plays it at 50%, the replay will be identical in nature, only played back with every other frame dropped. The key here is the timing is based in the emulator, which means its not hardware specific.

You've missed my point entirely. The point is the gameplay experience for different people on different setups can be slightly or vastly different - hence, an uneven playing field. It doesn't matter that the replay works identically in 2 places, I guarantee that if I replace a microswitch in your joystick with an unreliable one, you'll perform worse than you would otherwise. This concept extends to how reactive your system is generally speaking, whether or not your monitor blurs, and so on.

There is no way to verify any score without being there in person, even a video of someone playing a pcb can be doctored. That shouldn't stop people from having friendly competition, much like the high score boards here at shmups.

Quoted for truth. This all came about because of a lack of trust, coupled with people wanting to play the way they prefer, not on some ghetto emulator hooked up with USB on Windows The whole proof of concept thing purely came about because of that deluded guy who thought replays were bulletproof.

[freddiebamboo]

There is no way to verify any score without being there in person, even a video of someone playing a pcb can be doctored. That shouldn't stop people from having friendly competition, much like the high score boards here at shmups.

Quoted for truth. This all came about because of a lack of trust, coupled with people wanting to play the way they prefer, not on some ghetto emulator hooked up with USB on Windows The whole proof of concept thing purely came about because of that deluded guy who thought replays were bulletproof.

Well said.

I don't really feel this "inp's can be faked" has much impact on this board though.

I've always thought the goal here on the board has been to try and get better and learn from other players, rather than scoring elitism.

Members get more respect round here for contributing and helping out the strategy section. There really isn't much to be gained by faking a score.

At least, until we do some kind of country vs country competition

[BBH]

Display modes change how things look and how fast they draw depending on system. Some USB controllers and peripherals are faster than others. Some drivers are better than others, some motherboards are better than others, some PCs have more latency basically. When I made those INP files, wolfmame told me I was getting 60fps. Certainly didn't look like it to me, stuttering all over the place. Even a user with a poor quality LCD screen can be at a disadvantage to one with a CRT.

When you said emulation isn't identical on all people's machines, I assumed you were referring to the game's emulation itself, not the emulation "experience".

I know there are factors like that that can have an impact on things, but if you want to get technical there are always differences that can exist with PCB's/ports too. Say you're playing in an arcade with a joystick that doesn't hit diagonals very well, and a monitor that hasn't been adjusted correctly so that part of the display is cut off. That's a bit of a disadvantage if somebody else is playing on a cabinet without these problems, and has an extra button set up for autofire. Ports are the same way, some people will play with a custom stick instead of a pad, some people will play vertical games with TATE and others will play with the display squashed. There's always going to be differences like this no matter which way you decide to play. Unless everything is standardized in a setting where everyone is in the same room playing the same game in the same way, there's inevitably going to be differences.

As for the frame thing, perhaps you had autoframeskip on and while it stayed at 60 fps most of the time, there were little jumps that caused the autoframeskip to kick in for a brief second and make it a little choppier. You can always disable that.

Added to that, even games in MAME that are allegedly perfect, frequently feel different to the arcade boards, games running at 59.x HZ on a 60 HZ display are already slightly wrong in look. So actually it's inconsistent /and/ innaccurate.

Take that up with the MAME team then, I'm sure they'd just love to hear about it.

[sikraiken]

As for the frame thing, perhaps you had autoframeskip on and while it stayed at 60 fps most of the time, there were little jumps that caused the autoframeskip to kick in for a brief second and make it a little choppier. You can always disable that.

I was messing around with another computer that had an LCD monitor, and on one of the settings (I was messing around with the display settings and stretching and stuff like that) the game (galaga) played really choppy, but the game said frameskip 0 100% 61/61. After changing the settings some more, it played normally. I think that type of thing is fixable for most, anyway. I play on an LCD from 2002 and I've never had a problem playing video games on it vs. a CRT.

Along with what BBH was saying about how you can never really have exact setups... didn't Dave K say something about different DDP boards not running at the same speeds? That's just for another example.

[system11]

Take that up with the MAME team then, I'm sure they'd just love to hear about it.

I'm sure they already know. Fact is, if you've got a display that refreshes at 60hz (for example, an LCD), and you want to display a game that updates at 59hz, you are going to have tearing or jumping somewhere. You can try this with 19XX for example. Someone tried to tell me I was wrong, and that the original board suffers the jumping issue. Er.. whatever (I own one, it doesn't ) Only way around this is a setup where the refresh rate is fully controllable. That very much depends on your graphics card, monitor, OS, and IIRC using AdvanceMAME.

[BiQ]

It may be possible to make that version of Mame more secure by having it regularly "phone home" from the user's machine (e.g. every 5 seconds) to a central MARP authentication server of some kind which is not easily spoofed (e.g. via encryption keys), and imbedd the additional authentication information into a signed replay file. It would be a more secure way to authenticate replays, possibly preventing VMWare save state misuse. If the MARP community feels strongly about validation security, they may want to investigate such avenues.

In the distant future where broadband speed and latency is pretty freaking quick there can be something like a mame which runs on a secured server and players access this mame with a client which only handles user input / output. Game runs on the server.

Ok, there is of course the issue that how can it be checked that no one manages to hack the client to insert a cheated inp file in place of genuine user input? Making the game pause at server side at unknown (yet as unobtrusive as possible) moment so the pre-timed inp file would fail from that point forward? That would also be problematic.

Another issue would be the copyrights issue when the rom file is located at a central server for playing.

But really, all this is pointless. I don't really give a rat's ass about prize competitions and the thought that someone would even want to fake their scores on a niche gaming genre forums is nothing but p-a-t-h-e-t-i-c. Really. When one tries to get his kicks out of faking shmup scores on a hobbyist www board he TRULY has hit the rock bottom. That's just so sad I'd probalby have a hard time to even be angry at them. </rant>

[Dave_K.]

I'm sure they already know. Fact is, if you've got a display that refreshes at 60hz (for example, an LCD), and you want to display a game that updates at 59hz, you are going to have tearing or jumping somewhere. You can try this with 19XX for example. Someone tried to tell me I was wrong, and that the original board suffers the jumping issue. Er.. whatever (I own one, it doesn't ) Only way around this is a setup where the refresh rate is fully controllable. That very much depends on your graphics card, monitor, OS, and IIRC using AdvanceMAME.

Even with a graphics card that can be tweaked to a specific refresh rate, you will still not be able to get it "perfect" as the card has one clock, and your CPU another. I've worked for a month trying to get the perfect mame setup with perfect picture, and always have a ripple running through the display because DDP uses a 57.550644Hz refresh rate Even speeding up the emulated framerate to lock everything in at 60hz, my tri-res monitor sees it as 59.99998...and still get a slow ripple.