** Disclaimer**-- my opinions on this are based on outputting a real N64 to a 36" trinitron via RGB with the old school RGB amp method.
Ive been playing around with the MiSTer and its awesome N64 core in the past couple weeks, and found some rather interesting stuff that led down yet another interesting rabbit hole. The MiSTer, like the emulator ParaLLEl N64, emulates the N64 RCP at a low level, and enables you to set certain VI (video interface) registers. Whilst playing around on my 1440p monitor with the MiSTer, I tried different settings, but came across two that really make the N64 "pop". VI Dither, and VI Divot. Turning off dither removes a vaseline like color filter that hides dithering present in 16 bit color mode games. VI Divot does something really unique to some textures, such as the windows on the castle, the trunks of the puffy trees, and fences and bannisters in SM64---it removes some sort of specific blur filter on such objects but leaves untouched everything else. It makes them more easily visible at a distance, yet does not make them appear pixelated.
The VI Antialias, which infamously and unfairly takes most of the blame for many N64 games blurry look, actually doesnt do anything at all to add blur to textures or bitmaps, all it does is smooth line edges and intersections, and I have found that it most definitely needs to stay On/ at default for games that use AA (most of them). On games that dont use AA, toggling it makes no difference and will not apply AA to a non AA game. Anyways, enough about N64 on MiSTer.
On a real N64 and games, many of the same options available in the MiSTer core can be either enabled via game shark or pre-patched into the game ROMs. Below is a great page on ways to do this. Ive found that the U64aap044.zip program set that Saturnu created is superior to the set of .ips/.aps patches by Porregon because the U64AAP044 program allows you to individually toggle dither or divot on or off apart from turning off the AA. This is significant because when you turn off the AA via the known methods, it doesnt just turn off AA, it turns off the dither filter and the divot filter as well.
Ive found that just turning off the
dither filter is the single best sharpening mod, as it seems to lift a veil of vaseline-like smear from all the textures of 3D rendered objects. It sharpens them quite visibly. The downside to this is that on darker scenery especially (Castlevania is a prime example), the dithering is very visible and can be distracting. Personally though, for me, even heavily dithered scenes are worth the tradeoff for the blur removal. Interestingly enough, even on pure 2D games like Rampage: World Tour, turning this dither filter off really sharpens the image, making it look more crisp and with better contrast between colors, equaling or bettering the PS1 2D output, which Ive always found noticeably sharper than N64.
Second is the Divot filter. As mentioned above, it only works on seemingly 2D objects working in a 3D space, but turning it off removes a level of blur from them and looks superior in all cases that Ive seen, with no downsides that I know of.
Last is AA. Using the .ips patches and possibly Gameshark codes (not sure if you can piecemeal those) to turn off AA is misleading, because they also turn off the dither and divot filters-- the two most responsible for cleaning up the image. Those aside, turning off the AA itself is not good. Line and polygon edges become remarkably rough and jagged, and Ive not found a single game that I prefer to turn AA off vs just turning off the divot and dither filters. From all the methods Ive seen, turning off AA also turns off divot and dither, and theres no way to just turn off AA by itself-- except on the MiSTer.
Oh and also, the horizontal 640/320 blur is not possible as of yet to turn off with any of these registers, but when gaming on a very sharp 36" Trinitron I really couldnt see the difference when I tried it on an N64 RGB a few years ago, try that I did. It is possible with the MiSTer though.
In any case, if you havent dont it yet, give U64aap044.zip a try and experiment with just dither and divot filters without touching AA. The difference from those alone is very impressive to me. Some games, like SoTE, Blast Corps, and KI Gold dont work with it and I havent seen Gameshark codes for them yet, (Well I did see one for SoTE PAL version, but havent tried it) so Im guessing no one was able to find the VI register addresses for those for some reason. I have half a mind to try myself but dont know if I ever will.
https://consolemods.org/wiki/N64:N64blur