Unlucky for me, but the ADC Gain did not solve the issues on the image shown in the previous post. I have extracted some more data from the message window of the UI. I started the Neo Geo system, printed some info, got the video timings, set the video to 640x480, set the system to 1280x960 and got the video timings again. Here is the output:
Spoiler
SD mode
resetDigital
hcoast: 1728
Base: 1802 Best: 1821 Diff: 19
post preset done
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:80
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^_ ign:64 stat:2.10 deb:5a0 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a0 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:584 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a0 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5b4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:_^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:_^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^_ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:431 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
h:430 v:527 PLL:^^ ign:64 stat:2.10 deb:5a4 m:1 ht:2132 vt:261 hpw:161
----
htotal: 1821
HS ST: 65
HS SP: 279
HB ST (display): 1820
HB SP (display): 496
HB ST (memory): 1744
HB SP (memory): 352
----
vtotal: 1000
VS ST: 987
VS SP: 990
VB ST (display): 986
VB SP (display): 26
VB ST (memory): 21
VB SP (memory): 22
NTSC
resetDigital
hcoast: 1724
Base: 857 Best: 866 Diff: 9
post preset done
NTSC
SD mode
resetDigital
hcoast: 1728
Base: 1802 Best: 1821 Diff: 19
post preset done
----
htotal: 1821
HS ST: 65
HS SP: 279
HB ST (display): 1820
HB SP (display): 496
HB ST (memory): 1744
HB SP (memory): 352
----
vtotal: 1000
VS ST: 987
VS SP: 990
VB ST (display): 986
VB SP (display): 26
VB ST (memory): 21
VB SP (memory): 22
Images with "vanilla" GBS: