Dmian
As you know, that was my first idea when I came across the FabGL board. It seemed like the perfect target. It could even have been implemented similar to the PicoMite VGA, but instead of one core driving the video, it could have been a second ESP32 board. They are not particularly expensive. I think this is the approach that the Argon uses. It has the Z80 compatible CPU (and also soon the 6502 compatible one) and then uses an ESP32 as the video chip.
One of the good decisions Uli made in the first version of the BASIC Engine was the choice of the SNES controller. It uses a lot less I/O lines to give a lot of buttons, and only and extra 1 I/O line for extra controllers.
I keep an eye on as many of the similar projects as possible.