There will hopefully be more optimizations to speed this up.

Speculative, Jacobi, or lookahead decoding could speed things up quite a bit.