8004 has six DDR5 channels afaik. That takes care of the memory bandwidth. The only issue would be an SP6 motherboard.

3090 doesn’t support PCIE 5.0, only 4.0

The 4090 does, and it makes some sense to use them in x8 5.0 configuration, but only if you have a pallet of these GPUs.

Which 34B? CodeLLaMAs or Yi-34B?

Hmm… With what models?

What settings did you use? I didn’t run into this issue, in fact, min-P sampling helps taming the chaotic nature of some models without putting them on rails.

Isn’t DynaTemp just that?