Debian bookworm (kernel 6.1) with ROCm 6.4.0.
The problem is not a FAH problem but probably an amdgpu problem. The folding core is using as much memory as it needs. It is the DRM framebuffer subsystem that fails to allocate memory (despite GTT being available) and kills the graphics, logging an error that it failed to pin the framebuffer due to error -12 (which corresponds to ENOMEM i.e. insufficient memory for the allocation). The folding core keeps merrily chugging along the whole time while using its 504 MB VRAM. It's not even aware that anything happened.
I have since blacklisted 12922-12925 so it is no longer a problem for me.