Is there anyway to force the client to compress the return work package?
If not is there a wish list of features I can add it to?
Last couple of days I have seen 350 mb file returns and my daily usage has exploded to 15 - 20 gb.
Normal usage is around 4 - 6 gb.
Thanks
Compressing Returned Work
Moderators: Site Moderators, FAHC Science Team
-
muziqaz
- Posts: 2291
- Joined: Sun Dec 16, 2007 6:22 pm
- Hardware configuration: 9950x, 9950x3d, 7950x3d, 5950x, 5800x3d
7900xtx, RX9070, Radeon 7, 5700xt, 6900xt, RX550, Intel B580 - Location: London
- Contact:
Re: Compressing Returned Work
All WUs are compressed when returned. This is set by the researcher before being released
FAH Omega tester
-
Artoria2e5
- Posts: 6
- Joined: Mon Dec 15, 2025 2:48 pm
Re: Compressing Returned Work
The "see also" links mainly concern the packaging of the cores. At that time I didn't look too much into returning work as my bottleneck was download time (download time has the processor idle, upload... i don't know actually!). In addition, multi-threaded LZMA2/xz isn't really necessary as single-thread decompression is already much faster than bzip. It's just a nice-to-have.
I cannot confirm that there's result compression, though naturally one would expect some. This is because the code for making wuresult_01.dat is not in the FoldingAtHome github organization -- I think the core is responsible for doing that and there's just no code for the core-adapter part. Perhaps I can use some data-recovery software to dig up one of the done files and see how it's packed.
I think a more immediate concern is https://github.com/FoldingAtHome/fah-cl ... issues/350, which causes some expansion in code size due to transferring stuff as base64.
I cannot confirm that there's result compression, though naturally one would expect some. This is because the code for making wuresult_01.dat is not in the FoldingAtHome github organization -- I think the core is responsible for doing that and there's just no code for the core-adapter part. Perhaps I can use some data-recovery software to dig up one of the done files and see how it's packed.
I think a more immediate concern is https://github.com/FoldingAtHome/fah-cl ... issues/350, which causes some expansion in code size due to transferring stuff as base64.
-
Artoria2e5
- Posts: 6
- Joined: Mon Dec 15, 2025 2:48 pm
Re: Compressing Returned Work
I still don't have access to a sample of wuresult_01.dat. But based on the current bunch of files in the work folders, my guess is that it includes at least the final positions.xtc. This XTC format from gromacs is compressed both by reducing precision and by a kind of delta-coding described in https://github.com/Pappulab/xdrf/blob/2 ... o.txt#L103. I don't expect its output to be very amenable to further compression, so I guess this is basically it.
Because making wuresult_01.dat seems to be the core's job, it's also possible that different cores have wildly different ideas about what should be in them. The best you can do for figuring out what's going on is to copy one of these huge dats to somewhere FAHclient won't delete (like your desktop!) for analysis. Very likely it's in a format that 7-zip can open (I'm guessing tar), so you can do that and make a screenshot of the file names and sizes. Alternatively you can just write on the forum which core is doing that and someone else running the same core might do it for you.
I am running mainly GPU so I probably won't be of much help. My current Core27 job has 30 minutes to go. If I remember to come back I might be able to the wuresult_01.dat and check out what's in it.
Because making wuresult_01.dat seems to be the core's job, it's also possible that different cores have wildly different ideas about what should be in them. The best you can do for figuring out what's going on is to copy one of these huge dats to somewhere FAHclient won't delete (like your desktop!) for analysis. Very likely it's in a format that 7-zip can open (I'm guessing tar), so you can do that and make a screenshot of the file names and sizes. Alternatively you can just write on the forum which core is doing that and someone else running the same core might do it for you.
I am running mainly GPU so I probably won't be of much help. My current Core27 job has 30 minutes to go. If I remember to come back I might be able to the wuresult_01.dat and check out what's in it.