Segfault on Project: 19230 (Run 6802, Clone 20, Gen 0)

[bikeaddict]

This WU segfaults and restarts endlessly. The client is unable to detect that it's crashing multiple times and then halt it.

Code: Select all

23:10:58:WU02:FS00:0xa8:Project: 19230 (Run 6802, Clone 20, Gen 0)
23:10:58:WU02:FS00:0xa8:Unit: 0x00000000000000000000000000000000
23:10:58:WU02:FS00:0xa8:Reading tar file core.xml
23:10:58:WU02:FS00:0xa8:Reading tar file md0.tpr
23:10:58:WU02:FS00:0xa8:Digital signatures verified
23:10:58:WU02:FS00:0xa8:Calling: mdrun -c md0.gro -s md0.tpr -x md0.xtc -cpt 15 -nt 31 -ntmpi 1
23:10:58:WU02:FS00:0xa8:Steps: first=0 total=5000000
23:10:58:WU02:FS00:0xa8:Completed 1 out of 5000000 steps (0%)
23:10:59:WU02:FS00:FahCore returned: INTERRUPTED (102 = 0x66)
23:10:59:WU02:FS00:Starting
23:10:59:WU02:FS00:Running FahCore: /usr/bin/FAHCoreWrapper /var/lib/fahclient/cores/cores.foldingathome.org/lin/64bit-avx2-256/a8-0.0.12/Core_a8.fah/FahCore_a8 -dir 02 -suffix 01 -version 706 -lifeline 1602938 -checkpoint 15 -np 31
23:10:59:WU02:FS00:Started FahCore on PID 3646387
23:10:59:WU02:FS00:Core PID:3646391
23:10:59:WU02:FS00:FahCore 0xa8 started

Code: Select all

Jul 30 23:10:04 fedora kernel: FahCore_a8[3673135]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d5b1ffad0 error 5
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673133]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d5c5ffad0 error 5
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000] likely on CPU 15 (core 15, socket 0)
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000] likely on CPU 9 (core 9, socket 0)
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673139]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d589ffad0 error 5
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673136]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d5a7ffad0 error 5
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673126]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d60bffad0 error 5
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673130]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d5e3ffad0 error 5
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673127]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d601ffad0 error 5
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000] likely on CPU 30 (core 14, socket 0)
Jul 30 23:10:04 fedora kernel: FahCore_a8[3673134]: segfault at fffffffe2865d200 ip 0000000000cfda6b sp 00007f7d5bbffad0 error 5
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000]
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000]
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000]
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000]
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: likely on CPU 18 (core 2, socket 0)
Jul 30 23:10:04 fedora kernel: likely on CPU 2 (core 2, socket 0) 
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: likely on CPU 16 (core 0, socket 0) 
Jul 30 23:10:04 fedora kernel: likely on CPU 13 (core 13, socket 0)
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: in FahCore_a8[406000+e6d000]
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: likely on CPU 24 (core 8, socket 0)
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: 
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Jul 30 23:10:04 fedora kernel: Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4

Code: Select all

Process 3673111 (FahCore_a8) of user 0 dumped core.#012#012
Stack trace of thread 3673129:#012#0  0x0000000000cfda6b n/a (FahCore_a8 + 0x8fda6b)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673111:#012#0  0x0000000000ef6f5e n/a (FahCore_a8 + 0xaf6f5e)#012#1  0x0000000000ef5fc6 n/a (FahCore_a8 + 0xaf5fc6)#012#2  0x00000000008c5a92 n/a (FahCore_a8 + 0x4c5a92)#012#3  0x00000000008933a0 n/a (FahCore_a8 + 0x4933a0)#012#4  0x000000000072be66 n/a (FahCore_a8 + 0x32be66)#012#5  0x0000000000727f3d n/a (FahCore_a8 + 0x327f3d)#012#6  0x000000000048c0e1 n/a (FahCore_a8 + 0x8c0e1)#012#7  0x0000000000454d2c n/a (FahCore_a8 + 0x54d2c)#012#8  0x0000000000440041 n/a (FahCore_a8 + 0x40041)#012#9  0x0000000000f1d923 n/a (FahCore_a8 + 0xb1d923)#012#10 0x0000000000f1da45 n/a (FahCore_a8 + 0xb1da45)#012#11 0x000000000043f46e n/a (FahCore_a8 + 0x3f46e)#012#12 0x00007f7d71608088 __libc_start_call_main (libc.so.6 + 0x2a088)#012#13 0x00007f7d7160814b __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a14b)#012#14 0x000000000044600a n/a (FahCore_a8 + 0x4600a)#012#012
Stack trace of thread 3673116:#012#0  0x0000000000cfeb04 n/a (FahCore_a8 + 0x8feb04)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673115:#012#0  0x00007f7d71810404 cbrtf32 (libm.so.6 + 0x41404)#012#1  0x0000000000cfe54d n/a (FahCore_a8 + 0x8fe54d)#012#2  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#3  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#4  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#5  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673117:#012#0  0x0000000000cfdbf6 n/a (FahCore_a8 + 0x8fdbf6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673118:#012#0  0x0000000000cfdbc6 n/a (FahCore_a8 + 0x8fdbc6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673119:#012#0  0x0000000000cfdbc6 n/a (FahCore_a8 + 0x8fdbc6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673120:#012#0  0x0000000000cfdbaa n/a (FahCore_a8 + 0x8fdbaa)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673121:#012#0  0x00007f7d71810375 cbrtf32 (libm.so.6 + 0x41375)#012#1  0x0000000000cfe3db n/a (FahCore_a8 + 0x8fe3db)#012#2  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#3  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#4  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#5  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673122:#012#0  0x0000000000cfdc2e n/a (FahCore_a8 + 0x8fdc2e)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673135:#012#0  0x0000000000cfda6b n/a (FahCore_a8 + 0x8fda6b)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673123:#012#0  0x0000000000cfdbaa n/a (FahCore_a8 + 0x8fdbaa)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673124:#012#0  0x0000000000cfdbf6 n/a (FahCore_a8 + 0x8fdbf6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673125:#012#0  0x0000000000cfdbc6 n/a (FahCore_a8 + 0x8fdbc6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673128:#012#0  0x0000000000cfdbaa n/a (FahCore_a8 + 0x8fdbaa)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673131:#012#0  0x0000000000cfd33e n/a (FahCore_a8 + 0x8fd33e)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673137:#012#0  0x00007f7d71810432 cbrtf32 (libm.so.6 + 0x41432)#012#1  0x0000000000cfe3db n/a (FahCore_a8 + 0x8fe3db)#012#2  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#3  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#4  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#5  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673133:#012#0  0x0000000000cfda6b n/a (FahCore_a8 + 0x8fda6b)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673132:#012#0  0x0000000000cfdbc6 n/a (FahCore_a8 + 0x8fdbc6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673138:#012#0  0x00000000004063a0 n/a (FahCore_a8 + 0x63a0)#012#1  0x0000000000cfe54d n/a (FahCore_a8 + 0x8fe54d)#012#2  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#3  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#4  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#5  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673140:#012#0  0x0000000000cfdbc6 n/a (FahCore_a8 + 0x8fdbc6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673141:#012#0  0x0000000000ef6f5e n/a (FahCore_a8 + 0xaf6f5e)#012#1  0x0000000000ef4b6a n/a (FahCore_a8 + 0xaf4b6a)#012#2  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#3  0x00007f7d716f940c __clone3 (libc.so.6 + 0x11b40c)#012#012
Stack trace of thread 3673142:#012#0  0x0000000000cfdbc6 n/a (FahCore_a8 + 0x8fdbc6)#012#1  0x00000000008c48b4 n/a (FahCore_a8 + 0x4c48b4)#012#2  0x0000000000ef4b5e n/a (FahCore_a8 + 0xaf4b5e)#012#3  0x00007f7d71675507 start_thread (libc.so.6 + 0x97507)#012#4  0x00007f7d716f940c __clo

[toTOW]

I reported it to the researcher.

[YInMn]

Im having a similar issue with the WU: Project 19230 (Run 332, Clone 20, Gen 0)
The WU fails indefinitely.

Info from the logfile:

Code: Select all

18:46:24:WU01:FS00:FahCore 0xa8 started
18:46:24:WU01:FS00:0xa8:Project: 19230 (Run 332, Clone 20, Gen 0)
18:46:24:WU01:FS00:0xa8:Unit: 0x00000000000000000000000000000000
18:46:24:WU01:FS00:0xa8:Digital signatures verified
18:46:24:WU01:FS00:0xa8:Calling: mdrun -c md0.gro -s md0.tpr -x md0.xtc -cpt 3 -nt 8 -ntmpi 1
18:46:24:WU01:FS00:0xa8:Steps: first=0 total=5000000
18:46:25:WU01:FS00:0xa8:Completed 1 out of 5000000 steps (0%)
18:46:25:WU01:FS00:FahCore returned: INTERRUPTED (102 = 0x66)

error in the system log:

Code: Select all

Aug 03 20:46:26 systemd-coredump[27456]: Process 27443 (FahCore_a8) of user 62464 dumped core.
Stack trace of thread 27450:
#0  0x0000000000cfda6b n/a (/var/lib/private/fah/cores/cores.foldingathome.org/lin/64bit-avx2-256/a8-0.0.12/Core_a8.fah/FahCore_a8 + 0x8fda6b)
ELF object binary architecture: AMD x86-64

dmesg entry:

Code: Select all

FahCore_a8[21824]: segfault at fffffffe14cc5000 ip 0000000000cfda6b sp 0000758ad73ffad0 error 5
FahCore_a8[21826]: segfault at fffffffe14cc5000 ip 0000000000cfda6b sp 0000758ad5fffad0 error 5
in FahCore_a8[8fda6b,406000+e6d000]
in FahCore_a8[8fda6b,406000+e6d000]
likely on CPU 30 (core 14, socket 0)
likely on CPU 29 (core 13, socket 0)

Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4
Code: 00 c5 fa 10 35 5e 93 58 00 4c 8b 84 24 e0 01 00 00 c4 c1 72 59 ce c5 72 59 9c 24 c8 01 00 00 c4 c1 7a 2c c3 c5 fa 2a c0 48 98 <c4> c1 7a 10 14 81 c5 22 5c e0 c4 41 1a 59 6c 81 04 c4 41 4a 5c c4

[bikeaddict]

Just got similar crashes with these:

Project: 19230 (Run 783, Clone 20, Gen 0)
Project: 19230 (Run 1023, Clone 20, Gen 0)

[toTOW]

The researcher is looking into this issue.

[Frontiers]

No any issues with any R-C-G of P19230 with my setup. With different distro though - Mint 21.3 with kernel 6.5, few days ago updated to kernel 6.8, but still no new P19230 with this updated kernel. It's fastest CPU project that my hardware ever received.

[toTOW]

I forgot to post an update here.

Clone 20 on all Runs has an issue. The researcher turned this particular clone off. He will restart this clone as a new project once the issue has been fixed.