F@h Keeps crashing, Not sure where to start
Posted: Thu Feb 02, 2012 10:49 pm
Man, my app keeps crashing with the following:
This recently started happening. I get the following crash. Constantly. Sometimes one of my GPUs, Sometimes my CPU.
My log is below.
It's on my M17x R2. I'm using the latest V7 beta. 
This recently started happening. I get the following crash. Constantly. Sometimes one of my GPUs, Sometimes my CPU.
Code: Select all
Problem signature:
Problem Event Name: APPCRASH
Application Name: FahCore_a4.exe
Application Version: 0.0.0.0
Application Timestamp: 4d23eafc
Fault Module Name: ntdll.dll
Fault Module Version: 6.1.7601.17725
Fault Module Timestamp: 4ec49b8f
Exception Code: c0000029
Exception Offset: 00090812
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789Code: Select all
*********************** Log Started 2012-02-02T21:28:06 ************************
21:28:06:************************* Folding@home Client *************************
21:28:06: Website: http://folding.stanford.edu/
21:28:06: Copyright: (c) 2009-2012 Stanford University
21:28:06: Author: Joseph Coffland <joseph@cauldrondevelopment.com>
21:28:06: Args: --lifeline 3000 --command-port=36330
21:28:06: Config: C:/Users/D.Tillman/AppData/Roaming/FAHClient/config.xml
21:28:06:******************************** Build ********************************
21:28:06: Version: 7.1.43
21:28:06: Date: Jan 2 2012
21:28:06: Time: 12:33:05
21:28:06: SVN Rev: 3223
21:28:06: Branch: fah/trunk/client
21:28:06: Compiler: Intel(R) C++ MSVC 1500 mode 1200
21:28:06: Options: /TP /nologo /EHa /Qdiag-disable:4297,4103,1786,279 /Ox -arch:SSE
21:28:06: /QaxSSE2,SSE3,SSSE3,SSE4.1,SSE4.2 /Qopenmp /Qrestrict /MT
21:28:06: Platform: win32 XP
21:28:06: Bits: 32
21:28:06: Mode: Release
21:28:06:******************************* System ********************************
21:28:06: CPU: Intel(R) Core(TM) i7 CPU X 940 @ 2.13GHz
21:28:06: CPU ID: GenuineIntel Family 6 Model 30 Stepping 5
21:28:06: CPUs: 8
21:28:06: Memory: 15.99GiB
21:28:06: Free Memory: 14.02GiB
21:28:06: Threads: WINDOWS_THREADS
21:28:06: On Battery: false
21:28:06: UTC offset: -5
21:28:06: PID: 5664
21:28:06: CWD: C:/Users/D.Tillman/AppData/Roaming/FAHClient
21:28:06: OS: Windows 7 Ultimate
21:28:06: OS Arch: AMD64
21:28:06: GPUs: 2
21:28:06: GPU 0: ATI:4 Whistler XT [AMD Radeon HD 6970M]
21:28:06: GPU 1: ATI:4 Whistler XT [AMD Radeon HD 6970M]
21:28:06: CUDA: Not detected
21:28:06:Win32 Service: false
21:28:06:***********************************************************************
21:28:07:<config>
21:28:07: <!-- FahCore Control -->
21:28:07: <checkpoint v='20'/>
21:28:07:
21:28:07: <!-- Folding Slot Configuration -->
21:28:07: <gpu v='true'/>
21:28:07:
21:28:07: <!-- Network -->
21:28:07: <proxy v=':8080'/>
21:28:07:
21:28:07: <!-- User Information -->
21:28:07: <passkey v='********************************'/>
21:28:07: <team v='213698'/>
21:28:07: <user v='DR650SE'/>
21:28:07:
21:28:07: <!-- Folding Slots -->
21:28:07: <slot id='0' type='GPU'>
21:28:07: <next-unit-percentage v='90'/>
21:28:07: </slot>
21:28:07: <slot id='1' type='GPU'>
21:28:07: <next-unit-percentage v='90'/>
21:28:07: </slot>
21:28:07: <slot id='2' type='SMP'>
21:28:07: <cpus v='6'/>
21:28:07: <next-unit-percentage v='90'/>
21:28:07: </slot>
21:28:07:</config>
21:28:07:Trying to access database...
21:28:07:Successfully acquired database lock
21:28:07:Enabled folding slot 00: READY gpu:0:"Whistler XT [AMD Radeon HD 6970M]"
21:28:07:Enabled folding slot 01: READY gpu:1:"Whistler XT [AMD Radeon HD 6970M]"
21:28:07:Enabled folding slot 02: READY smp:6
21:28:07:WU03:FS01:Starting
21:28:07:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/D.Tillman/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 03 -suffix 01 -version 701 -checkpoint 20 -gpu 1
21:28:07:WU03:FS01:Started FahCore on PID 1692
21:28:07:WU03:FS01:Core PID:4016
21:28:07:WU03:FS01:FahCore 0x16 started
21:28:07:WU01:FS00:Starting
21:28:07:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/D.Tillman/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 01 -suffix 01 -version 701 -checkpoint 20 -gpu 0
21:28:07:WU01:FS00:Started FahCore on PID 1980
21:28:07:WU01:FS00:Core PID:4212
21:28:07:WU01:FS00:FahCore 0x16 started
21:28:07:WU02:FS02:Starting
21:28:07:WU02:FS02:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/D.Tillman/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/Core_a4.fah/FahCore_a4.exe -dir 02 -suffix 01 -version 701 -checkpoint 20 -np 6
21:28:07:WU02:FS02:Started FahCore on PID 3940
21:28:07:WU02:FS02:Core PID:6856
21:28:07:WU02:FS02:FahCore 0xa4 started
21:28:07:WU03:FS01:0x16:
21:28:07:WU03:FS01:0x16:*------------------------------*
21:28:07:WU03:FS01:0x16:Folding@Home GPU Core
21:28:07:WU03:FS01:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
21:28:07:WU03:FS01:0x16:
21:28:07:WU01:FS00:0x16:
21:28:07:WU03:FS01:0x16:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
21:28:07:WU01:FS00:0x16:*------------------------------*
21:28:07:WU03:FS01:0x16:Build host: user-f6d030f24f
21:28:07:WU01:FS00:0x16:Folding@Home GPU Core
21:28:07:WU03:FS01:0x16:Board Type: AMD/OpenCL
21:28:07:WU01:FS00:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
21:28:07:WU03:FS01:0x16:Core : x=16
21:28:07:WU02:FS02:0xa4:
21:28:07:WU01:FS00:0x16:
21:28:07:WU03:FS01:0x16: Window's signal control handler registered.
21:28:07:WU02:FS02:0xa4:*------------------------------*
21:28:07:WU01:FS00:0x16:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
21:28:07:WU03:FS01:0x16:Preparing to commence simulation
21:28:07:WU02:FS02:0xa4:Folding@Home Gromacs GB Core
21:28:07:WU01:FS00:0x16:Build host: user-f6d030f24f
21:28:07:WU03:FS01:0x16:- Looking at optimizations...
21:28:07:WU02:FS02:0xa4:Version 2.27 (Dec. 15, 2010)
21:28:07:WU01:FS00:0x16:Board Type: AMD/OpenCL
21:28:07:WU03:FS01:0x16:- Files status OK
21:28:07:WU02:FS02:0xa4:
21:28:07:WU01:FS00:0x16:Core : x=16
21:28:07:WU03:FS01:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
21:28:07:WU02:FS02:0xa4:Preparing to commence simulation
21:28:07:WU01:FS00:0x16: Window's signal control handler registered.
21:28:07:WU03:FS01:0x16:- Expanded 42495 -> 171163 (decompressed 402.7 percent)
21:28:07:WU02:FS02:0xa4:- Looking at optimizations...
21:28:07:WU01:FS00:0x16:Preparing to commence simulation
21:28:08:WU03:FS01:0x16:Called DecompressByteArray: compressed_data_size=42495 data_size=171163, decompressed_data_size=171163 diff=0
21:28:08:WU02:FS02:0xa4:- Files status OK
21:28:08:WU01:FS00:0x16:- Looking at optimizations...
21:28:08:WU03:FS01:0x16:- Digital signature verified
21:28:08:WU02:FS02:0xa4:- Expanded 2079520 -> 5386224 (decompressed 259.0 percent)
21:28:08:WU01:FS00:0x16:- Files status OK
21:28:08:WU03:FS01:0x16:
21:28:08:WU02:FS02:0xa4:Called DecompressByteArray: compressed_data_size=2079520 data_size=5386224, decompressed_data_size=5386224 diff=0
21:28:08:WU01:FS00:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
21:28:08:WU03:FS01:0x16:Project: 11293 (Run 1, Clone 407, Gen 0)
21:28:08:WU02:FS02:0xa4:- Digital signature verified
21:28:08:WU01:FS00:0x16:- Expanded 44853 -> 171163 (decompressed 381.6 percent)
21:28:08:WU03:FS01:0x16:
21:28:08:WU02:FS02:0xa4:
21:28:08:WU01:FS00:0x16:Called DecompressByteArray: compressed_data_size=44853 data_size=171163, decompressed_data_size=171163 diff=0
21:28:08:WU03:FS01:0x16:Assembly optimizations on if available.
21:28:08:WU02:FS02:0xa4:Project: 7809 (Run 3, Clone 327, Gen 33)
21:28:08:WU01:FS00:0x16:- Digital signature verified
21:28:08:WU03:FS01:0x16:Entering M.D.
21:28:08:WU02:FS02:0xa4:
21:28:08:WU01:FS00:0x16:
21:28:08:WU02:FS02:0xa4:Assembly optimizations on if available.
21:28:08:WU01:FS00:0x16:Project: 11293 (Run 0, Clone 243, Gen 2)
21:28:08:WU02:FS02:0xa4:Entering M.D.
21:28:08:WU01:FS00:0x16:
21:28:08:WU01:FS00:0x16:Assembly optimizations on if available.
21:28:08:WU01:FS00:0x16:Entering M.D.
21:28:09:WU03:FS01:0x16:Will resume from checkpoint file 03/wudata_01.ckp
21:28:09:WU03:FS01:0x16:Tpr hash 03/wudata_01.tpr: 428829383 4171448299 3993183890 4073803069 1795910398
21:28:09:WU03:FS01:0x16:Working on ALZHEIMER DISEASE AMYLOID
21:28:09:WU03:FS01:0x16:Client config unavailable.
21:28:09:WU03:FS01:0x16:Starting GUI Server
21:28:09:WU01:FS00:0x16:Will resume from checkpoint file 01/wudata_01.ckp
21:28:09:WU01:FS00:0x16:Tpr hash 01/wudata_01.tpr: 537407621 3550239141 1616066183 4199144504 2505667728
21:28:09:WU01:FS00:0x16:Working on ALZHEIMER DISEASE AMYLOID
21:28:09:WU01:FS00:0x16:Client config unavailable.
21:28:09:WU01:FS00:0x16:Starting GUI Server
21:28:10:Server connection id=1 on 0.0.0.0:36330 from 127.0.0.1
21:28:13:WU02:FS02:0xa4:Using Gromacs checkpoints
21:28:13:WU02:FS02:0xa4:Mapping NT from 6 to 6
21:28:14:WU02:FS02:0xa4:Resuming from checkpoint
21:28:14:WU02:FS02:0xa4:Verified 02/wudata_01.log
21:28:14:WU02:FS02:0xa4:Verified 02/wudata_01.trr
21:28:14:WU02:FS02:0xa4:Verified 02/wudata_01.xtc
21:28:14:WU02:FS02:0xa4:Verified 02/wudata_01.edr
21:28:14:WU02:FS02:0xa4:Completed 775880 out of 1500000 steps (51%)
21:28:16:WU03:FS01:0x16:Resuming from checkpoint
21:28:16:WU03:FS01:0x16:fcCheckPointResume: retreived and current tpr file hash:
21:28:16:WU03:FS01:0x16: 0 428829383 428829383
21:28:16:WU03:FS01:0x16: 1 4171448299 4171448299
21:28:16:WU03:FS01:0x16: 2 3993183890 3993183890
21:28:16:WU03:FS01:0x16: 3 4073803069 4073803069
21:28:16:WU03:FS01:0x16: 4 1795910398 1795910398
21:28:16:WU03:FS01:0x16:fcCheckPointResume: file hashes same.
21:28:16:WU03:FS01:0x16:fcCheckPointResume: state restored.
21:28:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.log Verified 03/wudata_01.log
21:28:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.trr Verified 03/wudata_01.trr
21:28:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.xtc Verified 03/wudata_01.xtc
21:28:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.edr Verified 03/wudata_01.edr
21:28:16:WU03:FS01:0x16:fcCheckPointResume: state restored 2
21:28:16:WU03:FS01:0x16:Resumed from checkpoint
21:28:16:WU03:FS01:0x16:Setting checkpoint frequency: 500000
21:28:16:WU03:FS01:0x16:Completed 41500001 out of 50000000 steps (83%).
21:28:16:WU01:FS00:0x16:Resuming from checkpoint
21:28:16:WU01:FS00:0x16:fcCheckPointResume: retreived and current tpr file hash:
21:28:16:WU01:FS00:0x16: 0 537407621 537407621
21:28:16:WU01:FS00:0x16: 1 3550239141 3550239141
21:28:16:WU01:FS00:0x16: 2 1616066183 1616066183
21:28:16:WU01:FS00:0x16: 3 4199144504 4199144504
21:28:16:WU01:FS00:0x16: 4 2505667728 2505667728
21:28:16:WU01:FS00:0x16:fcCheckPointResume: file hashes same.
21:28:16:WU01:FS00:0x16:fcCheckPointResume: state restored.
21:28:16:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.log Verified 01/wudata_01.log
21:28:16:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.trr Verified 01/wudata_01.trr
21:28:16:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.xtc Verified 01/wudata_01.xtc
21:28:16:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.edr Verified 01/wudata_01.edr
21:28:17:WU01:FS00:0x16:fcCheckPointResume: state restored 2
21:28:17:WU01:FS00:0x16:Resumed from checkpoint
21:28:17:WU01:FS00:0x16:Setting checkpoint frequency: 500000
21:28:17:WU01:FS00:0x16:Completed 37500001 out of 50000000 steps (75%).
21:32:28:WU03:FS01:0x16:Completed 42000000 out of 50000000 steps (84%).
21:32:29:WU01:FS00:0x16:Completed 38000000 out of 50000000 steps (76%).
21:33:27:WU02:FS02:0xa4:Completed 780000 out of 1500000 steps (52%)
21:37:06:WU03:FS01:0x16:Completed 42500000 out of 50000000 steps (85%).
21:37:06:WU01:FS00:0x16:Completed 38500000 out of 50000000 steps (77%).
21:41:43:WU03:FS01:0x16:Completed 43000000 out of 50000000 steps (86%).
21:41:43:WU01:FS00:0x16:Completed 39000000 out of 50000000 steps (78%).
21:46:21:WU03:FS01:0x16:Completed 43500000 out of 50000000 steps (87%).
21:46:22:WU01:FS00:0x16:Completed 39500000 out of 50000000 steps (79%).
21:51:42:WU02:FS02:0xa4:Completed 795000 out of 1500000 steps (53%)
22:08:26:WU02:FS02:0xa4:Completed 810000 out of 1500000 steps (54%)
22:24:52:WU03:FS01:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
22:24:52:WARNING:WU03:FS01:FahCore returned an unknown error code which probably indicates that it crashed
22:24:52:WU03:FS01:Starting
22:24:52:WU03:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/D.Tillman/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 03 -suffix 01 -version 701 -checkpoint 20 -gpu 1
22:24:52:WU03:FS01:Started FahCore on PID 5624
22:24:52:WU03:FS01:Core PID:5680
22:24:52:WU03:FS01:FahCore 0x16 started
22:24:52:WU03:FS01:0x16:
22:24:52:WU03:FS01:0x16:*------------------------------*
22:24:52:WU03:FS01:0x16:Folding@Home GPU Core
22:24:52:WU03:FS01:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
22:24:52:WU03:FS01:0x16:
22:24:52:WU03:FS01:0x16:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
22:24:52:WU03:FS01:0x16:Build host: user-f6d030f24f
22:24:52:WU03:FS01:0x16:Board Type: AMD/OpenCL
22:24:52:WU03:FS01:0x16:Core : x=16
22:24:52:WU03:FS01:0x16: Window's signal control handler registered.
22:24:52:WU03:FS01:0x16:Preparing to commence simulation
22:24:52:WU03:FS01:0x16:- Ensuring status. Please wait.
22:24:53:WU01:FS00:FahCore returned: UNKNOWN_ENUM (-1073741819 = 0xc0000005)
22:24:53:WARNING:WU01:FS00:FahCore returned an unknown error code which probably indicates that it crashed
22:24:53:WU01:FS00:Starting
22:24:53:WU01:FS00:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/D.Tillman/AppData/Roaming/FAHClient/cores/www.stanford.edu/~pande/Win32/AMD64/ATI/R600/Core_16.fah/FahCore_16.exe -dir 01 -suffix 01 -version 701 -checkpoint 20 -gpu 0
22:24:53:WU01:FS00:Started FahCore on PID 4040
22:24:53:WU01:FS00:Core PID:5512
22:24:53:WU01:FS00:FahCore 0x16 started
22:24:53:WU01:FS00:0x16:
22:24:53:WU01:FS00:0x16:*------------------------------*
22:24:53:WU01:FS00:0x16:Folding@Home GPU Core
22:24:53:WU01:FS00:0x16:Version 2.11 (Thu Dec 9 15:00:14 PST 2010)
22:24:53:WU01:FS00:0x16:
22:24:53:WU01:FS00:0x16:Compiler : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.30729.01 for 80x86
22:24:53:WU01:FS00:0x16:Build host: user-f6d030f24f
22:24:53:WU01:FS00:0x16:Board Type: AMD/OpenCL
22:24:53:WU01:FS00:0x16:Core : x=16
22:24:53:WU01:FS00:0x16: Window's signal control handler registered.
22:24:53:WU01:FS00:0x16:Preparing to commence simulation
22:24:53:WU01:FS00:0x16:- Ensuring status. Please wait.
22:25:01:WU03:FS01:0x16:- Looking at optimizations...
22:25:01:WU03:FS01:0x16:- Working with standard loops on this execution.
22:25:01:WU03:FS01:0x16:- Previous termination of core was improper.
22:25:01:WU03:FS01:0x16:- Files status OK
22:25:01:WU03:FS01:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
22:25:01:WU03:FS01:0x16:- Expanded 42495 -> 171163 (decompressed 402.7 percent)
22:25:01:WU03:FS01:0x16:Called DecompressByteArray: compressed_data_size=42495 data_size=171163, decompressed_data_size=171163 diff=0
22:25:01:WU03:FS01:0x16:- Digital signature verified
22:25:01:WU03:FS01:0x16:
22:25:01:WU03:FS01:0x16:Project: 11293 (Run 1, Clone 407, Gen 0)
22:25:02:WU03:FS01:0x16:
22:25:02:WU03:FS01:0x16:Entering M.D.
22:25:02:WU01:FS00:0x16:- Looking at optimizations...
22:25:02:WU01:FS00:0x16:- Working with standard loops on this execution.
22:25:02:WU01:FS00:0x16:- Previous termination of core was improper.
22:25:02:WU01:FS00:0x16:- Files status OK
22:25:02:WU01:FS00:0x16:sizeof(CORE_PACKET_HDR) = 512 file=<>
22:25:02:WU01:FS00:0x16:- Expanded 44853 -> 171163 (decompressed 381.6 percent)
22:25:02:WU01:FS00:0x16:Called DecompressByteArray: compressed_data_size=44853 data_size=171163, decompressed_data_size=171163 diff=0
22:25:02:WU01:FS00:0x16:- Digital signature verified
22:25:02:WU01:FS00:0x16:
22:25:02:WU01:FS00:0x16:Project: 11293 (Run 0, Clone 243, Gen 2)
22:25:02:WU01:FS00:0x16:
22:25:02:WU01:FS00:0x16:Entering M.D.
22:25:04:WU03:FS01:0x16:Will resume from checkpoint file 03/wudata_01.ckp
22:25:04:WU03:FS01:0x16:Tpr hash 03/wudata_01.tpr: 428829383 4171448299 3993183890 4073803069 1795910398
22:25:04:WU03:FS01:0x16:Working on ALZHEIMER DISEASE AMYLOID
22:25:04:WU03:FS01:0x16:Client config unavailable.
22:25:04:WU03:FS01:0x16:Starting GUI Server
22:25:04:WU01:FS00:0x16:Will resume from checkpoint file 01/wudata_01.ckp
22:25:04:WU01:FS00:0x16:Tpr hash 01/wudata_01.tpr: 537407621 3550239141 1616066183 4199144504 2505667728
22:25:04:WU01:FS00:0x16:Working on ALZHEIMER DISEASE AMYLOID
22:25:04:WU01:FS00:0x16:Client config unavailable.
22:25:05:WU01:FS00:0x16:Starting GUI Server
22:25:10:WU02:FS02:0xa4:Completed 825000 out of 1500000 steps (55%)
22:25:16:WU03:FS01:0x16:Resuming from checkpoint
22:25:16:WU03:FS01:0x16:fcCheckPointResume: retreived and current tpr file hash:
22:25:16:WU03:FS01:0x16: 0 428829383 428829383
22:25:16:WU03:FS01:0x16: 1 4171448299 4171448299
22:25:16:WU03:FS01:0x16: 2 3993183890 3993183890
22:25:16:WU03:FS01:0x16: 3 4073803069 4073803069
22:25:16:WU03:FS01:0x16: 4 1795910398 1795910398
22:25:16:WU03:FS01:0x16:fcCheckPointResume: file hashes same.
22:25:16:WU03:FS01:0x16:fcCheckPointResume: state restored.
22:25:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.log Verified 03/wudata_01.log
22:25:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.trr Verified 03/wudata_01.trr
22:25:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.xtc Verified 03/wudata_01.xtc
22:25:16:WU03:FS01:0x16:fcCheckPointResume: name 03/wudata_01.edr Verified 03/wudata_01.edr
22:25:16:WU03:FS01:0x16:fcCheckPointResume: state restored 2
22:25:16:WU03:FS01:0x16:Resumed from checkpoint
22:25:16:WU03:FS01:0x16:Setting checkpoint frequency: 500000
22:25:16:WU03:FS01:0x16:Completed 43000001 out of 50000000 steps (86%).
22:25:19:WU01:FS00:0x16:Resuming from checkpoint
22:25:19:WU01:FS00:0x16:fcCheckPointResume: retreived and current tpr file hash:
22:25:19:WU01:FS00:0x16: 0 537407621 537407621
22:25:19:WU01:FS00:0x16: 1 3550239141 3550239141
22:25:19:WU01:FS00:0x16: 2 1616066183 1616066183
22:25:19:WU01:FS00:0x16: 3 4199144504 4199144504
22:25:19:WU01:FS00:0x16: 4 2505667728 2505667728
22:25:19:WU01:FS00:0x16:fcCheckPointResume: file hashes same.
22:25:19:WU01:FS00:0x16:fcCheckPointResume: state restored.
22:25:19:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.log Verified 01/wudata_01.log
22:25:19:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.trr Verified 01/wudata_01.trr
22:25:19:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.xtc Verified 01/wudata_01.xtc
22:25:19:WU01:FS00:0x16:fcCheckPointResume: name 01/wudata_01.edr Verified 01/wudata_01.edr
22:25:19:WU01:FS00:0x16:fcCheckPointResume: state restored 2
22:25:19:WU01:FS00:0x16:Resumed from checkpoint
22:25:19:WU01:FS00:0x16:Setting checkpoint frequency: 500000
22:25:19:WU01:FS00:0x16:Completed 39000001 out of 50000000 steps (78%).
22:29:27:WU01:FS00:0x16:Completed 39500000 out of 50000000 steps (79%).
22:29:33:WU03:FS01:0x16:Completed 43500000 out of 50000000 steps (87%).
22:33:57:WU01:FS00:0x16:Completed 40000000 out of 50000000 steps (80%).
22:34:07:WU03:FS01:0x16:Completed 44000000 out of 50000000 steps (88%).
