FAHWatch7

[darkbasic]

Yes, thank you for your work, we really appreciate it.

[GreyWhiskers]

Yeah I didn't fix that before uploading, or uploaded from the wrong directory, sorry. I'll upload the fixed source and new installer later today. My apologies.

None needed. Great work.

[jimerickson]

yes MtM great work on your part. keep it up.

[MtM]

Thanks for the kind words, I appreciate it. But I still feel bad since it's not going as quick or as efficient as I want it to

Took a day longer then I wanted, went back and forth with replacing all the old parser code ( since the installer does not remove the database, it should pick up where it stopped after the update ) but that didn't work either as some of the logs which were not marked as 'finished' had the old layout. But I think it's good now, haven't found a problem testing here that is. Will be the first time you all can check the new parser code though ( last revision had lots of changes already, many of those should make it quicker ).

One note, the psummary parser can appear to 'hang' when accessing an url, it's because the webrequest code runs on the same thread as the user interface ( which I should really change ) so give it a moment and it should work out in the end.

Some of the options don't work ( auto reparse, start with windows, start from tray ( I think )). Other do work, columns can be resized and moved but it will not be stored. The large graph window when selected will not update when selecting a work unit from the history browser, I'll fix that as well.

Installer -> http://code.google.com/p/cftunity/downl ... p&can=2&q=

Live monitoring is coming as well, but not today ( ow the form is empty.. don't worry this isn't the test code for live monitoring ).

Edit:

I have a question, with the current code, a frame ( let's take 1% to 2% for example ) lasts from the first time 'completed 1%' ( since you might stop the core, and checkpoints are not the same as 'completed x' messages ) to the first time 'completed 2%' is found. This is the reason that if you pause a slot, the frame graph will have those 'mountains'. If you do this less then 10 times or so for a work unit, it shouldn't affect AVG time span that much, in fact when I generate tpf by disregarding frames which are more then 120% of the average and less then 80% of the average I don't get that much of a difference and in some cases it will even cause the AVG to be adjusted longer instead of shorter as there might be more frames just shorter as the minimum cut off then is offset by cutting off the big mountain caused by a paused slot.

In short, I would like to think it's fine as is, and infact the graph is reflecting real time usage right now and should stay as is. But, it is possible to cut out the pause duration of those peaks and normalize them, and replace the real time effective frame time with an actual frame time. It will require some additions to the log parser code, and while I think it shouldn't be that hard I might run into complications who knows.

What is your opinion, should I keep the effective frame times or go for the actual frame time, and if the last should that be only in the frame graph or also in hardware statistics ( which is where I think it has merit opposed to the frame graph so maybe it should just go there and not in the graph )??

Edit2: changelog didn't include the fix for the double log entries as provided earlier in the thread, but it should be fixed as well. I also didn't include the changes to hwinfo regarding no more relying on calcons.exe, cal detection should no longer cause a hung state.

[Grandpa_01]

The latest version appears to work good on my Laptop I have not installed it on anything else yet. I did notice a problem with the project info, when I quarry the project info from Stanford for some reason it does not download the 8XXX projects it goes from project 7905 to project 10009 for some reason it skips the 8XXX series. I had it use all 3 psumary pages and it had the same results with all of them. I checked and the 8XXX projects are listed on all of the psumary pages. I do not know if any other lines are getting missed or not I did not check.

As far as the first question I think how you currently have it is good. As far as frame times go I think most people seem to prefer actual frame time, it does not matter to me.

[MtM]

The project browser did not list those projects, confirmed. But, my work units where credited so the project was known. I updated the summary using the browser from the default psummary linked at the top of the forums, and the projects then did show up in the browser. The browser did however not sort them correctly and they were added to the bottom of the list.



I will make a ticket to sort this so I don't forget, thanks for the report!

Edit: ticket created.

*fixed url*

[jimerickson]

effective frame time is fine with me.

[bruce]

I have a question, with the current code, a frame ( let's take 1% to 2% for example ) lasts from the first time 'completed 1%' ( since you might stop the core, and checkpoints are not the same as 'completed x' messages ) to the first time 'completed 2%' is found. This is the reason that if you pause a slot, the frame graph will have those 'mountains'.

If the core is restarted from a checkpoint that occurred at the end of a frame, the first frame will be "complete" and there will be no mountain. If the last checkpoint before the shutdown was NOT at the end of a frame, the first frame will be short and you will see something like this:

Code: Select all

01:35:32:WU00:FS01:0xa4:Completed 152580 out of 250000 steps  (61%)
01:37:06:WU00:FS01:0xa4:Completed 155000 out of 250000 steps  (62%)
01:38:42:WU00:FS01:0xa4:Completed 157500 out of 250000 steps  (63%)

Note that if you divide 152580 by 250000 you do not get 61% but rather 61.032%. If you divide the step count on the other lines, you do get 62.0% and 63.0%
My recommendation would be to check the first message to see if it represents the exact beginning of a frame and simply discard the entry at 01:35:32 for the incomplete frame that starts from step 152580 rather than at step 152500.

[MtM]

For my gpu clients, steps are no displayed in the log. The remote interface does show them through 'simulation-info' afaik, but a log parser has no method if determining the actual step completion. If the gpu cores will display steps in the logs that would be nice ( uniformity among all cores ).

But, it wouldn't prevent a 'mountain' as I described it as I'm already discarding that line, not because I use the steps but because the work unit's current percentage is the same as being printed at that line. The mountain results from the increase in tpf because of this ->

Code: Select all

00:53:59:WU00:FS01:0xa4:Completed 152500 out of 250000 steps  (61%)
00:54:12:FS01:Paused
01:35:29:FS01:Unpaused
<snip I don't know from the back of my head>
01:35:32:WU00:FS01:0xa4:Completed 152580 out of 250000 steps  (61%)
01:37:06:WU00:FS01:0xa4:Completed 155000 out of 250000 steps  (62%)
01:38:42:WU00:FS01:0xa4:Completed 157500 out of 250000 steps  (63%)

I can prevent the mountain by substracting the time between 'Paused' and the second 'Completed 61%'.

The question was also meant to inquiry people's thoughts about using effective versus actual frame times. I will probably end up adding code which can determine the actual running time or a frame using slot status only for hardware performance statistics it now uses the same frame time as displayed in the tpf graph and that's incorrect ( ticket ).

[bruce]

A FahCore is mainly a wrapper for code developed elsewhere. Stanford cannot produce uniformity among all cores. That would be up to the developers of the various analysis code-bases.

[MtM]

A FahCore is mainly a wrapper for code developed elsewhere. Stanford cannot produce uniformity among all cores. That would be up to the developers of the various analysis code-bases.

While this is true, it only applies if the cores write directly to the log. If FAHClient would be the only process writing to the logs it could format any message there any way it likes, and as I said afaik the remote interface does provide simulation info including steps completed even for gpu clients ( I could very well be wrong here which is why I say afaik ).

It shouldn't be hard to change to code in the cores which now writes to the log file to write to process.standardout and have that tied to FAHClient, or I don't think it should but I might be wrong there idk.

That would make 3rd party very happy as different core's wouldn't influence the log format breaking existing parsing code, and I think donors in general who do look at their logs would feel it's easier to read if everything in it followed the same formatting rules.

I would like to request this and if possible I hope it will be made a requested enhancement even if get's a trivial status.

[bruce]

The the design goals for V7, there's a statement that log files may change without notice and that 3rd party designers shall NOT parse the logs. On that basis, requesting standardization of the logs so that they can be parsed will be automatically denied. Log file formats have changed several times during the development from 7.0.0 to 7.1.43 and still may change at any time without Stanford having to request permission from the 3rd party developers or wait while they update their parsing code.

If the developer of a 3rd party tool which parses the log is no longer available to update it and a change to the log files breaks it, whose fault is that? Stanford is committed to maintaining the socket interface and 3rd party tools developed to use that interface should keep working without additional maintenance -- at least after the bugs are fixed.

[MtM]

I made some progress on some better psummary handling, it will no longer run on the ui thread. About the missing projects, the parser code did not miss projects I'm sure of that. I did notice that Calxalot's summary does not list the 8xxx range. I placed a copy in txt form on my project page here. Since Calxalot is the default summary, it's likely that after updating with the stanford copies you missed the projects because of the sorting problem ( see below ).

The summary browser did not sort the projects correctly though, mostly because the code was 4 years old for the most part, and did not offer the functionality it needed to easily sort projects. It now does, and it's much simpler as well so that makes me happy

The eoc update handling is running nicely as well, I think the fade is slow but that can be tweaked easily. What I have no tried is running with custom formatting option for the signature image.

I've also dumped some code which supported upgrading from older versions to newer one's, as it wasn't needed anymore and without it it's much easier.

Some might have noticed that debugout is enabled by default now even when starting without the flag, this will stay this way until the project leaves beta. It helps me allot with catching problems, and it doesn't impact performance much since I changed the log handling methods.

I do know there are some duplicate entries in the log with debug output, those will be removed ( already done a few of those ).

Link for build here. Replace the executable if upgrading from build44.

[MtM]

The the design goals for V7, there's a statement that log files may change without notice and that 3rd party designers shall NOT parse the logs. On that basis, requesting standardization of the logs so that they can be parsed will be automatically denied. Log file formats have changed several times during the development from 7.0.0 to 7.1.43 and still may change at any time without Stanford having to request permission from the 3rd party developers or wait while they update their parsing code.

If the developer of a 3rd party tool which parses the log is no longer available to update it and a change to the log files breaks it, whose fault is that? Stanford is committed to maintaining the socket interface and 3rd party tools developed to use that interface should keep working without additional maintenance -- at least after the bugs are fixed.

1. I know about the design goals pointing to the socket interface, but that will only work if that interface works. Currently, it doesn't. Harlam has also said he is using log parsing to obtain tpf, credit, ppd, eta estimates just as pre V7, and there is no incentive to stop doing that until the interface works. There is merit to saying it's a lower priority then getting FAHClient and FAHControl itself up to standards, I won't say otherwise.

2. Using the socket interface will give simulation info giving steps vs steps completed, and datetime stamps for downloaded. It will also give an eta and a tpf field. But, how are those calculated? Just as my question above, tpf can be looked at in different ways. And if FAHClient gives TPF in one manner, does that mean it's not important to be able to obtain it another way if so desired?

Third and most important ( ) look at the changes made, are they already working towards a singular format? I think so. Off course some core's output different information and that information will have formats which are linked to those cores ( think the startup info from the gpu core's listing board type for instance ). But, I didn't request that being standardized, I only asked that gpu clients also logged steps completed vs steps total if possible.

Short answer: I don't want a full standardization as I don't think that's possible, but I would like as much standardization as feasible when looking at both time needed to make it work versus return value for others. And like I said, I think more people not only those parsing the logs but also those just looking at them with notepad or FAHControl or through the remote interface, would enjoy the highest possible grade of standardization where possible.

But I'll leave it at this, if you say it won't be considered then I'm out of luck I guess

[Grandpa_01]

On initial install and parse the 8xxx WU's were not listed but I did and update using an alternative psumary and the were listed and in the correct position. ( sorry I did not use the default psumary to update I will when I install it on another rig). May I congratulate you on the improvements you have made over the last few days it fells much quicker in use and behaves very nicely. I also figured out how to use the PPD calculator and it appears to work quite well thanks.