Extracting Stats

[batman113]

Does anyone know if it is possible to get the os client data and put it into a database automatically?
Also any idea how these third party stats websites do it? eg Extreme Overclockers
Thanks

[MtM]

Does anyone know if it is possible to get the os client data and put it into a database automatically?
Also any idea how these third party stats websites do it? eg Extreme Overclockers
Thanks

Os client data is not offerd by external feeds like the donor list and teams lists are. You're not allowed to crawl their page afaik.

EOC ect use the external feeds.

[batman113]

How do you get these external feeds?

[MtM]

http://folding.stanford.edu/English/Stats

Donor list and Team list

[batman113]

I know where the stats are but how can i get them to be automatically added to my own database?

[batman113]

I am trying to make a similar database to what one of the external stats sites has however i do not know how. I am relatively familiar with C# and know some VB6 and PHP. Is There any way of getting the stats automatically added to the database?
Thanks!

[MtM]

I am trying to make a similar database to what one of the external stats sites has however i do not know how. I am relatively familiar with C# and know some VB6 and PHP. Is There any way of getting the stats automatically added to the database?
Thanks!

You can try something like this, it's in vb.net so you have to use other declares, and I have an dataset with tableadapters which you have to declare as well. Also I only am storing my own team members myself that's easilly changed. My problem was dat the code I was using created tables on the fly named by the update time, this worked perfectly for me but saving them to the dataset did not save them to the sql database ( copy if never. db accepts adds. can alter any exising table, clear them, remove them, but not add them for some strange reason. )

Keep in mind btw you can only call the stats feed 10 times a day, the donor list is 25mb +- so if you keep running and running it you're putting needless strain on them. And this should work in one try, it's not as quick as downloading the file and then parsing it as well, but I want to host this on my isp soon and don't have that much space there

You can edit the code abit, it checks the team table with the known last updates and clears it before refilling current standing, that you can use to create a new table for storing the entire update ( which I'm doing, I got the past weeks for my team but I stored them in text files not a database. If I figure out how to keep my tables from dissapearing I will post the updated code here sometime if you want

But I'm not a pro coder btw, and there are probably allot off inefficient code in here

Code: Select all

'some imports you need
Imports system.io
imports system.net
imports system.data.sqlclient

    Public Const UrlDonors As String = "http://fah-web.stanford.edu/daily_user_summary.txt"
    Public Function ParsePGstring(ByVal strDT As String) As DateTime
        Dim tmp1 As String = Mid(strDT, strDT.LastIndexOf(" ") + 1).Trim
        strDT = strDT.Replace(" PST " & tmp1, "")
        Dim tmp2 As String = Mid(strDT, strDT.LastIndexOf(" ") + 1).Trim
        strDT = strDT.Replace(tmp2, "").TrimEnd
        strDT = strDT & " " & tmp1 & " " & tmp2
        Dim Dtu As DateTime
        Try
            Dtu = DateTime.Parse(strDT)
        Catch ex As Exception
            Debug.Print("Parsing to datetime object failed!")
            Debug.Print(Err.Description)
            Return Nothing
        End Try
        Return Dtu
    End Function
    Public ReadOnly Property LastUpdateOnServer() As DateTime
        Get
            Try
'You could change this to using the responce object last edited property but I was already using the time in the file itself. This does mean my app makes a connection to their server a couple of times an hour ( depends on how I set the timer.
                Dim request As HttpWebRequest = WebRequest.Create(UrlDonors)
                Dim response As HttpWebResponse = request.GetResponse()
                Dim reader As StreamReader = New StreamReader(response.GetResponseStream())
                Dim tmpString As String = reader.ReadLine
                reader.Close()
                response.Close()
                Return ConvertToLocalDT(ParsePGstring(tmpString))
            Catch ex As Exception
                Return Nothing
            End Try
        End Get
    End Property
    Public ReadOnly Property LastProcesedUpdate() As DateTime
        Get
            Try
                If XtremeTeam.Count > 0 Then
                    Dim tRow As DataRow = XtremeTeam.Item(0)
                    Return tRow("uDT")
                Else
                    Return DateTime.MinValue
                End If
            Catch ex As Exception
                Return DateTime.MinValue
            End Try
        End Get
    End Property
    Public Function ConvertToLocalDT(ByVal dTU) As DateTime
        Try
            Dim nZone As TimeZoneInfo = TimeZoneInfo.FindSystemTimeZoneById("Pacific Standard Time")
            Try
                Dtu = TimeZoneInfo.ConvertTime(Dtu, nZone, TimeZoneInfo.Local)
            Catch e As TimeZoneNotFoundException
                Return Nothing
            Catch e As InvalidTimeZoneException
                Return Nothing
            End Try
            Return dTU
        Catch ex As Exception
            Return Nothing
        End Try
    End Function
    Public Function ImportLastDonors(ByVal lblStatus As System.Windows.Forms.ToolStripStatusLabel, Optional ByVal DgView As DataGridView = Nothing) As Boolean
        Try
            _IsImporting = True
            Try
                DgView.SuspendLayout()
            Catch ex As Exception

            End Try
            Dim doADD As Boolean = False
            Dim dtU As DateTime
            Dim request As HttpWebRequest = WebRequest.Create(UrlDonors)
            Using response As HttpWebResponse = request.GetResponse()
                Using reader As StreamReader = New StreamReader(response.GetResponseStream())
                    Using mRead As New Microsoft.VisualBasic.FileIO.TextFieldParser(reader)
                        mRead.TextFieldType = FileIO.FieldType.Delimited
                        mRead.SetDelimiters(vbTab)
                        Dim XlineNum As Integer = 1
                        While Not mRead.EndOfData
                            Try
                                Dim currentRow As String()
                                currentRow = mRead.ReadFields()
                                If XlineNum = 1 Then
                                    Dim utime As String = currentRow.ElementAt(0)
                                    dtU = ConvertToLocalDT(ParsePGstring(utime))
                                    If XtremeTeam.Count > 0 Then
                                        Dim tRow As DataRow = XtremeTeam.Item(0)
                                        If tRow("uDT") = LastUpdateOnServer Then
                                            Return True
                                        Else
                                            XtremeTeam.Rows.Clear()
                                            XtremeTeam.AcceptChanges()
                                            XtremeTeamTableAdapter.Update(XtremeTeam)
                                        End If
                                    End If
                                ElseIf XlineNum >= 3 Then
                                    Dim xStep As Integer : xStep = 1
                                    Dim nDonor As New sDonors
                                    For Each currentField In currentRow
                                        If xStep = 1 Then
                                            'name
                                            nDonor.Name = currentField
                                        ElseIf xStep = 2 Then
                                            'New Total
                                            nDonor.NewTotal = currentField
                                        ElseIf xStep = 3 Then
                                            'Las
                                            nDonor.WUs = currentField
                                        ElseIf xStep = 4 Then
                                            'Team
                                            nDonor.Team = currentField
                                        End If
                                        xStep += 1
                                    Next
                                    If nDonor.Team = "36362" Then
                                        Dim nRow As DataRow = XtremeTeam.NewRow
                                        nRow("Member") = Mid(nDonor.Name, 1, 25)
                                        nRow("TotalPoints") = CDbl(nDonor.NewTotal)
                                        nRow("WUs") = CInt(nDonor.WUs)
                                        nRow("Udt") = dtU
                                        Try
                                            XtremeTeam.AddXtremeTeamRow(nRow)
                                            lblStatus.Text = "Added " & nDonor.Name
                                            Application.DoEvents()
                                        Catch ex As Exception
                                            'add errorhandling
                                        End Try
                                    Else
                                        lblStatus.Text = "Skipped " & nDonor.Name
                                        Application.DoEvents()
                                    End If
                                End If
                                If XlineNum <= 2 Then XlineNum += 1
                            Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
                                Debug.Print(ex.Message)
                            End Try
                            Application.DoEvents()
                        End While
                    End Using
                End Using
            End Using
            XtremeTeamTableAdapter.Update(XtremeTeam)
            _IsImporting = False
            Try
                DgView.ResumeLayout()
                DgView.Refresh()
            Catch ex As Exception

            End Try
            Return True
        Catch ex As Exception
            _IsImporting = False
            Return False
        End Try
    End Function

[batman113]

Thanks MtM!
I will give this a go but vb is probably my weakest language so ill probably mess it up. I am trying to find a way using C# but im struggling atm. Will post updates!

[MtM]

hehe maybe we could help eachother, I'm using webchart which is documented mostly in c#, and had some troubles converting that to vb.net. When I have something I could need someone's help with who knows c# better then I do I'll ask you

[batman113]

Thanks for the code MtM but i cant get it to work! What proggram did you write it in? Im using Visual Studio 2008 and i keep getting tis error:

Code: Select all

Error	1	Statement is not valid in a namespace.	

Any ideas?

[MtM]

Put the code in a module

I have to admit I'm looking at it now feeling ashamed I posted it as there are so many things which aren't really efficient/good coding.

All the for each statements with the currentrow can replaced with simpler value = currentrow(x) statements ect, but I thought you just needed to look at something to get you started.

[batman113]

I have so far got it to read the file and i am beggining to seperate it into 4 arrays (team number, team name, points and wu's) but i am struggling because i seperated them with the .Split sunction but it seperates everthing not just tabbed spaces. I will update when i have worked out how to do it!