Page 1 of 1
Extracting Stats
Posted: Thu Nov 20, 2008 7:32 pm
by batman113
Does anyone know if it is possible to get the os client data and put it into a database automatically?
Also any idea how these third party stats websites do it? eg Extreme Overclockers
Thanks
Re: Extracting Stats
Posted: Thu Nov 20, 2008 7:34 pm
by MtM
batman113 wrote:Does anyone know if it is possible to get the os client data and put it into a database automatically?
Also any idea how these third party stats websites do it? eg Extreme Overclockers
Thanks
Os client data is not offerd by external feeds like the donor list and teams lists are. You're not allowed to crawl their page afaik.
EOC ect use the external feeds.
Re: Extracting Stats
Posted: Thu Nov 20, 2008 8:09 pm
by batman113
How do you get these external feeds?
Re: Extracting Stats
Posted: Fri Nov 21, 2008 12:05 am
by MtM
Re: Extracting Stats
Posted: Fri Nov 21, 2008 6:17 pm
by batman113
I know where the stats are but how can i get them to be automatically added to my own database?
Re: Extracting Stats
Posted: Fri Nov 21, 2008 9:33 pm
by batman113
I am trying to make a similar database to what one of the external stats sites has however i do not know how. I am relatively familiar with C# and know some VB6 and PHP. Is There any way of getting the stats automatically added to the database?
Thanks!
Re: Extracting Stats
Posted: Fri Nov 21, 2008 10:46 pm
by MtM
batman113 wrote:I am trying to make a similar database to what one of the external stats sites has however i do not know how. I am relatively familiar with C# and know some VB6 and PHP. Is There any way of getting the stats automatically added to the database?
Thanks!
You can try something like this, it's in vb.net so you have to use other declares, and I have an dataset with tableadapters which you have to declare as well. Also I only am storing my own team members myself that's easilly changed. My problem was dat the code I was using created tables on the fly named by the update time, this worked perfectly for me but saving them to the dataset did not save them to the sql database ( copy if never. db accepts adds. can alter any exising table, clear them, remove them, but not add them for some strange reason. )
Keep in mind btw you can only call the stats feed 10 times a day, the donor list is 25mb +- so if you keep running and running it you're putting needless strain on them. And this should work in one try, it's not as quick as downloading the file and then parsing it as well, but I want to host this on my isp soon and don't have that much space there
You can edit the code abit, it checks the team table with the known last updates and clears it before refilling current standing, that you can use to create a new table for storing the entire update ( which I'm doing, I got the past weeks for my team but I stored them in text files not a database. If I figure out how to keep my tables from dissapearing I will post the updated code here sometime if you want
But I'm not a pro coder btw, and there are probably allot off inefficient code in here
Code: Select all
'some imports you need
Imports system.io
imports system.net
imports system.data.sqlclient
Public Const UrlDonors As String = "http://fah-web.stanford.edu/daily_user_summary.txt"
Public Function ParsePGstring(ByVal strDT As String) As DateTime
Dim tmp1 As String = Mid(strDT, strDT.LastIndexOf(" ") + 1).Trim
strDT = strDT.Replace(" PST " & tmp1, "")
Dim tmp2 As String = Mid(strDT, strDT.LastIndexOf(" ") + 1).Trim
strDT = strDT.Replace(tmp2, "").TrimEnd
strDT = strDT & " " & tmp1 & " " & tmp2
Dim Dtu As DateTime
Try
Dtu = DateTime.Parse(strDT)
Catch ex As Exception
Debug.Print("Parsing to datetime object failed!")
Debug.Print(Err.Description)
Return Nothing
End Try
Return Dtu
End Function
Public ReadOnly Property LastUpdateOnServer() As DateTime
Get
Try
'You could change this to using the responce object last edited property but I was already using the time in the file itself. This does mean my app makes a connection to their server a couple of times an hour ( depends on how I set the timer.
Dim request As HttpWebRequest = WebRequest.Create(UrlDonors)
Dim response As HttpWebResponse = request.GetResponse()
Dim reader As StreamReader = New StreamReader(response.GetResponseStream())
Dim tmpString As String = reader.ReadLine
reader.Close()
response.Close()
Return ConvertToLocalDT(ParsePGstring(tmpString))
Catch ex As Exception
Return Nothing
End Try
End Get
End Property
Public ReadOnly Property LastProcesedUpdate() As DateTime
Get
Try
If XtremeTeam.Count > 0 Then
Dim tRow As DataRow = XtremeTeam.Item(0)
Return tRow("uDT")
Else
Return DateTime.MinValue
End If
Catch ex As Exception
Return DateTime.MinValue
End Try
End Get
End Property
Public Function ConvertToLocalDT(ByVal dTU) As DateTime
Try
Dim nZone As TimeZoneInfo = TimeZoneInfo.FindSystemTimeZoneById("Pacific Standard Time")
Try
Dtu = TimeZoneInfo.ConvertTime(Dtu, nZone, TimeZoneInfo.Local)
Catch e As TimeZoneNotFoundException
Return Nothing
Catch e As InvalidTimeZoneException
Return Nothing
End Try
Return dTU
Catch ex As Exception
Return Nothing
End Try
End Function
Public Function ImportLastDonors(ByVal lblStatus As System.Windows.Forms.ToolStripStatusLabel, Optional ByVal DgView As DataGridView = Nothing) As Boolean
Try
_IsImporting = True
Try
DgView.SuspendLayout()
Catch ex As Exception
End Try
Dim doADD As Boolean = False
Dim dtU As DateTime
Dim request As HttpWebRequest = WebRequest.Create(UrlDonors)
Using response As HttpWebResponse = request.GetResponse()
Using reader As StreamReader = New StreamReader(response.GetResponseStream())
Using mRead As New Microsoft.VisualBasic.FileIO.TextFieldParser(reader)
mRead.TextFieldType = FileIO.FieldType.Delimited
mRead.SetDelimiters(vbTab)
Dim XlineNum As Integer = 1
While Not mRead.EndOfData
Try
Dim currentRow As String()
currentRow = mRead.ReadFields()
If XlineNum = 1 Then
Dim utime As String = currentRow.ElementAt(0)
dtU = ConvertToLocalDT(ParsePGstring(utime))
If XtremeTeam.Count > 0 Then
Dim tRow As DataRow = XtremeTeam.Item(0)
If tRow("uDT") = LastUpdateOnServer Then
Return True
Else
XtremeTeam.Rows.Clear()
XtremeTeam.AcceptChanges()
XtremeTeamTableAdapter.Update(XtremeTeam)
End If
End If
ElseIf XlineNum >= 3 Then
Dim xStep As Integer : xStep = 1
Dim nDonor As New sDonors
For Each currentField In currentRow
If xStep = 1 Then
'name
nDonor.Name = currentField
ElseIf xStep = 2 Then
'New Total
nDonor.NewTotal = currentField
ElseIf xStep = 3 Then
'Las
nDonor.WUs = currentField
ElseIf xStep = 4 Then
'Team
nDonor.Team = currentField
End If
xStep += 1
Next
If nDonor.Team = "36362" Then
Dim nRow As DataRow = XtremeTeam.NewRow
nRow("Member") = Mid(nDonor.Name, 1, 25)
nRow("TotalPoints") = CDbl(nDonor.NewTotal)
nRow("WUs") = CInt(nDonor.WUs)
nRow("Udt") = dtU
Try
XtremeTeam.AddXtremeTeamRow(nRow)
lblStatus.Text = "Added " & nDonor.Name
Application.DoEvents()
Catch ex As Exception
'add errorhandling
End Try
Else
lblStatus.Text = "Skipped " & nDonor.Name
Application.DoEvents()
End If
End If
If XlineNum <= 2 Then XlineNum += 1
Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
Debug.Print(ex.Message)
End Try
Application.DoEvents()
End While
End Using
End Using
End Using
XtremeTeamTableAdapter.Update(XtremeTeam)
_IsImporting = False
Try
DgView.ResumeLayout()
DgView.Refresh()
Catch ex As Exception
End Try
Return True
Catch ex As Exception
_IsImporting = False
Return False
End Try
End Function
Re: Extracting Stats
Posted: Fri Nov 21, 2008 11:02 pm
by batman113
Thanks MtM!
I will give this a go but vb is probably my weakest language so ill probably mess it up. I am trying to find a way using C# but im struggling atm. Will post updates!
Re: Extracting Stats
Posted: Fri Nov 21, 2008 11:14 pm
by MtM
hehe maybe we could help eachother, I'm using webchart which is documented mostly in c#, and had some troubles converting that to vb.net. When I have something I could need someone's help with who knows c# better then I do I'll ask you
Re: Extracting Stats
Posted: Sun Nov 23, 2008 6:22 pm
by batman113
Thanks for the code MtM but i cant get it to work! What proggram did you write it in? Im using Visual Studio 2008 and i keep getting tis error:
Code: Select all
Error 1 Statement is not valid in a namespace.
Any ideas?
Re: Extracting Stats
Posted: Sun Nov 23, 2008 8:21 pm
by MtM
Put the code in a module
I have to admit I'm looking at it now feeling ashamed I posted it as there are so many things which aren't really efficient/good coding.
All the for each statements with the currentrow can replaced with simpler value = currentrow(x) statements ect, but I thought you just needed to look at something to get you started.
Re: Extracting Stats
Posted: Tue Nov 25, 2008 9:56 pm
by batman113
I have so far got it to read the file and i am beggining to seperate it into 4 arrays (team number, team name, points and wu's) but i am struggling because i seperated them with the .Split sunction but it seperates everthing not just tabbed spaces. I will update when i have worked out how to do it!