Welcome Guest ( Log In | Register )



 
Reply to this topicStart new topic
> Suggestions Needed For Latest Ycc Bot Maker Breakdown
tansqrx
post May 24 2008, 09:00 AM
Post #1


Super Member
Group Icon

Group: [HOSTED]
Posts: 522
Joined: 25-April 05
Member No.: 4,374



This is not specifically geared toward Astahost because it is a plea for help for one of the programs that I am working on. I know there are a lot of smart people here so I decided to give it a shot.

About a week ago I received word that version 1.2 of YCC Bot Maker had stopped working. This was not a huge surprise as Yahoo! continuously changes its registration process and YCC Bot Maker is very dependant on the data held in the registration page. I hoped to have a fix put out fairly quickly but this time I am stuck and have yet to find a solution.

From what I can gather, Yahoo! has not changed any of the pages or any of the processes used during registration. One day it was working and the next it was broken. This leads to be believe that Yahoo! is doing some sort of new browser profiling or timing analysis. YCC Bot Maker uses the HttpWebRequest object built into .NET. By adding accept, referer, user agent, and others I can fairly approximate the look of IE7.

I start Fiddler (http://www.fiddlertool.com/fiddler/) and make a side by side comparison of a session with IE7 and then YCC Bot Maker. The requests are almost identical except a few header locations are swapped.

Here is the last request during the registration process using IE7 on the India server. The response shows a congratulations page for creating a new account.

CODE
POST /registration;_ylt=Amw.PhwB5E.stLUBnuRzxoSZ2PAI HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-silverlight, */*
Referer: https://edit.india.yahoo.com/registration?....omo=&.last=
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
UA-CPU: x86
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)
Host: edit.india.yahoo.com
Content-Length: 807
Connection: Keep-Alive
Pragma: no-cache
Cookie: B=4dnko1h43ek4p&b=3&s=97

u=5b178pd43ek5n&dracs=&t=x2F6Q4.MzeGZRlr35zVb1DkywcQjtqkCMrdWMjcL5jX.V6Y1do4hqQ8aOF2HAHDw3jr30lHHDA
NXTIXGXaohokYjurJtWrgBEDE3ucfh2EOavo0VGwzarjnQ4VJVw_kLYlX4.XVpbLNai8H1BhVbwb8.iO
vNBnMSdx1yaaDHdSh6zml1DMIEHZ143m0LGzz8Rxn6nnHe8JcWdRh1en0AJC.s9eYumSrm1taEZAoQ_S
oCEt00C8MtUQ--%7EB&done=http%3A%2F%2Fmail.yahoo.com&last=&partner=yahoo_default&intl=us&src=ym&.scrumb=&jsenabled=0&preferredcontent=us&firstname=sadfdas&secondname=ewerq&gender=m&mm=2&dd=1&yyyy=1981&country=us&postalcode=58443&yahooid=dfk58443lkalsdfk&domain=yahoo.com&password=bobobob&passwordconfirm=bobobob&altemail=&secquestion=Who+was+your+childhood+hero%3F&secquestionanswer=dsfdsfadfs&cword=fw8nyvl&cdata=ftqaZeJZFelVDTGYXguGQO75qbdYepa6qbxfs5c2jew_iXlBPSPrIVt8DMAhTLjTWv7KGD
DVEErJLca0hg--&showc=1&tos_agreed=y&IAgreeBtn=Create+My+Account


Here is the same output from YCC Bot Maker on the India server. This time I receive a “please try this code instead” and a failed attempt. Additionally, if I make another request with the previous failed attempt using the new CAPTCHA, I get a successful account creation page.

CODE
POST /registration;_ylt=AkbfFrpAnjNO5TMJr36fEHeZ2PAI HTTP/1.0
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, application/x-silverlight, */*
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)
Content-Type: application/x-www-form-urlencoded
Referer: https://edit.india.yahoo.com/registration?....omo=&.last=
Host: edit.india.yahoo.com
Cookie: B=11uird143f8gg&b=3&s=2v
Content-Length: 804
Connection: Keep-Alive

u=f1p49q543f8gj&dracs=&t=c.kVhRIoP.HFU6QfWH.b0CftVs9X73rAw.U_hBx.y9JCxffF2ctaHKrOcaKQnOBNm7iClR402h
HW8RM8BozEdxaxki05iRoJ.Dag.P3_jRAw9QILVBTONHiCwpdt50_E0q81D9Nd16qrsR7tpnSvMJ6zZh
JP0hRIBkNgFzXDuR8l4A9ndFkQtuMvJnrtFyGdngja5d16AAi0Tq5VeYqKBDTWuTm6iIlQurTLJyvYQ0
EPYbheZ3_esw--%7EB&done=http%3A%2F%2Fmail.yahoo.com&last=&partner=yahoo_default&intl=us&src=ym&.scrumb=&jsenabled=0&preferredcontent=us&firstname=MQYhLVgujy&secondname=BrXTSVyGEk&gender=m&mm=5&dd=12&yyyy=1947&country=us&postalcode=40076&yahooid=igbl49U&domain=yahoo.com&password=Nb6s5cR&passwordconfirm=Nb6s5cR&altemail=&secquestion=Where+did+you+meet+your+spouse%3F&secquestionanswer=7IiKII&cword=8teyj4e&cdata=5wtKjOJZFemg3x.fc7zjW2fNv.BxMnIccglRTwNkIaMYMDzucE_SZ7cB5ZHefjZetpKvKL
tJpHLekJpb4g--&showc=1&tos_agreed=y&IAgreeBtn=Create+My+Account


You can see that the requests are exactly the same with the exception of user data and Yahoo! specific variables. I have no way of explaining this because from everything I know it should. I need your help getting the next version of YCC Bot Maker working. If you have suggestions or know of some new technology Yahoo! is using please let me know.
Go to the top of the page
 
+Quote Post
faulty.lee
post May 29 2008, 01:52 AM
Post #2


Premium Member
Group Icon

Group: [HOSTED]
Posts: 479
Joined: 5-November 06
Member No.: 17,016



I would really like to help. I did a same tool to login into astahost and fetch the remaining point, so can I can keep an eye on it without loading the astahost main page. Anyway, do you mind to show me the section of the coding which is doing the login? Or better still, strip the login coding out and put inside a mini project, so we can just try out that portion.
Go to the top of the page
 
+Quote Post
tansqrx
post Jun 1 2008, 06:50 AM
Post #3


Super Member
Group Icon

Group: [HOSTED]
Posts: 522
Joined: 25-April 05
Member No.: 4,374



Thank you for your offer. Let me do some clarification. YCC Bot Maker is a program that signs up for Yahoo! accounts the same way you would if you used Internet Explorer or Firefox. One difference is that it is a VB.NET form that puts all the signup information in one spot and is much quicker than using a web browser. Another difference for this particular question is that YCC Bot Maker uses a series of HttpWebRequest objects and not the Webbrowser object (i.e. the Internet Explorer ActiveX control) to make the requests. I have all the code needed to create the HTTP requests and get valid responses back but I suspect that Yahoo! has added code on their servers to profile the type of “browser” used ask for accounts. More specifically I have changed the UserAgent, Accept, AcceptLanguage, and UA-CPU properties of the HttpWebRequest to mimic Internet Explorer. Even with all of this the Yahoo! server some how knows that my program is not a regular web browser and rejects the request (more on this later).

Up until about a month ago YCC Bot Maker was working fine. Then over night I started getting a prompt to reenter the CAPTCHA. After a through review of the Yahoo! signup page I found that nothing had changed.

There is a series of three requests, get the mail login page which is used to grab the Yahoo! cookie, get the registration page where you actually enter the user data, and finally submit a HTML form POST with the user data and see if it was successful. The first two requests are still fine but when I send the POST, the account is denied and a prompt to enter a new CAPTCHA is shown. After playing around with the sequence I found that I can get a valid account if I reenter the second CAPTCHA and submit once again. An account is only granted if two successful CAPTCHAs are entered. Once again if I use Internet Explorer or FireFox, I do not have to enter the second CAPTCHA. Because I can finally get an account, but only after more verification, I have come to the conclusion that the Yahoo! server knows this is not actually Internet Explorer and adds extra validation.

My question is how does this technology work and how can I get around it.

NOTE:

Here is the code that I use to generate my HTTP requests. It is not really needed but I did put a lot of hard work into it and it is a common question asked. The full library is called HTTPHelper and can be downloaded at http://ycoderscookbook.com/code.htm. The original intent was to make up for many of the short comings in version 1.1 of the .NET Framework but it is still very valuable even with the latest version.

Here is an excerpt of a HTTP GET request. HTTPDataIn and HTTPDataOut are structures defined within the class and hold needed variables.

CODE
Public Shared Function HTTPGET(ByVal HTTPDataIn As HTTPDataIn) As HTTPDataOut
Dim HTTPDataOut As New HTTPDataOut
Dim httpRequest As HttpWebRequest
Dim httpResponse As HttpWebResponse = Nothing
Dim responseStream As Stream = Nothing
Dim responseEncoding As Encoding
Dim responseStreamReader As StreamReader = Nothing
'Dim cookieContainerTempOut As New CookieCollection
Dim iIndex As Integer

Try
HTTPDataOut.bUnsucessful = False
HTTPDataOut.strErrorCode = ""
httpRequest = CType(WebRequest.Create(HTTPDataIn.strServer), HttpWebRequest)

'proxy
If HTTPDataIn.bEnableProxy = True Then
Dim wpRequest As New WebProxy
Dim uriPrxy As New Uri(HTTPDataIn.piProxy.strServer + ":" + HTTPDataIn.piProxy.strPort)
wpRequest.Address = uriPrxy
If HTTPDataIn.piProxy.strUsername IsNot Nothing Then
wpRequest.Credentials = New NetworkCredential(HTTPDataIn.piProxy.strUsername, HTTPDataIn.piProxy.strPassword)
End If
httpRequest.Proxy = wpRequest
End If

'set the timeout
If HTTPDataIn.bEnforceTimeout = True Then
httpRequest.Timeout = HTTPDataIn.iTimeout * 1000
End If

'auto redirect
If HTTPDataIn.bAllowAutoRedirect = True Then
httpRequest.AllowAutoRedirect = True
Else
httpRequest.AllowAutoRedirect = False
End If

'accept
If HTTPDataIn.strAccept.Length > 0 Then
httpRequest.Accept = HTTPDataIn.strAccept
End If

'Referer
If HTTPDataIn.strReferer IsNot Nothing Then
httpRequest.Referer = HTTPDataIn.strReferer
End If

httpRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-us")
httpRequest.Headers.Add("UA-CPU", "x86")

'useragent
If HTTPDataIn.strUserAgent.Length > 0 Then
httpRequest.UserAgent = HTTPDataIn.strUserAgent
End If

'httpRequest.KeepAlive = True

'Cookies
httpRequest.CookieContainer = New CookieContainer()
If HTTPDataIn.cookieCollection IsNot Nothing Then
For iIndex = 0 To HTTPDataIn.cookieCollection.Count - 1
httpRequest.CookieContainer.Add(HTTPDataIn.cookieCollection)
Next
End If

httpRequest.ProtocolVersion = HttpVersion.Version10


'Console Logging Out
LoggingOut(HTTPDataIn, httpRequest, HTTPDataIn.cookieCollection, True)

httpResponse = CType(httpRequest.GetResponse(), HttpWebResponse)

responseStream = httpResponse.GetResponseStream()
responseEncoding = System.Text.Encoding.GetEncoding("utf-8")

HTTPDataOut.strResponseCode = httpResponse.StatusCode.ToString

'Return header information
Dim colHeaders As New System.Net.WebHeaderCollection
colHeaders = httpResponse.Headers
HTTPDataOut.strLocation = colHeaders("Location")
HTTPDataOut.strEncoding = colHeaders("Content-Encoding")
HTTPDataOut.strServer = httpResponse.ResponseUri.OriginalString
'Return cookies
Dim ccTemp As CookieCollection = httpResponse.Cookies

'This is required because of a "bug" in the cookies class. Its not really a bug
'because it is RFC complient but not all cookies will be set in some cases. An
'example of this is www.aol.com. When sending cookies to other subdomains such as
'new.aol.com the cookie will not be sent even though the PATH variable is set to '/'
If HTTPDataIn.bSetCookieToRoot = True Then
Dim strSplit() As String
For iIndex = 0 To ccTemp.Count - 1
strSplit = ccTemp.Item(iIndex).Domain.Split("."c)
'Use only the domain, i.e. .aol.com
ccTemp.Item(iIndex).Domain = "." + strSplit(strSplit.Length - 2) + "." + strSplit(strSplit.Length - 1)
strSplit = Nothing
Next
End If
HTTPDataOut.cookieContainer = ccTemp


'Decode if gzip is used
If HTTPDataOut.strEncoding = "gzip" Then
Dim gzipDLL As New ICSharpCode.SharpZipLib.GZip.GZipInputStream(responseStream)
responseStreamReader = New StreamReader(gzipDLL, responseEncoding)
Else
responseStreamReader = New StreamReader(responseStream, responseEncoding)
End If

If Not responseStreamReader Is Nothing Then
HTTPDataOut.strResponse = responseStreamReader.ReadToEnd
Else
HTTPDataOut.strResponse = ""
HTTPDataOut.strErrorCode = "Null Response"
HTTPDataOut.bUnsucessful = True
End If

'Console Logging In
LoggingIn(HTTPDataIn, HTTPDataOut, httpResponse)
httpResponse.Close()

'handle thrown exceptions
Catch ex As WebException
WebExceptionHandling(ex, HTTPDataOut)
Catch ex As InvalidOperationException
HTTPDataOut.strResponse = ""
HTTPDataOut.strErrorCode = "Invalid Operation Exception"
HTTPDataOut.bUnsucessful = True
Catch ex As UriFormatException
HTTPDataOut.strResponse = ""
HTTPDataOut.strErrorCode = "Uri Format Exception"
HTTPDataOut.bUnsucessful = True
Catch ex As NullReferenceException
HTTPDataOut.strResponse = ""
HTTPDataOut.strErrorCode = "Null Reference Exception"
HTTPDataOut.bUnsucessful = True
Catch ex As Exception
HTTPDataOut.strResponse = ""
HTTPDataOut.strErrorCode = "General Exception"
HTTPDataOut.bUnsucessful = True
Finally
If Not httpResponse Is Nothing Then
httpResponse.Close()
End If
If Not responseStream Is Nothing Then
responseStream.Close()
End If
If Not responseStreamReader Is Nothing Then
responseStreamReader.Close()
End If
End Try
Return HTTPDataOut
End Function


Here is a simple call using HTTPHelper, in this case requesting the Yahoo! mail signin page.

CODE
Private Function requestMailPage() As Boolean
_HTTPDataIn.strServer = strRegionalServer + "/config/mail"
_HTTPDataIn.bAllowAutoRedirect = True
_HTTPDataIn.cookieCollection = Nothing

Try
_HTTPDataOut = HTTPGET(_HTTPDataIn)
If _HTTPDataOut.bUnsucessful = True Then
BeginInvoke(_cmd_BeforeFailDeleg, New Object() {_strLoginName, _strPassword, _HTTPDataOut.strErrorCode})
Else
_strReferer = _HTTPDataOut.strServer
_cookieHomepage = _HTTPDataOut.cookieContainer

'Signup server
_strSignupServer = HTTPStringFind(_HTTPDataOut.strResponse, "<p class=""yreglgsub""><a href=""", """>")

requestSigninPage()
End If
Catch ex As Exception
MessageBox.Show(ex.ToString)
End Try
End Function
Go to the top of the page
 
+Quote Post
faulty.lee
post Jun 1 2008, 08:54 AM
Post #4


Premium Member
Group Icon

Group: [HOSTED]
Posts: 479
Joined: 5-November 06
Member No.: 17,016



I look into that in the coming few days.

One i think i can suggest is using packet capturing tools, maybe wireshark(free) or commview(my favourite), so study the diff in the data exchanged during the login process.

I've got wireshark installed, will try that do. I'll post the result here.
Go to the top of the page
 
+Quote Post
tansqrx
post Jun 2 2008, 08:55 PM
Post #5


Super Member
Group Icon

Group: [HOSTED]
Posts: 522
Joined: 25-April 05
Member No.: 4,374



I have relied heavily on capture tools to produce what I have now. I primarily use Wireshark but most of the Yahoo! pages are SSL so Wireshark is useless. There is a way to decode SSL packets with Wireshark but you have to have the server private key which I don’t think Yahoo! is willing to give up anytime soon. The next best thing is Fiddler (http://www.fiddlertool.com/fiddler/) which is an Internet Explorer sniffer. It adds a certificate to the system that acts like a man in the middle attack and you are able to see all of the SSL traffic coming into and out of Internet Explorer. It does have a downside in that you can not see the entire packet but only the TCP payload which is usually enough. The best part is that it is free and the most powerful of this class of tool.

Currently my httpWebRequest mimics Internet Explorer in all of the HTTP fields but not the order. The only thing left that I can see is that the COOKIE field is placed ahead of the KEEP-ALIVE field. No matter what I do I can not change the order to place the COOKIE at the end just before the /r/n/r/n. I suppose the httpWebRequest class automatically places the keep alive at the end during the automatic generation of the request headers and the placement is hard coded. From a specification standpoint this should not matter but this could be the thing that Yahoo! is punishing me for.

My next task is to create my own httpWebRequest class where I can have finer control of the headers. This will be a lot of extra work but it looks like that’s the only way to get around this problem. I have also been exploring using the .NET Webbrowser object to just script my way through the registration process but that has opened a whole new set of problems and I am not happy with the results. Even when using the unmanaged ActiveX control I keep getting scripting error pop-ups and it is about 10 times slower than using my current method. I have successfully created accounts using this method but it was not painless and there is much work to be done before this is a viable method. This just once again shows that there is some fundamental difference between using httpWebRequest and Internet Explorer to create Yahoo! accounts.

P.S. I done some extra exploratory work this weekend to compare the results from HTTPHelper to Internet Explorer when requesting www.google.com (because of the SSL problem mentioned earlier). When the captures from Wireshark are run through diff there is no difference at all. The packets are identical down to the smallest detail.
Go to the top of the page
 
+Quote Post
faulty.lee
post Jun 3 2008, 04:51 AM
Post #6


Premium Member
Group Icon

Group: [HOSTED]
Posts: 479
Joined: 5-November 06
Member No.: 17,016



Looks like you've done most of the thing i can think of doing. Doesn't look like i can help much here. Anyway i'll still give it a try.

I got a few more "crazy" suggestion. Well, I'ven't touch anything from your side yet, so not sure if this is workable., and it might sound stupid.

1. Login to yahoo without SSL? I've tried turning off ssl suport then it won't allow me to login into yahoo. Not sure if there's way to disable the requirement so "older" browser would work. That way you can capture and compare the packet properly.

2. When you're trying doing a packet comparison, due to the fact that ssl encrypted the data, you can't see it's content. But, isn't the public key that's used to encrypt the data is viewable in the browser. If not the browser also won't be able to encrypt the data. I know it's useless to have just the public key, then again you can use the same key to encrypt the data you generated, and see if it resulted to the same output. This might be tedious, considering you need to try a few different timestamp to get it right.

3. Save the login page of yahoo, modified the post action, point to a non ssl localhost php page, to capture what it's trying to post. Then you can see if it's the same as what you expected.

I think the 3rd one might be more feasible. This sounds like a HUGE workaround to solve a very simple problem, which have yet to find another better solution tongue.gif

EDIT: i wonder if YPOP! is still working?

This post has been edited by faulty.lee: Jun 3 2008, 04:53 AM
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic

Collapse

> Similar Topics

Topics Topics
  1. Visual Basic.NET Help Needed.(7)


 



- Lo-Fi Version Time is now: 8th September 2008 - 07:03 AM