HTTP Problem

This is the forum for miscellaneous technical/programming questions.

Moderator: 2ffat

HTTP Problem

Postby theLizard » Tue Jul 18, 2017 9:41 pm

Hello All,

Using TIdHttp and up until sometime in April this year everything working fine with the following

http->Get("http://www.pvoutput.org");

The only thing that has changed is that the url name went from an HTTP to an HTTPS which by my understanding the http://www.pvoutput.org should be automatically redirected to https://www.pvoutput.org but when attempting to connect using GET I get an 301 Moved Permanently error.

I cannot understand what is going on here, is there anything, a setting for TIdHttp component that would address this problem?

Cheers
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby rlebeau » Mon Jul 24, 2017 4:33 pm

theLizard wrote:The only thing that has changed is that the url name went from an HTTP to an HTTPS which by my understanding the http://www.pvoutput.org should be automatically redirected to https://www.pvoutput.org


Yes, and that is done using an HTTP redirect, which all web browsers handle by default.

theLizard wrote:but when attempting to connect using GET I get an 301 Moved Permanently error.


301 is one of the defined HTTP redirect response codes. An EIdHTTPProtocolException will be thrown for a 301 response if the TIdHTTP::HandleRedirects property is false (which it is by default). Make sure it is set to true instead (or handle the TIdHTTP::OnRedirect event to get the new URL, and then call TIdHTTP::Get() again with it).

Also, make sure you have an SSLIOHandler component, like TIdSSLIOHandlerSocketOpenSSL, assigned to the TIdHTTP::IOHandler property to handle the SSL/TLS aspect of the new URL when following the redirect.

And then stop using the old URL, since it is clearly no longer valid (as it has been *permanently* moved to the new URL, as opposed to other 3xx redirects which are *temporary* instead and the old URL should still be used).
Last edited by rlebeau on Wed Jul 26, 2017 12:44 pm, edited 2 times in total.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1457
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: HTTP Problem

Postby theLizard » Tue Jul 25, 2017 4:56 pm

theLizard wrote:Also, make sure you have an SSLIOHandler assigned to handle the SSL/TLS aspect of the new URL.


Thanks Remy,

Since original post I tried HandleRedirect and assigning an SSLIOHandler which I thought would be the correct way to go about it your reply confirms this but I get an "Could not load SSL Library" error and I am not sure where to find the library on my system or what it is actually called

Cheers
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby theLizard » Tue Jul 25, 2017 7:07 pm

So, I have found libeay32 and libssl32 dll's and included these in my project but still get EidOSSLCouldNotLoadSSLLibrary error what do I need to do to get these loaded as they appear to be the libs missing?
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby HsiaLin » Tue Jul 25, 2017 10:35 pm

HsiaLin
BCBJ Master
BCBJ Master
 
Posts: 281
Joined: Sun Jul 08, 2007 6:29 pm

Re: HTTP Problem

Postby theLizard » Tue Jul 25, 2017 11:54 pm

HsiaLin wrote:http://docwiki.embarcadero.com/RADStudi ... en/OpenSSL


Thanks for the link, I put libs in my projects folder instead, will try putting them in the right place tomorrow.

Cheers
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby theLizard » Wed Jul 26, 2017 3:53 pm

OK, the libs are loading, no longer getting "Cannot Load Library" error but getting Socket Error 10060 - Connection Timed Out

Should I be doing something with the SSLIOHandler properties or HTTP properties

Cheers
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby rlebeau » Wed Jul 26, 2017 6:40 pm

theLizard wrote:OK, the libs are loading, no longer getting "Cannot Load Library" error but getting Socket Error 10060 - Connection Timed Out

Should I be doing something with the SSLIOHandler properties or HTTP properties


Indy's defaults are usually sufficient for most cases. But not always.

When I use the latest version of Indy to connect to "http://www.pvoutput.org", using all of Indy's defaults (other than HandleRedirects, which I set to true), TIdHTTP follows the redirect to "https://www.pvoutput.org", and then the HTTPS server immediately closes the TCP connection gracefully as soon as it receives Indy's SSL/TLS handshake hello. TIdSSLIOHandlerSocketOpenSSL throws an OpenSSL exception accordingly, not a timeout error (specifically, it throws an error that says "encountered EOF that violated the protocol", which is normal for an unexpected disconnect during the SSL/TLS handshake process).

Maybe in your case, the server is not closing the connection gracefully, or your OS is not detecting it gracefully. Use a packet sniffer, like Wireshark, to verify. If the connection is not closed gracefully, that might be causing the timeout error (TIdSSLIOHandlerSocketOpenSSL does enable socket timeouts on Windows Vista+).

The same test works fine in web browsers. The difference being that TIdSSLIOHandlerSocketOpenSSL enables only TLS 1.0 by default, but modern web browsers also enable TLS 1.1 and 1.2 as well. When I enable TLS 1.1 in TIdSSLIOHandlerSocketOpenSSL, TIdHTTP is able to connect to the HTTPS URL and retrieve its HTML normally.

So clearly, the HTTPS server in question does not like Indy's use of TLS 1.0 and wants TLS 1.1+ instead.
Last edited by rlebeau on Mon Jul 31, 2017 11:21 am, edited 1 time in total.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1457
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: HTTP Problem

Postby theLizard » Sat Jul 29, 2017 12:43 am

theLizard wrote: The difference being that TIdSSLIOHandlerSocketOpenSSL enables only TLS 1.0 by default, but modern web browsers also enable TLS 1.1 and 1.2 as well. When I enable TLS 1.1 in TIdSSLIOHandlerSocketOpenSSL, TIdHTTP is able to connect to the HTTPS URL and retrieve its HTML normally.


Thanks Remy,

With your input I have got my problem sorted out however, using http->Get(...) I get EIdException "Error Connecting with SSL" this in itself is not a problem because the http->Post(...) does it's job correctly and that is what I am after so all I would like to know at this point is what could be causing the "Error Connecting with SSL" and is there a specific EId Exception I could be testing for.

Cheers
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby rlebeau » Mon Jul 31, 2017 11:30 am

theLizard wrote:however, using http->Get(...) I get EIdException "Error Connecting with SSL"


I am not able to reproduce that given the URL you provided. Are you using up-to-date versions of Indy and OpenSSL?

theLizard wrote:this in itself is not a problem because the http->Post(...) does it's job correctly and that is what I am after


Depending on the server's requirements, a GET may or may not be required before a POST, in order to receive server cookies that are required by the POST handler.

Also, how are you able to post data to the server without a successful SSL/TLS connection? Or, is the post URL using HTTP and not HTTPS?

theLizard wrote:so all I would like to know at this point is what could be causing the "Error Connecting with SSL"


I can't answer that without more details, such as a packet capture of the actual SSL/TLS handshake. There are many possibilities for why an SSL/TLS handshake would fail. Maybe there is a TLS version mismatch between your client and the server, for instance.

theLizard wrote:is there a specific EId Exception I could be testing for.


The "Error Connecting with SSL" error is thrown as an EIdOSSLConnectError exception.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1457
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: HTTP Problem

Postby theLizard » Mon Jul 31, 2017 2:32 pm

rlebeau wrote:I am not able to reproduce that given the URL you provided. Are you using up-to-date versions of Indy and OpenSSL?


The app uses c++ 2010 so am using the indy components that came with it which I would say are not upto date.

rlebeau wrote:Depending on the server's requirements, a GET may or may not be required before a POST, in order to receive server cookies that are required by the POST handler.


In my case I am using GET to establish that a connection can be made before a post is attempted, it is possible that a GET is not required but have not tested that.

rlebeau wrote:Also, how are you able to post data to the server without a successful SSL/TLS connection? Or, is the post URL using HTTP and not HTTPS?


POST is using HTTP as is GET, I have not made changes to the URL code leaving the component to handle redirects. will change to https and see the result.

rlebeau wrote:I can't answer that without more details, such as a packet capture of the actual SSL/TLS handshake. There are many possibilities for why an SSL/TLS handshake would fail. Maybe there is a TLS version mismatch between your client and the server, for instance.


Would love to be able to show packet capture but I cannot install wireshark (I am comfortable with it) on my dev machine it causes an hissy fit for wireshark, am in the process of migrating my dev machine to a newer updated system.

rlebeau wrote:The "Error Connecting with SSL" error is thrown as an EIdOSSLConnectError exception.


Will test for this error, cheers.


Thanks again for your assistance Remy.
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm

Re: HTTP Problem

Postby rlebeau » Mon Jul 31, 2017 3:07 pm

theLizard wrote:The app uses c++ 2010 so am using the indy components that came with it which I would say are not upto date.


Not even close. Please upgrade.

theLizard wrote:In my case I am using GET to establish that a connection can be made before a post is attempted


That is a waste of a connection, especially if HTTP keep-alives are not used. There is no difference in establishing a connection to the server for a GET vs a POST, so just do the POST unconditionally and handle any errors it may throw. HTTP is stateless, it doesn't care why a connection is being made, and it does not guarantee that the connection will remain open after the server's response is sent. Since each request will potentially require a new connection anyway, just use as few requests as needed to get the job done. Like I said, there is only really one reason to ever perform a GET before a POST, and that is if server generated cookies/token are required in the POST, such as when posting an HTML webform. Otherwise, don't do it, unless a POST-based API requires it.

theLizard wrote:POST is using HTTP as is GET, I have not made changes to the URL code leaving the component to handle redirects. will change to https and see the result.


If GET requires HTTPS, then POST most certainly will as well.

theLizard wrote:Would love to be able to show packet capture but I cannot install wireshark (I am comfortable with it) on my dev machine it causes an hissy fit for wireshark, am in the process of migrating my dev machine to a newer updated system.


Wireshark is not the only way to capture network traffic.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1457
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: HTTP Problem

Postby theLizard » Thu Aug 17, 2017 9:11 pm

Thanks Remy, have followed your suggestion eliminating GET all is fine now..

Cheers
theLizard
BCBJ Master
BCBJ Master
 
Posts: 447
Joined: Wed Mar 18, 2009 2:14 pm


Return to Technical

Who is online

Users browsing this forum: Google [Bot] and 11 guests