Tuesday, August 16, 2011

 

LWP::UserAgent (and by extension WWW::Mechanize) proxying https requests

IO::Socket::SSL vs Net::SSL the fatal showdown. Ripped into the internals of LWP::UserAgent to find out how it sets up proxying for https connections. All the info on the net says that you cannot do $ua->proxy('https','1.2.3.4:8080); because it just wont fucking work for https. Also you can go ahead and set up shit like $ENV{HTTPS_PROXY} but scratch you head as to why its not actually being used by the code. So then you call env_proxy and wonder why the fuck its all of a sudden again not working for https. ... anyway, at least on my system, turns out Net::HTTPS is doing some decision making about what socket module to use ... and if you want it to actually correctly make use of your HTTPS_PROXY environment variable, you need to find a way to tell it to use Net::SSL. ... I'm not quite sure on that part yet, but I wanted to share that which took me 7 (SEVEN) dang hours to figure out. I sometimes think I am in the wrong line of work when it takes me so long to figure something out like this. And this problem exists since like 2001 for some people too judging by the unresolved posts I found scattered throughout the web.

Anyway next I think I need to do some testing and more reading of code but finally seeing the proxy IP in my ssl log and ... AND --> __AND__ seeing the proper html from the response, makes me really manically (maniacally even) happy. ... hoping to update this with elegant workaround/solution/fix that doesnt involve making hacked versions of the LWP and related modules.

Ok, protip: put this in your BEGIN block:

$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS}='Net::SSL';

and you just might achieve https proxying (via ENV settings) win. I seem to be. What this does is tells Net::HTTPS to use Net::SSL as the socket object. And Net::SSL (I havent looked inside) _seems_ to do stuff right so far (using ENV settings).


This page is powered by Blogger. Isn't yours?