You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
urllib.open sends full URL after GET command instead of local path #49322
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=<Date2009-02-18.14:38:21.246>created_at=<Date2009-01-26.19:22:53.170>labels= ['type-bug', 'library']
title='urllib.open sends full URL after GET command instead of local path'updated_at=<Date2009-02-18.14:38:21.245>user='https://bugs.python.org/olemis'
The first thing I have to say is that I searched the open issues and I
found nothing similar to what I am going to report hereinafter. If this
ticket is duplicate , I apologize ...
Yesterday I was testing how to access the wiki pages in a
Trac [1]_ site and I realized that something wrong was happening
(a bug? ...)
Initially the behavior was as follows :
{{{
#!python
>>> u = urllib.urlopen('http://localhost:8000/trac-dev')
>>> u.read()
'Environment not found'
>>> u.close()
}}}
The same picture is shown when using both Python 2.5.1 and 2.5.2 ...
I have not installed Python 2.6.x so I am not sure about whether this
issue has propagated onto newer versions of Python ... and I don't
know euther if this issue is also present in urllib2 or not ...
... so further research is needed, but IMO this is a serious bug :(
PD: If this is a bug ... how could it be hidden so far ? Is there any
test case written to assert this kind of things ? I checked out test.test_urllib and test.test_urllibnet modules and I saw
nothing at all ...
Ooops ... sorry, remove the print statement. The patch is as follows :
{{{
#!diff
--- /usr/lib/python2.5/urllib.py 2008-07-31 13:40:40.000000000 -0500+++ /media/urllib_unix.py 2009-01-26 09:48:54.000000000 -0500@@ -270,6 +270,7 @@
def open_http(self, url, data=None):
"""Use HTTP protocol."""
import httplib
+ from urlparse import urlparse
user_passwd = None
proxy_passwd= None
if isinstance(url, str):
@@ -312,12 +313,17 @@
else:
auth = None
h = httplib.HTTP(host)
+ target = ''.join(sep + part for sep, part in \+ zip(['', ';', '?', '#'], \+ urlparse(selector)[2:]) \+ if part)
if data is not None:
- h.putrequest('POST', selector)+ h.putrequest('POST', target)
h.putheader('Content-Type', 'application/x-www-form-
urlencoded')
h.putheader('Content-Length', '%d' % len(data))
else:
- h.putrequest('GET', selector)+ h.putrequest('GET', target)
if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' %
proxy_auth)
if auth: h.putheader('Authorization', 'Basic %s' % auth)
if realhost: h.putheader('Host', realhost)
}}}
I apologize once again ...
ggenellina commented on Jan 27, 2009
ggenellinamannequin
Mannequin
More actions
I could not reproduce this issue neither with Python 2.6 nor 2.5.2
If I print host and selector near line 313, I get 'localhost:8000' and
'/trac-dev', the expected results.
Do you have an HTTP proxy? running at the *same* port? (!)
olemis commented on Jan 27, 2009
olemismannequin
MannequinAuthor
More actions
Actually I am using a proxy hosted in some other machine (i.e. not my
PC ... sorry, I didnt mention :S ...) I «debugged» urllib and, when
branching at this point (see below ;) in URLopener.open_http :
My IP is 172.18.2.99 ... so the else branch is the one being executed
If you need further details ... dont hesitate and ask anything you
want ;)
PD: What d'u mean when you said?
Do you have an HTTP proxy? running at the *same* port? (!)
I dont understand this since *I already said* that *I accessed* my Trac
environment using my web browser (Opera 9.63, I dont know whether this
is relevant at all ... ), *I sent you* the lines outputted by tracd to
stdout (or stderr ... I am not very sure right now ... ;) and *I told
you* that, once I applied the path *I submitted*, everything was *back
to normal* ...
I dont understand how could all this be possible if I were running
tracd and an HTTP proxy in the same port, or even in case http_proxy envvar be set to the hostname + port where my Trac
instance is listening for incoming connections ...
Anyway ... CMIIW ...
I also checked that immediately before executing the following
statements ...
... selector is bound to 'http://localhost:8000/trac-dev' ... BTW the else clause is the one executed in this case, and this is
consistent with tracd reports I sent before and is logical since data arg is missing in the code snippet I sent before.
I suppose 172.18.2.7:3128 is the address:port of the your proxy, right?
In which case, urllib seems to do the right thing. When talking to an
HTTP proxy, requests are of the form "GET http://site.com/path", rather
than "GET /path". It's up to the proxy to strip the host part of the URL
when forwarding the request to the target server.
(but I suppose tracd could also be more permissive and allow the "GET http://site.com/path" variant. It seems Apache does)
olemis commented on Jan 27, 2009
olemismannequin
MannequinAuthor
More actions
Quoting Antoine Pitrou ...
I suppose 172.18.2.7:3128 is the address:port of the your proxy,
right?
Yes ...
In which case, urllib seems to do the right thing. When talking to an
HTTP proxy, requests are of the form "GET http://site.com/path", rather
than "GET /path". It's up to the proxy to strip the host part of the URL
when forwarding the request to the target server.
This being said ...
(but I suppose tracd could also be more permissive and allow the "GET http://site.com/path" variant. It seems Apache does)
... It works with Apache (I am talking about trac once again ...)
therefore I will report this issue to Trac devs instead ...
Thnx a lot ! Sorry if I caused you any trouble ...
ggenellina commented on Jan 28, 2009
ggenellinamannequin
Mannequin
More actions
> Do you have an HTTP proxy? running at the *same* port?
(!)
I dont understand this since *I already said* that *I
accessed* my Trac
environment using my web browser (Opera 9.63, I dont know
whether this
is relevant at all ... ), *I sent you* the lines outputted
by tracd to
stdout (or stderr ... I am not very sure right now ... ;)
and *I told
you* that, once I applied the path *I submitted*,
everything was *back
to normal* ...
If you had configured a proxy at localhost:8000, and *also* a Trac instance at that port, and Trac had "won the race" for the port, then you would observe exactly the symthoms you describe. That is, urllib talking to 8000 as it were a proxy, and the Trac instance actually there getting confused.
Your patch, as you surely understand now, is not correct; in fact, the code is OK as it is. urllib builds the request in that specific way *because* he thinks there is a proxy. If the proxy is buggy, misconfigured, or inexistent, it's not the library's fault :)
--
Gabriel Genellina
I dont understand how could all this be possible if I were
running
tracd and an HTTP proxy in the same port, or even in case
http_proxy envvar be set to the hostname + port where my
Trac
instance is listening for incoming connections ...
Anyway ... CMIIW ...
I also checked that immediately before executing the
following
statements ...
{{{
#!python
urllib,py
h = httplib.HTTP(host)
if data is not None:
h.putrequest('POST', selector)
h.putheader('Content-Type',
... selector is bound to
'http://localhost:8000/trac-dev' ... BTW the else clause is the one executed in this case, and this
is
consistent with tracd reports I sent before and is
logical since data arg is missing in the code snippet I sent before.
Activity
olemis commentedon Jan 26, 2009
Hello ...
The first thing I have to say is that I searched the open issues and I
found nothing similar to what I am going to report hereinafter. If this
ticket is duplicate , I apologize ...
Yesterday I was testing how to access the wiki pages in a
Trac [1]_ site and I realized that something wrong was happening
(a bug? ...)
Initially the behavior was as follows :
{{{ #!python >>> u = urllib.urlopen('http://localhost:8000/trac-dev') >>> u.read() 'Environment not found' >>> u.close() }}}And tracd reported a line like this
{{{
127.0.0.1 - - [25/Jan/2009 17:32:08] "GET http://localhost:8000/trac-
dev HTTP/1.0" 404 -
}}}
Which means that a 'Not found' error code was sent back to urllib
client.
I tried to access the same page from my browser and tracd reported
{{{
127.0.0.1 - - [25/Jan/2009 18:05:44] "GET /trac-dev HTTP/1.0" 200 -
}}}
The problem is obvious ... urllib was sending the full URL after GET
and it should send only the string after the network location.
I applied the following patch to urllib (yours will be better, I am
sure about that ;)
{{{
#!diff
}}}
And everithing was «back» to normal ...
{{{ #!python >>> u = urllib.urlopen('http://localhost:8000/trac-dev') >>> u.read() ... # Lots of beautiful HTML code ;) >>> u.close() }}}... tracd outputted ...
{{{
127.0.0.1 - - [25/Jan/2009 18:05:44] "GET /trac-dev HTTP/1.0" 200 -
}}}
The same picture is shown when using both Python 2.5.1 and 2.5.2 ...
I have not installed Python 2.6.x so I am not sure about whether this
issue has propagated onto newer versions of Python ... and I don't
know euther if this issue is also present in urllib2 or not ...
... so further research is needed, but IMO this is a serious bug :(
PD: If this is a bug ... how could it be hidden so far ? Is there any
test case written to assert this kind of things ? I checked out
test.test_urllibandtest.test_urllibnetmodules and I sawnothing at all ...
.. [1] Trac
(http://trac.edgewall.org)
olemis commentedon Jan 26, 2009
Ooops ... sorry, remove the print statement. The patch is as follows :
{{{
#!diff
}}}
I apologize once again ...
ggenellina commentedon Jan 27, 2009
I could not reproduce this issue neither with Python 2.6 nor 2.5.2
If I print host and selector near line 313, I get 'localhost:8000' and
'/trac-dev', the expected results.
Do you have an HTTP proxy? running at the *same* port? (!)
olemis commentedon Jan 27, 2009
Actually I am using a proxy hosted in some other machine (i.e. not my
PC ... sorry, I didnt mention :S ...) I «debugged» urllib and, when
branching at this point (see below ;) in URLopener.open_http :
{{{
#!python
# urllib,py
}}}
url variable is bound to the following binary tuple
{{{
#!python
('172.18.2.7:3128', 'http://localhost:8000/trac-dev')
}}}
My IP is 172.18.2.99 ... so the
elsebranch is the one being executedIf you need further details ... dont hesitate and ask anything you
want ;)
PD: What d'u mean when you said?
I dont understand this since *I already said* that *I accessed* my Trac
environment using my web browser (Opera 9.63, I dont know whether this
is relevant at all ... ), *I sent you* the lines outputted by tracd to
stdout (or stderr ... I am not very sure right now ... ;) and *I told
you* that, once I applied the path *I submitted*, everything was *back
to normal* ...
I dont understand how could all this be possible if I were running
tracd and an HTTP proxy in the same port, or even in case
http_proxyenvvar be set to the hostname + port where my Tracinstance is listening for incoming connections ...
Anyway ... CMIIW ...
I also checked that immediately before executing the following
statements ...
{{{
#!python
# urllib,py
}}}
...
selectoris bound to 'http://localhost:8000/trac-dev' ... BTW theelseclause is the one executed in this case, and this isconsistent with tracd reports I sent before and is logical since
dataarg is missing in the code snippet I sent before.pitrou commentedon Jan 27, 2009
I suppose 172.18.2.7:3128 is the address:port of the your proxy, right?
In which case, urllib seems to do the right thing. When talking to an
HTTP proxy, requests are of the form "GET http://site.com/path", rather
than "GET /path". It's up to the proxy to strip the host part of the URL
when forwarding the request to the target server.
(but I suppose tracd could also be more permissive and allow the "GET
http://site.com/path" variant. It seems Apache does)
olemis commentedon Jan 27, 2009
Yes ...
This being said ...
... It works with Apache (I am talking about trac once again ...)
therefore I will report this issue to Trac devs instead ...
Thnx a lot ! Sorry if I caused you any trouble ...
ggenellina commentedon Jan 28, 2009
If you had configured a proxy at localhost:8000, and *also* a Trac instance at that port, and Trac had "won the race" for the port, then you would observe exactly the symthoms you describe. That is, urllib talking to 8000 as it were a proxy, and the Trac instance actually there getting confused.
Your patch, as you surely understand now, is not correct; in fact, the code is OK as it is. urllib builds the request in that specific way *because* he thinks there is a proxy. If the proxy is buggy, misconfigured, or inexistent, it's not the library's fault :)
--
Gabriel Genellina
Recetas prácticas y comida saludable
http://ar.mujer.yahoo.com/cocina/
devdanzin commentedon Feb 12, 2009
Anyone against closing this as "works for me"?
orsenthil commentedon Feb 18, 2009
Yup, This should be closed too. Thanks.