Repository navigation
Closed
Labels
3.7 (EOL)end of lifeend of lifestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-securityA security issueA security issue
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/vstinner'
closed_at = <Date 2017-07-26.02:48:01.344>
created_at = <Date 2017-05-29.04:04:12.437>
labels = ['type-security', '3.7', 'library']
title = '[security] urllib connects to a wrong host'
updated_at = <Date 2019-05-10.17:59:50.407>
user = 'https://github.com/postmasters'bugs.python.org fields:
activity = <Date 2019-05-10.17:59:50.407>
actor = 'ned.deily'
assignee = 'vstinner'
closed = True
closed_date = <Date 2017-07-26.02:48:01.344>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2017-05-29.04:04:12.437>
creator = 'Nam.Nguyen'
dependencies = []
files = []
hgrepos = []
issue_num = 30500
keywords = []
message_count = 20.0
messages = ['294667', '294671', '294694', '295318', '296422', '296426', '296427', '296430', '296431', '296432', '296433', '296436', '296439', '296442', '296446', '296448', '296455', '297939', '298211', '299187']
nosy_count = 6.0
nosy_names = ['vstinner', 'larry', 'ned.deily', 'martin.panter', 'Nam.Nguyen', 'serhiy.storchaka']
pr_nums = ['1849', '1850', '2289', '2290', '2291', '2292', '2293', '2295', '2296', '2294']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue30500'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7']Metadata
Metadata
Assignees
Labels
3.7 (EOL)end of lifeend of lifestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-securityA security issueA security issue
Activity
postmasters commentedon May 29, 2017
Reported by Orange Tsai:
==========
Hi, Python Security Team
I have tested on the latest version of Python 2.7.13.
==========
vadmium commentedon May 29, 2017
See also bpo-18140, where it looks like people _want_ the hash (#) to be part of the username and/or password.
Another option may be to raise an exception.
postmasters commentedon May 29, 2017
I think the best behavior is to do what popular web browsers do. Chrome and Firefox, for example, parses this is host 127.0.0.1, path /, fragment #@evil.com.
If the code does want to support username/password, it should do a custom opener (with basic HTTP authentication) instead.
vstinner commentedon Jun 7, 2017
I agree that in case of doubt, it's better to follow the implementation of most popular web browser which indirectly define the "standard".
[-]urllib connects to a wrong host[/-][+][security] urllib connects to a wrong host[/+]vstinner commentedon Jun 20, 2017
When porting the change to Python 3.4, I found this older change:
made by:
commit 44eceb6
Author: Serhiy Storchaka <storchaka@gmail.com>
Date: Tue Mar 3 20:21:35 2015 +0200
With this change, newlines are now accepted in URLs.
@serhiy: Was it a deliberate change?
vstinner commentedon Jun 20, 2017
New changeset 4899d84 by Victor Stinner in branch '3.5':
bpo-30500: urllib: Simplify splithost by calling into urlparse. (bpo-1849) (bpo-2290)
4899d84
vstinner commentedon Jun 20, 2017
New changeset 536c1f1 by Victor Stinner in branch '3.6':
bpo-30500: urllib: Simplify splithost by calling into urlparse. (bpo-1849) (bpo-2289)
536c1f1
vstinner commentedon Jun 20, 2017
New changeset 82acabd by Victor Stinner in branch '3.6':
bpo-30500: Fix the NEWS entry (bpo-2296)
82acabd
vstinner commentedon Jun 20, 2017
New changeset 4108606 by Victor Stinner in branch '3.5':
bpo-30500: Fix the NEWS entry (bpo-2295)
4108606
vstinner commentedon Jun 20, 2017
New changeset 8457706 by Victor Stinner in branch 'master':
bpo-30500: Fix the NEWS entry (bpo-2293)
8457706
serhiy-storchaka commentedon Jun 20, 2017
Oh, I didn't expected that newlines can be in a host name. In any case if newlines are a problem, it is better to check explicitly whether a host name contains CR, LF or other special characters. And it is better to do such checks when format a request rather than when parse an URL.
vstinner commentedon Jun 20, 2017
New changeset d4324ba by Victor Stinner in branch '2.7':
bpo-30500: urllib: Simplify splithost by calling into urlparse. (bpo-1849) (bpo-2294)
d4324ba
vstinner commentedon Jun 20, 2017
Oh, I was too fast. I wanted to see an agreement on DOTALL before merging this one. I missed that the 2.7 change also added DOTALL.
I had to handle many branches (2.7, 3.3, 3.4, 3.5, 3.6, master), conflicts (especially in 2.7!), and there was a mistake in the NEWS entry (http://... => //...). Sorry, I missed the DOTALL in 2.7 :-p
vstinner commentedon Jun 20, 2017
I tested my system python2 (Python 2.7.13 on Fedora 25):
=> Newline is accepted in the hostname, but not in the URL path.
With my change (adding DOTALL), newlines are accepted in the hostname and in the URL:
More generally, it seems like the urllib module doesn't try to validate URL content. It just try to "split" them.
Who is responsible to validate URLs? Is it the responsability of the application developer to implement a whitelist?
serhiy-storchaka commentedon Jun 20, 2017
The urllib package consists of two parts: urllib.parse and urllib.request. I think urllib.request is responsible for making valid requests and validate arguments if needed.
serhiy-storchaka commentedon Jun 20, 2017
Or more low-level modules used by urllib.request: http, ftplib, etc.
vstinner commentedon Jun 20, 2017
I created bpo-30713: "Reject newline character (U+000A) in URLs in urllib.parse", to discuss how to handle newlines in URLs.
ned-deily commentedon Jul 8, 2017
New changeset b0fba88 by Ned Deily (Victor Stinner) in branch '3.6':
bpo-30500: urllib: Simplify splithost by calling into urlparse. (bpo-1849) (bpo-2289)
b0fba88
larryhastings commentedon Jul 12, 2017
New changeset cc54c1c by larryhastings (Victor Stinner) in branch '3.4':
bpo-30500: urllib: Simplify splithost by calling into urlparse. (bpo-1849) (bpo-2291)
cc54c1c
ned-deily commentedon Jul 26, 2017
New changeset 052f9d6 by Ned Deily (Victor Stinner) in branch '3.3':
[3.3] bpo-30500: urllib: Simplify splithost by calling into urlparse. (bpo-1849) (bpo-2292)
052f9d6