Skip to content

urllib2 needs to remove scope from IPv6 address when creating Host header #67636

Open

Description

@ngierman
mannequin
BPO 23448
Nosy @gpshead, @vadmium

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-02-11.18:40:23.810>
labels = ['type-bug', 'library']
title = 'urllib2 needs to remove scope from IPv6 address when creating Host header'
updated_at = <Date 2017-01-26.21:22:28.565>
user = 'https://bugs.python.org/ngierman'

bugs.python.org fields:

activity = <Date 2017-01-26.21:22:28.565>
actor = 'JonathanGuthrie'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-02-11.18:40:23.810>
creator = 'ngierman'
dependencies = []
files = []
hgrepos = []
issue_num = 23448
keywords = []
message_count = 3.0
messages = ['235762', '235768', '286334']
nosy_count = 4.0
nosy_names = ['gregory.p.smith', 'martin.panter', 'ngierman', 'JonathanGuthrie']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23448'
versions = ['Python 2.7']

Activity

  1. ngierman commented on Feb 11, 2015

    ngiermanmannequin
    MannequinAuthor

    Using a scoped IPv6 address with urllib2 creates an invalid Host header that Apache will not accept.

            IP = "fe80::0000:0000:0000:0001%eth0"
            req = urllib2.Request("http://[" + IP + "]/")
            req.add_header('Content-Type', 'application/json')
            res = urllib2.urlopen(req, json.dumps(data))

    Apache will reject the above request because the Host header is "[fe80::0000:0000:0000:0001%eth0]". This behavior was reported to Apache at https://issues.apache.org/bugzilla/show_bug.cgi?id=35122 and the Apache devs will not fix this as there are new RFCs prohibiting scopes in the Host header. Firefox had the same issue and their fix was to strip out the scope from the Host header: https://bugzilla.mozilla.org/show_bug.cgi?id=464162 and http://hg.mozilla.org/mozilla-central/rev/bb80e727c531.

    My suggestion is to change urllib2.py's do_request_ method from:

            if not request.has_header('Host'):
                request.add_unredirected_header('Host', sel_host)

    to:

            if not request.has_header('Host'):
                request.add_unredirected_header('Host', re.compile(r"%.*$").sub("", sel_host, 1))

    I have not tested this patch to urllib2.py however I am now using similar logic in my code to override the Host header when I create my request:

            IP = "fe80::0000:0000:0000:0001%eth0"
            req = urllib2.Request("http://[" + IP + "]/")
            req.add_header('Host', '[' + re.compile(r"%.*").sub("", IP, 1) + ']')
            req.add_header('Content-Type', 'application/json')
            res = urllib2.urlopen(req, json.dumps(data))
  2. added
    stdlibStandard Library Python modules in the Lib/ directory
    type-bugAn unexpected behavior, bug, or error
    on Feb 11, 2015
  3. vadmium commented on Feb 11, 2015

    @vadmium
    Member

    I’m no IPv6 expert, but there seems to be a few standards:

    • <https://tools.ietf.org/html/rfc6874\> (Feb 2013). Encodes as http://[fe80::1%25eth0]/; says Windows uses this form. Also mentions the unencoded http://[fe80::1%eth0]/ form. Says that the HTTP Host header should not include the scope zone identifier, since it is not necessarily relevant to the server.

    • <https://tools.ietf.org/html/draft-sweet-uri-zoneid-01\> (Nov 2013). Encodes as http://[v1.fe80::1+eth0]/; says CUPS uses this form. Also acknowledges the RFC %25 form. Says that the Host header _should_ include the scope, to help with servers that send back self-referencing absolute URLs.

    Also, I would probably find IP.split('%', 1)[0] easier to read than a regular expression.

  4. JonathanGuthrie commented on Jan 26, 2017

    JonathanGuthriemannequin
    Mannequin

    Michael Sweet's draft RFC requiring that the scope should be included in the Host line expired in May 2014 and I can't find where it ever went anywhere. Does anyone have any updated information?

  5. transferred this issue fromon Apr 10, 2022
  6. elupus commented on Nov 18, 2026

    @elupus

    I think they kinda dropped the ball here. The draft replacement https://www.rfc-editor.org/info/rfc9844/ is targeting browsers and UI's. Here we are talking about backend services/programs that need a way to describe how to connect to something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions