<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>urllib2 – Library for opening URLs. — Python Module of the Week</title>
<link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '1.132',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="author" title="About these documents" href="../about.html" />
<link rel="top" title="Python Module of the Week" href="../index.html" />
<link rel="up" title="Internet Protocols and Support" href="../internet_protocols.html" />
<link rel="next" title="urlparse – Split URL into component pieces." href="../urlparse/index.html" />
<link rel="prev" title="urllib – simple interface for network resource access" href="../urllib/index.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="../urlparse/index.html" title="urlparse – Split URL into component pieces."
accesskey="N">next</a> |</li>
<li class="right" >
<a href="../urllib/index.html" title="urllib – simple interface for network resource access"
accesskey="P">previous</a> |</li>
<li><a href="../contents.html">PyMOTW</a> »</li>
<li><a href="../internet_protocols.html" accesskey="U">Internet Protocols and Support</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="../contents.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">urllib2 – Library for opening URLs.</a><ul>
<li><a class="reference internal" href="#http-get">HTTP GET</a><ul>
<li><a class="reference internal" href="#encoding-arguments">Encoding Arguments</a></li>
</ul>
</li>
<li><a class="reference internal" href="#http-post">HTTP POST</a></li>
<li><a class="reference internal" href="#working-with-requests-directly">Working with Requests Directly</a><ul>
<li><a class="reference internal" href="#adding-outgoing-headers">Adding Outgoing Headers</a></li>
<li><a class="reference internal" href="#posting-form-data">Posting Form Data</a></li>
<li><a class="reference internal" href="#uploading-files">Uploading Files</a></li>
</ul>
</li>
<li><a class="reference internal" href="#custom-protocol-handlers">Custom Protocol Handlers</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="../urllib/index.html"
title="previous chapter">urllib – simple interface for network resource access</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="../urlparse/index.html"
title="next chapter">urlparse – Split URL into component pieces.</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/urllib2/index.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="module-urllib2">
<span id="urllib2-library-for-opening-urls"></span><h1>urllib2 – Library for opening URLs.<a class="headerlink" href="#module-urllib2" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field"><th class="field-name">Purpose:</th><td class="field-body">A library for opening URLs that can be extended by defining custom protocol handlers.</td>
</tr>
<tr class="field"><th class="field-name">Python Version:</th><td class="field-body">2.1</td>
</tr>
</tbody>
</table>
<p>The <a class="reference internal" href="#module-urllib2" title="urllib2: Library for opening URLs."><tt class="xref py py-mod docutils literal"><span class="pre">urllib2</span></tt></a> module provides an updated API for using internet
resources identified by URLs. It is designed to be extended by
individual applications to support new protocols or add variations to
existing protocols (such as handling HTTP basic authentication).</p>
<div class="section" id="http-get">
<h2>HTTP GET<a class="headerlink" href="#http-get" title="Permalink to this headline">¶</a></h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The test server for these examples is in BaseHTTPServer_GET.py, from the
PyMOTW examples for <a class="reference internal" href="../BaseHTTPServer/index.html#module-BaseHTTPServer" title="BaseHTTPServer: Provides base classes for implementing web servers."><tt class="xref py py-mod docutils literal"><span class="pre">BaseHTTPServer</span></tt></a>. Start the server in one
terminal window, then run these examples in another.</p>
</div>
<p>As with <a class="reference internal" href="../urllib/index.html#module-urllib" title="urllib: Accessing remote resources that don't need authentication, cookies, etc."><tt class="xref py py-mod docutils literal"><span class="pre">urllib</span></tt></a>, an HTTP GET operation is the simplest use of
<a class="reference internal" href="#module-urllib2" title="urllib2: Library for opening URLs."><tt class="xref py py-mod docutils literal"><span class="pre">urllib2</span></tt></a>. Pass the URL to <tt class="xref py py-func docutils literal"><span class="pre">urlopen()</span></tt> to get a “file-like”
handle to the remote data.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s">'http://localhost:8080/'</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'RESPONSE:'</span><span class="p">,</span> <span class="n">response</span>
<span class="k">print</span> <span class="s">'URL :'</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">geturl</span><span class="p">()</span>
<span class="n">headers</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">info</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'DATE :'</span><span class="p">,</span> <span class="n">headers</span><span class="p">[</span><span class="s">'date'</span><span class="p">]</span>
<span class="k">print</span> <span class="s">'HEADERS :'</span>
<span class="k">print</span> <span class="s">'---------'</span>
<span class="k">print</span> <span class="n">headers</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'LENGTH :'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'DATA :'</span>
<span class="k">print</span> <span class="s">'---------'</span>
<span class="k">print</span> <span class="n">data</span>
</pre></div>
</div>
<p>The example server accepts the incoming values and formats a plain
text response to send back. The return value from <tt class="xref py py-func docutils literal"><span class="pre">urlopen()</span></tt>
gives access to the headers from the HTTP server through the
<tt class="xref py py-func docutils literal"><span class="pre">info()</span></tt> method, and the data for the remote resource via
methods like <tt class="xref py py-func docutils literal"><span class="pre">read()</span></tt> and <tt class="xref py py-func docutils literal"><span class="pre">readlines()</span></tt>.</p>
<div class="highlight-python"><pre>$ python urllib2_urlopen.py
RESPONSE: <addinfourl at 11940488 whose fp = <socket._fileobject object at 0xb573f0>>
URL : http://localhost:8080/
DATE : Sun, 19 Jul 2009 14:01:31 GMT
HEADERS :
---------
Server: BaseHTTP/0.3 Python/2.6.2
Date: Sun, 19 Jul 2009 14:01:31 GMT
LENGTH : 349
DATA :
---------
CLIENT VALUES:
client_address=('127.0.0.1', 55836) (localhost)
command=GET
path=/
real path=/
query=
request_version=HTTP/1.1
SERVER VALUES:
server_version=BaseHTTP/0.3
sys_version=Python/2.6.2
protocol_version=HTTP/1.0
HEADERS RECEIVED:
accept-encoding=identity
connection=close
host=localhost:8080
user-agent=Python-urllib/2.6</pre>
</div>
<p>The file-like object returned by <tt class="xref py py-func docutils literal"><span class="pre">urlopen()</span></tt> is iterable:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s">'http://localhost:8080/'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
<span class="k">print</span> <span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
</pre></div>
</div>
<p>This example strips the trailing newlines and carriage returns before
printing the output.</p>
<div class="highlight-python"><pre>$ python urllib2_urlopen_iterator.py
CLIENT VALUES:
client_address=('127.0.0.1', 55840) (localhost)
command=GET
path=/
real path=/
query=
request_version=HTTP/1.1
SERVER VALUES:
server_version=BaseHTTP/0.3
sys_version=Python/2.6.2
protocol_version=HTTP/1.0
HEADERS RECEIVED:
accept-encoding=identity
connection=close
host=localhost:8080
user-agent=Python-urllib/2.6</pre>
</div>
<div class="section" id="encoding-arguments">
<h3>Encoding Arguments<a class="headerlink" href="#encoding-arguments" title="Permalink to this headline">¶</a></h3>
<p>Arguments can be passed to the server by encoding them with
<a class="reference internal" href="../urllib/index.html#urllib-urlencode"><em>urllib.urlencode()</em></a> and appending them to the
URL.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">query_args</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'q'</span><span class="p">:</span><span class="s">'query string'</span><span class="p">,</span> <span class="s">'foo'</span><span class="p">:</span><span class="s">'bar'</span> <span class="p">}</span>
<span class="n">encoded_args</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">query_args</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Encoded:'</span><span class="p">,</span> <span class="n">encoded_args</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">'http://localhost:8080/?'</span> <span class="o">+</span> <span class="n">encoded_args</span>
<span class="k">print</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>The list of client values returned in the example output contains the
encoded query arguments.</p>
<div class="highlight-python"><pre>$ python urllib2_http_get_args.py
Encoded: q=query+string&foo=bar
CLIENT VALUES:
client_address=('127.0.0.1', 55849) (localhost)
command=GET
path=/?q=query+string&foo=bar
real path=/
query=q=query+string&foo=bar
request_version=HTTP/1.1
SERVER VALUES:
server_version=BaseHTTP/0.3
sys_version=Python/2.6.2
protocol_version=HTTP/1.0
HEADERS RECEIVED:
accept-encoding=identity
connection=close
host=localhost:8080
user-agent=Python-urllib/2.6</pre>
</div>
</div>
</div>
<div class="section" id="http-post">
<h2>HTTP POST<a class="headerlink" href="#http-post" title="Permalink to this headline">¶</a></h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">The test server for these examples is in BaseHTTPServer_POST.py, from the
PyMOTW examples for the <a class="reference internal" href="../BaseHTTPServer/index.html#module-BaseHTTPServer" title="BaseHTTPServer: Provides base classes for implementing web servers."><tt class="xref py py-mod docutils literal"><span class="pre">BaseHTTPServer</span></tt></a>. Start the server in one
terminal window, then run these examples in another.</p>
</div>
<p>To POST form-encoded data to the remote server, instead of using GET,
pass the encoded query arguments as data to <tt class="xref py py-func docutils literal"><span class="pre">urlopen()</span></tt>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">query_args</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'q'</span><span class="p">:</span><span class="s">'query string'</span><span class="p">,</span> <span class="s">'foo'</span><span class="p">:</span><span class="s">'bar'</span> <span class="p">}</span>
<span class="n">encoded_args</span> <span class="o">=</span> <span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">query_args</span><span class="p">)</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">'http://localhost:8080/'</span>
<span class="k">print</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">encoded_args</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>The server can decode the form data and access the individual values
by name.</p>
<div class="highlight-python"><pre>$ python urllib2_urlopen_post.py
Client: ('127.0.0.1', 55943)
User-agent: Python-urllib/2.6
Path: /
Form data:
q=query string
foo=bar</pre>
</div>
</div>
<div class="section" id="working-with-requests-directly">
<h2>Working with Requests Directly<a class="headerlink" href="#working-with-requests-directly" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-func docutils literal"><span class="pre">urlopen()</span></tt> is a convenience function that hides some of the
details of how the request is made and handled for you. For more
precise control, you may want to instantiate and use a
<tt class="xref py py-class docutils literal"><span class="pre">Request</span></tt> object directly.</p>
<div class="section" id="adding-outgoing-headers">
<h3>Adding Outgoing Headers<a class="headerlink" href="#adding-outgoing-headers" title="Permalink to this headline">¶</a></h3>
<p>As the examples above illustrate, the default <em>User-agent</em> header
value is made up of the constant <tt class="docutils literal"><span class="pre">Python-urllib</span></tt>, followed by the
Python interpreter version. If you are creating an application that
will access other people’s web resources, it is courteous to include
real user agent information in your requests, so they can identify the
source of the hits more easily. Using a custom agent also allows them
to control crawlers using a <tt class="docutils literal"><span class="pre">robots.txt</span></tt> file (see
<a class="reference internal" href="../robotparser/index.html#module-robotparser" title="robotparser: Internet spider access control"><tt class="xref py py-mod docutils literal"><span class="pre">robotparser</span></tt></a>).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s">'http://localhost:8080/'</span><span class="p">)</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_header</span><span class="p">(</span><span class="s">'User-agent'</span><span class="p">,</span> <span class="s">'PyMOTW (http://www.doughellmann.com/PyMOTW/)'</span><span class="p">)</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">print</span> <span class="n">data</span>
</pre></div>
</div>
<p>After creating a <tt class="xref py py-class docutils literal"><span class="pre">Request</span></tt> object, use <tt class="xref py py-func docutils literal"><span class="pre">add_header()</span></tt> to
set the user agent value before opening the request. The last line of
the output shows our custom value.</p>
<div class="highlight-python"><pre>$ python urllib2_request_header.py
CLIENT VALUES:
client_address=('127.0.0.1', 55876) (localhost)
command=GET
path=/
real path=/
query=
request_version=HTTP/1.1
SERVER VALUES:
server_version=BaseHTTP/0.3
sys_version=Python/2.6.2
protocol_version=HTTP/1.0
HEADERS RECEIVED:
accept-encoding=identity
connection=close
host=localhost:8080
user-agent=PyMOTW (http://www.doughellmann.com/PyMOTW/)</pre>
</div>
</div>
<div class="section" id="posting-form-data">
<h3>Posting Form Data<a class="headerlink" href="#posting-form-data" title="Permalink to this headline">¶</a></h3>
<p>You can set the outgoing data on the <tt class="xref py py-class docutils literal"><span class="pre">Request</span></tt> to post it to
the server.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="n">query_args</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'q'</span><span class="p">:</span><span class="s">'query string'</span><span class="p">,</span> <span class="s">'foo'</span><span class="p">:</span><span class="s">'bar'</span> <span class="p">}</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s">'http://localhost:8080/'</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Request method before data:'</span><span class="p">,</span> <span class="n">request</span><span class="o">.</span><span class="n">get_method</span><span class="p">()</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_data</span><span class="p">(</span><span class="n">urllib</span><span class="o">.</span><span class="n">urlencode</span><span class="p">(</span><span class="n">query_args</span><span class="p">))</span>
<span class="k">print</span> <span class="s">'Request method after data :'</span><span class="p">,</span> <span class="n">request</span><span class="o">.</span><span class="n">get_method</span><span class="p">()</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_header</span><span class="p">(</span><span class="s">'User-agent'</span><span class="p">,</span> <span class="s">'PyMOTW (http://www.doughellmann.com/PyMOTW/)'</span><span class="p">)</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'OUTGOING DATA:'</span>
<span class="k">print</span> <span class="n">request</span><span class="o">.</span><span class="n">get_data</span><span class="p">()</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'SERVER RESPONSE:'</span>
<span class="k">print</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>The HTTP method used by the <tt class="xref py py-class docutils literal"><span class="pre">Request</span></tt> changes from GET to POST
automatically after the data is added.</p>
<div class="highlight-python"><pre>$ python urllib2_request_post.py
Request method before data: GET
Request method after data : POST
OUTGOING DATA:
q=query+string&foo=bar
SERVER RESPONSE:
Client: ('127.0.0.1', 56044)
User-agent: PyMOTW (http://www.doughellmann.com/PyMOTW/)
Path: /
Form data:
q=query string
foo=bar</pre>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Although the method is <tt class="xref py py-func docutils literal"><span class="pre">add_data()</span></tt>, its effect is <em>not</em>
cumulative. Each call replaces the previous data.</p>
</div>
</div>
<div class="section" id="uploading-files">
<h3>Uploading Files<a class="headerlink" href="#uploading-files" title="Permalink to this headline">¶</a></h3>
<p>Encoding files for upload requires a little more work than simple forms. A complete MIME
message needs to be constructed in the body of the request, so that the server can
distinguish incoming form fields from uploaded files.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">itertools</span>
<span class="kn">import</span> <span class="nn">mimetools</span>
<span class="kn">import</span> <span class="nn">mimetypes</span>
<span class="kn">from</span> <span class="nn">cStringIO</span> <span class="kn">import</span> <span class="n">StringIO</span>
<span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="k">class</span> <span class="nc">MultiPartForm</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="sd">"""Accumulate the data to be used when posting a form."""</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_fields</span> <span class="o">=</span> <span class="p">[]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">files</span> <span class="o">=</span> <span class="p">[]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">boundary</span> <span class="o">=</span> <span class="n">mimetools</span><span class="o">.</span><span class="n">choose_boundary</span><span class="p">()</span>
<span class="k">return</span>
<span class="k">def</span> <span class="nf">get_content_type</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'multipart/form-data; boundary=</span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">boundary</span>
<span class="k">def</span> <span class="nf">add_field</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="sd">"""Add a simple field to the form data."""</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_fields</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
<span class="k">return</span>
<span class="k">def</span> <span class="nf">add_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fieldname</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">fileHandle</span><span class="p">,</span> <span class="n">mimetype</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="sd">"""Add a file to be uploaded."""</span>
<span class="n">body</span> <span class="o">=</span> <span class="n">fileHandle</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">if</span> <span class="n">mimetype</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="n">mimetype</span> <span class="o">=</span> <span class="n">mimetypes</span><span class="o">.</span><span class="n">guess_type</span><span class="p">(</span><span class="n">filename</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">or</span> <span class="s">'application/octet-stream'</span>
<span class="bp">self</span><span class="o">.</span><span class="n">files</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">fieldname</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mimetype</span><span class="p">,</span> <span class="n">body</span><span class="p">))</span>
<span class="k">return</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">"""Return a string representing the form data, including attached files."""</span>
<span class="c"># Build a list of lists, each containing "lines" of the</span>
<span class="c"># request. Each part is separated by a boundary string.</span>
<span class="c"># Once the list is built, return a string where each</span>
<span class="c"># line is separated by '\r\n'. </span>
<span class="n">parts</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">part_boundary</span> <span class="o">=</span> <span class="s">'--'</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">boundary</span>
<span class="c"># Add the form fields</span>
<span class="n">parts</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span>
<span class="p">[</span> <span class="n">part_boundary</span><span class="p">,</span>
<span class="s">'Content-Disposition: form-data; name="</span><span class="si">%s</span><span class="s">"'</span> <span class="o">%</span> <span class="n">name</span><span class="p">,</span>
<span class="s">''</span><span class="p">,</span>
<span class="n">value</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">form_fields</span>
<span class="p">)</span>
<span class="c"># Add the files to upload</span>
<span class="n">parts</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span>
<span class="p">[</span> <span class="n">part_boundary</span><span class="p">,</span>
<span class="s">'Content-Disposition: file; name="</span><span class="si">%s</span><span class="s">"; filename="</span><span class="si">%s</span><span class="s">"'</span> <span class="o">%</span> \
<span class="p">(</span><span class="n">field_name</span><span class="p">,</span> <span class="n">filename</span><span class="p">),</span>
<span class="s">'Content-Type: </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">content_type</span><span class="p">,</span>
<span class="s">''</span><span class="p">,</span>
<span class="n">body</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">field_name</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">content_type</span><span class="p">,</span> <span class="n">body</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">files</span>
<span class="p">)</span>
<span class="c"># Flatten the list and add closing boundary marker,</span>
<span class="c"># then return CR+LF separated data</span>
<span class="n">flattened</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">itertools</span><span class="o">.</span><span class="n">chain</span><span class="p">(</span><span class="o">*</span><span class="n">parts</span><span class="p">))</span>
<span class="n">flattened</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">'--'</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">boundary</span> <span class="o">+</span> <span class="s">'--'</span><span class="p">)</span>
<span class="n">flattened</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">''</span><span class="p">)</span>
<span class="k">return</span> <span class="s">'</span><span class="se">\r\n</span><span class="s">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">flattened</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="c"># Create the form with simple fields</span>
<span class="n">form</span> <span class="o">=</span> <span class="n">MultiPartForm</span><span class="p">()</span>
<span class="n">form</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">'firstname'</span><span class="p">,</span> <span class="s">'Doug'</span><span class="p">)</span>
<span class="n">form</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">'lastname'</span><span class="p">,</span> <span class="s">'Hellmann'</span><span class="p">)</span>
<span class="c"># Add a fake file</span>
<span class="n">form</span><span class="o">.</span><span class="n">add_file</span><span class="p">(</span><span class="s">'biography'</span><span class="p">,</span> <span class="s">'bio.txt'</span><span class="p">,</span>
<span class="n">fileHandle</span><span class="o">=</span><span class="n">StringIO</span><span class="p">(</span><span class="s">'Python developer and blogger.'</span><span class="p">))</span>
<span class="c"># Build the request</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="s">'http://localhost:8080/'</span><span class="p">)</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_header</span><span class="p">(</span><span class="s">'User-agent'</span><span class="p">,</span> <span class="s">'PyMOTW (http://www.doughellmann.com/PyMOTW/)'</span><span class="p">)</span>
<span class="n">body</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">form</span><span class="p">)</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_header</span><span class="p">(</span><span class="s">'Content-type'</span><span class="p">,</span> <span class="n">form</span><span class="o">.</span><span class="n">get_content_type</span><span class="p">())</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_header</span><span class="p">(</span><span class="s">'Content-length'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">body</span><span class="p">))</span>
<span class="n">request</span><span class="o">.</span><span class="n">add_data</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'OUTGOING DATA:'</span>
<span class="k">print</span> <span class="n">request</span><span class="o">.</span><span class="n">get_data</span><span class="p">()</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'SERVER RESPONSE:'</span>
<span class="k">print</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
</div>
<p>The <tt class="xref py py-class docutils literal"><span class="pre">MultiPartForm</span></tt> class can represent an arbitrary form as a
multi-part MIME message with attached files.</p>
<div class="highlight-python"><pre>$ python urllib2_upload_files.py
OUTGOING DATA:
--192.168.1.17.527.30074.1248020372.206.1
Content-Disposition: form-data; name="firstname"
Doug
--192.168.1.17.527.30074.1248020372.206.1
Content-Disposition: form-data; name="lastname"
Hellmann
--192.168.1.17.527.30074.1248020372.206.1
Content-Disposition: file; name="biography"; filename="bio.txt"
Content-Type: text/plain
Python developer and blogger.
--192.168.1.17.527.30074.1248020372.206.1--
SERVER RESPONSE:
Client: ('127.0.0.1', 57126)
User-agent: PyMOTW (http://www.doughellmann.com/PyMOTW/)
Path: /
Form data:
lastname=Hellmann
Uploaded biography as "bio.txt" (29 bytes)
firstname=Doug</pre>
</div>
</div>
</div>
<div class="section" id="custom-protocol-handlers">
<h2>Custom Protocol Handlers<a class="headerlink" href="#custom-protocol-handlers" title="Permalink to this headline">¶</a></h2>
<p><a class="reference internal" href="#module-urllib2" title="urllib2: Library for opening URLs."><tt class="xref py py-mod docutils literal"><span class="pre">urllib2</span></tt></a> has built-in support for HTTP(S), FTP, and local file
access. If you need to add support for other URL types, you can
register your own protocol handler to be invoked as needed. For
example, if you want to support URLs pointing to arbitrary files on
remote NFS servers, without requiring your users to mount the path
manually, would create a class derived from <tt class="xref py py-class docutils literal"><span class="pre">BaseHandler</span></tt> and
with a method <tt class="xref py py-func docutils literal"><span class="pre">nfs_open()</span></tt>.</p>
<p>The protocol <tt class="xref py py-func docutils literal"><span class="pre">open()</span></tt> method takes a single argument, the
<tt class="xref py py-class docutils literal"><span class="pre">Request</span></tt> instance, and it should return an object with a
<tt class="xref py py-func docutils literal"><span class="pre">read()</span></tt> method that can be used to read the data, an
<tt class="xref py py-func docutils literal"><span class="pre">info()</span></tt> method to return the response headers, and
<tt class="xref py py-func docutils literal"><span class="pre">geturl()</span></tt> to return the actual URL of the file being read. A
simple way to achieve that is to create an instance of
<tt class="xref py py-class docutils literal"><span class="pre">urllib.addurlinfo</span></tt>, passing the headers, URL, and open file
handle in to the constructor.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">mimetypes</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">tempfile</span>
<span class="kn">import</span> <span class="nn">urllib</span>
<span class="kn">import</span> <span class="nn">urllib2</span>
<span class="k">class</span> <span class="nc">NFSFile</span><span class="p">(</span><span class="nb">file</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tempdir</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">tempdir</span> <span class="o">=</span> <span class="n">tempdir</span>
<span class="nb">file</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="s">'rb'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">close</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'NFSFile:'</span>
<span class="k">print</span> <span class="s">' unmounting </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">tempdir</span>
<span class="k">print</span> <span class="s">' when </span><span class="si">%s</span><span class="s"> is closed'</span> <span class="o">%</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">file</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">FauxNFSHandler</span><span class="p">(</span><span class="n">urllib2</span><span class="o">.</span><span class="n">BaseHandler</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tempdir</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">tempdir</span> <span class="o">=</span> <span class="n">tempdir</span>
<span class="k">def</span> <span class="nf">nfs_open</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">req</span><span class="p">):</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">get_selector</span><span class="p">()</span>
<span class="n">directory_name</span><span class="p">,</span> <span class="n">file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="n">server_name</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">get_host</span><span class="p">()</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'FauxNFSHandler simulating mount:'</span>
<span class="k">print</span> <span class="s">' Remote path: </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">directory_name</span>
<span class="k">print</span> <span class="s">' Server : </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">server_name</span>
<span class="k">print</span> <span class="s">' Local path : </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">tempdir</span>
<span class="k">print</span> <span class="s">' File name : </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="n">file_name</span>
<span class="n">local_file</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tempdir</span><span class="p">,</span> <span class="n">file_name</span><span class="p">)</span>
<span class="n">fp</span> <span class="o">=</span> <span class="n">NFSFile</span><span class="p">(</span><span class="n">tempdir</span><span class="p">,</span> <span class="n">local_file</span><span class="p">)</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">mimetypes</span><span class="o">.</span><span class="n">guess_type</span><span class="p">(</span><span class="n">file_name</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">or</span> <span class="s">'application/octet-stream'</span>
<span class="n">stats</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">stat</span><span class="p">(</span><span class="n">local_file</span><span class="p">)</span>
<span class="n">size</span> <span class="o">=</span> <span class="n">stats</span><span class="o">.</span><span class="n">st_size</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span> <span class="s">'Content-type'</span><span class="p">:</span> <span class="n">content_type</span><span class="p">,</span>
<span class="s">'Content-length'</span><span class="p">:</span> <span class="n">size</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">urllib</span><span class="o">.</span><span class="n">addinfourl</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">req</span><span class="o">.</span><span class="n">get_full_url</span><span class="p">())</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">tempdir</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">mkdtemp</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="c"># Populate the temporary file for the simulation</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tempdir</span><span class="p">,</span> <span class="s">'file.txt'</span><span class="p">),</span> <span class="s">'wt'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">'Contents of file.txt'</span><span class="p">)</span>
<span class="c"># Construct an opener with our NFS handler</span>
<span class="c"># and register it as the default opener.</span>
<span class="n">opener</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">build_opener</span><span class="p">(</span><span class="n">FauxNFSHandler</span><span class="p">(</span><span class="n">tempdir</span><span class="p">))</span>
<span class="n">urllib2</span><span class="o">.</span><span class="n">install_opener</span><span class="p">(</span><span class="n">opener</span><span class="p">)</span>
<span class="c"># Open the file through a URL.</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="s">'nfs://remote_server/path/to/the/file.txt'</span><span class="p">)</span>
<span class="k">print</span>
<span class="k">print</span> <span class="s">'READ CONTENTS:'</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'URL :'</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">geturl</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'HEADERS:'</span>
<span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">info</span><span class="p">()</span><span class="o">.</span><span class="n">items</span><span class="p">()):</span>
<span class="k">print</span> <span class="s">' </span><span class="si">%-15s</span><span class="s"> = </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
<span class="n">response</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">os</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tempdir</span><span class="p">,</span> <span class="s">'file.txt'</span><span class="p">))</span>
<span class="n">os</span><span class="o">.</span><span class="n">removedirs</span><span class="p">(</span><span class="n">tempdir</span><span class="p">)</span>
</pre></div>
</div>
<p>The <tt class="xref py py-class docutils literal"><span class="pre">FauxNFSHandler</span></tt> and <tt class="xref py py-class docutils literal"><span class="pre">NFSFile</span></tt> classes print
messages to illustrate where a real implementation would add mount and
unmount calls. Since this is just a simulation,
<tt class="xref py py-class docutils literal"><span class="pre">FauxNFSHandler</span></tt> is primed with the name of a temporary
directory where it should look for all of its files.</p>
<div class="highlight-python"><pre>$ python urllib2_nfs_handler.py
FauxNFSHandler simulating mount:
Remote path: /path/to/the
Server : remote_server
Local path : /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmppv5Efn
File name : file.txt
READ CONTENTS: Contents of file.txt
URL : nfs://remote_server/path/to/the/file.txt
HEADERS:
Content-length = 20
Content-type = text/plain
NFSFile:
unmounting /var/folders/9R/9R1t+tR02Raxzk+F71Q50U+++Uw/-Tmp-/tmppv5Efn
when file.txt is closed</pre>
</div>
<div class="admonition-see-also admonition seealso">
<p class="first admonition-title">See also</p>
<dl class="last docutils">
<dt><a class="reference external" href="http://docs.python.org/library/urllib2.html">urllib2</a></dt>
<dd>The standard library documentation for this module.</dd>
<dt><a class="reference internal" href="../urllib/index.html#module-urllib" title="urllib: Accessing remote resources that don't need authentication, cookies, etc."><tt class="xref py py-mod docutils literal"><span class="pre">urllib</span></tt></a></dt>
<dd>Original URL handling library.</dd>
<dt><a class="reference internal" href="../urlparse/index.html#module-urlparse" title="urlparse: Split URL into component pieces."><tt class="xref py py-mod docutils literal"><span class="pre">urlparse</span></tt></a></dt>
<dd>Work with the URL string itself.</dd>
<dt><a class="reference external" href="http://www.voidspace.org.uk/python/articles/urllib2.shtml">urllib2 – The Missing Manual</a></dt>
<dd>Michael Foord’s write-up on using urllib2.</dd>
<dt><a class="reference external" href="http://www.voidspace.org.uk/python/cgi.shtml#upload">Upload Scripts</a></dt>
<dd>Example scripts from Michael Foord that illustrate how to upload a file
using HTTP and then receive the data on the server.</dd>
<dt><a class="reference external" href="http://code.activestate.com/recipes/146306/">HTTP client to POST using multipart/form-data</a></dt>
<dd>Python cookbook recipe showing how to encode and post data, including files,
over HTTP.</dd>
<dt><a class="reference external" href="http://www.w3.org/TR/REC-html40/interact/forms.html#h-17.13.4">Form content types</a></dt>
<dd>W3C specification for posting files or large amounts of data via HTTP forms.</dd>
<dt><tt class="xref py py-mod docutils literal"><span class="pre">mimetypes</span></tt></dt>
<dd>Map filenames to mimetype.</dd>
<dt><tt class="xref py py-mod docutils literal"><span class="pre">mimetools</span></tt></dt>
<dd>Tools for parsing MIME messages.</dd>
</dl>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="../urlparse/index.html" title="urlparse – Split URL into component pieces."
>next</a> |</li>
<li class="right" >
<a href="../urllib/index.html" title="urllib – simple interface for network resource access"
>previous</a> |</li>
<li><a href="../contents.html">PyMOTW</a> »</li>
<li><a href="../internet_protocols.html" >Internet Protocols and Support</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright Doug Hellmann.
Last updated on Oct 24, 2010.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a>.
<br/><a href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/" rel="license"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-nc-sa/3.0/us/88x31.png"/></a>
</div>
</body>
</html>