<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Communication Between Processes — Python Module of the Week</title>
<link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '1.132',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="author" title="About these documents" href="../about.html" />
<link rel="top" title="Python Module of the Week" href="../index.html" />
<link rel="up" title="multiprocessing – Manage processes like threads" href="index.html" />
<link rel="next" title="Implementing MapReduce with multiprocessing" href="mapreduce.html" />
<link rel="prev" title="multiprocessing Basics" href="basics.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="mapreduce.html" title="Implementing MapReduce with multiprocessing"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="basics.html" title="multiprocessing Basics"
accesskey="P">previous</a> |</li>
<li><a href="../contents.html">PyMOTW</a> »</li>
<li><a href="../optional_os.html" >Optional Operating System Services</a> »</li>
<li><a href="index.html" accesskey="U">multiprocessing – Manage processes like threads</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="../contents.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Communication Between Processes</a><ul>
<li><a class="reference internal" href="#passing-messages-to-processes">Passing Messages to Processes</a></li>
<li><a class="reference internal" href="#signaling-between-processes">Signaling between Processes</a></li>
<li><a class="reference internal" href="#controlling-access-to-resources">Controlling Access to Resources</a></li>
<li><a class="reference internal" href="#synchronizing-operations">Synchronizing Operations</a></li>
<li><a class="reference internal" href="#controlling-concurrent-access-to-resources">Controlling Concurrent Access to Resources</a></li>
<li><a class="reference internal" href="#managing-shared-state">Managing Shared State</a></li>
<li><a class="reference internal" href="#shared-namespaces">Shared Namespaces</a></li>
<li><a class="reference internal" href="#process-pools">Process Pools</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="basics.html"
title="previous chapter">multiprocessing Basics</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="mapreduce.html"
title="next chapter">Implementing MapReduce with multiprocessing</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/multiprocessing/communication.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="communication-between-processes">
<h1>Communication Between Processes<a class="headerlink" href="#communication-between-processes" title="Permalink to this headline">¶</a></h1>
<p>As with threads, a common use pattern for multiple processes is to
divide a job up among several workers to run in parallel. Effective
use of multiple processes usually requires some communication between
them, so that work can be divided and results can be aggregated.</p>
<div class="section" id="passing-messages-to-processes">
<span id="multiprocessing-queues"></span><h2>Passing Messages to Processes<a class="headerlink" href="#passing-messages-to-processes" title="Permalink to this headline">¶</a></h2>
<p>A simple way to communicate between process with
<a class="reference internal" href="index.html#module-multiprocessing" title="multiprocessing: Manage processes like threads."><tt class="xref py py-mod docutils literal"><span class="pre">multiprocessing</span></tt></a> is to use a <a class="reference internal" href="../Queue/index.html#module-Queue" title="Queue: Provides a thread-safe FIFO implementation"><tt class="xref py py-class docutils literal"><span class="pre">Queue</span></tt></a> to pass messages
back and forth. Any pickle-able object can pass through a
<a class="reference internal" href="../Queue/index.html#module-Queue" title="Queue: Provides a thread-safe FIFO implementation"><tt class="xref py py-class docutils literal"><span class="pre">Queue</span></tt></a>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">class</span> <span class="nc">MyFancyClass</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="k">def</span> <span class="nf">do_something</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">proc_name</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="s">'Doing something fancy in </span><span class="si">%s</span><span class="s"> for </span><span class="si">%s</span><span class="s">!'</span> <span class="o">%</span> <span class="p">(</span><span class="n">proc_name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">(</span><span class="n">q</span><span class="p">):</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="n">obj</span><span class="o">.</span><span class="n">do_something</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">queue</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">queue</span><span class="p">,))</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">MyFancyClass</span><span class="p">(</span><span class="s">'Fancy Dan'</span><span class="p">))</span>
<span class="c"># Wait for the worker to finish</span>
<span class="n">queue</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">queue</span><span class="o">.</span><span class="n">join_thread</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>This short example only passes a single message to a single worker,
then the main process waits for the worker to finish.</p>
<div class="highlight-python"><pre>$ python multiprocessing_queue.py
Doing something fancy in Process-1 for Fancy Dan!</pre>
</div>
<p>A more complex example shows how to manage several workers consuming
data from a <tt class="xref py py-class docutils literal"><span class="pre">JoinableQueue</span></tt> and passing results back to the
parent process. The <em>poison pill</em> technique is used to stop the
workers. After setting up the real tasks, the main program adds one
“stop” value per worker to the job queue. When a worker encounters
the special value, it breaks out of its processing loop. The main
process uses the task queue’s <tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> method to wait for all of
the tasks to finish before processin the results.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">class</span> <span class="nc">Consumer</span><span class="p">(</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_queue</span><span class="p">,</span> <span class="n">result_queue</span><span class="p">):</span>
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span> <span class="o">=</span> <span class="n">task_queue</span>
<span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span> <span class="o">=</span> <span class="n">result_queue</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">proc_name</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">next_task</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">if</span> <span class="n">next_task</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="c"># Poison pill means shutdown</span>
<span class="k">print</span> <span class="s">'</span><span class="si">%s</span><span class="s">: Exiting'</span> <span class="o">%</span> <span class="n">proc_name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
<span class="k">break</span>
<span class="k">print</span> <span class="s">'</span><span class="si">%s</span><span class="s">: </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">proc_name</span><span class="p">,</span> <span class="n">next_task</span><span class="p">)</span>
<span class="n">answer</span> <span class="o">=</span> <span class="n">next_task</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">answer</span><span class="p">)</span>
<span class="k">return</span>
<span class="k">class</span> <span class="nc">Task</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">a</span> <span class="o">=</span> <span class="n">a</span>
<span class="bp">self</span><span class="o">.</span><span class="n">b</span> <span class="o">=</span> <span class="n">b</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span> <span class="c"># pretend to take some time to do the work</span>
<span class="k">return</span> <span class="s">'</span><span class="si">%s</span><span class="s"> * </span><span class="si">%s</span><span class="s"> = </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">a</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">a</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'</span><span class="si">%s</span><span class="s"> * </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">a</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">b</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="c"># Establish communication queues</span>
<span class="n">tasks</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">JoinableQueue</span><span class="p">()</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
<span class="c"># Start consumers</span>
<span class="n">num_consumers</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">cpu_count</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span>
<span class="k">print</span> <span class="s">'Creating </span><span class="si">%d</span><span class="s"> consumers'</span> <span class="o">%</span> <span class="n">num_consumers</span>
<span class="n">consumers</span> <span class="o">=</span> <span class="p">[</span> <span class="n">Consumer</span><span class="p">(</span><span class="n">tasks</span><span class="p">,</span> <span class="n">results</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_consumers</span><span class="p">)</span> <span class="p">]</span>
<span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">consumers</span><span class="p">:</span>
<span class="n">w</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="c"># Enqueue jobs</span>
<span class="n">num_jobs</span> <span class="o">=</span> <span class="mi">10</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_jobs</span><span class="p">):</span>
<span class="n">tasks</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">Task</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="p">))</span>
<span class="c"># Add a poison pill for each consumer</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="n">num_consumers</span><span class="p">):</span>
<span class="n">tasks</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="bp">None</span><span class="p">)</span>
<span class="c"># Wait for all of the tasks to finish</span>
<span class="n">tasks</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="c"># Start printing results</span>
<span class="k">while</span> <span class="n">num_jobs</span><span class="p">:</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">results</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'Result:'</span><span class="p">,</span> <span class="n">result</span>
<span class="n">num_jobs</span> <span class="o">-=</span> <span class="mi">1</span>
</pre></div>
</div>
<p>Although the jobs enter the queue in order, since their execution is
parallelized there is no guarantee about the order they will be
completed.</p>
<div class="highlight-python"><pre>$ python -u multiprocessing_producer_consumer.py
Creating 4 consumers
Consumer-1: 0 * 0
Consumer-2: 1 * 1
Consumer-4: 2 * 2
Consumer-3: 3 * 3
Consumer-1: 4 * 4
Consumer-2: 5 * 5
Consumer-4: 6 * 6
Consumer-3: 7 * 7
Consumer-1: 8 * 8
Consumer-2: 9 * 9
Consumer-4: Exiting
Consumer-3: Exiting
Consumer-1: Exiting
Consumer-2: Exiting
Result: 0 * 0 = 0
Result: 1 * 1 = 1
Result: 2 * 2 = 4
Result: 3 * 3 = 9
Result: 4 * 4 = 16
Result: 5 * 5 = 25
Result: 6 * 6 = 36
Result: 7 * 7 = 49
Result: 8 * 8 = 64
Result: 9 * 9 = 81</pre>
</div>
</div>
<div class="section" id="signaling-between-processes">
<h2>Signaling between Processes<a class="headerlink" href="#signaling-between-processes" title="Permalink to this headline">¶</a></h2>
<p>The <tt class="xref py py-class docutils literal"><span class="pre">Event</span></tt> class provides a simple way to communicate state
information between processes. An event can be toggled between set
and unset states. Users of the event object can wait for it to change
from unset to set, using an optional timeout value.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">wait_for_event</span><span class="p">(</span><span class="n">e</span><span class="p">):</span>
<span class="sd">"""Wait for the event to be set before doing anything"""</span>
<span class="k">print</span> <span class="s">'wait_for_event: starting'</span>
<span class="n">e</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'wait_for_event: e.is_set()->'</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">is_set</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">wait_for_event_timeout</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="n">t</span><span class="p">):</span>
<span class="sd">"""Wait t seconds and then timeout"""</span>
<span class="k">print</span> <span class="s">'wait_for_event_timeout: starting'</span>
<span class="n">e</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'wait_for_event_timeout: e.is_set()->'</span><span class="p">,</span> <span class="n">e</span><span class="o">.</span><span class="n">is_set</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">e</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Event</span><span class="p">()</span>
<span class="n">w1</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'block'</span><span class="p">,</span>
<span class="n">target</span><span class="o">=</span><span class="n">wait_for_event</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">e</span><span class="p">,))</span>
<span class="n">w1</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">w2</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'non-block'</span><span class="p">,</span>
<span class="n">target</span><span class="o">=</span><span class="n">wait_for_event_timeout</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="mi">2</span><span class="p">))</span>
<span class="n">w2</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'main: waiting before calling Event.set()'</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="n">e</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'main: event is set'</span>
</pre></div>
</div>
<p>When <tt class="xref py py-func docutils literal"><span class="pre">wait()</span></tt> times out it returns without an error. The caller
is responsible for checking the state of the event using
<tt class="xref py py-func docutils literal"><span class="pre">is_set()</span></tt>.</p>
<div class="highlight-python"><pre>$ python -u multiprocessing_event.py
main: waiting before calling Event.set()
wait_for_event: starting
wait_for_event_timeout: starting
wait_for_event_timeout: e.is_set()-> False
main: event is set
wait_for_event: e.is_set()-> True</pre>
</div>
</div>
<div class="section" id="controlling-access-to-resources">
<h2>Controlling Access to Resources<a class="headerlink" href="#controlling-access-to-resources" title="Permalink to this headline">¶</a></h2>
<p>In situations when a single resource needs to be shared between
multiple processes, a <tt class="xref py py-class docutils literal"><span class="pre">Lock</span></tt> can be used to avoid conflicting
accesses.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">worker_with</span><span class="p">(</span><span class="n">lock</span><span class="p">,</span> <span class="n">stream</span><span class="p">):</span>
<span class="k">with</span> <span class="n">lock</span><span class="p">:</span>
<span class="n">stream</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">'Lock acquired via with</span><span class="se">\n</span><span class="s">'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">worker_no_with</span><span class="p">(</span><span class="n">lock</span><span class="p">,</span> <span class="n">stream</span><span class="p">):</span>
<span class="n">lock</span><span class="o">.</span><span class="n">acquire</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">stream</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">'Lock acquired directly</span><span class="se">\n</span><span class="s">'</span><span class="p">)</span>
<span class="k">finally</span><span class="p">:</span>
<span class="n">lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="n">lock</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
<span class="n">w</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker_with</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">lock</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">))</span>
<span class="n">nw</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker_no_with</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">lock</span><span class="p">,</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">))</span>
<span class="n">w</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">nw</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">w</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="n">nw</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>In this example, the messages printed to the console may be jumbled
together if the two processes do not synchronize their access of the
output stream with the lock.</p>
<div class="highlight-python"><pre>$ python multiprocessing_lock.py
Lock acquired via with
Lock acquired directly</pre>
</div>
</div>
<div class="section" id="synchronizing-operations">
<h2>Synchronizing Operations<a class="headerlink" href="#synchronizing-operations" title="Permalink to this headline">¶</a></h2>
<p><tt class="xref py py-class docutils literal"><span class="pre">Condition</span></tt> objects can be used to synchronize parts of a
workflow so that some run in parallel but others run sequentially,
even if they are in separate processes.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">stage_1</span><span class="p">(</span><span class="n">cond</span><span class="p">):</span>
<span class="sd">"""perform first stage of work, then notify stage_2 to continue"""</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="s">'Starting'</span><span class="p">,</span> <span class="n">name</span>
<span class="k">with</span> <span class="n">cond</span><span class="p">:</span>
<span class="k">print</span> <span class="s">'</span><span class="si">%s</span><span class="s"> done and ready for stage 2'</span> <span class="o">%</span> <span class="n">name</span>
<span class="n">cond</span><span class="o">.</span><span class="n">notify_all</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">stage_2</span><span class="p">(</span><span class="n">cond</span><span class="p">):</span>
<span class="sd">"""wait for the condition telling us stage_1 is done"""</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="s">'Starting'</span><span class="p">,</span> <span class="n">name</span>
<span class="k">with</span> <span class="n">cond</span><span class="p">:</span>
<span class="n">cond</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'</span><span class="si">%s</span><span class="s"> running'</span> <span class="o">%</span> <span class="n">name</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">condition</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Condition</span><span class="p">()</span>
<span class="n">s1</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'s1'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">stage_1</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">condition</span><span class="p">,))</span>
<span class="n">s2_clients</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'stage_2[</span><span class="si">%d</span><span class="s">]'</span> <span class="o">%</span> <span class="n">i</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">stage_2</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">condition</span><span class="p">,))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s2_clients</span><span class="p">:</span>
<span class="n">c</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">s1</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">s1</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s2_clients</span><span class="p">:</span>
<span class="n">c</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>In this example, two process run the second stage of a job in
parallel, but only after the first stage is done.</p>
<div class="highlight-python"><pre>$ python multiprocessing_condition.py
Starting s1
s1 done and ready for stage 2
Starting stage_2[1]
stage_2[1] running
Starting stage_2[2]
stage_2[2] running</pre>
</div>
</div>
<div class="section" id="controlling-concurrent-access-to-resources">
<h2>Controlling Concurrent Access to Resources<a class="headerlink" href="#controlling-concurrent-access-to-resources" title="Permalink to this headline">¶</a></h2>
<p>Sometimes it is useful to allow more than one worker access to a
resource at a time, while still limiting the overall number. For
example, a connection pool might support a fixed number of
simultaneous connections, or a network application might support a
fixed number of concurrent downloads. A <tt class="xref py py-class docutils literal"><span class="pre">Semaphore</span></tt> is one way
to manage those connections.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">random</span>
<span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">class</span> <span class="nc">ActivePool</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">super</span><span class="p">(</span><span class="n">ActivePool</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">mgr</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Manager</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">active</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">mgr</span><span class="o">.</span><span class="n">list</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">lock</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Lock</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">makeActive</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">lock</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">active</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">makeInactive</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">lock</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">active</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">lock</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">active</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">pool</span><span class="p">):</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">with</span> <span class="n">s</span><span class="p">:</span>
<span class="n">pool</span><span class="o">.</span><span class="n">makeActive</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Now running: </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="nb">str</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">random</span><span class="o">.</span><span class="n">random</span><span class="p">())</span>
<span class="n">pool</span><span class="o">.</span><span class="n">makeInactive</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">pool</span> <span class="o">=</span> <span class="n">ActivePool</span><span class="p">()</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Semaphore</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">i</span><span class="p">),</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">pool</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span>
<span class="n">j</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span>
<span class="n">j</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'Now running: </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="nb">str</span><span class="p">(</span><span class="n">pool</span><span class="p">)</span>
</pre></div>
</div>
<p>In this example, the <tt class="xref py py-class docutils literal"><span class="pre">ActivePool</span></tt> class simply serves as a
convenient way to track which processes are running at a given
moment. A real resource pool would probably allocate a connection or
some other value to the newly active process, and reclaim the value
when the task is done. Here, the pool is just used to hold the names
of the active processes to show that only three are running
concurrently.</p>
<div class="highlight-python"><pre>$ python multiprocessing_semaphore.py
Now running: ['0', '1', '5']
Now running: ['0', '1', '5']
Now running: ['0', '1', '5']
Now running: ['1', '5', '2']
Now running: ['2', '6', '3']
Now running: ['1', '2', '6']
Now running: ['6', '8', '9']
Now running: ['9', '7', '4']
Now running: ['6', '8', '9']
Now running: ['8', '9', '7']
Now running: ['1', '5', '2']
Now running: ['2', '6', '3']
Now running: ['6', '8', '9']
Now running: ['6', '8', '9']
Now running: ['9', '7']
Now running: ['9', '7']
Now running: ['9', '7']
Now running: []
Now running: []
Now running: []</pre>
</div>
</div>
<div class="section" id="managing-shared-state">
<h2>Managing Shared State<a class="headerlink" href="#managing-shared-state" title="Permalink to this headline">¶</a></h2>
<p>In the previous example, the list of active processes is maintained
centrally in the <tt class="xref py py-class docutils literal"><span class="pre">ActivePool</span></tt> instance via a special type of
list object created by a <tt class="xref py py-class docutils literal"><span class="pre">Manager</span></tt>. The <tt class="xref py py-class docutils literal"><span class="pre">Manager</span></tt> is
responsible for coordinating shared information state between all of
its users.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="n">d</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">mgr</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Manager</span><span class="p">()</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">mgr</span><span class="o">.</span><span class="n">dict</span><span class="p">()</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="o">*</span><span class="mi">2</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span>
<span class="n">j</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span>
<span class="n">j</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'Results:'</span><span class="p">,</span> <span class="n">d</span>
</pre></div>
</div>
<p>By creating the list through the manager, it is shared and updates are
seen in all processes. Dictionaries are also supported.</p>
<div class="highlight-python"><pre>$ python multiprocessing_manager_dict.py
Results: {0: 0, 1: 2, 2: 4, 3: 6, 4: 8, 5: 10, 6: 12, 7: 14, 8: 16, 9: 18}</pre>
</div>
</div>
<div class="section" id="shared-namespaces">
<h2>Shared Namespaces<a class="headerlink" href="#shared-namespaces" title="Permalink to this headline">¶</a></h2>
<p>In addition to dictionaries and lists, a <tt class="xref py py-class docutils literal"><span class="pre">Manager</span></tt> can create a
shared <tt class="xref py py-class docutils literal"><span class="pre">Namespace</span></tt>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">ns</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="n">ns</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="s">'This is the value'</span>
<span class="n">event</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">ns</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">ns</span><span class="o">.</span><span class="n">value</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">,</span> <span class="n">err</span><span class="p">:</span>
<span class="k">print</span> <span class="s">'Before event, consumer got:'</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">err</span><span class="p">)</span>
<span class="n">event</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'After event, consumer got:'</span><span class="p">,</span> <span class="n">ns</span><span class="o">.</span><span class="n">value</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">mgr</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Manager</span><span class="p">()</span>
<span class="n">namespace</span> <span class="o">=</span> <span class="n">mgr</span><span class="o">.</span><span class="n">Namespace</span><span class="p">()</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Event</span><span class="p">()</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">producer</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="n">event</span><span class="p">))</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">consumer</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="n">event</span><span class="p">))</span>
<span class="n">c</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">c</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>Any named value added to the <tt class="xref py py-class docutils literal"><span class="pre">Namespace</span></tt> is visible to all of
the clients that receive the <tt class="xref py py-class docutils literal"><span class="pre">Namespace</span></tt> instance.</p>
<div class="highlight-python"><pre>$ python multiprocessing_namespaces.py
Before event, consumer got: 'Namespace' object has no attribute 'value'
After event, consumer got: This is the value</pre>
</div>
<p>It is important to know that <em>updates</em> to the contents of mutable
values in the namespace are <em>not</em> propagated automatically.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">producer</span><span class="p">(</span><span class="n">ns</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="n">ns</span><span class="o">.</span><span class="n">my_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">'This is the value'</span><span class="p">)</span> <span class="c"># DOES NOT UPDATE GLOBAL VALUE!</span>
<span class="n">event</span><span class="o">.</span><span class="n">set</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">consumer</span><span class="p">(</span><span class="n">ns</span><span class="p">,</span> <span class="n">event</span><span class="p">):</span>
<span class="k">print</span> <span class="s">'Before event, consumer got:'</span><span class="p">,</span> <span class="n">ns</span><span class="o">.</span><span class="n">my_list</span>
<span class="n">event</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'After event, consumer got:'</span><span class="p">,</span> <span class="n">ns</span><span class="o">.</span><span class="n">my_list</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">mgr</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Manager</span><span class="p">()</span>
<span class="n">namespace</span> <span class="o">=</span> <span class="n">mgr</span><span class="o">.</span><span class="n">Namespace</span><span class="p">()</span>
<span class="n">namespace</span><span class="o">.</span><span class="n">my_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Event</span><span class="p">()</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">producer</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="n">event</span><span class="p">))</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">consumer</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">namespace</span><span class="p">,</span> <span class="n">event</span><span class="p">))</span>
<span class="n">c</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">c</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>To update the list, attach it to the namespace object again.</p>
<div class="highlight-python"><pre>$ python multiprocessing_namespaces_mutable.py
Before event, consumer got: []
After event, consumer got: []</pre>
</div>
</div>
<div class="section" id="process-pools">
<h2>Process Pools<a class="headerlink" href="#process-pools" title="Permalink to this headline">¶</a></h2>
<p>The <tt class="xref py py-class docutils literal"><span class="pre">Pool</span></tt> class can be used to manage a fixed number of
workers for simple cases where the work to be done can be broken up
and distributed between workers independently. The return values from
the jobs are collected and returned as a list. The pool arguments
include the number of processes and a function to run when starting
the task process (invoked once per child).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">do_calculation</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="k">return</span> <span class="n">data</span> <span class="o">*</span> <span class="mi">2</span>
<span class="k">def</span> <span class="nf">start_process</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">inputs</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
<span class="k">print</span> <span class="s">'Input :'</span><span class="p">,</span> <span class="n">inputs</span>
<span class="n">builtin_outputs</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="n">do_calculation</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Built-in:'</span><span class="p">,</span> <span class="n">builtin_outputs</span>
<span class="n">pool_size</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">cpu_count</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span>
<span class="n">pool</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Pool</span><span class="p">(</span><span class="n">processes</span><span class="o">=</span><span class="n">pool_size</span><span class="p">,</span>
<span class="n">initializer</span><span class="o">=</span><span class="n">start_process</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">pool_outputs</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">do_calculation</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
<span class="n">pool</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="c"># no more tasks</span>
<span class="n">pool</span><span class="o">.</span><span class="n">join</span><span class="p">()</span> <span class="c"># wrap up current tasks</span>
<span class="k">print</span> <span class="s">'Pool :'</span><span class="p">,</span> <span class="n">pool_outputs</span>
</pre></div>
</div>
<p>The result of the <tt class="xref py py-func docutils literal"><span class="pre">map()</span></tt> method is functionally equivalent to the
built-in <tt class="xref py py-func docutils literal"><span class="pre">map()</span></tt>, except that individual tasks run in parallel.
Since the pool is processing its inputs in parallel, <tt class="xref py py-func docutils literal"><span class="pre">close()</span></tt> and
<tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> can be used to synchronize the main process with the task
processes to ensure proper cleanup.</p>
<div class="highlight-python"><pre>$ python multiprocessing_pool.py
Input : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Built-in: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
Starting PoolWorker-3
Starting PoolWorker-4
Starting PoolWorker-1
Starting PoolWorker-2
Pool : [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]</pre>
</div>
<p>By default <tt class="xref py py-class docutils literal"><span class="pre">Pool</span></tt> creates a fixed number of worker processes
and passes jobs to them until there are no more jobs. Setting the
<em>maxtasksperchild</em> parameter tells the pool to restart a worker
process after it has finished a few tasks. This can be used to avoid
having long-running workers consume ever more system resources.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">do_calculation</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="k">return</span> <span class="n">data</span> <span class="o">*</span> <span class="mi">2</span>
<span class="k">def</span> <span class="nf">start_process</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">inputs</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">))</span>
<span class="k">print</span> <span class="s">'Input :'</span><span class="p">,</span> <span class="n">inputs</span>
<span class="n">builtin_outputs</span> <span class="o">=</span> <span class="nb">map</span><span class="p">(</span><span class="n">do_calculation</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Built-in:'</span><span class="p">,</span> <span class="n">builtin_outputs</span>
<span class="n">pool_size</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">cpu_count</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span>
<span class="n">pool</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Pool</span><span class="p">(</span><span class="n">processes</span><span class="o">=</span><span class="n">pool_size</span><span class="p">,</span>
<span class="n">initializer</span><span class="o">=</span><span class="n">start_process</span><span class="p">,</span>
<span class="n">maxtasksperchild</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">pool_outputs</span> <span class="o">=</span> <span class="n">pool</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="n">do_calculation</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
<span class="n">pool</span><span class="o">.</span><span class="n">close</span><span class="p">()</span> <span class="c"># no more tasks</span>
<span class="n">pool</span><span class="o">.</span><span class="n">join</span><span class="p">()</span> <span class="c"># wrap up current tasks</span>
<span class="k">print</span> <span class="s">'Pool :'</span><span class="p">,</span> <span class="n">pool_outputs</span>
</pre></div>
</div>
<p>The pool restarts the workers when they have completed their allotted
tasks, even if there is no more work. In this output, eight workers
are created, even though there are only 10 tasks, and each worker can
complete two of them at a time.</p>
<div class="highlight-python"><pre>$ python multiprocessing_pool_maxtasksperchild.py
Input : [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Built-in: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
Starting PoolWorker-1
Starting PoolWorker-2
Starting PoolWorker-4
Starting PoolWorker-3
Starting PoolWorker-5
Starting PoolWorker-6
Starting PoolWorker-7
Starting PoolWorker-8
Pool : [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="mapreduce.html" title="Implementing MapReduce with multiprocessing"
>next</a> |</li>
<li class="right" >
<a href="basics.html" title="multiprocessing Basics"
>previous</a> |</li>
<li><a href="../contents.html">PyMOTW</a> »</li>
<li><a href="../optional_os.html" >Optional Operating System Services</a> »</li>
<li><a href="index.html" >multiprocessing – Manage processes like threads</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright Doug Hellmann.
Last updated on Oct 24, 2010.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a>.
<br/><a href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/" rel="license"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-nc-sa/3.0/us/88x31.png"/></a>
</div>
</body>
</html>