<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>multiprocessing Basics — Python Module of the Week</title>
<link rel="stylesheet" href="../_static/sphinxdoc.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '1.132',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="author" title="About these documents" href="../about.html" />
<link rel="top" title="Python Module of the Week" href="../index.html" />
<link rel="up" title="multiprocessing – Manage processes like threads" href="index.html" />
<link rel="next" title="Communication Between Processes" href="communication.html" />
<link rel="prev" title="multiprocessing – Manage processes like threads" href="index.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="communication.html" title="Communication Between Processes"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="index.html" title="multiprocessing – Manage processes like threads"
accesskey="P">previous</a> |</li>
<li><a href="../contents.html">PyMOTW</a> »</li>
<li><a href="../optional_os.html" >Optional Operating System Services</a> »</li>
<li><a href="index.html" accesskey="U">multiprocessing – Manage processes like threads</a> »</li>
</ul>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="../contents.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">multiprocessing Basics</a><ul>
<li><a class="reference internal" href="#importable-target-functions">Importable Target Functions</a></li>
<li><a class="reference internal" href="#determining-the-current-process">Determining the Current Process</a></li>
<li><a class="reference internal" href="#daemon-processes">Daemon Processes</a></li>
<li><a class="reference internal" href="#waiting-for-processes">Waiting for Processes</a></li>
<li><a class="reference internal" href="#terminating-processes">Terminating Processes</a></li>
<li><a class="reference internal" href="#process-exit-status">Process Exit Status</a></li>
<li><a class="reference internal" href="#logging">Logging</a></li>
<li><a class="reference internal" href="#subclassing-process">Subclassing Process</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="index.html"
title="previous chapter">multiprocessing – Manage processes like threads</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="communication.html"
title="next chapter">Communication Between Processes</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="../_sources/multiprocessing/basics.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="../search.html" method="get">
<input type="text" name="q" size="18" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="multiprocessing-basics">
<h1>multiprocessing Basics<a class="headerlink" href="#multiprocessing-basics" title="Permalink to this headline">¶</a></h1>
<p>The simplest way to spawn a second is to instantiate a
<tt class="xref py py-class docutils literal"><span class="pre">Process</span></tt> object with a target function and call <tt class="xref py py-func docutils literal"><span class="pre">start()</span></tt>
to let it begin working.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">():</span>
<span class="sd">"""worker function"""</span>
<span class="k">print</span> <span class="s">'Worker'</span>
<span class="k">return</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">)</span>
<span class="n">jobs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<p>The output includes the word “Worker” printed five times, although it
may not be entirely clean depending on the order of execution.</p>
<div class="highlight-python"><pre>$ python multiprocessing_simple.py
Worker
Worker
Worker
Worker
Worker</pre>
</div>
<p>It usually more useful to be able to spawn a process with arguments to
tell it what work to do. Unlike with <a class="reference internal" href="../threading/index.html#module-threading" title="threading: Manage several concurrent threads of execution."><tt class="xref py py-mod docutils literal"><span class="pre">threading</span></tt></a>, to pass
arguments to a <a class="reference internal" href="index.html#module-multiprocessing" title="multiprocessing: Manage processes like threads."><tt class="xref py py-mod docutils literal"><span class="pre">multiprocessing</span></tt></a> <tt class="xref py py-class docutils literal"><span class="pre">Process</span></tt> the argument
must be able to be serialized using <a class="reference internal" href="../pickle/index.html#module-pickle" title="pickle: Python object serialization"><tt class="xref py py-mod docutils literal"><span class="pre">pickle</span></tt></a>. This example
passes each worker a number so the output is a little more interesting.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">(</span><span class="n">num</span><span class="p">):</span>
<span class="sd">"""thread worker function"""</span>
<span class="k">print</span> <span class="s">'Worker:'</span><span class="p">,</span> <span class="n">num</span>
<span class="k">return</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">i</span><span class="p">,))</span>
<span class="n">jobs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<p>The integer argument is now included in the message printed by each worker:</p>
<div class="highlight-python"><pre>$ python multiprocessing_simpleargs.py
Worker: 0
Worker: 1
Worker: 2
Worker: 4
Worker: 3</pre>
</div>
<div class="section" id="importable-target-functions">
<h2>Importable Target Functions<a class="headerlink" href="#importable-target-functions" title="Permalink to this headline">¶</a></h2>
<p>One difference between the <a class="reference internal" href="../threading/index.html#module-threading" title="threading: Manage several concurrent threads of execution."><tt class="xref py py-mod docutils literal"><span class="pre">threading</span></tt></a> and <a class="reference internal" href="index.html#module-multiprocessing" title="multiprocessing: Manage processes like threads."><tt class="xref py py-mod docutils literal"><span class="pre">multiprocessing</span></tt></a>
examples is the extra protection for <tt class="docutils literal"><span class="pre">__main__</span></tt> used in the
<a class="reference internal" href="index.html#module-multiprocessing" title="multiprocessing: Manage processes like threads."><tt class="xref py py-mod docutils literal"><span class="pre">multiprocessing</span></tt></a> examples. Due to the way the new processes are
started, the child process needs to be able to import the script
containing the target function. Wrapping the main part of the
application in a check for <tt class="docutils literal"><span class="pre">__main__</span></tt> ensures that it is not run
recursively in each child as the module is imported. Another approach
is to import the target function from a separate script.</p>
<p>For example, this main program:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">multiprocessing_import_worker</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">multiprocessing_import_worker</span><span class="o">.</span><span class="n">worker</span><span class="p">)</span>
<span class="n">jobs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<p>uses this worker function, defined in a separate module:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">worker</span><span class="p">():</span>
<span class="sd">"""worker function"""</span>
<span class="k">print</span> <span class="s">'Worker'</span>
<span class="k">return</span>
</pre></div>
</div>
<p>and produces output like the first example above:</p>
<div class="highlight-python"><pre>$ python multiprocessing_import_main.py
Worker
Worker
Worker
Worker
Worker</pre>
</div>
</div>
<div class="section" id="determining-the-current-process">
<h2>Determining the Current Process<a class="headerlink" href="#determining-the-current-process" title="Permalink to this headline">¶</a></h2>
<p>Passing arguments to identify or name the process is cumbersome, and
unnecessary. Each <tt class="xref py py-class docutils literal"><span class="pre">Process</span></tt> instance has a name with a default
value that can be changed as the process is created. Naming processes
is useful for keeping track of them, especially in applications with
multiple types of processes running simultaneously.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">():</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="n">name</span><span class="p">,</span> <span class="s">'Starting'</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="n">name</span><span class="p">,</span> <span class="s">'Exiting'</span>
<span class="k">def</span> <span class="nf">my_service</span><span class="p">():</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="n">name</span><span class="p">,</span> <span class="s">'Starting'</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="k">print</span> <span class="n">name</span><span class="p">,</span> <span class="s">'Exiting'</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">service</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'my_service'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">my_service</span><span class="p">)</span>
<span class="n">worker_1</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'worker 1'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">)</span>
<span class="n">worker_2</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">)</span> <span class="c"># use default name</span>
<span class="n">worker_1</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">worker_2</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">service</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<p>The debug output includes the name of the current process on each
line. The lines with <tt class="docutils literal"><span class="pre">Process-3</span></tt> in the name column correspond to
the unnamed process <tt class="docutils literal"><span class="pre">worker_1</span></tt>.</p>
<div class="highlight-python"><pre>$ python multiprocessing_names.py
worker 1 Starting
worker 1 Exiting
Process-3 Starting
Process-3 Exiting
my_service Starting
my_service Exiting</pre>
</div>
</div>
<div class="section" id="daemon-processes">
<h2>Daemon Processes<a class="headerlink" href="#daemon-processes" title="Permalink to this headline">¶</a></h2>
<p>By default the main program will not exit until all of the children
have exited. There are times when starting a background process that
runs without blocking the main program from exiting is useful, such as
in services where there may not be an easy way to interrupt the
worker, or where letting it die in the middle of its work does not
lose or corrupt data (for example, a task that generates “heart beats”
for a service monitoring tool).</p>
<p>To mark a process as a daemon, set its <tt class="xref py py-attr docutils literal"><span class="pre">daemon</span></tt> attribute with a
boolean value. The default is for processes to not be daemons, so
passing True turns the daemon mode on.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">daemon</span><span class="p">():</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'Starting:'</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">pid</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Exiting :'</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">pid</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">non_daemon</span><span class="p">():</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'Starting:'</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">pid</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'Exiting :'</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">pid</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'daemon'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">daemon</span><span class="p">)</span>
<span class="n">d</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'non-daemon'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">non_daemon</span><span class="p">)</span>
<span class="n">n</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">d</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">n</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<p>The output does not include the “Exiting” message from the daemon
process, since all of the non-daemon processes (including the main
program) exit before the daemon process wakes up from its 2 second
sleep.</p>
<div class="highlight-python"><pre>$ python multiprocessing_daemon.py
Starting: daemon 13952
Starting: non-daemon 13953
Exiting : non-daemon 13953</pre>
</div>
<p>The daemon process is terminated automatically before the main program
exits, to avoid leaving orphaned processes running. You can verify
this by looking for the process id value printed when you run the
program, and then checking for that process with a command like
<tt class="docutils literal"><span class="pre">ps</span></tt>.</p>
</div>
<div class="section" id="waiting-for-processes">
<h2>Waiting for Processes<a class="headerlink" href="#waiting-for-processes" title="Permalink to this headline">¶</a></h2>
<p>To wait until a process has completed its work and exited, use the
<tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> method.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">daemon</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting:'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Exiting :'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">def</span> <span class="nf">non_daemon</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting:'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="s">'Exiting :'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'daemon'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">daemon</span><span class="p">)</span>
<span class="n">d</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'non-daemon'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">non_daemon</span><span class="p">)</span>
<span class="n">n</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">d</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">n</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">d</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="n">n</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>Since the main process waits for the daemon to exit using
<tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt>, the “Exiting” message is printed this time.</p>
<div class="highlight-python"><pre>$ python multiprocessing_daemon_join.py
Starting: non-daemon
Exiting : non-daemon
Starting: daemon
Exiting : daemon</pre>
</div>
<p>By default, <tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> blocks indefinitely. It is also possible to
pass a timeout argument (a float representing the number of seconds to
wait for the process to become inactive). If the process does not
complete within the timeout period, <tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> returns anyway.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">daemon</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting:'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Exiting :'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">def</span> <span class="nf">non_daemon</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting:'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">print</span> <span class="s">'Exiting :'</span><span class="p">,</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'daemon'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">daemon</span><span class="p">)</span>
<span class="n">d</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s">'non-daemon'</span><span class="p">,</span> <span class="n">target</span><span class="o">=</span><span class="n">non_daemon</span><span class="p">)</span>
<span class="n">n</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">False</span>
<span class="n">d</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">n</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">d</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'d.is_alive()'</span><span class="p">,</span> <span class="n">d</span><span class="o">.</span><span class="n">is_alive</span><span class="p">()</span>
<span class="n">n</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>Since the timeout passed is less than the amount of time the daemon
sleeps, the process is still “alive” after <tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> returns.</p>
<div class="highlight-python"><pre>$ python multiprocessing_daemon_join_timeout.py
Starting: non-daemon
Exiting : non-daemon
d.is_alive() True</pre>
</div>
</div>
<div class="section" id="terminating-processes">
<h2>Terminating Processes<a class="headerlink" href="#terminating-processes" title="Permalink to this headline">¶</a></h2>
<p>Although it is better to use the <em>poison pill</em> method of signaling to
a process that it should exit (see <a class="reference internal" href="communication.html#multiprocessing-queues"><em>Passing Messages to Processes</em></a>), if
a process appears hung or deadlocked it can be useful to be able to
kill it forcibly. Calling <tt class="xref py py-func docutils literal"><span class="pre">terminate()</span></tt> on a process object kills
the child process.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">slow_worker</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Starting worker'</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'Finished worker'</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">slow_worker</span><span class="p">)</span>
<span class="k">print</span> <span class="s">'BEFORE:'</span><span class="p">,</span> <span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'DURING:'</span><span class="p">,</span> <span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'TERMINATED:'</span><span class="p">,</span> <span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'JOINED:'</span><span class="p">,</span> <span class="n">p</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">is_alive</span><span class="p">()</span>
</pre></div>
</div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">It is important to <tt class="xref py py-func docutils literal"><span class="pre">join()</span></tt> the process after terminating it
in order to give the background machinery time to update the
status of the object to reflect the termination.</p>
</div>
<div class="highlight-python"><pre>$ python multiprocessing_terminate.py
BEFORE: <Process(Process-1, initial)> False
DURING: <Process(Process-1, started)> True
TERMINATED: <Process(Process-1, stopped[SIGTERM])> False
JOINED: <Process(Process-1, stopped[SIGTERM])> False</pre>
</div>
</div>
<div class="section" id="process-exit-status">
<h2>Process Exit Status<a class="headerlink" href="#process-exit-status" title="Permalink to this headline">¶</a></h2>
<p>The status code produced when the process exits can be accessed via
the <tt class="xref py py-attr docutils literal"><span class="pre">exitcode</span></tt> attribute.</p>
<p>For <tt class="xref py py-attr docutils literal"><span class="pre">exitcode</span></tt> values</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">==</span> <span class="pre">0</span></tt> – no error was produced</li>
<li><tt class="docutils literal"><span class="pre">></span> <span class="pre">0</span></tt> – the process had an error, and exited with that code</li>
<li><tt class="docutils literal"><span class="pre"><</span> <span class="pre">0</span></tt> – the process was killed with a signal of <tt class="docutils literal"><span class="pre">-1</span> <span class="pre">*</span> <span class="pre">exitcode</span></tt></li>
</ul>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">def</span> <span class="nf">exit_error</span><span class="p">():</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">exit_ok</span><span class="p">():</span>
<span class="k">return</span>
<span class="k">def</span> <span class="nf">return_value</span><span class="p">():</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="k">def</span> <span class="nf">raises</span><span class="p">():</span>
<span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s">'There was an error!'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">terminated</span><span class="p">():</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="p">[</span><span class="n">exit_error</span><span class="p">,</span> <span class="n">exit_ok</span><span class="p">,</span> <span class="n">return_value</span><span class="p">,</span> <span class="n">raises</span><span class="p">,</span> <span class="n">terminated</span><span class="p">]:</span>
<span class="k">print</span> <span class="s">'Starting process for'</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">func_name</span>
<span class="n">j</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">f</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="n">f</span><span class="o">.</span><span class="n">func_name</span><span class="p">)</span>
<span class="n">jobs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">j</span><span class="p">)</span>
<span class="n">j</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">jobs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span>
<span class="n">j</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">print</span> <span class="s">'</span><span class="si">%s</span><span class="s">.exitcode = </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">j</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">j</span><span class="o">.</span><span class="n">exitcode</span><span class="p">)</span>
</pre></div>
</div>
<p>Processes that raise an exception automatically get an
<tt class="xref py py-attr docutils literal"><span class="pre">exitcode</span></tt> of 1.</p>
<div class="highlight-python"><pre>$ python multiprocessing_exitcode.py
Starting process for exit_error
Starting process for exit_ok
Starting process for return_value
Starting process for raises
Starting process for terminated
Process raises:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python
2.7/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python
2.7/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "multiprocessing_exitcode.py", line 24, in raises
raise RuntimeError('There was an error!')
RuntimeError: There was an error!
exit_error.exitcode = 1
exit_ok.exitcode = 0
return_value.exitcode = 0
raises.exitcode = 1
terminated.exitcode = -15</pre>
</div>
</div>
<div class="section" id="logging">
<h2>Logging<a class="headerlink" href="#logging" title="Permalink to this headline">¶</a></h2>
<p>When debugging concurrency issues, it can be useful to have access to
the internals of the objects provided by <a class="reference internal" href="index.html#module-multiprocessing" title="multiprocessing: Manage processes like threads."><tt class="xref py py-mod docutils literal"><span class="pre">multiprocessing</span></tt></a>.
There is a convenient module-level function to enable logging called
<tt class="xref py py-func docutils literal"><span class="pre">log_to_stderr()</span></tt>. It sets up a logger object using
<a class="reference internal" href="../logging/index.html#module-logging" title="logging: Report status, error, and informational messages."><tt class="xref py py-mod docutils literal"><span class="pre">logging</span></tt></a> and adds a handler so that log messages are sent to the
standard error channel.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Doing some work'</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">log_to_stderr</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">)</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>By default the logging level is set to <tt class="docutils literal"><span class="pre">NOTSET</span></tt> so no messages are
produced. Pass a different level to initialize the logger to the
level of detail you want.</p>
<div class="highlight-python"><pre>$ python multiprocessing_log_to_stderr.py
[INFO/Process-1] child process calling self.run()
Doing some work
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers</pre>
</div>
<p>To manipulate the logger directly (change its level setting or add
handlers), use <tt class="xref py py-func docutils literal"><span class="pre">get_logger()</span></tt>.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">def</span> <span class="nf">worker</span><span class="p">():</span>
<span class="k">print</span> <span class="s">'Doing some work'</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">multiprocessing</span><span class="o">.</span><span class="n">log_to_stderr</span><span class="p">()</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">get_logger</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">worker</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">p</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>The logger can also be configured through the <a class="reference internal" href="../logging/index.html#module-logging" title="logging: Report status, error, and informational messages."><tt class="xref py py-mod docutils literal"><span class="pre">logging</span></tt></a>
configuration file API, using the name <tt class="docutils literal"><span class="pre">multiprocessing</span></tt>.</p>
<div class="highlight-python"><pre>$ python multiprocessing_get_logger.py
[INFO/Process-1] child process calling self.run()
Doing some work
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/MainProcess] process shutting down</pre>
</div>
</div>
<div class="section" id="subclassing-process">
<h2>Subclassing Process<a class="headerlink" href="#subclassing-process" title="Permalink to this headline">¶</a></h2>
<p>Although the simplest way to start a job in a separate process is to
use <tt class="xref py py-class docutils literal"><span class="pre">Process</span></tt> and pass a target function, it is also possible
to use a custom subclass.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="k">class</span> <span class="nc">Worker</span><span class="p">(</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">print</span> <span class="s">'In </span><span class="si">%s</span><span class="s">'</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">name</span>
<span class="k">return</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">jobs</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">Worker</span><span class="p">()</span>
<span class="n">jobs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="n">jobs</span><span class="p">:</span>
<span class="n">j</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
</pre></div>
</div>
<p>The derived class should override <tt class="xref py py-meth docutils literal"><span class="pre">run()</span></tt> to do its work.</p>
<div class="highlight-python"><pre>$ python multiprocessing_subclass.py
In Worker-1
In Worker-2
In Worker-3
In Worker-5
In Worker-4</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="../genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="../py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="communication.html" title="Communication Between Processes"
>next</a> |</li>
<li class="right" >
<a href="index.html" title="multiprocessing – Manage processes like threads"
>previous</a> |</li>
<li><a href="../contents.html">PyMOTW</a> »</li>
<li><a href="../optional_os.html" >Optional Operating System Services</a> »</li>
<li><a href="index.html" >multiprocessing – Manage processes like threads</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright Doug Hellmann.
Last updated on Oct 24, 2010.
Created using <a href="http://sphinx.pocoo.org/">Sphinx</a>.
<br/><a href="http://creativecommons.org/licenses/by-nc-sa/3.0/us/" rel="license"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-nc-sa/3.0/us/88x31.png"/></a>
</div>
</body>
</html>