Calling Python from .NET/C#

In case you want to jump ahead and just see how the code works, the source code (Visual Studio solution, projects, C# code and python scripts) for this post can be found in Github.com at https://github.com/softwarepronto/Blog under the CSharpPythonMaster folder.

This posting will demonstrate how to invoke a Python script using C# and the Process class from the System.Diagnostic namespace. It is assumed Python is already installed on the machine on which a C# application will invoke the python script. An example of the steps required to install Python on Windows can be found at: Python: Installing on Windows. The C# application can communicate directly with the Python script by passing in command-line arguments. The Python script will communicate with the C# executable by writing to standard output. Obviously communication could be handled between C# and Python using files or sockets or dozens of other mechanisms.

C# is able to invoke any scripting language including Python. With regards to Python, running a script simply a matter of invoking (from C#) the python.exe executable and passing in a command-line argument corresponding to the name of the python script to be run. An example Python script, Python101.py, is as follows:

Note that the previous code displays all the arguments passed to Python during execution (sys.argv corresponds to the command-line arguments).

This script can be invoked from any console window provided the Python-related installation folders were added to the PATH variable during installation (see the following for more details: Python: Installing on Windows). The script is simple enough to execute as is shown below:

The general idea to run Python from C# is to include "using System.Diagnostics;" at the top of the C# file (making the items within the System.Diagnostics namespace available). The ProcessStartInfo class is used to specify the executable filename ("python") and the lone command-line argument ("Python101.py") to be invoked by the Process class instance's Start method. An example of the rough code is follows:

    ProcessStartInfo processStartInfo = new ProcessStartInfo()
    {
        Arguments = "Python101.py",
        FileName = "python"
    };

    using (Process process = new Process())
    {
        process.StartInfo = processStartInfo;
        process.Start();
        process.WaitForExit();
        process.Close();
    }

Recall also that we wanted to have C# specify command-line options to be processed by Python so more realistic rough code is as follows:

    private static void RunPythonScript(

                            string script, string scriptArgs)

    {

        ProcessStartInfo processStartInfo = new ProcessStartInfo()

        {

            Arguments = _pythonScriptToExecute + " " + scriptArgs,

            FileName = "python",

        };

        using (Process process = new Process())

        {

            // assign start information to the process 

            process.StartInfo = processStartInfo;

            process.Start();

            process.WaitForExit();

            process.Close();

        }

    }

    const string _pythonScriptToExecute = "Python101.py";

    private static void InvokePythonScript()

    {

        RunPythonScript(_pythonScriptToExecute, "Arg0 Arg1 Arg2");

The previous code handles passing command-line arguments from C# to Python (see the code demarcated by boldface). What is missing in the previous code is to have C# handle the processing of the standard output generated by the Python script.

In order for a C# application to read the standard output stream from Python or any application the ProcessStartInfo class's properties must be set as follows:
    UseShellExecute = false,
    RedirectStandardOutput = true,

Reading standard output adds complexity. There are deadlock scenarios that take place where the parent (the C# application) can wait forever for the child process (the Python script) to write to standard output. The documentation for RedirectStandardOutput (ProcessStartInfo.RedirectStandardOutput Property)states:

There are a dozen questions asked on Stackoverflow.com on the deadlock topic related to reading standard output form a child process spawned with Process.Start. The simplest way to address it is to:

1. Insure that both standard output and standard error area read completely
2. Only read the streams associated with standard output and standard error using asynchronous methods

An example that invokes a Python script, passes in command-line parameters to the script and reads standard output from the script without deadlocking needs a bit of infrastructure. The following events are used to indicate when the asynchronously invoked methods have read standard output and standard error:

private static AutoResetEvent _doneHandlingSTDOUT = 
                                  new AutoResetEvent(false);

private static AutoResetEvent _doneHandlingSTERR = 
                                  new AutoResetEvent(false);

private static AutoResetEvent[] _allToWaitOn = 
    { _doneHandlingSTDOUT, _doneHandlingSTERR };

The following methods are invoke asynchronously to handle the reading of standard error and standard output respectively:

private static void HandleSTDERR(
            object sendingProcess,
            DataReceivedEventArgs stderr)
{
    // Empty stream so done handling standard error stream
    if (String.IsNullOrEmpty(stderr.Data))
    {
        _doneHandlingSTERR.Set();
    }

    else
    {
        Console.Write("There was an error: ");
        Console.WriteLine(stderr.Data);
    }
}

private static void HandleSTDOUT(
            object sendingProcess,
            DataReceivedEventArgs stdout)
{
    // Empty stream so done handling standard output stream
    if (String.IsNullOrEmpty(stdout.Data))
    {
        _doneHandlingSTDOUT.Set();
    }

    else
    {
        Console.WriteLine(stdout.Data);
    }
}

The previous methods receive text form standard error and standard output respectively from the parameter of type DataReceivedEventArgs which exposes the Data property. The Data property contains a value of null when there is no data left in the stream. When Data equals null, each of the previous methods sets an event indicating that processing the stream is complete.

The code that invokes the Python script using Process.Start and that also sets up to asynchronously read of standard output and standard error is as follows:

private static void RunPythonScript(
                         string script, string scriptArgs)
{
    ProcessStartInfo processStartInfo = new ProcessStartInfo()
    {
        Arguments = script + " " + scriptArgs,
        FileName = "python",
        // can only redirect STDIO when UseShellExecute=false
        UseShellExecute = false, 
        RedirectStandardOutput = true,
        RedirectStandardError = true
    };

    using (Process process = new Process())
    {
        process.StartInfo = processStartInfo;
        process.OutputDataReceived += HandleSTDOUT;
        process.ErrorDataReceived += HandleSTDERR;
        if (!process.Start())
        {
            Console.WriteLine("Process failed to start.");

            return;
        }
        process.BeginErrorReadLine();
        process.BeginOutputReadLine();
        process.WaitForExit();
        Console.WriteLine("Process exit code:" + process.ExitCode);
        process.Close();
    }

    WaitHandle.WaitAll(_allToWaitOn);
}

The key elements of the previous method are to set the ProcessStartInfo properties as:

        UseShellExecute = false, 
        RedirectStandardOutput = true,
        RedirectStandardError = true

The Process class instance in the previous method exposes two events, OutputDataReceived and ErrorDataReceived, that are assigned to the the methods used to handle the asynchronous reading of standard output and standard error:

        process.OutputDataReceived += HandleSTDOUT;
        process.ErrorDataReceived += HandleSTDERR;

After Process.Start is invoked the following methods of the Process instance must be called in order begin the asychronsous reading of standard output and standard error:

        process.BeginErrorReadLine();
        process.BeginOutputReadLine();

What is provided is a C# shell sufficient to invoke a Python script and read the standard output generated from said script. 

Detecting Errors in the Invoked Python Script

The source code on Github includes an example of invoking a Python script, JustGarbage.py, that will generate an error when run by Python.exe:

When a script is invoked Process.Start will return false if the process failed to start. Additional the Process class's Exit code property should return non-zero on an error:

        if (!process.Start())
        {
            Console.WriteLine("Process failed to start.");

            return;
        }
        process.BeginErrorReadLine();
        process.BeginOutputReadLine();
        process.WaitForExit();
        Console.WriteLine("Process exit code:" + process.ExitCode);

It makes sense that Process.Start returns true when Python.exe is passed an nonsensical Python script like JustGarabage.py because Python does successfully get invoked  The invoked process, Python.exe, will detect and report on the error. The value for ExitCode when running the nonsensical script is one indicating an error. Python writes text to standard error indicating an error occurred. The output from th C# application is as follows with JustGarbage.py is specified as the Python script to run:

For those who know music, the Python script, JustGarbage.py, contains the names of the members of the band Garbage. 

Can Process.Start invoke the Python script directly by assigning it to ProcessStartInfo.FileName?

It is not required that ProcessStartInfo.FileName be set to "Python.exe". Instead the actually value of the python script could be specified. To demonstrated this consider the following Python script which creates a file:

import os
import sys

scriptpath = os.path.dirname(os.path.realpath(__file__))
filename = os.path.join(scriptpath, sys.argv[1])
print(filename)
file = open(filename, "w")
file.write("She came from Greece. She had a third for knowledge.")
file.close()

The file create in the previous script is named using first command-line parameter passed into the script (sys.argv[1]).

The C# code to invoke this script is follows where the script name is ProofByFile.py and the name of the Python script is assigned directly to ProcessStartInfo.FileName:

const string _pythonScriptToExecuteDirectly = "ProofByFile.py";

private static void RunScriptDirectly()
{
    string filenameToCreate = 
               DateTime.Now.ToString("yyyyMMddhhmmssfff") + ".txt";
    ProcessStartInfo processStartInfo = new ProcessStartInfo()
    {
        Arguments = filenameToCreate,
        FileName = _pythonScriptToExecuteDirectly,
    };

    using (Process process = new Process())
    {
        process.StartInfo = processStartInfo;
        process.Start();
        process.Close();
    }

    string commandLineOfExecutable = 
        Path.GetDirectoryName(Environment.GetCommandLineArgs()[0]);
    string fileCreatedByPython = 
        Path.Combine(commandLineOfExecutable, filenameToCreate);
    bool fileFound = false;

    for (int i = 0; i < 5; i++)
    {
        fileFound = File.Exists(fileCreatedByPython);
        if (fileFound)
        {
            break;
        }

        Thread.Sleep(100);
    }

    if (fileFound)
    {
        Console.WriteLine("Python created file as expected.");
    }

    else
    {
        Console.WriteLine(
            "Python did not create the file as expected.");
    }
}

Notice at the end of the previous code a delay is used to give the Python script a chance to commit the file craete. This example of polling and Thread.Sleep is not meant to be an example of production code. 

The reason it was possible to invoke ProofByFile.py directly in the previous code was because of how ProcessStartInfo was configured (see the code demarcated by boldface above). By default the UseShellExecute property of ProcessStartInfo  is set to true. This means that the previous code executed using with UseShellExecute=true.

Under the covers this means that Windows used the ShellExecute function to invoke the Python script. If UseShellExecute had been set to false then Windows would have used the CreateProcess function to attempt to invoke the script. It turns out that invoking from the Windows shell (the ShellExecute function) can run the Python script, ProofByFile.py. This is because the shell uses the "PY" file extension to look up the executable used to run said extension  which is Python.exe.

The original premise was that the C# application would receive data from the Python script using standard output from the invoked script. It is only possible for C# to access standard output from the child process when UseShellExecute=false. This is why the original C# example invoked Python.exe and not the Python script directly.