Introduction

In a previous post I blogged about a mediator pattern, for python development to direct the instantiation, execution and management of subprocesses in an application. In that post I concluded with a promise that I'd have more to say about using processes and threads in the context of Graphical User Interface Development, and, well, here we are. My aim in this present piece is to tie together the concepts and patterns from those previous articles -- those revolving around using asynchronous python code, sub-processes and threads -- with a focus on applying them to GUI development.

Background

Back in olden days, before GUIs became ubiquitous, programmers used to create applications using command-line interfaces (CLIs). Life seems to have been so much simpler back then compared to the exponentially increasing complexity that surrounds us today. But we are where we are only by virtue of standing (and building!) on the shoulders of giants, and so, perhaps ironically, in order to move forward one must look backward every now and then.

My Favorite Programming Tool: The Command Line

Of course I say all that "tongue-in-cheek". As anyone with any interest in programming must surely know, the CLI remains a very important tool to this day and many programs are produced requiring interaction with the command line. That said, the CLI can, at times, feel mysterious, somewhat arcane, and in need of demystification. A big part of the beauty of the command-line lies in the simplicity of the interface. Interacting with a well-designed command-line tool or application can feel like having a conversation. You provide a command, the system might respond with some requests for information, and then it generates your results.

Enter the GUI

All that being said, as applications grow and become increasingly complex the need for a GUI to facilitate user-interaction becomes manifest. To make the discussion somewhat concrete, lately I've been working on implementing a networking protocol and needed a prototype to establish the basis for a distributed system. In order to conduct the analyses I needed to build the prototype, I reached a point where I needed to stand up a GUI as a sort of harness for development. In part, this series of posts documents some of the issues I encountered and some of the pain-points along the way.

Reading Subprocess Output with Threads

One of the problems I encountered revolved around the need to execute and coordinate subprocesses which had been previously defined as python console-applications. As noted and discussed in detail elsewhere, the problem with executing subprocesses -- and indeed any asynchronous code -- in a GUI app is that GUIs are event driven. They execute in a continuous loop that will block if it gets caught up in a lengthy routine or if a subprocess spawned by the GUI blocks for some reason. Again, I've discussed GUI event loops and asynchronous development elsewhere, so won't get into the details of the event-loop here. Instead I'd like to extend the previous discussions with a focus on reading and writing to subprocesses using threads.

Visualizing the Solution

I am a visual thinker -- I like visualizing solutions and I'm big fan of a good diagram. So before jumping into the code I've created a visualization of the solution we're about to dig into. Please indulge me as I walk through it.

Figure 1 illustrates reading subprocess output with threads in a GUI driven application. Here's the analysis.

1. First, notice the illustration of the process pipe representing the sub-process standard output. I kind of like the pipe analogy, so much so that I've gone ahead and drawn a little valve on it. I did so to highlight the point that this is what will block the GUI in an application that reads output from a sub-process. If the subprocess reaches a point where it awaits input (obtained through another pipe -- namely stdin) Then it's stdout which will be blocked. The valve is a reminder that the pipe can be shut off at times.

2. So if you want to interact with a long-running process from a GUI, you'll want to spin off the reading of that process's stdout pipe to a dedicated thread.

3. Under such conditions, the safest way to communicate information back to the main thread (here, the one executing the tkinter loop) is to use a queue. Recall that a queue is a data structure that encapsulates a "First-In-First-Out" (FIFO) data processing routine. The queue defines methods for retrieving data in the order it's entered. The good news is that python provides pre-defined queue-type data structures that are said to be "thread safe". Probably.

4. The thread reading the output (the output reader) executes a read loop wherein it simply; (1) reads a line at a time from the pipe, and (2) puts each line read into a shared queue.

5. Meanwhile, the main thread sits there continuously executing the tkinter event loop. This is where we can tie in to periodically update tkinter GUI views using await to read new line items should they become available on the queue.

In essence, what we have here is a classic producer/consumer use-case. The output reader is a producer who's role is to populate the process output queue with data as it becomes available over the course of the application life cycle. The main thread is the consumer, which periodically dequeues information and updates a view in the GUI.

With this visualization in mind, let's look at a concrete example...

Example

Listing 1

The first listing is a just some scaffolding I put up to test the process. It's just a simple python script that prints some output, asks for end-user input, and echos that input back to the end-user. All in all, pretty straightforward. The thing to notice though, is the part that gets user input using the python input() function (which I've highlighted in red). input causes the program to block and wait for the end-user to enter data. Normally, this is what we'd want since it enables interaction via the console through stdin (by default the keyboard in a console app). The problem is that if we run this script from a tkinter GUI app the GUI will freeze up when the script hits this part of the application.

import sys
import argparse
import asyncio

def send_output ( text ) :
    print( text )
    sys.stdout.flush()

def get_input () : 
    print( "Enter Text Input ... " )
    test_in = input()
    sys.stdout.flush(  )
    return test_in

def echo_input( obtained_in ) : 
    send_output( obtained_in )

async def send_async_output (text) :
    print( text ) 

def test_send_async ( text ) :
    loop.run_until_complete( send_async_output( text ) )

if __name__ == "__main__" :
    print( "----    START TEST    ----" )
    parser = argparse.ArgumentParser (
        prog = 'nn_testscript',
        description = 'Test scaffolding for process mediator pattern',
        epilog = '\u266B Always look on the bright side of life... '
    )
    parser.add_argument( 'instance_label' )
    args = parser.parse_args()
    print( sys.argv[0] )
    instance_label = args.instance_label
    print(f"Instance: {instance_label}")
    loop = asyncio.new_event_loop()
    send_output ( "Testing 1, 2, 3" )
    test_in = get_input()
    test_send_async( f"ECHO: {test_in}" )
    print( "----    END TEST    ----" )
    

Listing 2

Listing 2 (below) comprises a simple test-GUI (using tkinter) which I whipped up to demonstrate the process. The listing shows how the GUI code:

1. Launches the test script above using a process mediator, and

2. Enables interaction through GUI controls.

In this case there are only a few widgets:

1. A text area to display the process output,

2. An entry field to enable an end-user to enter data,

3. A launch button to launch the subprocess (executing the test script), and

4. A send button to send data to the process via its stdin pipe.

import tkinter as tk
from process_mediator import ProcessDirector

# Main application class with tkinter GUI
class App:
    def __init__( self, root ):
        self.root = root
        self.root.title("Read/Write Process Thread Example")
        # Instantiate a 'process mediator'
        self.process_mediator = ProcessDirector()
        # Create GUI elements
        self.text_output = tk.Text(root, wrap='word', height=20, width=40)
        self.text_output.pack(pady=5)
        self.entry_input = tk.Entry(root)
        self.entry_input.pack(pady=5)

        self.start_button1 = tk.Button( 
            root, 
            text="Launch Subprocess", 
            command=lambda:self.start_process( TEST_SCRIPT )
        )
        self.start_button1.pack( side=tk.LEFT, padx=5 )

        self.send_button1 = tk.Button(
            root, 
            text="Send Input", 
            command=self.send_input
        )
        self.send_button1.pack(side=tk.LEFT, padx=5)

        # Start updating output display
        self.update_output()

    def start_process(self, script_name):
        """Start the specified process."""
        self.subProcId = self.process_mediator.launch_process( 
            script_name, 
            ["TEST_PROCESS"] 
        )

    def send_input( self ):
        """Send input to the selected process."""
        input_text = self.entry_input.get()
        self.process_mediator.send_input( 
            self.subProcId, input_text 
        ) 
        self.entry_input.delete(0, tk.END)

    def update_output(self):
        """Update output view(s)."""
        if  hasattr( self, 'subProcId' ) :
            output = self.process_mediator.process_q()
            self.text_output.insert( tk.END, output )
        self.root.after(100, self.update_output)  # Update every 100ms

if __name__ == "__main__":
    TEST_SCRIPT = "nn_testscript.py"
    root = tk.Tk()
    print( "====    START GUI TEST    ====" )
    app = App(root)
    root.mainloop()
    print( "====    END GUI TEST    ====" )
    

Notice that the GUI is very thin. It doesn't 'know' or 'care' how to orchestrate communication with the sub-process. All that is the responsibility of the process mediator (ProcessDirector). To get script output to update the relevant view all the client code has to worry about is calling: self.process_mediator.process_q()

Listing 3: The Process Mediator

The next listing contains selected portions of the class ProcessDirector. I've provided a full listing of the class as an appendix but here I've in-lined those parts most relevent to the present discussion for convenience.

import queue
import subprocess
import time
...
from threading import Thread

class ProcessDirector :

    def __init__ ( self ) : 
        self.processes = [] 
        self.t_queue = queue.Queue()

    def launch_process( self, python_script, cmd_ln_args ) : 
        launch_sequence = [ sys.executable, python_script ] + cmd_ln_args
        proc = subprocess.Popen(
            launch_sequence,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            bufsize=1
        )
        self.processes.append( proc )
        t = Thread(
            target=self.read_proc_output,
            args=[proc.pid]
        )
        t.daemon = True
        t.start()
        return proc.pid

    ...

    def read_proc_output( self, pid ) : 
        for process in self.processes :
            if process.pid == pid :
                proc = process
        while True: 
            # sleep. otherwise you work too hard and heat up the box...
            time.sleep( 0.02 )

            output = proc.stdout.readline()
            if not output :
                continue
            self.t_queue.put( output )

    ...

    def process_q ( self ) : 
        output = ""
        while not self.t_queue.empty () :
            output += self.t_queue.get()
        return output

    ...

    

Notice:

1. The ProcessDirector is responsible for launching subprocesses and orchestrating process interactions. It launches subprocesses in the launch_process method using subprocess.Popen. Notice further that the process director obtains handles to the standard process pipes; stdout, stdin, and stderror. The buffsize=1 argument sets the buffer to a line.

2. Next, it spins off a thread targeting the read_proc_output method. read_proc_output defines the read-loop that reads from the subprocess stdout pipe and populates the process output queue (owned by the process director) a line at a time. It's this read operation that has the potential to block (and freeze up) the GUI so this is the part that gets spun off to its own thread.

3. Finally -- notice the process_q method. This is the method of concern for ProcessDirector client code that wants to update views associated with the process output. Client code would call this method to dequeue output for display.

And that's it. Taken as a whole the code we've walked through here is a basic implementation of a pattern that enables subprocess management and the integration of asynchronous code in Python GUI driven applications. The following screenshot shows the GUI displaying the output of the test script following an interactive session.

Discussion

The code that I've provided for this blog post is pretty bare-bones. It's intended to serve as reference material and also to provide a basis for further development. The ProcessDirector class is a first-pass implementation of what I've described as a mediator pattern for subprocess management. The GUI code is simple enough that it should be readily adaptable to other use-cases or to play with the process mediator.

The Process Mediator Pattern

The main considerations in adapting the process mediator are that it owns relevant data structures and manages the life-cycle of subprocesses tailored to execute specific sub-routines. The example we walked through here uses a queue to manage process output and defines a public method intended to enable client code to process the queue from a "consumer" thread (in the present case the main tkinter loop).

Additional Considerations

Another issue that came up for me concerns the subprocess output. Handling output from subprocesses can sometimes be a bit tricky. Python buffers standard output which can lead to issues with subprocesses that use print statements to write to standard output. So if you're working with subprocesses and encounter unexpected issues like missing output, process blocking, or race conditions consider the following.

1. Python print statements are usually line buffered. But when redirected to a pipe the system may switch to block buffering. So it is important to ensure that the buffer gets flushed in in the subprocess.

2. You can do this on print statements with: print( "ipsum lorum ... ", flush=True ) .

3. You can also flush the buffer from the subprocess with sys.stdout.flush() .

4. If you cannot modify your subprocess script you ca try the workaround of running the process with the environment variable: PYTHONUNBUFFERED=1 .

And as a final consideration, here are some useful linux commands if you find yourself working with subprocess and need to troubleshoot issues.

$ ps aux | grep [[PATTERN]]

$ kill [[ pid ]]
    

1. The first command will get you a list of processes on your system. You can grep on the script name if you want to see instances of script processes launched from your application. Use this if you need to get process IDs.

2. The second command can be used to kill a process given the pid. Use kill -9 to force termination if its gone unresponsive.

Summary

This post concludes a series of blog entries revolving around python GUI development with subprocesses, asynchronous routines, and multi-threaded process I/O. The present article provided a rationale and discussion for using threads to prevent UI blocking in tkinter application development. A number of code listings are provide for reference and potentially to bootstrap development.

Appendix 1: The Process Mediator

Below is a complete listing of the class ProcessDirector along with a bit of associated test scaffolding. ProcessDirector is an implementation of a mediator pattern which I've described in an earlier entry to this blog series. I've also made the ProcessDirector source file and the test GUI described in this and prior articles available on github.

import queue
import subprocess
import time
import sys
import os
from threading import Thread

class ProcessDirector :
    '''
    Responsible for spawning and directing processes to 
    execute python scripts. 
    '''
    def __init__ ( self ) : 
        self.processes = [] 
        self.t_queue = queue.Queue()

    def launch_process( self, python_script, cmd_ln_args ) : 
        '''
        Launch a process and add it to your list given...

        arguments: 
            python_script: name of script to launch
            cmd_lin_args: a sequence of command line 
            arguments...

        returns: the new process id. The client should hold onto it.
        '''
        launch_sequence = [ sys.executable, python_script ] + cmd_ln_args

        proc = subprocess.Popen(
            launch_sequence,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            bufsize=1
        )

        self.processes.append( proc )

        t = Thread(
            target=self.read_proc_output,
            args=[proc.pid]
        )
        t.daemon = True
        t.start()

        return proc.pid

    def terminate_children( self ) : 
        '''
        Dispose of all spawned child processes
        '''
        for i in range( len( self.processes ) - 1, -1, -1 ) : 
            target = self.processes[i]
            target.terminate()
            target.wait()
            if target.returncode is not None:
                self.processes.pop( i )

    def read_proc_output( self, pid ) : 
        '''
        Read-loop target for read thread. Reads stdout of child 
        process given the process ID. Note: The pid enables the
        director to determine which pipe to read...

        Arguments: 
            pid: The pid (process id) of the process 
            (should be obtained on launch...) 
        '''
        for process in self.processes :
            if process.pid == pid :
                proc = process
        while True: 
            # sleep. otherwise you work too hard and heat up the box...
            time.sleep( 0.02 )

            output = proc.stdout.readline()
            if not output :
                continue
            self.t_queue.put( output )

    def send_input(self, pid, input_text):
        '''
        Send input to the specified process.

        Arguments:
          pid: Process id to send to
          input_text: text to send
        '''
        for process in self.processes :
            if process.pid == pid :
                proc = process
        proc.stdin.write(input_text + '\n')
        # proc.stdin.write(input_text)
        proc.stdin.flush()

    def process_q ( self ) : 
        '''
        This is the function clients should call to get data
        from the `process mediator`. The queue is expected to
        hold data accumulated since the last dequeue operation...
        '''
        output = ""
        while not self.t_queue.empty () :
            output += self.t_queue.get()
        return output

    def terminate( self, pid ) :
        '''
        Kill the specified process given ...

        Arguments: 
            pid : process id (obtained at launch)
        '''
        for i, process in enumerate( self.processes ) :
            if process.pid == pid :
                target = process
                target_idx = i
                break
        if not target :
            return
        target.terminate()
        target.wait()
        if target.returncode is not None:
            self.processes.pop( target_idx )


if __name__ == "__main__" : 
    print( "----    START TEST     ----" )
    CWD = os.getcwd()
    TEST_SCRIPT_NAME = CWD + "/path/to/your/python_script.py"
    pd = ProcessDirector()
    test_proc_1_pid = pd.launch_process( TEST_SCRIPT_NAME, [] )
    test_proc_2_pid = pd.launch_process( TEST_SCRIPT_NAME, [] )

    print("TESTING. WAITA MINNIT...")
    time.sleep( 5 )
    print( f"Q SIZE PRE: {pd.t_queue.qsize()}" )
    test_output = pd.process_q()
    print( test_output )
    print( f"Q SIZE POST: {pd.t_queue.qsize()}" )

    pd.terminate_children()
    print( "----    END   TEST     ----" )
    

Resources

1. Graphical User Interfaces with Tk

2. The TkInter Threading Model

3. The Python Threading API

4. The Python Thread Object

5. queue -- A synchronized queue class

6. Python Subprocess Management

7. process-mediator on github