MVC, AJAX and REST - Breaking out of the sandbox, part 2

In Part 1 of this series I described in outline how to set up Bottle so a browser can act as a UI for a Python program running on the same computer. Now I'll put some flesh on the bones.

The File Manager

Here I'll start to code my file manager, which will accept requests from the browser on my computer and return corresponding data. To avoid this article becoming a book I'll stick with a very simple set of functions, being to return the contents of the top-level directory on the computer, permit the user to dive into folders and show the content of text files.

We'll start with 2 new endpoints. One is needed to cover the case where no directory name is given; it returns the contents of the system root directory, e.g. / or C:/. The other takes a path name so it can list any directory in the system.

Note that the code here is all written for Linux. It should also work on Mac but some changes to the root path will be needed for Windows.

A 'list files' endpoint

# Endpoint: GET localhost:8080/listfiles
# Lists all files in the system root directory
@app.get('/listfiles')
def listRoot():
    return listFiles('')

# Endpoint: GET localhost:8080/listfiles/name
# Lists all files in a specified directory
@app.get('/listfiles/<name:path>')
def listFiles(name):
    name = f'/{name}'
    print(f'List files in {name}')
    dd = []
    ff = []
    for file in os.listdir(name):
        path = os.path.join(name, file)
        if os.path.isdir(path):
            dd.append(file)
        else:
            ff.append(file)
    dd.sort()
    ff.sort()
    d = json.dumps(dd)
    f = json.dumps(ff)
    return '{"dirs":' + d + ',"files":' + f + '}'

The endpoint lists all directories and files at the location given and returns them as a JSON formatted string containing 2 lists. One, called dirs, is a list of directories; the other, files, is a list of the files.

You can use this endpoint in Postman, on the browser command line or in a JavaScript file, to examine anything inside the server directory, including subdirectories to any level. The decorator

@app.get('/listFiles/<name:path>')

tells Bottle to catch all GET calls where the first part of the URL is listFiles. The rest of the URL is the path, relative to the server root, and name is also the parameter received by the listFiles() function that follows the decorator. The filter :path will cause it to accept anything that looks like a directory pathname, so it will match any of these:

http://localhost:8080/listFiles/
http://localhost:8080/listFiles/static/html
http://localhost:8080/listFiles/resource/img/png

but not

http://localhost:8080/listFiles
http://localhost:8080/listFiles./html

The listFiles() function itself is passed everything in the URL that follows listfiles/. In this case it reads the directory, extracting directories and files separately and combining them as described above. The return value is passed back to the caller.

As with the double() endpoint in Part 1, this one is a specific endpoint sharing the same pattern as getFile(), so it must be placed higher in the script file or Bottle will see the generic endpoint first and try to use it.

We also need an endpoint to read a file from anywhere on the computer. The @app.get('/<filename:path>') endpoint we created in Part 1 is no good as it can only operate on files in the server's own subtree. So we need another one:

# Endpoint: GET localhost:8080/readfile/content
# Read a file
@app.get('/readfile/<name:path>')
def readFile(name):
    f = open(f'/{name}', 'r')
    file = f.read().replace('\t', '%09').replace('\n', '%0a');
    f.close()
    return file

This will read any text file, given the full path to it. To avoid problems in browsers, tab and newline characters are encoded before the content is returned.

With these endpoints you can now get the contents of any file in any folder on your entire computer.

MVC revisited

The code we've written so far is all part of the Model. We've not yet thought much about what to display (Controller) or how to display it (View). The Model data returned from a call to the listfiles endpoint is just that; a list. Two lists, in fact; directories and files. The Controller was you, typing a URL into your browser's address bar, and the View was the list of files, still in JSON format, that appeared in your browser window.

The overall aim is to create an application that can display a list of files and do something useful when one of the names is clicked. It's not the job of the Controller to decide what the list should look like; its role is to get the list from the Model and give it to the View. Then it should wait until notified by the View that a file name was clicked, and take whatever action is appropriate. This latter part is solely the Controller's responsibility; neither the View nor the Model have any say in the matter.

Right in the middle of all this is the 2-way communications channel between the browser and the server. The Controller sits astride this, hiding it from both the View and the Model, so in principle they could be on opposite sides of the globe without being aware of it.

At the server, the Controller and Model may be a little hard to distinguish. Everything happens in REST endpoints, some of which deal directly with system resources (directories and files) which are part of the Model. Other endpoints, though, make decisions about how they act, and these can be regarded as Controller functions. So an endpoint that fetches a file is Model, but another that calls the first with a specific file name or a set of options is Controller. We may decide to create endpoints that carry information at quite a high level of abstraction; information that has a meaning such as next or previous and causes specific decisions about what Model data is being referred to.

Similarly, at the browser the data returning from the server may require some processing before it can displayed. This doesn't prevent raw HTML being supplied as the response to a request; such a scenario is typical when using HTML templates. It's the job of the Controller to manage templates and populate them with appropriate data, either obtained by a separate call to the server or by using knowledge of the current state of the View.

Server-side only

I noted in Part 1 that the entire job could be done in the server, sending finished pages ready for display, so let's look at that first. The first thing we need is an HTML file:

<html>
  <head>
    <title>/TITLE/</title>
    <meta content="">
    <style></style>
  </head>
  <body>
    /CONTENT/
  </body>
</html>

This file will be used as the main template for all the pages we generate. We'll call it server.html. It's just a basic HEAD and BODY with a couple of replacement tags, /TITLE/ and /CONTENT/.

The Python script needs an endpoint that starts off the file manager. I've decided to set up one that will list files in any directory, using the REST format

http://localhost:8080/flist/<title>/<name>

where flist indicates I want a file listing, <title> is the title of the page and <name> the path to the directory to be listed. The endpoint to handle all this will return a complete HTML page comprising the template above with the page title substituted for /TITLE/ and a list of directories and files substituted for /CONTENT/.

There are actually 2 endpoints; the first one deals with the special case of there being no path, that is, we're listing the root directory. I'll start by defining a constant to hold the root URL of the webserver:

def HOST():
    return 'http://localhost:8080'

Here are the file listing endpoints in full:

# Endpoint: GET localhost:8080/flist/title
# List directories and files in the root directory
@app.get('/flist/<title>')
def flistroot(title):
    return flist(title, '')

# Endpoint: GET localhost:8080/flist/title/content
# List directories and files
@app.get('/flist/<title>/<name:path>')
def flist(title, name):
    # Get the files in the named directory
    flist = json.loads(listFiles(f'/{name}'))
    dirs = flist['dirs']
    files = flist['files']
    html = ''
    for file in dirs:
        if not file.startswith('.'):
            if name is '':
                path = file
            else:
                path = f'{name}/{file}'
            html += f'<div><a href="{HOST()}/flist/{title}/{path}">{file}</a></div>'
    html += '<br>'
    for file in files:
        if not file.startswith('.'):
            if name is '':
                path = file
            else:
                path = f'{name}/{file}'
        html += f'<div><a href="{HOST()}/fview/{title}/{path}">{file}</a></div>'
    # Get the server template
    f = open('template/server.html', 'r')
    template = f.read()
    f.close()
    # Get the HTML to insert
    f = open(f'html/file-lister', 'r')
    content = f.read()
    f.close()
    content = content.replace('/DIR/', name).replace('/CONTENT/', html)
    return template.replace('/TITLE/', title).replace('/CONTENT/', content)

Each of the directory names is hyperlinked to call the same endpoint recursively, traversing the computer's filing system. There's also a hyperlink to view a file, which requires another endpoint:

# Endpoint: GET localhost:8080/fview/title/content
# View a file
@app.get('/fview/<title>/<name:path>')
def fview(title, name):
    content = ''
    # Get the server template
    f = open('template/server.html', 'r')
    template = f.read()
    f.close()
    # Get the HTML to insert
    f = open(f'html/file-viewer', 'r')
    content = f.read()
    f.close()
    # Read the file and escape special characters
    file = readFile(name).replace('<', '&lt;').replace('>', '&gt;') \
        .replace('%09', '   ').replace('   ', '&nbsp;&nbsp;&nbsp;').replace('%0a', '<br>')
    content = content.replace('/FILE/', name).replace('/CONTENT/', file)
    return template.replace('/TITLE/', title).replace('/CONTENT/', content)

Part of this code deals with the problem of displaying HTML or Python files in the browser. If nothing special is done it will interpret content as HTML markup, so each of the critical characters such as angle braces must be converted to HTML codes so they will be displayed properly.

This endpoint is incomplete; it ignores images, for example. The code to do that is not difficult but it could be quite bulky, so I'll leave it as an exercise.

The two endpoints above require HTML templates, file-lister and file-viewer, to format their content. Here they are:

<h2 style="text-align:center">Files in //DIR/</h2>
<div style="background:#ffffee; border:1px solid gray; font-family:mono; margin-left:auto; margin-right:auto; padding:0.5em; width:90%">/CONTENT/</div>

<h2 style="text-align:center">Content of //FILE/</h2>
<div style="background:#ffffee; border:1px solid gray; font-family:mono; margin-left:auto; margin-right:auto; padding:0.5em; width:90%">/CONTENT/</div>

The final thing to do is to update the code at the top of our Python file that calls for the browser to start up, by supplying it the endpoint that will read the top-level directory. As before, if you are not using the Chromium browser on a Linux system some adjustment will be needed:

# start up the browser (version for Linux/Chromium)
cmd = f'chromium-browser {HOST()}/flist/File%20Manager'
subprocess.call(cmd, shell=True)

The result of all this is a very basic - and very incomplete - file manager. When you start the Python program a new tab opens in the chosen browser and displays the files at the root of the file system.

Given the speed of modern computers, the need for a complete screen redraw is not an issue here; the update will be barely noticeable. However, there are a number of reasons why this will not suit all applications, the most important of which is that client-server code of this kind is reactive, not proactive, meaning that things only happen when the user takes some action. So if you wanted to monitor the free disk space, or display the current time, there is no way to "push" changes to the browser; all the server can do is wait for a request to come in following a user action. To do better requires JavaScript and AJAX.

In Part 3 I'll move some of the Controller functionality into the browser.

Photo by Markus Spiske on Unsplash