A Gopher Client in Rust - 02 Core Client

Alright! After a few days of playing with the Gopher protocol, I have the core of it working and added a whole bunch of things that I wanted as I was going around gopherholes.

• Gopherholes are people's gopher directories/servers, maybe

First of all, some caveats. This project really highlighted how much of a rust and it's style I still don't get. Every thing works but something about the code doesn't feel right and it bothers me.

Second, I very much wrote this only for myself, so there isn't much in the way of robustness and error handling is only match statements and some if checks. I think that is something worth fixing.

Third and lastly, I really enjoyed traveling the internet under my own power! Here I am viewing and reading people's thoughts with something that I wrote myself. As I read, I wanted features like bookmarks, and history and it was powerful to be able to just open up vim and add it.

Gopher is a very human place, at least the places I was in, it's all people's thoughts and ideas and there is nothing industrial about it. I had learned about gopher on hackernews.

https://news.ycombinator.com/item?id=23161922

The link points to a screenshot of a GUI browser pointed at baud.baby. I ended up learning about gopher and telneting in and I was hooked on his content. Just the writing style, and the fact I was telneting in scratched an itch that I always felt.

I very much recommend implementing this on your own and then reading this to see where we may have done things differently.

The full code is available in the third chapter.

The full list of commands I've implemented are:

• visit - Go to a gopherhole
• ls - List current menu
• cd - Switch into a directory
• more - Read a document
• add - Add a directory to our bookmarks
• b - List our bookmarks
• b{num} - Select a bookmark and go there
• h - List the documents we've read
• h{num} - Show the document that we read
• where - Show where we currently are
• bye - Quit our client
• Search - Search the menu for something

Feel free to add more or leave the one's you don't care for.

The first 4 commands are really the ones you need for a minimal client.

Anyway! Let's move to the code.

Gopher Client

I'm going to link the entire source at the bottom along with the one dependency which is libc in Cargo.toml.

I'm not going to go over every line but just the functions I think were fun/hard to implement.

Visit Function

The core of gopher is you connect on a port, submit a selector string and ideally get either a list of lines which is a navigation or you get a dump of text which is a document.

With that said, the first step is to connect to a gopherhole and submit a selector string.

...
fn visit(domain: &str, port: &str, file_path: &str) -> Result<String, std::io::Error> {
    let gopher_hole = format!("{}:{}", domain, port);
    let stream = TcpStream::connect(&gopher_hole);

    match stream {
        Ok(mut stream) => {
            let selector = format!("{}\r\n", file_path);
            stream.write(selector.as_bytes()).unwrap();
            stream.flush().unwrap();

            let mut data: Vec<u8> = vec![];
            stream.read_to_end(&mut data).unwrap();
            Ok(String::from_utf8_lossy(&data).to_string())
        },
        Err(e) => {
            Err(e)
        }
    }
}
...

The visit function opens a connection to a specified domain, port and submits the file_path. This is the selector but as I worked on the client, selector has other meanings for me(I was thinking of css and jquery) so I changed it to something that made more intuitive sense for me.

Once we have our socket(stream) open we add a CRLF, carriage-return line-feed to our file_path and we write it as bytes to the socket. Voila! We have just done what everything does on the internet. You can't send anything but binary data over sockets so to send something we need to do the conversion. This is true for HTTP just as much as gopher!

We then read from the socket and add everything to a vector. We could read this into an array but we don't know how much data we'll be getting back.

We have now received binary data from the server! All we have to do is convert this binary data into text and we can then view the data. We use the from_utf8_lossy function because if there are letters we can't deal with, it will be replaced with a ?.

We then return the data as a string to our calling function.

The visit function is the core of our gopher client, I really like the term visit because browsing the gopherholes very much feels like visiting.

Parsing the Data

Let's now look at what happens to the data we received from the server. One thing to note here is visit is assumed to always return data for a menu. I think this may be in the spec, but it could also not be.

...
else if tokens[0] == "v" || tokens[0] == "visit" {
            let data = visit(tokens[1], "70", "");
            match data {
                Ok(data) => {
                    prompt = String::from(tokens[1]);
                    pwd.push(Location::new(tokens[1].to_string(), "70".to_string(), "".to_string(), tokens[1].to_string()));
                    stack.push(GopherListing::parse(data));
                },
                Err(e) => {
                    println!("{}", e);
                }
            }

...

In our program, when we type in visit, it will hit this if statement. We can do something like visit baud.baby and it will then run the visit function return with the menu for baud.baby.

I ended up adding a promp, a list of places we are nested in and a history. Everything is a stack which is neat!

The key part of this visit command is the parsing of the data.

Let's take a look.

...
#[derive(Debug)]
struct GopherListing {
    id: u32,
    gopher_type: String,
    user_visible_name: String,
    file_path: String,
    domain: String,
    port: String
}

impl GopherListing {
    fn parse(data: String) -> Vec<GopherListing> {
        let mut listings: Vec<GopherListing> = vec![];

        let mut counter = 0;
        for gopher_line in data.split("\r\n") {
            if gopher_line == "." {
                break;
            }

            if gopher_line == "" {
                continue;
            }

            let gopher_type = gopher_line[..1].to_string();

            if gopher_type == "0".to_string() || gopher_type == "1".to_string() {
                let line: Vec<&str> = gopher_line[1..].split("\t").collect(); 

                counter = counter + 1;
                listings.push(GopherListing {
                    id: counter,
                    gopher_type,
                    user_visible_name: line[0].to_string(),
                    file_path: line[1].to_string(),
                    domain: line[2].to_string(),
                    port: line[3].to_string()
                });
            }
        }

        listings
    }
}
...

We want to take the gopher menu data and create a GopherListing out of it. This way we can have a model for the data we receive. Our parsing function is very simple and the RFC make's quite clear that this was one of the goals.

All the parser does is, split on tabs, if it is type 0 or 1, create a GopherListing. I know there are more types in the wild and i is very much used everywhere but I didn't want to deal with it so I stayed with the what made sense to me.

Once we run our visit function, we then parse the data into a vector of gopher listings and then save that to our stack!

• After finding that there were a number of gopherholes using the I type, I added a simple one line change to allow i types to get through the parsing step. As soon as I did, Type 1 Files, open into a listing of type I lines which I can see on the screen.

...
 if gopher_type == "0".to_string() || gopher_type == "1".to_string() || gopher_type == "i" {
...

The if statement simply lets i types get added to the GopherListing.

LS - List Gopher Items

Now that we have a vector of gopher listings, we can now start displaying them.

...
fn display(listings: &Vec<GopherListing>, needle: &str) {
    for listing in listings {
        if needle == "" || listing.user_visible_name.to_lowercase().contains(&needle.to_lowercase()) {
            if listing.gopher_type == "i" {
                println!("{}", listing.user_visible_name);
            } else {
                println!("{}. {}", listing.id, listing.user_visible_name);
            }
        }
    }
}
...

This function will just print out the user visible name and the id. I had originally had just the visible name but I changed it quickly after getting tired of typing in the full name to view the document. I should have saw in the RFC that numbers as menu options is also a core part of gopher!

This function actually does double duty as I also added rudimentary search here, this was a later addition.

This function also removes some of the clutter that would happen if we printed the i types like normal, this way we don't add unuseable numbers for the i type lines.

...

        if tokens[0] == "ls" {
            if stack.len() > 0 {
                display(&stack.last().unwrap(), "");
            } else {
                println!("Nothing to show.");
            }
            continue;
        }
...

The ls command calls the display function with the last item on the stack. This means that we are saving all of our visits to various gopher holes so we don't have to do another network call.

Now we can view a list of gopher items on our screen! The next step is to move into a type 1 gopher item which is a directory.

CD - Change Directory

...
       } else if tokens[0] == "cd" {
            let selected = tokens[1].trim();

            if stack.len() == 0 {
                println!("Nothing to change directory into.");
                continue;
            }

            if selected == ".." {
                pwd.pop();
                stack.pop();
                continue;
            }

            for listing in stack.last().unwrap() {
                if selected == listing.user_visible_name || selected == listing.id.to_string() {
                    if listing.gopher_type == "1" {
                        let data = visit(&listing.domain, &listing.port, &listing.file_path);
                        match data {
                            Ok(data) => {
                                prompt = String::from(listing.domain.clone());
                                pwd.push(Location::new(listing.domain.clone(), listing.port.clone(), listing.file_path.clone(), listing.user_visible_name.clone()));
                                stack.push(GopherListing::parse(data));
                            }, 
                            Err(e) => {
                                println!("{}", e);
                            }
                        }
                    } else {
                        println!("Not a directory.");
                    }
                    break;
                }
            }
...

Once we display the listing of gopher items, a user can now do cd 2 to mean that they want to go inside the second line in the listing.

So we first figure out where we're going. We find the listing in our vector and visit it. Now instead of passing CRLF to get the base menu, we will pass in what the listing has for it's file_path. Once we get the data back, we will do what we did in our visit command.

We will add the listing to our pwd(path of working directory), we will then add it to our stack. Our stack is a vector of all the places we've been.

Now the next time we go to do ls, to list our gopher items, we will pick up the listings in the last part of our stack which is the newest place we're in.

Now we can travel down menus! The final thing we need to be able to do is view second type of gopher items - the documents!

More - Viewing Gopher Documents

...
        } else if tokens[0] == "more" {
            let selected = tokens[1].trim();
            if stack.len() == 0 {
                println!("Nothing to print.");
                continue;
            }

            for listing in stack.last().unwrap() {
                if selected == listing.id.to_string() {
                    if listing.gopher_type == "0" {
                        let data = visit(&listing.domain, &listing.port, &listing.file_path);

                        let document = Location::new(
                            listing.domain.clone(),
                            listing.port.clone(),
                            listing.file_path.clone(),
                            format!("{} ({})", listing.user_visible_name, listing.domain)
                        );

                        save_in_file(history_path, &document);
                        history.push(document);

                        match data {
                            Ok(data) => display_document(data),
                            Err(e) => println!("{}", e)
                        }

                    } else {
                        println!("Not a file.");
                    }
                    break;
                }
            }
        }
...

More acts very similar to cd, we type in more 3 meaning we want to view 3, and if it is a type 0, it will output to the screen.

We use the visit function again but the difference here is that instead of creating a GopherListing from the data, we will simply pass the data to our display_document function.

fn display_document(data: String) {
    let (_, window_height) = term_dimensions();

    let lines: Vec<&str> = data.split("\n").collect();
    let mut current_pos = 0;
    let mut done = false;

    while !done {
        for i in current_pos..(current_pos + window_height - 1) {
            if i >= lines.len()  {
                done = true;
                break;
            }
            println!("{}", lines[i]);
            current_pos = i;
        }

        if !done {
            myp!("\x1b[92m[Press Enter for Next page]\x1b[0m");
            let mut command = String::new();
            io::stdin().read_line(&mut command).unwrap();
            if command.trim() == "q" {
                done = true;
            }
        }
    }
}

The core of this function is that we split the data by newlines and display each line.

The extra logic is for pagination. We get the height of the window and then only display the lines that fit. We allow the user to hit anything to go to the next page or q to exit in the middle. The window height is actually stolen from the term_size crate. The rust crates site, docs.rs is very easy to use and viewing the source of rust crates is quite friendly.

use libc::{STDOUT_FILENO, c_int, c_ulong, winsize};
use std::mem::zeroed;
static TIOCGWINSZ: c_ulong = 0x5413;

extern "C" {
    fn ioctl(fd: c_int, request: c_ulong, ...) -> c_int;
}

unsafe fn get_dimensions_out() -> (usize, usize) {
    let mut window: winsize = zeroed();
    ioctl(STDOUT_FILENO, TIOCGWINSZ, &mut window);
    (window.ws_col as usize, window.ws_row as usize)
}

fn term_dimensions() -> (usize, usize) {
    unsafe { get_dimensions_out() }
}

This code was mostly taken from term_size because I didn't want to add any dependencies, adding libc in my mind isn't that bad and term_size was really just a wrapped for libc so I went directly to the source. There are some magic numbers here that I don't understand but it works!

! We have the core functions of our client at this point, we have the ability to connect to a gopherhole, display the menu, travel the menu, and view a document.

That's it! In the next chapter, I'll go over some of the features I added such as history, bookmarks and search.