inconsequentia

Using Hugo to Generate Gopher Sites

Posted at — Jul 10, 2019

Last week, after spending a lot of time bashing my head against Hugo, I realised the behaviour of uglyURLs had changed (thanks to old mate #4428), left the entire gopher site ugly, and rage quit for a while. Well, I say rage, but mostly I was too busy working on our kitchen renovation to really worry about it.

This evening I got back to it and eventually got what looks like a pretty functional setup running. This is still really heavily based on jfm’s work, for which I’m eternally grateful, but I’ve had to update it a fair bit for Hugo v0.55.6.

Hugo configuration

config.toml

To start with, I globally enabled uglyURLs. Weirdly, you can’t turn uglyURLs on in an output format, but you can turn it off. So we turn it on globally, and then disable it for the HTML output format.

uglyURLs = true

The next step is defining a new output format for gopher. This is the part where we also turn uglyURLs back off for HTML.

The outputs section says to keep generating the usual HTML and XML content, but also to generate our new gopher format alongside it. The gopher output format is just plain text, with ‘list’ files being named gophermap. Well, actually, they’ll end up being named gophermap.txt. Well, actually, only the root _index.md gets named properly. We’ll fix the rest up later.

[outputs]
  home = ["HTML", "gopher", "RSS"]
  section = ["HTML", "gopher", "RSS"]
  taxonomy = ["HTML", "gopher", "RSS"]
  taxonomyTerm = ["HTML", "gopher", "RSS"]
  page = ["HTML", "gopher"]

[outputFormats.gopher]
  mediaType = "text/plain"
  baseName = "gophermap"
  isPlainText = true
  protocol = "gopher://"

[outputFormats.HTML]
  noUgly = true

Finally, I needed a few extra entries in my params section. gophermaps need a hostname, so I stuck that in params, and obviously I needed a figlet title. Escaping all of the backslashes, obvs.

[params]
  hostname = "example.com"
  figletTitle = """
 _                                                  _   _
(_)_ __   ___ ___  _ __  ___  __ _ _   _  ___ _ __ | |_(_) __ _
| | '_ \\ / __/ _ \\| '_ \\/ __|/ _` | | | |/ _ \\ '_ \\| __| |/ _` |
| | | | | (_| (_) | | | \\__ \\ (_| | |_| |  __/ | | | |_| | (_| |
|_|_| |_|\\___\\___/|_| |_|___/\\__, |\\__,_|\\___|_| |_|\\__|_|\\__,_|
                                |_|"""

Templates

There are three templates I’m using. One for the root index page (*cough* sorry, root gophermap), one for list pages, and one for individual item pages. I’ve put them all in the layouts directory at the root of my Hugo tree, but will eventually move them in to a theme, I suppose.

layouts/index.gopher.txt

layouts/index.gopher.txt generates the root gophermap for the site. It displays the figletTitle, the contents of my root _index.md, then section links and finally the most recent updates. But we can’t just dump raw content, it has to be formatted as a valid gophermap. The template has to iterate over raw content for all of the bits we want to include and give every line a light massage. Note that this file is also full of tabs; briefly, every output line is <selector type><content>\t<path>\t<host>\t<port>. For plain text lines (i selector) I went with the convention I found after five minutes looking at other people’s maps of leaving the path blank, an obvious placeholder hostname, and a port of 1.

!{{ .Site.Title }}
{{- range (split .Site.Params.figletTitle "\n") }}
i{{- . }}		null.host	1
{{- end }}
{{- range (split .RawContent "\n") }}
i{{- . }}		null.host	1
{{- end }}
i		null.host	1
iSite sections:		null.host	1
{{- range .Site.Menus.main }}
1{{- .Name }}	{{ .URL }}
{{- end }}
i		null.host	1
iMost recent articles		null.host	1
{{- range first 3 .Pages.ByPublishDate.Reverse }}
0{{- .Title }}	{{ with .OutputFormats.Get "gopher" -}}{{ .RelPermalink }}{{ end }}
{{- end }}
.

layouts/_default/list.gopher.txt

This template is used for all other list pages across the site (I belive this is all other index.md and _index.md files). Again, it’s a gophermap so you can’t just dump raw content. A simple template that formats the content and then lists the rest of the appropriate pages in reverse chronological order looks like this.

!{{ .Title }}
{{- range (split .RawContent "\n") }}
i{{- . }}		null.host	1
{{- end }}
i		null.host	1
{{ range .Pages.ByPublishDate.Reverse }}
0{{ .Title }}	{{ with .OutputFormats.Get "gopher" -}}{{ .RelPermalink }}	{{ $.Site.Params.hostname}}	70 {{ end }}
{{ end }}

layouts/_default/single.gopher.txt

Finally, a template to display an individual page. This one is just regular text, so all my template does is append the title with a markdown header, and dump the raw content. Obviously anything else can go in here too.

# {{ .Title }}

{{ .RawContent }}

Generating the site

At this point, running hugo will now generate HTML and gopher text files in the same output directory. But that doesn’t bother me, because I’m lazy. Yes, if you wanted to you could request HTML files from the gopher server and vice-versa. But who cares?

But because of old mate #4428 the file names and locations of our gophermaps is all hinky. A file named content/blog/_index.md in our source tree will end up as public/blog.txt when what we want is public/blog/gophermap.txt. So I wrote a deploy script that, after generating the site, moves text files in it to the proper place. It assumes that if a text file has a directory alongside with the same name (minus extension), and the first character of the file is ! (the gophermap selector for a map title, which all of my gophermap templates use), then it’s a gophermap and moves it to the right place. Finally it rsyncs the site to my server. I put this script in deploy.sh in the root of my Hugo tree, so it will play nicely with easy-hugo.

#!/bin/sh
#
# deploy.sh
# Run hugo to generate static content. Do some post-processing to ensure
# gophermaps are in the right place with the right name. And then rsync
# the result to my web|gopher server.

destination=example.com:/var/www/html
# Refresh generated content
rm -rf public
hugo --destination public
ret=$?
if [ $ret -ne 0 ]; then
    echo "hugo generation failed, bailing"
    exit $ret
fi

for file in $(find public -name "*.txt"); do
    echo "Considering $file:"
    dirname=${file%.txt}
    if [ -d $dirname ]; then
        echo "  $dirname exists. Checking for gophermap."
        if [ "$(head -c 1 $file)" = \! ]; then
            echo "    File is gophermap. Moving to $dirname/gophermap.txt"
            mv $file $dirname/gophermap.txt
        else
            echo "    File is content. Ignoring."
        fi
    else
        echo "  $dirname doesn't exist. File is content. Ignoring."
    fi
done

rsync -rtplz --delete public/ $destination

Setting up the gopher server

I’m using gophernicus because it’s simple, secure, and comes ready to be packaged for Debian (I’m really, really lazy, and tired of a thousand python venvs strewn all over my home Puppet manifests). I installed it, and added -r /var/www/html -g gophermap.txt to the end of the options in /etc/default/gophernicus to set the gopher root and change the name of the gophermap file to gophermap.txt.

DONE

And, well, gopher://hardy.dropbear.id.au/ works now.

Caveats

I, too, have had to rethink how I’m typing Markdown. I’ve been slowly replacing inline links whenever I see them in documentation at work, so reference-style links aren’t a big thing for me. But I’m finding that keeping the links themselves close to where the labels are used, instead of all bunched together at the end, feels more readable. It’s also really important to note that gophermaps by default have a lower maximum length, so I’m careful to wrap all of my index files at 72 characters to deal with this.

And it’s still not a perfect solution either. The biggest issue I have is that Hugo shortcodes are just being dumped raw in to the gopher pages, when ideally I’d like, for example, relref shortcodes to be expanded to a real link (I’d live with HTTP, but ideally Hugo would come up with an appropriate gopher link). I think this is possible with custom shortcode templates, but I also think that’s going to be a problem for another day.