Sad beautiful truths

Serendipity played it’s part yesterday. A series of events that lead me to an open invitation track day at Hampton Downs. With summer drawing to a close here in New Zealand, and my bike going back into storage due to overseas committments, “scratching” on a track day was the perfect way to end the season.

But it wasn’t the thrill of the 20 somethings on their sports bikes being left in my wake on my “tourer” that I enjoyed most. Nor the power wheelies leaving the apex on the hair-pin turns. Or even the circuit scrutineer on his prupose built track bike letting me know after the session he had trouble catching me. What I enjoyed most was the 60 something year old widower I befriended in the pits.

Unbeknownst to most, motorcyclists are a friendly, supportive and curious bunch. It’s the same reason we always wave to one another on the road, to uphold and respect the unwritten code. Maybe it was the thought of me in years to come, or that I’m always eager to learn, but this day I pulled up beside a grey and aged owner, kitted in his suit, supping on a cuppa tea.

With a purpose built Triumph track bike, a van full of spares and all the racing kit to see you run an entire season, I was surpised when he metioned he was heading out in the “slow” group. I know full well that age isn’t a limiting factor on the back of a motorcycle and have certainly been “schooled” for thinking so. I enquired as to why and he replied.

“This is the first time I have been back out on the track since my wifes’ death. I took a year and a half off to nurse her until her passing.”

He told me that she had cancer of the throat so was unable to speak. A hole cut in her larynx to allow an airway to breathe. He feed her directly via a tube to the stomach.

“She would be here now.” he said, “She would always come to my track meets, we travelled the country together, just us and these two wheels.”

As the day progressed, I would confirm his last session, how he was going. Was he finding his rhythm, or getting obstructed by other riders? We would exchange tips, entry and brake reference points, preferred lines… and then there was a silence.

For whatever reason, I don’t really know why, he decided to tell me the last written words his dying wife wrote.

“I love you. Kiss, kiss. Hug, hug”

He said he didn’t really think much of it at the time, replying of course “I love you too”. He regrets not properly saying goodbye as she passed later that evening.

A tear rolled down my cheek.

With his words on my mind, I headed out for my next session, eager in some sense to get back to my new friend.

He didn’t finish the remainder of the day. I guess he knew he really didn’t need to. Packing up early, we left each other with a handshake and a hat tip goodbye.

There’s something to be said for any being who loses their life partner, to stand up, to get up, to get out and do the things that brought them so much enjoyment in past.

These poignant reminders, that none of us have very long within this realm.

Personally, I’m choosing a life without fear nor regrets.

MyDigest - A learned journey

Luckily and happily, I’m always in an ongoing pursuit to level up my knowledge. For me personally, I’ve found the best way I learn is not by reading, but by building instead. With that, I present to you my latest online application:

Starting simple and small, MyDigests’ current purpose is to deliver a daily sales digest to retailers using the Vend POS Retail platform. We have much larger future goals, but we plan to grow with the feedback of our users into different data sources and broadcasting mediums. So far it’s been an interesting journey. Without a doubt, I’m an engineer not a product manager!

Although the techs enclosed aren’t necessarily bleeding edge in all circumstances, good tech does not stay static long! And more importantly, why be good with a technology when you can have a mastery instead?

By no means a definitive list (it will most likely grow) please stay tuned as I share my learnings on:

  • Why Golang?
  • An Application Structure
  • Mongo Much?
  • Schedules and Queues
  • API JSON Spec
  • Frontend Folly
  • Templating
  • Google Cloud vs Amazon
  • Kubernetes

Hopefully my learnings can be used as a guide, and if nothing else, I hope you’re inspired to build something for yourself too! Have a specific request or question? Please reach out and I will gladly oblige where able.

Golang Templates - An Addendum

Dealing with DOM Ready.

Note: If you haven’t been playing the at home game, then I suggest you read the first post here, otherwise let’s assume that you have a directory structure like the following.

├── _footer.html
├── _header.html
├── layout.html
├── templates.go
└── user
    ├── dashboard.html

The problem: ensuring that any JS snippets defined throughout your application templates are executed with a valid DOM available. Granted, it would be best practice for you to combine all of your .js files and then load the src in the head using the async attribute. But even a snippet in the _header.html may still fail without jQuery loaded.

We can do better.

Let’s collect any/all js snippets from any/all of the available templates and then render them at the last possible moment in the DOM.


First off we will create an accessible template function to pass our js snippets.

jsSnippets := []template.JS{}
tplFuncs := template.FuncMap{
    // This template method just appends js snippets. The
    // Render() method is responsible for concatenation and
    // adding to the base layouts content.
    "domReady": func(js string) string {
        jsSnippets = append(jsSnippets, template.JS(js))
        return ""

A couple things to note, any function in a template.FuncMap must return either a single value or x2 values where the 2nd is of type error. As we are not concerned with the return value here, we just return an empty string.

The purpose of the function is to store your js string snippets as a template.JS value for future rendering. Then in your template files call

// The use of back ticks `s removes any single/double
// quote shenanigans.
{{domReady `console.log("All of the JS")`}}

First failures.

Ok so now we have our snippets gathered from our templates, we just need to pass them up to our outer layout…..ummm sorry? Let’s recap on our render method.

func Render(.....) {
    // At first we thought that we can iterate over
    // our js snippets now and assign to the template
    // data. But it's always empty!
    var domReadyJS template.JS
    if 0 < len(jsSnippets) {
        for _, jq := range jsSnippets {
            domReadyJS = domReadyJS + jq
    data["domReadyJS"] = domReadyJS
    layoutClone, _ := GetLayout(l).Clone()
    err := layoutClone.Execute(w, data)
    if err != nil {
        fmt.Fprintf(w, errorTemplate, name, err.Error())

My first attempt to implement of course failed due to a chicken/egg scenario. When the Render() method is called, the layout will always have an empty slice of js snippets! This is because it’s not until the enclosed named template is executed that any calls to the domReady function will be invoked.

You can always get out!

If you were thinking about how to hook into the templates to pass values to the layouts, take it from me, don’t. I tried. The better solution is of course to buffer the content of the layout, check for js snippets and then write the results to an outer template.

For a full example, refer to the this snippet -

With this logic, you can call {{domReady}} multiple times from any layout/template and have the results combined and rendered at the base of the page. Much better!

PS: In case you’re wondering, the chicken ALWAYS came before the egg.

Golang Templates - What I missed!


Don’t include {{define}} calls in your templates parsed with template.ParseFiles(). Know that template.ParseFiles() will overwrite any reference to a template with the same filename, not filename and path! Go look at the way I now parse my application templates here - Otherwise feel free to stick around and read why =]

So what’s the issue?

Having happily built out the backend of a little Golang app I’ve been working on, I’ve refocused my attention to the frontend. In general, I thought I had a ok understanding of how my html templates were being referred to and rendered. But of course all that changed when I wanted to add a partial to my main application layout.

As you read on, I will refer back to the official docs for some aha moments. To be honest however, I haven’t found the official docs terribly helpful in my day to day template usage. Yes of course “everything is in there”, but I’m yet to find a good online resource with examples for Go templating? In fact a lot of my templating knowledge came from this website which is built with Hugo! Let’s see if I can shed a little more light on the subject with this contrived example.

A typical setup?

├── _footer.html
├── _header.html
├── layout.html
├── sign
│   ├── index.html
├── templates.go
└── user
    ├── dashboard.html
    ├── index.html

// And here are the contents of the layout.html file.
{{define "app/layout"}}

With this single layout, we have defined one template func yield which will be used to render an inner templates contents. Of course to get the most out of my apps, I parse all of the available templates on the init() call. This means on execute, we only need to parse the data to render the template. The structure above and layout contents look fairly innocuous but I’m sure to the seasoned trained eye, you can see a couple problems.

How many layouts???

To parse the layout, we may use this code block

var (
    layout *template.Template
    layoutFuncs = template.FuncMap{
        "yield": func() (string, error) {
            return "", fmt.Errorf("yield called unexpectedly.")

func init() {
    layout = template.New("layout").Funcs(layoutFuncs)
    layout = template.Must(

We create a package level var of layout and we populate this value with layout = template.New("layout").Funcs(layoutFuncs). By assigning a default template func, the code will at least compile rather than error with function "yield" not defined. Then we attempt to parse our actual layout file with template.Must(layout.ParseFiles("./templates/layout.html")). Note the use of template.Must() which will panic on any error.

The interesting thing about this example call is that layout does not now contain one template instance, it actually contains three! Let’s check this by ranging over the layout templates with this extremely useful snippet.

for _, t := range layout.Templates() {

// Which would output
~$ go run main.go

The layout and the layout.html made sense to me. One created by the template.New("layout") call, the other being the filename returned on template.ParseFiles(). But the third value of app/layout showed a hole in my knowledge. It’s due to using the define keyword in the layout

// And here are the contents of the layout.html file.
{{define "app/layout"}}

This also defines a new template as denoted in the docs. Of course writing it here it’s plain as day but my confusion was well founded as it stemmed from a book I referenced early on in my Go development. Here’s an excerpt

As you can see, we have an html file layout with an embedded {{define}} call. Effectively we are now parsing a file template that has an embedded template to parse too! Of course you’re free to refer to your templates this way but personally I wouldn’t. I much rather computing less and using the explicit value of the filename. Especially with partials eg;

<!DOCTYPE html>
{{template "_header.html"}}

Sorry how many templates???

Ok so far so good, I’ve removed all/any of my {{define calls inside my html files. Now lets parse the rest of my templates using the templates.ParseGlob() function.

├── _footer.html
├── _header.html
├── layout.html
├── sign
│   ├── index.html
└── user
    ├── dashboard.html
    ├── index.html

templates = template.New("templates")
templates = template.Must(

The pattern ./templates/*/*.html tells ParseGlob to find any folder nested inside the templates directory and find any file with an .html extension. From the directory structure above, you’d be forgiven for thinking you will have four templates. One for the initial New() call of templates, one for index.html in the sign directory, and another x2 files in the user directory. But you be wrong! Lets foreach and see what we get back

// Only THREE templates!
~$ go run main.go

The docs are pretty clear on this one, but it’s still a trick for young players. Any duplicate files found will be overridden. The last file trumps and this came as a real surprise to me! In all my experience developing thus far, a template has always been referred to by its name and its path to stop overwrites from occurring.

A solution?

Ok so now we know a little more, let’s see if we can get the desired outcome. We know that the ParseGlob hands off to the _parseFiles which in turn will return the basename on any file found. We could write our own _parseFiles method and return what we need, but I have a better solution. Instead you can use the "path/filepath" package and walk the files directly. On each file you find, you can call New() on a template instance and then name the template to whatever you like! Something similar to

const trimPrefix = "templates/"

ts, err := filepath.Glob("./templates/*/*.html")
if err != nil {

templates = template.New("templates")
for _, t := range ts {
    b, err := ioutil.ReadFile(t)
    if err != nil {
    name := strings.TrimPrefix(t, trimPrefix)

This way I name each of my files to that of the filename and path removing the prefix “/templates”. So templates/sign/index.html becomes sign/index.html, much better! If you want the full example of how I am currently rendering my application templates, head over to the snippet

Wrap up

Initially I thought I was going nuts when trying to render my application templates. But after a little more digging and a little more reading I’m happy with the solution I’ve thus far settled with. Hopefully my learnings can be used to help another would be Gopher!


The culprit of single template names

The excerpt came from this book which was a great introduction into programming with Go Level up your web app with Go. But like all books, the art of doing is a much better teacher!

Qupid - A Golang Queue

In my experience working with Golang thus far, I encounter a pretty regular requirement to fetch some sort of data, process the data accordingly and then persist or send the output. Naturally the use of Golangs’ channels really shines here, but you still need a little orchestration to tie your channels together. Publish to one and subscribe to another along with the ability to stop processing as and when required too!

But how do you stop processing duplicate records? Take the following simple example.

                     |           |         
                     | Datastore |         
                ^^^  |           | ^^^     
                |||  |           | |||     
                |||  +-----------+ |||     
                |||                |||     
          Poll  |||                ||| Save
                |||                |||     
                |||   +---------+  |||     
                |||   |         |  |||     
                |||   |   App   |  |||     
                vvv   |         |  |||     
                      |         |          

In this example the App polls a datastore to find records that need to be processed and then persists them back. Note that there is no pushing or event dispatchers here =] When it comes to ensuring consistency, one would often leave that up to a 3rd party vendor, eg database transactions.

But rather than force that logic down the chain into a black box, I find it’s much better to bring that logic into plain view at the code level. In this way anyone involved has the same viewpoint, can identify issues and contribute with patches and tests. In addition, what do you do when unable to rely on a database transaction, when retrieving data from multiple external sources? Something more distributed like this:

+-----------+         +---------+        +-----------+
|           |         |         |        |           |
|    API    |   Poll  |   App   |  Save  |    NOT    |
|  Resource |<------->|         |------->|   S3!!!   |
|           |         |         |        |           |
+-----------+         +---------+        +-----------+

Clearly you can’t rely on a database transaction here! Because I found myself repeatedly dealing with the generic structure above, I’ve created a Golang package to take the hassle out of queues and channels so you can concentrate on the important stuff!

Called Qupid, you can head over to Gitlab and view the source code

To handle state locking and keeping things mutually exclusive, Golang offers the sync package and it is utilised so that completed queue entries do not (by default) get reprocessed. Although you can of course explicitly remove entries as required or auto purge completed results! If you want to know more, you can see all of that and more via the source code and the example file.

As always I’m happy to take on comments and feedback and of course with ANY distributed system, YOU need to ensure that your work is idempotent!!!!

Happy coding =]

It's where the magic happens © 2018 - KyleHQ