Golang Templates - What I missed!


Don’t include {{define}} calls in your templates parsed with template.ParseFiles(). Know that template.ParseFiles() will overwrite any reference to a template with the same filename, not filename and path! Go look at the way I now parse my application templates here - https://gitlab.com/snippets/1662623. Otherwise feel free to stick around and read why =]

So what’s the issue?

Having happily built out the backend of a little Golang app I’ve been working on, I’ve refocused my attention to the frontend. In general, I thought I had a ok understanding of how my html templates were being referred to and rendered. But of course all that changed when I wanted to add a partial to my main application layout.

As you read on, I will refer back to the official docs for some aha moments. To be honest however, I haven’t found the official docs terribly helpful in my day to day template usage. Yes of course “everything is in there”, but I’m yet to find a good online resource with examples for Go templating? In fact a lot of my templating knowledge came from this website which is built with Hugo! Let’s see if I can shed a little more light on the subject with this contrived example.

A typical setup?

├── _footer.html
├── _header.html
├── layout.html
├── sign
│   ├── index.html
├── templates.go
└── user
    ├── dashboard.html
    ├── index.html

// And here are the contents of the layout.html file.
{{define "app/layout"}}

With this single layout, we have defined one template func yield which will be used to render an inner templates contents. Of course to get the most out of my apps, I parse all of the available templates on the init() call. This means on execute, we only need to parse the data to render the template. The structure above and layout contents look fairly innocuous but I’m sure to the seasoned trained eye, you can see a couple problems.

How many layouts???

To parse the layout, we may use this code block

var (
    layout *template.Template
    layoutFuncs = template.FuncMap{
        "yield": func() (string, error) {
            return "", fmt.Errorf("yield called unexpectedly.")

func init() {
    layout = template.New("layout").Funcs(layoutFuncs)
    layout = template.Must(

We create a package level var of layout and we populate this value with layout = template.New("layout").Funcs(layoutFuncs). By assigning a default template func, the code will at least compile rather than error with function "yield" not defined. Then we attempt to parse our actual layout file with template.Must(layout.ParseFiles("./templates/layout.html")). Note the use of template.Must() which will panic on any error.

The interesting thing about this example call is that layout does not now contain one template instance, it actually contains three! Let’s check this by ranging over the layout templates with this extremely useful snippet.

for _, t := range layout.Templates() {

// Which would output
~$ go run main.go

The layout and the layout.html made sense to me. One created by the template.New("layout") call, the other being the filename returned on template.ParseFiles(). But the third value of app/layout showed a hole in my knowledge. It’s due to using the define keyword in the layout

// And here are the contents of the layout.html file.
{{define "app/layout"}}

This also defines a new template as denoted in the docs. Of course writing it here it’s plain as day but my confusion was well founded as it stemmed from a book I referenced early on in my Go development. Here’s an excerpt

As you can see, we have an html file layout with an embedded {{define}} call. Effectively we are now parsing a file template that has an embedded template to parse too! Of course you’re free to refer to your templates this way but personally I wouldn’t. I much rather computing less and using the explicit value of the filename. Especially with partials eg;

<!DOCTYPE html>
{{template "_header.html"}}

Sorry how many templates???

Ok so far so good, I’ve removed all/any of my {{define calls inside my html files. Now lets parse the rest of my templates using the templates.ParseGlob() function.

├── _footer.html
├── _header.html
├── layout.html
├── sign
│   ├── index.html
└── user
    ├── dashboard.html
    ├── index.html

templates = template.New("templates")
templates = template.Must(

The pattern ./templates/*/*.html tells ParseGlob to find any folder nested inside the templates directory and find any file with an .html extension. From the directory structure above, you’d be forgiven for thinking you will have four templates. One for the initial New() call of templates, one for index.html in the sign directory, and another x2 files in the user directory. But you be wrong! Lets foreach and see what we get back

// Only THREE templates!
~$ go run main.go

The docs are pretty clear on this one, but it’s still a trick for young players. Any duplicate files found will be overridden. The last file trumps and this came as a real surprise to me! In all my experience developing thus far, a template has always been referred to by its name and its path to stop overwrites from occurring.

A solution?

Ok so now we know a little more, let’s see if we can get the desired outcome. We know that the ParseGlob hands off to the _parseFiles which in turn will return the basename on any file found. We could write our own _parseFiles method and return what we need, but I have a better solution. Instead you can use the "path/filepath" package and walk the files directly. On each file you find, you can call New() on a template instance and then name the template to whatever you like! Something similar to

const trimPrefix = "templates/"

ts, err := filepath.Glob("./templates/*/*.html")
if err != nil {

templates = template.New("templates")
for _, t := range ts {
    b, err := ioutil.ReadFile(t)
    if err != nil {
    name := strings.TrimPrefix(t, trimPrefix)

This way I name each of my files to that of the filename and path removing the prefix “/templates”. So templates/sign/index.html becomes sign/index.html, much better! If you want the full example of how I am currently rendering my application templates, head over to the snippet https://gitlab.com/snippets/1662623

Wrap up

Initially I thought I was going nuts when trying to render my application templates. But after a little more digging and a little more reading I’m happy with the solution I’ve thus far settled with. Hopefully my learnings can be used to help another would be Gopher!


The culprit of single template names https://golang.org/src/text/template/helper.go

The excerpt came from this book which was a great introduction into programming with Go Level up your web app with Go. But like all books, the art of doing is a much better teacher!

Qupid - A Golang Queue

In my experience working with Golang thus far, I encounter a pretty regular requirement to fetch some sort of data, process the data accordingly and then persist or send the output. Naturally the use of Golangs’ channels really shines here, but you still need a little orchestration to tie your channels together. Publish to one and subscribe to another along with the ability to stop processing as and when required too!

But how do you stop processing duplicate records? Take the following simple example.

                     |           |         
                     | Datastore |         
                ^^^  |           | ^^^     
                |||  |           | |||     
                |||  +-----------+ |||     
                |||                |||     
          Poll  |||                ||| Save
                |||                |||     
                |||   +---------+  |||     
                |||   |         |  |||     
                |||   |   App   |  |||     
                vvv   |         |  |||     
                      |         |          

In this example the App polls a datastore to find records that need to be processed and then persists them back. Note that there is no pushing or event dispatchers here =] When it comes to ensuring consistency, one would often leave that up to a 3rd party vendor, eg database transactions.

But rather than force that logic down the chain into a black box, I find it’s much better to bring that logic into plain view at the code level. In this way anyone involved has the same viewpoint, can identify issues and contribute with patches and tests. In addition, what do you do when unable to rely on a database transaction, when retrieving data from multiple external sources? Something more distributed like this:

+-----------+         +---------+        +-----------+
|           |         |         |        |           |
|    API    |   Poll  |   App   |  Save  |    NOT    |
|  Resource |<------->|         |------->|   S3!!!   |
|           |         |         |        |           |
+-----------+         +---------+        +-----------+

Clearly you can’t rely on a database transaction here! Because I found myself repeatedly dealing with the generic structure above, I’ve created a Golang package to take the hassle out of queues and channels so you can concentrate on the important stuff!

Called Qupid, you can head over to Gitlab and view the source code


To handle state locking and keeping things mutually exclusive, Golang offers the sync package and it is utilised so that completed queue entries do not (by default) get reprocessed. Although you can of course explicitly remove entries as required or auto purge completed results! If you want to know more, you can see all of that and more via the source code and the example file.

As always I’m happy to take on comments and feedback and of course with ANY distributed system, YOU need to ensure that your work is idempotent!!!!

Happy coding =]

What JSON?

If you’re one to consume JSON api’s (who isn’t really?), then it’s likely you’ve bumped into the issue of figuring out whether the resource you just called is returning a single JSON object or an array payload. As per the JSON API Spec,

The document’s “primary data” is a representation of the resource
or collection of resources targeted by a request.

Primary data MUST be either:

a single resource object, a single resource identifier object, or null, for
requests that target single resources

an array of resource objects, an array of resource identifier objects, or
an empty array ([]), for requests that target resource collections

To help out all the Gophers out there, I created a simple little helper package to figure it out for you.

Checkout the code here https://gitlab.com/kylehqcom/whatjson.

It’s pretty simple really, so instead of having a dependency to my repo, just copy paste into your project instead.


Just call WhatJSON() with a body of type []byte and an optional prefix. The code will check the very next char after the prefix has been matched to determine whether the body is a object or an array. Go check out the code already!

Short n sweet, enjoy =]

Lessons in Golang - Goroutines and Channels

Recently I made publicly available an in memory scheduler written in Golang, https://gitlab.com/kylehqcom/kevin.

Writing code for public consumption is always something of a scary exercise. Knowing full well the critiques, scrutiny and chin rubbing that will come from your peers and non peers alike. But being an optimist I appreciate any feedback. Good or bad it is always greatly received as it all just helps with the learning. My goal here is to share some of that learning, mistakes and “a ha” moments I have found in developing with Go.

Long may it continue!


In my own time I have been writing a new SaaS. It’s currently private but details will emerge as and when the beta rolls out. The project itself is made up of 3 distinctive parts, a getter, a cruncher and a sender. Starting with the “getter”, it became evident pretty early on that I would need a way to schedule “gets”. But such a scheduler would come in super handy later on with my “sender” too. I first took a look at the existing open source packages.

From what I found, most were overkill for what I needed. Sure AirBnB’s chronos and even the “trimmed” down version kala would do the job, but have way too much overhead for me at this time.

At the other end of the scale, most took a direct linux cron job approach. gocron will certainly repeat tasks but didn’t have a nice way to manipulate running tasks or retrieve details from these tasks for introspection. It’s at this point that I took inspiration from the above and rolled up my sleeves. For reference, take a look at Kevin’s source.

Goroutines and Channels

Because my own requirements didn’t require a persistent store to track running jobs, implementing a schedule pool was as simple as creating a map key’d on a string.

type schedules map[string]*schedule

This allowed for the returning of a schedule based on a generic name, easy enough. When adding a Job to a Schedule, after validation, the details of the job are submitted in a non blocking *Goroutine. This ensured that any surrounding code can continue to execute and the job would happily run in the background. On submission, if an ID was not passed, a generated ID was returned to introspect the job in future. The job runner itself utilises Golang’s built in


to repeat the actual function calls assigned to the job. So nothing terribly scary or out of the ordinary happening here. Well that was until I wanted to use an ID to stop a running job. Although at the time my “knowledge” of the problem was a wee bit lacking, I was pretty confident I could get the desired outcome.

The question was, how can I access the values of a function running in a background Goroutine?

As far as I was concerned “at the time”, each Goroutine is living in some “forbidden zone” away from variable scope. This left me with one option, using some well intentioned channels. Channels (among other things) are a way for Goroutines to communicate with each other. But how would I use a **stop channel for a pool of schedules with numerous jobs??? The internet is your friend.

After screening though numerous Stackoverflow results, my face made a rye smile when I finally found my way to @matreyer - I had been in contact with Mat earlier in the year in regards to a GoLang position so I wasn’t surprised to find his repo to handle my exact use case - https://github.com/matryer/runner Bam happy days! I now had a way to stop a Goroutine from the inside out.

But like most things, I wasn’t entirely happy with this initial implementation. I took issue with the duration between stop being called from the outside and the job timer checking for the stopped status. In short, a ticker is an infinite “for loop” in which you “break” out of or stop().

// do is called via Run in a go routine.
func do(r *runner) {
      // This will tick on "Every" forever.
      e := time.NewTicker(r.job.Every)
      for range e.C {
          // Work happens
          // But you can stop via a channel and 
          // break from the timer.
          if <- stop {

Take for example a job that gets an “Every” value of one week. If we are one day into the week and a stop is called on the job, the runner will then sit and wait for another 6 days before the ticker ticks and stops itself. I didn’t want that wait so set out to find a solution.

Curiosity for the win.

After hours of research, code attempts, Goroutine and channel examples I “happened” upon this playground example. Please go take a look.


From this example, it is clear that Goroutines still have access to variables in package scope. Notice that var x int is being updated by one Goroutine, and then being rendered from another. Now that I know that Goroutines can access a variable in scope, then surely an out of scope variable can be replaced with a pointer instead?

A pointer is a memory address to a variable. In past code I had used pointers for memory optimisations or to ensure a single point of truth, I just hadn’t applied the same thoughts to Goroutines. So I’m sitting here thinking, if a Goroutine has access to a variable pointer from inside the ticker loop, then why do we even need these channels at all? Answer is you don’t!

*** My current implementation uses a pointer of a runner instance https://gitlab.com/kylehqcom/kevin/blob/master/runner.go#L172 and the runner instance is assigned against the schedule map via a JobID. This ensures that any updates to the stopped flag on the runner instance can be read by the ticker and will stop/break as required. Winning!

As much as I enjoyed implementing @matreyer’s package, it’s much more enjoyable losing an external dependency!

The Lesson

Channels and Goroutines are awesome and fun, but use them only when required. Often a simpler solution can be found. Even without Mat’s runner package, I am still bound to a ticker being stopped based on it’s Every duration. But I’m glad I looked for a better solution. I now have a deeper understanding and my Golang just levelled up.

*There are plenty of resources about Goroutines online so please “go” have a read if they are new to you.

** Interestingly, I could still have used a single channel sending JobID’s. Though as I currently see it, this would mean that every runner would get ALL job updates. JobID values would be sent down the channel and runners would have to self check themselves for a JobID match. Terribly inefficient.

*** Addendum: the current implementation uses the closing of an empty struct channel to exit the ticker.

29 Days in Murica No.2

Las Vegas

I certainly had expectations before entering Las Vegas. Most were met. But other experiences I encountered I hadn’t expected at all!

I mean the bar was well ‘n truly set exiting the plane “offramp”, walking through the airport doors and being faced with an entire wall of slot machines. Forget airport security, just give us some coin! The start of an underlying theme was beginning to occur.

Making our way from our gated “trailer park” community towards the “strip”, the shear scale began to bed in. Where else would you find an Eiffel tower outside of Paris, or a pyramid outside of Egypt?

In a photo frenzy we walk past the MGM, through Caesars palace, find the most ridiculously sized candy bars and rollercoaster our way between casinos. We continue past all the “minions” busking for photo moments, the stampede of “stag dooz” and “hen’s nights” that are all “so original”.

Finally reaching the strips’ end. With the constant fend against hustlers passing out their stripper flyers, we take refuge from all the flashy lights, and enjoy a moment to reminisce the Oceans’ Eleven closing scenes.

And then, after leaving the calm of the water cascades, in a subtle instant, it all becomes perfectly clear.

The whole thing just wreaks of desperation. And I found it to be really, really sad.

Barhumbug I know! Maybe if I was in my early twenties, drugged to the eyeballs with a fist full of dollars. On the lookout for shady shenanigans and the like, then this place makes a whole lotta sense. But that’s not what I saw here. Instead I saw middle aged men ++, clinging to some glimmering hope that their ticket or the next dice roll is “the” winner. That the extra forty pounds hanging around their midriff is sexually appealing. That the greasy slicked back hair is the way all the “kool kids” are doing it.

And regrettably, the desperation wasn’t gender specific. Although mostly paraded as eye candy in paying “professional” places of work, women acting as public patrons fell victim too. Plastic breasts that practically chin touch, aim for the sunrise or shoot for the stars. Short shorts and skirts that act as reminders of a cheerleading beer commercial career that “could have been”


Sometimes it’s just very hard to write about an experience without inferring a cast or judgement. I clearly have no prejudice against the people or actions I have seen. And I in no way have any knowledge of the hardships or experiences any of these people may have had. I guess I just got to look through the looking glass and came out confidently saying “that’s not the life for me”.

I am going to go an earn my successes, not gamble them. I only wish the same for you, whatever your successes may be.

It's where the magic happens © 2019 - KyleHQ