Learn Go with tests
main
Contribute on github
Search…
⌃K
Links

IO and sorting

​In the previous chapter we continued iterating on our application by adding a new endpoint /league. Along the way we learned about how to deal with JSON, embedding types and routing.
Our product owner is somewhat perturbed by the software losing the scores when the server was restarted. This is because our implementation of our store is in-memory. She is also not pleased that we didn't interpret the /league endpoint should return the players ordered by the number of wins!

The code so far

// server.go
package main
​
import (
"encoding/json"
"fmt"
"net/http"
"strings"
)
​
// PlayerStore stores score information about players
type PlayerStore interface {
GetPlayerScore(name string) int
RecordWin(name string)
GetLeague() []Player
}
​
// Player stores a name with a number of wins
type Player struct {
Name string
Wins int
}
​
// PlayerServer is a HTTP interface for player information
type PlayerServer struct {
store PlayerStore
http.Handler
}
​
const jsonContentType = "application/json"
​
// NewPlayerServer creates a PlayerServer with routing configured
func NewPlayerServer(store PlayerStore) *PlayerServer {
p := new(PlayerServer)
​
p.store = store
​
router := http.NewServeMux()
router.Handle("/league", http.HandlerFunc(p.leagueHandler))
router.Handle("/players/", http.HandlerFunc(p.playersHandler))
​
p.Handler = router
​
return p
}
​
func (p *PlayerServer) leagueHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("content-type", jsonContentType)
json.NewEncoder(w).Encode(p.store.GetLeague())
}
​
func (p *PlayerServer) playersHandler(w http.ResponseWriter, r *http.Request) {
player := strings.TrimPrefix(r.URL.Path, "/players/")
​
switch r.Method {
case http.MethodPost:
p.processWin(w, player)
case http.MethodGet:
p.showScore(w, player)
}
}
​
func (p *PlayerServer) showScore(w http.ResponseWriter, player string) {
score := p.store.GetPlayerScore(player)
​
if score == 0 {
w.WriteHeader(http.StatusNotFound)
}
​
fmt.Fprint(w, score)
}
​
func (p *PlayerServer) processWin(w http.ResponseWriter, player string) {
p.store.RecordWin(player)
w.WriteHeader(http.StatusAccepted)
}
// in_memory_player_store.go
package main
​
func NewInMemoryPlayerStore() *InMemoryPlayerStore {
return &InMemoryPlayerStore{map[string]int{}}
}
​
type InMemoryPlayerStore struct {
store map[string]int
}
​
func (i *InMemoryPlayerStore) GetLeague() []Player {
var league []Player
for name, wins := range i.store {
league = append(league, Player{name, wins})
}
return league
}
​
func (i *InMemoryPlayerStore) RecordWin(name string) {
i.store[name]++
}
​
func (i *InMemoryPlayerStore) GetPlayerScore(name string) int {
return i.store[name]
}
// main.go
package main
​
import (
"log"
"net/http"
)
​
func main() {
server := NewPlayerServer(NewInMemoryPlayerStore())
log.Fatal(http.ListenAndServe(":5000", server))
}
You can find the corresponding tests in the link at the top of the chapter.

Store the data

There are dozens of databases we could use for this but we're going to go for a very simple approach. We're going to store the data for this application in a file as JSON.
This keeps the data very portable and is relatively simple to implement.
It won't scale especially well but given this is a prototype it'll be fine for now. If our circumstances change and it's no longer appropriate it'll be simple to swap it out for something different because of the PlayerStore abstraction we have used.
We will keep the InMemoryPlayerStore for now so that the integration tests keep passing as we develop our new store. Once we are confident our new implementation is sufficient to make the integration test pass we will swap it in and then delete InMemoryPlayerStore.

Write the test first

By now you should be familiar with the interfaces around the standard library for reading data (io.Reader), writing data (io.Writer) and how we can use the standard library to test these functions without having to use real files.
For this work to be complete we'll need to implement PlayerStore so we'll write tests for our store calling the methods we need to implement. We'll start with GetLeague.
//file_system_store_test.go
func TestFileSystemStore(t *testing.T) {
​
t.Run("league from a reader", func(t *testing.T) {
database := strings.NewReader(`[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
​
store := FileSystemPlayerStore{database}
​
got := store.GetLeague()
​
want := []Player{
{"Cleo", 10},
{"Chris", 33},
}
​
assertLeague(t, got, want)
})
}
We're using strings.NewReader which will return us a Reader, which is what our FileSystemPlayerStore will use to read data. In main we will open a file, which is also a Reader.

Try to run the test

# github.com/quii/learn-go-with-tests/io/v1
./file_system_store_test.go:15:12: undefined: FileSystemPlayerStore

Write the minimal amount of code for the test to run and check the failing test output

Let's define FileSystemPlayerStore in a new file
//file_system_store.go
type FileSystemPlayerStore struct{}
Try again
# github.com/quii/learn-go-with-tests/io/v1
./file_system_store_test.go:15:28: too many values in struct initializer
./file_system_store_test.go:17:15: store.GetLeague undefined (type FileSystemPlayerStore has no field or method GetLeague)
It's complaining because we're passing in a Reader but not expecting one and it doesn't have GetLeague defined yet.
//file_system_store.go
type FileSystemPlayerStore struct {
database io.Reader
}
​
func (f *FileSystemPlayerStore) GetLeague() []Player {
return nil
}
One more try...
=== RUN TestFileSystemStore//league_from_a_reader
--- FAIL: TestFileSystemStore//league_from_a_reader (0.00s)
file_system_store_test.go:24: got [] want [{Cleo 10} {Chris 33}]

Write enough code to make it pass

We've read JSON from a reader before
//file_system_store.go
func (f *FileSystemPlayerStore) GetLeague() []Player {
var league []Player
json.NewDecoder(f.database).Decode(&league)
return league
}
The test should pass.

Refactor

We have done this before! Our test code for the server had to decode the JSON from the response.
Let's try DRYing this up into a function.
Create a new file called league.go and put this inside.
//league.go
func NewLeague(rdr io.Reader) ([]Player, error) {
var league []Player
err := json.NewDecoder(rdr).Decode(&league)
if err != nil {
err = fmt.Errorf("problem parsing league, %v", err)
}
​
return league, err
}
Call this in our implementation and in our test helper getLeagueFromResponse in server_test.go
//file_system_store.go
func (f *FileSystemPlayerStore) GetLeague() []Player {
league, _ := NewLeague(f.database)
return league
}
We haven't got a strategy yet for dealing with parsing errors but let's press on.

Seeking problems

There is a flaw in our implementation. First of all, let's remind ourselves how io.Reader is defined.
type Reader interface {
Read(p []byte) (n int, err error)
}
With our file, you can imagine it reading through byte by byte until the end. What happens if you try to Read a second time?
Add the following to the end of our current test.
//file_system_store_test.go
​
// read again
got = store.GetLeague()
assertLeague(t, got, want)
We want this to pass, but if you run the test it doesn't.
The problem is our Reader has reached the end so there is nothing more to read. We need a way to tell it to go back to the start.
​ReadSeeker is another interface in the standard library that can help.
type ReadSeeker interface {
Reader
Seeker
}
Remember embedding? This is an interface comprised of Reader and Seeker​
type Seeker interface {
Seek(offset int64, whence int) (int64, error)
}
This sounds good, can we change FileSystemPlayerStore to take this interface instead?
//file_system_store.go
type FileSystemPlayerStore struct {
database io.ReadSeeker
}
​
func (f *FileSystemPlayerStore) GetLeague() []Player {
f.database.Seek(0, 0)
league, _ := NewLeague(f.database)
return league
}
Try running the test, it now passes! Happily for us strings.NewReader that we used in our test also implements ReadSeeker so we didn't have to make any other changes.
Next we'll implement GetPlayerScore.

Write the test first

//file_system_store_test.go
t.Run("get player score", func(t *testing.T) {
database := strings.NewReader(`[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
​
store := FileSystemPlayerStore{database}
​
got := store.GetPlayerScore("Chris")
​
want := 33
​
if got != want {
t.Errorf("got %d want %d", got, want)
}
})

Try to run the test

./file_system_store_test.go:38:15: store.GetPlayerScore undefined (type FileSystemPlayerStore has no field or method GetPlayerScore)

Write the minimal amount of code for the test to run and check the failing test output

We need to add the method to our new type to get the test to compile.
//file_system_store.go
func (f *FileSystemPlayerStore) GetPlayerScore(name string) int {
return 0
}
Now it compiles and the test fails
=== RUN TestFileSystemStore/get_player_score
--- FAIL: TestFileSystemStore//get_player_score (0.00s)
file_system_store_test.go:43: got 0 want 33

Write enough code to make it pass

We can iterate over the league to find the player and return their score
//file_system_store.go
func (f *FileSystemPlayerStore) GetPlayerScore(name string) int {
​
var wins int
​
for _, player := range f.GetLeague() {
if player.Name == name {
wins = player.Wins
break
}
}
​
return wins
}

Refactor

You will have seen dozens of test helper refactorings so I'll leave this to you to make it work
//file_system_store_test.go
t.Run("get player score", func(t *testing.T) {
database := strings.NewReader(`[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
​
store := FileSystemPlayerStore{database}
​
got := store.GetPlayerScore("Chris")
want := 33
assertScoreEquals(t, got, want)
})
Finally, we need to start recording scores with RecordWin.

Write the test first

Our approach is fairly short-sighted for writes. We can't (easily) just update one "row" of JSON in a file. We'll need to store the whole new representation of our database on every write.
How do we write? We'd normally use a Writer but we already have our ReadSeeker. Potentially we could have two dependencies but the standard library already has an interface for us ReadWriteSeeker which lets us do all the things we'll need to do with a file.
Let's update our type
//file_system_store.go
type FileSystemPlayerStore struct {
database io.ReadWriteSeeker
}
See if it compiles
./file_system_store_test.go:15:34: cannot use database (type *strings.Reader) as type io.ReadWriteSeeker in field value:
*strings.Reader does not implement io.ReadWriteSeeker (missing Write method)
./file_system_store_test.go:36:34: cannot use database (type *strings.Reader) as type io.ReadWriteSeeker in field value:
*strings.Reader does not implement io.ReadWriteSeeker (missing Write method)
It's not too surprising that strings.Reader does not implement ReadWriteSeeker so what do we do?
We have two choices
  • Create a temporary file for each test. *os.File implements ReadWriteSeeker. The pro of this is it becomes more of an integration test, we're really reading and writing from the file system so it will give us a very high level of confidence. The cons are we prefer unit tests because they are faster and generally simpler. We will also need to do more work around creating temporary files and then making sure they're removed after the test.
  • We could use a third party library. Mattetti has written a library filebuffer which implements the interface we need and doesn't touch the file system.
I don't think there's an especially wrong answer here, but by choosing to use a third party library I would have to explain dependency management! So we will use files instead.
Before adding our test we need to make our other tests compile by replacing the strings.Reader with an os.File.
Let's create some helper functions which will create a temporary file with some data inside it, and abstract our score tests
//file_system_store_test.go
func createTempFile(t testing.TB, initialData string) (io.ReadWriteSeeker, func()) {
t.Helper()
​
tmpfile, err := os.CreateTemp("", "db")
​
if err != nil {
t.Fatalf("could not create temp file %v", err)
}
​
tmpfile.Write([]byte(initialData))
​
removeFile := func() {
tmpfile.Close()
os.Remove(tmpfile.Name())
}
​
return tmpfile, removeFile
}
​
func assertScoreEquals(t testing.TB, got, want int) {
t.Helper()
if got != want {
t.Errorf("got %d want %d", got, want)
}
}
​CreateTemp creates a temporary file for us to use. The "db" value we've passed in is a prefix put on a random file name it will create. This is to ensure it won't clash with other files by accident.
You'll notice we're not only returning our ReadWriteSeeker (the file) but also a function. We need to make sure that the file is removed once the test is finished. We don't want to leak details of the files into the test as it's prone to error and uninteresting for the reader. By returning a removeFile function, we can take care of the details in our helper and all the caller has to do is run defer cleanDatabase().
//file_system_store_test.go
func TestFileSystemStore(t *testing.T) {
​
t.Run("league from a reader", func(t *testing.T) {
database, cleanDatabase := createTempFile(t, `[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
defer cleanDatabase()
​
store := FileSystemPlayerStore{database}
​
got := store.GetLeague()
​
want := []Player{
{"Cleo", 10},
{"Chris", 33},
}
​
assertLeague(t, got, want)
​
// read again
got = store.GetLeague()
assertLeague(t, got, want)
})
​
t.Run("get player score", func(t *testing.T) {
database, cleanDatabase := createTempFile(t, `[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
defer cleanDatabase()
​
store := FileSystemPlayerStore{database}
​
got := store.GetPlayerScore("Chris")
want := 33
assertScoreEquals(t, got, want)
})
}
Run the tests and they should be passing! There were a fair amount of changes but now it feels like we have our interface definition complete and it should be very easy to add new tests from now.
Let's get the first iteration of recording a win for an existing player
//file_system_store_test.go
t.Run("store wins for existing players", func(t *testing.T) {
database, cleanDatabase := createTempFile(t, `[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
defer cleanDatabase()
​
store := FileSystemPlayerStore{database}
​
store.RecordWin("Chris")
​
got := store.GetPlayerScore("Chris")
want := 34
assertScoreEquals(t, got, want)
})

Try to run the test

./file_system_store_test.go:67:8: store.RecordWin undefined (type FileSystemPlayerStore has no field or method RecordWin)

Write the minimal amount of code for the test to run and check the failing test output

Add the new method
//file_system_store.go
func (f *FileSystemPlayerStore) RecordWin(name string) {
​
}
=== RUN TestFileSystemStore/store_wins_for_existing_players
--- FAIL: TestFileSystemStore/store_wins_for_existing_players (0.00s)
file_system_store_test.go:71: got 33 want 34
Our implementation is empty so the old score is getting returned.

Write enough code to make it pass

//file_system_store.go
func (f *FileSystemPlayerStore) RecordWin(name string) {
league := f.GetLeague()
​
for i, player := range league {
if player.Name == name {
league[i].Wins++
}
}
​
f.database.Seek(0, 0)
json.NewEncoder(f.database).Encode(league)
}
You may be asking yourself why I am doing league[i].Wins++ rather than player.Wins++.
When you range over a slice you are returned the current index of the loop (in our case i) and a copy of the element at that index. Changing the Wins value of a copy won't have any effect on the league slice that we iterate on. For that reason, we need to get the reference to the actual value by doing league[i] and then changing that value instead.
If you run the tests, they should now be passing.

Refactor

In GetPlayerScore and RecordWin, we are iterating over []Player to find a player by name.
We could refactor this common code in the internals of FileSystemStore but to me, it feels like this is maybe useful code we can lift into a new type. Working with a "League" so far has always been with []Player but we can create a new type called League. This will be easier for other developers to understand and then we can attach useful methods onto that type for us to use.
Inside league.go add the following
//league.go
type League []Player
​
func (l League) Find(name string) *Player {
for i, p := range l {
if p.Name == name {
return &l[i]
}
}
return nil
}
Now if anyone has a League they can easily find a given player.
Change our PlayerStore interface to return League rather than []Player. Try to re-run the tests, you'll get a compilation problem because we've changed the interface but it's very easy to fix; just change the return type from []Player to League.
This lets us simplify our methods in file_system_store.
//file_system_store.go
func (f *FileSystemPlayerStore) GetPlayerScore(name string) int {
​
player := f.GetLeague().Find(name)
​
if player != nil {
return player.Wins
}
​
return 0
}
​
func (f *FileSystemPlayerStore) RecordWin(name string) {
league := f.GetLeague()
player := league.Find(name)
​
if player != nil {
player.Wins++
}
​
f.database.Seek(0, 0)
json.NewEncoder(f.database).Encode(league)
}
This is looking much better and we can see how we might be able to find other useful functionality around League that can be refactored.
We now need to handle the scenario of recording wins of new players.

Write the test first

//file_system_store_test.go
t.Run("store wins for new players", func(t *testing.T) {
database, cleanDatabase := createTempFile(t, `[
{"Name": "Cleo", "Wins": 10},
{"Name": "Chris", "Wins": 33}]`)
defer cleanDatabase()
​
store := FileSystemPlayerStore{database}
​
store.RecordWin("Pepper")
​
got := store.GetPlayerScore("Pepper")
want := 1
assertScoreEquals(t, got, want)
})

Try to run the test

=== RUN TestFileSystemStore/store_wins_for_new_players#01
--- FAIL: TestFileSystemStore/store_wins_for_new_players#01 (0.00s)
file_system_store_test.go:86: got 0 want 1

Write enough code to make it pass

We just need to handle the scenario where Find returns nil because it couldn't find the player.
//file_system_store.go
func (f *FileSystemPlayerStore) RecordWin(name string) {
league := f.GetLeague()
player := league.Find(name)
​
if player != nil {
player.Wins++
} else {
league = append(league, Player{name, 1})
}
​
f.database.Seek(0, 0)
json.NewEncoder(f.database).Encode(league)
}
The happy path is looking ok so we can now try using our new Store in the integration test. This will give us more confidence that the software works and then we can delete the redundant InMemoryPlayerStore.
In TestRecordingWinsAndRetrievingThem replace the old store.
//server_integration_test.go
database, cleanDatabase := createTempFile(t, "")
defer cleanDatabase()
store := &FileSystemPlayerStore{database}
If you run the test it should pass and now we can delete InMemoryPlayerStore. main.go will now have compilation problems which will motivate us to now use our new store in the "real" code.
//main.go
package main
​
import (
"log"
"net/http"
"os"
)
​
const dbFileName = "game.db.json"
​
func main() {
db, err := os.OpenFile(dbFileName, os.O_RDWR|os.O_CREATE, 0666)
​
if err != nil {
log.Fatalf("problem opening %s %v", dbFileName, err)
}
​
store := &FileSystemPlayerStore{db}
server := NewPlayerServer(store)
​
if err := http.ListenAndServe(":5000", server); err != nil {
log.Fatalf("could not listen on port 5000 %v", err)
}
}
  • We create a file for our database.
  • The 2nd argument to os.OpenFile lets you define the permissions for opening the file, in our case O_RDWR means we want to read and write and os.O_CREATE means create the file if it doesn't exist.
  • The 3rd argument means sets permissions for the file, in our case, all users can read and write the file. (See superuser.com for a more detailed explanation).
Running the program now persists the data in a file in between restarts, hooray!

More refactoring and performance concerns

Every time someone calls GetLeague() or GetPlayerScore() we are reading the entire file and parsing it into JSON. We should not have to do that because FileSystemStore is entirely responsible for the state of the league; it should only need to read the file when the program starts up and only need to update the file when data changes.
We can create a constructor which can do some of this initialisation for us and store the league as a value in our FileSystemStore to be used on the reads instead.
//file_system_store.go
type FileSystemPlayerStore struct {
database io.ReadWriteSeeker
league League
}
​
func NewFileSystemPlayerStore(database io.ReadWriteSeeker) *FileSystemPlayerStore {
database.Seek(0, 0)
league, _ := NewLeague(database)
return &FileSystemPlayerStore{
database: database,
league: league,
}
}
This way we only have to read from disk once. We can now replace all of our previous calls to getting the league from disk and just use f.league instead.
//file_system_store.go
func (f *FileSystemPlayerStore) GetLeague() League {
return f.league
}
​
func (f *FileSystemPlayerStore) GetPlayerScore(name string) int {
​
player := f.league.Find(name)
​
if player != nil {
return player.Wins
}
​
return 0
}
​
func (f *FileSystemPlayerStore) RecordWin(name string) {
player := f.league.Find(name)
​
if player != nil {
player.Wins++
} else {
f.league = append(f.league, Player{name, 1})
}
​
f.database.Seek(0, 0)
json.NewEncoder(f.database).Encode(f.league)
}
If you try to run the tests it will now complain about initialising FileSystemPlayerStore so just fix them by calling our new constructor.

Another problem

There is some more naivety in the way we are dealing with files which could create a very nasty bug down the line.
When we RecordWin, we Seek back to the start of the file and then write the new data—but what if the new data was smaller than what was there before?
In our current case, this is impossible. We never edit or delete scores so the data can only get bigger. However, it would be irresponsible for us to leave the code like this; it's not unthinkable that a delete scenario could come up.
How will we test for this though? What we need to do is first refactor our code so we separate out the concern of the kind of data we write, from the writing. We can then test that separately to check it works how we hope.
We'll create a new type to encapsulate our "when we write we go from the beginning" functionality. I'm going to call it Tape. Create a new file with the following:
//tape.go
package main
​
import "io"
​
type tape struct {
file io.ReadWriteSeeker
}
​
func (t *tape) Write(p []byte) (n int, err error) {
t.file.Seek(0, 0)
return t.file.Write(p)
}
Notice that we're only implementing Write now, as it encapsulates the Seek part. This means our FileSystemStore can just have a reference to a Writer instead.
//file_system_store.go
type FileSystemPlayerStore struct {
database io.Writer
league League
}
Update the constructor to use Tape
//file_system_store.go
func NewFileSystemPlayerStore(database io.ReadWriteSeeker) *FileSystemPlayerStore {
database.Seek(0, 0)
league, _ := NewLeague(database)
​
return &FileSystemPlayerStore{
database: &tape{database},
league: league,
}
}
Finally, we can get the amazing payoff we wanted by removing the Seek call from RecordWin. Yes, it doesn't feel much, but at least it means if we do any other kind of writes we can rely on our Write to behave how we need it to. Plus it will now let us test the potentially problematic code separately and fix it.
Let's write the test where we want to update the entire contents of a file with something that is smaller than the original contents.

Write the test first

Our test will create a file with some content, try to write to it using the tape, and read it all again to see what's in the file. In tape_test.go:
//tape_test.go
func TestTape_Write(t *testing.T) {
file, clean := createTempFile(t, "12345")
defer clean()
​
tape := &tape{file}
​
tape.Write([]byte("abc"))
​
file.Seek(0, 0)
newFileContents, _ := io.ReadAll(file)
​
got := string(newFileContents)
want := "abc"
​
if got != want {
t.Errorf("got %q want %q", got, want)
}
}

Try to run the test

=== RUN TestTape_Write
--- FAIL: TestTape_Write (0.00s)
tape_test.go:23: got 'abc45' want 'abc'
As we thought! It writes the data we want, but leaves the rest of the original data remaining.

Write enough code to make it pass

os.File has a truncate function that will let us effectively empty the file. We should be able to just call this to get what we want.
Change tape to the following:
//tape.go
type tape struct {
file *os.File
}
​
func (t *tape) Write(p []byte) (n int, err error) {
t.file.Truncate(0)
t.file.Seek(0, 0)
return t.file.Write(p)
}
The compiler will fail in a number of places where we are expecting an io.ReadWriteSeeker but we are sending in *os.File. You should be able to fix these problems yourself by now but if you get stuck just check the source code.
Once you get it refactoring our TestTape_Write test should be passing!

One other small refactor

In RecordWin we have the line json.NewEncoder(f.database).Encode(f.league).
We don't need to create a new encoder every time we write, we can initialise one in our constructor and use that instead.
Store a reference to an Encoder in our type and initialise it in the constructor:
//file_system_store.go
type FileSystemPlayerStore struct {
database *json.Encoder
league League
}
​