Golang JSON Marshal and Unmarshal

Posted on | Edited on | In

Highly recommend to read article JSON and Go if you are new to this area or want to bring the memory back.

In Golang, we usually use struct with struct tag to unmarshl/marshal JSON object data, for example, if the JSON object is:

1
2
3
4
5
6
7
{
	"page": 10,
	"user_info": {
		"age": 18,
		"height": 7.2
	}
}

The corresponding go struct:

1
2
3
4
5
6
7
8
9
type Response struct {
	Page     int      `json:"page"      xml:"PAGE"`
	UserInfo userInfo `json:"user_info" xml:"USER_INFO"`
}

type userInfo struct {
	Age    uint32  `json:"age"    xml:"AGE"`
	Height float32 `json:"height" xml:"HEIGHT"`
}

What is struct tag in golang please see: https://stackoverflow.com/questions/10858787/what-are-the-uses-for-struct-tags-in-go

Please note that only exported field(upper case) in struct will be encoded/decoded, and struct tag can help rename, if there is no struct tag, the exact struct field name will be used as JSON key name in marshalling.

For unmarshal if there is no corresponding JSON field, the struct field will be assigned the default zero value of that type, for example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// using raw string quote `` to avoid any escape chars
d := []byte(`
		{"page": 1}
	`)

resp := Response{}
err := json.Unmarshal(d, &resp)
if err != nil {
  fmt.Println("%w", err)
}
fmt.Printf("%+v\n", resp)

// output
{Page:1 UserInfo:{Age:0 Height:0}}

Because the user_info JSON data is missing, so UserInfo gets a struct value with default value on all fields UserInfo:{Age:0 Height:0}, but this cannot tell if the user_info is really missing or it has value but all 0.

To solve this, we can use pointer for nested struct field:

1
2
3
4
type Response struct {
	Page     int       `json:"page"`
	UserInfo *userInfo `json:"user_info"`  // change to pointer
}

And run again, the output will be:

1
{Page:1 UserInfo:<nil>}

By checking it is nil, we can tell the JSON data is missing.

For Arbitrary JSON Object Data

If there are unknown fields in JSON data, we can use interface and type assertion to help decode the JSON data, for example:

From blog JSON and Go, the json module uses interface{} to store arbitrary JSON objects or arrays and access them by type asseration with underlying map[string]interface{}.

I have encountered the case that a struct field the marshal and unmarshal have to use different type due to some reasons, so I have to use interface{} for that struct field and use assertion to access it.

Omit The Field If Not Exist

The “omitempty” option specifies that the field should be omitted from the marshal(encoding) if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string.

1
2
3
4
type Example struct {
	Count     int      `json:"count,omitempty"` // will not appear in JSON if empty value
	Name      string   `json:"name"`
}

Exclude The Field

This field will be skipped in either Marshal or Unmarshal.

1
2
3
type Example struct {
	Count     int      `json:"-"`
}

Non-Object JSON Data

The JSON is not always in object format, it can be a array, single string or number, so to unmarshal them we need to use right golang data type:

  1. JSON array -> golang slice
  2. JSON string -> golang string
  3. JSON number -> golang number

For example:

1
2
3
4
5
6
blob := `"hello, json"`
var s string
if err := json.Unmarshal([]byte(blob), &s); err != nil {
	log.Fatal(err)
}
fmt.Printf("\n%+v\n", s) // hello, json

Customizing Marshal/Unmarshal

This UnmarshalJSON method can be used to customize the unmarshal process, it is from Unmarshaler interface.

In this example, we want to unmarshal a JSON string to a golang struct, the JSON string is like:

1
"{\"name\": \"peter\", \"code\": 20001, \"message\": \"peter is sick off today!\"}"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
type ExampleError struct {
	Name    string `json:"name"`
	Code    uint   `json:"code"`
	Message string `json:"message"`
}

// Implement this func and it will work when you call json.Unmarshal
func (a *ExampleError) UnmarshalJSON(b []byte) error {
	// b is the raw byte data
	// here we first unarmshal it to string to remove the `\`, the string will be
	// `{"name":"peter","code":20001,"message":"peter is sick off today!"}`
	var s string
	if err := json.Unmarshal(b, &s); err != nil {
		return err
	}
	// then we further unmarshal it to golang struct 
	temp := struct {
		Name    string `json:"name"`
		Code    uint   `json:"code"`
		Message string `json:"message"`
	}{}
	// if the b is not valid, we keep it in Message
	if err := json.Unmarshal([]byte(s), &temp); err != nil {
		a.Message = s
		return nil
	}
	a.Name = temp.Name
	a.Code = temp.Code
	a.Message = temp.Message
	return nil
}

var ee = ExampleError{}
err := json.Unmarshal(<byte data>, &ee)

There is another example about customized marshal and unmarshal, redact on top of golang json module example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

type Animal int

const (
	Unknown Animal = iota
	Gopher
	Zebra
)

func (a *Animal) UnmarshalJSON(b []byte) error {
	// b is the element from the JSON array
	var s string
	if err := json.Unmarshal(b, &s); err != nil {
		return err
	}
	switch strings.ToLower(s) {
	default:
		*a = Unknown
	case "gopher":
		*a = Gopher
	case "zebra":
		*a = Zebra
	}
	return nil
}

func (a Animal) MarshalJSON() ([]byte, error) {
	// a is the Animal element from the golang array
	var s string
	switch a {
	default:
		s = "unknown"
	case Gopher:
		s = "gopher"
	case Zebra:
		s = "zebra"
	}
	return json.Marshal(s)
}

func main() {
	animalArray := [3]Animal{Gopher, Zebra, Unknown}
	blob, err := json.Marshal(animalArray)
	if err != nil {
		log.Fatal(err)
	}
	// blob now is:
	// `["gopher","armadillo","zebra","unknown","gopher","bee","gopher","zebra"]`

	var zoo []Animal
	if err := json.Unmarshal([]byte(blob), &zoo); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("\n%+v\n", zoo) // [1 2 0]
}

Unmarshal String to Number

This is tricky and useful in some cases, we want to unmarshal a string encoded number to golang uint64, so in the struct you can use tag string, for example:

1
2
3
type Example struct {
	ProjectNumber uint64 `json:"project_number,string"`
}

From document: The string option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types.

JSON Encoder/Decoder

When to use marshal/unmarshal and encoder/decoder, reference question:

  • Use json.Decoder if your data is coming from an io.Reader stream, or you need to decode multiple values from a stream of data, for example the HTTP response, this can be memory efficiency because it does not load all data into memory.

  • Use json.Unmarshal if you already have the JSON data in memory.

Also from JSON and Go: Due to the ubiquity of Readers and Writers, these Encoder and Decoder types can be used in a broad range of scenarios, such as reading and writing to HTTP connections, WebSockets, or files.

0%