jq/yq Cheat Sheet

Posted on | Edited on | In

Summary of some jq use cases and caveats daily. Quick reference:

Need to read through some common use cases:

yq is another open source tool relies on jq to query yaml file, tt has most common operations available and syntactically like jq.

  1. Must quote the expression For example:
1
2
# -r: raw string
k get pod banzai-vault-0 -o json | jq -r '.spec.containers[].name'
  1. Filter out null value For example, I want to get the configMap name from a deployment, the volumes may have multiple subitems and one of them is config map, we need to filter out null:
1
kubectl get deploy nwiaas -o json | jq -r '.spec.template.spec.volumes[].configMap.name | select(. != null)'

This is a simple case to introduce select function

  1. Replace field values Using |= operator, see here.

For example, I want to create one job from cronjob but with different args:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# --dry-run=client: print without sending object
kubectl create job \
--from=cronjob/<cronjob name> \
<job name> \
--dry-run=client \
--output json > original.json

# use jq docker
docker run -it --rm \
-v $(pwd)/original.json:/original.json \
--entrypoint=/bin/bash \
stedolan/jq \
-c "cat /original.json |jq -r '.spec.template.spec.containers[].args |= [\"python\", \"make.py\", \"do_something\"]' | cat" \
| kubectl apply -f -

Here I use cat at end to remove color supplement, otherwise kubectl apply will fail.

  1. Loop and extract specified values For example, the Elasticsearch API returns a list of map object and each map has the same number of fields, I want to extract few of the fields and output with specific format.
1
2
3
# [.index,.shard,.prirep,.node]: generate [] array format and put items in
curl -s "http://localhost:9200/_cat/shards/.ds-*?h=index,shard,prirep,state,node&format=json" | \
jq -r '.[] | [.index,.shard,.prirep,.node] | @csv' | sed 's/\"//g'

It will output csv style string, for example:

1
"xxx","yyy","zzz","qqq"

Or format as a table:

1
2
3
4
5
# [.index,.shard,.prirep,.node]: generate an array
# join(":|:"): join array elements
# column -s : -t: use `:` as separator and make table
curl -s "http://localhost:9200/_cat/shards/.ds-*?h=index,shard,prirep,state,node&format=json" | \
jq -r '.[] | [.index, .shard, .prirep, .node] | join(":|:")' | column -s : -t | sort

The output is like:

1
2
3
.ds-k8s-nodes-filebeat-2022.04.28-000020  |  0  |  p  |  172.16.0.172
.ds-k8s-nodes-filebeat-2022.04.28-000020  |  1  |  p  |  172.16.0.164
.ds-k8s-nodes-filebeat-2022.04.28-000020  |  2  |  p  |  172.16.0.247
0%