shell中的json解析工具-jq
jq命令允许直接在命令行下对JSON进行操作，包括分片、过滤、转换等 , jq是用C编写。
jq是一款命令行下处理JSON数据的工具。其可以接受标准输入，命令管道或者文件中的JSON数据，
经过一系列的过滤器(filters)和表达式的转后形成我们需要的数据结构并将结果输出到标准输出中。
严格遵循 JSON 格式的标准。所有的属性名必须是以双引号包括的字符串。对象的最后一个属性的末尾或者数组的最后一个元素的末尾不能有逗号。否则 jq 会抛出无法解析 JSON 的错误。
手册 https://stedolan.github.io/jq/manual/#Invokingjq
1、jq安装
centos
yum -y install jq
2、jq基础表达式
2.1 JSON对象操作
0.示例数据
复制示例数据存储为 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
1.引用json数据
. 符号：单独的一个 "." 或者 ".?" 符号用来表示对作为表达式输入的整个 JSON 对象的引用
如果 JSON 对象不包含指定的属性则返回 null
当输入不是 JSON 对象或数组时，"." ,表达式会抛出异常
当输入不是 JSON 对象或数组时， ".?" 表达式无任何输出
推荐使用 jq "."
[root@10-40-0-213 ~]# cat 1.json | jq .
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
##-c选项则是压缩到1行输出
[root@10-40-0-213 ~]# jq "." 1.json | jq -c
[{"date":"2022-11-28 11:40:50","version":"dev-0.0.1-1f4f44dd","ci.job.id":293776},{"date":"2022-11-28 11:57:36","version":"dev-0.0.1-08090349","ci.job.id":293780},{"date":"2022-11-28 14:17:07","version":"dev-0.0.1-bc59cf57","ci.job.id":293790},{"date":"2022-11-28 14:25:18","version":"dev-0.0.1-96c586f2","ci.job.id":293792}]
[root@10-40-0-213 ~]# cat 1.json | jq "."
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq . 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
2.2 数组操作
jq提供3种基础表达式来操作数组
迭代器操作( '.[]' ). 该表达式的输入可以是数组或者 JSON 对象。输出的是基于数组元素或者 JSON 对象属性值的 iterator。
访问特定元素的操作( '.[index]' 或 '.[attributename]' )。用来访问数组元素或者 JSON 对象的属性值。输出是单个值
数组切片操作( '.[startindex:endindex]' )，其行为类似于 python 语言中数组切片操作
1.迭代器操作( '.[]' )
该表达式的输入可以是数组或者 JSON 对象。输出的是基于数组元素或者 JSON 对象属性值的 iterator。
[root@10-40-0-213 ~]# jq "." 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[]"
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
2.访问特定元素的操作
( '.[index]' 或 '.[attributename]' )。用来访问数组元素或者 JSON 对象的属性值。输出是单个值
[root@10-40-0-213 ~]# jq "." 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[]'
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
##内建函数-key,用来获取JSON中的key元素
[root@10-40-0-213 ~]# jq "." 1.json | jq 'keys'
[
0,
1,
2,
3
]
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[1]'
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[2]'
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[3]'
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[4]'
null
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'| jq 'keys'
[
"ci.job.id",
"date",
"version"
]
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'| jq 'keys'
[
"ci.job.id",
"date",
"version"
]
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[0].date'
"2022-11-28 11:40:50"
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[1].date'
"2022-11-28 11:57:36"
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[2].date'
"2022-11-28 14:17:07"
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[3].date'
"2022-11-28 14:25:18"
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[3].version'
"dev-0.0.1-96c586f2"
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[3].version' -r
dev-0.0.1-96c586f2
##解析不存在的元素，会返回null
[root@10-40-0-213 ~]# jq "." 1.json | jq '.[3].version1' -r
null
输出多个索引的值，可以用逗号分割
[root@10-40-0-213 ~]# jq '.' 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq '.' 1.json| jq ".[0].date , .[0].version "
"2022-11-28 11:40:50"
"dev-0.0.1-1f4f44dd"
has是用来是判断是否存在某个key
[root@10-40-0-213 ~]# jq '.' 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq '.' 1.json | jq .[]
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq 'keys'
[
0,
1,
2,
3
]
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'| jq 'has("date")'
true
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'| jq 'has("version")'
true
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'| jq 'has("ci.job.id")'
true
# jq '.' 1.json | jq '.[0]'| jq 'has("version")'
true
获取数组的长度 jq '.|length'
length求长度，如果是字符串是求的字符串的长度，如果是数组则求得是数组的长度
[root@10-40-0-213 ~]# jq '.' 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0]'
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0].date|length'
19
[root@10-40-0-213 ~]# jq '.' 1.json | jq '.[0].version|length'
18
用管道符号｜可以对其进行再次处理
[root@10-40-0-213 ~]# jq '.' 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]#
[root@10-40-0-213 ~]# jq '.' 1.json| jq ".[]"
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
[root@10-40-0-213 ~]# jq '.' 1.json| jq ".[]|.date"
"2022-11-28 11:40:50"
"2022-11-28 11:57:36"
"2022-11-28 14:17:07"
"2022-11-28 14:25:18"
[root@10-40-0-213 ~]# jq '.' 1.json| jq ".[]|.version"
"dev-0.0.1-1f4f44dd"
"dev-0.0.1-08090349"
"dev-0.0.1-bc59cf57"
"dev-0.0.1-96c586f2"
3.数组切片操作
[root@10-40-0-213 ~]# jq "." 1.json | jq 'keys'
[
0,
1,
2,
3
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[1:]"
[
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[2:]"
[
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[3:]"
[
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[4:]"
[]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[]"
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[0:0]"
[]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[0:1]"
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[0:2]"
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[0:3]"
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[0:4]"
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
2.3.jq表达式操作
数学运算
[root@10-40-0-213 ~]# echo 1 | jq '(.+0)'
1
[root@10-40-0-213 ~]# echo 2 | jq '(.+0)'
2
[root@10-40-0-213 ~]# echo 1 | jq '(.+1)'
2
[root@10-40-0-213 ~]# echo 2 | jq '(.+1)'
3
[root@10-40-0-213 ~]# echo 1 | jq '(.+2)*2'
6
[root@10-40-0-213 ~]# echo {1,2,3,4} | jq '(.+2)*2'
6
8
10
12
字符串操作
精简json返回
[root@10-40-0-213 ~]# jq "." 1.json
[
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
},
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
},
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
},
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
]
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[]"
{
"date": "2022-11-28 11:40:50",
"version": "dev-0.0.1-1f4f44dd",
"ci.job.id": 293776
}
{
"date": "2022-11-28 11:57:36",
"version": "dev-0.0.1-08090349",
"ci.job.id": 293780
}
{
"date": "2022-11-28 14:17:07",
"version": "dev-0.0.1-bc59cf57",
"ci.job.id": 293790
}
{
"date": "2022-11-28 14:25:18",
"version": "dev-0.0.1-96c586f2",
"ci.job.id": 293792
}
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[]"|jq "{date}"
{
"date": "2022-11-28 11:40:50"
}
{
"date": "2022-11-28 11:57:36"
}
{
"date": "2022-11-28 14:17:07"
}
{
"date": "2022-11-28 14:25:18"
}
[root@10-40-0-213 ~]# jq "." 1.json | jq ".[]"|jq "{date},{version}"
{
"date": "2022-11-28 11:40:50"
}
{
"version": "dev-0.0.1-1f4f44dd"
}
{
"date": "2022-11-28 11:57:36"
}
{
"version": "dev-0.0.1-08090349"
}
{
"date": "2022-11-28 14:17:07"
}
{
"version": "dev-0.0.1-bc59cf57"
}
{
"date": "2022-11-28 14:25:18"
}
{
"version": "dev-0.0.1-96c586f2"
}
3、内置运算
jq --help
# jq --help
jq - commandline JSON processor [version 1.6]
Usage: jq [options] <jq filter> [file...]
jq [options] --args <jq filter> [strings...]
jq [options] --jsonargs <jq filter> [JSON_TEXTS...]
jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.
The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).
For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq
Example:
$ echo '{"foo": 0}' | jq .
{
"foo": 0
}
Some of the options include:
-c compact instead of pretty-printed output;
-n use `null` as the single input value;
-e set the exit status code based on the output;
-s read (slurp) all inputs into an array; apply filter to it;
-r output raw strings, not JSON texts;
-R read raw strings, not JSON texts;
-C colorize JSON;
-M monochrome (don't colorize JSON);
-S sort keys of objects on output;
--tab use tabs for indentation;
--arg a v set variable $a to value <v>;
--argjson a v set variable $a to JSON value <v>;
--slurpfile a f set variable $a to an array of JSON texts read from <f>;
--rawfile a f set variable $a to a string consisting of the contents of <f>;
--args remaining arguments are string arguments, not files;
--jsonargs remaining arguments are JSON arguments, not files;
-- terminates argument processing;
Named arguments are also available as $ARGS.named[], while
positional arguments are available as $ARGS.positional[].
See the manpage for more options.
