shell脚本处理JSON数据工具jq

shell脚本如何方便地处理JSON格式的数据呢，这里介绍一个工具：jq。

官方教程简单翻译如下。

1、获取JSON数据

我们以github上jq项目最新5条评论的JSON数据为例。获取数据如下:

curl ‘https://api.github.com/repos/stedolan/jq/commits?per_page=5‘

结果如下：

[
  {
    "sha": "d25341478381063d1c76e81b3a52e0592a7c997f",
    "commit": {
      "author": {
        "name": "Stephen Dolan",
        "email": "[email protected]",
        "date": "2013-06-22T16:30:59Z"
      },
      "committer": {
        "name": "Stephen Dolan",
        "email": "[email protected]",
        "date": "2013-06-22T16:30:59Z"
      },
      "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161",
      "tree": {
        "sha": "6ab697a8dfb5a96e124666bf6d6213822599fb40",
        "url": "https://api.github.com/repos/stedolan/jq/git/trees/6ab697a8dfb5a96e124666bf6d6213822599fb40"
      },
      "url": "https://api.github.com/repos/stedolan/jq/git/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
      "comment_count": 0
    },
    "url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
    "html_url": "https://github.com/stedolan/jq/commit/d25341478381063d1c76e81b3a52e0592a7c997f",
    "comments_url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f/comments",
    "author": {
      "login": "stedolan",
    ...

2、高亮并按属性排序显示

用 jq ‘.‘即可：

curl ‘https://api.github.com/repos/stedolan/jq/commits?per_page=5‘ | jq ‘.‘

结果：

[
  {
    "parents": [
      {
        "html_url": "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
        "url": "https://api.github.com/repos/stedolan/jq/commits/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
        "sha": "54b9c9bdb225af5d886466d72f47eafc51acb4f7"
      },
      {
        "html_url": "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
        "url": "https://api.github.com/repos/stedolan/jq/commits/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
        "sha": "8b1b503609c161fea4b003a7179b3fbb2dd4345a"
      }
    ],
    "committer": {
      "type": "User",
      "received_events_url": "https://api.github.com/users/stedolan/received_events",
      "events_url": "https://api.github.com/users/stedolan/events{/privacy}",
      "repos_url": "https://api.github.com/users/stedolan/repos",
      "organizations_url": "https://api.github.com/users/stedolan/orgs",
...

这里的评论内容比较多，我们现在想拿第一个评论。

3、获取数组某一项

curl ‘https://api.github.com/repos/stedolan/jq/commits?per_page=5‘ | jq ‘.[0]‘

结果：

{
  "parents": [
    {
      "html_url": "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
      "url": "https://api.github.com/repos/stedolan/jq/commits/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
      "sha": "54b9c9bdb225af5d886466d72f47eafc51acb4f7"
    },
    {
      "html_url": "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
      "url": "https://api.github.com/repos/stedolan/jq/commits/8b1b503609c161fea4b003a7179b3fbb2dd4345a",
      "sha": "8b1b503609c161fea4b003a7179b3fbb2dd4345a"
    }
  ],
  "committer": {
    "type": "User",
    "received_events_url": "https://api.github.com/users/stedolan/received_events",
    "events_url": "https://api.github.com/users/stedolan/events{/privacy}",
    "repos_url": "https://api.github.com/users/stedolan/repos",
    "organizations_url": "https://api.github.com/users/stedolan/orgs",
    "subscriptions_url": "https://api.github.com/users/stedolan/subscriptions",
    "starred_url": "https://api.github.com/users/stedolan/starred{/owner}{/repo}",
    "gists_url": "https://api.github.com/users/stedolan/gists{/gist_id}",
    "login": "stedolan",
    "id": 79765,
    "avatar_url": "https://1.gravatar.com/avatar/31de909d8e55dd07ed782d92ece59842?d=https%3A%2F%2Fidenticons.github.com%2Ffc5b6765b1c9cfaecea48ae71df4d279.png",
    "gravatar_id": "31de909d8e55dd07ed782d92ece59842",
    "url": "https://api.github.com/users/stedolan",
    "html_url": "https://github.com/stedolan",
    "followers_url": "https://api.github.com/users/stedolan/followers",
    "following_url": "https://api.github.com/users/stedolan/following{/other_user}"
  },
  "author": {
    "type": "User",
    "received_events_url": "https://api.github.com/users/stedolan/received_events",
    "events_url": "https://api.github.com/users/stedolan/events{/privacy}",
    "repos_url": "https://api.github.com/users/stedolan/repos",
    "organizations_url": "https://api.github.com/users/stedolan/orgs",
    "subscriptions_url": "https://api.github.com/users/stedolan/subscriptions",
    "starred_url": "https://api.github.com/users/stedolan/starred{/owner}{/repo}",
    "gists_url": "https://api.github.com/users/stedolan/gists{/gist_id}",
    "login": "stedolan",
    "id": 79765,
    "avatar_url": "https://1.gravatar.com/avatar/31de909d8e55dd07ed782d92ece59842?d=https%3A%2F%2Fidenticons.github.com%2Ffc5b6765b1c9cfaecea48ae71df4d279.png",
    "gravatar_id": "31de909d8e55dd07ed782d92ece59842",
    "url": "https://api.github.com/users/stedolan",
    "html_url": "https://github.com/stedolan",
    "followers_url": "https://api.github.com/users/stedolan/followers",
    "following_url": "https://api.github.com/users/stedolan/following{/other_user}"
  },
  "comments_url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f/comments",
  "html_url": "https://github.com/stedolan/jq/commit/d25341478381063d1c76e81b3a52e0592a7c997f",
  "url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
  "commit": {
    "comment_count": 0,
    "url": "https://api.github.com/repos/stedolan/jq/git/commits/d25341478381063d1c76e81b3a52e0592a7c997f",
    "tree": {
      "url": "https://api.github.com/repos/stedolan/jq/git/trees/6ab697a8dfb5a96e124666bf6d6213822599fb40",
      "sha": "6ab697a8dfb5a96e124666bf6d6213822599fb40"
    },
    "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161",
    "committer": {
      "date": "2013-06-22T16:30:59Z",
      "email": "[email protected]",
      "name": "Stephen Dolan"
    },
    "author": {
      "date": "2013-06-22T16:30:59Z",
      "email": "[email protected]",
      "name": "Stephen Dolan"
    }
  },
  "sha": "d25341478381063d1c76e81b3a52e0592a7c997f"
}

可以看到，我们已经拿到了一条完整的评论内容。但我们真正关心的只是评论内容和用户名，下面来获取这两项内容。

4、自定义格式输出某一项

jq ‘.[0] | {message: .commit.message, name: .commit.committer.name}‘

结果：

{
  "name": "Stephen Dolan",
  "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
}

我们可以看到，已经拿到了想要的内容，并且已经按我们自己定义的格式显示了。

这里 | 后面的内容是以前面的内容为输入的，.commit 中的 . 就是指 .[0] 中的内容。

5、自定义格式输出多项

jq ‘.[] | {message: .commit.message, name: .commit.committer.name}‘

结果：

{
  "name": "Stephen Dolan",
  "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
}
{
  "name": "Stephen Dolan",
  "message": "Reject all overlong UTF8 sequences."
}
{
  "name": "Stephen Dolan",
  "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences."
}
{
  "name": "Stephen Dolan",
  "message": "Fix example in manual for `floor`. See #155."
}
{
  "name": "Nicolas Williams",
  "message": "Document floor"
}

这里 .[] 获取的是数组中的所有项。

我们看到，结果是一个个独立的JSON对象，如何把结果组合成一个数组呢？

6、以数组形式自定义输出多项

jq ‘[.[] | {message: .commit.message, name: .commit.committer.name}]‘

结果：

[
  {
    "name": "Stephen Dolan",
    "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
  },
  {
    "name": "Stephen Dolan",
    "message": "Reject all overlong UTF8 sequences."
  },
  {
    "name": "Stephen Dolan",
    "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences."
  },
  {
    "name": "Stephen Dolan",
    "message": "Fix example in manual for `floor`. See #155."
  },
  {
    "name": "Nicolas Williams",
    "message": "Document floor"
  }
]

我们可以看到，只要在上一步的命令中内容的两端加个中括号即可。

最后，我们如果想获取每个评论的引用评论的url（在parents节点中，有一个或多个）呢？

7、获取其他内容

jq ‘[.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]}]‘

结果：

[
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7",
      "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a"
    ],
    "name": "Stephen Dolan",
    "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161"
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/ff48bd6ec538b01d1057be8e93b94eef6914e9ef"
    ],
    "name": "Stephen Dolan",
    "message": "Reject all overlong UTF8 sequences."
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7"
    ],
    "name": "Stephen Dolan",
    "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences."
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/3dcdc582ea993afea3f5503a78a77675967ecdfa"
    ],
    "name": "Stephen Dolan",
    "message": "Fix example in manual for `floor`. See #155."
  },
  {
    "parents": [
      "https://github.com/stedolan/jq/commit/7c4171d414f647ab08bcd20c76a4d8ed68d9c602"
    ],
    "name": "Nicolas Williams",
    "message": "Document floor"
  }
]

这里用 .parents[].html_url 获取当前项的 parents 节点中的所有项的 html_url 属性的内容，然后两边加个中括号组装成数组输出。

怎么样，经过这个例子可以看出，用jq处理JSON数据还是很方便强大的吧。

8、注意点

jq能处理的需要是严格的JSON格式数据，JSON对象和JSON字符串是不行的，如下面的两种格式数据jq是不能处理的：

json对象：

{
    a: 1,
    b: {
        c: "abc"
    }
}

json字符串：

‘{"a":1,"b":{"c":"abc"}}‘

正确的JSON格式：

{
    "a": 1,
    "b": {
        "c": "abc"
    }
}

关于什么是JSON格式数据，请参考：http://www.json.org/

本文原文出自十年后的卢哥哥：http://www.cnblogs.com/lurenjiashuo/，转载请注明出处。

时间： 2024-11-12 14:22:02

shell脚本处理JSON数据工具jq的相关文章

Zabbix监控之Linux命令行/Shell脚本解析json

本文主要写了在Linux下如何用Shell脚本解析json数据,以便于在Zabbix中添加适合于业务应用的监控项. 为什么要使用json? json作为一种文本化的数据格式(文本化协议),符合UNIX编程的哲学,既符合透明性(透明性:设计可见,便于审查和调试)又符合文本性(文本性:数据应该保存为文本文件).关键在于json对Web程序或者java程序非常友好,对于开发人员来说,他们也喜欢使用json数据.业务逻辑.内部的依赖关系以及状态信息由程序员去做,运维人员只需要调用程序员开放出来的API接

shell脚本修改json中某个字段的值

shell脚本修改json中某个字段的值 思路:通过awk来找到旧数据,然后用sed来替换旧数据 源码 config.json { "name": "the_name", "id": "132869", "content_url": "https://hot.example.com/", "enable_feature1": "true", &qu

Shell学习---Shell脚本的静态检查工具shellcheck

Shell脚本的静态检查工具shellcheck ubuntu下 apt install shellcheck ,即可安装shellcheck.写完shell脚本,记得用它检查一下,能给你点建议的.要检查现有项目的所有的脚本, find your_project_folder -name "*.sh" | xargs -i shellcheck {} 即可实现批量检查 原文地址:https://www.cnblogs.com/ftl1012/p/9568635.html

用fastjson框架解析Json数据工具类

在Android应用开发中,APP端与server交互的数据一般是json格式的数据,那么为了方便操作,我们通常会用json解析框架来帮助我们进行数据操作.经常使用的有Gjson和fastjson框架.在项目中用到了fastjson解析数据,当然前提是要准备要对应的jar包工具,把工具类(主要是json数据与实体bean之间的转换操作)贴出来方便以后使用: public class FastJsonTools { /** * 用fastjson 将json字符串解析为一个 JavaBean *

shell脚本处理大数据系列之(一)方法小结

转自:http://longriver.me/?p=57 方法1: 单进程处理大规模的文件速度如(上million量级)比较慢,可以采用awk取模的方法,将文件分而治之,这样可以利用充分的利用多核CPU的优势 1 2 3 4 for((i=0;i<5;i++));do     cat query_ctx.20k | awk 'NR%5=='$i'' |\      wc -l  1> output_$i 2>err_$i &  done 方法2: 另外也可以使用split的方法,

Linux Shell脚本编写——呈现数据(五)

Linux系统将每个对象当做文件来处理.这包括输入和输出的过程.Linux用文件描述符来标识每个文件对象.文件描述符是一个非负整数,可以唯一地标识会话中打开的文件.每个过程一次最多可以有9个文件描述符.出于特殊目的,bash shell保留了最早的3个文件描述符(0.1和2) Linux的标准文件描述符 文件描述符 缩写 描述 0 STDIN 标准输入 1 STDOUT 标准输出 2 SRDERR 标准错误 STDIN STDIN文件描述代表shell的标准输入.对于终端界面来说,标准输入就是键

详解Shell脚本：sed命令工具 ，awk命令工具

-----------------------------概述----------------------------------- Linux文本处理工具(以下都是命令和工具,而且只是作为展示,不能作为修改动作) grep(过滤,但是不支持正则表达式)egrep(支持正则表达式)sed (行过滤)awk(列过滤) - 正则表达式概念 正则表达式:使用单个字符串来描述.匹配一系列符合某个句法规则的字符串 由普通字符与特殊字符组成,一般用在脚本编程.文本编辑器中,如php.python.she等,

shell脚本，tee小工具的用法。

解释: tee是个工具 , 它的作用就是把标准输出,复制一份,扔文件里 ,原标准输出还输出,-a就相当于 >> 追加到文件里的意思. 不加就是 > 重定向到文件里去.

shell脚本之使用bc工具实现数值转换（浮点转二进制）

#! /bin/bash echo -n input float: read num echo power of the operation: echo $num^2 |bc echo converted to binary echo "obase=2;$num" |bc [[email protected] Desktop]# ./zhuanhuan.sh input float:3.1415926 power of the operation: 9.8696040 converte