Hive分析窗口函数(四) LAG,LEAD,FIRST_VALUE,LAST_VALUE

1.LAG功能是什么？

2.LEAD与LAG功能有什么相似的地方那个？

3.FIRST_VALUE与LAST_VALUE分别完成什么功能？

继续学习这四个分析函数。

注意： 这几个函数不支持WINDOW子句。

Hive版本为 apache-hive-0.13.1

数据准备：

水电费

    cookie1,2015-04-10 10:00:02,url2
    cookie1,2015-04-10 10:00:00,url1
    cookie1,2015-04-10 10:03:04,1url3
    cookie1,2015-04-10 10:50:05,url6
    cookie1,2015-04-10 11:00:00,url7
    cookie1,2015-04-10 10:10:00,url4
    cookie1,2015-04-10 10:50:01,url5
    cookie2,2015-04-10 10:00:02,url22
    cookie2,2015-04-10 10:00:00,url11
    cookie2,2015-04-10 10:03:04,1url33
    cookie2,2015-04-10 10:50:05,url66
    cookie2,2015-04-10 11:00:00,url77
    cookie2,2015-04-10 10:10:00,url44
    cookie2,2015-04-10 10:50:01,url55

    CREATE EXTERNAL TABLE lxw1234 (
    cookieid string,
    createtime string,  --页面访问时间
    url STRING       --被访问页面
    ) ROW FORMAT DELIMITED
    FIELDS TERMINATED BY ','
    stored as textfile location '/tmp/lxw11/';

    hive> select * from lxw1234;
    OK
    cookie1 2015-04-10 10:00:02     url2
    cookie1 2015-04-10 10:00:00     url1
    cookie1 2015-04-10 10:03:04     1url3
    cookie1 2015-04-10 10:50:05     url6
    cookie1 2015-04-10 11:00:00     url7
    cookie1 2015-04-10 10:10:00     url4
    cookie1 2015-04-10 10:50:01     url5
    cookie2 2015-04-10 10:00:02     url22
    cookie2 2015-04-10 10:00:00     url11
    cookie2 2015-04-10 10:03:04     1url33
    cookie2 2015-04-10 10:50:05     url66
    cookie2 2015-04-10 11:00:00     url77
    cookie2 2015-04-10 10:10:00     url44
    cookie2 2015-04-10 10:50:01     url55

LAG

LAG(col,n,DEFAULT) 用于统计窗口内往上第n行值

第一个参数为列名，第二个参数为往上第n行（可选，默认为1），第三个参数为默认值（当往上第n行为NULL时候，取默认值，如不指定，则为NULL）

    SELECT cookieid,
    createtime,
    url,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,
    LAG(createtime,1,'1970-01-01 00:00:00') OVER(PARTITION BY cookieid ORDER BY createtime) AS last_1_time,
    LAG(createtime,2) OVER(PARTITION BY cookieid ORDER BY createtime) AS last_2_time
    FROM lxw1234;

    cookieid createtime             url    rn       last_1_time             last_2_time
    -------------------------------------------------------------------------------------------
    cookie1 2015-04-10 10:00:00     url1    1       1970-01-01 00:00:00     NULL
    cookie1 2015-04-10 10:00:02     url2    2       2015-04-10 10:00:00     NULL
    cookie1 2015-04-10 10:03:04     1url3   3       2015-04-10 10:00:02     2015-04-10 10:00:00
    cookie1 2015-04-10 10:10:00     url4    4       2015-04-10 10:03:04     2015-04-10 10:00:02
    cookie1 2015-04-10 10:50:01     url5    5       2015-04-10 10:10:00     2015-04-10 10:03:04
    cookie1 2015-04-10 10:50:05     url6    6       2015-04-10 10:50:01     2015-04-10 10:10:00
    cookie1 2015-04-10 11:00:00     url7    7       2015-04-10 10:50:05     2015-04-10 10:50:01
    cookie2 2015-04-10 10:00:00     url11   1       1970-01-01 00:00:00     NULL
    cookie2 2015-04-10 10:00:02     url22   2       2015-04-10 10:00:00     NULL
    cookie2 2015-04-10 10:03:04     1url33  3       2015-04-10 10:00:02     2015-04-10 10:00:00
    cookie2 2015-04-10 10:10:00     url44   4       2015-04-10 10:03:04     2015-04-10 10:00:02
    cookie2 2015-04-10 10:50:01     url55   5       2015-04-10 10:10:00     2015-04-10 10:03:04
    cookie2 2015-04-10 10:50:05     url66   6       2015-04-10 10:50:01     2015-04-10 10:10:00
    cookie2 2015-04-10 11:00:00     url77   7       2015-04-10 10:50:05     2015-04-10 10:50:01

    last_1_time: 指定了往上第1行的值，default为'1970-01-01 00:00:00'
                 cookie1第一行，往上1行为NULL,因此取默认值 1970-01-01 00:00:00
                 cookie1第三行，往上1行值为第二行值，2015-04-10 10:00:02
                 cookie1第六行，往上1行值为第五行值，2015-04-10 10:50:01
    last_2_time: 指定了往上第2行的值，为指定默认值
                                                     cookie1第一行，往上2行为NULL
                                                     cookie1第二行，往上2行为NULL
                                                     cookie1第四行，往上2行为第二行值，2015-04-10 10:00:02
                                                     cookie1第七行，往上2行为第五行值，2015-04-10 10:50:01

LEAD

与LAG相反

LEAD(col,n,DEFAULT) 用于统计窗口内往下第n行值

第一个参数为列名，第二个参数为往下第n行（可选，默认为1），第三个参数为默认值（当往下第n行为NULL时候，取默认值，如不指定，则为NULL）

    SELECT cookieid,
    createtime,
    url,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,
    LEAD(createtime,1,'1970-01-01 00:00:00') OVER(PARTITION BY cookieid ORDER BY createtime) AS next_1_time,
    LEAD(createtime,2) OVER(PARTITION BY cookieid ORDER BY createtime) AS next_2_time
    FROM lxw1234;

    cookieid createtime             url    rn       next_1_time             next_2_time
    -------------------------------------------------------------------------------------------
    cookie1 2015-04-10 10:00:00     url1    1       2015-04-10 10:00:02     2015-04-10 10:03:04
    cookie1 2015-04-10 10:00:02     url2    2       2015-04-10 10:03:04     2015-04-10 10:10:00
    cookie1 2015-04-10 10:03:04     1url3   3       2015-04-10 10:10:00     2015-04-10 10:50:01
    cookie1 2015-04-10 10:10:00     url4    4       2015-04-10 10:50:01     2015-04-10 10:50:05
    cookie1 2015-04-10 10:50:01     url5    5       2015-04-10 10:50:05     2015-04-10 11:00:00
    cookie1 2015-04-10 10:50:05     url6    6       2015-04-10 11:00:00     NULL
    cookie1 2015-04-10 11:00:00     url7    7       1970-01-01 00:00:00     NULL
    cookie2 2015-04-10 10:00:00     url11   1       2015-04-10 10:00:02     2015-04-10 10:03:04
    cookie2 2015-04-10 10:00:02     url22   2       2015-04-10 10:03:04     2015-04-10 10:10:00
    cookie2 2015-04-10 10:03:04     1url33  3       2015-04-10 10:10:00     2015-04-10 10:50:01
    cookie2 2015-04-10 10:10:00     url44   4       2015-04-10 10:50:01     2015-04-10 10:50:05
    cookie2 2015-04-10 10:50:01     url55   5       2015-04-10 10:50:05     2015-04-10 11:00:00
    cookie2 2015-04-10 10:50:05     url66   6       2015-04-10 11:00:00     NULL
    cookie2 2015-04-10 11:00:00     url77   7       1970-01-01 00:00:00     NULL

    --逻辑与LAG一样，只不过LAG是往上，LEAD是往下。

FIRST_VALUE

取分组内排序后，截止到当前行，第一个值

    SELECT cookieid,
    createtime,
    url,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,
    FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS first1
    FROM lxw1234;

    cookieid  createtime            url     rn      first1
    ---------------------------------------------------------
    cookie1 2015-04-10 10:00:00     url1    1       url1
    cookie1 2015-04-10 10:00:02     url2    2       url1
    cookie1 2015-04-10 10:03:04     1url3   3       url1
    cookie1 2015-04-10 10:10:00     url4    4       url1
    cookie1 2015-04-10 10:50:01     url5    5       url1
    cookie1 2015-04-10 10:50:05     url6    6       url1
    cookie1 2015-04-10 11:00:00     url7    7       url1
    cookie2 2015-04-10 10:00:00     url11   1       url11
    cookie2 2015-04-10 10:00:02     url22   2       url11
    cookie2 2015-04-10 10:03:04     1url33  3       url11
    cookie2 2015-04-10 10:10:00     url44   4       url11
    cookie2 2015-04-10 10:50:01     url55   5       url11
    cookie2 2015-04-10 10:50:05     url66   6       url11
    cookie2 2015-04-10 11:00:00     url77   7       url11

LAST_VALUE

取分组内排序后，截止到当前行，最后一个值

    SELECT cookieid,
    createtime,
    url,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,
    LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS last1
    FROM lxw1234;

    cookieid  createtime            url    rn       last1
    -----------------------------------------------------------------
    cookie1 2015-04-10 10:00:00     url1    1       url1
    cookie1 2015-04-10 10:00:02     url2    2       url2
    cookie1 2015-04-10 10:03:04     1url3   3       1url3
    cookie1 2015-04-10 10:10:00     url4    4       url4
    cookie1 2015-04-10 10:50:01     url5    5       url5
    cookie1 2015-04-10 10:50:05     url6    6       url6
    cookie1 2015-04-10 11:00:00     url7    7       url7
    cookie2 2015-04-10 10:00:00     url11   1       url11
    cookie2 2015-04-10 10:00:02     url22   2       url22
    cookie2 2015-04-10 10:03:04     1url33  3       1url33
    cookie2 2015-04-10 10:10:00     url44   4       url44
    cookie2 2015-04-10 10:50:01     url55   5       url55
    cookie2 2015-04-10 10:50:05     url66   6       url66
    cookie2 2015-04-10 11:00:00     url77   7       url77

如果不指定ORDER BY，则默认按照记录在文件中的偏移量进行排序，会出现错误的结果

    SELECT cookieid,
    createtime,
    url,
    FIRST_VALUE(url) OVER(PARTITION BY cookieid) AS first2
    FROM lxw1234;

    cookieid  createtime            url     first2
    ----------------------------------------------
    cookie1 2015-04-10 10:00:02     url2    url2
    cookie1 2015-04-10 10:00:00     url1    url2
    cookie1 2015-04-10 10:03:04     1url3   url2
    cookie1 2015-04-10 10:50:05     url6    url2
    cookie1 2015-04-10 11:00:00     url7    url2
    cookie1 2015-04-10 10:10:00     url4    url2
    cookie1 2015-04-10 10:50:01     url5    url2
    cookie2 2015-04-10 10:00:02     url22   url22
    cookie2 2015-04-10 10:00:00     url11   url22
    cookie2 2015-04-10 10:03:04     1url33  url22
    cookie2 2015-04-10 10:50:05     url66   url22
    cookie2 2015-04-10 11:00:00     url77   url22
    cookie2 2015-04-10 10:10:00     url44   url22
    cookie2 2015-04-10 10:50:01     url55   url22

    SELECT cookieid,
    createtime,
    url,
    LAST_VALUE(url) OVER(PARTITION BY cookieid) AS last2
    FROM lxw1234;

    cookieid  createtime            url     last2
    ----------------------------------------------
    cookie1 2015-04-10 10:00:02     url2    url5
    cookie1 2015-04-10 10:00:00     url1    url5
    cookie1 2015-04-10 10:03:04     1url3   url5
    cookie1 2015-04-10 10:50:05     url6    url5
    cookie1 2015-04-10 11:00:00     url7    url5
    cookie1 2015-04-10 10:10:00     url4    url5
    cookie1 2015-04-10 10:50:01     url5    url5
    cookie2 2015-04-10 10:00:02     url22   url55
    cookie2 2015-04-10 10:00:00     url11   url55
    cookie2 2015-04-10 10:03:04     1url33  url55
    cookie2 2015-04-10 10:50:05     url66   url55
    cookie2 2015-04-10 11:00:00     url77   url55
    cookie2 2015-04-10 10:10:00     url44   url55
    cookie2 2015-04-10 10:50:01     url55   url55

如果想要取分组内排序后最后一个值，则需要变通一下：

    SELECT cookieid,
    createtime,
    url,
    ROW_NUMBER() OVER(PARTITION BY cookieid ORDER BY createtime) AS rn,
    LAST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime) AS last1,
    FIRST_VALUE(url) OVER(PARTITION BY cookieid ORDER BY createtime DESC) AS last2
    FROM lxw1234
    ORDER BY cookieid,createtime;

    cookieid  createtime            url     rn     last1    last2
    -------------------------------------------------------------
    cookie1 2015-04-10 10:00:00     url1    1       url1    url7
    cookie1 2015-04-10 10:00:02     url2    2       url2    url7
    cookie1 2015-04-10 10:03:04     1url3   3       1url3   url7
    cookie1 2015-04-10 10:10:00     url4    4       url4    url7
    cookie1 2015-04-10 10:50:01     url5    5       url5    url7
    cookie1 2015-04-10 10:50:05     url6    6       url6    url7
    cookie1 2015-04-10 11:00:00     url7    7       url7    url7
    cookie2 2015-04-10 10:00:00     url11   1       url11   url77
    cookie2 2015-04-10 10:00:02     url22   2       url22   url77
    cookie2 2015-04-10 10:03:04     1url33  3       1url33  url77
    cookie2 2015-04-10 10:10:00     url44   4       url44   url77
    cookie2 2015-04-10 10:50:01     url55   5       url55   url77
    cookie2 2015-04-10 10:50:05     url66   6       url66   url77
    cookie2 2015-04-10 11:00:00     url77   7       url77   url77

时间： 2024-10-08 03:12:21

Hive分析窗口函数(四) LAG,LEAD,FIRST_VALUE,LAST_VALUE的相关文章

Hive函数：LAG,LEAD,FIRST_VALUE,LAST_VALUE

参考自大数据田地:http://lxw1234.com/archives/2015/04/190.htm 测试数据准备: create external table test_data ( cookieid string, createtime string, --页面访问时间 url string --被访问页面 ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' stored as textfile location '/user/jc_rc_ft

Hive分析窗口函数

Hive分析窗口函数(四) LAG,LEAD,FIRST_VALUE,LAST_VALUE https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics

Hive分析窗口函数(二) NTILE,ROW_NUMBER,RANK,DENSE_RANK

问题导读: 1.NTILE作用是什么? 2.按照pv降序排列,生成分组内每天的pv名次可使用哪个窗口函数? 3.RANK 和 DENSE_RANK作用是什么? 接上篇:Hive分析窗口函数(一)SUM,AVG,MIN,MAX 本文中介绍前几个序列函数,NTILE,ROW_NUMBER,RANK,DENSE_RANK,下面会一一解释各自的用途. Hive版本为 apache-hive-0.13.1 注意: 序列函数不支持WINDOW子句.(什么是WINDOW子句,Hive分析窗口函数(一)SUM,

Hive分析窗口函数(一) SUM,AVG,MIN,MAX

Hive分析窗口函数(一) SUM,AVG,MIN,MAX Hive分析窗口函数(一) SUM,AVG,MIN,MAX Hive中提供了越来越多的分析函数,用于完成负责的统计分析.抽时间将所有的分析窗口函数理一遍,将陆续发布. 今天先看几个基础的,SUM.AVG.MIN.MAX. 用于实现分组内所有和连续累积的统计. 数据准备 CREATE EXTERNAL TABLE lxw1234 ( cookieid string, createtime string, --day pv INT ) RO

Hive分析窗口函数(五) GROUPING SETS,GROUPING__ID,CUBE,ROLLUP

1.GROUPING SETS与另外哪种方式等价? 2.根据GROUP BY的维度的所有组合进行聚合由哪个关键字完成? 3.ROLLUP与ROLLUP关系是什么? GROUPING SETS,GROUPING__ID,CUBE,ROLLUP 这几个分析函数通常用于OLAP中,不能累加,而且需要根据不同维度上钻和下钻的指标统计,比如,分小时.天.月的UV数. Hive版本为 apache-hive-0.13.1 数据准备: 2015-03,2015-03-10,cookie1 2015-03,20

Hive学习之路 （十五）Hive分析窗口函数(三) CUME_DIST和PERCENT_RANK

这两个序列分析函数不是很常用,这里也练习一下. 数据准备 数据格式 cookie3.txt d1,user1,1000 d1,user2,2000 d1,user3,3000 d2,user4,4000 d2,user5,5000 创建表 use cookie; drop table if exists cookie3; create table cookie3(dept string, userid string, sal int) row format delimited fields te

[Android]Volley源码分析（四）

上篇中有提到NetworkDispatcher是通过mNetwork(Network类型)来进行网络访问的,现在来看一下关于Network是如何进行网络访问的. Network部分的类图: Network有一个实现类BasicNetwork,它有一个mHttpStack的属性,实际的网络请求是由这个mHttpStack来进行的,看BasicNetwork的performRequest()方法, 1 @Override 2 public NetworkResponse performRequest

Linux内核分析（四）----进程管理|网络子系统|虚拟文件系统|驱动简介

Linux内核分析(四) 两天没有更新了,上次博文我们分析了linux的内存管理子系统,本来我不想对接下来的进程管理子系统.网络子系统.虚拟文件系统在这个阶段进行分析的,但是为了让大家对内核有个整体的把握,今天还是简单的介绍一下剩余的几个子系统,我们对这几个子系统的分析,只要了解其作用和部分内容即可,不必深究,等我们写上几个驱动,到时候按照驱动再来分析这几个子系统我们就清晰多了. 在http://www.cnblogs.com/wrjvszq/p/4257164.html一文我们提到过linux

u-boot分析（四）---设置异常向量表|设置SVC模式

u-boot分析(四) 通过前三篇的分析,我们对u-boot已经有了整体的认识和掌握,但是我们仍然对于其部分硬件是如何初始化的不太清楚,所以接下来几篇博文我将会对我们在http://www.cnblogs.com/wrjvszq/archive/2015/01/10/4215627.html一文中总结出的u-boot的工作流程中的重要环节,结合文档加以分析. 今天我们会用到的文档: 1.        ARM Architecture Reference Manual:http://downlo