HIVE: Transform应用实例

数据文件内容

steven:100;steven:90;steven:99^567^22
ray:90;ray:98^456^30
Tom:81^222^33

期望最终放到数据库的数据格式如下：

steven    100    567     22
steven    90      567     22
steven    99      567     22
ray       90      456    30
ray       98      456    30
Tom       81      222    33

Specifically, if you want to return a different number of columns, or a different number of rows for a given input row, then yu need to perform what hive calls a transform.

1.创建表存储原始数据

create table u_data(col1 string, code int, age int) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘^‘ STORED AS TEXTFILE;

2.加载数据

load data local inpath ‘/home/stevenxia/data1‘ overwrite into table u_data;

3.编写transform脚本

#!/usr/bin/python
import sys
for line in sys.stdin:
 values = line.split()
 tmp = values[0]
 key_values = tmp.split(";")
 for kv in key_values:
  k = kv.split(":")[0]
  v = kv.split(":")[1]
  print ‘\t‘.join([k,v,values[1],values[2]])

4.把脚本部署到node节点, 位置 /home/stevenxia/u.py

5.这样hive就可以使用了

select transform(u.col1, u.code, u.age) using ‘/home/stevenxia/u.py‘ as (col1, col2, col3, col4) from (select * from u_data) as u;

运行结果