http://www.codelast.com/?p=4249
http://www.klshu.com/656.html
o_data = LOAD ‘hdfs://hadoop/user/xuting/analy/userDevtmp‘ using PigStorage(‘`‘) AS (userId,userMac,Time_Stamp,reqUserAgent,device,time);
pig
cd hdfs:///user/ xuting/analy/userDevtmp
data2015 = FILTER o_data BY time MATCHES ‘2015.*‘;
因为我们自己添加了两个字段 设备号和时间 这个是肯定没有问题的 你可以匹配最后一个时间的字段 如果不满足‘2015.*’这样滤掉数据
data_head_error = FILTER data2015 BY userId MATCHES ‘^A‘;
汇总
B = GROUP data all;
C = FOREACH B GENERATE COUNT(data);
DUMP C;
B = GROUP data all;
C = FOREACH B GENERATE COUNT_STAR(data);
DUMP C;
B = GROUP data by userId;
C = FOREACH B GENERATE group, COUNT(data);
空值
item = FOREACH data GENERATE reqUserAgent;
A = FILTER item BY reqUserAgent is null;
B = GROUP A all;
C = FOREACH B GENERATE COUNT_STAR(A) ;
dump C;
Missing
item = FOREACH data GENERATE reqUserAgent;
A = FILTER item BY reqUserAgent ==‘Missing‘;
B = GROUP A all;
C = FOREACH B GENERATE COUNT_STAR(A) ;
dump C;
异常
item = FOREACH data GENERATE reqUserAgent;
A = FILTER item BY reqUserAgent ==‘-‘;
B = GROUP A all;
C = FOREACH B GENERATE COUNT_STAR(A) ;
dump C;
匹配
item = FOREACH data GENERATE userId;
A = FILTER data BY NOT userId MATCHES ‘^U[0-9a-zA-Z]{47}$‘;
B = GROUP A all;
C = FOREACH B GENERATE COUNT_STAR(A) ;
dump C;
1. userId : 以U开头，后面接47个[0-9a-zA-Z]范围内的字符
‘^U[0-9a-zA-Z]{47}$‘
2. userMac : ‘^[0-9a-zA-Z]{2}:[0-9a-zA-Z]{2}:[0-9a-zA-Z]{2}:[0-9a-zA-Z]{2}:[0-9a-zA-Z]{2}:[0-9a-zA-Z]{2}$‘
3. Time_Stamp : [0-9]范围内的数字共10位
‘^[0-9]{10}$‘
4. reqUserAgent 这个数据的异常情况先不用分析
数据样例：
Dalvik/1.6.0 (Linux; U; Android 4.3; ZTE Q805T Build/Q805TV1.0.0B02)
Mozilla/5.0 (Linux; U; Android 4.4.4; zh-cn; YL-Coolpad 8297-T01 Build/KVT49L) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30