02.友盟项目--原始日志数据生成

友盟架构图1

友盟架构图2

日志生成类

package com.star.umeng.phone.domain;

/**
 * App日志的公共属性
 */
public class AppBaseLog {
    private Long createdAtMs;           //日志创建时间
    private String appId;               //应用唯一标识
    private String tenantId;            //租户唯一标识,企业用户
    private String deviceId;            //设备唯一标识
    private String appVersion;          //版本
    private String appChannel;          //渠道,安装时就在清单中制定了，appStore等。
    private String appPlatform;         //平台
    private String osType;              //操作系统
    private String deviceStyle;         //机型

    public Long getCreatedAtMs() {
        return createdAtMs;
    }

    public void setCreatedAtMs(Long createdAtMs) {
        this.createdAtMs = createdAtMs;
    }

    public String getAppId() {
        return appId;
    }

    public void setAppId(String appId) {
        this.appId = appId;
    }

    public String getTenantId() {
        return tenantId;
    }

    public void setTenantId(String tenantId) {
        this.tenantId = tenantId;
    }

    public String getDeviceId() {
        return deviceId;
    }

    public void setDeviceId(String deviceId) {
        this.deviceId = deviceId;
    }

    public String getAppVersion() {
        return appVersion;
    }

    public void setAppVersion(String appVersion) {
        this.appVersion = appVersion;
    }

    public String getAppChannel() {
        return appChannel;
    }

    public void setAppChannel(String appChannel) {
        this.appChannel = appChannel;
    }

    public String getAppPlatform() {
        return appPlatform;
    }

    public void setAppPlatform(String appPlatform) {
        this.appPlatform = appPlatform;
    }

    public String getOsType() {
        return osType;
    }

    public void setOsType(String osType) {
        this.osType = osType;
    }

    public String getDeviceStyle() {
        return deviceStyle;
    }

    public void setDeviceStyle(String deviceStyle) {
        this.deviceStyle = deviceStyle;
    }
}

AppBaseLog-App日志的公共属性

package com.star.umeng.phone.domain;

/**
 * 错误日志
 */
public class AppErrorLog extends AppBaseLog {
    private String errorBrief;        //错误摘要
    private String errorDetail;       //错误详情

    public String getErrorBrief() {
        return errorBrief;
    }

    public void setErrorBrief(String errorBrief) {
        this.errorBrief = errorBrief;
    }

    public String getErrorDetail() {
        return errorDetail;
    }

    public void setErrorDetail(String errorDetail) {
        this.errorDetail = errorDetail;
    }
}

AppErrorLog-错误日志

package com.star.umeng.phone.domain;

import java.util.Map;

/**
 * 事件日志
 */
public class AppEventLog extends AppBaseLog {
    private String eventId;                                //事件唯一标识
    private Long eventDurationSecs;                        //事件持续时长
    private Map<String, String> paramKeyValueMap;        //参数名/值对

    public String getEventId() {
        return eventId;
    }

    public void setEventId(String eventId) {
        this.eventId = eventId;
    }

    public Long getEventDurationSecs() {
        return eventDurationSecs;
    }

    public void setEventDurationSecs(Long eventDurationSecs) {
        this.eventDurationSecs = eventDurationSecs;
    }

    public Map<String, String> getParamKeyValueMap() {
        return paramKeyValueMap;
    }

    public void setParamKeyValueMap(Map<String, String> paramKeyValueMap) {
        this.paramKeyValueMap = paramKeyValueMap;
    }
    //getter/setter
}

AppEventLog-事件日志

package com.star.umeng.phone.domain;

/**
 * 页面日志
 */
public class AppPageLog extends AppBaseLog {
    private int pageViewCntInSession = 0;
    private String pageId;                        //页面id
    private int visitIndex = 0;                    //访问顺序号，0为第一个页面
    private String nextPage;                    //下一个访问页面，空表示为退出应用的页面
    private Long stayDurationSecs = (long) 0;     //当前页面停留时长

    public int getPageViewCntInSession() {
        return pageViewCntInSession;
    }

    public void setPageViewCntInSession(int pageViewCntInSession) {
        this.pageViewCntInSession = pageViewCntInSession;
    }

    public String getPageId() {
        return pageId;
    }

    public void setPageId(String pageId) {
        this.pageId = pageId;
    }

    public int getVisitIndex() {
        return visitIndex;
    }

    public void setVisitIndex(int visitIndex) {
        this.visitIndex = visitIndex;
    }

    public String getNextPage() {
        return nextPage;
    }

    public void setNextPage(String nextPage) {
        this.nextPage = nextPage;
    }

    public Long getStayDurationSecs() {
        return stayDurationSecs;
    }

    public void setStayDurationSecs(Long stayDurationSecs) {
        this.stayDurationSecs = stayDurationSecs;
    }
}

AppPageLog --页面日志

package com.star.umeng.phone.domain;

/**
 * 启动日志
 */
public class AppStartupLog extends AppBaseLog {
    private String country;           //国家，终端不用上报，服务器自动填充该属性
    private String province;          //省份，终端不用上报，服务器自动填充该属性
    private String ipAddress;         //ip地址

    private String network;           //网络
    private String carrier;           //运营商

    private String brand;             //品牌
    private String screenSize;        //分辨率

    public String getCountry() {
        return country;
    }

    public void setCountry(String country) {
        this.country = country;
    }

    public String getProvince() {
        return province;
    }

    public void setProvince(String province) {
        this.province = province;
    }

    public String getIpAddress() {
        return ipAddress;
    }

    public void setIpAddress(String ipAddress) {
        this.ipAddress = ipAddress;
    }

    public String getNetwork() {
        return network;
    }

    public void setNetwork(String network) {
        this.network = network;
    }

    public String getCarrier() {
        return carrier;
    }

    public void setCarrier(String carrier) {
        this.carrier = carrier;
    }

    public String getBrand() {
        return brand;
    }

    public void setBrand(String brand) {
        this.brand = brand;
    }

    public String getScreenSize() {
        return screenSize;
    }

    public void setScreenSize(String screenSize) {
        this.screenSize = screenSize;
    }
}

AppStartupLog --启动日志

package com.star.umeng.phone.domain;

/**
 * 使用日志
 */
public class AppUsageLog extends AppBaseLog {
    private Long singleUseDurationSecs;      //单次使用时长
    private Long singleUploadTraffic;        //单次使用过程中的上传流量
    private Long singleDownloadTraffic;      //单次使用过程中的下载流量

    public Long getSingleUseDurationSecs() {
        return singleUseDurationSecs;
    }

    public void setSingleUseDurationSecs(Long singleUseDurationSecs) {
        this.singleUseDurationSecs = singleUseDurationSecs;
    }

    public Long getSingleUploadTraffic() {
        return singleUploadTraffic;
    }

    public void setSingleUploadTraffic(Long singleUploadTraffic) {
        this.singleUploadTraffic = singleUploadTraffic;
    }

    public Long getSingleDownloadTraffic() {
        return singleDownloadTraffic;
    }

    public void setSingleDownloadTraffic(Long singleDownloadTraffic) {
        this.singleDownloadTraffic = singleDownloadTraffic;
    }
}

AppUsageLog --使用日志

package com.star.umeng.phone.domain;

import java.util.ArrayList;
import java.util.List;

/**
 * 日志聚合体
 */
public class AppLogAggEntity {
    private String appId;               //应用唯一标识
    private String tenantId;            //租户唯一标识,企业用户
    private String deviceId;            //设备唯一标识
    private String appVersion;          //版本
    private String appChannel;          //渠道,安装时就在清单中制定了，appStore等。
    private String appPlatform;         //平台
    private String osType;              //操作系统
    private String deviceStyle;         //机型

    //5中类型的日志集合
    private List<AppStartupLog> startupLogs = new ArrayList<AppStartupLog>();
    private List<AppErrorLog> errorLogs = new ArrayList<AppErrorLog>();
    private List<AppEventLog> eventLogs = new ArrayList<AppEventLog>();
    private List<AppUsageLog> usageLogs = new ArrayList<AppUsageLog>();
    private List<AppPageLog> pageLogs = new ArrayList<AppPageLog>();

    public String getAppId() {
        return appId;
    }

    public void setAppId(String appId) {
        this.appId = appId;
    }

    public String getTenantId() {
        return tenantId;
    }

    public void setTenantId(String tenantId) {
        this.tenantId = tenantId;
    }

    public String getDeviceId() {
        return deviceId;
    }

    public void setDeviceId(String deviceId) {
        this.deviceId = deviceId;
    }

    public String getAppVersion() {
        return appVersion;
    }

    public void setAppVersion(String appVersion) {
        this.appVersion = appVersion;
    }

    public String getAppChannel() {
        return appChannel;
    }

    public void setAppChannel(String appChannel) {
        this.appChannel = appChannel;
    }

    public String getAppPlatform() {
        return appPlatform;
    }

    public void setAppPlatform(String appPlatform) {
        this.appPlatform = appPlatform;
    }

    public String getOsType() {
        return osType;
    }

    public void setOsType(String osType) {
        this.osType = osType;
    }

    public String getDeviceStyle() {
        return deviceStyle;
    }

    public void setDeviceStyle(String deviceStyle) {
        this.deviceStyle = deviceStyle;
    }

    public List<AppStartupLog> getStartupLogs() {
        return startupLogs;
    }

    public void setStartupLogs(List<AppStartupLog> startupLogs) {
        this.startupLogs = startupLogs;
    }

    public List<AppErrorLog> getErrorLogs() {
        return errorLogs;
    }

    public void setErrorLogs(List<AppErrorLog> errorLogs) {
        this.errorLogs = errorLogs;
    }

    public List<AppEventLog> getEventLogs() {
        return eventLogs;
    }

    public void setEventLogs(List<AppEventLog> eventLogs) {
        this.eventLogs = eventLogs;
    }

    public List<AppUsageLog> getUsageLogs() {
        return usageLogs;
    }

    public void setUsageLogs(List<AppUsageLog> usageLogs) {
        this.usageLogs = usageLogs;
    }

    public List<AppPageLog> getPageLogs() {
        return pageLogs;
    }

    public void setPageLogs(List<AppPageLog> pageLogs) {
        this.pageLogs = pageLogs;
    }
}

AppLogAggEntity-- 日志聚合体

测试生成

package com.star.umeng.phone;

import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;

/**
 * 入口点程序
 */
public class Main {
    public static void main(String[] args) throws Exception {
        String strURL = "http://localhost/1.html" ;
        URL url = new URL(strURL) ;
        HttpURLConnection conn = (HttpURLConnection) url.openConnection();
        //设置请求方式
        conn.setRequestMethod("POST");
        //设置发送的内容类型
        conn.setRequestProperty("Content-Type" , "application/json");
        //允许输出到服务器
        conn.setDoOutput(true);
        OutputStream out = conn.getOutputStream() ;

        String json = "{\n" + "  \"appChannel\": \"anroid bus\",\n" + "  \"appId\": \"sohuvideo\",\n" + "  \"appPlatform\": \"ios\",\n" + "  \"appVersion\": \"1.1.0\",\n" + "  \"deviceStyle\": \"oppo 1\",\n" + "  \"errorLogs\": [\n" + "    {\n" + "      \"appChannel\": \"umeng\",\n" + "      \"appId\": \"gaodemap\",\n" + "      \"appPlatform\": \"blackberry\",\n" + "      \"appVersion\": \"1.1.0\",\n" + "      \"deviceStyle\": \"vivo 3\",\n" + "      \"errorBrief\": \"at cn.lift.dfdf.web.AbstractBaseController.validInbound(AbstractBaseController.java:72)\",\n" + "      \"errorDetail\": \"at cn.lift.dfdfdf.control.CommandUtil.getInfo(CommandUtil.java:67) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)\",\n" + "      \"osType\": \"android 4.0\",\n" + "      \"tenantId\": \"tnt023\"\n" + "    }\n" + "  ],\n" + "  \"eventLogs\": [\n" + "    {\n" + "      \"appChannel\": \"anroid bus\",\n" + "      \"appId\": \"tianya\",\n" + "      \"appPlatform\": \"android\",\n" + "      \"appVersion\": \"2.0.0\",\n" + "      \"deviceStyle\": \"红米\",\n" + "      \"eventId\": \"popmenu\",\n" + "      \"osType\": \"ios11\",\n" + "      \"tenantId\": \"tnt009\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"appstore\",\n" + "      \"appId\": \"gaodemap\",\n" + "      \"appPlatform\": \"android\",\n" + "      \"appVersion\": \"1.1.0\",\n" + "      \"deviceStyle\": \"iphone 7 plus\",\n" + "      \"eventId\": \"popmenu\",\n" + "      \"osType\": \"android 4.0\",\n" + "      \"tenantId\": \"tnt009\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"appstore\",\n" + "      \"appId\": \"faobengplay\",\n" + "      \"appPlatform\": \"blackberry\",\n" + "      \"appVersion\": \"1.0.0\",\n" + "      \"deviceStyle\": \"vivo 3\",\n" + "      \"eventId\": \"autoImport\",\n" + "      \"osType\": \"ios11\",\n" + "      \"tenantId\": \"tnt009\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"anroid bus\",\n" + "      \"appId\": \"tianya\",\n" + "      \"appPlatform\": \"blackberry\",\n" + "      \"appVersion\": \"1.1.0\",\n" + "      \"deviceStyle\": \"iphone 7 plus\",\n" + "      \"eventId\": \"popmenu\",\n" + "      \"osType\": \"mi 5.5\",\n" + "      \"tenantId\": \"tnt009\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"anroid bus\",\n" + "      \"appId\": \"tianya\",\n" + "      \"appPlatform\": \"android\",\n" + "      \"appVersion\": \"1.2.0\",\n" + "      \"deviceStyle\": \"iphone 7\",\n" + "      \"eventId\": \"bookstore\",\n" + "      \"osType\": \"ios11\",\n" + "      \"tenantId\": \"tnt501\"\n" + "    }\n" + "  ],\n" + "  \"osType\": \"ios11\",\n" + "  \"pageLogs\": [\n" + "    null,\n" + "    null,\n" + "    null,\n" + "    null,\n" + "    null\n" + "  ],\n" + "  \"startupLogs\": [\n" + "    {\n" + "      \"appChannel\": \"anroid bus\",\n" + "      \"appId\": \"faobengplay\",\n" + "      \"appPlatform\": \"ios\",\n" + "      \"appVersion\": \"1.2.0\",\n" + "      \"brand\": \"魅族\",\n" + "      \"carrier\": \"中国铁通\",\n" + "      \"country\": \"america\",\n" + "      \"deviceStyle\": \"vivo 3\",\n" + "      \"network\": \"wifi\",\n" + "      \"osType\": \"android 4.0\",\n" + "      \"province\": \"guangdong\",\n" + "      \"screenSize\": \"480 * 320\",\n" + "      \"tenantId\": \"tnt501\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"appstore\",\n" + "      \"appId\": \"sohuvideo\",\n" + "      \"appPlatform\": \"blackberry\",\n" + "      \"appVersion\": \"2.0.0\",\n" + "      \"brand\": \"Apple\",\n" + "      \"carrier\": \"中国铁通\",\n" + "      \"country\": \"china\",\n" + "      \"deviceStyle\": \"iphone 7\",\n" + "      \"network\": \"3g\",\n" + "      \"osType\": \"ios11\",\n" + "      \"province\": \"guangxi\",\n" + "      \"screenSize\": \"1136 * 640\",\n" + "      \"tenantId\": \"tnt501\"\n" + "    }\n" + "  ],\n" + "  \"tenantId\": \"tnt009\",\n" + "  \"usageLogs\": [\n" + "    {\n" + "      \"appChannel\": \"umeng\",\n" + "      \"appId\": \"gaodemap\",\n" + "      \"appPlatform\": \"winphone\",\n" + "      \"appVersion\": \"1.1.0\",\n" + "      \"deviceStyle\": \"iphone 7\",\n" + "      \"osType\": \"ios11\",\n" + "      \"tenantId\": \"tnt009\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"anroid bus\",\n" + "      \"appId\": \"tianya\",\n" + "      \"appPlatform\": \"android\",\n" + "      \"appVersion\": \"1.2.0\",\n" + "      \"deviceStyle\": \"iphone 6\",\n" + "      \"osType\": \"android 4.0\",\n" + "      \"tenantId\": \"tnt501\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"umeng\",\n" + "      \"appId\": \"tianya\",\n" + "      \"appPlatform\": \"winphone\",\n" + "      \"appVersion\": \"1.0.0\",\n" + "      \"deviceStyle\": \"vivo 3\",\n" + "      \"osType\": \"mi 5.5\",\n" + "      \"tenantId\": \"tnt023\"\n" + "    },\n" + "    {\n" + "      \"appChannel\": \"umeng\",\n" + "      \"appId\": \"sohuvideo\",\n" + "      \"appPlatform\": \"android\",\n" + "      \"appVersion\": \"1.0.0\",\n" + "      \"deviceStyle\": \"iphone 6 plus\",\n" + "      \"osType\": \"mi 5.5\",\n" + "      \"tenantId\": \"tnt023\"\n" + "    }\n" + "  ]\n" + "}" ;
        out.write(json.getBytes());
        out.flush();
        out.close();
        int code = conn.getResponseCode() ;
        if(code == 200){
            System.out.println("发送ok!!");
        }
    }
}

测试代码-生成数据

s102--s104查看生成的数据

原文地址：https://www.cnblogs.com/star521/p/9865381.html

时间： 2024-09-30 07:14:45

02.友盟项目--原始日志数据生成的相关文章

Poseidon 系统是一个日志搜索平台——认证看链接ppt，本质是索引的倒排列表和原始日志数据都存在HDFS，而文档和倒排的元数据都在NOSQL里，同时针对单个filed都使用了独立索引，使用MR来索引和搜索

Poseidon 系统是一个日志搜索平台,可以在百万亿条.100PB 大小的日志数据中快速分析和检索.360 公司是一个安全公司,在追踪 APT(高级持续威胁)事件,经常需要在海量的历史日志数据中检索某些信息,例如某个恶意样本在某个时间段内的活动情况.在 Poseidon 系统出现之前,都是写 Map/Reduce 计算任务在 Hadoop 集群中做计算,一次任务所需的计算时间从数小时到数天不等,大大制约了 APT 事件的追踪效率.Poseidon 系统就是解决这个需求,能在数百万亿条规模的数据

iOS 友盟统计崩溃日志查询

1.在Finder中找到你打出去的那个包,YourAppName.xcarchive. 2.然后Show Package Contents,在dSYMs 和 Products文件夹内找到这两个文件:YourAppName, YourAppName.app.dSYM 3.把这两个文件放到一个文件夹内,在终端进入该位置. 4.根据友盟统计上的错误分析,输入相应的如下命令行即可定位到出错内容: atos -arch armv7 -o YourAppName.app/YourAppName 0x306d

12 友盟项目--配置nginx的日志滚动

配置nginx的日志滚动-------------------------- 使用linux的crod调度器,周期性生成新文件. [编写滚动脚本 /usr/local/bin/umeng_roll_log.sh]     chmod  777     修改可执行权限 #!/bin/bash dateStr=`date '+%Y-%m-%d-%H-%M'` mv /usr/local/openresty/nginx/logs/access.log /usr/local/openresty/ngin

01.友盟项目--nginx服务器配置

Nginx实现反向代理 openresty-nginx增强版(现在使用的) localhost  反向代理到 s102--s104 http://openresty.org/cn/ 1.windows下 安装openresty,并修改nginx.conf配置文件 nginx.conf配置文件 #使用指定用户 #user root; #使用的处理器进程数 worker_processes 4; #错误日志文件 #error_log logs/error.log; #日志文件与日志级别 #error

15 友盟项目--资源文件工具类(ResourceUtil)、sql执行工具类(ExecSQLUtil）

资源文件工具类把sql脚本转换为String字符串--->交给sql工具类ExecSQLUtil执行sql 1.资源文件工具类(ResourceUtil) 把sql脚本转换为String字符串 /** * 资源文件工具类 */ public class ResourceUtil { /** * 以String方式读取整个资源串 */ public static String readResourceAsString(String resource ,String charset) throws

实现友盟在线统计--日志跟踪

#pragma mark --- 初始化 -(void) initUMengSDK { [MobClick startWithAppkey:umeng_app_key reportPolicy:BATCH channelId:nil];    [MobClick checkUpdateWithDelegate:self selector:@selector(displayUpdateNote:)]; //注册在线参数更新的通知    [[NSNotificationCenter defaultC

22 友盟项目--sparkstreaming对接kafka、集成redis--从redis中查询月留存率

 实时存储到redis,更新用户使用时间的最大.最小值-->来求留存率  避免全表扫描问题 1.spark 对接kafka 消费者   解析json 2.concat(appid,'#',appversion,'#',brand,'#',appplatform,'#',devicestyle,'#',ostype,'#',deviceid) ---> 作为key  各个维度 3.选出用户使用app的最小时间戳min , 最大时间戳max  -->作为value 4.存储到redis 依赖

友盟推出“数据工场”助跑开发者，“用户评级”和“微社区”齐亮相

今天,友盟推出了新战略“友盟数据工场”和数据产品“用户评级”.业界500多名开发者与我们一同见证了这一时刻.在此,感谢大家的支持!同时,想把我们最新的动态分享给所有盟友. “数据工场”能给开发者带来什么价值? 友盟高级技术总监叶谦介绍了“数据工场”的运转流程,其挖掘系统构建在开源大数据处理平台,通过设备识别.行为分析.设备评级等手段,日夜不停地对近十亿移动设备数据进行挖掘和分析,产出包括用户属性标签.用户兴趣标签.用户地理位置 POI .移动设备标识体系在内的多个数据成果. 这些数据挖掘成果将不

友盟让用户数据动起来——给app增加运营

让用户数据动起来——给app增加运营 一.初识友盟 友盟大家都听说过,在给app集成友盟之前对友盟的认识没有那么深刻.用了友盟之后,才发现友盟很强大. 集成友盟能够获取那些数据呢? 用户的基本信息:比如用户的数量.活跃用户的占比.用户地域的分析.用户增长速度等等. 用户的行为统计:用户对某模块的使用情况,对某个按钮的点击次数,按数量和值统计用户行为.让开发者更加清晰的分析用户行为,更清楚的了解自己,所谓知己知彼. app异常错误统计:monkey测试跑了一个礼拜都没有跑出来问题,用了友盟之后一个