对于INNODB,主键就是聚集索引,如果没有主键定义,则第一个唯一非空索引被作为聚集索引.
如果没有主键也没有合适的唯一索引,那么innodb内部会生成一个隐藏的主键作为聚集索引,
这个隐藏的主键类似一个自增的id(int).
删除与重新添加主键
alter table tbname drop primary key;
alter table tbname add primary key(col1,col2,col3);
CREATE INDEX IDX_METE_BASE_AREA_STATION_CABINET ON CMS.T_CABINET_ELECTRICITY_HH_HIVE (
BASE_NAME ASC,
AREA_NAME ASC,
STATION_NAME ASC,
CABINET_NAME ASC
);
SELECT DISTINCT DEVICE_NAME ,REGEXP_EXTRACT(DEVICE_NAME,"-(.*)",0)
FROM T_METE_CONFIG
WHERE LSC_ID IN (30,31)
AND DEVICE_TYPE_NAME=‘UPS配电‘
AND STATION_NAME LIKE ‘%数据机房%‘
ORDER BY DEVICE_NAME;
SELECT REGEXP_EXTRACT(‘ASDFSADF-JOB‘,‘-([A-Z])‘,0) FROM CMS.DUAL LIMIT 1;
复合分区示例,
先以时间粒度天为例进行外分区,然后根据地域id进行分区,
mysql仅支持range和list的子分区,而且子分区只能是hash或key.
CREATE TABLE t_cabinet_electricity_hh_hive2
(
base_id int NOT NULL,
base_name varchar(20),
lsc_name varchar(255),
area_id bigint,
area_name varchar(255),
station_id int NOT NULL,
station_name varchar(255),
device_type_id varchar(255),
device_type_name varchar(255),
device_id varchar(255) NOT NULL,
device_name varchar(255),
mete_id int NOT NULL,
mete_name varchar(255),
max_report_time varchar(20),
max_mete_value double,
min_report_time varchar(20),
min_mete_value double,
avg_mete_value double,
hour_sum_value double,
hour_times int,
dt_hh varchar(10) NOT NULL,
hour_last_value double,
cabinet_name varchar(255),
dt varchar(20) NOT NULL,
lsc_id int NOT NULL,
PRIMARY KEY (dt,dt_hh,base_id,lsc_id,mete_id,device_id,station_id,mete_name)
)
PARTITION BY RANGE(TO_DAYS(DT))
SUBPARTITION BY HASH(LSC_ID)
SUBPARTITIONS 6
(
PARTITION P0 VALUES LESS THAN (TO_DAYS(‘20170810‘)),
PARTITION P1 VALUES LESS THAN (TO_DAYS(‘20170811‘)),
PARTITION P2 VALUES LESS THAN (TO_DAYS(‘20170812‘)),
PARTITION P3 VALUES LESS THAN (TO_DAYS(‘20170813‘)),
PARTITION P4 VALUES LESS THAN (TO_DAYS(‘20170814‘)),
PARTITION P5 VALUES LESS THAN (TO_DAYS(‘20170815‘)),
PARTITION P6 VALUES LESS THAN (TO_DAYS(‘20170816‘)),
PARTITION P7 VALUES LESS THAN (TO_DAYS(‘20170817‘)),
PARTITION P8 VALUES LESS THAN (TO_DAYS(‘20170818‘)),
PARTITION P9 VALUES LESS THAN (TO_DAYS(‘20170819‘)),
PARTITION P10 VALUES LESS THAN (TO_DAYS(‘20170820‘))
)
;
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P11 VALUES LESS THAN (TO_DAYS(‘20170821‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P12 VALUES LESS THAN (TO_DAYS(‘20170822‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P13 VALUES LESS THAN (TO_DAYS(‘20170823‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P14 VALUES LESS THAN (TO_DAYS(‘20170824‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P15 VALUES LESS THAN (TO_DAYS(‘20170825‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P16 VALUES LESS THAN (TO_DAYS(‘20170826‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P17 VALUES LESS THAN (TO_DAYS(‘20170827‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P18 VALUES LESS THAN (TO_DAYS(‘20170828‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P19 VALUES LESS THAN (TO_DAYS(‘20170829‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P20 VALUES LESS THAN (TO_DAYS(‘20170830‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P21 VALUES LESS THAN (TO_DAYS(‘20170831‘)));
ALTER TABLE t_cabinet_electricity_hh_hive2 ADD PARTITION (PARTITION P22 VALUES LESS THAN (TO_DAYS(‘20170901‘)));
--查看分区信息
select
partition_name,table_rows
from information_schema.partitions where table_name=‘xxx‘;
问题,hive表中的日期类型 yyyy-mm-dd 到mysql中对应的date字段,但是无法导入成功
其实看sqoop转换的java代码,可以看出这是sqoop的一个bug.,
里面有类似 (java.sql.Date)value
转换失败为空,这样进入我含有非空约束的字段就失败了.
原因是日期格式是不确定的,所以直接转有问题,
建议是直接设置临时表的字段为varchar,后面带做处理.
sqoop eval --connect jdbc:mysql://xx:3306/cms --username root --password xxx \
--query "truncate table xx.stage_t_xx"
sqoop export --connect jdbc:mysql://xxx:3306/xxx\
--username root --password xxx --table stage_t_xxe \
--hcatalog-database xxx--hcatalog-table stage_txxx
sqoop eval --connect jdbc:mysql://xxx:3306/cms --username root --password xxx \
--query "insert into xxx.t_xxx_xx_hh_xx select * from xxx.stage_t_xx_xx_hh_hive"