【解决】hive动态添加partitions不能超过100的问题
Author: kwu
【解决】hive动态添加partitions不能超过100的问题,全量动态生成partitions超过100会出现例如以下异常:
The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100
解决100限制,可设置例如以下參数:
set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.dynamic.partitions=100000; SET hive.exec.max.dynamic.partitions.pernode=100000; insert overwrite table ods.cms_entity PARTITION (DAY) select ENTITY_ID , ENTITY_NAME , ENTITY_DESC , ENTITY_TYPE , ENTITY_PID , ENTITY_TIME , ENTITY_PRIORITY , ENTITY_STATUS , ENTITY_CHANNEL , ENTITY_EDITOR , ENTITY_TEMPLATE , ENTITY_URL , ENTITY_CATEGORY , ENTITY_PARAM , ENTITY_SHORTNAME , ENTITY_SUBTYPE , ENTITY_COMPDELAY , day from stage.cms_entity_by_day;