songchaolin - 博客园

2020年4月

摘要：阅读全文

posted @ 2020-04-30 10:59 songchaolin 阅读(438) 评论(0) 推荐(0) 编辑

摘要：接到新的需求,在spark中增加埋点，解析任务的血缘关系，包括sql和代码方式，不包括中间临时视图(createOrReplaceTempView(XXX表))。有位同学已经https://www.cnblogs.com/wuxilc/p/9326130.html 做了hive解析相关的，但是sp 阅读全文

posted @ 2020-04-30 10:25 songchaolin 阅读(2496) 评论(0) 推荐(0) 编辑

idea java项目引入checkstyle检查

摘要：一 maven项目 pom引入checkstyle依赖 plugins 里面添加 <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-checkstyle-plugin</artifactId> <versio 阅读全文

posted @ 2020-04-16 17:18 songchaolin 阅读(3422) 评论(0) 推荐(0) 编辑

2020年3月

spark查询hive表因库跟表名反引号报错Table or view not found

摘要：使用spark查询hive表时，报错如下: 20/03/23 11:24:10 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint Error in query: Table or view not fou 阅读全文

posted @ 2020-03-23 11:29 songchaolin 阅读(1898) 评论(0) 推荐(0) 编辑

2019年12月

hivesql迁移spark2.4 cannot resolve '(ctime >= start_time) due to data type mismatch: differing types in '(ctime` >= `start_time`)'(timestamp and bigint).; line 99 pos 10

摘要： hivesql 一些默认类型转换。但spark需要手动强转比如以下部分: SELECT 1 as id, avid as avid, mid as mid, TRIM(LOWER(tag)) as tag, ctime as ctime FROM archive.dws_archive_daily 阅读全文

posted @ 2019-12-31 14:32 songchaolin 阅读(1729) 评论(0) 推荐(0) 编辑

sparksql报错 cannot resolve 'coalesce(d.`field`, map(NULL, NULL))' due to data type mismatch: input to function coalesce should all be the same type

摘要： hivesql可运行的sql 在spark上运行报错如下 Error in query: cannot resolve 'coalesce(d.`subarea_duration_l30`, map(NULL, NULL))' due to data type mismatch: input to 阅读全文

posted @ 2019-12-26 17:30 songchaolin 阅读(1369) 评论(0) 推荐(0) 编辑

hivesql 迁移spark3.0 sparksql报错如Cannot safely cast '字段':StringType to IntegerType的问题

摘要：一问题 hivesql可以正常运行，spark3.0运行报错如图 spark3.0配置查看源码新增一个 val STORE_ASSIGNMENT_POLICY = buildConf("spark.sql.storeAssignmentPolicy") .doc("When inserting 阅读全文

posted @ 2019-12-25 19:58 songchaolin 阅读(6930) 评论(0) 推荐(0) 编辑

sparksql 报错Container killed by YARN for exceeding memory limits. xGB of x GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead or disabling

摘要：对此提高了对外内存 spark.executor.memoryOverhead = 4096m 重新执行sql 改报下面的错误 19/12/25 15:49:02 ERROR ShuffleBlockFetcherIterator: Failed to get block(s) from bigd 阅读全文

posted @ 2019-12-25 18:55 songchaolin 阅读(3383) 评论(0) 推荐(1) 编辑

hiveSql 迁移spark2.4时报错Error in query: Window function row_number() requires window to be ordered, please add ORDER BY clause

摘要： hivesql对语法检查较弱像下面的语法 hive是可以通过的 partition by 后没有跟order by row_number() over(partition by buvid,version_code,app_id) as rn 原因看下hive 源码(hive 已经做了补充) sp 阅读全文

posted @ 2019-12-25 16:08 songchaolin 阅读(1177) 评论(0) 推荐(0) 编辑

spark 执行报错 java.io.EOFException: Premature EOF from inputStream

摘要：使用spark2.4跟spark2.3 做替代公司现有的hive选项。跑个别任务spark有以下错误 java.io.EOFException: Premature EOF from inputStream at com.hadoop.compression.lzo.LzopInputStream 阅读全文

posted @ 2019-12-23 19:42 songchaolin 阅读(2619) 评论(0) 推荐(0) 编辑

公告