org.apache.spark.sql.AnalysisException: Can not create the managed table

spark执行过程中偶发性出现错误。

Traceback (most recent call last):
  File "/dfs/data9/nm-local-dir/usercache/hadoop/appcache/application_1666879209698_29104/container_e26_1666879209698_29104_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/dfs/data9/nm-local-dir/usercache/hadoop/appcache/application_1666879209698_29104/container_e26_1666879209698_29104_01_000001/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o193.saveAsTable.
: org.apache.spark.sql.AnalysisException: Can not create the managed table('`xy_fx`.`mdl_uniswapv3_stat_protocol_hour_daytmp01`'). The associated location('hdfs://ns1/user/hive/warehouse/xy_fx.db/mdl_uniswapv3_stat_protocol_hour_daytmp01') already exists.;
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.validateTableLocation(SessionCatalog.scala:336)
	at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:170)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)

最终找到问题原因:表已经删除,但是hdfs目录仍然存在,所以导致以上的报错。
解决方法:spark增加以下配置参数

.set("spark.sql.legacy.allowCreatingManagedTableUsingNonemptyLocation","true")
posted @ 2022-12-07 17:35  硅谷工具人  阅读(400)  评论(0编辑  收藏  举报
成功之道,在于每个人生阶段都要有不同的目标,并且通过努力实现自己的目标,毕竟人生不过百年! 所有奋斗的意义在于为个人目标实现和提升家庭幸福,同时能推进社会进步和国家目标! 正如古人讲的正心诚意格物致知,修身齐家治国平天下。