dremio24 版本copy into from 简单说明

copy into from 是dremio 24 版本很不错的一个新功能，可以提升数据加载处理的速度（以前对于iceberg 数据的处理有insert into 以及通过ctas 模式）

简单机制说明

目前看到copy into from 并不象其他sql 命令基于sql SimpleDirectHandler 模式，而是通过sql 关系转换的
DremioSqlToRelConverter 中的

 

  private RelRoot convertOther(SqlNode query, boolean top, RelDataType targetRowType) {

    if (query instanceof SqlOptimize) {

      NamespaceKey path = ((SqlOptimize)query).getPath();

      Prepare.PreparingTable nsTable = catalogReader.getTable(path.getPathComponents());

      return RelRoot.of(new TableOptimizeCrel(cluster, cluster.traitSetOf(Convention.NONE), nsTable.toRel(createToRelContext()), nsTable, null, OptimizeOptions.DEFAULT), SqlKind.OTHER);

    } else if (query instanceof SqlCopyIntoTable) {

      NamespaceKey path = ((SqlCopyIntoTable)query).getPath();

      Prepare.PreparingTable nsTable = catalogReader.getTable(path.getPathComponents());

      CopyIntoTableContext  copyIntoTableContext= new CopyIntoTableContext((SqlCopyIntoTable)query);

      return RelRoot.of(new CopyIntoTableCrel(cluster, cluster.traitSetOf(Convention.NONE), nsTable,  copyIntoTableContext), SqlKind.OTHER);

    } else  {

      return super.convertQueryRecursive(query, top, targetRowType);

    }

  }

几个核心类

主要是CopyIntoTableContext，CopyIntoTableCrel ，CopyIntoTableRel ，CopyIntoTablePrule,CopyIntoTableRule
CopyIntoTableRule 主要是为PlannerPhase 逻辑阶段提供规则（CopyIntoTableRule 使用了CopyIntoTableContext，CopyIntoTableCrel ，CopyIntoTableRel ）
CopyIntoTablePrule 主要是为PlannerPhase 物理计划规则提供规则，实际物理规则的生成是通过了CopyIntoTablePlanBuilder ，里边包含了实际物理规则的处理，值得学习参考

说明

以上是简单的说明，内部好多细节是值得学习参考的，而且目前copy into from 属于append 模式的（也比较符合日常数据湖数据处理模式）

参考资料

sabot/kernel/src/main/java/com/dremio/exec/planner/sql/handlers/query/CopyIntoTableContext.java
sabot/kernel/src/main/java/com/dremio/exec/planner/sql/DremioSqlToRelConverter.java
sabot/kernel/src/main/java/com/dremio/exec/calcite/logical/CopyIntoTableCrel.java
sabot/kernel/src/main/java/com/dremio/exec/planner/physical/CopyIntoTablePrule.java
sabot/kernel/src/main/java/com/dremio/exec/planner/PlannerPhase.java
sabot/kernel/src/main/java/com/dremio/exec/planner/CopyIntoTablePlanBuilder.java
sabot/kernel/src/main/java/com/dremio/exec/planner/logical/CopyIntoTableRule.java
https://docs.dremio.com/software/data-formats/copying-data-into-tables/