dremio 23 反射异常问题原因分析简单说明
通过几天的分析，大致可以确认dremio 23 的问题，应该是一个exception 引起的，以下是通过使用jpropfiler 发现的一个exception 信息
现象
- 参考图
- 说明
从上图可以看出是一个MatchFailed 的exception 引起的，但是目前没有影响业务，因为异常之后使用了默认的
参考代码
public static Optional<List<RexNode>> rewriteRexNodeList(RexRewriteContext rexRewriteContext, LogicalProject materializationNode, RelNode query, List<? extends RexNode> nodeListToRewrite, boolean unifyLiterals) {
TargetMapper shuttle = createRexShuttle(rexRewriteContext, materializationNode, query, unifyLiterals);
boolean failed = false;
ImmutableList.Builder<RexNode> transformedNodes = ImmutableList.builder();
Iterator var8 = nodeListToRewrite.iterator();
while(var8.hasNext()) {
RexNode nodeToRewrite = (RexNode)var8.next();
try {
RexNode node = (RexNode)nodeToRewrite.accept(shuttle);
transformedNodes.add(node);
} catch (MatchFailed var11) {
failed = true;
}
}
return failed ? Optional.empty() : Optional.of(transformedNodes.build());
}
都次方法调用信息
watch com.dremio.reflection.rules.RexRewrite rewriteRexNodeList '{params, target, returnObj, throwExp}' -x 2
[arthas@5289]$ watch com.dremio.reflection.rules.RexRewrite rewriteRexNodeList '{params, target, returnObj, throwExp}' -x 2
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 2) cost in 432 ms, listenerId: 3
method=com.dremio.reflection.rules.RexRewrite.rewriteRexNodeList location=AtExit
ts=2022-12-11 11:54:54; [cost=2.673281ms] result=@ArrayList[
@Object[][
@RexRewriteContext[com.dremio.reflection.rules.RexRewriteContext@58dbea81],
@LogicalProject[rel#9000:LogicalProject.NONE.ANY([]).[](input=ScanCrel#8967,id=$0,code=$1,name=$2,parent_id=$3,short_name=$4,bstype_code=$5,org_id=$6,logo=$7,creator=$8,create_time=$9,modifier=$10,modify_time=$11,deletor=$12,delete_time=$13,status=$14,remark=$15,ts=$16,dr=$17,mdm_ts=$18,mdm_code=$19,res_code=$20)],
@ReflectionPtr[ReflectionPtr#9021],
@RegularImmutableList[isEmpty=false;size=21],
@Boolean[true],
],
null,
@Optional[
EMPTY=@Optional[Optional.empty],
value=null,
],
null,
]
method=com.dremio.reflection.rules.RexRewrite.rewriteRexNodeList location=AtExit
ts=2022-12-11 11:54:54; [cost=0.872951ms] result=@ArrayList[
@Object[][
@RexRewriteContext[com.dremio.reflection.rules.RexRewriteContext@4bbf5895],
@LogicalProject[rel#9000:LogicalProject.NONE.ANY([]).[](input=ScanCrel#8967,id=$0,code=$1,name=$2,parent_id=$3,short_name=$4,bstype_code=$5,org_id=$6,logo=$7,creator=$8,create_time=$9,modifier=$10,modify_time=$11,deletor=$12,delete_time=$13,status=$14,remark=$15,ts=$16,dr=$17,mdm_ts=$18,mdm_code=$19,res_code=$20)],
@ScanCrel[rel#8930:ScanCrel.NONE.ANY([]).[](table=mysql.boss_admin.prd_line,columns=`id`, `code`, `name`, `parent_id`, `short_name`, `bstype_code`, `org_id`, `logo`, `creator`, `create_time`, `modifier`, `modify_time`, `deletor`, `delete_time`, `status`, `remark`, `ts`, `dr`, `mdm_ts`, `mdm_code`, `res_code`,splits=1,tableDigest=-866816580|mysql|aff76261-76d1-4852-8f3d-716542dd55ac)],
@TransformingRandomAccessList[isEmpty=false;size=21],
@Boolean[false],
],
null,
@Optional[
EMPTY=@Optional[Optional.empty],
value=null,
],
null,
]
投影处理时候对于调用
public UnifyResult apply(ReflectionRuleContext context, UnifyRuleCall call) {
LogicalProject target = (LogicalProject)call.getTarget();
LogicalProject query = (LogicalProject)call.getQuery();
// dremio 22 版本 看着是直接就返回了null
if (RelOptUtil.areRowTypesEqual(target.getRowType(), query.getRowType(), false) && target.getProjects().toString().equals(query.getProjects().toString())) {
return null;
} else {
// dremio 23 版本执行了此处，会有异常
Optional<List<RexNode>> newProjects = RexRewrite.rewriteRexNodeList(new RexRewriteContext(context.getHintFeatureGatherer(), query, target), target, query.getInput(), query.getProjects(), true);
if (!newProjects.isPresent()) {
return null;
} else {
LogicalProject newProject = LogicalRels.createProject(query.getRowType(), call.getReflection(), (List)newProjects.get());
RelNode newProject2 = LogicalRels.strip(newProject);
if (newProject2 == newProject.getInput()) {
return call.result(newProject2);
} else {
newProjects = RexRewrite.rewriteRexNodeList(new RexRewriteContext(context.getHintFeatureGatherer(), query, target), target, query, query.getProjects(), false);
if (!newProjects.isPresent()) {
return null;
} else {
newProject = LogicalRels.createProject(query.getRowType(), call.getReflection(), (List)newProjects.get());
return call.result(newProject);
}
}
}
}
}
实际上dremio 界面上也会有关于反射使用的profile 信息，推荐开启planner.verbose_profile=true, 从效果来看应该是23 之后要么是类型处理，或者查询计划处理的问题
我尝试过直接替换ce 的包，发现也是不行的，应该是kernel 部分的一些东西也是有关系的，大致对比代码发现了一些sql 转换到关系算子部分变化还是很大的，比如PrelTransformer，整体上参考github dremio 08-23 之后的changelog
说明
以上是一个简单的初步分析，dremio 里边的一些东西还是比较复杂的，还得好好分析学习，才能明确实际产生问题的原因
