dremio formatPlugin 调用链
格式化插件在dremio 中是比较核心的，包含了不少格式化插件，了解格式化插件的执行可以更好的开发自己的扩展以及了解dremio的内部机制
以下是基于arthas 分析的一个调用链可以参考
调用链
- stack parquet format plugin
stack com.dremio.exec.store.parquet.ParquetFormatPlugin getRecordWriter
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 113 ms, listenerId: 2
ts=2022-03-02 05:49:52;thread_name=e0 - 1de0fa7f-5211-b4a8-fe08-9b77891b3800:frag:0:0;id=63;is_daemon=false;priority=5;TCCL=sun.misc.Launcher$AppClassLoader@18b4aac2
@com.dremio.exec.store.parquet.ParquetFormatPlugin.getRecordWriter()
at com.dremio.exec.store.parquet.ParquetFormatPlugin.getWriterBatch(ParquetFormatPlugin.java:129)
at com.dremio.exec.store.parquet.ParquetWriterBatchCreator.create(ParquetWriterBatchCreator.java:26)
at com.dremio.exec.store.parquet.ParquetWriterBatchCreator.create(ParquetWriterBatchCreator.java:22)
at com.dremio.sabot.driver.OperatorCreatorRegistry.getSingleInputOperator(OperatorCreatorRegistry.java:73)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:257)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:116)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter(AbstractPhysicalVisitor.java:67)
at com.dremio.exec.physical.base.AbstractWriter.accept(AbstractWriter.java:29)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:261)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:116)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter(AbstractPhysicalVisitor.java:57)
at com.dremio.exec.physical.config.WriterCommitterPOP.accept(WriterCommitterPOP.java:107)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:261)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:116)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:82)
at com.dremio.exec.physical.config.Project.accept(Project.java:55)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:261)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:116)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriter(AbstractPhysicalVisitor.java:67)
at com.dremio.exec.physical.base.AbstractWriter.accept(AbstractWriter.java:29)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:261)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:116)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter(AbstractPhysicalVisitor.java:57)
at com.dremio.exec.physical.config.WriterCommitterPOP.accept(WriterCommitterPOP.java:107)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:261)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitOp(PipelineCreator.java:116)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:82)
at com.dremio.exec.physical.config.Project.accept(Project.java:55)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen(PipelineCreator.java:235)
at com.dremio.sabot.driver.PipelineCreator$CreatorVisitor.visitScreen(PipelineCreator.java:116)
at com.dremio.exec.physical.config.Screen.accept(Screen.java:63)
at com.dremio.sabot.driver.PipelineCreator.get(PipelineCreator.java:108)
at com.dremio.sabot.driver.PipelineCreator.get(PipelineCreator.java:102)
at com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution(FragmentExecutor.java:394)
at com.dremio.sabot.exec.fragment.FragmentExecutor.run(FragmentExecutor.java:269)
at com.dremio.sabot.exec.fragment.FragmentExecutor.access$1600(FragmentExecutor.java:94)
at com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run(FragmentExecutor.java:747)
at com.dremio.sabot.task.AsyncTaskWrapper.run(AsyncTaskWrapper.java:112)
at com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop(SlicingThread.java:243)
at com.dremio.sabot.task.slicing.SlicingThread.run(SlicingThread.java:171)
- stack parquet writer
stack com.dremio.exec.physical.base.AbstractWriter accept
Press Q or Ctrl+C to abort.
Affect(class count: 4 , method count: 1) cost in 286 ms, listenerId: 7
ts=2022-03-02 05:58:04;thread_name=1de0f894-0b7e-2d88-cd3a-a6265fa2f200/0:foreman-planning;id=d1;is_daemon=true;priority=10;TCCL=sun.misc.Launcher$AppClassLoader@18b4aac2
@com.dremio.exec.physical.base.AbstractWriter.accept()
at com.dremio.exec.planner.sql.handlers.PrelTransformer$PopCollector.visitOp(PrelTransformer.java:876)
at com.dremio.exec.planner.sql.handlers.PrelTransformer$PopCollector.visitOp(PrelTransformer.java:870)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitWriterCommiter(AbstractPhysicalVisitor.java:57)
at com.dremio.exec.physical.config.WriterCommitterPOP.accept(WriterCommitterPOP.java:107)
at com.dremio.exec.planner.sql.handlers.PrelTransformer$PopCollector.visitOp(PrelTransformer.java:876)
at com.dremio.exec.planner.sql.handlers.PrelTransformer$PopCollector.visitOp(PrelTransformer.java:870)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitProject(AbstractPhysicalVisitor.java:82)
at com.dremio.exec.physical.config.Project.accept(Project.java:55)
at com.dremio.exec.planner.sql.handlers.PrelTransformer$PopCollector.visitOp(PrelTransformer.java:876)
at com.dremio.exec.planner.sql.handlers.PrelTransformer$PopCollector.visitOp(PrelTransformer.java:870)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitStore(AbstractPhysicalVisitor.java:142)
at com.dremio.exec.physical.base.AbstractPhysicalVisitor.visitScreen(AbstractPhysicalVisitor.java:200)
at com.dremio.exec.physical.config.Screen.accept(Screen.java:63)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToPlan(PrelTransformer.java:862)
at com.dremio.exec.planner.sql.handlers.PrelTransformer.convertToPlan(PrelTransformer.java:867)
at com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan(NormalHandler.java:137)
at com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan(HandlerToExec.java:59)
at com.dremio.exec.work.foreman.AttemptManager.plan(AttemptManager.java:488)
at com.dremio.exec.work.foreman.AttemptManager.lambda$run$1(AttemptManager.java:387)
at com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3(ReleasableBoundCommandPool.java:138)
at com.dremio.service.commandpool.CommandWrapper.run(CommandWrapper.java:62)
at com.dremio.context.RequestContext.run(RequestContext.java:95)
at com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3(ContextMigratingExecutorService.java:199)
at com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run(ContextMigratingExecutorService.java:180)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
参考资料
sabot/kernel/src/main/java/com/dremio/exec/store/dfs/FormatPlugin.java
