RowListener

前言

代码说明

TableInputRowAdapter

TableOutputRowAdapter

其他代码段

后记

完整交换代码

前言

本次目的是监控数据，然后能操作数据，同时继承TableInput,TableInputMeta和TableOutput，TableOutputMeta类，能实现其中主要的方法--processRow()。

首先来看监控数据，也就是BaseStep类中的addRowListener()方法，当然其中还要很多其他方法。

这个方法里面需要一个RowListener接口，里面有三个方法，如下所示，注解很清晰的说明了每个方法被调用的情况，此处我用的是RowAdapter类，实现了这个RowListener接口。

public interface RowListener {
  /**
   * This method is called when a row is read from another step
   *
   * @param rowMeta
   *          the metadata of the row
   * @param row
   *          the data of the row
   * @throws KettleStepException
   *           an exception that can be thrown to hard stop the step
   */
  public void rowReadEvent( RowMetaInterface rowMeta, Object[] row ) throws KettleStepException;

  /**
   * This method is called when a row is written to another step (even if there is no next step)
   *
   * @param rowMeta
   *          the metadata of the row
   * @param row
   *          the data of the row
   * @throws KettleStepException
   *           an exception that can be thrown to hard stop the step
   */
  public void rowWrittenEvent( RowMetaInterface rowMeta, Object[] row ) throws KettleStepException;

  /**
   * This method is called when the error handling of a row is writing a row to the error stream.
   *
   * @param rowMeta
   *          the metadata of the row
   * @param row
   *          the data of the row
   * @throws KettleStepException
   *           an exception that can be thrown to hard stop the step
   */
  public void errorRowWrittenEvent( RowMetaInterface rowMeta, Object[] row ) throws KettleStepException;
}

还是上篇博客那个mysql交换，首先为了能得到error数据，走errorRowWritetenEvent方法，将dest的表name从varchar改为int。

kettle调试查看源码数据_lw18751836671的专栏-CSDN博客目录前言运行TableInput描述TableOutput描述前言在之前我写了如何通过java 脚本来修改数据，从而确定有一个processRow()方法，该方法中能获取到数据信息等，那么接下来就是需要运行一个简单的表交换来看数据是怎么流的。首先写一个简单的tableInput->tableOutput的交换，这里代码就贴在最后，毕竟只是用mysql表交换，数据也不多，太简单了。定义一个源表，放两条数据，然后新建一个u...https://blog.csdn.net/lw18751836671/article/details/121519115?spm=1001.2014.3001.5501

CREATE TABLE `user_info_dest` (
  `id` varchar(36) NOT NULL,
  `name` int(36) DEFAULT NULL
)

源表里面一条是数字，一条是英文，这样就有一条异常数据了，

同时我们用如下图的流程来改造一下mysql交换，完整代码贴在最后，

上图这个图片流程在博客中也有，链接在下面，

kettle中错误处理_lw18751836671的专栏-CSDN博客目录一、图文说明二、错误处理代码上述的图片整体代码三、扩展图片说明完整代码一、图文说明在如上图中，有定义一个tableOutputError，如果在select节点发生错误，那么就将进入此tableOutputError，将错误数据输出到表。但是如果是tableOutput处发生了错误，那么就无法进行错误处理了，后面没有定义错误处理。界面选择错误处理如下，只有有定义错误处理选项的才能...https://blog.csdn.net/lw18751836671/article/details/121339655?spm=1001.2014.3001.5501

代码说明

如下图所示的结构，这些都是继承原来jar包中的类，

这里的重点是RowAdapter类，本来打算是修改processRow方法的，对其中异常进行一些包裹后处理，这样能让其他正常数据能走下去，但想了想其实也很简单，没啥意义。

TableInputRowAdapter

@Slf4j
public class TableInputRowAdapter extends RowAdapter {

    private String adapterName;

    public TableInputRowAdapter(String adapterName){
        this.adapterName = adapterName;
    }

    @Override
    public void rowReadEvent(RowMetaInterface rowMeta, Object[] row) throws KettleStepException {
        /**
         * org.pentaho.di.trans.step.BaseStep#getRow()
         *      org.pentaho.di.trans.step.BaseStep#handleGetRow() 此处调用 rowReadEvent
         */
//        super.rowReadEvent(rowMeta, row);
        PrintUtil.printlnData(adapterName+"-->rowReadEvent",rowMeta,row);
    }

    @Override
    public void rowWrittenEvent(RowMetaInterface rowMeta, Object[] row) throws KettleStepException {
        /*
        org.pentaho.di.trans.step.BaseStep.putRow
                org.pentaho.di.trans.step.BaseStep.handlePutRow 此处调用 rowWrittenEvent
         */
//        super.rowWrittenEvent(rowMeta, row);
        PrintUtil.printlnData(adapterName+"-->rowWrittenEvent",rowMeta,row);
    }

    @Override
    public void errorRowWrittenEvent(RowMetaInterface rowMeta, Object[] row) throws KettleStepException {
        //当发生错误时，将错误数据打印出来
        PrintUtil.printlnData(adapterName+"-->errorRowWrittenEvent",rowMeta,row);
    }


}

TableOutputRowAdapter

@Slf4j
public class TableOutputRowAdapter extends RowAdapter {

    private String adapterName;

    public TableOutputRowAdapter(String adapterName){
        this.adapterName = adapterName;
    }

    @Override
    public void rowReadEvent(RowMetaInterface rowMeta, Object[] row) throws KettleStepException {
        /**
         * org.pentaho.di.trans.step.BaseStep#getRow()
         *      org.pentaho.di.trans.step.BaseStep#handleGetRow() 此处调用 rowReadEvent
         */
//        super.rowReadEvent(rowMeta, row);
        PrintUtil.printlnData(adapterName+"-->rowReadEvent",rowMeta,row);
    }

    @Override
    public void rowWrittenEvent(RowMetaInterface rowMeta, Object[] row) throws KettleStepException {
        /*
        org.pentaho.di.trans.step.BaseStep.putRow
                org.pentaho.di.trans.step.BaseStep.handlePutRow 此处调用 rowWrittenEvent
         */
//        super.rowWrittenEvent(rowMeta, row);
        PrintUtil.printlnData(adapterName+"-->rowWrittenEvent",rowMeta,row);
    }

    @Override
    public void errorRowWrittenEvent(RowMetaInterface rowMeta, Object[] row) throws KettleStepException {
        /*
        定义了错误处理时会进入此处，如下文章定义错误处理
        https://blog.csdn.net/lw18751836671/article/details/121339655?spm=1001.2014.3001.5501

        org.pentaho.di.trans.step.BaseStep.putError
                org.pentaho.di.trans.step.BaseStep#handlePutError  此处调用 errorRowWrittenEvent

         */
        //当发生错误时，将错误数据打印出来
        PrintUtil.printlnData(adapterName+"-->errorRowWrittenEvent",rowMeta,row);

    }
}

将RowAdapter放入监控队列中，如下所示，在构造函数处添加，

public OwnTableInput(StepMeta stepMeta, StepDataInterface stepDataInterface,
                         int copyNr, TransMeta transMeta, Trans trans, RowAdapter rowAdapter) {
        super(stepMeta, stepDataInterface, copyNr, transMeta, trans);

        //添加行监控
        if (rowAdapter != null)
            addRowListener(rowAdapter);
    }

public OwnTableOutput(StepMeta stepMeta, StepDataInterface stepDataInterface,
                          int copyNr, TransMeta transMeta, Trans trans, RowAdapter rowAdapter) {
        super(stepMeta, stepDataInterface, copyNr, transMeta, trans);

        //添加行监控
        if (rowAdapter != null)
            addRowListener(rowAdapter);
    }

当运行后日志打印如下，tableInput只走进了rowWritten，也就是说只进行了putRow方法，tableOutput有读数据和写数据，那么也就是调用了getRow()，putRow()，putError()方法。

其中putError方法是用来处理异常数据的，从日志打印看一条正常，一条异常，和我造的数据结果一样。

tableErrorOutput读取了异常的数据，然后将此异常数据写入到异常表。

2021/11/25 14:23:09 [INFO] tableInput-->rowWrittenEvent： id[VARCHAR]=002e7210219b49819ae5485a4d06e3c3,name[VARCHAR]=1
2021/11/25 14:23:09 [INFO] tableInput-->rowWrittenEvent： id[VARCHAR]=0092d0758f5e4cbb8a54d116e10ae5ed,name[VARCHAR]=bbb

2021/11/25 14:23:09 [INFO] tableOutput-->rowReadEvent： id[VARCHAR]=002e7210219b49819ae5485a4d06e3c3,name[VARCHAR]=1
2021/11/25 14:23:09 [INFO] tableOutput-->rowReadEvent： id[VARCHAR]=0092d0758f5e4cbb8a54d116e10ae5ed,name[VARCHAR]=bbb

2021/11/25 14:23:09 [INFO] tableOutput-->rowWrittenEvent： id[VARCHAR]=002e7210219b49819ae5485a4d06e3c3,name[VARCHAR]=1
2021/11/25 14:23:09 [INFO] tableOutput-->errorRowWrittenEvent： id[VARCHAR]=0092d0758f5e4cbb8a54d116e10ae5ed,name[VARCHAR]=bbb

2021/11/25 14:23:09 [INFO] tableErrorOutput-->rowReadEvent： id[VARCHAR]=0092d0758f5e4cbb8a54d116e10ae5ed,name[VARCHAR]=bbb
2021/11/25 14:23:09 [INFO] tableErrorOutput-->rowWrittenEvent： id[VARCHAR]=0092d0758f5e4cbb8a54d116e10ae5ed,name[VARCHAR]=bbb

其他代码段

public class OwnTableInputMeta extends TableInputMeta {

    private RowAdapter rowAdapter;

    @Override
    public StepInterface getStep(StepMeta stepMeta, StepDataInterface stepDataInterface, int cnr,
                                 TransMeta transMeta, Trans trans ) {
        return new OwnTableInput( stepMeta, stepDataInterface, cnr, transMeta, trans,rowAdapter);
    }

    public void setRowAdapter(RowAdapter rowAdapter) {
        this.rowAdapter = rowAdapter;
    }
}

public class OwnTableOutputMeta extends TableOutputMeta {

    private RowAdapter rowAdapter;

    @Override
    public StepInterface getStep(StepMeta stepMeta, StepDataInterface stepDataInterface, int cnr, TransMeta transMeta, Trans trans) {
        return new OwnTableOutput(stepMeta, stepDataInterface, cnr, transMeta, trans,rowAdapter);
    }

    public void setRowAdapter(RowAdapter rowAdapter) {
        this.rowAdapter = rowAdapter;
    }
}

在交换代码中，使用自己继承的类，替换其中的TableInputMeta和TableOutputMeta,

OwnTableInputMeta tableInputMeta = new OwnTableInputMeta();
tableInputMeta.setRowAdapter(new TableInputRowAdapter("tableInput"));

OwnTableOutputMeta tableOutputMeta = new OwnTableOutputMeta();
tableOutputMeta.setRowAdapter(new TableOutputRowAdapter("tableOutput"));

OwnTableOutputMeta tableOutputMeta = new OwnTableOutputMeta();
tableOutputMeta.setRowAdapter(new TableOutputRowAdapter("tableErrorOutput"));

后记

本来打算再写一写关于kettle 8.2.0.0.342这个源码的，时间上来不及了，如果看源码的话这个入口可以是 org.pentaho.di.trans.Trans#execute,初略的看一下大概，后面断点调试比较好，可以从debug中看到堆栈信息。

关于kettle中博客的案例也上传gitee了，地址是下面的，当然里面代码不一定有用，因为我一开始用的是5.XXX 版本，后续改为了8.XXX 版本，所以改了版本后有些需要引入plugins中的包。

kettle简单案例: 使用kettle的简单案例https://gitee.com/king_software/simple-case-of-kettle

完整交换代码

@Slf4j
public class ExchangeWithExpandCode {

    @Before
    public void before() {
        try {
            // 初始化Kettle环境
            KettleEnvironment.init();
            EnvUtil.environmentInit();
        } catch (KettleException e) {
            e.printStackTrace();
        }
    }

    @Test
    public void exchange()throws KettleException{
        TransMeta transMeta = new TransMeta();
        transMeta.setName("交换");

        PluginRegistry registry = PluginRegistry.getInstance();

        StepMeta inputStep = getTableInputStep(transMeta,registry);
        StepMeta outputStep = getTableOutputStep(transMeta,registry);
        StepMeta errorOutputStep = getTableOutputErrorStep(transMeta,registry);

        //错误数据处理
        VariableSpace space = new  Variables();
        StepErrorMeta errorMeta = new StepErrorMeta(space,outputStep,errorOutputStep);
        errorMeta.setEnabled(true);
        outputStep.setStepErrorMeta(errorMeta);
        errorOutputStep.setStepErrorMeta(errorMeta);

        Trans trans = new Trans(transMeta);

        transMeta.addTransHop(new TransHopMeta(inputStep, outputStep));
        transMeta.addTransHop(new TransHopMeta(outputStep, errorOutputStep));

		//执行转换
		trans.execute(null);

		//等待完成
		trans.waitUntilFinished();
		if (trans.getErrors() > 0) {
			System.out.println("交换出错.");
			return;
		}

//        StepMetaInterface tableOutputMeta = outputStep.getStepMetaInterface();
//        tableOutputMeta.getStep(outputStep,tableOutputMeta.getStepData(),1,transMeta,trans).addRowListener();
    }


    /**
     * 获取表输入
     * @param transMeta
     * @param registry
     * @return
     */
    public StepMeta getTableInputStep(TransMeta transMeta,PluginRegistry registry) throws KettleException{
        /*
		1. 源数据库连接
		 */
        String mysql_src = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" +
                "<connection>" +
                "<name>src</name>" +
                "<server>192.168.10.64</server>" +
                "<type>MySQL</type>" +
                "<access>Native</access>" +
                "<database>test</database>" +
                "<port>3306</port>" +
                "<username>root</username>" +
                "<password>root</password>" +
                "</connection>";

        DatabaseMeta srcDatabaseMeta = new DatabaseMeta(mysql_src);

        transMeta.addDatabase(srcDatabaseMeta);

        OwnTableInputMeta tableInputMeta = new OwnTableInputMeta();

        tableInputMeta.setRowAdapter(new TableInputRowAdapter("tableInput"));

        String tableInputPluginId = registry.getPluginId(StepPluginType.class,
                tableInputMeta);

        tableInputMeta.setDatabaseMeta(srcDatabaseMeta);
        //设置查询条件
        String selectSql = "select id ,name from user_info_src";
        tableInputMeta.setSQL(selectSql);

        StepMeta tableInputStep = new StepMeta(tableInputPluginId,
                "tableInput", (StepMetaInterface) tableInputMeta);

        //给步骤添加在spoon工具中的显示位置
        tableInputStep.setDraw(true);
        tableInputStep.setLocation(100, 100);

        transMeta.addStep(tableInputStep);

        return tableInputStep;
    }

    /**
     * 表输出
     * @param transMeta
     * @param registry
     * @return
     * @throws KettleException
     */
    public StepMeta getTableOutputStep(TransMeta transMeta,PluginRegistry registry) throws KettleException{
        String mysql_dest = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" +
                "<connection>" +
                "<name>dest</name>" +
                "<server>192.168.10.64</server>" +
                "<type>MySQL</type>" +
                "<access>Native</access>" +
                "<database>test</database>" +
                "<port>3306</port>" +
                "<username>root</username>" +
                "<password>root</password>" +
                "</connection>";
        DatabaseMeta destDatabaseMeta = new DatabaseMeta(mysql_dest);

        OwnTableOutputMeta tableOutputMeta = new OwnTableOutputMeta();

        tableOutputMeta.setRowAdapter(new TableOutputRowAdapter("tableOutput"));

        tableOutputMeta.setDatabaseMeta(destDatabaseMeta);

        tableOutputMeta.setSchemaName(null);
        tableOutputMeta.setTablename("user_info_dest");

        String tableOutputPluginId = registry.getPluginId(StepPluginType.class, tableOutputMeta);
        StepMeta tableOutputStep = new StepMeta(tableOutputPluginId, "tableOutput", (StepMetaInterface) tableOutputMeta);

        //将步骤添加进去
        transMeta.addStep(tableOutputStep);

        //给步骤添加在spoon工具中的显示位置
        tableOutputStep.setDraw(true);
        tableOutputStep.setLocation(200, 200);

        return tableOutputStep;
    }

    /**
     * 表输出
     * @param transMeta
     * @param registry
     * @return
     * @throws KettleException
     */
    public StepMeta getTableOutputErrorStep(TransMeta transMeta,PluginRegistry registry) throws KettleException{
        String mysql_dest = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" +
                "<connection>" +
                "<name>dest</name>" +
                "<server>192.168.10.64</server>" +
                "<type>MySQL</type>" +
                "<access>Native</access>" +
                "<database>test</database>" +
                "<port>3306</port>" +
                "<username>root</username>" +
                "<password>root</password>" +
                "</connection>";
        DatabaseMeta destDatabaseMeta = new DatabaseMeta(mysql_dest);

        OwnTableOutputMeta tableOutputMeta = new OwnTableOutputMeta();

        tableOutputMeta.setRowAdapter(new TableOutputRowAdapter("tableErrorOutput"));

        tableOutputMeta.setDatabaseMeta(destDatabaseMeta);

        tableOutputMeta.setSchemaName(null);
        tableOutputMeta.setTablename("user_info_dest_error");

        String tableOutputPluginId = registry.getPluginId(StepPluginType.class, tableOutputMeta);
        StepMeta tableOutputStep = new StepMeta(tableOutputPluginId, "tableErrorOutput", (StepMetaInterface) tableOutputMeta);

        //将步骤添加进去
        transMeta.addStep(tableOutputStep);

        //给步骤添加在spoon工具中的显示位置
        tableOutputStep.setDraw(true);
        tableOutputStep.setLocation(300, 300);

        return tableOutputStep;
    }

}

posted @ 2021-11-25 15:10 伟衙内阅读(42) 评论(0) 编辑收藏举报

刷新页面返回顶部

weiyanei

RowListener

前言

代码说明

TableInputRowAdapter

TableOutputRowAdapter

其他代码段

后记

完整交换代码