canal-adapter-进行全量和增量到ES7.*源码修改(六)

文章主目录

• 说明
• 支持的同步
• 同步到ES7.*源码修改
• 消费不到binlog消息修改
• 订阅指定表配置
• 基本类型空指针

回到顶部

说明

不知道是否对7.*所有版本有效 我目前是7.4 如果按照下面方法不行 最好是下载源码调试

源码获取参考 https://www.cnblogs.com/LQBlog/p/12177295.html#autoid-3-0-0

回到顶部

支持的同步

输入端

canal adapter个人理解是canal实现的 数据同步增量全量到ES和和各个数据库 暂时只支持以下几种可以看出接口打了SPI注解 如果我们输出到其他端可以扩展

 

 

输出端

如以下配置 我大概看了源码 tcp增量数据来源是binlog 也可以是kafka和rocketMQ

canal.conf:
  mode: tcp # kafka rocketMQ 数据来源TCP则是binlog 其他则增量数据来源是kafka和rocketMQ

 

 

回到顶部

同步到ES7.*源码修改

因为各个ES版本客户端api都不一致，默认只支持到了6.4以下的版本，如果以上版本 则需要自行替换es客户端 然后某些不兼容的地方修改,这里我拿ES7.4做比较

/canal-canal-1.1.4/client-adapter/elasticsearch/pom.xml

1.修改POM文件 提高客户端版本

<!--        <dependency>-->
<!--            <groupId>org.elasticsearch</groupId>-->
<!--            <artifactId>elasticsearch</artifactId>-->
<!--            <version>6.4.3</version>-->
<!--        </dependency>-->
<!--        <dependency>-->
<!--            <groupId>org.elasticsearch.client</groupId>-->
<!--            <artifactId>transport</artifactId>-->
<!--            <version>6.4.3</version>-->
<!--        </dependency>-->
<!--        <dependency>-->
<!--            <groupId>org.elasticsearch.client</groupId>-->
<!--            <artifactId>elasticsearch-rest-client</artifactId>-->
<!--            <version>6.4.3</version>-->
<!--        </dependency>-->
<!--        <dependency>-->
<!--            <groupId>org.elasticsearch.client</groupId>-->
<!--            <artifactId>elasticsearch-rest-high-level-client</artifactId>-->
<!--            <version>6.4.3</version>-->
<!--        </dependency>-->
        <!--liqiangtodo 版本提高改动-->
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.4.0</version>
        </dependency>

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.4.0</version>
        </dependency>
        <!--7.4.0没有对应版本的transport 所以我还是使用6.4.3-->
                <dependency>
                    <groupId>org.elasticsearch.client</groupId>
                    <artifactId>transport</artifactId>
                    <version>6.4.3</version>
                </dependency>
        <dependency>

2.放开mapping判断

com.alibaba.otter.canal.client.adapter.es.support.ESConnection#getMapping

  public MappingMetaData getMapping(String index, String type) {
        MappingMetaData mappingMetaData = null;
        if (mode == ESClientMode.TRANSPORT) {
            ImmutableOpenMap<String, MappingMetaData> mappings;
            try {
                mappings = transportClient.admin()
                    .cluster()
                    .prepareState()
                    .execute()
                    .actionGet()
                    .getState()
                    .getMetaData()
                    .getIndices()
                    .get(index)
                    .getMappings();
            } catch (NullPointerException e) {
                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
            }
            mappingMetaData = mappings.get(type);

        } else {
            ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>> mappings;
            try {
                GetMappingsRequest request = new GetMappingsRequest();
                request.indices(index);
                GetMappingsResponse response;
                // try {
                // response = restHighLevelClient
                // .indices()
                // .getMapping(request, RequestOptions.DEFAULT);
                // // 6.4以下版本直接使用该接口会报错
                // } catch (Exception e) {
                // logger.warn("Low ElasticSearch version for getMapping");
                response = RestHighLevelClientExt.getMapping(restHighLevelClient, request, RequestOptions.DEFAULT);
                // }

                mappings = response.mappings();
            } catch (NullPointerException e) {
                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
            } catch (IOException e) {
                logger.error(e.getMessage(), e);
                return null;
            }
            mappingMetaData = mappings.get(index).get(type);
        
        }
        return mappingMetaData;
    }

将红色部分改为

    //liqiangtodo 版本提高改动
    public MappingMetaData getMapping(String index, String type) {
        MappingMetaData mappingMetaData = null;
        if (mode == ESClientMode.TRANSPORT) {
            ImmutableOpenMap<String, MappingMetaData> mappings;
            try {
                mappings = transportClient.admin()
                        .cluster()
                        .prepareState()
                        .execute()
                        .actionGet()
                        .getState()
                        .getMetaData()
                        .getIndices()
                        .get(index)
                        .getMappings();
            } catch (NullPointerException e) {
                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
            }
            mappingMetaData = mappings.get(type);

        } else {
            ImmutableOpenMap<String, ImmutableOpenMap<String, MappingMetaData>> mappings;
            try {
                GetMappingsRequest request = new GetMappingsRequest();
                request.indices(index);
                GetMappingsResponse response;
                response = RestHighLevelClientExt.getMapping(restHighLevelClient, request, RequestOptions.DEFAULT);
                mappings = response.mappings();
            } catch (NullPointerException e) {
                throw new IllegalArgumentException("Not found the mapping info of index: " + index);
            } catch (IOException e) {
                logger.error(e.getMessage(), e);
                return null;
            }
          //  mappingMetaData = mappings.get(index).get(type);
            //liqiangtodo 版本提高改动
            mappingMetaData = mappings.get(index).get("properties");
        }
        return mappingMetaData;
    }

3.bulk版本兼容改动

com.alibaba.otter.canal.client.adapter.es.support.ESConnection.ESBulkRequest#bulk

红色部分是我的改动，刷新策略是 我需要理解可见 因为用户下单后 同步到es 需要马上在订单列表可见

ublic BulkResponse bulk() {
            BulkResponse bulkResponse=null;
            if (mode == ESClientMode.TRANSPORT) {
                bulkResponse=bulkRequestBuilder.execute().actionGet();
            } else {
                try {
                    //详见文档https://blog.csdn.net/hanchao5272/article/details/89151166 刷新策略 立即可见
                    bulkRequest.setRefreshPolicy(WriteRequest.RefreshPolicy.IMMEDIATE);
                    //liqiangtodo 版本提高修改
                    bulkResponse =restHighLevelClient.bulk(bulkRequest,RequestOptions.DEFAULT);
                   // return restHighLevelClient.bulk(bulkRequest);
                } catch (IOException e) {
                    throw new RuntimeException(e);
                }
            }
            return bulkResponse;

        }

4.getEsType

com.alibaba.otter.canal.client.adapter.es.support.ESTemplate#getEsType

**
     * 获取es mapping中的属性类型
     *
     * @param mapping   mapping配置
     * @param fieldName 属性名
     * @return 类型
     */
    @SuppressWarnings("unchecked")
    private String getEsType(ESMapping mapping, String fieldName) {
        String key = mapping.get_index() + "-" + mapping.get_type();
        Map<String, String> fieldType = esFieldTypes.get(key);
        if (fieldType != null) {
            return fieldType.get(fieldName);
        } else {
            MappingMetaData mappingMetaData = esConnection.getMapping(mapping.get_index(), mapping.get_type());

            if (mappingMetaData == null) {
                throw new IllegalArgumentException("Not found the mapping info of index: " + mapping.get_index());
            }

            fieldType = new LinkedHashMap<>();

            //liqiangtodo 版本升级改动
            Map<String, Object> esMapping =mappingMetaData.getSourceAsMap();
//            Map<String, Object> sourceMap ==mappingMetaData.getSourceAsMap();
//            Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");
            for (Map.Entry<String, Object> entry : esMapping.entrySet()) {
                Map<String, Object> value = (Map<String, Object>) entry.getValue();
                if (value.containsKey("properties")) {
                    fieldType.put(entry.getKey(), "object");
                } else {
                    fieldType.put(entry.getKey(), (String) value.get("type"));
                }
            }
            esFieldTypes.put(key, fieldType);

            return fieldType.get(fieldName);
        }
    }

5.count

com.alibaba.otter.canal.client.adapter.es.ESAdapter#count

 @Override
    public Map<String, Object> count(String task) {
        ESSyncConfig config = esSyncConfig.get(task);
        ESMapping mapping = config.getEsMapping();
        SearchResponse response = this.esConnection.new ESSearchRequest(mapping.get_index(), mapping.get_type()).size(0)
            .getResponse();

        //liqiangtodo 提高版本改动
        //long rowCount = response.getHits().getTotalHits();
        TotalHits totalHits= response.getHits().getTotalHits();
        Map<String, Object> res = new LinkedHashMap<>();
        res.put("esIndex", mapping.get_index());
        res.put("count", totalHits.value);
        return res;
    }

6.修改es adapter的SPI的key

client-adapter/elasticsearch/src/main/resources/META-INF/canal/com.alibaba.otter.canal.client.adapter.OuterAdapter

#es=com.alibaba.otter.canal.client.adapter.es.ESAdapter
es7.4.0=com.alibaba.otter.canal.client.adapter.es.ESAdapter

 

回到顶部

消费不到binlog消息修改

com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker#CanalAdapterWorker

   public CanalAdapterWorker(CanalClientConfig canalClientConfig, String canalDestination, SocketAddress address,
                              List<List<OuterAdapter>> canalOuterAdapters){
        super(canalOuterAdapters);
        this.canalClientConfig = canalClientConfig;
        this.canalDestination = canalDestination;
        //liqiangtodo 消费不到改动
        connector = CanalConnectors.newClusterConnector(Arrays.asList(address), canalDestination, "", "");
        //connector = CanalConnectors.newSingleConnector(address, canalDestination, "", "");
    }

回到顶部

订阅指定表配置

防止订阅到实例里面其他表变动

com.alibaba.otter.canal.client.adapter.support.CanalClientConfig 增加配置

   //liqiangtodo canal订阅的消息
    private String subscribe;

com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterWorker#process

...... 
 //获取锁 com.alibaba.otter.canal.adapter.launcher.common.SyncSwitch.init 初始化位置 会读取zk
                /**
                 * /canal-adapter/sync-switch/{canalDestination} on为可获取锁 off为阻塞
                 */
                syncSwitch.get(canalDestination);

                logger.info("=============> Start to connect destination: {} <=============", this.canalDestination);
                connector.connect();
                logger.info("=============> Start to subscribe destination: {} <=============", this.canalDestination);
                //liqiangtodo 消费不到改动 订阅自己关心的
                if(StringUtils.isEmpty(canalClientConfig.getSubscribe())){
                    connector.subscribe("*.*");
                }else {
                    connector.subscribe(canalClientConfig.getSubscribe());
                }
                //connector.subscribe();
                logger.info("=============> Subscribe destination: {},subscribe:{} succeed <=============",this.canalDestination,StringUtils.isEmpty(canalClientConfig.getSubscribe())?"*.*":canalClientConfig.getSubscribe());
......

使用方式

canal.conf:
  subscribe: merge_test.pro_brand,merge_test.soa_ord_order_summary,merge_test.soa_ord_order,merge_test.soa_ord_order_item #我自己改源码加的防止订阅不关心的消息

回到顶部

基本类型空指针

如Interger 数据库存的是null 

com.alibaba.otter.canal.client.adapter.es.support.ESSyncUtil#typeConvert

 /**
     * 类型转换为Mapping中对应的类型
     */
    public static Object typeConvert(Object val, String esType) {
        if (val == null) {
            return null;
        }
        if (esType == null) {
            return val;
        }
        //liqiangtodo 增加代码
        //For input string: ""防止空转换异常指针异常
        if(StringUtil.isNullOrEmpty(val.toString())){
            return null;
        }
 ......
}