elasticsearch中runtime_mapping实战

背景：需要根据一个实时计算处理的结果值进行排序，数据从es中查询。（基于业务背景：佣金排序）

es版本：7.17.1；spring-data-elasticsearch版本：4.3.9

方式一：mysql新增字段：mysql根据业务操作，直接在在代码中刷取数据存储到mysql中（未采用）
优点：代码简单，后期查询时候排序简单

缺点：如果相关因素变化，需要扫描刷新大量相关数据

方式二：pipeline，数据往es同步的时候计算出来需排序的字段（暂未采用）

优点：实现简单，后期排序简单

缺点：当前业务背景下，也需要刷新大量数据，只是不用自己计算而已。相当于计算过程中部分字段在原始schema中也没有，复杂度和方式一差不多。

方式二.一：（暂未采用）
创建索引时指定mapping中设置runtime字段，支持数据插入的时候即生成runtime字段
优点：性能比查询时直接使用runtime mapping性能高
缺点：需要重新创建索引，reindex所有历史数据。

方式三：使用runtime mapping在查询过程中直接实现（采用）

优点：实现简单，直接用script脚本即可实现

缺点：数据量大的时候，会有性能问题，生产环境使用前，需要压测

下面重点讲解一下方式三实现：

版本一：

script是一个对象{}，可以动态传递参数，更加灵活。

POST /t_spu/_search
{
  "size": 1000, 
  "runtime_mappings": {
    "commission": {
      "type": "double",
      "script": {
        "source": """
        String commissionStr = doc['commission_price.keyword'].value;
        long price = doc['min_sku_sale_price'].value;
        int indexNo = commissionStr.indexOf('~');
        if (indexNo > 0) {
            double allCommission = Double.parseDouble(commissionStr.substring(0, indexNo))*100;
            emit(allCommission - price * params.platformCommissionRate);
        } else {
            emit(Double.parseDouble(commissionStr)*100 - price * params.platformCommissionRate);
        }
        """,
        "lang": "painless",
        "params": {
          "platformCommissionRate": 0.03
        }
      }
    }
  },
  "fields": [
    "commission"
  ],
  "query": {
    "bool": {
      "filter": [
      {"term": {
      "goods_source_type": {
        "value": "1"
      }
    }}
      ]
    }
  },
  "sort": [
    {
      "commission": {
        "order": "desc"
      }
    }
  ]
}

 java:（部分查询条件未具体实现）

1、这里用到的script是一个对象{}，ElasticsearchRestTemplate不支持。es版本：7.17.1；spring-data-elasticsearch版本：4.3.9

只能用RestHighLevelClient客户端实现java代码。

@Autowired
private RestHighLevelClient restHighLevelClient;

String commissionScript = "String commissionStr = doc['commission_price.keyword'].value;" +
                "long price = doc['min_sku_sale_price'].value;" +
                "int indexNo = commissionStr.indexOf('~');" +
                "if (indexNo > 0) {double allCommission = Double.parseDouble(commissionStr.substring(0, indexNo))*100;" +
                "emit(allCommission - price * params.platform_commission_rate);}" +
                "else {emit(Double.parseDouble(commissionStr)*100 - price * params.platform_commission_rate);}";
        //runtime_mappings
        final String COMMISSION = "commission";
        Map<String, Object> params = new HashMap<>();
        params.put(GoodsSearchEsConstant.PLATFORM_COMMISSION_RATE, realPlatformCommissionRate);
        Script script = new Script(ScriptType.INLINE, "painless", commissionScript, params);


        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
//        sourceBuilder.from(0);
//        sourceBuilder.size(10);
//        sourceBuilder.fetchSource(new String[]{"title"}, new String[]{});
//        MatchQueryBuilder matchQueryBuilder = QueryBuilders.matchQuery("title", "商品名称");
        TermQueryBuilder termQueryBuilder = QueryBuilders.termQuery("goods_source_type", 1);
//        RangeQueryBuilder rangeQueryBuilder = QueryBuilders.rangeQuery("payTime");
//        rangeQueryBuilder.gte("2023-01-26T08:00:00Z");
//        rangeQueryBuilder.lte("2023-01-26T20:00:00Z");
        BoolQueryBuilder boolBuilder = QueryBuilders.boolQuery();
//        boolBuilder.must(matchQueryBuilder);
        boolBuilder.must(termQueryBuilder);
//        boolBuilder.must(rangeQueryBuilder);
        sourceBuilder.query(boolBuilder);

        //runtime_mappings 部分
        HashMap<String, Object> runtimeMappings = new HashMap<>();
        HashMap<String, Object> commObj = new HashMap<>();
        commObj.put("script", script);
        commObj.put("type", FieldType.Double.getMappedName());
        runtimeMappings.put(COMMISSION, commObj);
//        sourceBuilder.fetchField("*");
        sourceBuilder.fetchField("commission");
        sourceBuilder.runtimeMappings(runtimeMappings);

        sourceBuilder.sort(COMMISSION, SortOrder.DESC);
        SearchRequest searchRequest = new SearchRequest("index_t_spu");
        searchRequest.source(sourceBuilder);
        try {
            SearchResponse response = restHighLevelClient.search(searchRequest, RequestOptions.DEFAULT);
            System.out.println(response);
        } catch (IOException e) {
            e.printStackTrace();
        }

版本二：

script直接传source,string类型脚本，这样的话动态参数params就无法添加了，只能自己提前拼接好完整的script。类型是String。

POST /tb-g-mysql.tb_retail_goods.t_spu/_search
{
  "size": 1120, 
  "runtime_mappings": {
    "commission": {
      "type": "double",
      "script": "String commissionStr = doc['commission_price.keyword'].value;long price = doc['min_sku_sale_price'].value / 100;int indexNo = commissionStr.indexOf('~');if (indexNo > 0) {double allCommission = Double.parseDouble(commissionStr.substring(0, indexNo))*100;emit(allCommission - price * 0.03);} else {emit(Double.parseDouble(commissionStr)*100 - price * 0.03);}"
    }
  },
  "fields": [
    "commission"
  ],
  "query": {
    "term": {
     "goods_source_type": {
       "value": "1"
     }
   }
  },
  "sort": [
    {
      "commission": {
        "order": "desc"
      }
    }
  ]
}

java代码实现：（部分查询条件未具体实现）

spring-data-elasticsearch的客户端ElasticsearchRestTemplate 不支持：因为：RuntimeField不支持Script对象参数，因此直接string替换动态参数拼接script

es版本：7.17.1；spring-data-elasticsearch版本：4.3.9

@Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;

NativeSearchQueryBuilder searchQueryBuilder = new NativeSearchQueryBuilder();
BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder().filter(QueryBuilders.termQuery("xxx", 1));

NativeSearchQuery nativeSearchQuery = searchQueryBuilder.withTrackScores(true)
        .withQuery(boolQueryBuilder)
        .withPageable(PageRequest.of(dto.getPage() - 1, dto.getLimit()))
        .withIndicesOptions(IndicesOptions.LENIENT_EXPAND_OPEN).build();

String realPlatformCommissionRate = BigDecimal.valueOf(platformCommissionRate).divide(BigDecimal.valueOf(1000), 2, RoundingMode.HALF_UP).toString();
        String commissionScript = ("String commissionRateStr = doc['commission_rate.keyword'].value;" +
                "long price = doc['min_sku_sale_price'].value;" +
                "int indexNo = commissionRateStr.indexOf('~');" +
                "if (indexNo > 0) {double allCommission = Double.parseDouble(commissionRateStr.substring(0, indexNo))*100;" +
                "emit(allCommission - price * realPlatformCommissionRate);}" +
                "else {emit(Double.parseDouble(commissionRateStr)*100 - price * realPlatformCommissionRate);}")
                .replace("realPlatformCommissionRate", realPlatformCommissionRate);
        //runtime_mappings
        final String COMMISSION = "commission";
        //注意：RuntimeField不支持Script对象参数，因此直接string替换动态参数拼接script
        RuntimeField runtimeField = new RuntimeField(COMMISSION, FieldType.Double.getMappedName(), commissionScript);
        searchQuery.addRuntimeField(runtimeField);
        searchQuery.addFields(COMMISSION);
        if (StringUtils.isNotBlank(dto.getSort()) && SqlKeyword.DESC.name().equals(dto.getSort().toUpperCase())) {
            searchQuery.addSort(Sort.by(Sort.Direction.DESC, COMMISSION));
        } else {
            searchQuery.addSort(Sort.by(Sort.Direction.ASC, COMMISSION));
        }