graylog 索引数据查询处理简单说明
参考请求地址
从ui 看到的,类似一个job 的执行,支持异步以及同步数据查询处理，还包含了相对时间、绝对时间查询的
/api/views/search/<id>/execute
/api/views/search
/api/views/search/metadata
查询适配
- 参考实现
主要是对于es6 以及es7 不同版本的兼容处理，对于查询支持了scroll 模式以及普通的search 模式
- 查询处理
如下图，对于查询可能会包含多个索引
public SearchResult search(Set<String> indices, Set<IndexRange> indexRanges, SearchesConfig config) {
final SearchSourceBuilder searchSourceBuilder = searchRequestFactory.create(config);
if (indexRanges.isEmpty()) {
return SearchResult.empty(config.query(), searchSourceBuilder.toString());
}
// 构造多索引的查询
final SearchRequest searchRequest = new SearchRequest(indices.toArray(new String[0]))
.source(searchSourceBuilder);
final SearchResponse searchResult = client.search(searchRequest, "Unable to perform search query");
final List<ResultMessage> resultMessages = extractResultMessages(searchResult);
final long totalResults = searchResult.getHits().getTotalHits().value;
final long tookMs = searchResult.getTook().getMillis();
final String builtQuery = searchSourceBuilder.toString();
return new SearchResult(resultMessages, totalResults, indexRanges, config.query(), builtQuery, tookMs);
}
查询任务执行
实际上是基于了多线程支持模式，graylog 同时提供了同步以及异步模式，具体执行都是SearchExecutor 提供的
具体会使用QueryEngine提供的查询能力，同时一次查询可能会处理多个stream 所以需要遍历查询
public SearchJob execute(SearchJob searchJob) {
searchJob.getSearch().queries().forEach(query -> searchJob.addQueryResultFuture(query.id(),
// generate and run each query, making sure we never let an exception escape
// if need be we default to an empty result with a failed state and the wrapped exception
// 使用了CompletableFuture
CompletableFuture.supplyAsync(() -> prepareAndRun(searchJob, query), queryPool)
.handle((queryResult, throwable) -> {
if (throwable != null) {
final Throwable cause = throwable.getCause();
final SearchError error;
if (cause instanceof SearchException) {
error = ((SearchException) cause).error();
} else {
error = new QueryError(query, cause);
}
LOG.debug("Running query {} failed: {}", query.id(), cause);
searchJob.addError(error);
return QueryResult.failedQueryWithError(query, error);
}
return queryResult;
})
));
searchJob.getSearch().queries().forEach(query -> {
final CompletableFuture<QueryResult> queryResultFuture = searchJob.getQueryResultFuture(query.id());
if (!queryResultFuture.isDone()) {
// this is not going to throw an exception, because we will always replace it with a placeholder "FAILED" result above
final QueryResult result = queryResultFuture.join();
} else {
LOG.debug("[{}] Not generating query for query {}", defaultIfEmpty(query.id(), "root"), query);
}
});
LOG.debug("Search job {} executing", searchJob.getId());
return searchJob.seal();
}
具体的查询结果是通过SearchJob 的results 获取的（获取数据异步线程的数据结果，利用了streamex 方便的stream 工具）
@JsonProperty("results")
public Map<String, QueryResult> results() {
return EntryStream.of(queryResults)
.mapValues(future -> future.getNow(QueryResult.incomplete()))
.filterKeys(queryId -> !queryId.isEmpty()) // the root query result is meaningless, so we don't include it here
.filterValues(r -> (r.state() == QueryResult.State.COMPLETED) || (r.state() == QueryResult.State.FAILED))
.toMap();
}
说明
尽管scroll 是一个请求大量数据的模式，但是es 并不是很推荐了，更推荐的是使用search-after，同时需要基于pit 进行更高效的进行数据查询处理
参考资料
https://docs.graylog.org/docs/elasticsearch
https://docs.graylog.org/docs/index-model
https://docs.graylog.org/docs/rest-api
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/searches/SearchesAdapter.java
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog2/indexer/searches/Searches.java
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/paginate-search-results.html#search-after
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/point-in-time-api.html
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog/plugins/views/search/engine/QueryEngine.java
https://github.com/Graylog2/graylog2-server/blob/626be1f0d80506705b5ba41fbea33c2ec0164bc0/graylog2-server/src/main/java/org/graylog/plugins/views/search/rest/SearchResource.java
