Logstash 7.7 x+导出官网 java 插件为离线包，重安装失败bug：ERROR: Something went wrong when installing file:///xxx, message: Illformed requirement ["=java_xxx"]

标题中bug：在最后按照离线包处说明，给出解决方式

公司需要使用最新版logstash来开发，我们自己开放性框架

  1. 解压windows 直接解压/linux  unzip xxxx.zip
  2. 生成java插件所需api 核心 jar 包文件：进入项目根目录执行-windows:gradlew.bat assemble  linux：./gradlew assemble
  1. plugin demo 根目录，增加java.properties 配置文件 gradle.properties，添加值：LOGSTASH_CORE_PATH=/data/logstashcpl/logstash-7.10.1/logstash-core  该核心路径指向上一步中核心jar包路径
  2. 打包插件：./gradlew gem  
  1. 解压即可使用，进入logstash根目录
  2. 安装上面打包的插件：bin/logstash-plugin install --no-verify --local 你打包出来的插件
    ?
    1
    bin/logstash-plugin install --no-verify --local /data/plugins/logstash-input-java_input_example-master/logstash-input-java_input_example-1.0.1.gem
  3.  运行该插件：
    ?
    1
    bin/logstash -e "input {java_input_example {}} output {stdout { codec => rubydebug }}"
  4. 查看该插件在logstash中的名称
    ?
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    [root@VM_centos logstash-7.10.1]# bin/logstash-plugin list
    Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
    WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
    ....
    logstash-input-imap
    logstash-input-java_input_example
    logstash-input-jm
    .....
     
    #logstash-input-java_input_example 即为插件名

    　　

  5. 导出该插件用于离线环境安装（我司内网，没办法在内网编译安装）bin/logstash-plugin prepare-offline-pack --overwrite --output [包名] 插件名，上一步获取的插件名即可
    ?
    1
    2
    3
    4
    5
    6
    7
    #打离线包
    [root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/java_input.zip logstash-input-java_input_example
    Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
    WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
    Offline package created at: /data/java_input.zip
     
    You can install it with this command `bin/logstash-plugin install file:///data/java_input.zip`

    　　

  6. 安装该离线包：bin/logstash-plugin install file:///data/java_input.zip

我在执行上述步骤是遇到问题如下：

  1. demo 插件问题：该错误来源：
    ?
    1
    No signature of method: org.gradle.api.internal.tasks.DefaultTaskDependency $TaskDependencySet.getAt() is applicable for argument types: (ArrayList) values:　　

    　　

    dependsOn [downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles] 修改如下
    dependsOn([downloadAndInstallJRuby, removeObsoleteJars, vendor, generateRubySupportFiles])
  2. demo插件问题：
    ?
    1
    2
    /data/plugins/logstash-input-java_input_example-master/src/main/java/org/logstashplugins/JavaInputExample.java:3: error: package co.elastic.logstash.api does not exist
    import co.elastic.logstash.api.Configuration;

    　　该错误原因为build.gradle 去匹配核心jar包时未匹配到，因为demo中

    ?
    1
    2
    3
    4
    implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.?.?.jar")
    #logstash-core-?.?.?.jar 中 ? 匹配一个字符，一旦 x.y.z.jar包中版本超过9，达到两个字符就会匹配不到
    #修改为：
    implementation fileTree(dir: LOGSTASH_CORE_PATH, include: "**/logstash-core-?.??.?.jar") #或指定你生成的jar包版本

    　　

  3. 安装离线包：
    ?
    1
    ERROR: Something went wrong when installing file:///data//plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]

    　　该错误是由于默认命名插件时以 java开头导致：build.gradle 文件中

    ?
    1
    2
    3
    pluginInfo.pluginType      = "input"
    pluginInfo.pluginClass     = "JavaInputExample"
    pluginInfo.pluginName      = "java_input_example" //此处命名java开头导致

    　　

分析安装离线包导致的该问题：

 根目录搜索该报错信息：

?
1
2
3
[root@VM_centos logstash-7.10.1]# grep Illformed -nr ./*
./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:110:      raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
[root@VM_63_147_centos logstash-7.10.1]# vim ./vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb +110

输出堆栈信息：puts caller

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
def self.parse(obj)
  return ["=", obj] if Gem::Version === obj
 
  unless PATTERN =~ obj.to_s
    puts caller
    raise BadRequirementError, "Illformed requirement [#{obj.inspect}]"
  end
 
  if $1 == ">=" && $2 == "0"
    DefaultRequirement
  else
    [$1 || "=", Gem::Version.new($2)]
  end
end

　　堆栈信息如下

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Installing file: /data/plugins/java-input.zip
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `block in initialize'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `map!'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:140:in `initialize'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/requirement.rb:65:in `create'
/data/logstash/logstash-7.10.1/vendor/jruby/lib/ruby/stdlib/rubygems/dependency.rb:60:in `initialize'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/dependency.rb:82:in `initialize'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:65:in `dependency'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `collect'
/data/logstash/logstash-7.10.1/lib/pluginmanager/bundler/logstash_injector.rb:52:in `inject!'
/data/logstash/logstash-7.10.1/lib/pluginmanager/pack_installer/local.rb:58:in `execute'
/data/logstash/logstash-7.10.1/lib/pluginmanager/install.rb:47:in `execute'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11:in `execute'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:68:in `run'
/data/logstash/logstash-7.10.1/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:133:in `run'
/data/logstash/logstash-7.10.1/lib/pluginmanager/main.rb:64:in `<main>'
ERROR: Something went wrong when installing file:///data/plugins/java-input.zip, message: Illformed requirement ["=java_input_example"]

　通过上述信息定位到：

?
1
vim ./lib/pluginmanager/pack_installer/pack.rb +65

　　

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
class Pack
  class GemInformation
    EXTENSION = ".gem"
    SPLIT_CHAR = "-"
    JAVA_PLATFORM_RE = /-java/
    DEPENDENCIES_DIR_RE = /dependencies/
 
    attr_reader :file, :name, :version, :platform
 
    def initialize(gem)
      @file = gem
      extracts_information
    end
 
    def dependency?
      @dependency
    end
 
    def plugin?
      !dependency?
    end
 
    private
    def extracts_information
      basename = ::File.basename(file, EXTENSION)
      parts = basename.split(SPLIT_CHAR)
      @dependency = ::File.dirname(file) =~ DEPENDENCIES_DIR_RE
      puts "basename: #{basename}"
      puts "parts: #{parts}"
      if basename.match(JAVA_PLATFORM_RE)
        @platform = parts.pop
        @version = parts.pop
        @name = parts.join(SPLIT_CHAR)
      else
        @platform = nil
        @version = parts.pop
        @name = parts.join(SPLIT_CHAR)
      end
    end
  end

　输出：basename、parts 可以分析出，因为打包名为 logstash-input-java_input_example-1.0.1 而 logstash 安装源码中，使用了 

?
1
JAVA_PLATFORM_RE = /-java/
?
1
if basename.match(JAVA_PLATFORM_RE) 此处匹配到了，/-java/ 认为打包平台为 java 不是 ruby 导致
?
1
2
3
4
5
6
7
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin install file:///data/plugins/java-input.zip
...
arguments: ["install", "file:///data/plugins/java-input.zip"]
subcmd_class: LogStash::PluginManager::Install
arguments: ["file:///data/plugins/java-input.zip"]
Installing file: /data/plugins/java-input.zip
basename: logstash-input-java_input_example-1.0.1

解决方案就比较多了，当然不建议修改源码，

直接修改插件名，别用 java 开头就好了

 

多说一点。

楼主排查的时候怀疑是不是打包就有问题，少打了 -java 参数到包名中去，捋了好久，具体过程就不说了，看结果吧

使用 bin/logstash-plugin prepare-offline-pack 去打包的话

?
1
2
3
[root@VM_centos logstash-7.10.1]# grep -rn "my collect:" ./
./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb:32:      puts "my collect: #{collect_required_gems}"
[root@VM_centos logstash-7.10.1]# vim ./vendor/bundle/jruby/2.5.0/gems/paquet-0.2.1/lib/paquet/gem.rb +32
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
module Paquet
  class Gem
    RUBYGEMS_URI = "https://rubygems.org/downloads"
 
    attr_reader :gems, :ignores
 
    def initialize(target_path, cache = nil)
      @target_path = target_path
      @gems = []
      @ignores = []
      @cache = cache
    end
 
    def add(name)
      @gems << name
    end
 
    def ignore(name)
      @ignores << name
    end
 
    def pack
      Paquet::ui.info("Cleaning existing target path: #{@target_path}")
 
      FileUtils.rm_rf(@target_path)
      FileUtils.mkdir_p(@target_path)
      puts "sq collect: #{collect_required_gems}"
      package_gems(collect_required_gems)
    end

输出打包命令去收集所需gems时参数都为：

?
1
2
3
4
5
6
7
8
[root@VM_centos logstash-7.10.1]# bin/logstash-plugin prepare-offline-pack --overwrite --output /data/plugins/java-input.zip logstash-input-java_input_example
Using JAVA_HOME defined java: /data/java/jdk1.8.0_271/
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
arguments: ["prepare-offline-pack", "--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"]
subcmd_class: LogStash::PluginManager::PrepareOfflinePack
arguments: ["--overwrite", "--output", "/data/plugins/java-input.zip", "logstash-input-java_input_example"]
my test: ["logstash-input-java_input_example"]
my collect: [#<Paquet::Dependency:0x51d76ad3 @version=#<Gem::Version "1.0.1">, @name="logstash-input-java_input_example", @platform="ruby">]

可以发现上面最后一行 collect 收集参数中 platform 就是ruby 不是 java
至此楼主也懒得研究了，研究这几天炸了。
反正 logstash 官网这插件 demo 质量太差，然后处理打包上，源码感觉也不够严谨，导致了这样的bug，分割字符串，正则匹配太SX了

 

 最后楼主将修改后打出的包放到内网成功如下：

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash-plugin install file:///data/java-input.zip
Using JAVA_HOME defined java: /data/share/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
 
Installing file: /data/java-input.zip
Install successful
SIS3.0.45.0 /home/fantom/party/logstash-7.10.1 #
SIS3.0.45.0 /data/logstash-7.10.1 # bin/logstash -e "input {input_example_java {}} output {stdout { codec => rubydebug }}"
Using JAVA_HOME defined java: /data/java/jdk
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
 
Sending Logstash logs to /data/logstash-7.10.1/logs which is now configured via log4j2.properties
[2020-12-21T21:14:01,451][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +indy +jit [linux-x86_64]"}
[2020-12-21T21:14:01,858][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/data/logstash-7.10.1/data/queue"}
[2020-12-21T21:14:01,878][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/fantom/party/logstash-7.10.1/data/dead_letter_queue"}
[2020-12-21T21:14:02,485][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-12-21T21:14:02,529][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"a33c9423-70fa-4794-9995-f02bc72e2192", :path=>"/home/fantom/party/logstash-7.10.1/data/uuid"}
[2020-12-21T21:14:05,000][INFO ][org.reflections.Reflections] Reflections took 88 ms to scan 1 urls, producing 23 keys and 47 values
[2020-12-21T21:14:08,510][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x21cc9c5e run>"}
[2020-12-21T21:14:11,199][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>2.66}
[2020-12-21T21:14:11,252][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-12-21T21:14:11,432][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
    "@timestamp" => 2020-12-21T13:14:11.308Z,
       "message" => "message        1 of 3       ",
      "@version" => "1"
}
{
    "@timestamp" => 2020-12-21T13:14:11.319Z,
       "message" => "message        2 of 3       ",
      "@version" => "1"
}
{
    "@timestamp" => 2020-12-21T13:14:11.319Z,
       "message" => "message        3 of 3       ",
      "@version" => "1"
}
[2020-12-21T21:14:12,511][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2020-12-21T21:14:13,048][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2020-12-21T21:14:13,060][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-21T21:14:13,450][INFO ][logstash.runner          ] Logstash shut down.

　　

posted @   G1733  阅读(2018)  评论(0)  编辑  收藏  举报
编辑推荐：
· Linux系列：如何用 C#调用 C方法造成内存泄露
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 记一次.NET内存居高不下排查解决与启示
· 探究高空视频全景AR技术的实现原理
· 理解Rust引用及其生命周期标识（上）
阅读排行：
· DeepSeek 开源周回顾「GitHub 热点速览」
· 物流快递公司核心技术能力-地址解析分单基础技术分享
· .NET 10首个预览版发布：重大改进与新特性概览！
· AI与.NET技术实操系列（二）：开始使用ML.NET
· 单线程的Redis速度为什么快？
点击右上角即可分享
微信分享提示
AI FOR CODE 大赛