storm本地python开发环境搭建
搭建storm本地python开发环境
需要使用python进行storm的开发,这里记录本地开发环境搭建,过程中使用的资源gitee中下载
- centos7 普通用户
- python 3.5.1
- jdk 1.8.0
- storm 1.1.0
- lein+streamparse
本地用户faron, 根目录/home/faron
这次使用普通用户安装
python 3.5.1安装
这里我之前使用pyenv已经部署好 /home/faron目录下的pthon环境为3.5.1, 这里可自行搜索
jdk 1.8.0安装
-
安装包下载
-
在~/.bashrc文间尾部追加java环境变量
# java export JAVA_HOME=//home/faron/lein/java_storm/jdk1.8.0_161 export JRE_HOME=$JAVA_HOME/jre # export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
-
激活环境变量
source ~/.bashrc
-
校验java环境
java versoin
终端显示
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
strom 1.1.0安装
-
安装包下载
-
在~/.bashrc文间尾部追加storm环境变量
# storm export STORM_HOME=/home/faron/lein/java_storm/apache-storm-1.1.0 export PATH=$STORM_HOME/bin:$PATH
-
激活环境变量
source ~/.bashrc
-
校验storm环境
storm version
终端显示
Running: //home/faron/lein/java_storm/jdk1.8.0_161/bin/java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/faron/lein/java_storm/apache-storm-1.1.0 -Dstorm.log.dir=/home/faron/lein/java_storm/apache-storm-1.1.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /home/faron/lein/java_storm/apache-storm-1.1.0/lib/storm-core-1.1.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/kryo-3.0.3.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/reflectasm-1.10.1.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/asm-5.0.3.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/minlog-1.3.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/objenesis-2.1.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/clojure-1.7.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/ring-cors-0.1.5.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/disruptor-3.3.2.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-api-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-core-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-slf4j-impl-2.8.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/slf4j-api-1.7.21.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/log4j-over-slf4j-1.6.6.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/servlet-api-2.5.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/lib/storm-rename-hack-1.1.0.jar:/home/faron/lein/java_storm/apache-storm-1.1.0/conf org.apache.storm.utils. VersionInfo
Storm 1.1.0
URL https://git-wip-us.apache.org/repos/asf/storm.git -r e40d213de7067f7d3aa4d4992b81890d8ed6ff31
Branch (no branch)
Compiled by ptgoetz on 2017-03-21T17:04Z
From source with checksum 4ddc442e8b804654454ab26fbec6348
lein+streamparse安装
-
安装过程中可能会出现lein脚本无法下载,leiningen-2.9.3-standalone.jar无法下载的情况
makdir ~/bin cd ~/bin cp ~/lein/lein ./ chmod +x ~/bin/lein # 运行lein安装脚本 lein
终端显示(下载速度可能会有点慢)
WARNING: You have $CLASSPATH set, probably by accident.
It is strongly recommended to unset this before proceeding.
Downloading Leiningen to /home/faron/.lein/self-installs/leiningen-2.9.3-standalone.jar now...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 637 100 637 0 0 393 0 0:00:01 0:00:01 --:--:-- 393
27 13.9M 27 3942k 0 0 34711 0 0:07:02 0:01:56 0:05:06 26468如果
leiningen-2.9.3-standalone.jar
下载失败cp ~/lein/java_storm/leiningen-2.9.3-standalone.zip ~/.lein/self-installs/
lein环境验证
lein version # >>>终端显示 # WARNING: You have $CLASSPATH set, probably by accident. # It is strongly recommended to unset this before proceeding. # Leiningen 2.9.3 on Java 1.8.0_161 Java HotSpot(TM) 64-Bit Server VM
-
streamparse安装
-
安装
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip pip install -i https://pypi.tuna.tsinghua.edu.cn/simple streamparse
-
启动wordcount
mkdir ~/storm_workplace cd ~/storm_workplace sparse quickstart wordcount cd wordcount sparse run
-
异常中心包无法下载
Could not transfer artifact org.apache.storm:storm-core:jar:1.1.0 from/to central (https://repo1.maven.org/maven2/): Connect to repo1.maven.org:443 [repo1.maven.org/151.101.40.209] failed: 拒绝连接 (Connection refused)
Could not find artifact org.apache.storm:storm-core:jar:1.1.0 in clojars (https://repo.clojars.org/)
Could not transfer artifact org.clojure:clojure:jar:1.7.0 from/to central (https://repo1.maven.org/maven2/): Connect to repo1.maven.org:443 [repo1.maven.org/151.101.40.209] failed: 拒绝连接 (Connection refused)
Could not find artifact org.clojure:clojure:jar:1.7.0 in clojars (https://repo.clojars.org/)
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.# 仅限storm1.1.0,因为我的repository里面是storm-core:1.1.0本地环境 tar -zxvf ~/lein/java_storm/repository.tar.gz -C ~/.m2/ # 再次运行 sparse run
-