centos 部署 SparkR
---恢复内容开始---
环境配置——
- 操作系统：CentOS 6.5
- JDK版本：1.7.0_67
- Hadoop集群版本：CDH 5.3.0
安装过程——
1、（1）安装R
yum install -y R
（2）出现问题：
yum 安装软件时，报错：No package XXX available.
[root@localhost ~]# yum -y install redis Loaded plugins: fastestmirror No package redis available. Nothing to do
（3）解决方案
配置一台CentOS 7.x或RHEL 7.x的系统使用Fedora EPEL repos和第三方软件包安装各种软件包。请注意，这些软件包没有正式获得任何的CentOS或Red Hat的支持，但提供了许多流行的包和应用程序。
显示以上错误，有以下2种原因：
1、dns问题；
2、镜像连接错误；
DNS问题
首先测试下网络连接，如果ping不通，需要设置下DNS。
[root@localhost ~]# ping www.baidu.com XXXXXXXXXXXXXXXXX [root@localhost ~]# echo "nameserver 8.8.8.8">>/etc/resolv.conf
再次ping即可ping通了。
镜像连接错误问题
具体原因：使用yum搜索某些rpm包，找不到包是因为CentOS是RedHat企业版编译过来的，去掉了所有关于版权问题的东西。安装EPEL后可以很好的解决这个问题。EPEL(Extra Packages for Enterprise Linux )即企业版Linux的扩展包，提供了很多可共Centos使用的组件，安装完这个以后基本常用的rpm都可以找到。
Method #1 » 从企业版Linux库配置包中安装扩展包。（推荐 ）
[root@localhost ~]# sudo yum install epel-release Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package epel-release.noarch 0:7-5 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: epel-release noarch 7-5 extras 14 k Transaction Summary ================================================================================ Install 1 Package Total download size: 14 k Installed size: 24 k Is this ok [y/d/N]: y Downloading packages: epel-release-7-5.noarch.rpm | 14 kB 00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : epel-release-7-5.noarch 1/1 Verifying : epel-release-7-5.noarch 1/1 Installed: epel-release.noarch 0:7-5 Complete!
(4)最后运行下列代码 即可完成安装
yum install -y R
2、安装curl-devel（非常重要！否则无法进行RCurl包的安装，进而无法安装devtools）
yum install -y curl-devel
3.在R shell中安装devtools
install.packages("RCurl")
install.packages("devtools")
install.packages("devtools")
出现错误：
[plain] view plain copy
------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libcurl was not found. Try installing:
* deb: libcurl4-openssl-dev (Debian, Ubuntu, etc)
* rpm: libcurl-devel (Fedora, CentOS, RHEL)
* csw: libcurl_dev (Solaris)
If libcurl is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libcurl.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------------------------------------------------
ERROR: configuration failed for package ‘curl’
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/curl’
* installing *source* package ‘whisker’ ...
** package ‘whisker’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (whisker)
* installing *source* package ‘digest’ ...
** package ‘digest’ successfully unpacked and MD5 sums checked
** libs
[plain] view plain copy
configure: error: in `/tmp/RtmpkQMD9t/R.INSTALL6c1166a0db0e/git2r':
configure: error:
---------------------------------------------
OpenSSL library required
Please install:
libssl-dev (deb) or openssl-devel (rpm)
---------------------------------------------
See `config.log' for more details
ERROR: configuration failed for package ‘git2r’
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/git2r’
* installing *source* package ‘stringr’ ...
** package ‘stringr’ successfully unpacked and MD5 sums checked
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (stringr)
* installing *source* package ‘xml2’ ...
** package ‘xml2’ successfully unpacked and MD5 sums checked
Package libxml-2.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libxml-2.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libxml-2.0' found
Package libxml-2.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libxml-2.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libxml-2.0' found
Using PKG_CFLAGS=
Using PKG_LIBS=-lxml2
------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libxml-2.0 was not found. Try installing:
* deb: libxml2-dev (Debian, Ubuntu, etc)
* rpm: libxml2-devel (Fedora, CentOS, RHEL)
* csw: libxml2_dev (Solaris)
If libxml-2.0 is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libxml-2.0.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------------------------------------------------
ERROR: configuration failed for package ‘xml2’
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/xml2’
* installing *source* package ‘memoise’ ...
** package ‘memoise’ successfully unpacked and MD5 sums checked
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (memoise)
ERROR: dependency ‘curl’ is not available for package ‘httr’
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/httr’
* installing *source* package ‘evaluate’ ...
** package ‘evaluate’ successfully unpacked and MD5 sums checked
** R
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (evaluate)
* installing *source* package ‘roxygen2’ ...
** package ‘roxygen2’ successfully unpacked and MD5 sums checked
** libs
g++ -I/usr/local/lib64/R/include -DNDEBUG -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include" -fpic -g -O2 -c RcppExports.cpp -o RcppExports.o
g++ -I/usr/local/lib64/R/include -DNDEBUG -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include" -fpic -g -O2 -c isComplete.cpp -o isComplete.o
g++ -I/usr/local/lib64/R/include -DNDEBUG -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include" -fpic -g -O2 -c leadingSpaces.cpp -o leadingSpaces.o
g++ -I/usr/local/lib64/R/include -DNDEBUG -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include" -fpic -g -O2 -c parser.cpp -o parser.o
g++ -I/usr/local/lib64/R/include -DNDEBUG -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include" -fpic -g -O2 -c wrapString.cpp -o wrapString.o
g++ -shared -L/usr/local/lib64 -o roxygen2.so RcppExports.o isComplete.o leadingSpaces.o parser.o wrapString.o -L/usr/local/lib64/R/lib -lR
installing to /home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/roxygen2/libs
** R
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (roxygen2)
ERROR: dependencies ‘curl’, ‘xml2’ are not available for package ‘rversions’
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/rversions’
ERROR: dependencies ‘httr’, ‘curl’, ‘rversions’, ‘git2r’ are not available for package ‘devtools’
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/devtools’
The downloaded source packages are in
‘/tmp/Rtmp1A16li/downloaded_packages’
Warning messages:
1: In install.packages("devtools") :
installation of package ‘curl’ had non-zero exit status
2: In install.packages("devtools") :
installation of package ‘git2r’ had non-zero exit status
3: In install.packages("devtools") :
installation of package ‘xml2’ had non-zero exit status
4: In install.packages("devtools") :
installation of package ‘httr’ had non-zero exit status
5: In install.packages("devtools") :
installation of package ‘rversions’ had non-zero exit status
6: In install.packages("devtools") :
installation of package ‘devtools’ had non-zero exit status
根据错误安装，我这里安装了
[plain] view plain copy
[root@feng03 ~]# yum install libcurl-devel
[root@feng03 ~]# yum install openssl-devel
[root@feng03 ~]# yum install libxml2-devel
进入R shell
>install.packages("git2r")
>install.packages("xml2")
>install.packages("rversions")
步骤4: 安装好rJava及devtools，接下来安装SparkR
library(devtools)
install_github("amplab-extras/SparkR-pkg", subdir="pkg")
出现错误
Launching sbt from sbt/sbt-launch-0.13.6.jar
Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.6.jar
make: *** [target/scala-2.10/sparkr-assembly-0.1.jar] 错误 1
ERROR: compilation failed for package ‘SparkR’
* removing ‘/home/ndscbigdata/R/x86_64-pc-linux-gnu-library/3.2/SparkR’
错误: Command failed (1)
解决：
进入spark安装包中的R目录，直接输入如下命令，直接就DONE了。
cd进入目录
/data1/bigdata/spark-1.6.1/R/lib/SparkR
R -e "devtools::install('.')"
---恢复内容结束---
