打赏

Hadoop源码的编译过程详细解读(各版本)

 

 

说在前面的话

   重新试多几次。编译过程中会出现下载某个包的时间太久,这是由于连接网站的过程中会出现假死,按ctrl+c,重新运行编译命令。 
如果出现缺少了某个文件的情况,则要先清理maven(使用命令 mvn clean) 再重新编译。 
 
 
 
 

前言

   Spark可以通过SBT和Maven两种方式进行编译,再通过make-distribution.sh脚本生成部署包。  

   SBT编译需要安装git工具,而Maven安装则需要maven工具,两种方式均需要在联网 下进行。

    尽管maven是Spark官网推荐的编译方式,但是sbt的编译速度更胜一筹。因此,对于spark的开发者来说,sbt编译可能是更好的选择。由于sbt编译也是基于maven的POM文件,因此sbt的编译参数与maven的编译参数是一致的。

 

 

 

 

心得

   有时间,自己一定要动手编译源码,想要成为高手和大数据领域大牛,前面的苦,是必定要吃的。

   无论是编译spark源码,还是hadoop源码。新手初次编译,一路会碰到很多问题,也许会花上个一天甚至几天,这个是正常。把心态端正就是!有错误,更好,解决错误,是最好锻炼和提升能力的。

       更不要小看它们,能碰到是幸运,能弄懂和深入研究,之所以然,是福气。

 

 

 

 

各大版本简介

  1、Apache版------可自己编译,也可采用预编译的版本

 2、CDH版---------无需自己编译

 

Cloudera Manager安装之利用parcels方式安装3节点集群(包含最新稳定版本或指定版本的安装)(添加服务)

 

 3、HDP版----------无需自己编译

 

Ambari安装部署搭建hdp集群(图文分五大步详解)(博主强烈推荐)

 

主流是这3大版本,其实,是有9大版本。CDH的CM是要花钱的,当然它的预编译包,是免费的。

 

 

 

hadoop/spark源码的下载方式:

  1、官网下载

  2、Github下载(仅source code)

    

以下是从官网下载:

 

以下是Github下载(仅source code)

 

 

CDH的下载

 http://archive-primary.cloudera.com/cdh5/cdh/5/

 

 

HDP的下载

http://zh.hortonworks.com/products/

 

 

好的,那我这里就以,官网为例。

         准备Linux系统环境(如CentOS6.5)

 

  • 为什么要源码安装? 
    是因为之前下的binary版,解压后直接用,发现每次执行命令都会报个WARN,如: 
    hd_admin@master:/usr/local/hadoop$ hadoop fs -ls 
    14/10/27 19:59:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 

  看了下解压后的目录,lib/naive下完全是空的,另外网上查也有说此文件夹下如果是32位环境下编译出来的.so文件,在64位系统上也会有此问题,因此直接从cloudera上下了src版手工编译一遍

  因此如非必要,完全可以选择Apache Hadoop发行版,并且非源码安装编译的话可以跳过。

前言,这编译是一个复杂和长知识的过程。

版本和位置得要有自己的习惯。

 

 

1  首先检查并安装gcc-c++

在root用户下在线下载:

yum -y install gcc-c++

 

2● 首先检查并安装lrzsz

在root用户下在线下载:

yum -y install lrzsz

3.图形界面安装,会自带有Centos6.5自带的JDK

 一般将获得如下信息:
    java version "1.7.0_45"

OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)

OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)


 第一步:查看Centos6.5自带的JDK是否已安装 

<1> 检测原OPENJDK版本  

# java -version    

一般将获得如下信息:      

tzdata-java-2013g-1.el6.noarch

java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64


 <2>进一步查看JDK信息

# rpm -qa|grep java   

 rpm -e --nodeps tzdata-java-2013g-1.el6.noarch

 rpm -e --nodeps java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

rpm -e --nodeps java-1.6.0-openjdk-1.6.0.0-1.66.1.13.0.el6.x86_64


 <3>卸载OPENJDK

自带的jdk已经没了。

在root用户下安装jdk-7u79-linux-x64.tar.gz

在/usr/local上传

解压,

tar -zxvf jdk-7u79-linux-x64.tar.gz

删去压缩包

rm -rf jdk-7u79-linux-x64.tar.gz

配置环境变量

vim /etc/profile

#java

export JAVA_HOME=/usr/local/jdk1.7.0_79

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/jre/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

文件生效

source /etc/profile

查看是否安装成功

java -version

  4、● 安装maven

下载apache-maven-3.3.3-bin.tar.gz

/usr/local/

上传apache-maven-3.3.3-bin.tar.gz


解压

tar -zxvf apache-maven-3.3.3-bin.tar.gz

删除压缩包

rm -rf apache-maven-3.3.3-bin.tar.gz


  5、● 安装ant

下载apache-ant-1.9.7-bin.tar.gz

/usr/local

上传

解压

tar -zxvf apache-ant-1.9.7-bin.tar.gz

删除压缩包

rm -rf apache-ant-1.9.7-bin.tar.gz

 

Maven和ant配置环境变量

vim /etc/profile
#maven
export MAVEN_HOME=/usr/local/apache-maven-3.3.3
export PATH=$PATH:$MAVEN_HOME/bin
 (这里错误啦!!)

PATH=$MAVEN_HOME/bin: $PATH
Export MAVEN_HOME   (对的)



#ant
export ANT_HOME=/usr/local/apache-ant-1.9.7
export PATH=$PATH:$ANT_HOME/bin

文件生效

source /etc/profile

查看是否安装成功

mvn -v
ant -v

如果出现如下内容,说明安装成功:

Buildfile: build.xml does not exist!

Build failed

说明ant安装成功!因为ant默认运行build.xml文件,这个文件需要我们建立。

 

6. ● 安装ProtocolBuffer   

下载protobuf-2.5.0.tar.gz

在/usr/local/

上传protobuf-2.5.0.tar.gz

解压

tar -zxvf protobuf-2.5.0.tar.gz

删除压缩包

rm -rf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
./configure --prefix=/usr/local/protobuf-2.5.0
make
make check

make install

配置环境变量

vim /etc/profile
# protobuf
export PROTOBUF_HOME=/usr/local/protobuf-2.5.0
export PATH=$PATH:$PROTOBUF_HOME/bin

文件生效

source /etc/profile

查看是否安装成功

protoc --version

 

6、安装cmake

yum -y install cmake


7● zlib安装

yum -y install zlib

8● openssl-devel

yum -y install openssl-devel
 

9 ● 安装snappy

下载snappy-1.1.3.tar.gz

上传/usr/local/

解压

tar -zxvf snappy-1.1.3.tar.gz

删除压缩包

rm -rf snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure
make

make install

如果安装成功的话进入【/usr/local/lib】目录下(默认位置),可以看到已经生成了snappy的库文件

此刻,已经是该装好的,装好了,准备执行!

 

10. 在编译hadoop源码之前,替换掉MAVEN_HOME/conf目录下的setting.xml

 

 

 
11 下载对应的Hadoop源码

 


其实啊,这之后也是hadoop-2.6.0-cdh5.4.5-src.tar.gz


编译,生成hadoop-2.6.0-cdh5.4.5.tar.gz,然后进行集群安装!



hadoop-2.6.0-cdh5.4.5-src.tar.gz


源码编译正在开始!


http://archive.cloudera.com/cdh5/cdh/5/


下载hadoop-2.6.0-cdh5.4.5-src.tar.gz


解压hadoop-2.6.0-cdh5.4.5-src.tar.gz源代码包


删除hadoop-2.6.0-cdh5.4.5-src.tar.gz源代码包

然后进入hadoop-2.6.0-cdh5.4.5目录,

mvn package -Pdist,native -DskipTests -Dtar

mvn编译时间很长,虚拟机里面编译了11分钟。

mvn package -Pdist,native -DskipTests -Dtar -Dbundle.snappy -Dsnappy.lib=/usr/local/lib(也可以尝试)

mvn package [-Pdist] [-Pdocs] [-Psrc] [-Pnative] [Dtar] 这是构建发布版

 

进入

mvn package -Pdist,native -DskipTests -Dtar

 

 

[INFO] Apache Hadoop Extras ............................... SKIPPED

[INFO] Apache Hadoop Pipes ................................ SKIPPED

[INFO] Apache Hadoop OpenStack support .................... SKIPPED

[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED

[INFO] Apache Hadoop Client ............................... SKIPPED

[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED

[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED

[INFO] Apache Hadoop Tools Dist ........................... SKIPPED

[INFO] Apache Hadoop Tools ................................ SKIPPED

[INFO] Apache Hadoop Distribution ......................... SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:24 h

[INFO] Finished at: 2016-09-14T02:04:36+08:00

[INFO] Final Memory: 49M/120M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-annotations: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test failed: Plugin org.apache.maven.plugins:maven-surefire-plugin:2.16 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-surefire-plugin:jar:2.16 -> org.apache.maven.surefire:maven-surefire-common:jar:2.16 -> org.apache.maven.plugin-tools:maven-plugin-annotations:jar:3.2: Failed to read artifact descriptor for org.apache.maven.plugin-tools:maven-plugin-annotations:jar:3.2: Could not transfer artifact org.apache.maven.plugin-tools:maven-plugin-annotations:pom:3.2 from/to nexus-osc (http://nexus.rc.dataengine.com/nexus/content/groups/public): Access denied to: http://nexus.rc.dataengine.com/nexus/content/groups/public/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.pom , ReasonPhrase:Forbidden. -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]   mvn <goals> -rf :hadoop-annotations

[root@Compiler hadoop-2.6.0-src]#

 

其实,同spark源码编译一样,见我的博客链接

遇到问题,就解决问题。

 

 Spark源码的编译过程详细解读(各版本)

 

然后,我们来解决这个问题:

解决方法:
vim ./hadoop-common-project/hadoop-auth/pom.xml

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-annotations</artifactId>
<scope>compile</scope>
</dependency>

将节点hadoop-annotations的值由provide 改成compile

[root@Compiler local]# ls

apache-ant-1.9.7  apache-maven-3.3.3  bin  etc  games  hadoop-2.6.0-src  include  jdk1.7.0_79  lib  lib64  libexec  protobuf-2.5.0  sbin  share  snappy-1.1.3  src

[root@Compiler local]# cd hadoop-2.6.0-src/

[root@Compiler hadoop-2.6.0-src]# ls

BUILDING.txt  hadoop-assemblies  hadoop-common-project  hadoop-hdfs-project       hadoop-maven-plugins  hadoop-project       hadoop-tools         LICENSE.txt  pom.xml

dev-support   hadoop-client      hadoop-dist            hadoop-mapreduce-project  hadoop-minicluster    hadoop-project-dist  hadoop-yarn-project  NOTICE.txt   README.txt

[root@Compiler hadoop-2.6.0-src]# vim ./hadoop-common-project/hadoop-auth/pom.xml

[root@Compiler hadoop-2.6.0-src]# mvn package -Pdist,native -DskipTests –Dtar


http://mvnrepository.com/search?q=

然后在/root/Downloads/目录下,下载

wget -c http://central.maven.org/maven2/org/apache/hadoop/hadoop-kms/2.6.0/hadoop-kms-2.6.0.war

然后,

mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-kms -Dversion=2.6.0 -Dfile=/root/Downloads/hadoop-kms-2.6.0.war

成功后,

继续执行

mvn package -Pdist,native -DskipTests -Dtar


往复下去 !  此处我不再赘述。



这里,就不一一赘述了。类似的问题,都是这样的一个解决方法。直至,把对应的包都下载好。然后就最后成功了!

网速好的话,1个小时吧,网速慢的话,好几个小时。

 setting.xml文件的内容是:

 

<?xml version="1.0" encoding="UTF-8"?>

<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

<!--
| This is the configuration file for Maven. It can be specified at two levels:
|
| 1. User Level. This settings.xml file provides configuration for a single user,
| and is normally provided in ${user.home}/.m2/settings.xml.
|
| NOTE: This location can be overridden with the CLI option:
|
| -s /path/to/user/settings.xml
|
| 2. Global Level. This settings.xml file provides configuration for all Maven
| users on a machine (assuming they're all using the same Maven
| installation). It's normally provided in
| ${maven.home}/conf/settings.xml.
|
| NOTE: This location can be overridden with the CLI option:
|
| -gs /path/to/global/settings.xml
|
| The sections in this sample file are intended to give you a running start at
| getting the most out of your Maven installation. Where appropriate, the default
| values (values used when the setting is not specified) are provided.
|
|-->
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<!-- localRepository
| The path to the local repository maven will use to store artifacts.
|
| Default: ${user.home}/.m2/repository
<localRepository>/path/to/local/repo</localRepository>
-->

<!-- interactiveMode
| This will determine whether maven prompts you when it needs input. If set to false,
| maven will use a sensible default value, perhaps based on some other setting, for
| the parameter in question.
|
| Default: true
<interactiveMode>true</interactiveMode>
-->

<!-- offline
| Determines whether maven should attempt to connect to the network when executing a build.
| This will have an effect on artifact downloads, artifact deployment, and others.
|
| Default: false
<offline>false</offline>
-->

<!-- pluginGroups
| This is a list of additional group identifiers that will be searched when resolving plugins by their prefix, i.e.
| when invoking a command line like "mvn prefix:goal". Maven will automatically add the group identifiers
| "org.apache.maven.plugins" and "org.codehaus.mojo" if these are not already contained in the list.
|-->
<pluginGroups>
<!-- pluginGroup
| Specifies a further group identifier to use for plugin lookup.
<pluginGroup>com.your.plugins</pluginGroup>
-->
</pluginGroups>

<!-- proxies
| This is a list of proxies which can be used on this machine to connect to the network.
| Unless otherwise specified (by system property or command-line switch), the first proxy
| specification in this list marked as active will be used.
|-->
<proxies>
<!-- proxy
| Specification for one proxy, to be used in connecting to the network.
|
<proxy>
<id>optional</id>
<active>true</active>
<protocol>http</protocol>
<username>proxyuser</username>
<password>proxypass</password>
<host>proxy.host.net</host>
<port>80</port>
<nonProxyHosts>local.net|some.host.com</nonProxyHosts>
</proxy>
-->
</proxies>

<!-- servers
| This is a list of authentication profiles, keyed by the server-id used within the system.
| Authentication profiles can be used whenever maven must make a connection to a remote server.
|-->
<servers>
<!-- server
| Specifies the authentication information to use when connecting to a particular server, identified by
| a unique name within the system (referred to by the 'id' attribute below).
|
| NOTE: You should either specify username/password OR privateKey/passphrase, since these pairings are
| used together.
|
<server>
<id>deploymentRepo</id>
<username>repouser</username>
<password>repopwd</password>
</server>
-->

<!-- Another sample, using keys to authenticate.
<server>
<id>siteServer</id>
<privateKey>/path/to/private/key</privateKey>
<passphrase>optional; leave empty if not used.</passphrase>
</server>
-->
</servers>

<!-- mirrors
| This is a list of mirrors to be used in downloading artifacts from remote repositories.
|
| It works like this: a POM may declare a repository to use in resolving certain artifacts.
| However, this repository may have problems with heavy traffic at times, so people have mirrored
| it to several places.
|
| That repository definition will have a unique id, so we can create a mirror reference for that
| repository, to be used as an alternate download site. The mirror site will be the preferred
| server for that repository.
|-->
<mirrors>
<!-- mirror
| Specifies a repository mirror site to use instead of a given repository. The repository that
| this mirror serves has an ID that matches the mirrorOf element of this mirror. IDs are used
| for inheritance and direct lookup purposes, and must be unique across the set of mirrors.
|
<mirror>
<id>mirrorId</id>
<mirrorOf>repositoryId</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
<url>http://my.repository.com/repo/path</url>
</mirror>
-->
<mirror>
<id>osc</id>
<mirrorOf>central</mirrorOf>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
</mirror>

<mirror>
<id>osc_thirdparty</id>
<mirrorOf>thirdparty</mirrorOf>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
</mirror>

</mirrors>

<!-- profiles
| This is a list of profiles which can be activated in a variety of ways, and which can modify
| the build process. Profiles provided in the settings.xml are intended to provide local machine-
| specific paths and repository locations which allow the build to work in the local environment.
|
| For example, if you have an integration testing plugin - like cactus - that needs to know where
| your Tomcat instance is installed, you can provide a variable here such that the variable is
| dereferenced during the build process to configure the cactus plugin.
|
| As noted above, profiles can be activated in a variety of ways. One way - the activeProfiles
| section of this document (settings.xml) - will be discussed later. Another way essentially
| relies on the detection of a system property, either matching a particular value for the property,
| or merely testing its existence. Profiles can also be activated by JDK version prefix, where a
| value of '1.4' might activate a profile when the build is executed on a JDK version of '1.4.2_07'.
| Finally, the list of active profiles can be specified directly from the command line.
|
| NOTE: For profiles defined in the settings.xml, you are restricted to specifying only artifact
| repositories, plugin repositories, and free-form properties to be used as configuration
| variables for plugins in the POM.
|
|-->
<profiles>
<!-- profile
| Specifies a set of introductions to the build process, to be activated using one or more of the
| mechanisms described above. For inheritance purposes, and to activate profiles via <activatedProfiles/>
| or the command line, profiles have to have an ID that is unique.
|
| An encouraged best practice for profile identification is to use a consistent naming convention
| for profiles, such as 'env-dev', 'env-test', 'env-production', 'user-jdcasey', 'user-brett', etc.
| This will make it more intuitive to understand what the set of introduced profiles is attempting
| to accomplish, particularly when you only have a list of profile id's for debug.
|
| This profile example uses the JDK version to trigger activation, and provides a JDK-specific repo.
<profile>
<id>jdk-1.4</id>

<activation>
<jdk>1.4</jdk>
</activation>

<repositories>
<repository>
<id>jdk14</id>
<name>Repository for JDK 1.4 builds</name>
<url>http://www.myhost.com/maven/jdk14</url>
<layout>default</layout>
<snapshotPolicy>always</snapshotPolicy>
</repository>
</repositories>
</profile>
-->

<!--
| Here is another profile, activated by the system property 'target-env' with a value of 'dev',
| which provides a specific path to the Tomcat instance. To use this, your plugin configuration
| might hypothetically look like:
|
| ...
| <plugin>
| <groupId>org.myco.myplugins</groupId>
| <artifactId>myplugin</artifactId>
|
| <configuration>
| <tomcatLocation>${tomcatPath}</tomcatLocation>
| </configuration>
| </plugin>
| ...
|
| NOTE: If you just wanted to inject this configuration whenever someone set 'target-env' to
| anything, you could just leave off the <value/> inside the activation-property.
|
<profile>
<id>env-dev</id>

<activation>
<property>
<name>target-env</name>
<value>dev</value>
</property>
</activation>

<properties>
<tomcatPath>/path/to/tomcat/instance</tomcatPath>
</properties>
</profile>
-->
<profile>
<id>jdk-1.4</id>
<activation>
<jdk>1.4</jdk>
</activation>

<repositories>

<repository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>

</repositories>

<pluginRepositories>

<pluginRepository>
<id>nexus</id>
<name>local private nexus</name>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</pluginRepository>

</pluginRepositories>
</profile>

<profile>
<id>osc</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>

<repositories>

<repository>
<id>osc</id>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
</repository>

<repository>
<id>osc_thirdparty</id>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
</repository>

</repositories>

<pluginRepositories>

<pluginRepository>
<id>osc</id>
<url>http://maven.aliyun.com/nexus/content/repositories/central</url>
</pluginRepository>

</pluginRepositories>
</profile>
</profiles>

<!-- activeProfiles
| List of profiles that are active for all builds.
|
<activeProfiles>
<activeProfile>alwaysActiveProfile</activeProfile>
<activeProfile>anotherAlwaysActiveProfile</activeProfile>
</activeProfiles>
-->
</settings>





setting.xml文件的内容是:


<?xml version="1.0" encoding="UTF-8"?>

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
 
  <pluginGroups>
  </pluginGroups>


  <proxies>
  </proxies>


  <servers>
  </servers>

  <mirrors>
    <mirror>
        <id>nexus-osc</id>
        <mirrorOf>*</mirrorOf>
        <name>Nexus osc</name>
        <url>http://nexus.rc.dataengine.com/nexus/content/groups/public</url>
    </mirror>
    <mirror>
        <id>nexus-osc</id>
        <mirrorOf>central</mirrorOf>
        <name>Nexus osc</name>
        <url>http://maven.oschina.net/content/groups/public</url>
    </mirror>
    <mirror>
        <id>nexus-osc-thirdparty</id>
        <mirrorOf>thirdparty</mirrorOf>
        <name>Nexus osc thirdparty</name>
        <url>http://maven.oschina.net/content/repositories/thirdparty</url>
    </mirror>
    <mirror>
        <id>central</id>
        <mirrorOf>central</mirrorOf>
        <name>central</name>
        <url>http://central.maven.org/maven2</url>
    </mirror>
    <mirror>
        <id>repol</id>
        <mirrorOf>central</mirrorOf>
        <name>repol</name>
        <url>http://repol.maven.org/maven2</url>
    </mirror>
  </mirrors>

  <profiles>
    <profile>
        <id>jdk-1.7</id>
        <activation>
            <jdk>1.7</jdk>
        </activation>

        <repositories>
            <repository>
                <id>rc</id>
                <name>rc nexus</name>
                <url>http://nexus.rc.dataengine.com/nexus/content/groups/public</url>
                <releases>
                        <enabled>true</enabled>
                </releases>
                <snapshots>
                        <enabled>false</enabled>
                </snapshots>
            </repository>
            <repository>
                <id>nexus</id>
                <name>local private nexus</name>
                <url>http://maven.oschina.net/content/groups/public</url>
                <releases>
                        <enabled>true</enabled>
                </releases>
                <snapshots>
                        <enabled>false</enabled>
                </snapshots>    
            </repository>
            <repository>
                <id>central</id>
                <name>central</name>
                <url>http://central.maven.org/maven2/</url>
                <releases>
                        <enabled>true</enabled>
                </releases>
                <snapshots>
                        <enabled>false</enabled>
                </snapshots>    
            </repository>
            <repository>
                <id>repol</id>
                <name>repol</name>
                <url>http://repol.maven.org/maven2/</url>
                <releases>
                        <enabled>true</enabled>
                </releases>
                <snapshots>
                        <enabled>false</enabled>
                </snapshots>    
            </repository>
      </repositories>
      <pluginRepositories>
                <pluginRepository>
                    <id>rc</id>
                    <name>rc nexus</name>
                    <url>http://nexus.rc.dataengine.com/nexus/content/groups/public</url>
                    <releases>
                        <enabled>true</enabled>
                    </releases>
                    <snapshots>
                        <enabled>false</enabled>
                     </snapshots>
                </pluginRepository>
                <pluginRepository>
                    <id>nexus</id>
                    <name>local private nexus</name>
                    <url>http://maven.oschina.net/content/groups/public</url>
                    <releases>
                            <enabled>true</enabled>
                    </releases>
                    <snapshots>
                            <enabled>false</enabled>
                    </snapshots>    
                </pluginRepository>
                <pluginRepository>
                    <id>central</id>
                    <name>central</name>
                    <url>http://central.maven.org/maven2/</url>
                    <releases>
                            <enabled>true</enabled>
                    </releases>
                    <snapshots>
                            <enabled>false</enabled>
                    </snapshots>    
                </pluginRepository>
                <pluginRepository>
                    <id>repol</id>
                    <name>repol</name>
                    <url>http://repol.maven.org/maven2/</url>
                    <releases>
                            <enabled>true</enabled>
                    </releases>
                    <snapshots>
                            <enabled>false</enabled>
                    </snapshots>    
                </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>    
</settings>
                       






感谢以下博主:
https://my.oschina.net/laigous/blog/356552  
http://blog.csdn.net/sunbow0/article/details/41961459 (黄美灵老师) Spark MLlib机器学习 算法、源码及实战详解作者 















欢迎大家,加入我的微信公众号:大数据躺过的坑        人工智能躺过的坑

 

 

 

同时,大家可以关注我的个人博客

   http://www.cnblogs.com/zlslch/   和     http://www.cnblogs.com/lchzls/      http://www.cnblogs.com/sunnyDream/   

   详情请见:http://www.cnblogs.com/zlslch/p/7473861.html

 

  人生苦短,我愿分享。本公众号将秉持活到老学到老学习无休止的交流分享开源精神,汇聚于互联网和个人学习工作的精华干货知识,一切来于互联网,反馈回互联网。
  目前研究领域:大数据、机器学习、深度学习、人工智能、数据挖掘、数据分析。 语言涉及:Java、Scala、Python、Shell、Linux等 。同时还涉及平常所使用的手机、电脑和互联网上的使用技巧、问题和实用软件。 只要你一直关注和呆在群里,每天必须有收获

 

      对应本平台的讨论和答疑QQ群:大数据和人工智能躺过的坑(总群)(161156071) 

 

 

 

 

 

 

 

 

 

 


 


 

 
posted @ 2016-09-13 20:49  大数据和AI躺过的坑  阅读(5892)  评论(0编辑  收藏  举报