安装sqoop并升级

未经允许不得转载

安装sqoop并升级过程详解

注意：tbds中根据实际情况查看jdk的版本和路劲信息在直接配置即可

1. 安装jdk

安装程序yum install java-1.7.0-openjdk.x86_64 -y

验证是否安装成功java -version

安装开发环境 yum install java-1.7.0-openjdk-devel-1.7.0.261-2.6.22.2.el7_8.x86_64  -y

验证是否安装成功 javac

设置环境变量 vim /etc/profile 新增export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk

使环境变量生效 source /etc/profile

2. 安装hadoop(****伪分布式)

创建文件目录mkdir /opt/software &&  cd /opt/software/

下载软件 

wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

解压软件 tar zxvf hadoop-2.7.7.tar.gz -C /opt/software  **(****根据之前的路劲自己定义****)**

修改环境变量 vim /etc/profile 

新增export HADOOP_HOME=/opt/software/hadoop-2.7.7

修改环境变量 vi /etc/profile 

新增export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin  

 配置生效 source /etc/profile

 反复检查是否生效 echo $JAVA_HOME  echo $HADOOP_HOME  echo $PATH

如图：

 修改配置文件vim  /opt/software/hadoop-2.7.7/etc/hadoop/core-site.xml  ##注意格式

<configuration>

​     <property>

​          <name>fs.defaultFS</name>

​          <value>hdfs://192.168.10.201:9000</value>

​     </property>

</configuration>

修改配置文件vim  /opt/software/hadoop-2.7.7/etc/hadoop/hdfs-site.xml

<configuration>

​     <property>

​          <name>dfs.replication</name>

​          <value>1</value>

​     </property>

</configuration>

配置hadoop内置环境变量

cd  /opt/software/hadoop-2.7.7/etc/hadoop/

  vim hadoop-env.sh

  export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk

集群初始化 hdfs namenode -format

注意：有时候会遇到版本不支持

报错：

更改版本如图：改为1.0

启动hdfs

 . /opt/software/hadoop-2.7.7/sbin/start-dfs.sh

启动yarn

cd /opt/software/hadoop-2.7.7/sbin/  && ./start-yarn.sh

验证hdfs启动是否成功 http://192.168.10.201:50070

验证yarn启动是否成功 http://192.168.10.201:8088

3. 安装mysql 当前我使用的是方法二安装

方法一：

l 下载yum repo

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm

l 安装yum repo

yum -y install mysql57-community-release-el7-10.noarch.rpm

l 安装mysql

yum -y install mysql-community-server

l 启动mysql

systemctl start  mysqld.service
systemctl status mysqld.service

l 修改mysql密码

mysql
use mysql
update user set password=password('Bgx123.com') where user='root'
flush privileges;

l 重启mysql

systemctl start  mysqld.service
systemctl status mysqld.service
 

方法二

#1.下载官方扩展源, 扩展源集成mysql5.6、5.7、8.0,仅5.7仓库是开启

rpm -ivh http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql57-community-release-el7-10.noarch.rpm

#2.安装mysql, 默认5.7版本, 文件过大可能会导致下载失败

yum install mysql-community-server -y

#3.启动数据库, 并加入开机自启动

systemctl start mysqld

systemctl enable mysqld

#4.登陆mysql数据库[password中填写上一步过滤的密码]

mysql -uroot -p$(awk '/temporary password/{print $NF}' /var/log/mysqld.log)

#5.重新修改数据库密码

 ALTER USER 'root'@'localhost' IDENTIFIED BY 'Bgx123.com';

\4. 安装sqoop1.4.5

下载软件

http://archive.apache.org/dist/sqoop/（选择__hadoop版本）

解压软件

tar zxvf sqoop-1.4.5.bin__hadoop-2.0.4.tar.gz -C /opt/software/

配置环境变量vim /etc/profile

export SQOOP_HOME=/opt/software/sqoop-1.4.5.bin__hadoop-0.20

修改配置文件 cd /opt/software/sqoop-1.4.5.bin__hadoop-0.20/conf

mv sqoop-env-template.sh sqoop-env.sh
vim sqoop-env.sh 
export HADOOP_COMMON_HOME=/opt/software/hadoop-2.7.7
export HADOOP_MAPRED_HOME=/opt/software/hadoop-2.7.7

下载mysql连接器（5.1.30.tar.gz）和（5.1.49.tar.gz）都测试过

http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/
wget  http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/mysql-connector-java-5.1.49.tar.gz

解压后将jar文件移到lib下

mv mysql-connector-java-5.1.30-bin.jar /opt/software/sqoop-1.4.5.bin__hadoop-0.20/lib

验证安装 cd /opt/software/sqoop-1.4.5.bin__hadoop-0.20/bin/

sqoop-version

验证连接mysql

sqoop list-databases --connect jdbc:mysql://127.0.0.1:3306/ --username root -P

输入密码后显示出mysql所有库信息

\5. 安装sqoop1.4.7

下载软件

http://archive.apache.org/dist/sqoop/（选择__hadoop版本）

解压软件

tar zxvf sqoop-1.4.7.bin__hadoop-2.0.4.tar.gz -C /opt/software/

配置环境变量vi /etc/profile

export SQOOP_HOME=/opt/software/sqoop-1.4.7.bin__hadoop-2.6.0

修改配置文件 cd /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0/conf

mv sqoop-env-template.sh sqoop-env.sh
vi sqoop-env.sh 
export HADOOP_COMMON_HOME=/opt/software/hadoop-2.10.0
export HADOOP_MAPRED_HOME=/opt/software/hadoop-2.10.0

下载mysql连接器（5.1.30.tar.gz）

http://ftp.ntu.edu.tw/MySQL/Downloads/Connector-J/

解压后将jar文件移到lib下

mv mysql-connector-java-5.1.30-bin.jar /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0/lib

验证安装 cd /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0/bin

更新量 vim /etc/profile

export SQOOP_HOME=/opt/software/sqoop-1.4.7.bin__hadoop-2.6.0

sqoop-version

验证连接mysql

sqoop list-databases --connect jdbc:mysql://127.0.0.1:3306/ --username root -P

输入密码后显示出mysql所有库信息

注意：-p (****代表数据库密码)

验证版本信息

6.****回滚

更改环境变量

vim /etc/profile

export SQOOP_HOME=/opt/software/sqoop-1.4.5.bin__hadoop-0.20

source /etc/profile

回滚之后的版本

软连接目录

ln -s /etc/sqoop/conf /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0/