Hadoop3.2.1-MiniCluster安装

官网指南

Purpose

Using the CLI MiniCluster, users can simply start and stop a single-node Hadoop cluster with a single command, and without the need to set any environment variables or manage configuration files. The CLI MiniCluster starts both a YARN/MapReduce & HDFS clusters.

This is useful for cases where users want to quickly experiment with a real Hadoop cluster or test non-Java programs that rely on significant Hadoop functionality.

安装

  • 基于mac、jdk8

protobuf

下载

安装

  • 解压
  • 进入根目录
$ ./configure
$ make
$ make install

查看

$ protoc --version
libprotoc 2.5.0

minicluster

下载

安装

  • 解压
  • 进入根目录
$ mvn clean install -DskipTests
$ mvn package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip
$ cd /hadoop-dist/target/hadoop-3.2.1
  • 不知道为啥一直报ByteBuffer的position方法找不到的错，直接下载个二进制包，3.2.1二进制下载，解压后替换这个目录hadoop-3.2.1
  • 确保目录权限没问题
  • 第一次运行时要加上-format参数
$ bin/mapred minicluster -format
  • 如果没指定端口，通过lsof -i|grep java查看LISTEN的端口
posted @ 2020-11-05 18:22  java拌饭  阅读(279)  评论(0)  编辑  收藏  举报