flink Standalone Cluster

Requirements

Software Requirements

Flink runs on all UNIX-like environments, e.g. Linux, Mac OS X, and Cygwin (for Windows) and expects the cluster to consist of one master node and one or more worker nodes. Before you start to setup the system, make sure you have the following software installed on each node:

  • Java 1.8.x or higher,
  • ssh (sshd must be running to use the Flink scripts that manage remote components)

If your cluster does not fulfill these software requirements you will need to install/upgrade it.

Having passwordless SSH and the same directory structure on all your cluster nodes will allow you to use our scripts to control everything.

ssh免密登录：

在源机器执行

ssh-keygen  //一路回车，生成公钥，位置~/.ssh/id_rsa.pub

ssh-copy-id -i ~/.ssh/id_rsa.pub root@目的机器  //将源机器生成的公钥拷贝到目的机器的~/.ssh/authorized_keys

或者 手动将源机器生成的公钥拷贝到目的机器的~/.ssh/authorized_keys。注意：不要有换行

完成！

 Back to top

JAVA_HOME Configuration 

系统环境配置了JAVA_HOME即可，无需如下操作

Flink requires the JAVA_HOME environment variable to be set on the master and all worker nodes and point to the directory of your Java installation.

You can set this variable in conf/flink-conf.yaml via the env.java.home key.

 Back to top

Go to the downloads page and get the ready-to-run package. Make sure to pick the Flink package matching your Hadoop version. If you don’t plan to use Hadoop, pick any version.

After downloading the latest release, copy the archive to your master node and extract it:

tar xzf flink-*.tgz
cd flink-*

After having extracted the system files, you need to configure Flink for the cluster by editing conf/flink-conf.yaml.

Set the jobmanager.rpc.address key to point to your master node. You should also define the maximum amount of main memory the JVM is allowed to allocate on each node by setting the jobmanager.heap.mb and taskmanager.heap.mb keys.

These values are given in MB. If some worker nodes have more main memory which you want to allocate to the Flink system you can overwrite the default value by setting the environment variable FLINK_TM_HEAP on those specific nodes.

Finally, you must provide a list of all nodes in your cluster which shall be used as worker nodes. Therefore, similar to the HDFS configuration, edit the file conf/slaves and enter the IP/host name of each worker node. Each worker node will later run a TaskManager.

The following example illustrates the setup with three nodes (with IP addresses from 10.0.0.1 to 10.0.0.3 and hostnames master, worker1, worker2) and shows the contents of the configuration files (which need to be accessible at the same path on all machines):

/path/to/flink/conf/flink-conf.yaml
jobmanager.rpc.address: 10.0.0.1
注意：需要在所有需要启动taskmanager的机器进行如上配置

 /path/to/flink/conf/slaves

10.0.0.2
10.0.0.3

 

The Flink directory must be available on every worker under the same path. You can use a shared NFS directory, or copy the entire Flink directory to every worker node.

Please see the configuration page for details and additional configuration options.

In particular,

  • the amount of available memory per JobManager (jobmanager.heap.mb),
  • the amount of available memory per TaskManager (taskmanager.heap.mb),
  • the number of available CPUs per machine (taskmanager.numberOfTaskSlots),
  • the total number of CPUs in the cluster (parallelism.default) and
  • the temporary directories (taskmanager.tmp.dirs)

are very important configuration values.

 Back to top

The following script starts a JobManager on the local node and connects via SSH to all worker nodes listed in the slaves file to start the TaskManager on each node. Now your Flink system is up and running. The JobManager running on the local node will now accept jobs at the configured RPC port.

Assuming that you are on the master node and inside the Flink directory:

bin/start-cluster.sh

To stop Flink, there is also a stop-cluster.sh script.

 Back to top

Adding JobManager/TaskManager Instances to a Cluster

You can add both JobManager and TaskManager instances to your running cluster with the bin/jobmanager.sh and bin/taskmanager.shscripts.

Adding a JobManager

bin/jobmanager.sh ((start|start-foreground) cluster)|stop|stop-all

Adding a TaskManager

bin/taskmanager.sh start|start-foreground|stop|stop-all

Make sure to call these scripts on the hosts on which you want to start/stop the respective instance.

posted @ 2018-09-26 17:59  大数据从业者FelixZh  阅读(370)  评论(0)  编辑  收藏  举报