flink+nacos实现参数化配置

导言

项目分为开发环境，测试环境，预发布环境以及生产环境。之前的flink都是在代码中隔开，发布环境改配置文件的，非常的不方便，而且是每个flink中写一个配置，太麻烦了。想着通过nacos来区分环境。节省垃圾代码的使用。

1.前置条件

nacos的安装，可以实现基于db的访问。

 

2.代码流程

项目结构

 

 

 

 

 

 实现思路

在resource目录下定义开发环境，测试环境，预发布环境，以及生产环境需要的配置文件

在nacos中配置需要的参数。通过指定不同环境，获得不同的参数。

 

父类pom

复制代码
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.hailong.flink</groupId>
    <artifactId>flink-learning</artifactId>
    <packaging>pom</packaging>
    <version>1.0-SNAPSHOT</version>
    <modules>
        <module>flink-nacos-learning</module>
    </modules>


    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <!-- Maven properties -->
        <maven.test.skip>false</maven.test.skip>
        <maven.javadoc.skip>true</maven.javadoc.skip>
        <!-- Compiler settings properties -->
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>

        <nacos.groupId>com.alibaba.nacos</nacos.groupId>
        <nacos.version>1.1.4</nacos.version>

        <flink.version>1.10.0</flink.version>
        <flink.groupId>org.apache.flink</flink.groupId>
    </properties>

    <!-- 所有的子项目默认依赖 -->
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <scope>test</scope>
            <version>4.12</version>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>

            <!--相关的配置包-->
            <dependency>
                <groupId>${flink.groupId}</groupId>
                <artifactId>flink-clients_2.11</artifactId>
                <version>${flink.version}</version>
                <scope>compile</scope>
            </dependency>

            <dependency>
                <groupId>${flink.groupId}</groupId>
                <artifactId>flink-core</artifactId>
                <version>${flink.version}</version>
                <scope>compile</scope>
            </dependency>

            <dependency>
                <groupId>${flink.groupId}</groupId>
                <artifactId>flink-scala_2.11</artifactId>
                <version>${flink.version}</version>
                <scope>compile</scope>
            </dependency>

            <dependency>
                <groupId>${flink.groupId}</groupId>
                <artifactId>flink-streaming-scala_2.11</artifactId>
                <version>${flink.version}</version>
                <scope>compile</scope>
            </dependency>

            <dependency>
                <groupId>${flink.groupId}</groupId>
                <artifactId>flink-streaming-java_2.11</artifactId>
                <version>${flink.version}</version>
                <scope>compile</scope>
            </dependency>


            <dependency>
                <groupId>${flink.groupId}</groupId>
                <artifactId>flink-jdbc_2.11</artifactId>
                <version>${flink.version}</version>
                <scope>compile</scope>
            </dependency>

            <!--NACOS 服务配置中心依赖包-->
            <dependency>
                <groupId>${nacos.groupId}</groupId>
                <artifactId>nacos-common</artifactId>
                <version>${nacos.version}</version>
            </dependency>
            <!--<dependency>
                <groupId>${nacos.groupId}</groupId>
                <artifactId>nacos-core</artifactId>
                <version>${nacos.version}</version>
            </dependency>-->
            <dependency>
                <groupId>${nacos.groupId}</groupId>
                <artifactId>nacos-client</artifactId>
                <version>${nacos.version}</version>
            </dependency>

            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-classic</artifactId>
                <version>1.2.3</version>
            </dependency>

            <dependency>
                <groupId>ch.qos.logback</groupId>
                <artifactId>logback-core</artifactId>
                <version>1.2.3</version>
            </dependency>

            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-core</artifactId>
                <version>2.11.2</version>
            </dependency>

            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-slf4j-impl</artifactId>
                <version>2.11.2</version>
            </dependency>




        </dependencies>
    </dependencyManagement>


</project>
复制代码

 

子类pom

复制代码
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>flink-learning</artifactId>
        <groupId>com.hailong.flink</groupId>
        <version>1.0-SNAPSHOT</version>
        <relativePath>../pom.xml</relativePath>
    </parent>
    <modelVersion>4.0.0</modelVersion>
    <artifactId>flink-nacos-learning</artifactId>





    <dependencies>

        <dependency>
            <groupId>${flink.groupId}</groupId>
            <artifactId>flink-clients_2.11</artifactId>
        </dependency>


        <dependency>
            <groupId>${flink.groupId}</groupId>
            <artifactId>flink-core</artifactId>
        </dependency>

        <dependency>
            <groupId>${flink.groupId}</groupId>
            <artifactId>flink-scala_2.11</artifactId>
        </dependency>

        <dependency>
            <groupId>${flink.groupId}</groupId>
            <artifactId>flink-streaming-java_2.11</artifactId>
        </dependency>


        <dependency>
            <groupId>${flink.groupId}</groupId>
            <artifactId>flink-jdbc_2.11</artifactId>
        </dependency>



        <dependency>
            <groupId>${nacos.groupId}</groupId>
            <artifactId>nacos-common</artifactId>
        </dependency>
        <!--<dependency>
            <groupId>${nacos.groupId}</groupId>
            <artifactId>nacos-core</artifactId>
        </dependency>-->
        <dependency>
            <groupId>com.alibaba.nacos</groupId>
            <artifactId>nacos-client</artifactId>
        </dependency>
        <!-- log -->
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
        </dependency>
    </dependencies>


</project>
复制代码

resource

sim.flink.nacos.enable=true
sim.flink.nacos.serverAddr=120.0.0.1:8848
.flink.nacos.namespace=public
sim.flink.nacos.groups=DEFAULT_GROUP
sim.flink.nacos.dataIds=test

nacos参数

 

 

 

 

 

 java代码

复制代码
package com.hailong.flink.nacos;

import com.alibaba.nacos.api.NacosFactory;
import com.alibaba.nacos.api.config.ConfigService;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.client.config.listener.impl.PropertiesListener;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;
import java.io.InputStream;
import java.io.StringReader;
import java.util.Properties;

/**
 * flink nacos support
 *
 * @author wxg
 */
public class FlinkNacosConfig {
    private static final Logger logger = LoggerFactory.getLogger(FlinkNacosConfig.class);
    private static final String ENABLE_NACOS_KEY = "sim.flink.nacos.enable";
    private static final String SERVER_ADDR_KEY = "sim.flink.nacos.serverAddr";
    private static final String NAMESPACE_KEY = "sim.flink.nacos.namespace";
    private static final String GROUPS_KEY = "sim.flink.nacos.groups";
    private static final String DATA_IDS_KEY = "sim.flink.nacos.dataIds";
    private static final String TIMEOUT_KEY = "sim.flink.nacos.timeout";

    private static final String DEFAULT_GROUP = "DEFAULT_GROUP";
    private static final String DEFAULT_DATA_ID = "test";
    private static final String DEFAULT_TIMEOUT = "3000";

    private static String SERVER_ADDR;
    private static String NAMESPACE;
    private static String[] GROUPS;
    private static String[] DATA_IDS;
    private static Integer TIMEOUT;

    private static Properties CONFIG_MAP = new Properties();
    private static Properties NACOS_CONFIG_MAP = new Properties();

    private static final String DEFAULT_CONFIG_FILE = "application.properties";

    static {
        InputStream inputStream = FlinkNacosConfig.class.getClassLoader().getResourceAsStream("application.properties");
//        int i = 0;
//        byte[] b = new byte[2];
//        while (true) {
//            try {
//                if (!((i = inputStream.read(b)) != -1)) break;
//            } catch (IOException e) {
//                e.printStackTrace();
//            }
//            String str = new String(b);
//            System.out.print(str);
//        }
        if (inputStream == null) {
            logger.info("no flink nacos config found");
        } else {
            try {
                CONFIG_MAP.load(inputStream);
                String enableNacos = CONFIG_MAP.getProperty(ENABLE_NACOS_KEY, "false");
                if (Boolean.valueOf(enableNacos)) {
                    SERVER_ADDR = CONFIG_MAP.getProperty(SERVER_ADDR_KEY);
                    NAMESPACE = CONFIG_MAP.getProperty(NAMESPACE_KEY);
                    GROUPS = CONFIG_MAP.getProperty(GROUPS_KEY, DEFAULT_GROUP).split(",");
                    DATA_IDS = CONFIG_MAP.getProperty(DATA_IDS_KEY, DEFAULT_DATA_ID).split(",");
                    TIMEOUT = Integer.valueOf(CONFIG_MAP.getProperty(TIMEOUT_KEY, DEFAULT_TIMEOUT));
                    initNacos();
                } else {
                    logger.debug("Naocs config is not enable");
                }
            } catch (IOException e) {
                e.printStackTrace();
                logger.error("", e);
            }

        }
    }

    private static void initNacos() {
        Properties properties = new Properties();
        properties.put("serverAddr", SERVER_ADDR);
        properties.put("namespace", NAMESPACE);


        String content = null;
        try {
            logger.info("loading config from nacos...");

            ConfigService configService = NacosFactory.createConfigService(properties);
            for (String group : GROUPS) {
                for (String dataId : DATA_IDS) {
                    content = configService.getConfig(dataId, group, TIMEOUT);
                    if (content != null) {
                        NACOS_CONFIG_MAP.load(new StringReader(content));
                        configService.addListener(dataId, group, new PropertiesListener() {
                            @Override
                            public void innerReceive(Properties properties) {
                                logger.info("config file {}:{} refreshed", group, dataId);
                                NACOS_CONFIG_MAP.putAll(properties);
                            }
                        });
                    }
                }
            }
            logger.info("load config from nacos success");

        } catch (NacosException | IOException e) {
            e.printStackTrace();
            logger.error("", e);
        }

    }

    public static String getConfigValue(String key) {
        return NACOS_CONFIG_MAP.getProperty(key);
    }

    public static String getConfigValue(String key, String defaultValue) {
        return NACOS_CONFIG_MAP.getProperty(key, defaultValue);
    }
}
复制代码

test

复制代码
package com.hailong.flink.nacos;

import com.alibaba.nacos.api.NacosFactory;
import com.alibaba.nacos.api.config.ConfigService;
import org.apache.flink.api.java.utils.ParameterTool;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Properties;

/**
 * Created by 袁海龙 on 2020/2/26.
 * https://www.bilibili.com/video/av90742627
 */
public class FlinkNacosConfigTest {
    private static final Logger logger = LoggerFactory.getLogger(FlinkNacosConfig.class);

    public static void main(String[] args) throws Exception {
        String configValue = FlinkNacosConfig.getConfigValue("shuaige.wdw.swf");
        String chenjie = FlinkNacosConfig.getConfigValue("chenjie1");
        String ceshi = FlinkNacosConfig.getConfigValue("shuaige.ceshi");
        logger.info("configValue:{}",configValue);

    }
}
复制代码

 

结果

 

 代码地址

https://gitee.com/chenjie950907/flink-learning

posted @   小陈子博客  阅读(1281)  评论(1)  编辑  收藏  举报
相关博文：
· SeaTa+nacos window安装
· Spring Cloud Alibaba 分布式事务 Seata 入门
· 微服务架构 | 服务注册发现中心/配置中心/消息总线 - [nacos]
· SpringCloud第一部分（认识微服务、RestTemplate、Eureka、Ribbon、Nacos、Feign）
· Nacos笔记
阅读排行：
· 被坑几百块钱后，我竟然真的恢复了删除的微信聊天记录！
· 【自荐】一款简洁、开源的在线白板工具 Drawnix
· 没有Manus邀请码？试试免邀请码的MGX或者开源的OpenManus吧
· 园子的第一款AI主题卫衣上架——"HELLO! HOW CAN I ASSIST YOU TODAY
· 无需6万激活码！GitHub神秘组织3小时极速复刻Manus，手把手教你使用OpenManus搭建本
点击右上角即可分享
微信分享提示