|NO.Z.00026|——————————|BigDataEnd|——|Hadoop&HDFS.V11|——|Hadoop.v11|日志采集案例|

一、日志采集综合案例
### --- 日志采集综合案例

~~~     [日志采集案例之需求分析]
~~~     [日志采集案例之调度功能实现]
~~~     [日志采集案例之调度功能实现]
~~~     [日志采集之采集上传功能实现]
~~~     [日志采集案例之验证及调优] 
### --- 需求分析

~~~     定时采集已滚动完毕日志文件
~~~     将待采集文件上传到临时目录
~~~     备份日志文件
二、代码实现：创建工程
### --- pom文件

    <!-- hadoop-common , hadoop-client, hadoop-hdfs-->
    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.8.2</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.9.2</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.9.2</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.9.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
三、代码实现：优化
### --- LogCollector

package com.yanqi.collect;

import java.util.Timer;

public class LogCollector {
    /*
    - 定时采集已滚动完毕日志文件
    - 将待采集文件上传到临时目录
    - 备份日志文件
     */
    public static void main(String[] args) {
        Timer timer = new Timer();
        //定时采集任务的调度
        // task：采集的业务逻辑
        //延迟时间
        //周期时间
        timer.schedule(new LogCollectorTask(), 0, 3600*1000);
    }
}

### --- LogCollectorTask

package com.yanqi.collect;

import com.yanqi.common.Constant;
import com.yanqi.singlton.PropTool2;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.File;
import java.io.FilenameFilter;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Properties;
import java.util.TimerTask;

public class LogCollectorTask extends TimerTask {

    @Override
    public void run() {

        Properties prop = null;
        try {
            prop = PropTool2.getProp();
        } catch (IOException e) {
            e.printStackTrace();
        }


        //采集的业务逻辑
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
        String todayStr = sdf.format(new Date());
        // 1 扫描指定目录，找到待上传文件,原始日志目录
        File logsDir = new File(prop.getProperty(Constant.LOGS_DIR));
        final String log_prefix = prop.getProperty(Constant.LOG_PREFIX);
        File[] uploadFiles = logsDir.listFiles(new FilenameFilter() {
            public boolean accept(File dir, String name) {
                return name.startsWith(log_prefix);
            }
        });
        //2 把待上传文件转移到临时目录
        //判断临时目录是否存在，
        File tmpFile = new File(prop.getProperty(Constant.LOG_TMP_FOLDER));
        if (!tmpFile.exists()) {
            tmpFile.mkdirs();
        }
        for (File file : uploadFiles) {
            file.renameTo(new File(tmpFile.getPath() + "/" + file.getName()));
        }

        //3 使用hdfs api上传日志文件到指定目录
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://linux121:9000");

        FileSystem fs = null;
        try {
            fs = FileSystem.get(conf);
            //判断hdfs目标路径是否存在，备份目录是否存在
            Path path = new Path(prop.getProperty(Constant.HDFS_TARGET_FOLDER) + todayStr);
            if (!fs.exists(path)) {
                fs.mkdirs(path);
            }
            File bakFolder = new File(prop.getProperty(Constant.BAK_FOLDER) + todayStr);
            if (!bakFolder.exists()) {
                bakFolder.mkdirs();
            }
            File[] files = tmpFile.listFiles();
            for (File file : files) {
                //按照日期分门别列存放
                fs.copyFromLocalFile(new Path(file.getPath()), new Path(prop.getProperty(Constant.HDFS_TARGET_FOLDER) + todayStr + "/" + file.getName()));
                //4 上传后的文件转移到备份目录
                file.renameTo(new File(bakFolder.getPath() + "/" + file.getName()));
            }
        } catch (IOException e) {
            e.printStackTrace();
        }


    }
}
### --- Common

package com.yanqi.common;

public class Constant {
    public static  final  String LOGS_DIR="LOGS.DIR";
    public static  final  String LOG_PREFIX="LOG.PREFIX";
    public static  final  String LOG_TMP_FOLDER="LOG.TMP.FOLDER";
    public static  final  String HDFS_TARGET_FOLDER="HDFS.TARGET.FOLDER";
    public static  final  String BAK_FOLDER="BAK.FOLDER";

}
### --- PropTool

package com.yanqi.singlton;

import com.yanqi.collect.LogCollectorTask;

import java.io.IOException;
import java.util.Properties;

public class PropTool {
    //类加载时初始化执行一次即可
    //使用静态代码块实现 饿汉式加载
    private static Properties prop=null;

    static {
        prop=new Properties();
        try {
            prop.load(LogCollectorTask.class.getClassLoader().getResourceAsStream("collector.properties"));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public static  Properties getProp(){
        return prop;
    }

}
### --- PropTool2

package com.yanqi.singlton;

import com.yanqi.collect.LogCollectorTask;

import java.io.IOException;
import java.util.Properties;

public class PropTool2 {

    //volatile关键字是java中禁止指令重排序的关键字，保证有序性和可见性
    private static volatile Properties prop=null;



    //出现线程安全问题
    public static  Properties getProp() throws IOException {
        if(prop ==null){
            synchronized ("lock"){
                if(prop ==null){
                    prop=new Properties();
                    prop.load(LogCollectorTask.class.getClassLoader()
                            .getResourceAsStream("collector.properties"));
                }
            }
        }

        return prop;
    }

}
四、打印输出
### --- 打印输出
~~~     因为它是一个定时任务；会一直运行，不会退出

D:\JAVA\jdk1.8.0_231\bin\java.exe "-javaagent:D:\IntelliJIDEA\IntelliJ IDEA 2019.3.3\lib\idea_rt.jar=55392:D:\IntelliJIDEA\IntelliJ IDEA 2019.3.3\bin" -Dfile.encoding=UTF-8 -classpath D:\JAVA\jdk1.8.0_231\jre\lib\charsets.jar;D:\JAVA\jdk1.8.0_231\jre\lib\deploy.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\access-bridge-64.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\cldrdata.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\dnsns.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\jaccess.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\jfxrt.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\localedata.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\nashorn.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\sunec.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\sunjce_provider.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\sunmscapi.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\sunpkcs11.jar;D:\JAVA\jdk1.8.0_231\jre\lib\ext\zipfs.jar;D:\JAVA\jdk1.8.0_231\jre\lib\javaws.jar;D:\JAVA\jdk1.8.0_231\jre\lib\jce.jar;D:\JAVA\jdk1.8.0_231\jre\lib\jfr.jar;D:\JAVA\jdk1.8.0_231\jre\lib\jfxswt.jar;D:\JAVA\jdk1.8.0_231\jre\lib\jsse.jar;D:\JAVA\jdk1.8.0_231\jre\lib\management-agent.jar;D:\JAVA\jdk1.8.0_231\jre\lib\plugin.jar;D:\JAVA\jdk1.8.0_231\jre\lib\resources.jar;D:\JAVA\jdk1.8.0_231\jre\lib\rt.jar;E:\NO.Z.10000——javaproject\NO.Z.00002.Hadoop\collect_log\target\classes;C:\Users\Administrator\.m2\repository\junit\junit\4.13.2\junit-4.13.2.jar;C:\Users\Administrator\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Users\Administrator\.m2\repository\org\apache\logging\log4j\log4j-core\2.8.2\log4j-core-2.8.2.jar;C:\Users\Administrator\.m2\repository\org\apache\logging\log4j\log4j-api\2.8.2\log4j-api-2.8.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-common\2.9.2\hadoop-common-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-annotations\2.9.2\hadoop-annotations-2.9.2.jar;C:\Users\Administrator\.m2\repository\com\google\guava\guava\11.0.2\guava-11.0.2.jar;C:\Users\Administrator\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-math3\3.1.1\commons-math3-3.1.1.jar;C:\Users\Administrator\.m2\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;C:\Users\Administrator\.m2\repository\org\apache\httpcomponents\httpclient\4.5.2\httpclient-4.5.2.jar;C:\Users\Administrator\.m2\repository\org\apache\httpcomponents\httpcore\4.4.4\httpcore-4.4.4.jar;C:\Users\Administrator\.m2\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;C:\Users\Administrator\.m2\repository\commons-io\commons-io\2.4\commons-io-2.4.jar;C:\Users\Administrator\.m2\repository\commons-net\commons-net\3.1\commons-net-3.1.jar;C:\Users\Administrator\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Users\Administrator\.m2\repository\javax\servlet\servlet-api\2.5\servlet-api-2.5.jar;C:\Users\Administrator\.m2\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;C:\Users\Administrator\.m2\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;C:\Users\Administrator\.m2\repository\org\mortbay\jetty\jetty-sslengine\6.1.26\jetty-sslengine-6.1.26.jar;C:\Users\Administrator\.m2\repository\javax\servlet\jsp\jsp-api\2.1\jsp-api-2.1.jar;C:\Users\Administrator\.m2\repository\com\sun\jersey\jersey-core\1.9\jersey-core-1.9.jar;C:\Users\Administrator\.m2\repository\com\sun\jersey\jersey-json\1.9\jersey-json-1.9.jar;C:\Users\Administrator\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;C:\Users\Administrator\.m2\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;C:\Users\Administrator\.m2\repository\org\codehaus\jackson\jackson-jaxrs\1.8.3\jackson-jaxrs-1.8.3.jar;C:\Users\Administrator\.m2\repository\org\codehaus\jackson\jackson-xc\1.8.3\jackson-xc-1.8.3.jar;C:\Users\Administrator\.m2\repository\com\sun\jersey\jersey-server\1.9\jersey-server-1.9.jar;C:\Users\Administrator\.m2\repository\asm\asm\3.1\asm-3.1.jar;C:\Users\Administrator\.m2\repository\commons-logging\commons-logging\1.1.3\commons-logging-1.1.3.jar;C:\Users\Administrator\.m2\repository\log4j\log4j\1.2.17\log4j-1.2.17.jar;C:\Users\Administrator\.m2\repository\net\java\dev\jets3t\jets3t\0.9.0\jets3t-0.9.0.jar;C:\Users\Administrator\.m2\repository\com\jamesmurty\utils\java-xmlbuilder\0.4\java-xmlbuilder-0.4.jar;C:\Users\Administrator\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;C:\Users\Administrator\.m2\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;C:\Users\Administrator\.m2\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;C:\Users\Administrator\.m2\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;C:\Users\Administrator\.m2\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar;C:\Users\Administrator\.m2\repository\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar;C:\Users\Administrator\.m2\repository\org\slf4j\slf4j-log4j12\1.7.25\slf4j-log4j12-1.7.25.jar;C:\Users\Administrator\.m2\repository\org\codehaus\jackson\jackson-core-asl\1.9.13\jackson-core-asl-1.9.13.jar;C:\Users\Administrator\.m2\repository\org\codehaus\jackson\jackson-mapper-asl\1.9.13\jackson-mapper-asl-1.9.13.jar;C:\Users\Administrator\.m2\repository\org\apache\avro\avro\1.7.7\avro-1.7.7.jar;C:\Users\Administrator\.m2\repository\com\thoughtworks\paranamer\paranamer\2.3\paranamer-2.3.jar;C:\Users\Administrator\.m2\repository\org\xerial\snappy\snappy-java\1.0.5\snappy-java-1.0.5.jar;C:\Users\Administrator\.m2\repository\com\google\protobuf\protobuf-java\2.5.0\protobuf-java-2.5.0.jar;C:\Users\Administrator\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-auth\2.9.2\hadoop-auth-2.9.2.jar;C:\Users\Administrator\.m2\repository\com\nimbusds\nimbus-jose-jwt\4.41.1\nimbus-jose-jwt-4.41.1.jar;C:\Users\Administrator\.m2\repository\com\github\stephenc\jcip\jcip-annotations\1.0-1\jcip-annotations-1.0-1.jar;C:\Users\Administrator\.m2\repository\net\minidev\json-smart\2.3\json-smart-2.3.jar;C:\Users\Administrator\.m2\repository\net\minidev\accessors-smart\1.2\accessors-smart-1.2.jar;C:\Users\Administrator\.m2\repository\org\ow2\asm\asm\5.0.4\asm-5.0.4.jar;C:\Users\Administrator\.m2\repository\org\apache\directory\server\apacheds-kerberos-codec\2.0.0-M15\apacheds-kerberos-codec-2.0.0-M15.jar;C:\Users\Administrator\.m2\repository\org\apache\directory\server\apacheds-i18n\2.0.0-M15\apacheds-i18n-2.0.0-M15.jar;C:\Users\Administrator\.m2\repository\org\apache\directory\api\api-asn1-api\1.0.0-M20\api-asn1-api-1.0.0-M20.jar;C:\Users\Administrator\.m2\repository\org\apache\directory\api\api-util\1.0.0-M20\api-util-1.0.0-M20.jar;C:\Users\Administrator\.m2\repository\org\apache\curator\curator-framework\2.7.1\curator-framework-2.7.1.jar;C:\Users\Administrator\.m2\repository\com\jcraft\jsch\0.1.54\jsch-0.1.54.jar;C:\Users\Administrator\.m2\repository\org\apache\curator\curator-client\2.7.1\curator-client-2.7.1.jar;C:\Users\Administrator\.m2\repository\org\apache\curator\curator-recipes\2.7.1\curator-recipes-2.7.1.jar;C:\Users\Administrator\.m2\repository\com\google\code\findbugs\jsr305\3.0.0\jsr305-3.0.0.jar;C:\Users\Administrator\.m2\repository\org\apache\htrace\htrace-core4\4.1.0-incubating\htrace-core4-4.1.0-incubating.jar;C:\Users\Administrator\.m2\repository\org\apache\zookeeper\zookeeper\3.4.6\zookeeper-3.4.6.jar;C:\Users\Administrator\.m2\repository\org\apache\commons\commons-compress\1.4.1\commons-compress-1.4.1.jar;C:\Users\Administrator\.m2\repository\org\tukaani\xz\1.0\xz-1.0.jar;C:\Users\Administrator\.m2\repository\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;C:\Users\Administrator\.m2\repository\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-client\2.9.2\hadoop-client-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-hdfs-client\2.9.2\hadoop-hdfs-client-2.9.2.jar;C:\Users\Administrator\.m2\repository\com\squareup\okhttp\okhttp\2.7.5\okhttp-2.7.5.jar;C:\Users\Administrator\.m2\repository\com\squareup\okio\okio\1.6.0\okio-1.6.0.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-app\2.9.2\hadoop-mapreduce-client-app-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-common\2.9.2\hadoop-mapreduce-client-common-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-shuffle\2.9.2\hadoop-mapreduce-client-shuffle-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-yarn-server-common\2.9.2\hadoop-yarn-server-common-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-yarn-registry\2.9.2\hadoop-yarn-registry-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\geronimo\specs\geronimo-jcache_1.0_spec\1.0-alpha-1\geronimo-jcache_1.0_spec-1.0-alpha-1.jar;C:\Users\Administrator\.m2\repository\org\ehcache\ehcache\3.3.1\ehcache-3.3.1.jar;C:\Users\Administrator\.m2\repository\com\zaxxer\HikariCP-java7\2.4.12\HikariCP-java7-2.4.12.jar;C:\Users\Administrator\.m2\repository\com\microsoft\sqlserver\mssql-jdbc\6.2.1.jre7\mssql-jdbc-6.2.1.jre7.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-yarn-api\2.9.2\hadoop-yarn-api-2.9.2.jar;C:\Users\Administrator\.m2\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;C:\Users\Administrator\.m2\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;C:\Users\Administrator\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-core\2.9.2\hadoop-mapreduce-client-core-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-yarn-client\2.9.2\hadoop-yarn-client-2.9.2.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-yarn-common\2.9.2\hadoop-yarn-common-2.9.2.jar;C:\Users\Administrator\.m2\repository\com\sun\jersey\jersey-client\1.9\jersey-client-1.9.jar;C:\Users\Administrator\.m2\repository\org\apache\hadoop\hadoop-mapreduce-client-jobclient\2.9.2\hadoop-mapreduce-client-jobclient-2.9.2.jar;C:\Users\Administrator\.m2\repository\io\netty\netty\3.6.2.Final\netty-3.6.2.Final.jar;C:\Users\Administrator\.m2\repository\org\fusesource\leveldbjni\leveldbjni-all\1.8\leveldbjni-all-1.8.jar com.yanqi.collect.LogCollector
### --- 验证结果

查看HDFS_UI上是否采集成功
查看备份日志是否采集
 
 
 
 
 
 
 
 
 
Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart
                                                                                                                                                   ——W.S.Landor
 

 

posted on   yanqi_vip  阅读(3)  评论(0)  编辑  收藏  举报

相关博文：
· |NO.Z.00015|——————————|BigDataEnd|——|Hadoop&PB级数仓.V07|——|PB数仓.v07|会员活跃度分析|自定义拦截器实现&测试|
· |NO.Z.00022|——————————|BigDataEnd|——|Hadoop&HDFS.V07|——|Hadoop.v07|HDFS读写解析|
· Java日志技术快速入门
· Spring-微服务-全-
· 又双叒叕来更新啦！！Hadoop——基础篇
阅读排行：
· 无需6万激活码！GitHub神秘组织3小时极速复刻Manus，手把手教你使用OpenManus搭建本
· Manus爆火，是硬核还是营销？
· 终于写完轮子一部分：tcp代理 了，记录一下
· 别再用vector＜bool＞了！Google高级工程师：这可能是STL最大的设计失误
· 单元测试从入门到精通
< 2025年3月 >
日 一 二 三 四 五 六
23 24 25 26 27 28 1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31 1 2 3 4 5

导航

统计

点击右上角即可分享
微信分享提示