|NO.Z.00042|——————————|BigDataEnd|——|Hadoop&Flink.V09|——|Flink.v09|Flink State|状态原理|原理剖析|状态存储|开启checkpoint|

一、开启 checkpoint

### --- 开启checkpoint

~~~     开启 checkpoint 后，
~~~     state backend 管理的 taskmanager 上的状态数据才会被定期备份到jobmanager 或 外部存储，
~~~     这些状态数据在作业失败恢复时会用到。我们可以通过以下代码开启和配置checkpoint：

二、编程实现方法类

### --- 编程实现

    StreamExecutionEnvironment env =
    StreamExecutionEnvironment.getExecutionEnvironment();

    //env.getConfig().disableSysoutLogging();
    //每 30 秒触发一次 checkpoint，checkpoint 时间应该远小于（该值 +MinPauseBetweenCheckpoints），否则程序会一直做checkpoint，影响数据处理速度env.enableCheckpointing(30000); // create a checkpoint every 30 seconds
    // set mode to exactly-once (this is the default)
    // flink 框架内保证 EXACTLY_ONCE
    
    env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
    // make sure 30 s of progress happen between checkpoints
    // 两个 checkpoints之间最少有 30s 间隔（上一个checkpoint完成到下一个checkpoint开始，默认为0，这里建议设置为非0值）env.getCheckpointConfig().setMinPauseBetweenCheckpoints(30000);
    // checkpoints have to complete within one minute, or are discarded
    // checkpoint 超时时间（默认 600 s）

    env.getCheckpointConfig().setCheckpointTimeout(600000);
    // allow only one checkpoint to be in progress at the same time
    // 同时只有一个checkpoint运行（默认）

    env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
    // enable externalized checkpoints which are retained after job cancellation
    // 取消作业时是否保留 checkpoint (默认不保留)

    env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
    
    // checkpoint失败时 task 是否失败( 默认 true， checkpoint失败时，task会失败)
    env.getCheckpointConfig().setFailOnCheckpointingErrors(true);

    // 对 FsStateBackend 刷出去的文件进行文件压缩，减小 checkpoint 体积
    env.getConfig().setUseSnapshotCompression(true);

### --- FsStateBackend 和 RocksDBStateBackend checkpoint 完成后最终保存到下面的目录：

hdfs:///your/checkpoint/path/{JOB_ID}/chk-{CHECKPOINT_ID}/

### --- JOB_ID 是应用的唯一 ID，CHECKPOINT_ID 是每次 checkpoint 时自增的数字 
~~~     ID我们可以从备份的 checkpoint 数据恢复当时的作业状态：

flink-1x.x/bin/flink run -s hdfs:///your/checkpoint/path/{JOB_ID}/chk-{CHECKPOINT_ID}/ path/to//your/jar

### --- 我们可以实现 CheckpointedFunction 方法，在程序初始化或者 checkpoint 时修改状态：

public class StatefulProcess extends KeyedProcessFunction<String, KeyValue, KeyValue> implements CheckpointedFunction {
    ValueState<Integer> processedInt;

    @Override
    public void open(Configuration parameters) throws Exception {
        super.open(parameters);
    }
    
    @Override
    public void processElement(KeyValue keyValue, Context context, Collector<KeyValue> collector) throws Exception {
        try{
            Integer a = Integer.parseInt(keyValue.getValue());
            processedInt.update(a);
            collector.collect(keyValue);
        }catch(Exception e){
            e.printStackTrace();
        }
    }

    @Override
    public void initializeState(FunctionInitializationContext functionInitializationContext) throws Exception {
        processedInt = functionInitializationContext.getKeyedStateStore().getState(new ValueStateDescriptor<>("processedInt", Integer.class));
        if(functionInitializationContext.isRestored()){
            //Apply logic to restore the data
        }
    }
    
    @Override
    public void snapshotState(FunctionSnapshotContext functionSnapshotContext)
    throws Exception {
        processedInt.clear();
    }
}     

---

 

 

 

 

 

 

 

 

 

---

Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart

                                                                                                                                                   ——W.S.Landor

---