Flink - TypeInformation - fxjwind

Flink - TypeInformation

Flink 自己创建一套独立的类型系统，

参考， https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/types_serialization.html

为何要自己搞一套，而不像其他的平台一样让编程语言或serialization framework来天然做掉？

Flink tries to know as much information about what types enter and leave user functions as possible. This stands in contrast to the approach to just assuming nothing and letting the programming language and serialization framework handle all types dynamically.

To allow using POJOs and grouping/joining them by referring to field names, Flink needs the type information to make checks (for typos and type compatibility) before the job is executed.
The more we know, the better serialization and data layout schemes the compiler/optimizer can develop. That is quite important for the memory usage paradigm in Flink (work on serialized data inside/outside the heap and make serialization very cheap).
For the upcoming logical programs (see roadmap draft) we need this to know the “schema” of functions.
Finally, it also spares users having to worry about serialization frameworks and having to register types at those frameworks.

Note. POJOs是什么？Plain Old Java Object（简单的Java对象）,即轻量java对象的花式叫法

主要的理由，

第一是要做类型检查，Flink支持比较灵活的基于field的join或group，需要先检查这个field是否可以作为key，或这个field是否可以做join或group

第二是性能优化，便于使用更好的序列化和数据的layout

Flink主要定义如下几种类型，

Internally, Flink makes the following distinctions between types:

Basic types: All Java primitives and their boxed form, plus void, String, and Date.
Primitive arrays and Object arrays
Composite types
- Flink Java Tuples (part of the Flink Java API)
- Scala case classes (including Scala tuples)
- POJOs: classes that follow a certain bean-like pattern
Scala auxiliary types (Option, Either, Lists, Maps, …)
Generic types: These will not be serialized by Flink itself, but by Kryo.

基本类型

数组（包含Primitive数组和对象数组）

组合类型，包含Flink Tuples， Scala case classes，和POJOS

Scala增加的辅助类型

泛型，这个Flink不处理，而是用kryo

这里尤其需要注意POJOs，因为它的field是可以直接用name引用的，非常方便

dataSet.join(another).where("name").equalTo("personName")

那么对于Flink的准确的POJO的定义是什么？

The class is public and standalone (no non-static inner class)
The class has a public no-argument constructor
All fields in the class (and all superclasses) are either public or or have a public getter and a setter method that follows the Java beans naming conventions for getters and setters.

很简单，只要满足上面的规范，就支持“by-name” field referencing

文档里面还描述了在Scala和Java API中的类型问题，

对于Scala，用manifest或typetag来解决了泛型擦除的问题，所以主要是Flink用macro实现了TypeInformation，便于使用

对于Java，就必须要解决泛型擦除的问题，

DataSet<SomeType> result = dataSet
    .map(new MyGenericNonInferrableFunction<Long, SomeType>())
        .returns(SomeType.class);

比如，上面的日志，如果不加最后的hints，在runtime其实是无法知道SomeType是什么的，在编译的时候已经被erase成Object

所以Flink使用returns原语来增加hints

来看看源码，

基类为，

package org.apache.flink.api.common.typeinfo;

TypeInformation

目的， This type information class acts as the tool

to generate serializers and comparators

to perform semantic checks such as whether the fields that are uses as join/grouping keys actually exist.

bridges between the programming languages object model and a logical flat schema

前两个目的好理解，

最后一个目的，搞清两个概念，

arity，the number of fields it contains directly

total number of fields，number of fields in the entire schema of this type, including nested types

举个例子，

* public class InnerType {
*   public int id;
*   public String text;
* }
* 
* public class OuterType {
*   public long timestamp;
*   public InnerType nestedType;
* }

对于Inner type，arity和fields都是2

但对于OuterType，虽然arity是2，但fields是3，因为要把嵌套类型的fields也算上，这就是把编程语言对象模型转换为flat的逻辑schema

如何算fields的规则如下：

*   <li>Basic types are indivisible and are considered a single field.</li>
*   <li>Arrays and collections are one field</li>
*   <li>Tuples and case classes represent as many fields as the class has fields</li>

IntegerTypeInfo

用这个作为例子，分析一下

public class IntegerTypeInfo<T> extends NumericTypeInfo<T>

public abstract class NumericTypeInfo<T> extends BasicTypeInfo<T>

public class BasicTypeInfo<T> extends TypeInformation<T> implements AtomicType<T>

可以看到Integer最终继承到BasicType，BasicType除了继承TypeInformation还实现AtomicType接口，

public interface AtomicType<T> {   

   TypeComparator<T> createComparator(boolean sortOrderAscending, ExecutionConfig executionConfig);
}

* An atomic type is a type that is treated as one indivisible unit and where the entire type acts
* as a key.
* In contrast to atomic types are composite types, where the type information is aware of the individual
* fields and individual fields may be used as a key.

atomic类型就是不可分的类型，不像composite类型还包含其他的field，所以atomic本身整个作为key，基本类型如int肯定是属于atomic类型的

在BasicTypeInfo中定义了所有基本类型的TypeInfo，

    public static final BasicTypeInfo<String> STRING_TYPE_INFO = new BasicTypeInfo<String>(String.class, new Class<?>[]{}, StringSerializer.INSTANCE, StringComparator.class);
    public static final BasicTypeInfo<Boolean> BOOLEAN_TYPE_INFO = new BasicTypeInfo<Boolean>(Boolean.class, new Class<?>[]{}, BooleanSerializer.INSTANCE, BooleanComparator.class);
    public static final BasicTypeInfo<Byte> BYTE_TYPE_INFO = new IntegerTypeInfo<Byte>(Byte.class, new Class<?>[]{Short.class, Integer.class, Long.class, Float.class, Double.class, Character.class}, ByteSerializer.INSTANCE, ByteComparator.class);
    public static final BasicTypeInfo<Short> SHORT_TYPE_INFO = new IntegerTypeInfo<Short>(Short.class, new Class<?>[]{Integer.class, Long.class, Float.class, Double.class, Character.class}, ShortSerializer.INSTANCE, ShortComparator.class);
    public static final BasicTypeInfo<Integer> INT_TYPE_INFO = new IntegerTypeInfo<Integer>(Integer.class, new Class<?>[]{Long.class, Float.class, Double.class, Character.class}, IntSerializer.INSTANCE, IntComparator.class);
    public static final BasicTypeInfo<Long> LONG_TYPE_INFO = new IntegerTypeInfo<Long>(Long.class, new Class<?>[]{Float.class, Double.class, Character.class}, LongSerializer.INSTANCE, LongComparator.class);
    public static final BasicTypeInfo<Float> FLOAT_TYPE_INFO = new FractionalTypeInfo<Float>(Float.class, new Class<?>[]{Double.class}, FloatSerializer.INSTANCE, FloatComparator.class);
    public static final BasicTypeInfo<Double> DOUBLE_TYPE_INFO = new FractionalTypeInfo<Double>(Double.class, new Class<?>[]{}, DoubleSerializer.INSTANCE, DoubleComparator.class);
    public static final BasicTypeInfo<Character> CHAR_TYPE_INFO = new BasicTypeInfo<Character>(Character.class, new Class<?>[]{}, CharSerializer.INSTANCE, CharComparator.class);
    public static final BasicTypeInfo<Date> DATE_TYPE_INFO = new BasicTypeInfo<Date>(Date.class, new Class<?>[]{}, DateSerializer.INSTANCE, DateComparator.class);
    public static final BasicTypeInfo<Void> VOID_TYPE_INFO = new BasicTypeInfo<Void>(Void.class, new Class<?>[]{}, VoidSerializer.INSTANCE, null);

可以看到Byte，short，int，long都用的是IntegerTypeInfo

创建的4个参数分别为，以INT_TYPE_INFO为例，

class对象，Integer.class

可能被cast成的类型，所以对于Integer，被cast成long，float，double，character都是可以的

Serializer对象

Comparator对象

可以看到flink重新封装了所有对象的Serializer和Comparator

我们看下LongSerializer，

@Override
public void serialize(Long record, DataOutputView target) throws IOException {
    target.writeLong(record.longValue());
}

很高效的，对于Long，只会序列化真正的longValue，而不会存多余的东西

而NumericTypeInfo，只是一种特殊的BasicTypeInfo

    private static final Set<Class<?>> numericalTypes = Sets.<Class<?>>newHashSet(
            Integer.class,
            Long.class,
            Double.class,
            Byte.class,
            Short.class,
            Float.class,
            Character.class
    );

只有上面这几种class对象，才被认为是NumericTypeInfo

而IntegerTypeInfo，只是范围的进一步缩小，

    private static final Set<Class<?>> integerTypes = Sets.<Class<?>>newHashSet(
            Integer.class,
            Long.class,
            Byte.class,
            Short.class,
            Character.class
    );

除了上面的AtomicType，还有如array的typeinfo

比如，BasicArrayTypeInfo

posted on 2017-01-17 19:09 fxjwind 阅读(1866) 评论(0) 编辑收藏举报

刷新页面返回顶部

fxjwind

Flink - TypeInformation

导航

公告