Export
语法文件
export_stmt ::= KW_EXPORT KW_TABLE base_table_ref:tblRef where_clause:whereExpr KW_TO STRING_LITERAL:path opt_properties:properties opt_broker:broker {: RESULT = new ExportStmt(tblRef, whereExpr, path, properties, broker); :} ;
export有类似于Load的语法，export语法分为五段：
● KW_EXPORT KW_TABLE base_table_ref:tblRef
● where_clause
● KW_TO STRING_LITERAL:path
● opt_properties - PROPERTIES (xxx)
● opt_broker - WITH xxx可以是Broker或者是HDFS,S3，最终都转成BrokerDesc
EXPORT TABLE test_export TO "hdfs://ctyunns/tmp/doris/" WITH BROKER "hdfs_broker"( "hadoop.security.authentication"="kerberos", "kerberos_principal"="hdfs@BIGDATA.CHINATELECOM.CN", "kerberos_keytab"="/etc/security/keytabs/hdfs_export.keytab", 'dfs.nameservices'='ctyunns', 'dfs.ha.namenodes.ctyunns'='nn1,nn2', 'dfs.namenode.rpc-address.ctyunns.nn1'='nm-bigdata-030017237.ctc.local:54310', 'dfs.namenode.rpc-address.ctyunns.nn2'='nm-bigdata-030017238.ctc.local:54310', 'dfs.client.failover.proxy.provider.ctyunns'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider' );
Repository
语法文件
Repository也有类似的语法，但细看实则不同
CREATE REPOSITORY `default_repo` WITH BROKER `hdfs_broker` ON LOCATION "hdfs://ctyunns/dorisRepo" PROPERTIES ( 'dfs.nameservices'='ctyunns', 'dfs.ha.namenodes.ctyunns'='nn1,nn2', 'dfs.namenode.rpc-address.ctyunns.nn1'='nm-bigdata-030017237.ctc.local:54310', 'dfs.namenode.rpc-address.ctyunns.nn2'='nm-bigdata-030017238.ctc.local:54310', 'dfs.client.failover.proxy.provider.ctyunns'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider', "hadoop.security.authentication"="kerberos", "kerberos_principal"="hdfs@BIGDATA.CHINATELECOM.CN", "kerberos_keytab"="/etc/security/keytabs/hdfs_export.keytab" );
XXX on LOCATION xxx PROPERTIES是一个整体组成StorageBackend
| KW_CREATE opt_read_only:isReadOnly KW_REPOSITORY ident:repoName KW_WITH storage_backend:storage {: RESULT = new CreateRepositoryStmt(isReadOnly, repoName, storage); :} storage_backend ::= | KW_BROKER ident:brokerName KW_ON KW_LOCATION STRING_LITERAL:location opt_properties:properties {: RESULT = new StorageBackend(brokerName, location, StorageBackend.StorageType.BROKER, properties); :} | KW_S3 KW_ON KW_LOCATION STRING_LITERAL:location opt_properties:properties {: RESULT = new StorageBackend("", location, StorageBackend.StorageType.S3, properties); :} | KW_HDFS KW_ON KW_LOCATION STRING_LITERAL:location opt_properties:properties {: RESULT = new StorageBackend("", location, StorageBackend.StorageType.HDFS, properties); :} | KW_LOCAL KW_ON KW_LOCATION STRING_LITERAL:location opt_properties:properties {: RESULT = new StorageBackend("", location, StorageBackend.StorageType.LOCAL, properties); :} ;
StorageBackend -> BlobStorage
BrokerDesc -> BlobStorage
Resource
语法文件
Resource可以理解为一系列配置的集合
CREATE EXTERNAL RESOURCE "spark0" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.jars" = "xxx.jar,yyy.jar", "spark.files" = "/tmp/aaa,/tmp/bbb", "spark.executor.memory" = "1g", "spark.yarn.queue" = "queue0", "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/doris", "broker" = "broker0", "broker.username" = "user0", "broker.password" = "password0" );
语法文件如下：
KW_CREATE opt_external:isExternal KW_RESOURCE opt_if_not_exists:ifNotExists ident_or_text:resourceName opt_properties:properties {: RESULT = new CreateResourceStmt(isExternal, ifNotExists, resourceName, properties); :}
创建过程
创建过程如下：
● Resource.fromStmt
○ 创建各种不同类型的Resource子类。如SparkResource，JdbcResource
○ 调用Resource.setProperties将语法定义时的properties设置进对应的Resource子类中
● createResource - 将Resource置入内存存储
支持的Resource类型见Resource.getResourceInstance
