nifi练习项目，聚合多flowfile的attribute，以单个邮件发放通知
nifi练习项目，聚合多flowfile的attribute，以单个邮件发放通知
习惯看代码的直接访问
https://github.com/cclient/nifi-email-bundle
Nifi SendEmail Processor
flowfile0 mail0(flowfile0)
flowfile1 PutMail(1:1) mail1(flowfile1)
flowfile2 mail2(flowfile2)
flowfile0
flowfile1 SendMail(n:1) mail(flowfile0,flowfile1,flowfile2)
flowfile2
需求
- nifi生态和自定义processor下通常有failure stream的处理,为了及时发现failure需统一发送通知邮件
- upstream 里会堆积多项flowfile,1:1发送邮件则通知过多，因此n:1 把多项flowfile合并在一个邮件里发送(只发送flowfile的attribute信息)
- 邮件内容为table,每列是一个attribute,attribule取并集，每行是一个flowfile的所有attribute
基于官方的PutMail 开发
可以组合多个官方processor实现该功能，例如在前置Processor完成拼接email msg内容，直接通过putMail发送邮件
通过简单定制实现也是为了熟悉nifi的开发/测试规范
邮件为html body如下
| a0 | a1 | a2 | a3 | filename | path | uuid | entryDate | size |
|---|---|---|---|---|---|---|---|---|
| v0 | v1 | v2 | v3 | 184775642434972.mockFlowFile | target | 1bdaef1f-960b-48c5-a739-a881f92d1466 | Thu May 21 15:37:00 CST 2020 | |
| v0 | v1 | v2 | v3 | 184323287499017.mockFlowFile | target | 45caab10-50c4-4727-ac42-bc345670cf85 | Thu May 21 15:37:00 CST 2020 | Thu Jan 01 08:00:00 CST 1970 |
deploy
1 compile
mvn package -Dmaven.test.skip=true
2 upload to one of
nifi.nar.library.directory=./lib
nifi.nar.library.directory.custom=./lib_custom
nifi.nar.library.autoload.directory=./extensions
nifi.nar.working.directory=./work/nar/
cp nifi-email-nar/target/nifi-email-nar-0.1.nar nifi/lib_custom/
3 restart nifi if need
nifi/bin/nifi.sh restart
实现方式
- 1 官方MergeRecord/MergeContent组件+官方组件PutEmail组件
需要结合merge的特点和规范来应用
- 2 自定制插件-其实主要是件这个基本的功能研究nifi插件的规范来为以后的深度定制作准备
目标是全线替换logstash，flink，一些数据同步工具，原因后续有机会谈
官方的文档较少，不过好在开源，可以参照官方的插件源码，熟悉nifi的开发规范(包括测试方式)
以http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.PutEmail/index.html一个官方插件为例
官方为每flowfile 作为一条邮件发放
我们的目村是改造为 合并多条flowfile,作为一条邮件发放，暂时放弃发送附件，只将多flowfile的attr合并表格发放
直接看代码吧
合并flowfile的attr为html的table结构，很轻量的需求，为了少依赖，直接拼字符串
public class Util {
public static String buildEmailHtmlBodyByAttr(List<FlowFile> flowFileList) {
List<Map<String, String>> attrList = flowFileList.stream().map(flowFile -> flowFile.getAttributes()).collect(Collectors.toList());
List<String> allKeys = attrList.stream().map(attr -> attr.keySet()).flatMap(keys -> keys.stream()).distinct().collect(Collectors.toList());
Collections.sort(allKeys);
StringBuilder sb = new StringBuilder();
sb.append("<table border=\"1\" cellspacing=\"0\" >");
sb.append("<thead align=\"center\" valign=\"middle\">");
sb.append("<tr>");
allKeys.forEach(key -> sb.append("<th>" + key + "</th>"));
sb.append("<th>entryDate</th>");
sb.append("<th>size</th>");
sb.append("</tr>");
sb.append("</thead>");
sb.append("<tbody>");
flowFileList.stream().forEach(flowFile -> {
Map<String, String> kvs = flowFile.getAttributes();
sb.append("<tr>");
allKeys.forEach(key -> sb.append("<td>" + kvs.getOrDefault(key, "") + "</td>"));
sb.append("<td>" + new Date(flowFile.getEntryDate()) + "</td>");
sb.append("<td>" + flowFile.getSize() + "</td>");
sb.append("</tr>");
});
sb.append("</tbody>");
sb.append("</table>");
return sb.toString();
}
}
发送邮件部分
@Override
public void onTrigger(final ProcessContext context, final ProcessSession session) {
//邮件分组合并，获取组最大个数
Integer groupSize = context.getProperty(GROUP_SIZE).asInteger();
List<FlowFile> flowFiles = session.get(groupSize);
if (flowFiles == null || flowFiles.size() == 0) {
return;
}
final Properties properties = this.getMailPropertiesFromFlowFile(context);
final Session mailSession = this.createMailSession(properties);
final Message message = new MimeMessage(mailSession);
final ComponentLog logger = getLogger();
try {
message.addFrom(toInetAddresses(context, FROM));
message.setRecipients(RecipientType.TO, toInetAddresses(context, TO));
message.setRecipients(RecipientType.CC, toInetAddresses(context, CC));
message.setRecipients(RecipientType.BCC, toInetAddresses(context, BCC));
this.setMessageHeader("X-Mailer", context.getProperty(HEADER_XMAILER).getValue(), message);
message.setSubject(context.getProperty(SUBJECT).getValue());
String messageText;
//合并flowFiles的attr为messageText
if (context.getProperty(MESSAGE) != null && context.getProperty(MESSAGE).getValue() != null) {
messageText = context.getProperty(MESSAGE).getValue();
} else {
messageText = getMessage(flowFiles);
}
String contentType = context.getProperty(CONTENT_TYPE).getValue();
message.setContent(messageText, contentType);
message.setSentDate(new Date());
//发送邮件
send(message);
session.transfer(flowFiles, REL_SUCCESS);
logger.info("Sent email as a result of receiving {}", new Object[]{flowFiles});
} catch (final ProcessException | MessagingException e) {
context.yield();
logger.error("Failed to send email for {}: {}; routing to failure", new Object[]{flowFiles, e.getMessage()}, e);
session.transfer(flowFiles, REL_FAILURE);
}
}
private String getMessage(final List<FlowFile> flowFiles) {
String messageText = Util.buildEmailHtmlBodyByAttr(flowFiles);
return messageText;
}
