PipelineDB On Kafka

PipelineDB　

安装
yum install https://s3-us-west-2.amazonaws.com/download.pipelinedb.com/pipelinedb-0.9.8u2-centos7-x86_64.rpm
或
sudo rpm -ivh pipelinedb-0.9.8u2-centos7-x86_64.rpm

初始化
pipeline-init -D <data directory>

启动
pipeline-ctl -D /var/lib/pgsql/pipeline/data -l logfile start

激活
psql -h localhost -p 5432 -d pipeline -c "ACTIVATE

修改配置文件

pipelinedb.conf 类似于postgres的　postgresql.conf
pg_hba.conf

登录

psql -h localhost -p 5432 -d pipeline

主从搭建

http://docs.pipelinedb.com/replication.html

与kafka集成的流式计算

https://www.pipelinedb.com/blog/sql-on-kafka
https://github.com/digoal/blog/blob/1acc4e721e8e320facc13a6980f50339f6fe71cd/201510/20151021_02.md

效果无敌

 

遇到问题，

url　正则不匹配时pipelinedb直接挂掉

 

创建滑窗

CREATE CONTINUOUS VIEW message_count_hour WITH （sw = '1 hour',step_factor = 20) AS SELECT COUNT(*) FROM logs_stream;

sw 滑窗大小，时间

step_factor 百分比，一个滑窗内多久为一个单位