Minio Bucket 通知试用&&说明

kafka 安装

docker run -p 2181:2181 -p 9092:9092 \
      --name kafka --rm \
      --env ADVERTISED_HOST=HOSTIP \
      --env ADVERTISED_PORT=9092 \
      spotify/kafka

minio (mac local)

wget https://dl.minio.io/server/minio/release/darwin-amd64/minio 
chmod +x minio 
mkdir data
./minio server ./data

config minio notify

路径 ~/.minio/config.json

"kafka": {
    "1": {
        "enable": true,
        "brokers": ["localhost:9092"],
        "topic": "bucketevents"
       }
    }

下载client 并配置minio server

wget https://dl.minio.io/client/mc/release/darwin-amd64/mc 
chmod +x mc 
./mc --help
./mc config host add myminio http://localhost:9000 GT481SA7MEF5D2G29N0L QqaFn67VTbko3FRE0IfZVTYpztb3UpjKW0RloHSt

创建bucket 并启用evnet

注意配置之前，如果没有配置kafka 通知，配置完成之后，minio server 需要重启，不然会报错

./mc mb myminio/images
./mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .png

测试

• 下载 kafkacat

  kafka 信息查看(mac)

brew install kafkacat

• 上传图片测试

./mc cp app.png myminio/images

• 效果
  

说明

我们可以添加多种通知，实现一个pipeline 的文件处理服务，还是比较方便的

• 添加另外的一个通知

./mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .txt

• 效果
  
• 查看配置的通知

 ./mc events list myminio/images --json
{"status":"success","id":"","events":["s3:ObjectCreated:*","s3:ObjectRemoved:*","s3:ObjectAccessed:*"],"prefix":"","suffix":".png","arn":"arn:minio:sqs::1:kafka"}
{"status":"success","id":"","events":["s3:ObjectCreated:*","s3:ObjectRemoved:*","s3:ObjectAccessed:*"],"prefix":"","suffix":".jpg","arn":"arn:minio:sqs::1:kafka"}
{"status":"success","id":"","events":["s3:ObjectCreated:*","s3:ObjectRemoved:*","s3:ObjectAccessed:*"],"prefix":"","suffix":".txt","arn":"arn:minio:sqs::1:kafka"}

信息存储

实际上默认的配置都是在一个.minio.sys 文件夹中

• 参考图
  
• 内容

<NotificationConfiguration>
<QueueConfiguration>
<Id></Id>
<Filter>
<S3Key>
<FilterRule>
<Name>suffix</Name>
<Value>.png</Value>
</FilterRule>
</S3Key>
</Filter>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>
<Event>s3:ObjectAccessed:*</Event>
<Queue>arn:minio:sqs::1:kafka</Queue>
</QueueConfiguration>
<QueueConfiguration>
<Id></Id>
<Filter>
<S3Key>
<FilterRule>
<Name>suffix</Name>
<Value>.jpg</Value>
</FilterRule>
</S3Key>
</Filter>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>
<Event>s3:ObjectAccessed:*</Event>
<Queue>arn:minio:sqs::1:kafka</Queue>
</QueueConfiguration>
<QueueConfiguration>
<Id></Id>
<Filter>
<S3Key>
<FilterRule>
<Name>suffix</Name>
<Value>.txt</Value>
</FilterRule>
</S3Key>
</Filter>
<Event>s3:ObjectCreated:*</Event>
<Event>s3:ObjectRemoved:*</Event>
<Event>s3:ObjectAccessed:*</Event>
<Queue>arn:minio:sqs::1:kafka</Queue>
</QueueConfiguration>
</NotificationConfiguration>

参考资料

https://minio.io/downloads.html
https://github.com/edenhill/kafkacat
https://hub.docker.com/r/spotify/kafka/
https://docs.minio.io/docs/minio-bucket-notification-guide
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html