streamsets http client 请求过大问题

http 请求body 过大问题

现象

com.streamsets.pipeline.api.ext.io.OverrunException: Reader exceeded the read limit '1048576'

解决

in version 2.6 and before - add the following to your ${SDC_JVM_OPTS}:
-DDataFactoryBuilder.OverRunLimit=10485760 -Doverrun.reader.read.limit=10485760
these parameters set the buffer size to 10MiB - adjust as appropriate. also, set the Max Object Length in the Data Format tab.
in our next version after 2.6 (so in 2.6.X or 2.7), you'll be able to fix this by setting a new property - parser.limit in the sdc.properties file eg: parser.limit=10485760

参考资料

https://ask.streamsets.com/question/69/comstreamsetspipelinestageorigins3badspoolobjectexception-comstreamsetspipelineapiextiooverrunexception-reader-exceeded-the-read-limit-1048576-when/