ffmpeg now says "Invalid data found when processing input" #216

catchingknives

Since version 0.5 ffmpeg comes back with the errors/warnings

Stream #0: not enough frames to estimate rate; consider increasing probesize

(but increasing probesize doesn't help) and

Invalid data found when processing input

when using the command line

gource -1280x720 -o - | ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -preset ultrafast -pix_fmt yuv420p -crf 1 -threads 0 -bf 0 gource.mp4

as used in this example. This also applies to WebM encoding and for the method where you first go via the raw (ppm) video. ffmpeg always fails at the same second mark in my input file so it seems there's something there which it doesn't like?

hankcs

Same issue here. My version is ffmpeg-4.2.1.

acaudwell

Does this happen with Gource 0.49 if you build it from source too? Wondering if its a change in behaviour with ffmpeg. Also what distro and release is this occuring on?

hankcs

Yes, it does. But 2.8.15 works fine.

acaudwell

Can confirm I see the same issue with ffmpeg 4.2.1. ffmpeg 3.4.6 is fine.

acaudwell

The Gource ppm header has a comment line # Generated by Gource. I tried generating a ppm without it and the Invalid data found when processing input error doesn't occur.

I see that the last change to the pnm_parser.c in FFmpeg is to the comment parsing:
FFmpeg/FFmpeg@10ea6c3

I think maybe bs = match + 1 should maybe be bs = match, but I haven't tried it.

acaudwell

Hmm. That doesn't seem to be the right bit of code.

Since the comment in the header Gource includes is superfluous. I've removed it as a workaround for now a3e7269.

apalchys

Gource 0.51
ffmpeg 4.2.1

Seems the issue still exists

acaudwell

Oh? The fix is in the sub module, in src/core (src/core/ppm.cpp), is it possible the submodule is out of date?

Tried it again seems to be working.

apalchys

is there any way how I can troubleshoot it? I checked and ppm.cpp is up to date.
I'm using MacOS (Catalina)

btw, it works if I set --stop-at-time 3 but when I set --stop-at-time 10 or longer, it starts failing.

acaudwell

Things you could try:

check if gource is adding the #Generated by Gource header to the ppm output: gource -o - |head -n 2. If you see that message then you are running the old code
at the bottom of ppm.cpp in the dumpImpl() method add:
fprintf(stderr, "width %d height %d rowstride %d\n", display.width, display.height, (int)rowstride);
recompile and check that you see that message when you are recording gource with ffmpeg to confirm its actually using that version of ppm.cpp.

If it is using the new code and its still an issue could you could supply the command line your using, what resolution your running at.

acaudwell

Would be useful to get other peoples experience with the fix.

apalchys

I did what you suggested above, recompiled and the issue still exists.

gource -o - |head -n 2

P6
2048 1280 255

How I run:
gource --viewport 1280x720 --seconds-per-day 1 --auto-skip-seconds 1 -o gource.ppm -r 25 --stop-at-time 20 --time-scale 2

ffmpeg -y -r 25 -f image2pipe -vcodec ppm -i gource.ppm -vcodec libx264 -preset ultrafast -pix_fmt yuv420p -threads 0 -bf 0 gource.mp4

btw, it works fine with ffmpeg 3.4.2

acaudwell

Thanks I will try those settings.

I notice the resolutions seem to be different in the output, I wonder if its has to do with Apple Retina display handling, maybe you could also try adding the SDL_WINDOW_ALLOW_HIGHDPI flag as detailed here: #74 (comment)

One other thing to rule out is if you have several versions of gource and its running an old one?

dch

kia ora @acaudwell I confirmed that this change resolves this issue for me on FreeBSD when moving from 0.47 to 0.51. It's being used in dwatch(1) and you can see it in action later on in https://www.invidio.us/watch?v=xqOl_HY86n8 debugging kernels!

acaudwell

@dch good to know. Didn't know about this thanks for the link.

davep

Having upgraded to 0.51 on macOS thanks to #225:

$ gource --help | head -1
Gource v0.51

I seem to still be running into the same issue. The generated log file's first couple of lines look like this:

$ head -2 prompt.ppm
P6
2560 1440 255

Display resolution is 2560x1600. The machine (Macbook Pro 2019, 13-inch, running 10.15.1) is also hooked up to two external displays, each of them displaying at 1920x1200. Gource was being run on the primary display (2560x1600).

ffmpeg was run like this:

$ ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i prompt.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 1 -threads 0 -bf 0 prompt.mp4

ffmpeg version is 4.2.1, installed via Homebrew.

If there's anything else I can provide to help diagnose this, apologies for omitting it and please don't hesitate to ask.

acaudwell

Maybe it's related to the image size? You could try and render a smaller window (e.g. add -640x480 or something).

You could also try this SDL window flag option: #216 (comment)

acaudwell

It seems to still happen at 2560x1440. So the thing to eliminate if its any ppm video at that resolution or if its specific to gource.

acaudwell

I should also really look at an alternative lossless video format with some basic compression. I might try adding a png output option with minimal compression so it's not too slow.

davep

Thanks @acaudwell, I'll look at checking those things (won't be back at my office desk until Tuesday so might not be until then -- unless I can get the data I was using down the wire at home).

I think I did forget to mention one thing: I'd run gource with -1280x720 as a parameter when generating the ppm. I'm afraid I don't know much about any of this, but does it make sense that it says "2560 1440" in the head of the ppm in that circumstance?

davep

Turns out I could get access to the log file. Running this all again on my personal Macbook Pro (2019, 13inch) it worked just fine. Created the ppm like this:

$ gource -1280x720 -c 4 prompt.log -o prompt.ppm

and converted like this:

$ ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i prompt.ppm -vcodec libx264 -preset medium -pix_fmt yuv420p -crf 1 -threads 0 -bf 0 prompt.mp4

and the conversion went through without incident.

The setup here, vs the one on my desk at work, is more or less identical in terms of the main machine. The only difference here is that I'm running with the lid closed and hooked up to an external display. running at 1920 x 1080, the default for that display.

If I'm reading the above correctly, this is where the flag you mention comes in? And I take it this is something to be done when building Gource from source? If so, when I get a chance and am sure I have the correct setup, I'll try and give that a go (unless the above is enough to help nail this).

In that setup I'm left with the head of the ppm saying:

P6
1280 720 255

If I run with the lid open, on the main display:

I get this in the ppm:

P6
2560 1440 255

That results pretty quickly in me getting a Invalid data found when processing input when running ffmpeg.

acaudwell

I think I've confirmed this is an ffmpeg bug with ppm image sequences.

Given a video that is 1920x1080 (also seems to be a problem with that resolution).

Convert it to a sequence of ppm images in one file:
ffmpeg -i video.mp4 -c:v ppm -f image2pipe output.ppm

Try to convert it to another format:
ffmpeg -i output.ppm result.mp4

During conversion there is the Invalid data found when processing input message!

If you do the same thing but with -c:v png to save as a sequence of pngs, there is no such error converting it.

acaudwell

Also looks like it's only the 4.2.1 release that hangs when you get the Invalid data found when processing input message.

If you use a nightly build/ build from master it just continues on if it gets an error so it's not really an issue at that point. I recommend that as a work around if you're having this problem.

Presumably for homebrew you can add --HEAD to build ffmpeg from master.

davep

Thanks, when I next get a chance I'll try that out.

enesfarukmeniz

Tried reinstalling ffmpeg with --HEAD and seems like it works.

Cartoonman

I just ran into this very problem yesterday when I upgraded my docker stack to use Alpine 3.11, which packages FFmpeg 4.2.1. Glad you were able to dig into it; I had feared it was yet another ffmpeg bug that would have taken me ages to investigate. Was this reported upstream to the ffmpeg devs?

FFmpeg 4.2.2 just released 13 days ago, I'll see if that release solves the 4.2.1 issue. The shortlog contained a lot of 'fixes'.

Cartoonman

I confirmed that the ffmpeg 4.2.2 release seems to have resolved the issue entirely; no Invalid data found when processing input error popped up on my test case that was consistently able to trigger it previously.

I'd recommend anyone still having the issue to either downgrade to 4.1.4, or upgrade to 4.2.2.

acaudwell

Good to know its no longer an issue in 4.2.2!

chenrui333

I had issue with ffmpeg 4.2.2 (Homebrew/homebrew-core#48445), my gource is 0.51.

I am switching to 2.8.15 now (seems working fine for me so far).

chenrui333

Looks like I still suffer the maxval issue:

Input stream #0:0 frame changed from size:2560x1440 fmt:rgb24 to size:2560x1440 fmt:gray
Input stream #0:0 frame changed from size:2560x1440 fmt:gray to size:2560x1440 fmt:monow drop=1
[ppm @ 0x7f8a4b001800] value 666 larger than maxval 255.76 bitrate=306217.4kbits/s dup=0 drop=2
Error while decoding stream #0:0: Invalid data found when processing input
Input stream #0:0 frame changed from size:2560x1440 fmt:monow to size:2560x1440 fmt:rgb24drop=2
Error while decoding stream #0:0: Invalid data found when processing input4kbits/s dup=0 drop=67
    Last message repeated 1 times
Error while decoding stream #0:0: Invalid data found when processing input4kbits/s dup=0 drop=68
    Last message repeated 17 times
[ppm @ 0x7f8a4b001800] value 666 larger than maxval 255
Error while decoding stream #0:0: Invalid data found when processing input
[ppm @ 0x7f8a4b001800] value 666 larger than maxval 255.76 bitrate=306217.4kbits/s dup=0 drop=68
Error while decoding stream #0:0: Invalid data found when processing input
Input stream #0:0 frame changed from size:2560x1440 fmt:rgb24 to size:2560x1440 fmt:monowdrop=91
[ppm @ 0x7f8a4b001800] value 777 larger than maxval 255.76 bitrate=306217.4kbits/s dup=0 drop=112
Error while decoding stream #0:0: Invalid data found when processing input
Input stream #0:0 frame changed from size:2560x1440 fmt:monow to size:2560x1440 fmt:rgb24drop=112
Input stream #0:0 frame changed from size:2560x1440 fmt:rgb24 to size:2560x1440 fmt:gray drop=178

SamMackrill

I also get this with ffmpeg 4.2.2 (also tried git-2020-02-18-ebee808) & gource 0.51
does 27k frames then dies, nothing obvious in the ffmpeg log
Update: also fails with 4.1.4
Update 2: also fails with 3.4.2 :(

OK forget this. Sorry. I had a corrupt .ppm due to resource issues on my machine.

Cartoonman

@SamMackrill Does it still run fine if FFmpeg is taken out of the loop? e.g. running just Gource.

SamMackrill

@SamMackrill Does it still run fine if FFmpeg is taken out of the loop? e.g. running just Gource.

Thanks, it was Gource exiting early all along, sorry!

Cartoonman

@SamMackrill No problem. An alternative to saving ppm first and then feeding that into FFmpeg is to feed the PPM stream directly from stdout of Gource straight into FFmpeg with FFmpeg's image2pipe input. This way you avoid having to handle a large ppm file and can just directly stream it into FFmpeg.

dyatko

Yes, the stable ffmpeg 4.2.2 fixes the issue

rado84-github

ffmpeg 4.4 just gave me the same error while converting mka (dca) to ac3. I'll try downgrading to 4.2.2 and see what happens.

acaudwell closed this as completed on Feb 26, 2020

uschen mentioned this issue on Feb 28, 2021

unable to play wmv alibaba/CicadaPlayer#674

Closed

martingercke mentioned this issue on Nov 2, 2021

streaming errors in ffmpeg 4.2.1 jacobbo/WebEye#91

Open

songs18 mentioned this issue on Jun 16, 2022

Certain m3u8s Reliably Fail songs18/m3u8_To_MP4#10

Closed

gioelecerati mentioned this issue on Jun 28, 2022

transmuxing MP4 recording output is broken livepeer/go-livepeer#2476

Closed

hugalafutro mentioned this issue on Jul 10, 2022

webrtc broken after update to core 2022.7 AlexxIT/WebRTC#321

Closed

ffmpeg now says "Invalid data found when processing input" #216

ffmpeg now says "Invalid data found when processing input" #216

ffmpeg now says "Invalid data found when processing input"#216

Comments

catchingknives commented on Sep 28, 2019 •

hankcs commented on Sep 28, 2019 •

acaudwell commented on Oct 3, 2019 •

hankcs commented on Oct 3, 2019

acaudwell commented on Oct 7, 2019

acaudwell commented on Oct 11, 2019 •

acaudwell commented on Oct 17, 2019

apalchys commented on Nov 14, 2019 •

acaudwell commented on Nov 14, 2019 •

apalchys commented on Nov 14, 2019

acaudwell commented on Nov 15, 2019 •

acaudwell commented on Nov 15, 2019

apalchys commented on Nov 16, 2019

acaudwell commented on Nov 17, 2019

dch commented on Nov 27, 2019

acaudwell commented on Nov 27, 2019

davep commented on Nov 28, 2019

acaudwell commented on Nov 29, 2019

acaudwell commented on Nov 29, 2019 •

acaudwell commented on Nov 29, 2019 •

davep commented on Nov 29, 2019

davep commented on Nov 30, 2019

acaudwell commented on Nov 30, 2019

acaudwell commented on Dec 6, 2019

davep commented on Dec 6, 2019

enesfarukmeniz commented on Dec 30, 2019

Cartoonman commented on Jan 14, 2020

Cartoonman commented on Jan 14, 2020

acaudwell commented on Jan 16, 2020

chenrui333 commented on Jan 28, 2020 •

chenrui333 commented on Jan 28, 2020

SamMackrill commented on Feb 20, 2020 •

Cartoonman commented on Feb 21, 2020

SamMackrill commented on Feb 21, 2020

Cartoonman commented on Feb 22, 2020

dyatko commented on Feb 25, 2020

rado84-github commented on May 13, 2021

公告

搜索

常用链接

我的标签

随笔档案

阅读排行榜

评论排行榜

推荐排行榜

最新评论