I've updated the configuration examples in the open source segmenter project to reflect Apple's recommended stream bitrates for iPad video streaming, added a few fixes and a few new features. If you are interested in streaming video on the iPad, iPhone or iPod Touch and haven't done so yet you it may help to start with my post on windowed streaming on for the iPhone, then read about iPhone HTTP streaming with FFMpeg and the open source segmenter and finally check out the iPad, iPhone, and iPod Touch live video streaming project page.
Here is a demo of the iPad streaming video created with the segmenter (I tried to show the progressive upgrade happening but it happens very quickly since the iPad is on WIFI, I also show that you can scrub without any issues and if you look in the background you can see the server log displaying entries as the segments are downloaded):
If you want to view the demo yourself I've created a demo for the iPad, iPhone, and iPod Touch. Note that those are two links, one for the iPad version and one for the iPhone/iPod Touch version. The main difference is the size of the video.
I used the open source video Big Buck Bunny (the 1920×1080 ogg version) for the above demos.
If you are interested in more details on what changed read on or skip to the bottom if you want to see what I'll be working towards in future versions of the segmenter.
Over the past few weeks I learned that it is important that each segment starts with an IDR frame. To accomplish this I thought setting the gop so that the segment time % (gop size / frame rate) = 0 would work but I haven't completely convinced myself of this yet. An example of would be a segment size of 10 seconds % (300 gop size / 30 frame rate) = 0. This should insure that each segment starts with the correct i-frame and while I have looked over the resulting segments with a hex editor and believe it works I still get errors when using certain tools on the individual segments that makes me think it isn't working like I think it should. Either way I have included updates to the gop size in the example configuration files. The gop size is controlled by the -g option for FFMpeg and in the examples I have set it so that for a 10 second segment with the given frame rate the gop size makes sure each segment starts correctly. If you want to know more you can dig into the resulting segments and use this forum post on how to extract an i-frame, this list of mpeg headers to verify that each has the correct i-frame at the start.
The Apple streaming tech notes where informative in a few other ways as well. The tech notes contain the supported h264 profiles for each device. For the iPhone/iPod Touch baseline level 3.0 is supported while the iPad supports baseline level 3.1. You can find out more about h264 encoding levels with FFMpeg and also review FFMpeg x264 encoding guide, FFMpeg option mappings and information on encoding with the x264 codec for more information.
Another bit of information disclosed in the Apple tech notes is the existence of a media stream validation tool. This tool can be downloaded from the Apple iPhone developer site and has to be run on OS X. Apple recommends that you use this tool to validate any streams that you create.
One other note that I have recently ran across is that Apple seems to be rejecting native apps that have video streams without a fall back audio stream. The correct way of generating the fallback 64k audio only stream is something that is lacking in the current version of the segmenter but I hope to fix that soon. It won't matter for those using the HTML5 video tag however.
If you are using HTML5 video tags it is important not to use my older posts as a guide for the iPad since I use a now outdated example. Those examples still work on the iPhone but they do not work on the iPad. Here is an updated example that will work on both the iPhone and iPad:
<html>
<head>
<title>Video Test</title>
</head>
<body style="background-color:#FFFFFF; ">
<center>
<video controls>
<source src="stream_multi.m3u8"/>
</video>
</center>
</body>
</html>
Finally I am currently using the latest version of FFMpeg to do tests and it seems to have become more reliable for me. The current version in git seems to be pretty good at producing usable videos from a variety of source videos.
The following is a quick summary of the recent changes I have made:
- Applied an audio patch for the segmenter from Scott Kidder. This patch skips any video processing if there is no video stream.
- Modified sample configs with newer FFMpeg string, gop sizes, frame rates and other parameters to match the Apple recomended values for iPhone/iPod Touch with aspect ratios of 4:3 and 6:9 on cell or wifi and iPad with aspect ratios of 4:3 and 6:9 on cell or wifi
- Added configuration sanity checks
- Made gems not required if not using those features that need them
- Added fix for deprecation warning from newest version of libavformat
- Made termination work better when killed with SIGINT/ctrl-c
- Fixed some minor issues with index file format
The following are enhancements I'm planning on making:
- I plan on creating a howto on getting everything set up on EC2.
- Dig more into verification that each segment starts with the correct I-frame.
- I'm going to work on refactoring everything so that instead of creating files and then transferring them it streams them all. This makes more sense with doing encryption as just another part of the pipeline.
- Add ability to do encryption. I have a proof of concept working but I decided I need to get everything set up in one continuous stream before integrating encryption into the mix.
- I had thoughtabout using the ruby daemon utils to wrap the segmenter but I decided it probably isn't necessary at this point.
Incredibly helpful as always. We'll be adding support to Panda asap!
Thanks – Damien.
Pingback: How to Create iPad Formatted Videos Using HandBrake or FFMpeg
In a desperate attempt to get the segmenter to produce Apple's required 64kbps baseline stream, I first tried rendering an audio-only ts, but the segmenter fails with a "segmentation fault" error.
Next, I rendered a .ts file with a still image as the video stream and a 32kbps audio stream. The only way I could get the muxing overhead low enough to fit everything into 64kbps was to drastically drop the frame rate. Since it's a still image anyway, I don't care. Dropping the frame rate to 3fps cut the structural overhead from around 50kbps to 7kbps; I can now fit 32kbps/44.1k audio and a still image in a .ts with plenty of room to spare for the 64kbps limit.
Here's where it gets weird – after dropping the frame rate, specifying a segment duration of 10 seconds gives me segments that are 1 minute 24 seconds each. Changing the segment duration argument to segmenter changes the text in the m3u8 file, but does not change the segment count or length – they're always 1:24.
Any ideas?
segmenter rev50
FFmpeg SVN-r22566
x264 0.88.x (x264-snapshot-20100315-2245)
@Curt there is actually more to it than that to be compliant. The stream requires a tag that contains the time stamp that syncs with the video segments. It is on the short list of modifications I'm going to make but there is a decent amount of work required to get it going.
Can this streaming ALSO be used in a browser like safari?
@Chris Yes, it will work for safari.
Update: After modifying the ffmpeg h264 params when inserting the still image, the segment times straightened out. They're not all 10 seconds, but apparently close enough; Apple approved the iPhone app!
I tying to start http_streamer.rb as a daemon. But nothing happens.
This is a simple control my_control.rb:
#!/usr/bin/env ruby
#daemon control
require 'daemons'
Daemons.run('http_streamer.rb')
config.yml hard included in http_streamer.rb
It's normally start through: ruby http_streamer.rb
But didn't start through: ruby my_control.rb start
in the same dir
wondering if anyone has tried with dnxhd files. i can transcode them ok (to ts). the segmenter doesn't complain. but ipad/iphone just can't play the video.
thanks,
jp
found the answer for DNxDH files:
1. apply the following patch to libavcodec/dnxhddec.c
=============================
ctx->cid = AV_RB32(buf + 0x28);
dprintf(ctx->avctx, "compression id %d\n", ctx->cid);
+ if (ctx->cid == 1242) {
+ ctx->height = 1080;
+ }
+
if (dnxhd_init_vlc(ctx, ctx->cid) <0)
return -1;
===========================
2. don't configure the ffmpeg with –enable-libxvid since it's supported by the internal ffmpeg and not through libxvid
all works well.
disclaimers: these are the solutions i compiled from folks in the other forums. can't take credits :)
jp
ok. That`s fine. I have to set full paths everywhere in .yml.
Has anyone had any luck getting encrypted streaming to work? It seems I'm almost there but my video doesn't play but I don't get any errors either (like "Video is unplayable" or "You don't have permission to play this video" when I got the key wrong).
#bash script:
keyFile="key.txt"
openssl rand 16 > $keyFile
hexKey=$(cat key.txt | hexdump -e '"%x"')
hexIV='0'
openssl aes-128-cbc -e -in $fileName -out $encryptedFileName -p -nosalt -iv ${hexIV} -K ${hexKey}
#my playlist file:
#EXTM3U
#EXT-X-TARGETDURATION:000020
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-KEY:METHOD=AES-128,URI="key.txt"
#EXTINF:20, no desc
test.ts.enc
#EXT-X-ENDLIST
Carson,
You're Live Http Streaming Video using VLC works great!
Have you looked at iOS4 yet? Do you think it's going to have improved performance, specifically smaller latency? I've been trying to read up on the new video capabilities, but I can't find anything specific. Do you know of any links?
Just wondering. You're so "in the know."
Thanks,
Estelle
Carson:
Thank you so much for your writeup on this. Unfortunately, I'm unable to get it to work, getting errors like:
WARNING: Playlist Content-Type is 'application/x-mpegurl', but should be one of 'application/vnd.apple.mpegurl', 'audio/x-mpegurl' or 'audio/mpegurl'.
WARNING: 258 samples (88.966 %) do not have timestamps in track 256 (avc1). 4: us2-1.ts ~~~~~~~~
WARNING: Media segment duration outside of expected duration by 47.733 % (5.23 vs. 10.00 seconds, limit is 20 %). 40: us2-19.ts ~~~~~~~~~
I happy to pay you 1.5x your regular consulting rate if you can help us get through this – it's holding up an important update to our iphone app :(
Please email me at: chessdev{@}gmail
Thank you!!
Erik
Chess.com
I am using Rhozet to encode the file to HTTP Live Streaming and that works fine. I need to encrypt the files and was about to use media stream segmenter and was wondering if I could, considering taht the files are already segmented by Rhozet Carbon Server.
All I need to do is the encryption, so can I use MediaStreamSegmenter just to do the encryption? If so how?
Thanks,
@JeanM I believe you can use it to do just encryption but I have never used it myself. If you look at the documentation for it they probably have a command line option that will turn it on.
Carson,
I tried to encrypt only using mediastreamsegmenter but it seems like it requires the encoder IP and port.
When I run it without it it just hanges there.
Do you have some working sample command line examples beside the ones listed in the man pages?
@JeanM I'm not sure how the encryption works with the Apple tools. I haven't yet added that to the streamer I have.
Carson,
1. Thank you for your contribution. This is really exciting, now that I got it working, I guess.
2. Can anyone explain what is in this segmenter's log? Can I ignore the error ?
E, [2010-09-24T14:23:22.006299 #26568] ERROR -- : Encoder ep_128k: [mpegts @ 0x8487110] max_analyze_duration reached
[mpegts @ 0x8487110] Estimating duration from bitrate, this may be inaccurate
3. What about this log entry? What do the numbers mean when the log says Transfer initiated with value=*1,2,0,ep_128k*?
D, [2010-09-24T13:09:10.323226 #25814] DEBUG -- : Segment command ep_128k: *1, 2, 0, ep_128k*
I, [2010-09-24T13:09:10.323468 #25814] INFO -- : Transfer initiated with value = *1, 2, 0, ep_128k*
D, [2010-09-24T13:09:10.323622 #25814] DEBUG -- : Creating index
D, [2010-09-24T13:09:10.323883 #25814] DEBUG -- : Done creating index
I, [2010-09-24T13:09:10.326086 #25814] INFO -- : Transfer done
3. When viewing the segmented files in their folders, I saw some that were "MPEG Transport Stream" files and others that were "message catalog" files.
Is that normal o should they all be MPEG Transport Stream files with the exception of the .m3u8 file?
Can I control whether a video file starts on a keyframe when I edit the file in a editor like Premiere Pro or Final Cut? I know I can set the keyframe distance manually, but I usually leave it unchecked to allow the computer to calculate it for me. I'm thinking that this would help find out why it has trouble encoding the first frame.
Thank you all for your posts. They have always been a help to me .
Harold
Curt: I'm curious how you got the still image included into the segments. Can you share your x264 settings?