iPad Streaming Video and More

I’ve updated the configuration examples in the open source segmenter project to reflect Apple’s recommended stream bitrates for iPad video streaming, added a few fixes and a few new features. If you are interested in streaming video on the iPad, iPhone or iPod Touch and haven’t done so yet you it may help to start with my post on windowed streaming on for the iPhone, then read about iPhone HTTP streaming with FFMpeg and the open source segmenter and finally check out the iPad, iPhone, and iPod Touch live video streaming project page.

Here is a demo of the iPad streaming video created with the segmenter (I tried to show the progressive upgrade happening but it happens very quickly since the iPad is on WIFI, I also show that you can scrub without any issues and if you look in the background you can see the server log displaying entries as the segments are downloaded):

</param></param></param></embed>

If you want to view the demo yourself I’ve created a demo for the iPad, iPhone, and iPod Touch. Note that those are two links, one for the iPad version and one for the iPhone/iPod Touch version. The main difference is the size of the video.

I used the open source video Big Buck Bunny (the 1920x1080 ogg version) for the above demos.

If you are interested in more details on what changed read on or skip to the bottom if you want to see what I’ll be working towards in future versions of the segmenter.

Over the past few weeks I learned that it is important that each segment starts with an IDR frame. To accomplish this I thought setting the gop so that the segment time % (gop size / frame rate) = 0 would work but I haven’t completely convinced myself of this yet. An example of would be a segment size of 10 seconds % (300 gop size / 30 frame rate) = 0. This should insure that each segment starts with the correct i-frame and while I have looked over the resulting segments with a hex editor and believe it works I still get errors when using certain tools on the individual segments that makes me think it isn’t working like I think it should. Either way I have included updates to the gop size in the example configuration files. The gop size is controlled by the -g option for FFMpeg and in the examples I have set it so that for a 10 second segment with the given frame rate the gop size makes sure each segment starts correctly. If you want to know more you can dig into the resulting segments and use this forum post on how to extract an i-frame, this list of mpeg headers to verify that each has the correct i-frame at the start.

The Apple streaming tech notes where informative in a few other ways as well. The tech notes contain the supported h264 profiles for each device. For the iPhone/iPod Touch baseline level 3.0 is supported while the iPad supports baseline level 3.1. You can find out more about h264 encoding levels with FFMpeg and also review FFMpeg x264 encoding guide, FFMpeg option mappings and information on encoding with the x264 codec for more information.

Another bit of information disclosed in the Apple tech notes is the existence of a media stream validation tool. This tool can be downloaded from the Apple iPhone developer site and has to be run on OS X. Apple recommends that you use this tool to validate any streams that you create.

One other note that I have recently ran across is that Apple seems to be rejecting native apps that have video streams without a fall back audio stream. The correct way of generating the fallback 64k audio only stream is something that is lacking in the current version of the segmenter but I hope to fix that soon. It won’t matter for those using the HTML5 video tag however.

If you are using HTML5 video tags it is important not to use my older posts as a guide for the iPad since I use a now outdated example. Those examples still work on the iPhone but they do not work on the iPad. Here is an updated example that will work on both the iPhone and iPad:

<html>
  <head>
    <title>Video Test</title>
  </head>
  <body style="background-color:#FFFFFF; ">
    <center>
      <video controls>
        <source src="stream_multi.m3u8"/>
      </video>
    </center>
  </body>
</html>

Finally I am currently using the latest version of FFMpeg to do tests and it seems to have become more reliable for me. The current version in git seems to be pretty good at producing usable videos from a variety of source videos.

The following is a quick summary of the recent changes I have made:

  • Applied an audio patch for the segmenter from Scott Kidder. This patch skips any video processing if there is no video stream.
  • Modified sample configs with newer FFMpeg string, gop sizes, frame rates and other parameters to match the Apple recomended values for iPhone/iPod Touch with aspect ratios of 4:3 and 6:9 on cell or wifi and iPad with aspect ratios of 4:3 and 6:9 on cell or wifi
  • Added configuration sanity checks
  • Made gems not required if not using those features that need them
  • Added fix for deprecation warning from newest version of libavformat
  • Made termination work better when killed with SIGINT/ctrl-c
  • Fixed some minor issues with index file format

The following are enhancements I’m planning on making:

  • I plan on creating a howto on getting everything set up on EC2.
  • Dig more into verification that each segment starts with the correct I-frame.
  • I'm going to work on refactoring everything so that instead of creating files and then transferring them it streams them all. This makes more sense with doing encryption as just another part of the pipeline.
  • Add ability to do encryption. I have a proof of concept working but I decided I need to get everything set up in one continuous stream before integrating encryption into the mix.
  • I had thoughtabout using the ruby daemon utils to wrap the segmenter but I decided it probably isn't necessary at this point.