Wednesday, March 14, 2012

The Visual Studio 2011 Developer Preview

Today I thought I'd give the new VS 2011 Developer Preview a bash. My main interest was to see how C++ AMP can be used in increasing code performance.

This time the migration of projects to the latest VS version was relatively painless in comparison with the upgrade to VS2010. The one error I ran into was

Cannot open include file: 'sal.h': No such file or directory

which was easily fixed by adding "C:\Program Files (x86)\Windows Kits\8.0\Include\shared" to the list of include directories.

Remembering how the performance of an H.263 codec improved *notably* when we migrated from VS2008 to 2010, I was curious to find out what the VC team has done in 2011. After having rebuilt the Video Processing Project solution and running the FrameGrabber application the following results were observed: 

Mode:    Total average         Per frame      Improvement %
0                872.10 ms             0.86 ms
1                859.45 ms              0.75ms                 13.42%
2                 816.97ms              0.56ms                 35.26%
3                 879.01ms              0.82ms                   5.04%
5                 N/A

Comparing these results to the ones obtained using VS2010 as posted in Improving live multimedia pipeline performance

Mode:    Total average         Per frame      Improvement %
0               1224.46ms            1.16ms
1               1105.63ms            1.02ms                  12.26%
2                 969.82ms            0.55ms                  53.10%
3               1572.18ms            1.24ms                   -6.74%
5               1106.09ms            0.59ms                  49.13%

The results were obtained using the FrameGrabber application, running the application 5 times and taking the average. Looks like the compiler team has done some serious work optimising the generated code. Even though the relative placement between the timings of the various modes is still similar, the gap has closed considerably. This is not to say, that optimisation is any less important when using newer compilers : a 35.26% improvement is nothing to frown upon.

Looks like the free lunch isn't quite over yet, as long as you can afford a new compiler/IDE :-)


Looks like this might have something to do with the auto-vectorizer in VS2011:

Improving live multimedia pipeline performance

In this post, we will discuss code optimisation techniques necessary in real-time media pipelines. Live video requires that the media is processed fast enough to achieve the desired framerate e.g. to have a 15 fps framerate means that each frame should take no longer than 1000/15 = 66.6 ms.

A video pipeline is typically comprised of a media source, colour converters, scalers, croppers, video mixers, video codecs and media sinks.

Media pipeline
This means that all the operations together can not take more than 66.6 ms.
Although operations such as colour conversion are considered light-weight in relation to the video encoding, each link in the chain should be written as efficient as possible, within reasonable means. (Shaving a ms of colour conversion is not really going to make much difference if it is the encoder that takes 50ms per frame.)

The question is how we can improve algorithm performance?
  • Using the fixed point arithmetic over floating point
  • Less copies
  • Lookup tables over computation
  • Multi-threading
  • Increase cache hits
  • Using processor-specific knowledge (e.g. SIMD)
  • Using GPUs?
  • Improved algorithms structure(the big picture)
There is usually some kind of trade-off between speed and memory usage. In the case of the look-up table approach, there could be a slight computational overhead on start-up to compute the look-up table, with the benefit of less computations once the application is in a steady state. One should also take factors such as the size of the look-up table, and the target environment (i.e. desktop vs. embedded device) into account.

In this post, we will try out various techniques to improve the performance of the RGB to YUV420 colour converter ( source code available at the Video Processing Project).
The FrameGrabber project builds a simple multimedia pipeline consisting of a source, a sample grabber and a video renderer.

Once the sample grabber callback is triggered, we do the following:
- convert from RGB to YUV420
- convert back to RGB
- render image for visual confirmation that the conversion is correct.

The original color conversion code looks as follows can be seen in RealRGB24toYUV420Converter.cpp.

We will try to improve on this by adding a lookup table to minimize the multiplications as can be seen in FastLookupTableRGB24toYUV420Converter.cpp.

Next, we approach the problem using fixed point arithmetic as can be seen in FastFixedPointRGB24toYUV420Converter.cpp.
Here the idea is to use integer arithmetic over floating point.

Finally we attempted to use SIMD instructions to improve the colour converter performance as can be seen in FastSimdRGB24toYUV420Converter.cpp.

The FrameGrabber application is called with the following parameters:
FrameGrabber <<File>>.avi mode=0
mode 0 = original algorithm
mode 1 = lookup table
mode 2 = fixed-point arithmetic
mode 3 = SIMD
mode 4 = GPU (unimplemented)
mode 5 = multi-threaded

The standard Foreman test video sequence with CIF resolution was used as the video source in this experiment. The application was run 5 times per mode using an automated script and the results were averaged.

Mode:    Total average         Per frame      Improvement %
0               1224.46ms            1.16ms
1               1105.63ms            1.02ms                  12.26% 
2                 969.82ms            0.55ms                  53.10%
3               1572.18ms            1.24ms                  -6.74%
5               1106.09ms            0.59ms                  49.13%

As expected using a look-up table yields a notable improvement on the original algorithm. The fixed-point arithmetic performs best of all and is roughly twice as fast as the original algorithm. Surprisingly, the SIMD approach yielded no improvements, in fact performs slightly worse than the original. This could however be an implementation issue. (If you have a better solution, please drop us a line). FYI, the question was  posted on stackoverflow. The multi-threaded approach yields also yields a performance gain though this approach should be taken with caution. I would not advise spawning additional threads for the purpose of optimising colour conversion.

Comments/criticism/suggestions/improvements? Please drop us a line. Feel free to download the source and give it a try.

In order to compile the solution with support for mode 5, USE_MULTI_THREADED must be defined in the Image and FrameGrabber projects. Additionally, boost::thread and boost::asio are used to scale the colour conversion across 2 processors and the relevant boost include and library paths need to be configured in Visual Studio.

Wednesday, March 7, 2012

H.264 implementation update

The H.264 implementation we have been working on is finally nearing completion and will be added to the Video Processing Project in the near future. The author of the H.264 codec wrote the following explanation regarding the usage of the codec:

Implementing a DirectShow H264 source filter

After not finding a suitable DirectShow source filter able to render raw H.264 files, we decided to roll our own one (available at the Video Processing Project. I'm sure that many developers have written one of these and perhaps it's time to stop reinventing the wheel. Should anyone want/like to contribute improvements/extensions to this filter, please drop us a line.

The H.264 reference software allows one to take a YUV file and encode it into a .264 file format. These .264 files consist of a sequence of NAL units, each prepended with a start code (0x00000001). A source filter would thus have to read one of these files, break it up into separate NAL units, and then pass one frame at a time to the decoder. Windows 7 features a built-in H.264 decoder.

The IFileSourceFilter interface is implemented to facilitate loading of .264 files. This causes GraphEdit/GraphStudio to display a dialog box in which one can select the desired .264 file.

One of the first steps in writing a source filter is to provide the correct output pin media type, that allows the DirectShow framework to render the graph. In this case, the MEDIA_SUBTYPE_H264 was selected since using it requires the least amount of effort. The implementation of GetMediaType looks as follows:

  HRESULT H264OutputPin::GetMediaType(CMediaType *pMediaType)
    CAutoLock cAutoLock(m_pFilter->pStateLock());
    CheckPointer(pMediaType, E_POINTER);

    VIDEOINFOHEADER2* pvi2 = (VIDEOINFOHEADER2*)pMediaType->AllocFormatBuffer(
    ZeroMemory(pvi2, sizeof(VIDEOINFOHEADER2));
    pvi2->bmiHeader.biBitCount = 24;
    pvi2->bmiHeader.biSize = 40;
    pvi2->bmiHeader.biPlanes = 1;
    pvi2->bmiHeader.biWidth = m_pFilter->m_iWidth;
    pvi2->bmiHeader.biHeight = m_pFilter->m_iHeight;
    pvi2->bmiHeader.biSize = m_pFilter->m_iWidth * m_pFilter->m_iHeight * 3;
    pvi2->bmiHeader.biSizeImage = DIBSIZE(pvi2->bmiHeader);
    pvi2->bmiHeader.biCompression = DWORD('1cva');
    const REFERENCE_TIME FPS_25 = UNITS / 25;
    pvi2->AvgTimePerFrame = FPS_25;
    SetRect(&pvi2->rcSource, 0, 0, m_pFilter->m_iWidth, m_pFilter->m_iHeight);
    pvi2->rcTarget = pvi2->rcSource;
    pvi2->dwPictAspectRatioX = m_pFilter->m_iWidth;
    pvi2->dwPictAspectRatioY = m_pFilter->m_iHeight;
    return S_OK;

This code is sufficient to allow DirectShow to insert the Windows H.264 decoder into the pipeline. Here the width and height seem to be of little importance since they are in any case communicated in the (H.264) Sequence Parameter Set. The parameter sets are found by scanning the .264 file for the appropriate NAL units. The NAL unit type can be extracted from the NAL unit header as follows:

  unsigned char uiNalUnitType = nalUnitHeader & 0x1f;

Sequence parameter sets have value 7, picture parameter sets 8, and IDR frames value 5. One typically needs to pass these to the decoder before other encoded frames.

In closing, the filter currently is also able to output a custom media type, namely MEDIASUBTYPE_H264M. This makes it easy to test our own H.264 decoder filter. In the property pages of the source filter, one can select what the output media type of the H.264 source filter should be. It should be noted, that our H.264 decoder has limitations regarding the implemented parts of the specification as described in this post.

Should you wish to to be able to drag and drop .264 files into GraphStudio, run the registry scripts in the videoprocessing\Projects\Win32\Launch directory. Unfortunately these have only been tested on Windows 7.

Improvements/suggestions/corrections are of course welcome!

Tuesday, March 6, 2012

Introducing a DirectShow YUV source filter

This post introduces a YUV source filter that can be used to load standard YUV test sequences into the DirectShow environment. Certain YUV test sequences (such as the Foreman sequence pictured on the left) have become widely used by researchers and developers active in the video coding field and many of them are currently available here.

The YUV source filter is part of the Video Processing Project and the source can be downloaded and reused under a BSD license.

Once registered with the OS (regsvr32), the YUV source filter appears under the DirectShow filters category in GraphStudio.

Once inserted into the graph, the user must select the YUV file in the IFileSourceFilter dialog:

The output pin of the YUV source filter can then be rendered.

Since the YUV file format of the test sequence videos contains no information regarding image dimensions and framerate, these must be manually configured using the property page of the filter. In the case where these are not configured correctly, the application may of course crash.

An attempt has been made to auto-configure the filter using a naming convention: If the filename contains CIF, QCIF, or a string of the form <<width>>x<<height>>, the dimensions of the filter are automatically configured. This approach may be refined/improved at a later stage if required.

Color conversion in DirectShow

In this post we will look at various aspects of writing DirectShow color conversion filters. In our Video Processing Project we released a set of filters that convert between the RGB24 to YUV420 Planar color formats. These filters were originally written to convert RGB to YUV for the purpose of encoding video as H.263.

During a later stage of the project, we undertook some compliance testing to make sure that the converted YUV format is compatible with the MS YUV formats i.e. the filters would be regarded as compliant if the YUV to RGB conversion filter is interchangeable with the standard AVI decompressor that is usually inserted by DirectShow. Further, our filters needed to be able to convert the standard test sequence videos from YUV420 to RGB. In particular, our RGB to YUV converter outputs MEDIASUBTYPE_I420 (which is the format that the test sequence videos use).

Getting the filters to be compatible with with both the MS formats and the test sequence formats required us to add a chrominance offset parameter in the filter, that allows the mapping into the unsigned short range [0, 255].

Further, we noticed that we needed to flip the image during color conversion.

As a final compatibility test, the video was dumped to file:

Then, using the YUV source filter (also available in the video processing project) we rendered the following graph:

Once the AVI decompressor was able to output our YUV format properly, the filters were regarded as compatible:

As an aside note, the color conversion filters allow one to configure the chrominance offset and the flipping of the image via their property pages. Both these properties can be changed while the graph is running.