Live RTSP streaming using the Video Processing Project

In this post, we take a look at how we can stream live video and audio from a web cam using the video processing project.

We will use the following VPP filters:

• CSIR VPP Scale Filter
• CSIR VPP H264 Encoder
• CSIR VPP RTSP Sink Filter
• CSIR VPP RTSP Source Filter

Additionally you will need either an AMR or AAC encoders/decoders. Igor Janos from Monogram made some available on his blog a couple of years ago.

You will also need GraphEdit, GraphStudio or GraphStudioNext (recommended) to construct the graphs.

The VPP RTSP filters use the Live555 Streaming Media libraries which implement the RTSP/RTP/RTCP stacks. 

To stream audio and video from a webcam or live capture source setup a graph as shown in Figure 1.

Figure 1: RTSP Sink Graph

It is important to limit the source resolution (either via IAMStreamConfig or via the VPP Scale Filter ) and framerate (either via IAMStreamConfig or via the VPP Frame Skipping Filter) as this affects both the bitrate and the time required to encode the image. The current VPP Encoder can comfortably encode CIF video at 15 frames a second when in Mode 1. Mode 1 means that the encoder limits the bits per frame to the specified Frame Bit Limit parameter. In this example we have selected 5000 bits per frame. At e.g. 10 frames a second, this equals 50kb/s. Note that the filters must be configured before the graph is played.

Once the graph is played, start another instance of GraphStudioNext.

Figure 2: RTSP Source Graph

Insert the CSIR VPP RTSP Source Filter into the graph which will bring up a dialog (see Figure 2) allowing you to input the RTSP URI which will be of the format rtsp://<<ip>[:<<rtsp_port>>]/live. The VPP RTSP Sink Filter creates a session named "live". Finally render the output pins of the RTSP Source Filter which results in a graph similar to the one in Figure 3. The decoders will vary depending on what is installed on the machine.

Figure 3: Rendered RTSP Source Graph 

 Or alternatively use VLC. "Ctrl + N" brings up a dialog allowing you to enter the RTSP URI as shown in Figure 4.

Figure 4: RTSP with VLC

The Caching parameter should be set sufficiently big (at least 500ms, or even better 1000ms).
When this parameter is too small, VLC struggles to synchronise the audio and video resulting in many frames being discarded. This can be seen on the logging window if the verbosity is high enough.

Using the FrameSkippingFilter to drop the framerate

Sometimes it is desirable to reduce the framerate of a video. In the FrameSkippingFilter the user can select the value n, where a frame will be dropped every n frames. This makes it simple to halve the framerate.
This filter could easily be extended to perform more complex dropping schemes, but does not currently support that.

An example media pipeline shows the results:

A FramerateDisplayFilter has been inserted before the FrameSkippingFilter, and one after, to illustrate the effect.

As can be seen in the screen capture, the framerate has been halved.

Using the FramerateDisplayFilter

The FramerateDisplayFilter is a filter that is useful for check what framerate you are achieving in a live multimedia pipeline. In a live multimedia pipeline, it could be that an encoder is not compressing the media fast enough. It is sometimes desirable to know what frame rate is achievable using different resolutions, encoder modes, etc. It could also be useful in a system where one performs dynamic bitrate adaption.

The FramerateDisplayFilter uses a moving average over the last 50 samples to calculate the average and renders the estimate on top of the video using GDI+.


The FramerateDisplayFilter inherits CTransInPlaceFilter and currently has the following configuration options:

• mode: time-stamp or system-time
• X: x- position of the estimate (This can be off the screen)
• Y: y- position of the estimate (This can be off the screen)

In time-stamp mode, the timestamps of the actual media samples is used in the average calculation.

In system-time mode, the time as the  sample passes through the filter, is used in the average calculation.

Depending on the pipeline, there may be a minor or larger difference between the two.

As per usual, all settings are programmatically configurable using the COM ISettingsInterface interface.

On a side note, if anyone is interested in contributing to the development of this filter, capabilties to set the font, font-color, etc via the property page are still required.

Building the VPP for 64-bit Windows

One difference between targeting 32 and 64 bit Windows is the VC compiler that is used. Under C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin one can find the directories containing the tools needed to target the various environments. More about that can be read on MSDN.

Other things that differ are the include and linker paths. Since we are launching VS with out custom batch file,  we need to update our environment to target Windows 64. In VsVersion, we need to set TARGET=X86_64.

In 64-bit builds a different configuration is used in Visual Studio:

Additionally, one has to be sure that x64 is set as the target for static libraries built in the project.

If any mismatches are detected between x86 and x64, the compilation will fail with the following error message:

fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'

Another couple of tips if you run into compilation errors is to 

• make sure that you clean the solution
• Check that the correct lib and bin directories are being used
• Doublecheck the obj files that are created during compilation are 64-bit. You can do this by running dumpbin /HEADERS somefile.obj

If all went well, you should be able to load the 64-bit version of the filter in the 64-bit version of GraphStudio. You might have to make some tweaks to the paths depending on the platform you're building on, and on the target platform.

Using the VPP H.264 DirectShow filter

In this post, we take a quick look at how the VPP H.264 filter can be used using GraphEdit. The H.264 filter accepts both the RGB24 as well as the I420 media types making it compatible with the VPP YUV source filter. The filter can be configured via the property page. Currently, the property page contains the options shown in Figure 1. More options will be added in the future.

Figure 1

At the bottom of the property page, one can tick a checkbox in order to use the standard Microsoft H.264 decoder that comes stock with Windows 7. If the box is unchecked, the VPP decoder will be used, which has a custom media type. Figure 2 shows a graph in which the stock MS H.264 decoder is used to decode the video.

Figure 2

Currently, there are two Modes of Operation that can be selected, mode 0 and mode 1. In mode 0, the Quality of the video can be configured. Valid values lie in the range [0 - 51] with 0 having the best quality and 51 the worst. Figure 2 shows a graph in which the first encoder has Quality 0 and the second encoder Quality 35.

Figure 2

Alternatively, the frame bit limit can be set by setting Mode of Operation equal to 1 which effectively limits the bitrate of the media stream. The frame bit limit is, as the name suggests, measured in bits per frame. That means that to achieve a rate of 128 kb/s with a source video of 10 frames a second, one would need to set the frame bit limit to 12,8 * 1024 = 13107 bits per frame. In a live media pipeline, the VPP Framerate estimator filter may be useful to measure the approximate framerate of the video source.

The other option of interest is the I-frame Period which creates periodic IDR-frames i.e. every n-th frame, will be encoded as an IDR frame.

All parameters are programmatically settable by using the COM ISettingsInterface which all all/most of the VPP DirectShow filters inherit, but that is a post for another day...

P.S. The Notify on I-frame and Prepend parameter sets are no longer used and will be removed in the next release of the software.

Implementing a DirectShow H264 source filter

After not finding a suitable DirectShow source filter able to render raw H.264 files, we decided to roll our own one (available at the Video Processing Project. I'm sure that many developers have written one of these and perhaps it's time to stop reinventing the wheel. Should anyone want/like to contribute improvements/extensions to this filter, please drop us a line.

The H.264 reference software allows one to take a YUV file and encode it into a .264 file format. These .264 files consist of a sequence of NAL units, each prepended with a start code (0x00000001). A source filter would thus have to read one of these files, break it up into separate NAL units, and then pass one frame at a time to the decoder. Windows 7 features a built-in H.264 decoder.

The IFileSourceFilter interface is implemented to facilitate loading of .264 files. This causes GraphEdit/GraphStudio to display a dialog box in which one can select the desired .264 file.

One of the first steps in writing a source filter is to provide the correct output pin media type, that allows the DirectShow framework to render the graph. In this case, the MEDIA_SUBTYPE_H264 was selected since using it requires the least amount of effort. The implementation of GetMediaType looks as follows:


  HRESULT H264OutputPin::GetMediaType(CMediaType *pMediaType)
  {
    CAutoLock cAutoLock(m_pFilter->pStateLock());
    CheckPointer(pMediaType, E_POINTER);

    pMediaType->InitMediaType();    
    pMediaType->SetType(&MEDIATYPE_Video);
    pMediaType->SetSubtype(&MEDIASUBTYPE_H264);
    pMediaType->SetFormatType(&FORMAT_VideoInfo2);
    VIDEOINFOHEADER2* pvi2 = (VIDEOINFOHEADER2*)pMediaType->AllocFormatBuffer(
                                                                       sizeof(VIDEOINFOHEADER2));
    ZeroMemory(pvi2, sizeof(VIDEOINFOHEADER2));
    pvi2->bmiHeader.biBitCount = 24;
    pvi2->bmiHeader.biSize = 40;
    pvi2->bmiHeader.biPlanes = 1;
    pvi2->bmiHeader.biWidth = m_pFilter->m_iWidth;
    pvi2->bmiHeader.biHeight = m_pFilter->m_iHeight;
    pvi2->bmiHeader.biSize = m_pFilter->m_iWidth * m_pFilter->m_iHeight * 3;
    pvi2->bmiHeader.biSizeImage = DIBSIZE(pvi2->bmiHeader);
    pvi2->bmiHeader.biCompression = DWORD('1cva');
    const REFERENCE_TIME FPS_25 = UNITS / 25;
    pvi2->AvgTimePerFrame = FPS_25;
    SetRect(&pvi2->rcSource, 0, 0, m_pFilter->m_iWidth, m_pFilter->m_iHeight);
    pvi2->rcTarget = pvi2->rcSource;
    pvi2->dwPictAspectRatioX = m_pFilter->m_iWidth;
    pvi2->dwPictAspectRatioY = m_pFilter->m_iHeight;
    return S_OK;
  }

This code is sufficient to allow DirectShow to insert the Windows H.264 decoder into the pipeline. Here the width and height seem to be of little importance since they are in any case communicated in the (H.264) Sequence Parameter Set. The parameter sets are found by scanning the .264 file for the appropriate NAL units. The NAL unit type can be extracted from the NAL unit header as follows:

  unsigned char uiNalUnitType = nalUnitHeader & 0x1f;


Sequence parameter sets have value 7, picture parameter sets 8, and IDR frames value 5. One typically needs to pass these to the decoder before other encoded frames.

In closing, the filter currently is also able to output a custom media type, namely MEDIASUBTYPE_H264M. This makes it easy to test our own H.264 decoder filter. In the property pages of the source filter, one can select what the output media type of the H.264 source filter should be. It should be noted, that our H.264 decoder has limitations regarding the implemented parts of the specification as described in this post.

Should you wish to to be able to drag and drop .264 files into GraphStudio, run the registry scripts in the videoprocessing\Projects\Win32\Launch directory. Unfortunately these have only been tested on Windows 7.

Improvements/suggestions/corrections are of course welcome!

Introducing a DirectShow YUV source filter

This post introduces a YUV source filter that can be used to load standard YUV test sequences into the DirectShow environment. Certain YUV test sequences (such as the Foreman sequence pictured on the left) have become widely used by researchers and developers active in the video coding field and many of them are currently available here.

The YUV source filter is part of the Video Processing Project and the source can be downloaded and reused under a BSD license.

Once registered with the OS (regsvr32), the YUV source filter appears under the DirectShow filters category in GraphStudio.

Once inserted into the graph, the user must select the YUV file in the IFileSourceFilter dialog:

The output pin of the YUV source filter can then be rendered.

Since the YUV file format of the test sequence videos contains no information regarding image dimensions and framerate, these must be manually configured using the property page of the filter. In the case where these are not configured correctly, the application may of course crash.

An attempt has been made to auto-configure the filter using a naming convention: If the filename contains CIF, QCIF, or a string of the form <<width>>x<<height>>, the dimensions of the filter are automatically configured. This approach may be refined/improved at a later stage if required.

Color conversion in DirectShow

In this post we will look at various aspects of writing DirectShow color conversion filters. In our Video Processing Project we released a set of filters that convert between the RGB24 to YUV420 Planar color formats. These filters were originally written to convert RGB to YUV for the purpose of encoding video as H.263.

During a later stage of the project, we undertook some compliance testing to make sure that the converted YUV format is compatible with the MS YUV formats i.e. the filters would be regarded as compliant if the YUV to RGB conversion filter is interchangeable with the standard AVI decompressor that is usually inserted by DirectShow. Further, our filters needed to be able to convert the standard test sequence videos from YUV420 to RGB. In particular, our RGB to YUV converter outputs MEDIASUBTYPE_I420 (which is the format that the test sequence videos use).

Getting the filters to be compatible with with both the MS formats and the test sequence formats required us to add a chrominance offset parameter in the filter, that allows the mapping into the unsigned short range [0, 255].

Further, we noticed that we needed to flip the image during color conversion.

As a final compatibility test, the video was dumped to file:

Then, using the YUV source filter (also available in the video processing project) we rendered the following graph:

Once the AVI decompressor was able to output our YUV format properly, the filters were regarded as compatible:

As an aside note, the color conversion filters allow one to configure the chrominance offset and the flipping of the image via their property pages. Both these properties can be changed while the graph is running.