Lay-off badges

The Israeli hi-tech market inspired me...

Showing video with Qt toolbox and ffmpeg libraries

I recently had to build a demo client that shows short video messages for Ubuntu environment.
After checking out GTK+ I decided to go with the more natively OOP Qt toolbox (GTKmm didn't look right to me), and I think i made the right choice.

So anyway, I have my video files encoded in some unknown format and I need my program to show them in a some widget. I went around looking for an exiting example, but i couldn't find anything concrete, except for a good tip here that led me here for an example of using ffmpeg's libavformat and libavcodec, but no end-to-end example including the Qt code.

The ffmpeg example was simple enough to just copy-paste into my project, but the whole painting over the widget's canvas was not covered. Turns out painting video is not as simple as overriding paintEvent()...

Firstly, you need a separate thread for grabbing frames from the video file, because you won't let the GUI event thread do that.
That makes sense, but when the frame-grabbing thread (I called VideoThread) actually grabbed a frame and inserted it somewhere in the memory, I needed to tell the GUI thread to take that buffered pixels and paint them over the widget's canvas.

This is the moment where I praise Qt's excellent Signals/Slots mechanism. So I'll have my VideoThread emit a signal notifying some external entity that a new frame is in the buffer.
Here's a little code:

void VideoThread::run() {
/*
... Initialize libavformat & libavcodec data structures.
You can see it in the example i referred to before
*/

// Open video file
if(av_open_input_file(&pFormatCtx,
"lala.avi",
NULL, 0, NULL)!=0)
return -1; // Couldn't open file

// Retrieve stream information
if(av_find_stream_info(pFormatCtx)<0)
return -1; // Couldn't find stream information

// Find the first video stream ...

// Get a pointer to the codec context for the video
// stream...

// Find the decoder for the video stream...

// Open codec...

// Allocate video frame
pFrame=avcodec_alloc_frame();

// Allocate an AVFrame structure
pFrameRGB=avcodec_alloc_frame();
if(pFrameRGB==NULL)
return -1;

int dst_fmt = PIX_FMT_RGB24;
int dst_w = 160;
int dst_h = 120;

// Determine required buffer size and allocate buffer
numBytes = avpicture_get_size(dst_fmt, dst_w, dst_h);
buffer = new uint8_t[numBytes + 64];

//put a PPM header on the buffer
int headerlen = sprintf((char *) buffer,
"P6\n%d %d\n255\n",
dst_w, dst_h);

_v->buf = (uchar*)buffer;
_v->len = avpicture_get_size(dst_fmt,dst_w,dst_h) +
headerlen;

// Assign appropriate parts of buffer to image planes
// in pFrameRGB...

// I use libswscale to scale the frames to the required
// size.
// Setup the scaling context:
SwsContext *img_convert_ctx;
img_convert_ctx = sws_getContext(
pCodecCtx->width, pCodecCtx->height,
pCodecCtx->pix_fmt,
dst_w, dst_h, dst_fmt,
SWS_BICUBIC, NULL, NULL, NULL);

// Read frames and notify
i=0;
while(av_read_frame(pFormatCtx, &packet)>=0)
{
// Is this a packet from the video stream?
if(packet.stream_index==videoStream)
{
// Decode video frame
avcodec_decode_video(pCodecCtx,
pFrame,
&frameFinished,
packet.data,
packet.size);

// Did we get a video frame?
if(frameFinished)
{
// Convert the image to RGB
sws_scale(img_convert_ctx,
pFrame->data,
pFrame->linesize,
0,
pCodecCtx->height,
pFrameRGB->data,
pFrameRGB->linesize);

emit frameReady();

//My video is 5FPS so sleep for 200ms.
this->msleep(200);
}
}

// Free the packet that was allocated by
// av_read_frame
av_free_packet(&packet);
}

// Free the RGB image
delete [] buffer;
av_free(pFrameRGB);

// Free the YUV frame
av_free(pFrame);

// Close the codec...

// Close the video file...
} //end VideoThread::run

Ok so I have a frame-grabber that emits a frameReady signal everytime the buffer is full and ready for painting.

A couple of things to notice:

• I convert the image format to PIX_FMT_RGB24 (avcodec.h), which is required by Qt's QImage::fromData() method.
• I scale the image using ffmpeg's libswscale. All conversion/scaling methods inside libavcodev are deprecated now.
  But it's fairly simple, here's a good example. Just remember you need a sws_getContext and then sws_scale.
• I totally disregard actual frame rate here, I just sleep for 200ms because i know my file is 5FPS. For a (far-) more sophisticated way to get the FPS, very important if this is not a constant frame-rate video, you can find here.
• I don't cover audio in this example, although the mechanism to extract it from the file exists... you just need to grabe the audio stream's frame. For playing audio you also need some Qt-external library. In a different project I used SDL very easily, here's an example online.

Now, for painting over the widget.
This is fairly easy:

void VideoWidget::paintEvent(QPaintEvent * e) {
QPainter painter(this);

if(buf) {
QImage i = QImage::fromData(buf,len,"PPM");
painter.drawImage(QPoint(0,0),i);
}
}

Two things to note:

• The widget needs to be given the pointer to the video frame buffer (buf).
• The frame buffer needs to be in a PPM format. That means it needs to get a PPM header, which looks something like this: "P6\n320 240\n255\n", and then all the pixels in 3-byte per-pixel format (RGB24). You can see that i take care of that in the previous code block.

Finally we need to orchestrate this whole mess.
So in my GUI-screen class I do:

....
vt = new VideoThread();
connect(vt,SIGNAL(frameReady()),this,SLOT(updateVideoWidget()));
vt->start();
....

And:

void playMessage::updateVideoWidget() {
videoWidget->repaint(); //or update().
}

This will make the widget repaint on each frame ready.

Note:

• In this example I don't take care of multi-threading issues. Since the GUI and the ffmpeg decoder threads share a memory buffer, I should probably have a mutex to protect it. It's a classic producer-consumer problem.
• Performance wise, Qt's paint mechanism is by far the worst way to go when displaying video... but it's great for a quick-and-dirty solution (I only needed 5fps). A more performance favorable solution will probably be using an overlay block and frame-serving with SDL.
  

Enjoy!
Roy.

Thinking of new services...

הלימודים עלינו

לא בטוח שלימודים בחינם זה הכוון הנכון...

Take that - Toyota Prius!

Inspired by this article in the Telegraph.

The strange case of the BackgroundWorker and the disappearing exception

I was recently building a simple GUI in .NET to operate an algorithm as part of a school project, and I encountered a weird problem using BackgroundWorkers. I spent a lot of time debugging it, mainly because the code seemed to be perfect (which was true) but the run-time behavior was so strange...

Anyway, to make my algorithm as weakly-coupled as possible decided not to use 'BackgroundWorker.ReportProgress', because then my algorithm will have to know what a BackgroundWorker is...
I decided to actually fire my own event whenever I wanted to report on the algorithm progress (which is rather lenghty). So I defined my delegate and event inside my one-function class that runs the algorithm:

delegate System::Void AlgProgressDelegate(unsigned int done, unsigned outOf);

ref class Algorithm {
event AlgProgressDelegate^ AlgProgress;

public: void StartAlgorithm() {
...
AlgProgress(done, outOf);
...
}
}

Meanwhile, back in my UI...

public: System::Void OnProgress(unsigned int done,unsigned int outOf) {
progressBar1->Maximum = outOf;
progressBar1->Value = done;
progressBar1->Invalidate();
if((done % 100) == 0) {
label4->Text = String::Format("{0}/{1}",done,outOf);
label4->Invalidate();
}
}

Of course I registered my form to recieve these events:

Algorithm a();
a.AlgProgress += gcnew AlgProgressDelegate(this, &Form1::OnProgress);

Naively, I thought this would run perfectly, as I could see nothing wrong in the code, can you?
Little did i know that .NET has a few surprises when it comes to updating UI controls from the event thread...

I ran my little program, expecting the nice progress bar to fill up as the algorithm progresses, but instead - the algorithm would just end abruptly, very close to the beginning, without any exceptions or anything.

So I started to debug the algorithm function to see whats wrong: Step over, Step over, Step over and WHAM! the instruction pointer just leaps 20 lines ahead totally disregarding for loops and ifs in the middle.. What the hell is going on?

The mysterious jump ahead occured over the "AlgProgress()" function call. Some line-remarking within that function revealed that the offending line is the "progressBar1->Maximum" property set. Strangely enough, the "progressBar1->Value" prop set was executed just fine... This is weird stuff.

I don't remember exactly how, but finally I came up with a thought that maybe an exception is thrown and no one is catching it.. So I wapped the inside of my "AlgProgress" function with a try-catch block.
And lo and behold, indeed an exception is thrown. A nice "System.InvalidOperationException:
Cross-thread operation not valid: Control 'progressBar1' accessed from a thread other than the thread it was created on."
But the thing is, I was kinda getting used to idea that when exceptions are thrown - something catches them, and if nothing catches them the run-time is somehow stopped... so us developers/users will be anounced of said exception and be able to deal with it/bash the screen into a pulp.
But I guess Microsoft couldn't go without a good Voodoo in their framework, so they put in the "Disappearing exception" concept.

From here on it was quite simple to find a solution. I used the "CheckForIllegalCrossThreadCalls" property of the progress bar, although the better solution would be to have the UI thread change the progress bar by using "Invoke(delegate..)", as explained here.

To summerize: When in doubt - try & catch.

DIY: Cow skin pattern for your GIMPed cows

I wanted to share with you a (very simple and short) method for creating cow-skin patterns, totally random each time, for your GIMPin needs.
This can actually work the exact same way in GIMP, PS or Paint.NET.

I will demonstrate with GIMP, but it is so simple you can do it in any of the aforementioned programs.

So here goes:

1. render some clouds

You may want to smooth it a bit maybe if you get a grainy clouds render.

2. set levels

Black level and white level to 127.

3. and there you go - cow skin:
Now you can cut it up to fit your cow...

Honest typo

Where all the babes hang

* Made with Paint.NET

Paint.NET is cool

Today I stumbled upon Paint.NET.
It seems like an awesome tool for quickly drawing stuff.

I also read that some compare it to The GIMP (which I adore...). I found Paint.NET is easier for simpler tasks that the GIMP can drive you nuts while trying to do them.
In the GIMP i find myself using the select tool A LOT. You want to draw a circle? select and fill. You want a simple rectangle? select and fill...
Sometimes I wish GIMP was more like plain ol' Paint...

Here I think Paint.NET really excels. It keeps the simplicity of Paint, but adding the open-sourceness and variety of GIMP.

Driving calmly

Hebrew version:

Slave-co software

Recruitment process in Slave-co

MOV eax, ebx

English version:

Note to non-hebrew speakers: Hamster and Register (registers like in the CPU) are the same word in hebrew.

Hi-Tech survivor

Checking out the babes

Language disclaimer

I would like to apologize right away before all of you for my poor english skills, as english is not my mother-tounge. Furthermore, this blog meant to be just another exercise for my written/spoken english.

Thanks for understanding.
Roy.