Pitfalls and Errors When Using AVAssetWriter to Save the OpenGL Backbuffer Into a Video.

For my current game I need to record the OpenGL back buffer of GLKView into a video using AVAssetWriter.

I will not go over all the code to do this(which is not that much).

However, I will go quickly over some of the pitfalls that might make you go scratching your head when you try to create a video on iOS.

The first thing to consider is that AVAssetWriter might fail without throwing an exception and without returning any value pointing that out.

To test if there was an error within AVAssetWriter methods you need to read the status and error properties of AVAssetWriter.

You can print the status and error like so:

NSLog(@"%ld %@", (long)videoWriter.status, videoWriter.error);

The first pitfall I encountered is that after startWritting my AVAssetWriter status change to AVAssetWriterStatusFailed.

(Or the number 3)

The reason I was getting this error is because I didn’t delete the file I tried to write into from the previous run session.

That was an easy error but after that I was getting an error when trying to append the first(and following) CVPixelBuffer.

I got AVAssetWriterStatusFailed again but the error value was Unknown Error or the OSStatus was -12902.

After a while I have found out that nothing was wrong with my code except for the fact that something was wrong with the resolution.

Since I was running my app on an iPad3 the resolution was 2048×1536. I don’t know why but the codec or AVAssetWriter couldn’t handle this resolution.

Simply reducing the resolution to something smaller allowed me to write the video without getting any error.

 

For the sake of completion here is part of the code I was using:

 

- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect
{
    RenderGame();
    @synchronized(self) {
        if (Recording)
        {
 //           sample = [self sampleBufferFromCGImage:[self snapshotRenderBuffer]];
            GLubyte * data = [self snapshotRenderBufferToData];
            CVPixelBufferRef pixelBuffer = [self pixelBufferFromBytes: &data withSize:CGSizeMake(videoWidth, videoHeight) withSourceSize: CGSizeMake(screenWidth, screenHeight)];
            if (adaptor.assetWriterInput.readyForMoreMediaData)
            {
                RecordFrame++;
                [adaptor appendPixelBuffer:pixelBuffer withPresentationTime:CMTimeMake(RecordFrame*10, 600)];
            }
            //                [writerInput appendSampleBuffer: sample];
            CFRelease(pixelBuffer);
            free (data);
            NSLog(@"%ld %@", (long)videoWriter.status, videoWriter.error);
            //            NSLog(@"%ld", (long)videoWriter.status);
        }
    }
}
-(IBAction)selectRecord:(id)sender
{
    if (DoneRecord)
        return;
    @synchronized(self) {
        NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
        NSString *documentsDirectory = [paths objectAtIndex:0];
        NSString *sourcePath = [documentsDirectory stringByAppendingPathComponent:@"video.mov"];
        if (Recording)
        {
            [writerInput markAsFinished];
            NSLog(@"%ld", (long)videoWriter.status);
            [videoWriter finishWritingWithCompletionHandler:^{
                UISaveVideoAtPathToSavedPhotosAlbum(sourcePath,nil,nil,nil);
            }];
            DoneRecord = YES;
        }
        else
        {
            videoWidth = screenWidth/2;
            videoHeight = screenHeight/2;
            NSError * error = nil;
            NSFileManager *fileManager = [NSFileManager defaultManager];
            BOOL success = [fileManager removeItemAtPath:sourcePath error:&error];
            if (success) {

            }
            else
            {
                NSLog(@"Could not delete file -:%@ ",[error localizedDescription]);
            }
            RecordFrame = 0;
            videoWriter = [[AVAssetWriter alloc] initWithURL:
                                          [NSURL fileURLWithPath:sourcePath] fileType:AVFileTypeQuickTimeMovie
                                                                      error:&error];
            NSLog(@"%ld", (long)videoWriter.status);
            NSParameterAssert(videoWriter);

            NSDictionary *videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
                                           AVVideoCodecH264, AVVideoCodecKey,
                                           [NSNumber numberWithInt:videoWidth], AVVideoWidthKey,
                                           [NSNumber numberWithInt:videoHeight], AVVideoHeightKey,
                                           nil];
            writerInput = [AVAssetWriterInput
                                                assetWriterInputWithMediaType:AVMediaTypeVideo
                                                outputSettings:videoSettings]; //retain should be removed if ARC

            NSParameterAssert(writerInput);
            NSParameterAssert([videoWriter canAddInput:writerInput]);
            writerInput.expectsMediaDataInRealTime = YES;
            [videoWriter addInput:writerInput];

            adaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerInput sourcePixelBufferAttributes:nil];

            [videoWriter startWriting];
//            NSLog(@"%ld %@", (long)videoWriter.status, videoWriter.error);
            [videoWriter startSessionAtSourceTime:kCMTimeZero];
        }
        Recording = !Recording;
    }
}

Location 0? location -1? glGetUniformLocation, C++ and bugs. (Android\iOS GLES 2)

In OpenGLES 2 glGetUniformLocation receives the program id and a string as parameters. It then attempts to return a location int that can be used to set uniform GLSL shader variables.

If the variable is found it will return a 0 or positive value. If it fails to find the uniform variable it will return -1.

In C++ we should initialize the location ints in the ctr. If we don’t initialize the locations we might have garbage values when in Release mode.

Using the locations with garbage values might overwrite uniform variables with values we did not intend them to have.

So what we should initialize the locations with? One might think that 0 is a good value to initialize but it is not.

Remember! 0 is a valid shader uniform variable location. If we set all the locations to 0 we might overwrite the uniform variable at location 0.

We should initialize the location ints with -1.

We should do this because -1  is the value that is returned in case the uniform variable was not found and setting a value at location -1 will be ignored.

GLSL(OpenGL Shader Language) compilation bug with for loop on Adreno 205, Android.

One of the biggest advantages of the OpenGL API specification is that OpenGL is language agnostic.

That means it can be implemented on almost any programming language which makes it a very portable library.

However, there is a serious issue with OpenGL. It’s shader language(GLSL) has no specification for compilation. You can’t rely on binary files of compiled shaders to work on different devices.

Not only that but compiling the GLSL source code while running the app on different devices might produce different results or even silent bugs(Depending on driver implementation).

My game Shotgun Practice was running perfectly on my device(Galaxy Note N7000) but didn’t work on my friend’s device(HTC Desire Z).

On my friend’s ‘HTC Desire Z‘ Android device with the ‘Adreno 205‘ GPU it had graphics artifacts.

After quite some tests I found that a specific shader was the culprit. That shader was the vertex shader of skinned objects.

It took me a lot of tests because the driver for HTC Desire Z didn’t report any error or warning upon compiling and validating the skinning shader.

Eventually it boiled down to the part of code that transforms the vertices with the relevant bones.

Doesn’t work on HTC Desire Z

for(int i = 0; i < 4; ++i)
{
	mat4 m = BoneTransform[Index[i]];
	posOut += (w[i]*m*vec4(position, 1.0)).xyz;
	normalOut += (w[i]*m*vec4(normal, 0.0)).xyz;
}

Works on HTC Desire Z

mat4 m = BoneTransform[Index[0]];
posOut += (w[0]*m*vec4(position, 1.0)).xyz;
normalOut += (w[0]*m*vec4(normal, 0.0)).xyz;
m = BoneTransform[Index[1]];
posOut += (w[1]*m*vec4(position, 1.0)).xyz;
normalOut += (w[1]*m*vec4(normal, 0.0)).xyz;
m = BoneTransform[Index[2]];
posOut += (w[2]*m*vec4(position, 1.0)).xyz;
normalOut += (w[2]*m*vec4(normal, 0.0)).xyz;
m = BoneTransform[Index[3]];
posOut += (w[3]*m*vec4(position, 1.0)).xyz;
normalOut += (w[3]*m*vec4(normal, 0.0)).xyz;

As you can see the code that doesn’t work has a ‘for loop’ and in the code that works I manually unrolled the ‘for loop’.

I also tested if the issue was that ‘mat4 m’ was inside the ‘for loop’ block or that using a hard coded number of iterations would cause a faulty loop unrolling.

Neither attempts worked. I don’t know exactly what is the driver issue with this but I was told you should use ‘for loops’ very cautiously in GLSL meant for mobile devices.

Conclusion

Beware of ‘for loops’ and generally branching in GLSL meant for mobile devices.

But even worse, some drivers(hopefully only old devices) might not warn you that the shader isn’t going to work on the device even though it passed all the validation.