我如何绘制在平面/ycbcr/420f/yuv/NV12/非rgb的CVPixelBufferRef上?

问题描述

我从系统API收到了一个CMSampleBufferRef，其中包含的CVPixelBufferRef不是RGBA(线性像素).缓冲区包含平面像素(例如420f aka kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange aka yCbCr aka YUV).

I have received a CMSampleBufferRef from a system API that contains CVPixelBufferRefs that are not RGBA (linear pixels). The buffer contains planar pixels (such as 420f aka kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange aka yCbCr aka YUV).

我想先修改此视频数据，然后再将其发送到VideoToolkit以编码为h264(绘制一些文本，覆盖徽标，旋转图像等)，但是我想希望它高效且实时. Buuuut平面图像数据看起来像suuuper一样杂乱无章-有色度平面和亮度平面，它们的大小不同，并且...在字节级别上使用它似乎是很多工作.

I would like to modify do some manipulation of this video data before sending it off to VideoToolkit to be encoded to h264 (drawing some text, overlaying a logo, rotating the image, etc), but I'd like for it to be efficient and real-time. Buuuut planar image data looks suuuper messy to work with -- there's the chroma plane and the luma plane and they're different sizes and... Working with this on a byte level seems like a lot of work.

我可能会使用CGContextRef并仅在像素上方绘制，但是据我所知，它仅支持RGBA像素.关于如何减少数据复制，减少代码行的任何建议?

I could probably use a CGContextRef and just paint right on top of the pixels, but from what I can gather it only supports RGBA pixels. Any advice on how I can do this with as little data copying as possible, yet as few lines of code as possible?

推荐答案

CGBitmapContextRef只能绘制为32ARGB之类的东西，正确.这意味着您将要创建ARGB(或RGBA)缓冲区，然后找到一种方法来非常快速地将YUV像素转移到此ARGB曲面上.此食谱包括使用CoreImage，通过池的自制CVPixelBufferRef，引用您的自制像素缓冲区的CGBitmapContextRef，然后重新创建类似于您的输入缓冲区但引用您的输出像素的CMSampleBufferRef.换句话说，

CGBitmapContextRef can only paint into something like 32ARGB, correct. This means that you will want to create ARGB (or RGBA) buffers, and then find a way to very quickly transfer YUV pixels onto this ARGB surface. This recipe includes using CoreImage, a home-made CVPixelBufferRef through a pool, a CGBitmapContextRef referencing your home made pixel buffer, and then recreating a CMSampleBufferRef resembling your input buffer, but referencing your output pixels. In other words,

1. 将传入的像素提取到CIImage中.
2. 使用要创建的像素格式和输出尺寸创建CVPixelBufferPool.您不想实时创建没有池的CVPixelBuffer:如果生产者速度太快，您将用光内存；您将碎片化RAM，因为您将不会重复使用缓冲区；这是浪费时间.
3. 使用将在缓冲区之间共享的默认构造函数创建一个CIContext.它不包含外部状态，但是文档表明在每个框架上重新创建它都是非常昂贵的.
4. 在传入帧上，创建一个新的像素缓冲区.请确保使用分配阈值，以免出现RAM使用率失控的情况.
5. 锁定像素缓冲区
6. 创建引用像素缓冲区中字节的位图上下文
7. 使用CIContext将平面图像数据渲染到线性缓冲区中
8. 在CGContext中执行特定于应用程序的绘图！
9. 解锁像素缓冲区
10. 获取原始样本缓冲区的计时信息
11. 通过询问像素缓冲区的确切格式来创建CMVideoFormatDescriptionRef
12. 为像素缓冲区创建一个示例缓冲区.完成！

1. Fetch the incoming pixels into a CIImage.
2. Create a CVPixelBufferPool with the pixel format and output dimensions you are creating. You don't want to create CVPixelBuffers without a pool in real time: you will run out of memory if your producer is too fast; you'll fragment your RAM as you won't be reusing buffers; and it's a waste of cycles.
3. Create a CIContext with the default constructor that you'll share between buffers. It contains no external state, but documentation says that recreating it on every frame is very expensive.
4. On incoming frame, create a new pixel buffer. Make sure to use an allocation threshold so you don't get runaway RAM usage.
5. Lock the pixel buffer
6. Create a bitmap context referencing the bytes in the pixel buffer
7. Use CIContext to render the planar image data into the linear buffer
8. Perform your app-specific drawing in the CGContext!
9. Unlock the pixel buffer
10. Fetch the timing info of the original sample buffer
11. Create a CMVideoFormatDescriptionRef by asking the pixel buffer for its exact format
12. Create a sample buffer for the pixel buffer. Done!

这是一个示例实现，我选择32ARGB作为要使用的图像格式，因为CGBitmapContext和CoreVideo都喜欢在iOS上使用:

Here's a sample implementation, where I have chosen 32ARGB as the image format to work with, as that's something that both CGBitmapContext and CoreVideo enjoys working with on iOS:

{
    CGPixelBufferPoolRef *_pool;
    CGSize _poolBufferDimensions;
}
- (void)_processSampleBuffer:(CMSampleBufferRef)inputBuffer
{
    // 1. Input data
    CVPixelBufferRef inputPixels = CMSampleBufferGetImageBuffer(inputBuffer);
    CIImage *inputImage = [CIImage imageWithCVPixelBuffer:inputPixels];

    // 2. Create a new pool if the old pool doesn't have the right format.
    CGSize bufferDimensions = {CVPixelBufferGetWidth(inputPixels), CVPixelBufferGetHeight(inputPixels)};
    if(!_pool || !CGSizeEqualToSize(bufferDimensions, _poolBufferDimensions)) {
        if(_pool) {
            CFRelease(_pool);
        }
        OSStatus ok0 = CVPixelBufferPoolCreate(NULL,
            NULL, // pool attrs
            (__bridge CFDictionaryRef)(@{
                (id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32ARGB),
                (id)kCVPixelBufferWidthKey: @(bufferDimensions.width),
                (id)kCVPixelBufferHeightKey: @(bufferDimensions.height),
            }), // buffer attrs
            &_pool
        );
        _poolBufferDimensions = bufferDimensions;
        assert(ok0 == noErr);
    }

    // 4. Create pixel buffer
    CVPixelBufferRef outputPixels;
    OSStatus ok1 = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(NULL,
        _pool,
        (__bridge CFDictionaryRef)@{
            // Opt to fail buffer creation in case of slow buffer consumption
            // rather than to exhaust all memory.
            (__bridge id)kCVPixelBufferPoolAllocationThresholdKey: @20
        }, // aux attributes
        &outputPixels
    );
    if(ok1 == kCVReturnWouldExceedAllocationThreshold) {
        // Dropping frame because consumer is too slow
        return;
    }
    assert(ok1 == noErr);

    // 5, 6. Graphics context to draw in
    CGColorSpaceRef deviceColors = CGColorSpaceCreateDeviceRGB();
    OSStatus ok2 = CVPixelBufferLockBaseAddress(outputPixels, 0);
    assert(ok2 == noErr);
    CGContextRef cg = CGBitmapContextCreate(
        CVPixelBufferGetBaseAddress(outputPixels), // bytes
        CVPixelBufferGetWidth(inputPixels), CVPixelBufferGetHeight(inputPixels), // dimensions
        8, // bits per component
        CVPixelBufferGetBytesPerRow(outputPixels), // bytes per row
        deviceColors, // color space
        kCGImageAlphaPremultipliedFirst // bitmap info
    );
    CFRelease(deviceColors);
    assert(cg != NULL);

    // 7
    [_imageContext render:inputImage toCVPixelBuffer:outputPixels];

    // 8. DRAW
    CGContextSetRGBFillColor(cg, 0.5, 0, 0, 1);
    CGContextSetTextDrawingMode(cg, kCGTextFill);
    NSAttributedString *text = [[NSAttributedString alloc] initWithString:@"Hello world" attributes:NULL];
    CTLineRef line = CTLineCreateWithAttributedString((__bridge CFAttributedStringRef)text);
    CTLineDraw(line, cg);
    CFRelease(line);

    // 9. Unlock and stop drawing
    CFRelease(cg);
    CVPixelBufferUnlockBaseAddress(outputPixels, 0);

    // 10. Timings
    CMSampleTimingInfo timingInfo;
    OSStatus ok4 = CMSampleBufferGetSampleTimingInfo(inputBuffer, 0, &timingInfo);
    assert(ok4 == noErr);

    // 11. VIdeo format
    CMVideoFormatDescriptionRef videoFormat;
    OSStatus ok5 = CMVideoFormatDescriptionCreateForImageBuffer(NULL, outputPixels, &videoFormat);
    assert(ok5 == noErr);

    // 12. Output sample buffer
    CMSampleBufferRef outputBuffer;
    OSStatus ok3 = CMSampleBufferCreateForImageBuffer(NULL, // allocator
        outputPixels, // image buffer
        YES, // data ready
        NULL, // make ready callback
        NULL, // make ready refcon
        videoFormat,
        &timingInfo, // timing info
        &outputBuffer // out
    );
    assert(ok3 == noErr);

    [_consumer consumeSampleBuffer:outputBuffer];
    CFRelease(outputPixels);
    CFRelease(videoFormat);
    CFRelease(outputBuffer);
}

这篇关于我如何绘制在平面/ycbcr/420f/yuv/NV12/非rgb的CVPixelBufferRef上?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！