How to realize rk3288 hevc/h265 HW decoder under linux

[youbainiao]

I would like the linux realize rk3288 hardware solution hevc/h265, because there is no linux platform library, so I would like to achieve by hybris, here are a few questions about it:
1, directly call hevc/h265 related libraries to implement, unlike ubuntu called as android's media to achieve, what the library needs? only libhevcdec.so and libvpu.so, or libvpu.so do not need?
2, where demo can refer to it? Can refer h264 call vpu achieve it?Or use a completely different way to call hevc/h265?

[RISCy Dude]

Freaktab member MAC_L1 has made great strides in this area. And his turnkey effort is here. You may have already seen these. He recently provided a script to make 15.04 or 15.04 SD-boot work with his bits too.

Welcome to Freaktab!

[youbainiao]

Thank you for your reply,I have read, but I still do not know how to call hevc / h265 decoding library(libhevcdec.so) directly that no android services need to be started in linux.

[MAC_L1]

just answered on firefly website, however this is the best place to answer as its freaktab is not device specific, so lets continue topic here:

hi youbainiao!

welcome to the dark world of hw acceleration for rockchip devices!
sorry for the long wait: busy cooking new machybris and mackodi packages...

1) you should use machybris (new release is upcoming) that enables access to the rockchip libs in android: libvpu and librkon2. librkon2 will load other libs like libhevcdec.so itself so no extra linking needed for you app. header files can be found in /usr/local/include/android/libon2/*, linking with -lvpu

2) i just made a github repo to share experiments/ examples/ demos. i added my first raw prototype i used to experiment with vpu (as is - didnt do any cleanup!) - it is based on an android vpudemo from chipspark that i stumbled upon using my best chinese and google translate...

https://github.com/mac-l1/machybris_examples.git

hope you can use this github too so we can share with others because there is no other documentation around...
i will update it with more samples like the demos i published on youtube: like the h264recorder i used to record the screen into h264 file using the vpu, and also the h264 movie player.

for your h265 player i wouldnt expect too much problems: basically you need ffmpeg libav to demux movies into raw h265 bytestreams and stuff them into the vpu and show the produces vpu frames (nv21) to screen.

if you need some help, just ask me (however my response times might be a bit long long due to my 2 kids - my most time-consuming projects)

cheers -- mac

[MAC_L1]

to add some info here:
libvpu is mainly needed for VPU/GPU memory access; librkon2 is needed for encoding/decoding; librkon2 uses other rk libs like libhevcdec to do its magic.
librkffplayer is a more complete library based on ffplayer lib to (de)mux and encoding/decoding streams and is used by androids stagefright, not yet available in machybris but can be quite easily added.
machybris also exposes the very high level api for androids media service by libmedia: ubuntu made this for their ubuntu touch/phablet products

hope this helps...

[MAC_L1]

the tricky part is performance: lets say you want to show a 4K h264/h265 movie with 60 fps on screen using VPU and GPU/GLES

- first vpu needs to decode a raw bit/bytestream to a 4K frame (nv21 format), this frame is 4Kx2Kx3/2 bytes ~ converting into 12MB pre frame
- might be that you want to convert this to RGBA format, more suitable for your purposes: i used rockchips hardware module rga2 to do this convert-copy before (for instance directly into framebuffer memory: did that in my youtube demo) ~ converting into 24MB or 32MB per frame
- or you can let the GPU do this by use a dedicated shader in GLES: but then first the 12MB should be made accessable to GPU too; the framebuffer EGL/GLES implementation of ARM uses memcpy() to do this and sharing memory using an eglimage isnt supported yet; i used this in another youtube demo. but luckily EGL exposed in machybris could do the this the Android way enabling memory sharing
- then it can be processed or displayed to screen using a double/tripple buffering scheme, preferably based on the vsync of the monitor, to avoid tearing

and this 60 times per second: so a lot of memory bandwidth and synchronization is needed, and actually this way 60fps isnt feasable: i get a maximum of around 40-50 fps

a more efficient way in order to make 60 fps feasible is to completely maximize memory sharing from path of decoding to display; so the vpu output frame memory should directly be shared with GPU (EGL/GLES) or framebuffer. this can be done by using ion or dma_buf memory sharing and is for instance implemented in android and the technique is called zero-copy; the android framework implements this in its native/media frameworks. basically you ask for a frame and get the result in an opengl texture. it's also quite tricky to implement this yourself.

ubuntu's implementation of libhybris exposes this part of androids framework in their libmedia; machybris is partly based on ubuntu's libhybris and exposes this too.

in my coming mackodi release i use this to display movies more efficiently; it supports 60fps and also 4k hevc/h265 decoding for instance.

cheers -- mac

[youbainiao]

hi,MAC_L1,Thank you for your reply, which helped me a lot, thank you very much, and now I have a problem, I run you give demo test_vpu.c, decoding hevc video, 1280x720 and 3840x2160 resolution is normal, but 1920x1080 resolution can not be decoded properly (I save the decoded data and play), Annex:http://s000.tinyupload.com/?file_id=...27788862788325, I'm not sure this is a problem decoding library,or can still be solved by setting the decoder parameters,Can you help me?

[MAC_L1]

hi youbainiao,

this has to do with frame image output size (width and height) of hevc vpu images coming out of the hevc hw decoder;
the width has a different stride than other formats, and also the height has a different slice than other formats.

to be specific:
stride of normal (i.e. non-hevc format) vpu image = image width in pixels rounded up to 16; and the slice of normal vpu image = image height in pixels rounded up to 16;
stride of hevc vpu frame image = (image width rounded up to 255) OR 256; and the slice of hevc vpu frame image = image height in pixels rounded up to 8;

you can look at my code that i used for kodi, see: https://github.com/mac-l1/machybris_...ybris.cpp#L908

      // RK quirks
      if( m_hints.codec == AV_CODEC_ID_HEVC ) {
[B]       stride = (((width + 255)&(~255))|256);
        slice_height = (height + 7)&(~7);[/B]
      } else {
        stride = (width + 15)&(~15);
        slice_height = (height + 15)&(~15);
      }

      // Y plane
      m_src_stride[0] = stride;
      m_src_offset[0] = crop_top * stride;
      m_src_offset[0]+= crop_left;

      // UV plane
      m_src_stride[1] = stride;
      // skip over the Y plane
      m_src_offset[1] = slice_height * stride;
      m_src_offset[1]+= crop_top * stride;
      m_src_offset[1]+= crop_left;

      m_videoBuffer.iLineSize[0] = width; // Y
      m_videoBuffer.iLineSize[1] = width; // UV
      m_videoBuffer.iLineSize[2] = 0;
      m_videoBuffer.iLineSize[3] = 0;

      unsigned int iPixels = width * height;
      unsigned int iChromaPixels = iPixels;
      m_videoBuffer.data[0] = (uint8_t*)malloc(16 + iPixels);
      m_videoBuffer.data[1] = (uint8_t*)malloc(16 + iChromaPixels);
      m_videoBuffer.data[2] = NULL;
      m_videoBuffer.data[3] = NULL;
      m_videoBuffer.format = RENDER_FMT_NV12;

these buffers go straight into kodi's NV12 opengl shader for further rendering.

you can also use a hardware-wise YUV-to-RGB converter to convert the frame to an RGB frame, like i did in my h265player: https://github.com/mac-l1/machybris_...layer.cpp#L894

void yuv2rgb_hevc(uint8_t* rgb, uint8_t *yuv420sp, int width, int height) {
    struct _rga_img_info_t src, dst;
    memset(&src, 0, sizeof(struct _rga_img_info_t));
    memset(&dst, 0, sizeof(struct _rga_img_info_t));
  
    src.yrgb_addr = (int)yuv420sp;
    //src.vir_w = (width + 15)&(~15);
[B]   src.vir_w = (((width + 255)&(~255))|256);[/B]
    //src.vir_h = ((height + 15)&(~15));
[B]   src.vir_h = ((height + 7)&(~7));[/B]
    src.format = RK_FORMAT_YCbCr_420_SP;
    src.act_w = width;
    src.act_h = height;
    src.x_offset = 0;
    src.y_offset = 0;

    dst.yrgb_addr = (uint32_t)rgb;
    dst.vir_w = (width + 31)&(~31);
    dst.vir_h = (height + 15)&(~15);
    dst.format = RK_FORMAT_RGBA_8888;
    dst.act_w = width;
    dst.act_h = height;
    dst.x_offset = 0;
    dst.y_offset = 0;

    int ret = copyBit.draw(&src, &dst, RK_MMU_SRC_ENABLE | RK_MMU_DST_ENABLE | RK_BT_601_MPEG);
    return;
}

similar stuff is done in android framework code in av media libstagefright

it also took me a bit to get this sorted out...
hope this helps you!

cheers! -- mac-l1

[youbainiao]

hi,MAC_L1, I use FB directly to display the decoded data, without using OpenGL, not quite understand how to set up parameters in these two methods:
rk_fb_set_par (FBI);
Rk_fb_pan_display (&fbi->var, FBI);
Can you help me see?https://bitbucket.org/T-Firefly/fire...e-view-default

[MAC_L1]

hi,MAC_L1, I use FB directly to display the decoded data, without using OpenGL, not quite understand how to set up parameters in these two methods:
rk_fb_set_par (FBI);
Rk_fb_pan_display (&fbi->var, FBI);
Can you help me see?https://bitbucket.org/T-Firefly/fire...e-view-default

hi youbainiauo,
can't help you here, i guess... i also used fb directly to display the decoded data; i used mmap() in user space to get a pointer to the framebuffer data and i used normal ioctl() calls including ioctl(fb,FBIOPAN_DISPLAY,&fb_var) from user space; actually thought of using FBIOPLAN_DISPLAY to implement double-buffering. See: https://github.com/mac-l1/machybris_...fbtools.c#L282
still hope this helps you...

[youbainiao]

hi MAC_L1,thank you very much,I have solved this problem, the most important thing is to calculate stride need to align:
stride = (((width + 255)&(~255))|256);