accelerated video / video processor (VPU) running on linux on RK3288 / firefly

[MAC_L1]

Hi guys,

Just to show I've got the RK3288 video processor (VPU) working on linux -both decoder and encoder-:

https://www.youtube.com/watch?v=PLHsnUpak5Q

https://www.youtube.com/watch?v=ZXMX1kR8Y3M

Technically I use
- Android's libraries: for access to the vpu device driver and for memory management – both encoding and decoding h264 byte-streams
- libhybris: to access these libraries on linux/ubuntu
- rga2 device driver for 2D HW accelerated blitting and converting frames (yuv/nv21 <=> rgb)
- ARMs OpenGL library, driver and examples (fbdev) to render rotated images using the GPU
- raw framebuffer device access to blit full-screen (full-screen player and recorder)
- avlib to demux movie containers (mp4/mov) to bytestreams (player) and vice versa (recorder)
- Movies from blender projects: http://archive.blender.org/features-gallery/movies/

Due to a clean low-level interface it should be quite easy to add video acceleration to xbmc, avlib-tools, gstreamer, vlc, … Also all hw supported formats should be supported (including HEVC).

Basically this should also work on RK3188 devices, as Androids libs are almost identical and libhybris is also working (my next project...)

Cheers!

- - MAC

[Gilgamesh]

WOW finally!! mo123 is it possible to make a Linux ROM only that uses most of the internal storage? instead of Dual boot? (in my case 32GB)

[Gormar]

Very impresive.
If only will be possible to wrap it into OMX lib or other interface used by Mplayer or VLC.
Lack of video acceleration is the only thing that detter me to use Linux on RockChip TV boxes.

[Gilgamesh]

how long could it take before we can see ROMs with fully working accelerated video?

[MAC_L1]

@mo123: indeed, using libhybris for decoding/ encoding, it works pretty efficient. For the 3D stuff I use the fbdev mali driver from ARM. But for full HD rendering they are just a bit too slow. Scaling ARMs own simple examples to full HD screen gives me about 24 fps. Maybe overclocking could do the trick here... However, the frame memory copy to GPU also more than halves the fps and is bottleneck. With current fbdev mali driver it is not yet possible to share memory with GPU so a copy is needed. I guess coming X11 mali EGL/OpenGL support should solve this. Androids mali 3D driver would solve this too but I only got it working partly for RK3288 (on RK3188 it works perfectly, hence 3188 is my next project...)

@Gormar: I checked the sources of Kodi, Mplayer, VLC, gstreamer, and integration with vpu library doesn't seem that difficult. They all use libav library as core and that is where I hooked on to the vpu lib. So integration would be easily possible. My first target is probably Kodi, seems the easiest. Don't know if someone has already Kodi working on RK3288 linux (without acceleration but with mali driver) and is willing to share this on github or so?

@Gilgamesh: it all depends on the availability of X11 mali EGL/OpenGL driver, probably in a few months. Add a few months extra to integrate one or more applications -depending on the number of volunteer developers-. In total 3-6 months is my best guess.

mac

[Ava]

Thanks for your great work! =)

[taki]

Before Omegamon worked to port Openelec on rk3188. He said that I booted openelec but no hdmi . If it works hw Accelerate and opengl mali driver. That can be work 3188 and 3288
That is good news for rk user

[miskol]

Hi guys,

Just to show I've got the RK3288 video processor (VPU) working on linux -both decoder and encoder-:

https://www.youtube.com/watch?v=PLHsnUpak5Q

https://www.youtube.com/watch?v=ZXMX1kR8Y3M

Technically I use
- Android's libraries: for access to the vpu device driver and for memory management – both encoding and decoding h264 byte-streams
- libhybris: to access these libraries on linux/ubuntu
- rga2 device driver for 2D HW accelerated blitting and converting frames (yuv/nv21 <=> rgb)
- ARMs OpenGL library, driver and examples (fbdev) to render rotated images using the GPU
- raw framebuffer device access to blit full-screen (full-screen player and recorder)
- avlib to demux movie containers (mp4/mov) to bytestreams (player) and vice versa (recorder)
- Movies from blender projects: http://archive.blender.org/features-gallery/movies/

Due to a clean low-level interface it should be quite easy to add video acceleration to xbmc, avlib-tools, gstreamer, vlc, … Also all hw supported formats should be supported (including HEVC).

Basically this should also work on RK3188 devices, as Androids libs are almost identical and libhybris is also working (my next project...)

Cheers!

- - MAC

Can you make step by step manual for this?
I would really like to see it work on my RK3288 board
btw I think that EGL image from OpenMAX(from android) should work with EGL from fbdev. Or MALI kernel drivers should wide range of unifiend memmory (DMA_BUF, UMP, some android)
And your don't need yuv/nv21 <=> rgb as opengl shaders can do(much much faster) it for you.

[memeka]

Don't have a Firefly, but I did the same thing on Odroid-XU with libhybris so I made this accpunt to help out.

You can use the Android 3D Mali drivers with libhybris on XBMC with this: https://github.com/mihailescu2m/xbmc...80a267fef50b4a - it's using hwcomposer, dunno if Mali android has hwcomposer or not, but at least it's a start.

More importantly, xbmc can use the libhybris VPU with this: https://github.com/mdrjr/xbmc/blob/o...odecHybris.cpp
I suppose you actually use the libhybris from Ubuntu, which has libmedia added by Canonical. libhybris from git does not have libmedia.
I also had to patch libmedia, see the end of this post for patches in case it's not working out of the box: http://forum.odroid.com/viewtopic.ph...;t=4073#p33162

All the best.

[memeka]

Don't have a Firefly, but I did the same thing on Odroid-XU with libhybris so I made this accpunt to help out.

You can use the Android 3D Mali drivers with libhybris on XBMC with this: https://github.com/mihailescu2m/xbmc...80a267fef50b4a - it's using hwcomposer, dunno if Mali android has hwcomposer or not, but at least it's a start.

More importantly, xbmc can use the libhybris VPU with this: https://github.com/mdrjr/xbmc/blob/o...odecHybris.cpp
I suppose you actually use the libhybris from Ubuntu, which has libmedia added by Canonical. libhybris from git does not have libmedia.
I also had to patch libmedia, see the end of this post for patches in case it's not working out of the box: http://forum.odroid.com/viewtopic.ph...;t=4073#p33162

All the best.

[MAC_L1]

Can you make step by step manual for this?
I would really like to see it work on my RK3288 board
btw I think that EGL image from OpenMAX(from android) should work with EGL from fbdev. Or MALI kernel drivers should wide range of unifiend memmory (DMA_BUF, UMP, some android)
And your don't need yuv/nv21 <=> rgb as opengl shaders can do(much much faster) it for you.

I wil publish on github but currently code is a bit experimental and a mess. I have to clean it up and especially find a way to "package" libhybris and a minimal recompiles android /system image that is essential.

EGLimage on android needs the android mali lib/driver that isnt fully working, and for eglcreateimagekhr for fbdev i am stuck at getting a decent pixmap, have to ask ARM-community there: have the feeling not all is supported yet (DMA_BUF, ...) - Androids ION using libhybris could be a hybrid solution; for now too much unclarity so better wait for decent X11 mali drivers.
Basically all I need is a shared pointer to access texture memory directly.

As for the yuv/nv21 <=> rgb conversion: I first implemented this using an opengl shader just like you mentioned. That worked fine but only to a max fps of 24 fps or so. Basically that seems the max for rendering 1920x1080 pixels using opengl. Also in the examples from ARM (rotozoom, ...) this seems the max. Maybe increasing GPU frequency can crank it up a bit. So for the full screen video I dont use opengl but directly blit to framebuffer device using a hw 2d acceleration. It is for sure the fastest colorconvert copy using specialized hardware, however still a copy that isnt desirable.

[MAC_L1]

Don't have a Firefly, but I did the same thing on Odroid-XU with libhybris so I made this accpunt to help out.

You can use the Android 3D Mali drivers with libhybris on XBMC with this: https://github.com/mihailescu2m/xbmc...80a267fef50b4a - it's using hwcomposer, dunno if Mali android has hwcomposer or not, but at least it's a start.

More importantly, xbmc can use the libhybris VPU with this: https://github.com/mdrjr/xbmc/blob/o...odecHybris.cpp
I suppose you actually use the libhybris from Ubuntu, which has libmedia added by Canonical. libhybris from git does not have libmedia.
I also had to patch libmedia, see the end of this post for patches in case it's not working out of the box: http://forum.odroid.com/viewtopic.ph...;t=4073#p33162

All the best.

Tnx memeka! Especially for the xbmc links, gives a good staring point!

Mali android is using hwcomposer but somehow it is not working OK for rk3288 on linux - on rk3188 i got it fully working so that is my focus now.

I do use the ubuntu libhybris indeed, but tried to get libmedia running for more that a year or so, patching stuff. Actually it is running, but i get unsupported media format errors. Therefore I now added a libvpu compat part to libhybris that is basically a proxy direct to the androids libvpu/librkon2 from rockchip so giving me direct access to low-level decoding and encoding api. Advantage is that no android services need to be started in linux.

Also just checked your patch and whole thread: very helpful! Tnx!

[memeka]

yuv rgb i think was done by odroid member @oversun in ASM NEON, and I remember it was running ~5 times better than using egl. you might want to search the odroid forums.

[memeka]

btw, i would like to try a similar approach for odroid-xu3, with mali-t628.
so before making an image for rk or publishing, can you explain how you got things working?
what parts of android did you use? any patches to android? did you use bsp or cm?
i can deal with the linux/hybris stuff, but the android bits were the ones misterious to me

[asoltesz]

MAC_L1 This is great work, many thanks and keep it up.

In the coming months, a lot of people will buy the new extreme-cheap Chromebooks with the RK3288 (e.g. ASUS C201 for $169, Haier and HiSense ones for $149).

This hw is strong enough to run proper Linux so quite a few of us will try to put Ubuntu and other desktop Linux ports.

VPU acceleration would be a real boon here because it will allow the Linux install to be used almost as well/efficiently as ChromeOS (actually better if normal vplayers can be used). With VPU and GPU fully working there would be little incentive to switch between the proper Linux desktop and ChromeOS.