PRACTICAL IMAGE PROCESSING ON ANDROID

Hello, Today, I am going to write about image processing on Android. Obviously, This is not new to you But I am hoping the audience of my blog will be more intrigued to do lots of new works in this field after they finish reading it.

I got the opportunity to unravel the mystery of Image processing and we mostly solve unsolved challenges of Computer Vision. As you can imagine, On the Android side, we had to deal with bitmaps. My experience before joining this company was very like nonexistent dealing with bitmaps. So I had to really dive into how to use bitmaps efficiently, how to apply image filtering to that, and all that kind of thing. I am going to tell you a long story. Sit tight, do not go anywhere.

Mistakes we Make:

Firstly I am going to tell the common mistakes we make while we deal with bitmaps.

1. Rendering project size vs display size: Let’s say you have an Image of 800 by 600 and you have to display it on 20 by 20. If you do not downsample it, you are going to get lost by overusing a lot of unnecessary memory.
2. Loading bitmaps synchronously: We block the main thread unusually by synchronous loading of bitmaps and the problem arises you all know.
3. Lots of Image processing Library: Yes sometimes having a lot of options in hand is quite bad. This is going to be the main focus of our talk.

What is a Bitmap:

Basic Data Structure: 2D array of pixels and each pixel contains four channels Red, Green, Blue channel, and alpha channel.

Loading a Bitmap:

I am sure everybody here used some frameworks, libraries to do so. But let's look at the options:

1. BitmapFactory (Android developers site)
2. Glide
3. Picasso
4. Fresco
5. ….

These libraries wrap up a whole bunch of complexities for you and do it in a more concise way than BitmapFactory. Let’s take a look one by one:

BitmapFactory:

The below code is straight from the developer documentation. Trust me it’s a hell lot of code for load up a bitmap and displaying it some way. The problem with that is there are a few things it does not do. It does not cache the bitmap, If you do loading the same bitmap again and again in the memory, It can take a really long time to do so. It also does not know how to fetch images from a web server if you have URLs.

Glide to the Rescue:

Glide handles scaling, caching, downloading all the complexities alone with lot less boilerplate code. It helps you to get rid of unexpected OutOfMemoryException.

Images need to be upscaled and downscaled:

You often need to upscale and downscale a bitmap. Android framework supports this and you can easily get the codes by reading their developer documentation. It’s boring and lots of code but with Glide its cool and Google recommends to use Glide in their documentation.

Check Glides documentation you will get the demo. Here is a sample code:

Applying Effects:

If you want to apply effects on some Image there are Lots of ways definitely. Today we are not going to cover OpenGL and Native(C++). Everyone can breathe a slight relief for that. But let’s see how far we can get.

a. getPixel() & setPixel():

If you google it you can find lots of code getPixel and setPixel.

Basically, you have to write a loop over the 2D array and update(aRGB) each array item by position(x,y) while iterating. But the problem is as it runs on your CPU, It’s super slow and trust me It’s a hell lot of data! But it is easy to understand and you can use Threads for parallelism and you will get the leverage of full control.

Imagine this context and don’t die Because I have been there and I survived.

Pros:

1. Easy to understand

2. Manipulate per pixel manually

Cons:

1. Suuper Slow

2. CPU Processing.

b. ColorMatrix:

In Android, you get this class and it is not a third-party library. It modifies the color of each pixel drawn with a certain Paint. Let’s have a look at how we use it:

ColorMatrix is a 4 * 5 matrix that you apply to each pixel. After doing some cool maths, it comes back with a new pixel on the bitmap.

If you want to get the same image before it was updated You can multiply it with its identity matrix. Let’s say we want to invert my image, I want to make it the opposite of what it really was. I can multiply all the pixel values by -1. This could be a sample code:

In short, ColorMatrix is simple to use and reduces lots of code. You can do warmth, exposure, saturation, contrast, grayscale, sepia the whole bunch you can do with it.

Pros:

1. Easy-ish to understand.

2.Fast because it runs on the GPU

3. You will understand the exact effect being applied

Cons:

1. Can not do all filters like sharpen, unsharp, mask.

c. Glide Image Transformation:

This procedure is well documented at Glide. Before loading an image you can say apply this filter or something and Glide will handle that.

Pros:
1. Easy to understand

2. Fast

3. Nice API

Cons:
1. Can not do everything

2 . Values are abstracted away from you

d. GPUImage:

GPUImage is a similar kind of Library to Glide do it. It is from the cyberagent of Github. You set the image effect and get the output as the image effect. voila.

Pros:
1. Easy to understand

2.Nice API

3.Fast

Cons:
1. Can not do everything

2. Values are abstracted away from you

d. RenderScript:

If you want to make image editing fast with a few lines of code and the computing power of your phone’s GPU without the complexity of OpenGL? Well, Renderscript is meant for you.

Still not convinced? Let’s look at the numbers:

I made a comparison between the renderscript blur to the java based fastblur that you can find here. The image of the mountain has a resolution of 4806×3604 pixels. When processing blur on it on Pixel 2 XL, it took 738ms for the renderscript. The fastblur didn’t even work (out of memory)! So I tried on a smaller image (1944×1944), and the fastblur worked in 1,354ms, so I tried again with renderscript and it took 160ms, it’s more than 8 times faster. I won’t talk here about NDK, but you can find a renderscript vs NDK comparison here. Renderscript is based on C99 (Ed. C language), so you need to be familiar with that language. It shouldn’t be hard to know the basics if you already know java.

In short, Renderscript is a framework on Android for computationally expensive tasks.

At Pictarine, They compute a complex auto enhancement script on 100,000+ pictures everyday. It uses a lot of scripts, and it takes generally less than one second to compute. Sounds like they are more than happy with this powerful tool.

If you plan to do image processing I strongly recommend you to use RenderScript.

I will write a separate blog on it so that I can elaborate on it in detail. I got inspired to write this blog from a developer friend of mine, I hope that person will like it. That is all for today. If you need any examples, you can check on my github Happy Image processing.