Tag Archives: Apps

Sneaky tricks for developing on small devices – Bitmap ‘folding’

DSC_1583One of the most problematic constraints when developing applications for mobile or Set Top Box is video memory (AKA VRAM). You often will not have control over how much video memory is allocated to your application, or what the fallback behaviour is when your application uses too much.

This can be a pain, especially when you wish to create some off-screen surfaces for caching or compositing to improve performance.

If your application runs too slowly, that’s an ISSUE; if your application crashes due to excessive memory usage, that’s a PROBLEM.

I recently built a feature into an application which required a bunch of external images to be loaded into a Set Top Box device for in-process caching. Since all Bitmap surfaces are allocated on the platform as ARGB, but the images were monochrome, I could store the images efficiently AND make them available for hardware-accelerated compositing by storing just the single monochrome channel of each image in a separate channel of an in-process cache Bitmap surface.

You can see in the attached image what a a mish-mash of logos is created. For debugging purposes, you can also see the same surface viewed with each other channel turned off. When a logo is requested, a dictionary finds the relevant Bitmap slot it exists in (given by a Rectangle and BitmapDataChannel number). When a new image is loaded, its single channel is copied into the next available slot in the cache, in FIFO fashion.

The alpha channel of the cache surface wasn’t used, due to the pre-multiplication problems you’ll get – though this can be worked around if you can ensure there are no zero-alpha pixels. The result is an in-process cache requiring no image decoding to composite images, storing 3 times as many images as a regular Bitmap FIFO. #WIN

Super Kickups returns… as an Android app

Super Kickups mobile gameAddictive as crack – apparently

I finally decided to relaunch one of my classic old Flash games as a mobile app. I picked Super Kickups since I think the game mechanic works nicely on a touchscreen – that, and someone once told me that it’s addictive as crack – which I take as a compliment. I’ve recently added a leaderboard and I’ll be adding various other features, such as pickups and achievements, in the near future.

I used Starling for the rendering and it runs smoothly at 60fps on both my shiny new HTC and my somewhat slower old Sony Xperia. It’d be good to know how it runs on a range of other devices – especially if anyone finds the performance slow.

I’m not planning to target iOS just yet though, being a bigger fan of Android (perhaps Apple could make iOS less of a pain in the proverbial). You can download it from the Google Play Store here.

IPTV development with AIR for TV

Having just finished building the UI for the YouView set top box, I thought I’d share some of my insights into best practices when building applications for such resource constrained devices. The YouView UI is AIR based, written in AS3 and runs in Stagecraft 2, also known as ‘AIR for TV’. As the name suggests, AIR for TV is a special version of the Flash player for embedded systems, such as set top boxes. The first incarnation of the YouView UI (back when it was just codenamed ‘canvas’) was for Stagecraft version 1, which means coding in AS2 and suffering the abysmal performance that comes with running on AVM1 (ActionScript Virtual Machine 1).

Despite the delays and the need to code the UI from scratch in AS3, I think it was ultimately the right decision. Stagecraft 2 is a much better platform – Stagecraft 2.5.1 to be precise. It was a great opportunity to learn how to write optimal code and use hardware acceleration effectively on resource constrained devices. I’ll be doing some tutorials on this in the near future, but here’s the key points to observe when developing for such platforms:

  • Limit the complexity of your display list heirarchy
    This may sound obvious, but ensure you nest as few things as possible, keeping the display list as shallow as possible. Stagecraft needs to traverse through the display list, working out which areas of the screen to redraw. This is similar to how the desktop Flash Player handles redraws, but with some key differences to how it decides what needs redrawing, how it tackles moving display objects and how it delegates the work of updating the frame buffer – a subject for another time. Mostly importantly, if you’re developing for a resource constrained device (such as mobile or set top box), you’ll have very limited CPU power, even if the device’s GPU (graphics processing unit) affords you great hardware acceleration capabilities. So, before Stagecraft can delegate any work to hardware, it enumerates changes in the display list in software. Complex display list heirarchies are a headache for some of the low-powered CPUs found in mobiles and set top boxes and this’ll show up as rocketing CPU usage, low framerates and few spare ‘DoPlays’ in Stagecraft (spare work cycles). By keeping your display list shallow, with only the bare minimum of display objects on stage at any one time, you’ll be making life easier for Stagecraft by doing less work on the CPU – whether or not graphics are drawn in software or hardware.
  • Benchmark everything
    When building an application for a resource constrained device, you should be able to run each component in isolation, to assess its drain on CPU and system/video memory. There’s no point optimising the hell out of one component, when it’s actually another one that is the source of your performance bottleneck.
  • Know thine hardware acceleration capabilities
    There’s no point blindly using cacheAsBitmap and cacheAsBitmapMatrix everywhere, if it’s not going to speed things up on the target device. Worse still, too many cacheAsBitmaps and you may be just wasting valuable video memory, or causing unnecessary redraws (again, the subject of a future article). A lot of platforms will accelerate bitmaps, even if stretched, but not necessarily if flipped or rotated. Alpha on bitmaps (or anything cached as bitmap) will usually be accelerated too, but this is not necessarily the case with all colour transforms. Benchmarking any component you’re building will quickly tell you where you might have pushed it too far, but you should also have a way of verifying that a particular set of transforms is indeed hardware accelerated. Stagecraft provides this when using its –showblit command line parameter. I’ll be going into more detail about this in another post.
  • Mind your memory
    When using various hardware acceleration tricks, especially on resource constrained devices, video memory is at a premium and usually in limited supply. You will need to know the limits and have a way of seeing how much video memory your application is using at any one time – ensuring you dispose and dereference any bitmaps you’re finished with too. If your platform uses DirectFB for its rendering, as YouView does, the executable ‘dfdump’ can show you just where your video memory is going. This is something else I’ll get into in another article.
  • Blit blit blit
    This refers to blitting, where blocks of pixels are copied from one bitmap to another. This technique is used a lot in games, where graphics performance is critical, you should arm yourself with the basics of how old video games used blitting of multiple things to a single bitmap for performance and video memory efficiency.

I’ll probably go into more depth on each of these things in forthcoming posts. Stay tuned.

Chinese Handwriting Recognition App

Unlike the iPad, the BlackBerry PlayBook has rather poor international keyboard support, with no method for entering chinese characters. I like the way iOS achieves this, so went about building my own version in ActionScript.

This was mainly an academic exercise and to help me to learn to write chinese. My approach was to sample the strokes drawn into the app as a series of up to 8 directions, including the relative position of a given stroke to the previous stroke, again as one of 8 directions. This pattern, represented as a string of numbers is then put through a smoothing algorithm, to remove some unnecessary noise and then compared with a dictionary of pattern keys, which may contain one or more suggested characters. If there are no hits, an advanced search occurs, by mutating the given pattern in specific ways, in order to find alternative suggestions. I can also find characters based on the next most likely character to the one you’ve just entered, using frequency analysis on given sample text.

The app will eventually be a PlayBook App, but is still unfinished and currently in ‘training mode’, so that character patterns can be trained into the database. It’s currently primed with some simplified sample data, from which it picks the most popular few characters to learn. If you write chinese, give it a go.

How the hell do I build this?

I had a conversation yesterday with a friend and colleague about how his company should standardise their development environment for all Flashers – be they contract or perm, junior to senior.

He, like many of us, was sick of contractors building projects and leaving them in various states of repair. Required libraries or fonts are often missing, bits of code never even get checked into source control and, frustratingly, it is often unclear how to build a project. The makeup of Flash projects can vary from an FLA file/s full of timeline based code, to source code set up to compile under one specific, mystery environment.

The problem:
You need to set some kind of reasonable standards, so that ActionScript projects can be easily verified, maintained and recompiled, not necessarily by someone with intimate knowledge of the OS, environment and the project – ideally even a developer without intimate knowledge of ActionScript or a copy of Flash CS5 and Flash Builder to hand.

The solution/s:
There are obviously many ways to skin this cat. But, the way I see it, the best solution needs also to be reasonable, achievable by everyone and cost-effective for a typical digital agency. As such, the ‘best’ solution may not be the ‘ideal’ solution – by which, I mean an idealistic solution based purely on software development ‘ideals’.

Flash IDE
We have to start somewhere and, at the risk of a flaming, I’ll act as its advocate for a bit. There will often be a need for those FLA files knocking around and you won’t get designers building their banner’s with the Flex SDK. However, since the Flash IDE isn’t free, open source, understood by non Flashers and (with the exception of CS5) creates nasty binary balls of mud (namely FLA files), we’ll assume from here on in that we’re talking about compiling project with the Flex SDK.

Flash Builder
Adobe’s own latest development environment for Flash/Flex, built on the very popular and mature Eclipse is certainly feature-packed and already industry standard. But is it the sanest choice to enforce that everyone use this particular environment, just so projects are more maintainable? Will it work and, since it’s not free software, is it even cost-effective? Personally, I think perhaps not. Since Flash Builder isn’t free, requiring its use for all Flash projects within a company will probably solve one problem and create all new ones.

FlashDevelop
FlashDevelop is my favoured editor and, without getting into the FDT vs FlashDevelop vs Flash Builder debate – I favour it primarily because, whereever I work, I can always get IT to install a copy on my machine, without having to wait for budget approval, bring my laptop instead, etc. However, FlashDevelop is currently for Windows only and is still just one development environment. Even though it’s free and open source, we want to abstract away a project’s configuration and setup from any software that isn’t also cross-platform and industry standard.

Maven
So why not enforce that everyone builds and configures their projects so that they can compile under Maven from a POM? I think it’s a little unrealistic to expect every calibre of ActionScripter to even know what Maven is. At very least, it would create a rather high barrier to entry for prospective developers, making recruitment even more difficult for any company. I agree that it’s a bonus for many, larger projects that may need to pull in dependencies from other projects.

Ant
Ant is a cross-platform solution for software automation, built on Java. Since it’s open source, free, industry standard and pretty easy to set up and use, I’d say its the best choice for configuring the building of projects in a standardised way, without tying anyone to a particular code editor, platform or requiring expensive any software.

Ant can be integrated into most development environments and provides a sensible ‘how the hell am I supposed to build this?’ answer to any project. This also means that, for those people that use it, the Ant script can be used for building with Maven/Hudson, for automatically ensuring all projects will build – even if you only use this to ensure a contractor has left the project in a buildable state.

I also found this helpful article on integrating Ant with FlashDevelop – so if you like that free, open-source feeling, then take a look.