A couple weeks ago I released mobile version of my project APEXvj to stores. It’s made with Adobe AIR 3.2 and it works in iOS and Android. Since then I’ve been collecting feedback, reacting to those and continued to work on get it solid as possible.

The first release was in many ways total disaster and in other hand a priceless learning experience. This post will cover that and explain the design of gadget. I would also want to make one point (oh no…). This post covers many topics so if you find one boring just skip to next. It’ll also be pretty safe from hardcore coding stuff instead there’s a lot of pretty pictures :)

I heard that first 72 hours is somehow crucial and one should try to get as much traffic as possible during that time. So I sent emails to all APEXvj desktop users who gave permission. There was something like 5500 of them. Also drummed about the release at Twitter and at APEXvj Facebook page (~8500 dudes). All fine so far.. Then I started to get a lot of emails about the app isn’t supported on high-end android devices. Like on Galaxy S II that was my test device. :E WTF is this?? After way too much time I found this post from @polyGeek. And it says: In the -app.xml manifest file there is a setting for uses-configuration android:reqFiveWayNav=”true”. If you leave that set to the default of true then when you put your app in the Android Marketplace it will not show up for people who have a phone without a trackball – most of them. YEAH AUTS!!

Apparently that’s pretty common newbie mistake. So put this rule deep into your memory: Make sure you have

in your application description xml.

So if a lot of people sent me email about it didn’t worked for them it means that it didn’t work for shit loads of people. There were other issues as well. There are currently 693 Androids were APEXvj should work. This means a lot of different screen sizes and resolutions. I didn’t have this covered in first release. Not at all. For example layout elements didn’t adjust to higher resolutions. This caused UI to have too small hit areas and text was way too small. Also for example HTC Sensation has a pretty much same size screen then iPhone but resolution is huge 540×980. Also the touch works differently. Apparently it’s very sensitive for touches. Not like iOS that ‘knows’ when user is touching vs. swiping. This caused a problem where user couldn’t press any buttons. Fixed it when Eugene kindly pointed it out. With more test devices I could have avoided all of this, but like I could buy my self a stack of Androids…

Everything is now pretty much fixed and if I found more issues I’ll fix them right away. Updating to GooglePlay takes about an hour. Gladly the iOS version worked pretty well. There the update phase is one week. Good thing it’s my project so no harm done to others :)

UI is a set of playlists that can be changed by swiping. Here are ‘this week top 200 from SoundCloud’- and ‘Search’-view.

Some of the song can be downloaded and they are collected to ‘Downloaded’-playlist. Also user can connect with SoundCloud. This gives ability to set song as favorite.

Once authorization with SoundCloud is done user see their favorite and own songs in separate lists.

Once user tap a song it starts to play. After slight delay view animates down and visualizations start. If user double taps song just starts to play. Song name is replaced with waveform that indicates load and duration. User can change the position of song by tapping the waveform. Currently playing song is written under the prev/play/next + visuals controls.

When song is playing a info-icon is set over the waveform. By pressing that user can read more info about the track, share song, favorite and possibly download it. Download is confirmed with a popup. If user decides to download the progress is shown in popup. This is a performance thing since downloading a song takes a lot of cpu.

User can share a song to twitter or copy url. Copying isn’t possible with all devices, but better save then sorry. That could be changed after future software updates.

By pressing the shared url user is taken to page that shows song title and user who uploaded it. By pressing the play-button APEXvj application is opened and song is added to ‘Shared to you’-list. After little delay the song starts to play with visualizations. Delay is here so that user knows what is happening. ‘Shared to you’-list shows 50 latest shared songs. Users can mark songs to their favorite list and download list if feels so.

The position in view row is indicated by dots at top of view. Every list has maximum of 50 tracks shown at one time. After that there’s a pager where current page is inside large circle. Previous is at left and next at right. Left circle also leads to last page when viewing the first page. This show instantly how many pages are there. User can move to visuals by swiping down or tapping the logo-circle. Android users can kill the application by touching the quit X at top right corner.

User can return to UI by swiping up. This logic is informed in different places. When user tap a visualization view a settings panel opens. User can adjust ‘cut frequency’ which is in other words: “How tight the camera cut phase should be?” Also user can choose render quality and pick a color scheme.

By default user get two visualizations. Six more can be bought from Eye Candy shop.

In first run user can preview the visualization for 10 seconds. After that preview time is smaller. After preview they are encouraged to buy visuals. Eye Candy Shop has button that opens an in-app purchase dialog. It takes about 5 seconds to buy a visual-pack. After all sort of blood suckers I get about an euro per visual-pack minus taxes.

I’m planning on creating more visuals in the future. I think I’ll have few days break though :)

A couple days ago I went to toilet with camera… nothing like that! I filmed the app in different devices. It would be nice to expand this collection of ‘APEXvj running in different devices’. Anyway here’s APEXvj in Samsung Galaxy S II.

There it is. If you have more usability issues or some other concerns please let me know so that I can fix them.

I want to be free from problems caused by ego at some point. Ego causes envy, low self-esteem, fear and leads person to vicious circle of never been enough. It’s ego doing its stuff when one seeks respect from others. This is all personal growing stuff, but lately I have come to realize that ego might be doing its devious works on my codings.

When I was doing those photoshop pictures there wasn’t Twitter. I had no place where to show them. I did not measure ‘success’ in RT’s or visitor counts. Did them just for my self. Just started to build something, fall into beautiful flow state where everything was easy, interesting and felt right. Afterward I was like “whoahh what happened? That was awesome”. These days I do get flows but I also always check if people RT my posts and even feel a bit sad when there’s only few of them. Shit. Social media is turning me into a narcissist idiot.

Denying ego issues is part of its scam. If I look back about my latest codings I see some dubious signs like “Look 1.4 million particles (in other words: Look I have skills to do that!)”. I have created stunts. That’s bullshit and not why I want do my stuff. Gladly I open sourced that code so that I don’t feel like complete douche bag now. Anyway it’s worrying.

For me the right reason is exploring my inner, learn and enjoy. Wrong reason is to seek for cheers, respect and all stuff coming from outside to feed my ego.

I want to get to right path again. So I think I won’t be releasing future codings right after I make them. I will wait for a week so that I get bored on them or might not release them at all.

Here’s the first glimpse of the application. Video shown at FITC Amsterdam 2012 at the Adobe Keynote. It shows the first demo version of APEXvj. Very old but contains some of the features of the final apparatus and helps understanding this post. (UPDATE 5.3.2012 The video is longer there :/ )
http://www.youtube.com/watch?v=ZgLbjva4O2o

Today I open a new chapter in my little quest for synthesize all online music with realtime graphics. The new APEXvj mobile application is waiting for review at AppleStore and Android Market. The App will be free for all. I only ask little fee for extra visuals. Hopefully someone buys them so that I can continue this quest. In this post I’ll open up the long story behind the development and will share some technical findings and ideas I’ve came across during the process. Project was epic so will be the post.

All along it was obvious that I must create a next generation of APEXvj apps in GPU. Finally in Spring 2011 I took from bulls balls and started to dig in. I believe for just a little bit smarter then average people the stage3d API is easy thing to coupe. I'm not one of those. I'm a slow learner and even worse I'm one of those who needs to have fun while learning or it doesn't happen. I went there where I found the most fun; Particles.

It was great learning bench since with those little bastards you'll always navigate very deep at API. When optimizing one must not compromise. So I learned the API and learned how to get the most performance out of it. This phase took me something like 8 months. Obviously I drink beer, play frisbee golf and did other stuff like work in between. Oh and one baby. My son Akseli ;) (not in the picture)

It’s easy to copy .js lib from web and do as told in tutorial. I need to know how technology smells like in the morning without a makeup before I can master it. And that happens slowly. This goes to all young devs: It is OK to spent months on some minor detail to deeply understand it.

Those conference speakers always say that there’s time if you want there to be. Well.. this is all bullshit. There isn’t time. There’s only you and your sorry ass bending. Weekends are decided to my wife and son, but here’s my average weekday from past 6 months.

1. Baby awakes me up for the last time about 6 clock.
2. Some coffee and 30 or 45 minutes of personal coding.
3. To work. 8 hours of coding like a mother fucker. Advertising Jippikayjey motherfucker stuff.
4. 30-90 minutes after work personal coding.
5. Home playing with boy
6. 20:00 little man goes to sleep.
7. 20:00 - 24:00 personal coding.

If someone wants to build something good they must either risk their finance or hand over they freedom or risk their health or just take time. All of these scenarios have their minuses. Most risk their finance. I don’t want to do that. I have a mortgage and a little kid. Also I have a feeling that investor will take the freedom eventually. I choose the “take time” and make it happen slowly.

No matter if no-one buys this stuff I learned a LOT. Here are some of it I can share with you. Technical mambo nachos coming up:

Adobe Mobile AIR Stage3D is fast. Flash on mobiles have limitations and that's CPU. Also obviously GPU is not even close to desktop performance. But when done right a lot can be achieved. Here's couple things that should be keep in mind when doing particles with Mobile Stage3D.

Important lesson here is to define your playground. What can be done and what should be leave to future for better GPU's. Stuff shown here are just something I thought will ably. I can possible be wrong is some cases.

A) Avoid alpha kill it’s slow. Do not do the following.

Instead use add blend and accept this as your playground. You can even ditch the z-sort in this approach.

Don’t try to face particle planes/triangles against camera. It’s slow.

Simply keep your camera facing planes/triangles enough all the time and do only one matrix calculation in vertex shader. You can rotate z-axel as much a you like though. Accept this as playground.

Still no matter what the big challenge in mobile world will always be the big variety of devices. You don’t want to build against worst case scenario and cannot build against the best case either. If you go middle way you show your ass to every direction. Gladly there’s a solution for this. Do the trick familiar from post processing. Draw your scene into a texture and render that texture in a plane to screen. You can now set the resolution (quality) of that texture and satisfy all needs.

Handling touches is also a performance matter. According to my tests it’s fastest to disable mouse right from the start and handle mouse events just from stage.

Seed the final word where ever you think it can be used.

Avoid calling functions. No matter how painful it might be. You need to find a way to create stuff without calling methods. For example I have created all visuals so that they take context3d and other stuff from shared class, but in their render method they all do the same basic routines. For example handle camera, post-process rendering and background rendering. It isn’t the most elegant way to do but performance gain is huge. And my visual engine doesn’t call visual-classes render function. It changes the ENTER_FRAME event listener.

APEXvj mobiles UI runs smooth 60fps. This was achieved by following few basic principals. First for all I set

in application description xml file. This means that every frame is send to GPU and it require sending render command to stage3d in every frame. So when visual isn’t running I have a little Clear-class that does just that.

Also all the scrolling between views are done by sprite.scrollRect. Which is by far the most efficient way to scroll. Plus to that all my content is set cacheAsBitmap = true and like I said before elements don’t have mouse enabled.

That’s all about performance at this point. Don’t worry. Epic post is not quite over yet :D

In old browser and desktop version of APEXvj I used the Sound.computerSpectrum method a lot. It has a very annoying feature in it. It fails if you have a youtube or other video open in browsers other tab. When people get that error and are asked to close other tabs they in most cases close the APEXvj. I came to a decision that this method cannot be used ever again. Alternative to that is the sound.extract() which is a failure too since it takes too much CPU to calculate FFT.

To check the intensity of a song I add leftPeak value to running variable move. I can now use move for example in rotation. If song has a intensive phase it rolls faster and slower when sound cools down.

If leftPeak value reach a certain peak like 0.6 I could do a flash or something, but this doesn’t work the same in every possible song. Some songs are quieter and some are extremely intense. So I figured the following method.

I collect peak values from past 10 frames. Calculate the average. Check if the current value is large versus the average. This gives me the information about when song changes dramatically. According to that difference I can do several effects. Like if it’s big enough I’ll do a camera cut, a flash or post-process effect.

This simple but clever method gives me a perfect music video type of cuts in visuals for every type of songs. I think I’ll do all my cuts in future demos this way. :) I was so overwhelm happy when I figured this out! That point where I know when song has a peak can be used so many ways. For example in some effects I check how long it took from last peak and use that value to determine how strong the shown effect will be.

Anyway a lot can be done with just leftPeak and best thing about it is that it works everywhere every time. Takes zero CPU and smells like Swedish designers (good and expensive).

OK that’s it. Hopefully you find something interesting from this exhausting post. The next one will be the grand release post. The approval process probably won’t take more then six months :)

The content is mostly Flash which is coded in ActionScript 3, Pixel Bender and AGAL. Used Chrome, Firefox and mainly Safari for developing and testing. Photoshop for layout and Flash IDE for vector graphics plus some Illustrator. Screenium for video capturing and Media Encoder for compressing. Videos are streamed from vimeo.com and fonts from Google. I used Coda for html/css/js/php coding, Sequel Pro for database and FDT5 for AS.

Buzzword kids can try to categories that. :) Probably no need to point out more…

Anyway, you can jump between posts with arrow keys and view all flash stuff in HD video. I do recommend that you view them in realtime if you got device with flash player. They are much faster, sharper and small in file size that way.

Now that this project is done I’ll move on to next one.

In this post I’m going to explain some of my findings when playing around with Adobe’s Stage3D and particles. I have pushed the current version of particle engine called evoSpicyParticleEngine into GitHub, will show some examples how to play with it and explain tricks used in it.

… but first I want to joyously announcing that I do not owe you shit. Nor does anyone else that open source stuff or release what ever. So if you don’t like the syntax or what ever don’t pressure me with feature requests. Get involved by forking my codes.

We are the most spoiled generation ever. The notorious West. At that nanosecond everything isn’t absolutely perfect we start to whine like children. One can just wonder how complaining when train schedule fails for 10 minutes sounds from majority perspective. (including those four billion people that live under $2.50 a day)
How utterly ridiculous it is to whine when some extremely sophisticated technology fails to work at that precious moment one felt like using it? (e.g. Internet)
Or how about whining when some refugee that came four months ago from country middle of oil war fails to pronounce the pizza delivery right?

Here’s a great sort youtube clip from Conan O’Briens show related to this.

We face this nonsense when perspective is fucked up. Lets say we ditch the whining and preserve efforts.

Here’s my next effort for our playgrounds wellbeing (no more preaching):

evoSpicyParticleEngine

The idea behind this engine was to be able to render massive amount of moving particles. So this meant some restrictions. For example it only renders full buffers, it renders triangles instead of quads and all particles share some properties.

All this stuff can be find from GitHub. You’ll get links at the bottom of this gigantic post. Lets take a look from outside. This is how you render basic particles with sphere explosion movement:

…

…

Code above in action
SpicyParticleBasic.

Another example that’s also in GitHub
SpicyParticleBlurPerlin.

That looks easy enough?

Lets take a look on some of the tricks.

Single particle
Like I mentioned earlier in this engine particles are triangles. The ‘normal’ way would have them in quad that contains two triangles. That takes 4 vertices which is more then 3 and triangle can draw a particle as well.

Common sense, huh? ;)

Face particle to camera
This is a trick that I got idea and code examples from blog post by Jean-Philippe Auclair who works at Frima Studio. It’s a great post about particles. Make sure to read it up if you haven’t already.
Trick goes like this:

1. Precalculate a single triangle and store it to GPU. Store those register numbers (10,11,12) into individual particle position component (x, y, z, w <- right there).

2. In render method create a matrix that’s invert from camera matrix and pass it on to GPU

3. In vertex shader get position of stored triangle, rotate that with camera invert. Scale it to desired size and transform to world space.

Tadaa! now it’s always facing the camera and this trick didn’t cost a penny.

Moving particles in vertex shader
In AGAL vertex shader we cannot add values to vertices. Like move them a bit in x-axel in every frame. The data isn’t stored anywhere. All values must be at memory and push to GPU and then it’s just drawn. Continue movement could be done by storing movement values into a texture but we do not have access to texture in vertex shader. This a challenge and there’s a trick to get around this: We store start and end point of every vertices. Give a start time value to every particle and send constantly increasing time value to vertex shader. Then simply tween the particle position between those points according to time and start time value.

This way we can ‘rebirth’ single particle by setting it’s start time value to current time value.
This is how it looks like in AGAL:

And that’s how we can move 1.4 million particles in flash.

Manipulate the movement
AGAL is not limiting language. (except that ‘no access to texture’-thing) It has a stack of mathematical functions. With those a lot is possible. Here’s how particle movement is eased in exponential fashion:

Get your forks polished
– evoSpicyParticleEngine at GitHub. Also check those examples.

Hopefully you learn something from it. You can use it anyway you like. Contributing for its development by forking at GitHub is desirable. I’m slowly developing it more as time goes by. Post process filters, background and overlays are on table. Ohh and lights of course. All those are actually in my classes commented out waiting for polishing.

Peace and love, Simo

That’s 200 000 particle version. That might run on your browser.

So like I’ve written on this blog my particle engine can move large amount of particles from A to B and I can do custom tweening algorithms for that transition. Great… what then. I realized that using perlin noise for setting the B positions of the particles could turn this thing look quite interesting.

On my OsX toy the difference between flash on browser and standalone is insane. 200 000-300 000 particles is pretty much the maximum until it won’t run smooth anymore. I wonder if this is memory related thing? or what? Who knows? Someone from Adobe might… Well anyways. Here’s the same thing exploding 1.4 million particles in 1920×1200 resolution with smooth 60fps.

There are exactly 1 398 080 animated particles. This is the maximum amount since 64 full buffers is the limitation. It surely is enough. All the contrast will disappear because the screen is full of stuff. Still looks pretty darn rad. Especially thinking it’s Flash and dead and all..

Hopefully some update in the future will unleash this power for flash player in browser too.

— (UPDATE SOME HOURS LATER) —
Here’s a 1.4 million particle demo for you to test. It’s heavy shit. Be patient. I mean it :D
Also added a standalone version that you can play with your flash standalone player. That will run well in decent machine. (after unoptimized initialization)
– Browser version.
– Standalone version.

Prime Sperm of Sibelius. Music http://soundcloud.com/djlo/sibelius-finlandia-op-26-version-for-male-choir-and-orchestra

and of course.

Real
time

Video
capture

Abandoned Sperm of Sibelius. Music http://soundcloud.com/yourhips/sibelius

These are a bit heavier then last ones. Too bad if you have slow computer and you hate classic music. :/

Here’s how I can ‘individually’ move massive amount of particles.
1. Give them start and end point.
2. Tween between those points in vertex shader.
3. Send only time value to vertex shader in every frame.

No there’s no catch. The movement doesn’t have to be linear. AGAL can do this thing called mathematics and is quite awesomely fast in it. Familiar easing functions can be rewritten into AGAL. Upper flash is an example use case of Robbie‘s legendary ExpoOut function.

And here’s example of ExpoIn function.

Real
time

Video
capture

I’m working on a new particle engine (evoSpicyParticleEngine). That I’ll open source when it’s ready enough. Here’s some taste of it. There are 200 000 individually moving z-sorted particles. :D

I found the browser player a bit slow. Or well.. I know it sounds ridiculous to whine “it can only run some hundred thousand particles… nyyh nyyh” when we only had few thousand couple months ago. BUT the local standalone player runs way way way faster! On my OSX it can run almost million in full hd resolution. Maybe browsers limit the power or what?

I just checked and yeah… I had almost a year break from releasing experiments. The very core content of my blog. What ever. Here’s another particle experiment. There are 87 380 balls doing their thing. I think this will lead to open source Stage3D particle engine lib and hopefully I end up adding these monsters to Away3D someday soon too.

This isn’t a perfect system yet and not fully optimized. Anyway.

What this thing does is:
1. Set start and ending points for particle vertices. (2304 in every frame)
2. GPU will tween between those points.
3. Plus all sort of funky stuff with colors and there’s light at the top of every particle stream. GPU will calculate distances to those and add more value to colors. As you can see.

Anyway that was the plan. In this case it started to behave funny when I adjusted my AGAL codes and the rule explained doesn’t fully apply anymore. So what I released here is a mistake, but I think it feels and looks great!

Here’s a link to that cool song.

You can watch it now or read this thing first. This post contains spoilers but it might help you to understand demo better.

There are several ways to view this demo:

1. The blazing performance and highest quality way (recommended):
a) Download the standalone player here. (NOT THE DEBUGGER)
b) Download the swf + mp3 here and open it with that player.

2. Play in browser. Great quality but not as good in performance.
a) By visiting this link

3. Watch the video.
a) HD Video at YouTube

Making of ‘Sparkle a Moment’ by Away3D Development Team and Evoflash

It was 18th of August when Rob Bateman, the head man of Away3D framework (the worlds most advanced open source flash 3D engine), sent me an email and asked for a demo for Robbie’s and Richie‘s session at Adobe MAX 2011. We did this sort of collaboration with Away3D guys last year. Although it was a insane job I was stoked to burn my self out a bit again for sake of art and passion for this sport.

I introduced an idea, made a little synapsis and created a rough mood board.
Minomus made the track for me last Spring for Assembly demo which I never coded so we had that with us from point zero. (a very good thing.)
Here’s the synapsis:

“Into the endless darkness two light spheres are born. Slowly giving meaning and life to surroundings. In gentle moving light of spheres objects are showing their forms. Objects are symbols of science, history, society, evolution. nature, liquid and time. When a sphere come close to object it’ll be shown by the light of sphere and can become life. It can start levitate, sparkle, move, glow or break apart. Eventually spheres will light up everything. All is glamorous and flourishing. Suddenly spheres pop into little sparkles and disappear. Everything turn into deep endless darkness again.”

People liked it (or was too busy to dislike it) so I got a green light to do the direction again. The Flash on the Beach was closing and everybody was way too busy and could not work for demo. We had a little meet up at the conference and could pick up models for the demo. Other then that it was just drinking and fooling around. After all kind of delays we ended up creating this demo in less then two weeks. Fabrice Closier and Richard Olsson was working with 3D models and Ringo Blanken did examples of physics. David Lenaerts committed a stack of his special effects and went to vacation. I tried to get things together. Worked epic hours with camera cuts, setup elements and desperately struggled to get something to screen so we could move on. Once we got things rolling the process was smooth as nylon. With this kind of abstract idea it’s very important to get something done and move on from there. It took a lot of everybody’s faith to get demo in state that it started to make at least some sort of sense. I’m quite sure it won’t make sense to most of the viewers after all, but at least we got it done!

Here are screenshots of scenes and some comments about them:

Sparkles are born in to darkness. Cold and warm light. You know… Symbols of Yin and yang, thermodynamics, sweet and bitter, life and death… all that kind of shit.

A clock. And cogs. Symbols of time obviously, but also progress.

Eagle. Is a historical symbol of strength, courage, farsightedness and immortality. It is considered to be the king of the air and the messenger of the highest Gods. This is obviously nonsense as we’ll witness in near future where everything eventually vanish. (talking about the demo not 2012…) Eagle is also an animal and represents the Animalia.

We dive under water and what we find? no.. it’s not salmon. It’s tuna and David’s awesome realtime mirror ball. Fish is of course part of Animalia, but also very important symbol in various of religions.

Lunar eclipse is here to remind us of cycling nature of world. As a symbol it’s very important in all sort of nonsenses like astrology. It also looked nice with radial blur.

The train leads us to humans and society. It’s developing extremely fast. The technological progress seems invincible. It slithers through all obstacles.

We’ll move on to military tanks. While band plays ironic smooth jazz they drive in perfect formation. Firmly and certainly jumping off the cliff and form pile of junk.

We follow sparks and see Hercules, the demigod, the son of Zeus at the bottom of highest stone pole. Eventually he will fall.

Demogod now fall into pile of stones. Gravity fade down. Sparkles vanish and void darkness continues.

Simply put demo is about lifespan. It’s constant property that everything has.

I’ve been mostly listening to these records during past two weeks while doing demo:
Tori Amos – Night of Hunters and Ladytron – Gravity The Seducer
They might explain this madness a bit.

Big hugs to Away3D guys Rob, Richard, David and especially to Fabrice and Ringo who did a lot of work and had solution to all problems we faced during this process.

Ahh it feels good to have GPU support in Flash at last! Until next time…

Ohh and here’s guys shared session with Flare3D guys at Adobe MAX.