WebGL animation with audio

I have created a panoramic demo, here:

Playing animation using WebGL works fine. Playing audio using the Audio tag works fine. However, playing both animation and audio has some problems:

  1. In Chrome, sometimes the audio will stop after a minute or so and won’t resume. But after Reload the music will probably finish. And yes, I DID use the audio.load() function.
  2. In Minefield it doesn’t work at all. So I had to state that demo works only in Chromium.

Did anyone had similar problems?

I’m having considerable difficulties with both the <audio> and <video> tags - Firefox’s implementation simply misses entire chunks of the specification - with no documentation or error/warning/info when you try to do something that it doesn’t support. Chrome’s interpretation of some of the requirements is also done in a non-useful way. Worst of all (for me) is that there appears to be no way to robustly tell the system that you want to download the audio in it’s entireity so it’s always ready to play at a moment’s notice (which is a serious pain in the butt for sound effects in games which need short effects that consume very little memory to be synchronized to the animation with great precision). Instead, they seem to insist on streaming even the shortest sounds from the server as if you were trying to play an hour-long HD movie or something!

So I’m not at all surprised that you’re having problems - but sadly, I don’t have anything specific I can offer in the way of advice about what to do about it.

Having said that, I doubt that WebGL is implicated - other than that it imposes a high CPU load and therefore will tend to make other problems more pronounced than they might otherwise be.

If anyone knows of a forum where HTML5 media tags are discussed with experts in the field I’d be eternally grateful for a link. This is a desperate situation for those of us trying to use WebGL to write games.

Couldn’t agree more. By the specs, the “preload” attribute of the <audio> tag can take one the below values:

  • “none” (won’t preload anything)
  • “metadata” (loads the audio description, not the actual audio)
  • “auto” (chooses between the above options)

Funny, but that’s all. There’s no option “preload_audio_samples”. So I don’t know why they called that attribute “preload”.

One more thing I discovered. When I’m playing audio from my local disk, when the song ends, the audio element sets the “ended” attribute to “true”. So I know when to stop my WebGL animation.

When I upload the same project to the remote web server, the “ended” attribute is never set. When the song ends, the timer continues running as if it’s still playing. There’s no sound in the speakers, but the timer runs. After leaving it to run for 20 minutes (and song lasts only 5 min) I assumed it would probably run forever. I’m not sure if it’s a bug in Chromium or something related to the server.

(This is WAY off-topic for a WebGL forum…but the more people know this, the better the chance that someone will be able to influence the right people to get it fixed)

There are really very few things that a basic game, a simulation, or a similar interactive application needs in the way of audio:

  1. Looping background music or ‘ambient’ sounds - crickets chirping, wind, surf crashing on the beach - whatever.
  2. The ability to play a short sound with low latency, on demand. Think “gunshot”…that’s the canonical use-case.
  3. The ability to prioritize your sounds to keep within whatever limits there are on the maximum number of sounds the system can mix.

With those three basic things, you can do all you need for a simple game, simulation, whatever. The Atari 2600 could do this back in 1977 - it’s hardly rocket-science!

But sadly…

  1. Both Firefox and Chrome fail to do looping adequately. (Firefox ignores the looping attribute - and Chrome puts a honking great delay in while it it thinks about how to re-open the streaming audio channel or whatever). Even triggering an event when the audio stops playing fails some of the time (as you’ve discovered)…and still results in a long gap in the performance. Looping can (and should) be perfect - accurate to the individual sound sample, so you don’t get a ‘click’.

  2. As you’ve found out - the specification itself completely fails us for uploading teeny-tiny (1 to 5 second) audio snippets because it insists on streaming them - giving you massively less controllable server hits - and no chance whatever of your players being able to pull the trigger and hear their guns go “BANG!” within less than a half second. Streaming is all anyone thought about…and it’s not the common use-case. For short audio snippets, there should be a “preload” option that says “just preload the whole damned thing for chrissakes!” - and the implementations should make sure that they can respond to a ‘play()’ command within (let’s say) 33mseconds if the sound is already in memory.

  3. There is no specification for the number of simultaneous sounds that the system has to be able to play…or even whether it can play more than one sound simultaneously at all !! There certainly isn’t a way to query the number of sounds it will support. So you can’t (for example) play your background music - and play some footstep sounds for your player’s avatar - and have the player shoot people - because playing two sounds at once might (or might not) result in the already-half-a-second-late “BANG” being muted in order to keep the footsteps playing! It’s kinda random right now - but Firefox often shuts off my music track when I make a gunshot - and doesn’t reliably turn it back on again afterwards…so even playing two audio sources at once isn’t reliable.

(I’m beating up on Chrome and Firefox mainly because I haven’t played with Safari, Opera, IE and others…I have no idea whether they are any better).

It needs a SERIOUS do-over. The specification needs some beefing up (a “preload” option that actually guarantees to preload the ENTIRE sound if there is room in memory). There should be a cast-iron guarantee that there will be not one single sound sample missed when you loop a sound and there need to be a specification for the minimum number of simultaneous sounds you can guarantee to be able to play (at least 4 please!) and a way to query the maximum number you can have to you can shut off the footsteps when the gun goes “BANG”). When that’s done, there needs to be a solid effort on behalf of the browser manufacturers to come up with a test suite and to work towards meeting the requirements to the letter before people claim to have implemented HTML5 <audio>.

Without that, the <audio> tag is mostly useful for people to put annoying ‘elevator music’ on their web sites…it’s certainly useless for interactive games and other somewhat demanding applications.

I have a proposal for the Khronos guys: Why don’t you do for audio what you did for OpenGL==>WebGL? Pick a well-liked, portable, pre-existing standard (OpenAL would be my choice - because it’s free/Free, it’s been around for a long time, it has been used in thousands of serious projects, including commercial games and it has good hardware support from sound card vendors). Let’s write a specification document and create the obvious JavaScript bindings for it that fit alongside the existing infrastructure (ie, the <audio> tag) and then promote the heck out of it to the corporate supporters of WebGL…Apple, Google, Mozilla. Let’s call it “WebAL”.

You have my vote! :slight_smile: