Playing sound and changing background color - revisited

[tonybabb]

Hello again John,

Happy New Year !!

Sorry this is such a long post but you may recall that several months ago you kindly assisted me to create a talking phrasebook as a smartphone app, when a user tapped a phrase on the screen a sound clip played for a few seconds and the background color changed. We got it working on my PC but when I converted it to an app using Intel XDK (New) and ran it on my Samsung Galaxy S4 smartphone it had the problem that after playing some (20+) sound clips the sounds just didn't play, no error message or anything.

To cut a long story short I have recreated the app using the Intel App Framework v2 in place of jQuery Mobile and the Cordova v2.9 js library in place of jPlayer and it now will play sounds as many times as I want - the underlying problem, I believe, was that the Android OS has finite audio resources and when Android plays a sound file it allocates some resources and does not release the resources when it has finished playing the sound file. It then allocates additional resources each time a sound file is played, eventually it runs out of resources and stops playing. The app programmer has to explicitly release the resources when the sound file has stopped playing. I also had some other issues with jQuery Mobile relating to speed and reliability when positioning the page header/ footers on my smartphone and these also appear to have been fixed with the new version of the app.

So, now that I have the sound working I'm reconstructing the look/ feel/ functionality I previously had with jQuery Mobile/ jPlayer. Specifically I'm trying to merge the code I created to play sound files using cordova with the code you created to play sounds using jPlayer and also to change the background color. I spent some time trying to do this but was unable to follow your js - still learning js but I am getting better.

I would really appreciate your assistance to merge these two versions. Below is the code I created and following that the code you created. Unfortunately I can't show you this on a website because the App Framework doesn't support PC type browsers very well and has device specific drivers.

CURRENT VERSION USING APP FRAMEWORK 2 / CORDOVA 2.9

I currently use a button to play the sound file, of course I'd like to replace this so the user taps anywhere on a phrase to play the associated sound

<input type="button" onClick="soundclip('ageCuantos2F.ogg')" value=" Play Cuantos Anos "> ;
and here's the soundclip function it calls, note that it needs to get the path from the web root and prepend that to the soundfile name. Also, when creating the Media Object that is where I specify the stop/ release functionality to be completed when the audio file stops onSuccess or onError. The cordova API is documented here http://cordova.apache.org/docs/en/2.9.0/cordova_media_media.md.html#Media

Here's the js function called when the button is pressed

<script type="text/javascript">
function soundclip(filename) {
var my_media = new Media(getWebRoot() + "/" + filename,
function() { my_media.stop(); my_media.release();},
function() { my_media.stop(); my_media.release(); console.log ("Audio play error - " + filename)});
my_media.play() ;
} ; // End soundclip
</script>

<script type="text/javascript">
function getWebRoot() {
"use strict" ;
var path = window.location.href ;
path = path.substring( 0, path.lastIndexOf( '/' ) ) ;
return path ;
}
</script>

jQUERY MOBILE/ jPLAYER VERSION

Here's the html


<div class="english"> <!-- Start English -->
How old are you?
</div> <!-- End English -->
<div class="wrapper">
<a class="formal spanish" data-trigger="howoldf" href="ageCuantos2F.mp3">F: Cuántos años tiene?</a>
<a class="formal trigger" id="howoldf" href="#">F: Cuántos años tiene?</a>
<div class="spacer"></div>
<a class="informal spanish" data-trigger="howoldi" href="ageCuantos2I.mp3">I: Cuántos años tienes?</a>
<a class="informal trigger" id="howoldi" href="#">I: Cuántos años tienes?</a>
</div> <!-- End .wrapper -->

and here's the js you created that I just couldn't understand - sorry.

<script type="text/javascript">
jQuery(function($){
var highlight = 'yellow', origcolor = 'transparent', istouch = !!('ontouchstart' in window) || !!('ontouchstart' in document.documentElement) || !!window.ontouchstart || (!!window.Touch && !!window.Touch.length) || !!window.onmsgesturechange || (window.DocumentTouch && window.document instanceof window.DocumentTouch);
var $s = $('.spanish').each(function(i, snd){
$(snd).jPlayer({
ready: function() {
$(snd).jPlayer("setMedia", {
mp3: snd.href,
ogg: snd.href.replace(/\.mp3$/, '.ogg') });
var playstop = function (e) { e && e.preventDefault();
var $this = $(this);
if(!$this.data('play')){
$this.stop(true, true).css({backgroundColor: highlight});
var $playing = $('a[data-playing="true"]');
if($playing.length){
if(istouch){
playstop.apply($playing.get(0));
} else {
$playing.trigger('click');
}
}
} else {
$this.stop(true, true).animate({backgroundColor: origcolor}, 1000);
}
$(snd).jPlayer($this.data('play')? "stop" : "play");
$this.data('play', !$this.data('play'));
$this.attr('data-playing', $this.data('play'));
};
if(istouch){
var dt = this.getAttribute('data-trigger'), $dt = $('#' + dt);
dt = document.getElementById(dt);

dt.addEventListener('touchstart', function(e){$dt.data({tstart: new Date().getTime(), tx: e.pageX, ty: e.pageY});}, false);
dt.addEventListener('touchend', function(e){
var ds = $dt.data('tstart');
if(!ds){return;}
var vx = Math.abs(e.pageX - $dt.data('tx')), vy = Math.abs(e.pageY - $dt.data('ty'));
if(vx < 10 && vy < 10 && new Date().getTime() - ds < 400){
playstop.apply(this, [e]);

$dt.data({tstart: null, tx: null, ty: null});
}
}, false);

} else {
$('#' + this.getAttribute('data-trigger')).click(playstop);
}
},

ended: function(){
$('#' + this.getAttribute('data-trigger')).attr('data-playing', 'false').data('play', false).stop(true, true).animate({backgroundColor: origcolor}, 1000);
},

swfPath: "./"
});
});
});
</script>

I would really appreciate your assistance with this, it's been a long road but I can finally see a light at the end of the tunnel.

Regards,

Tony

[jscheuer1]

As a javascript programmer, I can pretty much follow that. You could just manually supply the path, you don't really need getWebRoot. It's the release that we need.

You haven't mentioned whether or not you've dropped jQuery altogether. We were using that and a small part of the UI for the special effects (color fading). As long as you're still willing and able to do that, adapting this code should be fairly simple.

Are You?

And, for this exercise I would assume we no longer need to worry about it working on the PC, right?

[tonybabb]

Hi John,

Thanks for your comments.

you don't really need getWebRoot - that's right I could hard code the full path but then I'd need a separate version for the iPhone app as I'm pretty sure the iPhone path would not include something called "...android_asset..." as part of it's path.


You haven't mentioned whether or not you've dropped jQuery altogether. to be honest I had no idea what jQuery did but so far I have concentrated on fixing the disappearing sound problem so when I recreated the app using App Framework and cordova I just created a couple of pages and on each page had a series of buttons which each played a sound file. Now that the sound is working I'm trying to get back to the look/ feel I had previously. Your comments caused me to look up jQuery and that has helped a lot, I can see I have much to learn but at least now I think I can read and begin to understand some of your code, before this I had no idea what I was looking at.

So, am I right thinking the best approach is to include jQuery and then modify your code to call the function I created (soundclip ('filename')) instead of calling jPlayer? I took a look at the jPlayer documentation also and think I can better understand the syntax for the call from your code. Finally some of this stuff is starting to make sense.

As for it not working on the PC, we don't need to worry. The Intel Cross Platform Development Kit documentation said using App Framework probably wouldn't work on a PC, I tried and the documentation was correct so I'm doing all testing using the Intel emulator on my PC. There is a legal/ licensing issue with Intel playing mp3 format files apparently, so would it make a difference to your code if I called the ogg version in place of the mp3 version - doing this I can hear the audio using the emulator. I have ogg and mp3 versions of all audio files. Once I've tested using the emulator I upload/ Build/ Download into my smartphone. A bit convoluted but it works.

Regarding the ogg and mp3 format, did I understand a previous comment you made correctly that you are recommending I include both formats for all audio?

Thanks,

Tony

[jscheuer1]

The jPlayer code was more complicated than I think the Cordova code will need to be because we were using two links to play each sound. One was seen by the user. The other was unseen and initialized a jPlayer instance for each sound. With Cordova it looks as though we can initialize a sound instance at will to a variable any time we need it and through that variable (among other optional things) - play, stop, or remove that sound instance.

Including both formats for audio is the recommended method when using jPlayer because it's more cross platform/browser. But if all of your target devices/browsers support ogg with the Cordova code, then using only ogg is fine.

I was thinking about this and came up with two questions that should help me adapt this to Cordova. With the input:


<input type="button" onClick="soundclip('ageCuantos2F.ogg')" value=" Play Cuantos Anos ">

I get it that if you click it once, it plays. But what happens if:

You click it a second time before it's finished playing?

You click another one before this one stops playing?


I don't want to know what should happen, I know that, in both cases, the one that is playing should stop, and if you've clicked on another one one, that one should start. I need to know what actually happens. Does it play two at once?

[tonybabb]

Hi John,

I used buttons as a simple way to play the sound, but I prefer the look and feel we had previously when the user could tap anywhere on the Spanish phrase. To answer your question, I tried pressing multiple buttons before the previous one had finished playing and it plays multiple sounds at the same time.

Regards,

Tony

[jscheuer1]

OK, as I cannot test this there might have to be a bit of back and forth before we get it working. In the code that follows, you may need to modify the rules for the css selector .spanish a little (get it to point to your background image, etc.) in order to get each sound link to look initially the way you want it. After the style section notice there's an HTML comment that tells you where to include the external script tags for APP FRAMEWORK 2 and CORDOVA 2.9. Put them there. Following that are the external tags for jQuery 1.6.4 and the color fading part of the UI script. You should still have both of these scripts. You can change their paths if needed to reflect their actual locations. As written, they should be in the same folder as the page. I got rid of the getWebRoot function after all because, now that these are links, the browser will read their href properties as the full path and pass that to the playstop function which I built from the soundclip function you provided. About the links, the href refers to the location of the sound. If it's in another folder, that needs to be included in the href. These demo sounds are assumed to be in the same folder as the page.

Any questions or problems, just let me know:


<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Demo using APP FRAMEWORK 2 / CORDOVA 2.9</title>
<style type="text/css">
.wrapper {
margin: 8px;
}
.spanish {
display: table-cell;
padding: 3px 5px 3px 110px;
background: transparent url(audio-30.gif) 0 center no-repeat;
height: 78px;
width: 150px;
vertical-align: middle;
text-decoration: none;
font: bold 95% verdana, helvetica, arial, sans-serif;
border-radius: 15px;
}
.spacer {
height: 5px;
}
</style>
<!-- External Script Tags for APP FRAMEWORK 2 / CORDOVA 2.9 should go here -->
<script src="jquery-1.6.4.min.js" type="text/javascript"></script>
<script src="jquery-ui-1.8.24.color.min.js"></script>
<script type="text/javascript">
jQuery(function($){
var highlight = 'yellow', origcolor = 'transparent', curSnd = null,
istouch = !!('ontouchstart' in window) || !!('ontouchstart' in document.documentElement) || !!window.ontouchstart || (!!window.Touch && !!window.Touch.length) || !!window.onmsgesturechange || (window.DocumentTouch && window.document instanceof window.DocumentTouch);
function playstop(e){
e.preventDefault();
var $this = $(this);
if(curSnd){
if(this === curSnd.tag){
curSnd.sound.stop();
if(curSnd){curSnd.sound.release();}
curSnd = null;
$this.stop(true, true).animate({backgroundColor: origcolor}, 1000);
return;
}
curSnd.sound.stop();
if(curSnd){
curSnd.sound.release();
$(curSnd.tag).stop(true, true).animate({backgroundColor: origcolor}, 1000);
curSnd = null;
}
}
$this.stop(true, true).css({backgroundColor: highlight});
var filename = this.href.substring(this.href.lastIndexOf('/'));
curSnd = {tag: this, sound: new Media(
this.href,
function() { if(curSnd){curSnd.sound.stop(); curSnd.sound.release(); curSnd = null; $this.stop(true, true).animate({backgroundColor: origcolor}, 1000);}},
function() { if(curSnd){curSnd.sound.stop(); curSnd.sound.release(); console.log ("Audio play error - " + filename); curSnd = null; $this.stop(true, true).animate({backgroundColor: origcolor}, 1000);}}
)};
curSnd.sound.play();
} // End playstop
$('.spanish').click(function(e){e.preventDefault();}).each(function(i, snd){
if(istouch){
var $snd = $(snd);
snd.addEventListener('touchstart', function(e){$snd.data({tstart: new Date().getTime(), tx: e.pageX, ty: e.pageY});}, false);
snd.addEventListener('touchend', function(e){
var ds = $snd.data('tstart');
if(!ds){return;}
var vx = Math.abs(e.pageX - $snd.data('tx')), vy = Math.abs(e.pageY - $snd.data('ty'));
if(vx < 10 && vy < 10 && new Date().getTime() - ds < 400){
playstop.apply(snd, [e]);
$snd.data({tstart: null, tx: null, ty: null});
}
}, false);
}
});
});
</script>
</head>
<body>

<div class="wrapper">
<a class="spanish" href="test.ogg">Ay Caramba</a>

<div class="spacer"></div>
<a class="spanish" href="ringding.ogg">Ring Ding</a>

</div>

</body>
</html>

[tonybabb]

Hi John,

Thanks so much for this.

I created a new version and uploaded it to see if it would work at all on my PC. It partially works with IE or Firefox, doesn't work at all with Chrome . The uploaded version can be seen here http://www.lawenforcementspanish.com/mle37 . The only difference between using IE/FF on my PC and the Intel emulator on my PC is that the sound only plays when I use the emulator - which I think you'd expect as it's calling an Android specific API for sound.

Here are links to screenshots of the Intel emulator. The first shows the main menu - which works
http://www.lawenforcementspanish.com/mle37/test-1.png
The second shows the content page identified as Age, Weight & Height and again looks correct although I need to do a little work on layout
http://www.lawenforcementspanish.com/mle37/test-2.png
When I click on Ay Caramba or Ring Ding the emulator plays the sound and then goes to the page shown below, IE/FF just go direct to the page.
http://www.lawenforcementspanish.com/mle37/test-3.png

I also tried building the app as-is and installed on my phone. It behaves on my phone the same way that it behaves on the emulator. One thing I noticed when I played either of the sounds is that it displays the page with garbage before the sound file has finished playing. Also, Ring Ding is fairly long so after the garbage page displayed I clicked "back" in the top left corner and then clicked Ring Ding to stop the sound, the sound stopped and again the garbage page was displayed. One last thing, I played Ay Caramba 50+ times with no problems. Hope this helps you understand what happened.

I have a little work to do on layout and colors but this looks very close, I'll study the code some more to try and understand what you're doing.
Thanks,

Tony

[jscheuer1]

When you say it switches to the page shown in test-3.png after playing the sound, that's the actual ogg file. It shouldn't do that, and wouldn't do that if this were treated as a normal link. At least I'm pretty sure of that, because I have its click behavior default prevented.

In any case the menu and all other links appear to get treated thus:


If it has a hash href and that hash is also on the page as an id, it slides to that id.


If it has a hash href and there is no id to match, it does nothing.


If it has a link (href points to an actual page or other resource), that link is loaded via AJAX. (If it's a link and has a hash, if that hash is on the page loaded via AJAX as an id, it goes there after loading.)


Our sound links are to actual resources, so they are being treated like that after the sound plays. Is there a way to disable that? I seem to remember that there was a class name that could be added to a link to prevent it from loading a page/resource. If there is these links should have that added. Otherwise we will have to go back to using dummy links as triggers. Before we had to do that because jPlayer replaced the sound links with invisible players. Here we would have to do it in order to prevent the sound file from loading as a page after playing. That is, unless there's a class name or something we can use to prevent that.

I was just Googling the framework 2 docs and perhaps the data-ignore="true" attribute will work for that:


<div class="wrapper">
<a class="spanish" data-ignore="true" href="test.ogg">Ay Caramba</a>

<div class="spacer"></div>
<a class="spanish" data-ignore="true" href="ringding.ogg">Ring Ding</a>

</div>

[tonybabb]

Hi John,

Thanks again. You were right, - Data-ignore=true stopped the extra page from displaying. I'm not sure if I screwed up something else or not but I have now noticed that the background color doesn't change to white when the sound file has finished playing, it only changes to white if I tap before the sound has finished playing or if I tap a second time after it has finished playing. I don't recall it doing this before so I went back and recreated mle37, this time calling it mle38 and it still behaves the same way.

When I made the change to mle37 (for data-ignore) I'm pretty sure it worked correctly -I think - so I took the opportunity to clean up my code - just aligning div statements under each other, I didn't consciously change anything else. So I'm not sure what happened. I can try going back and recreating again from an earlier version but I'm not sure what I did to cause this so I'm hoping this is something you can more easily point out.

The recreated version can be seen here www.lawenforcementspanish.com/mle38 (http://www.lawenforcementspanish.com/mle38)

Thanks so much for your patience.

Tony

[jscheuer1]

I don't think you would have been able to tell that before because it would always go to the ogg file (test-3.png) before that would be able to happen.

In any case, I story boarded it and found that with a normal object there were problems with calling it's stop method in it's success function if the success function also called it's stop method. It was too redundant (looping), according to the docs success is run on completion of both stop and play, so browsers refused to do that part. I think the only crucial thing that needs to be run there is release(), that and any other things not related directly to the Media object that we want to do at that stage (end of a successful play or stop operation), like reverting the background.

To that end I came up with this:


<script type="text/javascript">
jQuery(function($){
var highlight = 'yellow', origcolor = 'transparent', curSnd = null,
istouch = !!('ontouchstart' in window) || !!('ontouchstart' in document.documentElement) || !!window.ontouchstart || (!!window.Touch && !!window.Touch.length) || !!window.onmsgesturechange || (window.DocumentTouch && window.document instanceof window.DocumentTouch);
function playstop(e){
e.preventDefault();
var $this = $(this);
if(curSnd && curSnd.sound){
if(this === curSnd.tag){
curSnd.sound.stop();
return;
}
curSnd.sound.stop();
}
$this.stop(true, true).css({backgroundColor: highlight});
var filename = this.href.substring(this.href.lastIndexOf('/')), myMedia = new Media(
this.href,
function() {
myMedia && myMedia.release();
curSnd = myMedia = null;
$this.stop(true, true).animate({backgroundColor: origcolor}, 1000);
},
function(e) {
myMedia && myMedia.release();
curSnd = myMedia = null;
$this.stop(true, true).animate({backgroundColor: origcolor}, 1000);
window.console && console.log ("Audio play error - " + filename + "\ncode: " + e.code + "\nmessage: " + e.message);
}
);
curSnd = {tag: this, sound: myMedia};
curSnd.sound.play();
} // End playstop
$('.spanish').click(function(e){e.preventDefault();}).each(function(i, snd){
if(istouch){
var $snd = $(snd);
snd.addEventListener('touchstart', function(e){$snd.data({tstart: new Date().getTime(), tx: e.pageX, ty: e.pageY});}, false);
snd.addEventListener('touchend', function(e){
var ds = $snd.data('tstart');
if(!ds){return;}
var vx = Math.abs(e.pageX - $snd.data('tx')), vy = Math.abs(e.pageY - $snd.data('ty'));
if(vx < 10 && vy < 10 && new Date().getTime() - ds < 400){
playstop.apply(snd, [e]);
$snd.data({tstart: null, tx: null, ty: null});
}
}, false);
}
});
});
</script>

Use it in place of its equivalent in the mle38 code.

[tonybabb]

Hi John,

Thanks. I substituted the code and it didn't fix the problem. When I click on Ay Caramba it changes to a yellow background, plays the sound and the background stays yellow when the sound finished. When I click again it doesn't play the sound and background stays yellow. When I click a third time it doesn't play sound and the background turns white. When I click a fourth time it plays the sound and background turns yellow.

When I played the longer clip (Ring Ding) I clicked before it finished playing and the sound stopped, background stayed yellow, then I clicked again and no sound but the background turned white, click a fourth time and it played the sound and background changed to yellow. Essentially exactly the same sequence whether the sound finished normally or is clicked on before the end.

All tests were run using the Intel emulator. The current version can be seen at www.lawenforcementspanish.com/mle38 (http://www.lawenforcementspanish.com/mle38)

Thanks again,

Tony

[tonybabb]

Hi John,

Good news - I think we're OK. The emulator gave the results above and previously behaved consistently with the smartphone. Anyway, I tried doing a build/ download and ran it on my phone and it worked perfectly. I'll do a little more testing but for now it looks good - well actually it looks great to me !! Thank you again for your patient support and generous explanations of what you were doing. It really helped.

Tony