Well i would say that if you want to say yeah, then like say yeah or sure, and make sure you smile :)
But then if you want to say no, then be like " Well.. uhm sorry i don't (stutter and stare at them sorry-faced) I gotta go, sorry bye " I think that's the easiest way to let them down.
HOPE IT HELPED!