Do you find it hard to say no?

Do you ever catch yourself saying "I can't do that" when what you really mean is "I won't do that" but you don't know how to say it nicely?