It's an offensive defense. It's talking loud enough that it drowns out the voices in their heads that say "Oh, but you do... You do."
By boasting how much you don't care what others think, you're actually allowing their thoughts to control you. If you didn't actually care what they think, you wouldn't care that they know how much you don't care.
Confidence isn't not caring what other people think. Confidence is to not let what other people think change you. Every story I write, every picture I draw, every site I design, every idea I conceive... I care a LOT what other people think about them. I care that they'll like them. I care that they'll enjoy them. I care that they'll react well to them. Any artist, performer or worker who tells you otherwise is lying.
Everyone cares what other people think about them and the things they do. It's human. Somewhere deep inside, you care. I care. We all care. Confidence is staring down the thoughts of others about who you are, what you're doing or how you're doing it, and saying "this will work." Confidence is knowing there will be consequences and doing whatever it is you set out to do regardless.
You might be wrong. No one said doing something with confidence would necessarily equate to doing it right. But without confidence, whatever it is you're doing doesn't actually belong to you. It belongs to everyone you're trying to please. Your words, your actions, your behavior... All controlled by the invisible strings being pulled by everyone around you.
Cut the strings. Do what you do; be who you are. Do so with confidence. Let people react and let them own those reactions. You own your moments.