Never had any "build up" in the loofa pad, not the sponge. My first has lasted over five months and is beginning to show wear, so I'll probably cough up another $3 for a new one at Wal-Mart. Cheap, feels great, I don't like the little gritty stuff in the exfoliating soaps and lotions. It gets stuck on my bath soap too. The loofa pad works for me, but each to his own.