I would definately replace the rivets. As for replacing the patch...Personal Preference prevails. If it were me, would replace the rivets and while the patch was off I'd radius the corners and re-install. The radiused corners make the patch look professional (My Opinion) and they reduce the likelyhood of the sharp corner catching your clothing, or worse flesh.
Keep us posted.
"You wouldn't worry so much about what others think of you if you realized how seldom they do."