I would definately replace the rivets. As for replacing the patch...Personal Preference prevails. If it were me, would replace the rivets and while the patch was off I'd radius the corners and re-install. The radiused corners make the patch look professional (My Opinion) and they reduce the likelyhood of the sharp corner catching your clothing, or worse flesh.
Keep us posted.
"One of the best lessons I've learned is that you don't worry about criticism from people you wouldn't seek advice from."
William C. Swinney