The drill sizes come from military specifications for buck rivets. The latest spec is MIL-R-47196A(MI).
When bucked rivets are properly set, they expand to completely fill the slightly oversized hole.
Olympic rivets and pop rivets, on the other hand, act by clamping two sheets together, without expansion of the shank.
I believe that the hole sizes for buck rivets got "inherited" by the blind rivets, and that is where the numbers come from.
Personally, when installing pop
rivets I just use the rivet size; 1/8" drill for 1/8" rivet. I never use Olympic rivets so I won't comment on the correct drill size. For bucked rivets I use the number size drills you listed.
Edit: Further research on Olympic rivets indicated that the 5/32" Olympic uses a #20 hole (.162-.167). That comes from http://www.boltproducts.com/marson/b...bulb-tite.html
,near the bottom of the page. So there are lots of choices for what size to use.