What's the best way/tool to cut or drill holes 1/8 inch to 5/16 inch in
diameter in thin gauge aluminum (for inserting potentiometers and
phono, power jacks, etc.)? I've tried using just a power drill and
genernal purpose drill bits and the results have been sort of ragged
(the edges of the hole are ragged and the process of drilling creates a
depression surrounding the drilled hole).

Get yourself a step bit. You just drill the starter hole, and then drill
with the step bit until you have the size hole you want. Much cheaper than
punches.
==========================
I am getting good results with 2 sizes of tapered reamer
Initial hole drilled 1/8 inch.
Small diameter reamer from 3 to 13 mm (1/2 inch)
Larger diameter reamer from minimum 6 mm to 30 mm ( approx 1.25 inch)

My reamers are Nipponese ,made by Futaba Tool Mfg. who produce 4 sizes ,of
which the 2 I use are size 1 and size 4

For larger holes (say for moving coil meters ) I drill multiple holes ,
knock out and finish with half round file.
The latter takes some time obviously , if it is to be done neatly.

Frank GM0CSZ / KN6WH