I'm from the U.S. and the phrases "potter around" and "potter in the garden" are new to me. I have always used "putter".

My pronunciation of putter is "puh-ter" (same sound as under) and potter is "pah-ter" (same sound as father).