It would drive me nuts after spending good money on a nice set of magnetized nut drivers and within a couple months of use, the fasteners wouldn’t even begin to hold due to the buildup of dirt and shavings. I needed a solution that involved equipment I had with me every day: plumber’s putty.
Take a ball of warm putty and push it down into the end of your nut driver, leaving a fair amount sticking out. Now freeze it, or get it as cold as possible. Ok, yes I admit I casually dropped my driver in a customer’s freezer a few times.