First thing I would want to know is if power and ground ground are available to the blower motor when it is failing. If so, then you have a faulty blower motor. If either the power or ground drops out,...then you need to find out why. Do you have a test light?
At the blower motor: I would want to see power available on the Pink White wire any time the car is running. I want to see some ground availability anytime the fan switch is selected to a speed.
I agree with Electro and would also suggest checking the circuits with the blower switch on high since on most cars that bypasses the resistor pack . In fact usually if the resistor pack fails the blower will still run when switched to high speed on most vehicles . Besides the resister pack the most likely bad guys are the speed selector switch and of course the blower motor . I almost forgot to mention it could be the relay. Since this sounds like an intermittent problem if you have another relay identical to the one for the blower you could swap the two as a process of elimination .