The chief problem I see with video-only answers is problems for moderators. Viewing a video and checking its contents can be very time-consuming, and ultimately all it ensures is that the video actually contains an answer.
It is not, however, a metric of whether or not the answer itself is bad.
An answer that contains nothing but a video, just like a link-only answer, or an answer that contains only a link and very little explanation, is of low-quality, but still could be an answer in itself.
If it is not an answer, it should be treated as such - flagged or deleted. Flags will help moderators weed out which of these low-quality video answers are non-answers or too low-quality to be answers in themselves.
Realistically, video linking should be more restricted than any other type of linking because of this burden, but until it becomes so, we should treat answers with a video link as we would any other answer. That is to say, a good answer is still good, and a bad answer is still bad, regardless of whether or not a video is attached.