As the help text in the Add Comment box states:
Use comments to ask for clarification or add more information. Avoid comments like "+1" or "thanks".
So as long as the comments are improving the post, they should be fine. If you are having a discussion about it, it belongs in chat/meta.
In the end, comments can be deleted later, so no big issues with having too many (so long as the post is improved and the comments are then cleaned up).
As far as how the system determines when it pops up, it is not strictly the number or comments. This question has 14 comments but I am not given the option to "Continue in chat". I would guess that it might stem from multiple users posting back and forth (which is a discussion and should be moved to chat).