Diagnosing erythema migrans (EM) skin lesion, the most common early symptom of Lyme disease using deep learning techniques can be effective to prevent long-term complications. Existing works on deep learning based EM recognition only utilizes lesion image due to the lack of a dataset of Lyme disease related images with associated patient data. Physicians rely on patient information about the background of the skin lesion to confirm their diagnosis. In order to assist the deep learning model with a probability score calculated from patient data, we elicited opinion from fifteen doctors. We also proposed a strategy for fusing the EM probability score from a deep learning based image classifier with the elicited probability score from patient data. The proposed approach ensures veto power for the patient data. The elicited probability score and the proposed fusing approach can be utilized to make image based deep learning Lyme disease pre-scanners robust.