Lyme disease is one of the most common infectious vector-borne diseases in the world. We extensively studied the effectiveness of convolutional neural networks for identifying Lyme dis-ease from images. Our research contribution includes dealing with lack of data, multimodal learning incorporating expert opinion elicitation, and automation of skin hair mask generation.