Abstract: Sketch Re-identification (Sketch Re-ID), which aims to accurately identify persons of interest from images captured by surveillance cameras using hand-drawn sketches, is crucial for criminal investigation, law enforcement, and missing person searches. Existing methods focus on alleviating the modality gap using semantic metrics, such as cross-entropy loss. However, fine-grained but modality-consistent information unfortunately becomes misaligned due to the inherent abstract nature of sketches. To address this issue, this paper proposes a novel $\textit{Optimal Transport-based Labor-free Text Prompt Modeling}$ (OLTM) network, which hierarchically extracts coarse- and fine-grained similarity representations guided by textual semantic information without any additional annotations. Specifically, a text prompt alignment module is well-designed, so that both fixed and learnable prompts on various person attributes are generated. Subsequently, an optimal transport algorithm is introduced to dig out interested local text representation, which plays a bridge to extract the consistent but semantic features from both sketch and image modalities. Additionally, instead of measuring the similarity of two samples by only computing their distance, a novel triplet assignment loss is further proposed, in which the whole data distribution also contributes to optimizing the inter/intra-class distances. Extensive experiments conducted on two public benchmarks consistently demonstrate the robustness and superiority of our OLTM over state-of-the-art methods.