Menü
SkillsMP hat 3 Skills aus openvinotoolkit/openvino_tokenizers gesammelt. Öffne einen Skill, um Quelle und Details zu prüfen.
Es werden 3 von 3 gesammelten Skills angezeigt.
Validate a HuggingFace tokenizer with OpenVINO Tokenizers and OpenVINO GenAI. Use when: checking if a tokenizer converts and works correctly, verifying tokenizer/detokenizer accuracy, testing normalization steps, checking GenAI Tokenizer compatibility.
Quellsprache: Englisch
Diagnose tokenizer conversion issues for OpenVINO Tokenizers. Use when: tokenizer-checker reports failures, need to pinpoint root cause location (Python conversion vs C++ operation), identify which pipeline stage diverges, determine whether to use…
Quellsprache: Englisch
Fix Python conversion issues in OpenVINO Tokenizers. Use when: tokenizer-diagnostics reports root_cause_location=python, unsupported types need new handlers in hf_parser.py, pipeline step mapping or merging is incorrect, tokenizer_pipeline.py step classes…
Quellsprache: Englisch