메뉴
R tokenizers package for text tokenization. Use for fast, consistent tokenization of text.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | tokenizers |
| description | R tokenizers package for text tokenization. Use for fast, consistent tokenization of text. |
Fast, consistent tokenization of natural language text.
library(tokenizers)
# Tokenize into words
tokenize_words("This is a test sentence.")
# Multiple texts
texts <- c("First sentence.", "Second sentence.")
tokenize_words(texts)
# Lowercase
tokenize_words(text, lowercase = TRUE)
# Keep punctuation
tokenize_words(text, strip_punct = FALSE)
# Keep numbers
tokenize_words(text, strip_numeric = FALSE)
# Stopwords
tokenize_words(text, stopwords = stopwords::stopwords("en"))
# Tokenize into sentences
tokenize_sentences("First sentence. Second sentence!")
# With abbreviations
tokenize_sentences(text, strip_punct = FALSE)
# Single characters
tokenize_characters("hello")
# Character shingles
tokenize_character_shingles("hello", n = 3)
# Word n-grams
tokenize_ngrams("This is a test", n = 2)
# Range of n-grams
tokenize_ngrams("This is a test", n = 2, n_min = 1)
# Skip-grams
tokenize_skip_ngrams("This is a test", n = 2, k = 1)
# Split by paragraphs
text <- "First paragraph.\n\nSecond paragraph."
tokenize_paragraphs(text)
# Split by lines
tokenize_lines("Line 1\nLine 2\nLine 3")
# Custom pattern
tokenize_regex(text, pattern = "\\s+")
# Tokenize and stem
tokenize_word_stems("running cats jumping")
# With language
tokenize_word_stems(text, language = "english")
# Penn Treebank style
tokenize_ptb("It's a test.")
# Twitter-aware tokenization
tokenize_tweets("Hello @user! Check out #rstats http://example.com")
# Count words
count_words("This is a test sentence.")
# Count sentences
count_sentences("First. Second. Third.")
# Count characters
count_characters("hello")
R language data analysis and visualization skill. Use when user asks to (1) run R scripts or code, (2) install/update R packages, (3) perform data analysis with R, (4) create visualizations with ggplot2/plotly, (5) statistical analysis, (6) data manipulation with tidyverse/dplyr/data.table. Triggers on keywords like "R语言", "R脚本", "ggplot", "tidyverse", "数据分析", "可视化".
R DALEX package for model explanations. Use for explaining complex machine learning models.
R iml package for interpretable ML. Use for model-agnostic interpretability methods.
R lime package for local explanations. Use for explaining individual predictions with local interpretable models.
R packages for ML interpretability. Use for explaining and interpreting machine learning models.
R vip package for variable importance. Use for computing and visualizing variable importance scores.