NVIDIA/cudf-spark

最近记录的来源活动: 2026年10月23日 04:56

SkillsMP 收录数据更新: 2026年11月5日 19:57

已收集 skills: 8

GitHub 星标: 1,011
GitHub Forks: 316

用 AI 助手安装

复制这段 Prompt，发给你正在使用的 AI 助手。

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/NVIDIA/cudf-spark.

仓库 README

仓库 README 来源：README.md

查看 README 源文件

展开完整 README

# NVIDIA cuDF plugin for Apache Spark NOTE: For the latest stable [README.md](https://github.com/NVIDIA/cudf-spark/blob/main/README.md) ensure you are on the main branch. The NVIDIA cuDF plugin for [Apache Spark](https://spark.apache.org) provides a plugin library that leverages GPUs to accelerate processing via the [cuDF](https://github.com/NVIDIA/cudf) (CUDA DataFrame) libraries. Documentation on the current release can be found [here](https://nvidia.github.io/cudf-spark/). To get started and try the plugin out use the [getting started guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/overview.html). [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/cudf-spark) Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be sunset over time. GitHub links from `spark-rapids` will redirect to `cudf-spark`. Artifact names will remain the same for now. ## Compatibility The SQL plugin tries to produce results that are bit for bit identical with Apache Spark. Operator compatibility is documented [here](./docs/compatibility.md) ## Tuning To get started tuning your job and get the most performance out of it please start with the [tuning guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/tuning-guide.html). ## Configuration The plugin has a set of Spark configs that control its behavior and are documented [here](docs/configs.md). ## Issues & Questions We use github to track bugs, feature requests, and answer questions. File an [issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose) for a bug or feature request. Ask or answer a question on the [discussion board](https://github.com/NVIDIA/cudf-spark/discussions). ## Download The jar files for the most recent release can be retrieved from the [download](https://nvidia.github.io/cudf-spark/docs/download.html) page. ## Building From Source See the [build instructions in the contributing guide](CONTRIBUTING.md#building-from-source). ## Testing Tests are described [here](tests/README.md). ## Integration The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications. It is described [here](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/ml-integration.html). Currently, we are working with XGBoost to try to provide this integration out of the box. You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding. ## Qualification and Profiling tools The Qualification and Profiling tools are available in the [NVIDIA/cudf-spark-tools](https://github.com/NVIDIA/cudf-spark-tools) repository. Please refer to [Qualification tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/qualification/overview.html) and [Profiling tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/profiling/overview.html) for more details on how to use the tools. ## Dependency for External Projects If you develop functionality on top of the cuDF plugin (we currently limit support to GPU-accelerated UDFs) we recommend you declare the distribution artifact as a `provided` dependency. ```xml <dependency> <groupId>com.nvidia</groupId> <artifactId>rapids-4-spark_2.12</artifactId> <version>26.12.0-SNAPSHOT</version> <scope>provided</scope> </dependency> ```

正在加载仓库 README…

这个仓库中的 skills

软件开发工程师软件质量保证分析师与测试员

2 个职业分类 · 已分类 88%

skill

职业分类

描述

更新

check-shim-compatibility

未分类

Check cudf-spark shim source selection on a PR combined with its current target branch, provide independent cross-PR advisories, and identify required profile validation. Use for versioned sources, shared shim callers, or Spark/Databricks build profiles.

原文语言：英语

2026-10-23

udf-convert-to-cuda

软件开发工程师

Assists with converting a non-aggregating Apache Spark UDF to a native CUDA RapidsUDF using JNI and libcudf. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-cuda -> udf-benchmark). Use this skill when you have a CPU UDF with…

原文语言：英语

2026-07-30

udf-gen-test

软件质量保证分析师与测试员

Assists with generating a unit test for an Apache Spark UDF. This is step 1 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-* -> udf-benchmark). Use this skill when you have a CPU UDF and need to create a unit test for the UDF before…

原文语言：英语

2026-07-30

udf-judge-conversion

软件质量保证分析师与测试员

Reviews generated UDF tests and GPU/SQL implementations for robustness, anti-cheating, and GPU execution integrity. Use when the user requests a judge/review-agent pass, or when manually reviewing a completed conversion.

原文语言：英语

2026-06-25

udf-convert-to-sql

软件开发工程师

Assists with converting an Apache Spark UDF to a functionally equivalent Spark SQL expression. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-sql -> udf-benchmark). Use this skill when you have a CPU UDF with a unit test…

原文语言：英语

2026-06-25

udf-benchmark

软件质量保证分析师与测试员

Assists with benchmarking and profiling the performance of an Apache Spark UDF on the GPU. This is step 3 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-* -> udf-benchmark). Use this skill when you have a CPU UDF and a RapidsUDF or SQL…

原文语言：英语

2026-06-22

udf-convert-to-cudf

软件开发工程师

Assists with converting an Apache Spark UDF to a GPU-accelerated RapidsUDF using cuDF Java APIs. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-cudf -> udf-benchmark). Use this skill when you have a CPU UDF with a unit test…

原文语言：英语

2026-06-22

udf-optimize-cudf

软件开发工程师

Iteratively optimizes a cuDF RapidsUDF implementation for GPU performance. Use after testing and benchmarking with udf-benchmark. Runs a loop of profiling, optimizing, testing, and benchmarking until performance converges or the iteration budget is exhausted.

原文语言：英语

2026-06-22

已展示 8 / 8 个已收集 Skill。