NVIDIA/cudf-spark

NVIDIA 프로필

최근 기록된 소스 활동: 2026년 7월 30일 16:50

SkillsMP 카탈로그 업데이트: 2026년 10월 17일 00:09

수집된 skills: 7

GitHub 스타: 1,010
GitHub 포크: 311

저장소 README

README 원본 파일: README.md

README 원본 보기

README 전체 펼치기

# NVIDIA cuDF plugin for Apache Spark NOTE: For the latest stable [README.md](https://github.com/NVIDIA/cudf-spark/blob/main/README.md) ensure you are on the main branch. The NVIDIA cuDF plugin for [Apache Spark](https://spark.apache.org) provides a plugin library that leverages GPUs to accelerate processing via the [cuDF](https://github.com/NVIDIA/cudf) (CUDA DataFrame) libraries. Documentation on the current release can be found [here](https://nvidia.github.io/cudf-spark/). To get started and try the plugin out use the [getting started guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/overview.html). [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/cudf-spark) Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be sunset over time. GitHub links from `spark-rapids` will redirect to `cudf-spark`. Artifact names will remain the same for now. ## Compatibility The SQL plugin tries to produce results that are bit for bit identical with Apache Spark. Operator compatibility is documented [here](./docs/compatibility.md) ## Tuning To get started tuning your job and get the most performance out of it please start with the [tuning guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/tuning-guide.html). ## Configuration The plugin has a set of Spark configs that control its behavior and are documented [here](docs/configs.md). ## Issues & Questions We use github to track bugs, feature requests, and answer questions. File an [issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose) for a bug or feature request. Ask or answer a question on the [discussion board](https://github.com/NVIDIA/cudf-spark/discussions). ## Download The jar files for the most recent release can be retrieved from the [download](https://nvidia.github.io/cudf-spark/docs/download.html) page. ## Building From Source See the [build instructions in the contributing guide](CONTRIBUTING.md#building-from-source). ## Testing Tests are described [here](tests/README.md). ## Integration The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications. It is described [here](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/ml-integration.html). Currently, we are working with XGBoost to try to provide this integration out of the box. You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding. ## Qualification and Profiling tools The Qualification and Profiling tools are available in the [NVIDIA/cudf-spark-tools](https://github.com/NVIDIA/cudf-spark-tools) repository. Please refer to [Qualification tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/qualification/overview.html) and [Profiling tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/profiling/overview.html) for more details on how to use the tools. ## Dependency for External Projects If you develop functionality on top of the cuDF plugin (we currently limit support to GPU-accelerated UDFs) we recommend you declare the distribution artifact as a `provided` dependency. ```xml <dependency> <groupId>com.nvidia</groupId> <artifactId>rapids-4-spark_2.12</artifactId> <version>26.12.0-SNAPSHOT</version> <scope>provided</scope> </dependency> ```

저장소 README를 불러오는 중…

이 저장소의 skills

소프트웨어 개발자 소프트웨어 품질 보증 분석가·테스터

직업 카테고리 2개 · 100% 분류됨

skill

직업 분류

설명

업데이트

udf-convert-to-cuda

소프트웨어 개발자

Assists with converting a non-aggregating Apache Spark UDF to a native CUDA RapidsUDF using JNI and libcudf. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-cuda -> udf-benchmark). Use this skill when you have a CPU UDF with…

원문 언어: 영어

2026-07-30

udf-gen-test

소프트웨어 품질 보증 분석가·테스터

Assists with generating a unit test for an Apache Spark UDF. This is step 1 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-* -> udf-benchmark). Use this skill when you have a CPU UDF and need to create a unit test for the UDF before…

원문 언어: 영어

2026-07-30

udf-judge-conversion

소프트웨어 품질 보증 분석가·테스터

Reviews generated UDF tests and GPU/SQL implementations for robustness, anti-cheating, and GPU execution integrity. Use when the user requests a judge/review-agent pass, or when manually reviewing a completed conversion.

원문 언어: 영어

2026-06-25

udf-convert-to-sql

소프트웨어 개발자

Assists with converting an Apache Spark UDF to a functionally equivalent Spark SQL expression. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-sql -> udf-benchmark). Use this skill when you have a CPU UDF with a unit test…

원문 언어: 영어

2026-06-25

udf-benchmark

소프트웨어 품질 보증 분석가·테스터

Assists with benchmarking and profiling the performance of an Apache Spark UDF on the GPU. This is step 3 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-* -> udf-benchmark). Use this skill when you have a CPU UDF and a RapidsUDF or SQL…

원문 언어: 영어

2026-06-22

udf-convert-to-cudf

소프트웨어 개발자

Assists with converting an Apache Spark UDF to a GPU-accelerated RapidsUDF using cuDF Java APIs. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-cudf -> udf-benchmark). Use this skill when you have a CPU UDF with a unit test…

원문 언어: 영어

2026-06-22

udf-optimize-cudf

소프트웨어 개발자

Iteratively optimizes a cuDF RapidsUDF implementation for GPU performance. Use after testing and benchmarking with udf-benchmark. Runs a loop of profiling, optimizing, testing, and benchmarking until performance converges or the iteration budget is exhausted.

원문 언어: 영어

2026-06-22

수집된 skill 7개 중 7개를 표시합니다.

AI 어시스턴트로 설치

이 프롬프트를 복사해 사용 중인 AI 어시스턴트에 입력하세요.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/NVIDIA/cudf-spark.