NVIDIA/cudf-spark

Letzte erfasste Quellaktivität: 23. Oktober 2026 um 04:56

SkillsMP-Katalog aktualisiert: 29. Oktober 2026 um 18:30

gesammelte Skills: 8

GitHub-Stars: 1.013
GitHub-Forks: 312

Mit einem KI-Assistenten installieren

Diesen Prompt kopieren und in den verwendeten KI-Assistenten einfügen.

Follow https://skillsmp.com/skill-install/prompt.md to install Agent Skills from https://github.com/NVIDIA/cudf-spark.

README des Repositorys

README-Quelldatei: README.md

README-Quelldatei lesen

Vollständige README anzeigen

# NVIDIA cuDF plugin for Apache Spark NOTE: For the latest stable [README.md](https://github.com/NVIDIA/cudf-spark/blob/main/README.md) ensure you are on the main branch. The NVIDIA cuDF plugin for [Apache Spark](https://spark.apache.org) provides a plugin library that leverages GPUs to accelerate processing via the [cuDF](https://github.com/NVIDIA/cudf) (CUDA DataFrame) libraries. Documentation on the current release can be found [here](https://nvidia.github.io/cudf-spark/). To get started and try the plugin out use the [getting started guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/overview.html). [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/NVIDIA/cudf-spark) Note: The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be sunset over time. GitHub links from `spark-rapids` will redirect to `cudf-spark`. Artifact names will remain the same for now. ## Compatibility The SQL plugin tries to produce results that are bit for bit identical with Apache Spark. Operator compatibility is documented [here](./docs/compatibility.md) ## Tuning To get started tuning your job and get the most performance out of it please start with the [tuning guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/tuning-guide.html). ## Configuration The plugin has a set of Spark configs that control its behavior and are documented [here](docs/configs.md). ## Issues & Questions We use github to track bugs, feature requests, and answer questions. File an [issue](https://github.com/NVIDIA/cudf-spark/issues/new/choose) for a bug or feature request. Ask or answer a question on the [discussion board](https://github.com/NVIDIA/cudf-spark/discussions). ## Download The jar files for the most recent release can be retrieved from the [download](https://nvidia.github.io/cudf-spark/docs/download.html) page. ## Building From Source See the [build instructions in the contributing guide](CONTRIBUTING.md#building-from-source). ## Testing Tests are described [here](tests/README.md). ## Integration The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications. It is described [here](https://docs.nvidia.com/spark-rapids/user-guide/latest/additional-functionality/ml-integration.html). Currently, we are working with XGBoost to try to provide this integration out of the box. You may need to disable RMM caching when exporting data to an ML library as that library will likely want to use all of the GPU's memory and if it is not aware of RMM it will not have access to any of the memory that RMM is holding. ## Qualification and Profiling tools The Qualification and Profiling tools are available in the [NVIDIA/cudf-spark-tools](https://github.com/NVIDIA/cudf-spark-tools) repository. Please refer to [Qualification tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/qualification/overview.html) and [Profiling tool documentation](https://docs.nvidia.com/spark-rapids/user-guide/latest/profiling/overview.html) for more details on how to use the tools. ## Dependency for External Projects If you develop functionality on top of the cuDF plugin (we currently limit support to GPU-accelerated UDFs) we recommend you declare the distribution artifact as a `provided` dependency. ```xml <dependency> <groupId>com.nvidia</groupId> <artifactId>rapids-4-spark_2.12</artifactId> <version>26.12.0-SNAPSHOT</version> <scope>provided</scope> </dependency> ```

README des Repositorys wird geladen…

Skills in diesem Repository

Softwareentwickler Softwarequalitätssicherungsanalysten und -tester

2 Berufskategorien · 88% klassifiziert

Skill

Beruf

Beschreibung

Aktualisiert

check-shim-compatibility

nicht klassifiziert

Check cudf-spark shim source selection on a PR combined with its current target branch, provide independent cross-PR advisories, and identify required profile validation. Use for versioned sources, shared shim callers, or Spark/Databricks build profiles.

Quellsprache: Englisch

2026-10-23

udf-convert-to-cuda

Softwareentwickler

Assists with converting a non-aggregating Apache Spark UDF to a native CUDA RapidsUDF using JNI and libcudf. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-cuda -> udf-benchmark). Use this skill when you have a CPU UDF with…

Quellsprache: Englisch

2026-07-30

udf-gen-test

Softwarequalitätssicherungsanalysten und -tester

Assists with generating a unit test for an Apache Spark UDF. This is step 1 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-* -> udf-benchmark). Use this skill when you have a CPU UDF and need to create a unit test for the UDF before…

Quellsprache: Englisch

2026-07-30

udf-judge-conversion

Softwarequalitätssicherungsanalysten und -tester

Reviews generated UDF tests and GPU/SQL implementations for robustness, anti-cheating, and GPU execution integrity. Use when the user requests a judge/review-agent pass, or when manually reviewing a completed conversion.

Quellsprache: Englisch

2026-06-25

udf-convert-to-sql

Softwareentwickler

Assists with converting an Apache Spark UDF to a functionally equivalent Spark SQL expression. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-sql -> udf-benchmark). Use this skill when you have a CPU UDF with a unit test…

Quellsprache: Englisch

2026-06-25

udf-benchmark

Softwarequalitätssicherungsanalysten und -tester

Assists with benchmarking and profiling the performance of an Apache Spark UDF on the GPU. This is step 3 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-* -> udf-benchmark). Use this skill when you have a CPU UDF and a RapidsUDF or SQL…

Quellsprache: Englisch

2026-06-22

udf-convert-to-cudf

Softwareentwickler

Assists with converting an Apache Spark UDF to a GPU-accelerated RapidsUDF using cuDF Java APIs. This is step 2 of 3 in the UDF conversion workflow (udf-gen-test -> udf-convert-to-cudf -> udf-benchmark). Use this skill when you have a CPU UDF with a unit test…

Quellsprache: Englisch

2026-06-22

udf-optimize-cudf

Softwareentwickler

Iteratively optimizes a cuDF RapidsUDF implementation for GPU performance. Use after testing and benchmarking with udf-benchmark. Runs a loop of profiling, optimizing, testing, and benchmarking until performance converges or the iteration budget is exhausted.

Quellsprache: Englisch

2026-06-22

Es werden 8 von 8 gesammelten Skills angezeigt.