Data integration and manipulation with cuDF and pandas: Quick Reference — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)

Data Integration and Manipulation with cuDF and pandas — Quick Reference

This quick reference provides essential facts and guidelines for using cuDF and pandas in GPU-accelerated data science workflows, a key skill area for the NVIDIA-Certified Associate: Accelerated Data Science exam.

Key Libraries

• pandas: Python library for data manipulation and analysis on CPU.
• cuDF: GPU-accelerated DataFrame library with pandas-like API, part of RAPIDS suite.

DataFrame Creation and Conversion

• pandas DataFrame: Created from dictionaries, CSV, Excel, SQL, etc.
• cuDF DataFrame: Created similarly; supports reading CSV, Parquet, ORC with GPU acceleration.
• Conversion: cudf.from_pandas(df) converts pandas to cuDF.df.to_pandas() converts cuDF back to pandas.

Common Data Manipulation Operations

• Selection: Use df[column] or df.loc[row_indexer, column_indexer] in both libraries.
• Filtering: Boolean indexing supported similarly, e.g., df[df['col'] > 5].
• Sorting: df.sort_values(by='col') works in both, with GPU acceleration in cuDF.
• Grouping: df.groupby('col').agg() supported with similar syntax.
• Joining: df.merge() supports inner, outer, left, right joins.

Handling Missing Data

• Detection: df.isnull() or df.isna() identify missing values.
• Removal: df.dropna() drops rows/columns with missing data.
• Imputation: df.fillna(value) fills missing values.

Data Types and Conversion

• cuDF supports common numeric, categorical, datetime types similar to pandas.
• Use df.astype() to convert data types.
• cuDF handles categorical data efficiently with GPU acceleration.

Performance Considerations

• cuDF: Best for large datasets that fit GPU memory; offers significant speedups over pandas.
• pandas: Preferred for smaller datasets or when GPU resources are unavailable.
• Minimize data transfer between CPU and GPU to reduce overhead.

File I/O Support

• cuDF: Accelerated reading/writing of CSV, Parquet, ORC files.
• pandas: Supports many formats but CPU-bound.

Example: Converting and Filtering DataFrames

Worked Example

Task: Convert a pandas DataFrame to cuDF, filter rows where column 'age' > 30, then convert back to pandas.

Solution:

• Import libraries: import pandas as pd, import cudf
• Create pandas DataFrame: pdf = pd.DataFrame({'age': [25, 35, 45], 'name': ['A', 'B', 'C']})
• Convert to cuDF: gdf = cudf.from_pandas(pdf)
• Filter: gdf_filtered = gdf[gdf['age'] > 30]
• Convert back to pandas: pdf_filtered = gdf_filtered.to_pandas()

Summary

• cuDF provides a GPU-accelerated, pandas-like API for efficient data integration and manipulation.
• Conversion between pandas and cuDF is straightforward, enabling hybrid CPU-GPU workflows.
• Core operations such as selection, filtering, grouping, and joining have similar syntax in both libraries.
• Effective use of cuDF requires attention to data transfer overhead and GPU memory constraints.

For more detailed documentation and examples, visit the official RAPIDS cuDF site: https://rapids.ai/start.html

More in this topic

Feature engineering for numerical and categorical variables — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Dimensionality reduction and data sampling: Worked Example — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data integration and manipulation with cuDF and pandas: Worked Example — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Efficient processing and storage with Parquet — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data integration and manipulation with cuDF and pandas: Practice Questions — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data cleaning, quality handling, and governance: Worked Example — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data cleaning, quality handling, and governance: Common Mistakes — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data cleaning, quality handling, and governance: Practice Questions — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data integration and manipulation with cuDF and pandas — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data integration and manipulation with cuDF and pandas: Common Mistakes — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Dimensionality reduction and data sampling: Practice Questions — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data cleaning, quality handling, and governance — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data cleaning, quality handling, and governance: Quick Reference — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)GPU-accelerated ETL with RAPIDS, Dask, or Spark — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Dimensionality reduction and data sampling — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Dimensionality reduction and data sampling: Quick Reference — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Dimensionality reduction and data sampling: Common Mistakes — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Handling class imbalance and generating synthetic data — Data Manipulation and Preparation (NVIDIA-Certified Associate: Accelerated Data Science)Data Manipulation and Preparation — NVIDIA-Certified Associate: Accelerated Data Science