Data cleansing and preprocessing with cuDF and pandas: Quick Reference — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)

Data Cleansing and Preprocessing with cuDF and pandas — Quick Reference

This quick reference covers essential facts and best practices for data cleansing and preprocessing using cuDF and pandas, key libraries in the NVIDIA Accelerated Data Science ecosystem.

Key Concepts

• Data Cleansing: Identifying and correcting errors or inconsistencies in data to improve quality.
• Preprocessing: Transforming raw data into a suitable format for analysis or modeling.
• cuDF: GPU-accelerated DataFrame library similar to pandas, optimized for large-scale data on NVIDIA GPUs.
• pandas: Widely-used CPU-based Python library for data manipulation and analysis.

Common Data Cleansing Tasks

• Handling Missing Values: Use dropna() to remove or fillna() to impute missing data.
• Removing Duplicates: Apply drop_duplicates() to eliminate repeated rows.
• Data Type Conversion: Use astype() to convert columns to appropriate types (e.g., numeric, categorical).
• String Cleaning: Strip whitespace, convert case, or replace substrings with str.strip(), str.lower(), str.replace().

Preprocessing Techniques

• Feature Scaling: Normalize or standardize numerical features using custom functions or libraries like scikit-learn (CPU) or RAPIDS cuML (GPU).
• Encoding Categorical Variables: Convert categories to numeric codes with astype('category').cat.codes or one-hot encoding.
• Filtering and Subsetting: Use boolean indexing to select relevant data rows.
• Data Type Optimization: Downcast numeric types to reduce memory footprint.

cuDF vs pandas: Key Differences

• Execution: cuDF operations run on GPU, enabling faster processing on large datasets.
• API Compatibility: cuDF mimics pandas API but some pandas functions may be unsupported or behave differently.
• Data Transfer: Minimize host-device data transfer for performance; keep data on GPU when possible.

Common cuDF Functions for Cleansing and Preprocessing

• cudf.DataFrame.dropna() – Remove missing values.
• cudf.DataFrame.fillna(value) – Impute missing values.
• cudf.DataFrame.drop_duplicates() – Remove duplicate rows.
• cudf.Series.astype() – Change data types.
• cudf.Series.str.lower(), str.strip(), str.replace() – String manipulations.
• cudf.Series.cat.codes – Encode categorical variables.

Best Practices

• Validate Data Types Early: Ensure columns have correct types before transformations.
• Leverage GPU Acceleration: Use cuDF for large datasets to reduce preprocessing time.
• Minimize Data Movement: Keep data on GPU throughout pipeline to avoid costly transfers.
• Test on Sample Data: Validate preprocessing steps on small samples before scaling.

Worked Example: Handling Missing Values with cuDF

Problem: A cuDF DataFrame has missing values in the age column. Replace missing values with the column mean.

Solution:

• Calculate mean: mean_age = df['age'].mean()
• Fill missing: df['age'] = df['age'].fillna(mean_age)

This preserves GPU acceleration and cleanses data efficiently.

More in this topic

Data cleansing and preprocessing with cuDF and pandas: Practice Questions — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)Data cleansing and preprocessing with cuDF and pandas: Common Mistakes — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)Monitoring pipeline bottlenecks — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)Data Preparation — NVIDIA-Certified Professional: Accelerated Data ScienceData cleansing and preprocessing with cuDF and pandas — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)Generating synthetic data with cuDF and RAPIDS — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)Data cleansing and preprocessing with cuDF and pandas: Worked Example — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)Transforming and standardizing features — Data Preparation (NVIDIA-Certified Professional: Accelerated Data Science)