CSV Data Pipeline

This project demonstrates a simple data pipeline that reads data from a CSV file, processes it, and loads it into a new, cleaned csv file. The pipeline is built using Python and utilizes libraries such as Pandas for data manipulation.

Technologies Used

PythonPandasFile I/OBasic ETL

Features

  • • Reads data from a CSV file
  • • Processes and cleans the data
  • • Loads the processed data into a CSV file

Code Snippet

import pandas as pd
import numpy as np

def clean_csv_data(input_file, output_file):
    """
    Read CSV file, clean the data, and save to new CSV
    """
    # Read the CSV file
    df = pd.read_csv(input_file)
    
    # Data cleaning operations
    df_cleaned = df.dropna()  # Remove null values
    df_cleaned = df_cleaned.drop_duplicates()  # Remove duplicates
    
    # Save cleaned data
    df_cleaned.to_csv(output_file, index=False)
    
    print(f"Data pipeline completed!")
    print(f"Original rows: {len(df)}")
    print(f"Cleaned rows: {len(df_cleaned)}")

# Usage
clean_csv_data('raw_data.csv', 'cleaned_data.csv')
← Back to Projects