df = pd.DataFrame('x': np.random.rand(500000))
pip install pandarallel[full] import pandas as pd from pandarallel import pandarallel Initialize (do this once before using parallel functions) pandarallel.initialize() Optional: with progress bar and custom settings pandarallel.initialize( progress_bar=True, nb_workers=4, # number of workers (default: all CPUs) verbose=1 ) Key Parallel Functions | Pandas Function | Pandarallel Equivalent | |----------------|------------------------| | df.apply() | df.parallel_apply() | | df.applymap() | df.parallel_applymap() | | series.apply() | series.parallel_apply() | | series.map() | series.parallel_map() | | groupby.apply() | groupby.parallel_apply() | Examples 1. Basic parallel_apply on DataFrame import pandas as pd from pandarallel import pandarallel pandarallel.initialize(progress_bar=True) pandarallel
def heavy_func(x): return sum(np.sin(x) * np.cos(x) for _ in range(100)) start = time.time() result_pd = df['x'].apply(heavy_func) print(f"Pandas: time.time() - start:.2fs") Pandarallel start = time.time() result_pll = df['x'].parallel_apply(heavy_func) print(f"Pandarallel: time.time() - start:.2fs") Common Issues & Solutions 1. PicklingError (lambdas with closures) # This will fail df.parallel_apply(lambda row: row['a'] + external_var) Solution: Define a regular function def add_external(row): return row['a'] + external_var df = pd
What is Pandarallel? Pandarallel is a Python library that provides easy parallel computing for pandas operations. It allows you to replace standard pandas apply , map , and other functions with parallelized versions, leveraging all CPU cores of your machine. Installation pip install pandarallel For full features (progress bars, etc.): Pandarallel is a Python library that provides easy