Merge CSV files in different folders using Python

Question

I have about 5600 directories structured as follows:

I need to merge all A files into one file, all B files into another file, and so on.

How can I do this?

So you need all "root directory/dir1" csv files into a single df? Also do they have the exact same structure? — ArchAngelPwn
– ArchAngelPwn, Commented Jul 27, 2022 at 15:24
I need to merge all the A files into one file, all the B files into one file, etc.... Basically, I need to output 7 files: A, B, C, D, E, F, G, and each X file contains within it the union of the X files contained in the 5600 dirs. All the files have the same header — Giammaria Giordano
– Giammaria Giordano, Commented Jul 27, 2022 at 15:30

René · Accepted Answer · 2022-07-27 16:26:26Z

1

IIUC, this should work for your case (I used a RootDir with 2 subdirectories Dir1 and Dir2 with in each 2 files A.csv and B.csv). You can change the value of rootdir to match your usecase:

import os
import pandas as pd
rootdir = 'RootDir' # Change when needed to your root directory
files = [os.path.join(dp, f) for dp, dn, filenames in os.walk(rootdir) for f in filenames if os.path.splitext(f)[1] == '.csv']
names = set([x.rstrip('.csv').split('/')[-1] for x in files])
df_dict = {key: pd.DataFrame() for key in names}
for file in files:
    key = file.rstrip('.csv').split('/')[-1]
    df = pd.read_csv(file)
    df_dict[key] = pd.concat([df_dict[key], df])

Output is a dictionary of dataframes df_dict with A and B as keys.

Use df_dict['A'] to access DataFrame A and so on...

edited Jul 27, 2022 at 16:26

answered Jul 27, 2022 at 16:20

René

4,9195 gold badges29 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Merge CSV files in different folders using Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related