Strategies reading large data from Microsoft CSV and writing to Microsoft Excel in AWS

Question

Kindly assist in deciding the strategy to do Extraction, Transformation and Loading (ETL) work flow with Amazon AWS offerings. I am a newbie in Amazon Cloud. My use case is to read thousands of record rows from Microsoft CSV file. My intention is to write out this file into Microsoft Excel document. I want to store this excel file in single piece within S3 bucket as an object. Currently, I am doing this proof of concept with AWS Lambda. My issue is /tmp size is exceeding beyond 512 MB even if I select 6 GB RAM size in lambda specification. I do not do any disk operation. I read all of S3 csv file content once, in RAM and write out to excel, in RAM.

Can you please check container lambdas. They have 10GB of space. — Marcin
– Marcin, Commented Mar 15, 2021 at 8:29
I mean 10GB of disk space. Existing lambda offer only 10 GB of ram and 512 MB disk space. — Marcin
– Marcin, Commented Mar 16, 2021 at 23:13

Emerson · Accepted Answer · 2021-03-15 09:59:46Z

0

You should consider using aws glue for large volume data transformations. Lambda may be Ill suited to the task

answered Mar 15, 2021 at 9:59

Emerson

1,1761 gold badge8 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Prakashsinha Bayas Over a year ago

We are doing aws glue work flow. I cannot export to Microsoft excel from both AWS Glue and AWS Athena

Emerson Over a year ago

You can do it in the python environment of glue. Do not use pyspark. U will need to do the transformation using pandas.

Collectives™ on Stack Overflow

Strategies reading large data from Microsoft CSV and writing to Microsoft Excel in AWS

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related