Reading a GZipped File from S3 to Memory in NodeJS

Posted on by Christopher Davis

Specifically, I had firehose landing some small-ish files in S3 that were gzipped. Each file has a JSON object per row and I need to get the contents of that file into a lambda for processing.

SO needed to use the AWS SDK to call S3, gunzip the contents, and then work with them. The S3 body attributes in the v3 AWS SDK are now readable streams and combined with some of the new-ish node utility consumers, the result:

import { createGunzip } from 'node:zlib';
import { text } from 'node:stream/consumers';
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';

async function getObject(s3Client: S3Client, bucket: string, objectKey: string): Promise<string> {
    const command = new GetObjectCommand({
        Bucket: bucket,
        Key: objectKey,
    });
    const response = await s3Client.send(command);

    const gunzip = createGunzip();
    const body = response.Body.pipe(gunzip);

    return text(body);
}