DEV Community

Vee Satayamas
Vee Satayamas

Posted on

Reading lines from a Zstd archive in Common Lisp

Reading lines from a Zstd archive in Common Lisp

I have a big line-separated JSON file, compressed in Zstd format. I will call this file data.ndjson.zst.

I want my program to read the file line by line. I can use https://github.com/glv2/cl-zstd to make a binary stream from the file. Still, I can run the function read-line of a binary stream. So I need to wrap the binary stream with Flexi-stream.

(ql:quickload 'flexi-streams)
(ql:quickload 'zstd)

(defpackage #:ex1
  (:use #:cl #:flexi-streams :zstd))
(in-package :ex1)

(with-open-file (f #P"data.ndjson.zst"
           :element-type '(unsigned-byte 8)
           :direction :input)
  (with-decompressing-stream (zstd-stream f)
    (let ((s (make-flexi-stream zstd-stream
                :external-format (make-external-format :utf-8))))
      (loop for line = (read-line s nil nil)
        until (null line)
        do (print line)))))
Enter fullscreen mode Exit fullscreen mode

Printing those lines is a realistic example. So you can replace (print line) with your practical applications, for example, parse a JSON line and extract a book title.

Oldest comments (0)