DEV Community 👩‍💻👨‍💻

Vee Satayamas
Vee Satayamas

Posted on

Reading lines from a Zstd archive in Common Lisp

Reading lines from a Zstd archive in Common Lisp

I have a big line-separated JSON file, compressed in Zstd format. I will call this file data.ndjson.zst.

I want my program to read the file line by line. I can use to make a binary stream from the file. Still, I can run the function read-line of a binary stream. So I need to wrap the binary stream with Flexi-stream.

(ql:quickload 'flexi-streams)
(ql:quickload 'zstd)

(defpackage #:ex1
  (:use #:cl #:flexi-streams :zstd))
(in-package :ex1)

(with-open-file (f #P"data.ndjson.zst"
           :element-type '(unsigned-byte 8)
           :direction :input)
  (with-decompressing-stream (zstd-stream f)
    (let ((s (make-flexi-stream zstd-stream
                :external-format (make-external-format :utf-8))))
      (loop for line = (read-line s nil nil)
        until (null line)
        do (print line)))))
Enter fullscreen mode Exit fullscreen mode

Printing those lines is a realistic example. So you can replace (print line) with your practical applications, for example, parse a JSON line and extract a book title.

Latest comments (0)


Thank you.

Thanks for visiting DEV, we’ve worked really hard to cultivate this great community and would love to have you join us. If you’d like to create an account, you can sign up here.