DEV Community 👩‍💻👨‍💻

Vee Satayamas
Vee Satayamas

Posted on

Reading lines from a Zstd archive in Common Lisp

Reading lines from a Zstd archive in Common Lisp

I have a big line-separated JSON file, compressed in Zstd format. I will call this file data.ndjson.zst.

I want my program to read the file line by line. I can use https://github.com/glv2/cl-zstd to make a binary stream from the file. Still, I can run the function read-line of a binary stream. So I need to wrap the binary stream with Flexi-stream.

(ql:quickload 'flexi-streams)
(ql:quickload 'zstd)

(defpackage #:ex1
  (:use #:cl #:flexi-streams :zstd))
(in-package :ex1)

(with-open-file (f #P"data.ndjson.zst"
           :element-type '(unsigned-byte 8)
           :direction :input)
  (with-decompressing-stream (zstd-stream f)
    (let ((s (make-flexi-stream zstd-stream
                :external-format (make-external-format :utf-8))))
      (loop for line = (read-line s nil nil)
        until (null line)
        do (print line)))))
Enter fullscreen mode Exit fullscreen mode

Printing those lines is a realistic example. So you can replace (print line) with your practical applications, for example, parse a JSON line and extract a book title.

Oldest comments (0)

Create an Account! The only reason people scroll to the bottom...  
is because they want to read more.

Create an account to bookmark, comment, and react to articles that interest you.