Reading lines from a Zstd archive in Common Lisp
I have a big line-separated JSON file, compressed in Zstd format. I will call this file data.ndjson.zst.
I want my program to read the file line by line. I can use https://github.com/glv2/cl-zstd to make a binary stream from the file. Still, I can run the function read-line of a binary stream. So I need to wrap the binary stream with Flexi-stream.
(ql:quickload 'flexi-streams)
(ql:quickload 'zstd)
(defpackage #:ex1
(:use #:cl #:flexi-streams :zstd))
(in-package :ex1)
(with-open-file (f #P"data.ndjson.zst"
:element-type '(unsigned-byte 8)
:direction :input)
(with-decompressing-stream (zstd-stream f)
(let ((s (make-flexi-stream zstd-stream
:external-format (make-external-format :utf-8))))
(loop for line = (read-line s nil nil)
until (null line)
do (print line)))))
Printing those lines is a realistic example. So you can replace (print line) with your practical applications, for example, parse a JSON line and extract a book title.
Top comments (0)