jsonlines-web

Web stream based jsonlines decoder/encoder

✅Deno
✅browser
✅Node.js

This library supports JSON in the following formats:

Line-delimited JSON (JSONLinesParseStream)
- NDJSON
- JSON lines
Record separator-delimited JSON (JSONLinesParseStream)
Concatenated JSON (ConcatenatedJSONParseStream)

See wikipedia for the use of each JSON.

install or import

Deno

https://deno.land/x/jsonlines/ https://doc.deno.land/https://deno.land/x/jsonlines/mod.ts

import {
  ConcatenatedJSONParseStream,
  ConcatenatedJSONStringifyStream,
  JSONLinesParseStream,
  JSONLinesStringifyStream,
} from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

browser

import {
  ConcatenatedJSONParseStream,
  ConcatenatedJSONStringifyStream,
  JSONLinesParseStream,
  JSONLinesStringifyStream,
} from "https://deno.land/x/jsonlines@v0.0.8/js/mod.js";

Node.js

https://www.npmjs.com/package/jsonlines-web

npm install jsonlines-web

import {
  ConcatenatedJSONParseStream,
  ConcatenatedJSONStringifyStream,
  JSONLinesParseStream,
  JSONLinesStringifyStream,
} from "jsonlines-web";
// if you need
// import { TextDecoderStream, TextEncoderStream } from "node:stream/web";
// import { fetch } from "undici";

Usage

A working example can be found at ./testdata/test.ts.

How to parse JSON Lines

./json-lines.jsonl

{"some":"thing"}
{"foo":17,"bar":false,"quux":true}
{"may":{"include":"nested","objects":["and","arrays"]}}

import { JSONLinesParseStream } from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

const { body } = await fetch(
  "https://deno.land/x/jsonlines@v0.0.8/testdata/json-lines.jsonl",
);

const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new JSONLinesParseStream());

for await (const data of readable) {
  console.log(data);
}

How to parse json-seq

./json-seq.json-seq

{"some":"thing\n"}
{
  "may": {
    "include": "nested",
    "objects": [
      "and",
      "arrays"
    ]
  }
}

import { JSONLinesParseStream } from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

const { body } = await fetch(
  "https://deno.land/x/jsonlines@v0.0.8/testdata/json-seq.json-seq",
);

const recordSeparator = "\x1E";
const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new JSONLinesParseStream({ separator: recordSeparator }));

for await (const data of readable) {
  console.log(data);
}

How to parse concat-json

./concat-json.concat-json

{"foo":"bar"}{"qux":"corge"}{"baz":{"waldo":"thud"}}

import { ConcatenatedJSONParseStream } from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

const { body } = await fetch(
  "https://deno.land/x/jsonlines@v0.0.8/testdata/concat-json.concat-json",
);

const readable = body!
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new ConcatenatedJSONParseStream());

for await (const data of readable) {
  console.log(data);
}

How to stringify JSON Lines

import { JSONLinesStringifyStream } from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

const target = [
  { foo: "bar" },
  { baz: 100 },
];
const file = await Deno.open(new URL("./tmp.jsonl", import.meta.url), {
  create: true,
  write: true,
});
const readable = new ReadableStream({
  pull(controller) {
    for (const chunk of target) {
      controller.enqueue(chunk);
    }
    controller.close();
  },
});

readable
  .pipeThrough(new JSONLinesStringifyStream())
  .pipeThrough(new TextEncoderStream())
  .pipeTo(file.writable)
  .then(() => console.log("write success"));

How to stringify json-seq

import { JSONLinesStringifyStream } from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

const recordSeparator = "\x1E";
const target = [
  { foo: "bar" },
  { baz: 100 },
];
const file = await Deno.open(new URL("./tmp.jsonl", import.meta.url), {
  create: true,
  write: true,
});
const readable = new ReadableStream({
  pull(controller) {
    for (const chunk of target) {
      controller.enqueue(chunk);
    }
    controller.close();
  },
});

readable
  .pipeThrough(new JSONLinesStringifyStream({ separator: recordSeparator }))
  .pipeThrough(new TextEncoderStream())
  .pipeTo(file.writable)
  .then(() => console.log("write success"));

How to stringify concat-json

import { ConcatenatedJSONStringifyStream } from "https://deno.land/x/jsonlines@v0.0.8/mod.ts";

const target = [
  { foo: "bar" },
  { baz: 100 },
];
const file = await Deno.open(new URL("./tmp.concat-json", import.meta.url), {
  create: true,
  write: true,
});
const readable = new ReadableStream({
  pull(controller) {
    for (const chunk of target) {
      controller.enqueue(chunk);
    }
    controller.close();
  },
});

readable
  .pipeThrough(new ConcatenatedJSONStringifyStream())
  .pipeThrough(new TextEncoderStream())
  .pipeTo(file.writable)
  .then(() => console.log("write success"));

limit

When parsing concat-json, it cannot be parsed unless it is blank immediately after the number, null, true and false.

In other words, this cannot be parsed:

100 200{"foo": "bar"}

But this can be parsed:

100 200 {"foo": "bar"}

note

This library contains ReadableStream.prototype[Symbol.asyncIterator] polyfills. Importing this library will automatically enable ReadableStream.prototype[Symbol.asyncIterator].

develop

need to manually deno task transpile before release.