Newline Delimited json Format
NDJSON is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It’s a great format for log files. It’s also a flexible format for passing messages between cooperating processes.
ndjson Newline Delimited JSON
This page describes the ndjson format, also called Newline delimited JSON. NDJSON is a convenient format for storing or streaming structured data that may be processed one record at a time. It works well with unix-style text processing tools and shell pipelines. It’s a great format for log files. It’s also a flexible format for passing messages between cooperating processes.
1. Line Separator is ‘\n’
This means ‘\r\n’ is also supported because trailing white space is ignored when parsing JSON values.
2. Each Line is a Valid JSON Value
The most common values will be objects or arrays, but any JSON value is permitted. See json.org for more information about JSON values.
Newline Delimited JSON (ndjson) Format
MIME type: application/x-ndjson
ID: ndjson
DataWeave represents the Newline Delimited JSON format (ndjson) as an array of objects. Each line of the ndjson format is mapped to one object in the array.
The following parser strategies are supported by the ndjson reader:
- In-memory
- Streaming
For details, see DataWeave Readers.
Examples
The following examples show uses of the ndjson format.
Example
This example shows how DataWeave represents a simple ndjson input.
Input
{“name”: “Leandro”,”lastName”: “Shokida”}
{“name”: “Mariano”,”lastName”: “De Achaval”}
Source
The DataWeave script transforms the ndjson input to the DataWeave (dw) format and MIME type.
%dw 2.0
output application/dw
—
payload
Output
The DataWeave (dw) format outputs the ndjson input into an array of comma-separated objects.
[
{“name”: “Leandro”,”lastName”: “Shokida”},
{“name”: “Mariano”,”lastName”: “De Achaval”}
]
Example
This example shows that the ndjson reader ignores all lines of ndjson data that are invalid if skipInvalid=true.
Input
The input to the DataWeave source includes valid and invalid lines of ndjson data. Assume that the input is from a file myInput.ndjson.
myInput.ndjson:
{“name”: “Christian”
{“name”: “Mariano”}
{“name”: “Tomo”
{“name”: “Shoki”}
Source
The DataWeave script inputs the contents of the input file myInput.ndjson, applies the skipInvalid=true reader property, and transforms the input to the JSON format and MIME type.
%dw 2.0
var myInput = readUrl(‘classpath://myInput.ndjson’, ‘application/x-ndjson, {skipInvalid=true})
output application/json
—
myInput
Output
The JSON output is an array of the valid objects from the ndjson input.
[
{
“name”: “Mariano”
},
{
“name”: “Shoki”
}
]
Configuration Properties
DataWeave supports the following configuration properties for this format.
Reader Properties
This format accepts properties that provide instructions for reading input data.
| Parameter | Type | Default | Description |
| ignoreEmptyLine | Boolean | true | Ignores empty lines.Valid values are true or false. |
| skipInvalid | Boolean | false | Skips invalid records and ignores values that are not valid in this format.Valid values are true or false. |
Writer Properties
This format accepts properties that provide instructions for writing output data.
| Parameter | Type | Default | Description |
| bufferSize | Number | 8192 | Size of the buffer writer. |
| deferred | Boolean | false | Generates the output as a data stream when set to true, and defers the script’s execution until consumed.Valid values are true or false. |
| encoding | String | ‘UTF-8’ | The character set to use for the output, such as UTF-8. |
| skipNullOn | String | null | Skips null values. By default, DataWeave does not skip.arrays Ignore and omit null values inside arrays from the JSON output, for example, with output application/x-ndjson skipNullOn=”arrays”. objects Ignore key-value pairs that have null as the value, for example, with output application/x-ndjson skipNullOn=”objects”. everywhere Apply skipNullOn to arrays and objects, for example, output application/x-ndjson skipNullOn=”everywhere”. Valid values are arrays or objects or everywhere. |
| writeAttributes | Boolean | false | Converts attributes of a key into child key-value pairs of that key. The attribute key name starts with @.Valid values are true or false. |
Supported MIME Types
This format supports the following MIME types.
| MIME Type |
| application/x-ndjson |
| application/x-ldjson |
Follow Me
If you like my post please follow me to read my latest post on programming and technology.
Leave a Comment