How frustrating is it if you want to load a JSON document, and you get a vague error like:
SyntaxError: JSON.parse: unexpected character at line 2 column 18 of the JSON data
How can you figure out what is wrong, without spending too much time on it? And, of course, how to solve the problem and fix JSON? If you’re a developer like me, you do not want to spend time manually fixing JSON. It pulls you out of your flow, and costs time and frustration. Is it possible to automatically fix common issues in JSON documents?
This article explains typical issues that you can have with JSON documents, and explains how to solve them: first see if the issue can be fixed automatically, and if not, have the editor clearly point out the issue, so you can fix it manually.
You can use the minimalist editor below to directly fix your broken JSON:
Why is my JSON document not valid?
There are a couple of common real-world cases where a JSON document needs fixing, otherwise the document cannot be used, for example to send to a REST API or to query for analysis. In most cases the document is not really corrupt, but isn’t valid JSON because JSON is very strict. In the following sections we describe the most common cases.
1 Convert from an article or a Word document
When putting a JSON document in an article or word processor like Word or Google Docs, the auto formatting of the word processor can change straight double quotes " into curly quotes “ and ”, which are nice for the eye, but not valid JSON. This can look like:
{ “stringified”: “content” }
This is not valid JSON. These issues with quotes are hard to spot, but when copying the document to use it, it will give parse errors.
2 Convert from objects with a JSON-like structure
In languages like JavaScript and Python, you can natively write JSON-like structures. However, the languages are less strict than JSON. So making plain JavaScript or Python objects valid JSON requires a conversion step:
- enclose all keys with double quotes, and replace single quotes with double quotes. For example in JavaScript the following is valid:
{stringified: 'content'} - replace Python constants None, True, and False with null, true, and false
- Strip trailing commas
- Strip comments like
/* ... */and// ... - Concatenate strings like
"long text" + "more text on next line"
3 Convert from a string containing a JSON document
In all programming languages, you may have a string containing a JSON document in your code. When the string uses double quotes, all double quotes of the JSON document will be escaped, like "{\\"stringified\\": \\"content\\"}". You will need to remove the escape characters to make this valid JSON.
4 Convert from MongoDB output
MongoDB is a powerful, JSON based NoSQL database. Internally, the documents are not stored in plain JSON, but in a BSON (Binary JSON): a JSON variant to store JSON in a compact, highly performant, binary format. The BSON data format comes with additional data types, like Date, Long, and ObjectId.
When querying data from MongoDB, the result may look like:
{
"_id": ObjectId("5effaa5662679b5af2c58129"),
"name": "Joe",
"updated": ISODate("2022-11-15T10:04:44Z")
}
Which is not valid JSON: it has function wrappers around the special data types ObjectId and ISODate.
5 Convert from logs
Logging systems often use a JSON variant named Newline Delimited JSON (or NDJSON). This is simply a list with JSON objects, separated by a newline character. It is meant for an infinite stream of objects rather than a fixed list with a start and end. This can look like (example from Wikipedia):
{"ts":"2020-06-18T10:44:12","started":{"pid":45678}}
{"ts":"2020-06-18T10:44:13","logged_in":{"username":"foo"},"connection":{"addr":"1.2.3.4","port":5678}}
{"ts":"2020-06-18T10:44:15","registered":{"username":"bar","email":"bar@example.com"},"connection":{"addr":"2.3.4.5","port":6789}}
{"ts":"2020-06-18T10:44:16","logged_out":{"username":"foo"},"connection":{"addr":"1.2.3.4","port":5678}}
To change this into regular JSON it needs to be enclosed in square brackets [ and ], and the rows need to be separated by a comma.
6 Convert from JSON dialects
There are popular JSON variants like JSON5 or YAML which are less strict and more suitable for human-readable configuration files than JSON itself. These standards for example support comments, and make the double quotes around keys and values optional.
7 Truncated JSON and wrapped lines
When transferring data or storing data, it could be that a connection is reset or the storage has a limitation. In those cases it can be that a JSON document gets truncated halfway, resulting in a corrupt JSON file. This can look like:
{"id":2,"timestamp":"2020-06-18T10:44:12","message":"The imp
This is really a case of corrupted data. The missing data cannot be restored, but in most cases it is possible to repair the document to make it at least usable again.
A case close to this is JSON output in a terminal: the output is typically wrapped at the end of the line, like every 80th character. It is possible to detect this pattern and glue the data together again in a single, valid document.
How do I fix JSON errors?
Manually repairing a JSON file can cost quite some time (and annoyance). Based on 20+ years of experience as a full-stack developer, Jos de Jong is developing a tool for this: JSON Editor Online. Amongst others, this tool comes with a feature to validate your JSON and automatically repair broken JSON. The auto repair functionality is developed as a standalone, open source library named jsonrepair. This library aims at automatically solving cases like discussed above for you.
So, to fix your JSON errors, head over to https://jsoneditoronline.org/, and paste your JSON in one of the two panels. The editor has two modes: tree and text mode. In tree mode, the pasted JSON will be automatically repaired when loading it, and you will see a confirmation message at the bottom of the editor. In case of text mode, the editor is more conservative and will show you the current error message with a button “Auto repair” to automatically repair JSON, if possible. The following screenshot shows a truncated list with numbers, which can be fixed automatically:
Not all issues can be repaired automatically. The issues that can’t be solved are pointed out with a clear error message and the location of the issue is highlighted.
When the editor does not report a parse error, it means that your document contains valid JSON.
How does jsonrepair work?
The jsonrepair library is basically an extended JSON parser. It parses the provided JSON document character by character, reading it into a data structure. When it encounters a parse error, instead of throwing an error, it looks whether this error has a cause that is solvable. For example, when encountering two unexpected characters //, it knows that this is a comment, and it will skip the comment and move on. And after encountering an opening bracket {, it expects a key wrapped in double quotes. When it encounters a key without quotes, or wrapped in single quote characters, it will add double quotes instead.
Handling ambiguous cases
Some cases like missing quotes around a key are clear for everyone how to solve. However, other cases are ambiguous and can be solved in different ways. For example:
[{"a":1,{"b":2}]
Was this meant as an array with two items, where the first item is missing a closing bracket? Or was this intended as an array with one item, and is the opening bracket of the second item redundant? Or take the following example:
[
Joe
Jones
]
Is this meant as an array with two items, or do the two strings belong together in a single string with a return character?
Future plans
The jsonrepair library so far is quite conservative. In case of doubt, it will not make changes. One of the future plans is to make it customizable so you can select the cases that you want to have fixed. That would allow to select only safe, conservative rules, or also apply heuristics to ambiguous cases. It would also allow you to turn off a rule like removing comments.
In general, the jsonrepair library can be improved further by supporting more of the syntax of for example JSON variants like the mentioned JSON5 and YAML. Not all the cases described above can be handled by jsonrepair at the moment of writing. If you come across missing cases, please report them in the issues section of the project. Contributions to the project are also very welcome!
Conclusion
Stop wasting time and start to fix JSON with ease. To see what is wrong with your JSON document, open it in JSON Editor Online, and use the auto repair function to repair JSON online and solve many common issues with a single click.