NOTE: This article presents 3 scripts. Scroll to the section “Full Path Json with Python (Better – version 2)” at the bottom for the best one.
Sometimes JSON files are nested so much that it is hard to look for output. Of course, one could use JQ but to be fast with JQ you first need to know the structure of the JSON file. Here is a proposed method to look thru JSON files by using full path json. Which is just a flat 1 layer deep representation of the json object where it looks like this { full-key-path : value output }. It makes it very easy to grep thru and also understand the full json object.
Note the input requirements are that the json files must be in string format saved to a file. Also they must be valid formats that json would understand. Example: strings must be surrounded by quotes.
So this would fail to be processed:
{
one: 1,
two: {
three: 3
},
four: {
five: 5,
six: {
seven: 7
},
eight: 8
},
nine: 9
}But this would process:
{
"one": 1,
"two": {
"three": 3
},
"four": {
"five": 5,
"six": {
"seven": 7
},
"eight": 8
},
"nine": 9
}The output should just be something that is easy to grep for words to get what you are looking for.
Example:
{
one: 1,
'two.three': 3,
'four.five': 5,
'four.six.seven': 7,
'four.eight': 8,
nine: 9
}
# Or
{'four.eight': 8,
'four.five': 5,
'four.six.seven': 7,
'nine': 9,
'one': 1,
'two.three': 3}
# Or
{'four/eight': 8,
'four/five': 5,
'four/six/seven': 7,
'nine': 9,
'one': 1,
'two/three': 3}As you can see you can easily grep for the word “six” in those outputs to get everything under the key value of “six”. However, in the original json string grepping for “six” would not return anything meaningfull. You would need to use JQ. For JQ you need to be familiar with the layout of the file, however with this full path json you see the layout at a glance.
Note: Below are presented 3 programs: 1 node js program and 2 python program. The worst one is the nodejs version as sometimes it returns objects or errors when the others return the output. The best one is the last python one (fullpathjson2.py). Feel free to use whichever you want though. Scroll to the section “Full Path Json with Python (Better – version 2)” at the bottom for the best one.
Full Path Json With Node.JS
Here is an attempt via node.js
Step 1. Install node.js
Examples:
For Ubuntu / Dabian Linux:
apt install nodejs;
For MAC:
brew install node;
For Centos Linux:
yum install nodejs;
Step 2. Create the following node.js program. Just put this content into a file called fullpathjson.js.
cat fullpathjson.js
/* ================================================================================
function: read in json file from first arg and print out full path json format.
consider this input file:
$ cat test.json
{
"one": 1,
"two": {
"three": 3
},
"four": {
"five": 5,
"six": {
"seven": 7
},
"eight": 8
},
"nine": 9
}
example:
$ node fullpathjson.js test.json
output:
{
one: 1,
'two.three': 3,
'four.five': 5,
'four.six.seven': 7,
'four.eight': 8,
nine: 9
}
================================================================================ */
const filename = process.argv[2];
const fs = require("fs");
fs.readFile(filename, "utf8", (err, jsonString) => {
if (err) {
console.log("Error reading file from disk:", err);
return;
}
try {
const obj = JSON.parse(jsonString);
// console.log(obj); // DEBUG: shows given json object
const flatObject = (obj, keyPrefix = null) =>
Object.entries(obj).reduce((acc, [key, val]) => {
const nextKey = keyPrefix ? `${keyPrefix}.${key}` : key
if (typeof val !== "object") {
return {
...acc,
[nextKey]: val
};
} else {
return {
...acc,
...flatObject(val, nextKey)
};
}
}, {});
console.log(flatObject(obj))
} catch (err) {
console.log("Error parsing JSON string:", err);
}
});Step 3. Run the program against some json output like so
node fullpathjson.js input.json
The output should be easier to interpret and is very easily greppable.
Here is example output from an fio job which saved its results as json. Then we ran the resulting json file agsint fullpathjson.js.
...skip...
'fio version': 'fio-3.7',
timestamp: 1668015571,
time: 'Wed Nov 9 09:39:31 2022',
'global options.iodepth': '1',
'global options.bs': '32k,100k',
'global options.direct': '1',
'global options.runtime': '180',
'global options.rwmixread': '70',
'global options.rw': 'randrw',
'global options.numjobs': '40',
'client_stats.0.jobname': 'job1',
'client_stats.0.groupid': 0,
'client_stats.0.error': 117,
'client_stats.0.job options.filename': '/mnt/nvme1/testfile.bin:/mnt/nvme2/testfile.bin:/mnt/nvme3/testfile.bin:/mnt/nvme4/testfile.bin:/mnt/nvme5/testfile.bin:/mnt/nv',
'client_stats.0.read.io_bytes': 4423680,
'client_stats.0.read.io_kbytes': 4320,
'client_stats.0.read.bw_bytes': 552960000,
'client_stats.0.read.bw': 540000,
'client_stats.0.read.iops': 20250,
'client_stats.0.read.runtime': 8,
'client_stats.0.read.total_ios': 162,
'client_stats.0.read.short_ios': 0,
'client_stats.0.read.drop_ios': 0,
'client_stats.0.read.slat_ns.min': 0,
'client_stats.0.read.slat_ns.max': 0,
'client_stats.0.read.slat_ns.mean': 0,
'client_stats.0.read.slat_ns.stddev': 0,
'client_stats.0.read.clat_ns.min': 324640,
'client_stats.0.read.clat_ns.max': 1766444,
'client_stats.0.read.clat_ns.mean': 604275.614815,
'client_stats.0.read.clat_ns.stddev': 297093.707537,
'client_stats.0.read.clat_ns.percentile.1.000000': 346112,
'client_stats.0.read.clat_ns.percentile.5.000000': 362496,
'client_stats.0.read.clat_ns.percentile.10.000000': 382976,
'client_stats.0.read.clat_ns.percentile.20.000000': 419840,
...skip...Full Path Json with Python
Here is an example of full path json using python.
Create the following python script call it fullpathjson.py
import json
import sys
import pprint
def full_path_json(json_string: str) -> dict:
def transform(obj, parent_key='', separator='.'):
items = []
for k, v in obj.items():
new_key = parent_key + separator + k if parent_key else k
if isinstance(v, dict):
items.extend(transform(v, new_key, separator).items())
else:
items.append((new_key, v))
return dict(items)
# Parse the JSON string and transform it
obj = json.loads(json_string)
return transform(obj)
if __name__ == "__main__":
# get filename from arg
filename = sys.argv[1]
# Read the JSON file and transform it
with open(filename, 'r') as f:
json_string = f.read()
# flattened dict
fdict = full_path_json(json_string)
# pretty print
pprint.pprint(fdict)Here is a small example json file which we can use as the input; its the same one from the fullpathjson.js comment header.
$ cat test.json
{
"one": 1,
"two": {
"three": 3
},
"four": {
"five": 5,
"six": {
"seven": 7
},
"eight": 8
},
"nine": 9
}Now apply the full path json.
$ python fullpathjson.py test.json
{'four.eight': 8,
'four.five': 5,
'four.six.seven': 7,
'nine': 9,
'one': 1,
'two.three': 3}The output is slightly different, however, it is still very grep-able and that is the whole goal.
Full Path Json with Python (Better – version 2)
Script fullpathjson2.py
import json
import sys
import pprint
def flatten_json(nested_json: dict, exclude: list=[], delim: str = "/") -> dict:
"""Flatten json object with nested keys into a single level.
Args:
nested_json {dict}: A nested json object.
exclude {list}: Keys to exclude from output.
delim {str}: path delimiter, default is "/"
Returns:
{dict}: The flattened json object if successful, None otherwise.
"""
out = {}
def flatten(x: dict, name: str='', exclude: list=[]):
if type(x) is dict:
for a in x:
if a not in exclude:
flatten(x[a], f"{name}{delim}{a}", exclude)
elif type(x) is list:
i = 0
for a in x:
flatten(a, f"{name}{delim}{i}", exclude)
i += 1
else:
out[name[1:]] = x
flatten(nested_json, exclude=exclude)
return out
if __name__ == "__main__":
filename = sys.argv[1]
file_content = open(filename,"r").read()
dict_content = json.loads(file_content)
flattened_dict = flatten_json(dict_content)
pprint.pprint(flattened_dict)Example:
# Input
$ cat test.json
{
"one": 1,
"two": {
"three": 3
},
"four": {
"five": 5,
"six": {
"seven": 7
},
"eight": 8
},
"nine": 9
}
# Output
$ python fullpathjson2.py test.json
{'four/eight': 8,
'four/five': 5,
'four/six/seven': 7,
'nine': 9,
'one': 1,
'two/three': 3}