Simple Steps for .grib to JSON Conversion
Crafting a Weather Forecast App with the MERN Stack (MongoDB, Express, React, Node.js) ... Part 3
Picking Up Where We Left Off
In the previous section, we established the foundation for our application by configuring MongoDB and designing effective models to store forecast data. If you overlooked any information regarding this vital setup, I recommend revisiting Part 2, where we thoroughly discussed these critical steps.
Today's Objectives
In this section, our goal is to convert .grib files into JSON format and examine the data we plan to work with. To accomplish this, we need to install the Java command line tool, grib2json. Now, you might wonder, why are we using Java in a JavaScript project.
Did I miss the Script in Java
To convert our .grib files into a useful JSON format, we need to utilize the npm package grib2json. As you might expect, it does precisely what we want. However, the downside of this package is that it serves as a wrapper for a Java program.
Java & JavaScript: BFFs!
The .grib file format is quite popular for weather forecasts, but sadly, it's not popular enough to have a suitable npm package for our needs without involving Java. Therefore, we must accept this reality. On the bright side, all we need to do is install Java, and grib2json will handle the rest.
Easy Peasy: grib2json Installation
Alright, let's begin by installing the package, and you'll see that it's quite a simple setup. Change the directory to our root folder and run npm install grib2json.
npm install grib2json
As usual, New Folder & index.js
To maintain organization, let's begin by creating a new folder. Inside this folder, we will add an index.js file. This step will ensure a structured approach to our project.
mkdir src/convert_grib
touch src/convert_grib/index.js
Npm & Java: grib2json Confusion
The npm package grib2json is essentially a wrapper around the command-line tool grib2json written in Java. To use the npm package, you'll also need to install the grib2json Java command-line tool. Please note that both the npm package and the Java command-line tool share the same name, which might be a little confusing at first.
To integrate the grib2json tool into our project, follow these steps:
Download the repository by clicking the "Code" button and selecting "Download ZIP."
Extract the downloaded ZIP file.
Copy the main folder and paste it into our project's newly created "src/convert_grib" folder.
Once copied, rename the main folder to just "grib2json"
Now, within the "grib2json" folder, delete all files and folders except the "src/" folder and the "pom.xml" file. This ensures that we keep only the necessary files needed to build and run the grib2json tool.
After completing these steps, our file structure should look like this:
src/
├── convert_grib/
│ ├──grib2json/
│ │ ├── src/
│ │ └── pom.xml
│ └── index.js
└── ...css
Java Download & Install: Easy as 1-2-3
Visit the official Oracle Java website.
Choose the appropriate Java version for your operating system (Windows, macOS, Linux).
Accept the license agreement and download Java.
Run the installer and follow the on-screen instructions to complete the installation.
Verify the installation by typing
java -versionin the terminal/command prompt.
After the installation, navigate to the src/convert_grib/grib2json folder and run mvn package to install the grib2json Java-command-line tool. To execute this command, you need to have Maven installed. If you have Homebrew installed, you can simply use brew install maven.
cd src/convert_grib/grib2json
mvn package
cd ../../../
File Shuffle
After running mvn package, you will need to unzip and untar the file target\grib2json~SNAPSHOT.tar.gz. This action will generate two subdirectories, namely bin and lib. Now, proceed to move both directories to the grib2json/src folder. Make sure to replace the existing bin folder with the new one.
Error? No Fear! I've Got Your Back!
That's all we needed. Now we can work with our npm package and don't need to worry about the grib2json (Java) command line tool. If you encounter any errors during the installation process, try checking the repository's issues or feel free to ask me for assistance.
Smooth Data Flow
As always we want to stay organized. We start by setting up a function named convertGrib() . In this function, we will take two arguments, an array of filenames, and the path to the files. The first step is to get the list of all spots from our database. Therefore we need to require our Spot model. Then we will loop over every file and call the poplateSpots() function with the filename and the spots array. The last step is to save the changes on the spot's forecasts. Again we will wrap everything in try and catch to avoid unexpected behavior in our app.
// src/convert_grib/index.js
...
const { Spot } = require('../models');
...
const convertGrib = async (filenames, path) => {
const spots = await Spot.find({}).populate('forecasts').exec();
if (!spots) {
return false;
}
try {
for (const filename of filenames) {
await populateSpots(`${path}/${filename}`, spots);
}
for (const spot of spots) {
await spot.save();
}
} catch (err) {
console.log(err);
return false;
}
return true;
};
module.exports = {
convertGrib,
};
Looping Magic
As you may have noticed, we are missing one function to make this work. The function populateSpots() serves the purpose of, as you can guess, populating the spots with our forecast data. We call this function file by file. Essentially, we convert a single .grib file and then save the forecast values for this specific forecast time in our spots. That's why we call this function in a loop; we have to do this over and over again for every file.
Enhanced Functionality
Let's examine the populateSpots() function. Since we want to use grib2json in this function, we need to require it. However, there is one issue with the package. As it hasn't been updated for a few years, it doesn't support promises out of the box. Thankfully, we can overcome this limitation with just two additional lines of code and leverage async/await instead of callbacks. By using util.promisify(), we can avoid falling into callback hell and get the functionality we need.
const util = require('util');
const grib2json = require('grib2json').default;
const getJson = util.promisify(grib2json);
Unraveling the Grib Goodies
To retrieve the JSON data from the .grib file, we can call the getJson() function, passing the name of the .grib file and an options object as arguments. The options object should include the script path, which, in our case, is: ./src/convert_grib/grib2json/src/bin/grib2json, along with two booleans for the data and names. Both of these booleans should be set to true.
By setting data: true, we indicate that we want to obtain not only the forecast information but also the forecast values themselves.
With the option names: true, we receive additional descriptions in the forecast information, which proves to be quite helpful, especially in the initial stages of our work.
// src/convert_grib/index.js
const util = require('util');
const grib2json = require('grib2json').default;
...
const getJson = util.promisify(grib2json);
...
const populateSpots = async (filename, spots) => {
const forecastJson = await getJson(filename, {
scriptPath: './src/convert_grib/grib2json/src/bin/grib2json',
names: true, // (default false): Return descriptive names too
data: true, // (default false): Return data, not just headers
});
...
};
...
A Glance at the Data
I called our function using a sample file and logged the data with console.log(). I removed many uninteresting values from the header for now. Take a first look at the JSON. Once you're done, we'll go through it line by line.
[
{
header: {
refTime: '2023-07-26T12:00:00.000Z',
parameterNumberName: 'Temperature',
parameterUnit: 'K',
forecastTime: 0,
gridDefinitionTemplateName: 'Latitude_Longitude',
numberPoints: 906390,
nx: 1215,
ny: 746,
lo1: 356.06,
la1: 43.18,
lo2: 20.34,
la2: 58.08,
dx: 0.02,
dy: 0.02,
... alot more items
},
data: [
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN', 'NaN',
'NaN', 'NaN', 'NaN', 'NaN',
... 906290 more items
]
}
]
Header Insights
In the header of the JSON, we will find some useful values. RefTime refers to the starting time of the forecast model. ForecastTime: 0 indicates that this is the .grib file for our first forecast hour, beginning at the refTime. These two values correspond to the date and the forecastTime in the filename.
ParameterNumberName and ParameterUnit represent the parameter and its unit in this .grib file, as you might guess. Since K stands for Kelvin, we need to convert it to either Fahrenheit or Celsius.
The gridDefinitionTemplateName also corresponds to the filename. The numberPoints value indicates the total number of points in the grid. Nx and Ny represent the number of values along the x-axis and y-axis of the grid, respectively. These values can also be calculated from the other values. Lo1, Lo2, La1, and La2 define the boundaries of the grid. Dx and Dy represent the spacing of the values along the x-axis and y-axis, respectively.
'NaN' sounds Cold
As you may have noticed, our data arrays consist solely of NaN values initially. However, this is only the case at the beginning of the array. By selecting the regular-lat-lon forecast model, we have chosen a data model that has already been converted. The original model is based on a triangular grid, which will be transformed into a rectangular grid for our use. You can see this in the image below.
When attempting to overlay a rectangle on the original forecast model, some red areas become visible. These values within these areas are not covered by the model.
1D Array for Maps
The forecast values are provided in a simple Array. This is not so easy to work with initially. When we think about storing values on a map, we might want to choose nested arrays since we can refer to the values using coordinates like array[lat][lon]. However, with the provided 1D array, we have to convert our coordinates into an index for the array first.
Delve deeper into the rabbit hole.
Alright, we will need to take a few more steps to ultimately save our forecast values in our database. I must end this article here because it is already quite lengthy. We will address these steps in the next part. So far, we have successfully installed our primary tool, grib2json, and learned how to obtain data from it. We have also examined this data and now understand what needs to be done to work with it.
In the next part, we will finally convert and interpolate our data to store it in MongoDB. Thank you for reading, and as always, your comments and feedback are greatly appreciated. See you in Part 4.
-Tobias