From .grib to MongoDB: A Weather Wizard's Guide to Storing Forecast Values

Where We Left Off

In the last part of this series, we managed to automatically pick, download and extract the .grib files we received from the FTP server. If you don't know what I am talking about, check out Part 1.

Effective Forecast Models in MongoDB

In this section, our main objective is to set up MongoDB and design models to store the forecast values that we will generate in the upcoming part of this series.

Cloud Power: MongoDB Atlas

We will use MongoDB Atlas since it's free for our use case. MongoDB Atlas is a cloud database, which means we don't have to struggle with installing one on our machine. You can sign up for free at MongoDB.com.

Step 1: Sign Up

1. Sign up for MongoDB Atlas.

2. After signing up, you'll be prompted to answer a few questions about your use case.

Step 2: Create a New Database

1. On the next screen, navigate to create a new database.

2. Choose the "free M0 tier" as it will suit your needs perfectly.

3. Select a provider, a location, and provide a name for your cluster.

4. Click on the "Create" button.

Step 3: Create a User

1. After your cluster is created, the next step is to create a user.

2. Enter a username and a password for the user.

3. Click on "Create User."

4. Click "Finish" and then "Close."

Step 4: Get the Connection String

1. In the Database Section of your MongoDB Atlas, find your newly created cluster.

2. Click on the "Connect" button next to the cluster.

3. In the Popup, choose "Drivers" in the "Connect to your application" section.

4. Select "Node.js" as the driver and the most recent version.

5. Skip Step 2 (installing the npm module) for now.

6. Look at Step 3 and save the listed connection string. Remember to replace <password> with the actual password of the recently created user.

Step 5: Add a Database

1. To add a new database to your cluster, click on "Browse Collections."

2. Click on "Add My Own Data."

3. Choose a name for the database and fill in the required fields for the name of the collection.

4. Click on "Create," and your database will be set up on the cluster.

The main goal here is to have a database on a cluster with the connection string securely saved for later use. Now, you have successfully set up your MongoDB cluster and are ready to start using it for your application! If you want to learn more about MongoDB, I can recommend its free education program.

Keeping Secrets Safe: Installing dotenv

Let's go back to the terminal and install some npm packages. The first package we want to install is dotenv. With the help of dotenv, we can keep our connection string, which contains the password and username for our database, safe. You likely want to use Git or another version control. If you were to upload your code to GitHub, your login information could be exposed. With dotenv, you can prevent this. So let's install it, and then we will look at how it will work.

npm install dotenv

Vital Setup: .gitignore + .env

To ensure proper functionality, we need to create two files in the root folder, where our package.json is located.

touch .env .gitignore

Igniting Ignored Files

In the .gitignore file, we specify intentionally untracked files that Git should ignore. This is exactly what we want for our .env file. We simply add .env to the file. Additionally, we can add node_modules and any downloaded .grib files to the ignore list.

// .gitignore
.env
node_modules
grib_data/*

Crafting the .env Magic

The .env file is pretty straightforward. We simply add our MongoDB connection string to it. As you may notice, I have included the name of my database in the string: ...mongodb.net/windspotter?... For establishing the connection, remember to add the name of your database too.

// .env
MONGODB_URI = "mongodb+srv://<username>:<password>!*******.mongodb.net/windspotter?retryWrites=true&w=majority"

Embracing Mongoose for MongoDB

As we explored in the connection instructions on the MongoDB website, it is possible to install a MongoDB package via npm. Instead of using the official MongoDB driver, we will opt for Mongoose. Mongoose is built on top of the official MongoDB Node.js driver and provides additional features, making it easier to work with your database.

npm install mongoose

Weaving the Database Connection

To establish the database connection with Node.js, we can achieve it with just a few lines of code added to our main index.js file. Initially, we need to require mongoose and dotenv/config to access the MONGODB_URI variable. Next, we run mongoose.connect() with the connection string and an options object. Subsequently, we store the connection in a variable, and if the connection fails, we log an error message.

// src/index.js
const mongoose = require('mongoose');
require('dotenv/config');
const { downloadFiles } = require('./ftp');

mongoose.connect(process.env.MONGODB_URI, {
  useUnifiedTopology: true,
  useNewUrlParser: true,
});
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'mongo connection error'));

downloadFiles();

Unraveling MongoDB Models

Working with Mongoose is a straightforward process. Once we establish a connection to our database, the first step is to define models. Models serve as Mongoose's way of easily describing the layout of MongoDB documents or schemas. Currently, we aim to create three distinct models: spot, forecast, and forecastInfo.

ForecastInfo Model: Weather Prediction Essentials

The ForecastInfo model will store essential information about weather prediction models like ICON, GFS, or ICWMF. This data includes the model name, grid size, and the last update time. As our application expands, it is likely that we will have multiple prediction models to cover various regions, offering forecasts for different time periods, such as 3-day or 10-day predictions.

Forecast Model: Spot-Specific Predictions

The Forecast model encompasses the forecastName, forecastTimestamp, and forecasted values specific to a particular spot.

Spot Model: Geolocations and Forecasts

The Spot model describes geolocations, representing spots with attributes such as spot name, latitude, longitude, and an array of forecasts for different prediction models. For now, we will store only the values from the corresponding .grib files, disregarding forecast values that are not covered by spots.

The Whole Grid Dilemma

Storing the entire grid is not feasible due to the sheer volume of data. Each .grib file contains over 900,000 values, and with at least 200 .grib files, we'd be dealing with an enormous number of values. This could lead to excessively large documents in our database, which is not suitable for our current free MongoDB Atlas plan.

Scalability and Flexibility

While accommodating 18 million values is possible, it surpasses the capabilities of our current MongoDB Atlas plan. Nonetheless, we can easily scale our system by adding more spots as the app grows. Additionally, we have the option to store the whole grid at a later stage when we can afford a more extensive Atlas plan or decide to host our database independently.

Organizational Approach

To ensure a well-structured project, we opt for a separate folder for our models, with each model having its dedicated .js file. In addition, we will consolidate all the models in an index file, simplifying management as the number of models increases.

mkdir src/models
touch src/models/spot.js src/models/forecast.js 
touch src/models/forecastinfo.js src/models/index.js

Streamlined Forecast Model Setup

We begin by importing Mongoose and using destructuring to extract the Schema. With that, we create a new Schema instance by invoking new Schema(). The object passed as a parameter to the Schema constructor contains essential properties for the forecast model. Unlike JavaScript, MongoDB requires specifying the variable type explicitly. Additionally, we can set options such as required or default values. Starting with these straightforward structures allows for potential expansion as our app evolves. Lastly, we export the Schema as a model for utilization throughout the application.

// src/models/forecastinfo.js
const mongoose = require('mongoose');

const { Schema } = mongoose;

const ForecastInfoSchema = new Schema({
  name: { type: String, required: true, maxLength: 100 },
  time: { type: Date, required: true },
  lo1: { type: Number, required: true },
  lo2: { type: Number, required: true },
  la1: { type: Number, required: true },
  la2: { type: Number, required: true },
  dy: { type: Number, required: true },
  dx: { type: Number, required: true },
});

module.exports = mongoose.model('ForecastInfo', ForecastInfoSchema);

Smart References: Forecast Info for Efficiency

Our primary objective is to store forecast values efficiently for a specific spot. To achieve this, we utilize a property called forecast Info, which serves as a reference to our Forecast Model. This reference allows us to access important forecast details such as time and name seamlessly. By referencing the location where this information is already stored, we avoid redundant data and ensure that any updates need only be made in one central place. This approach promotes data consistency and simplifies maintenance, making it a more practical choice for our application.

// src/models/forecast.js
const mongoose = require('mongoose');

const { Schema } = mongoose;

const ForecastSchema = new Schema({
  forecastInfo: { type: Schema.Types.ObjectId, ref: 'Forecast' },
  t: { type: Object },
  v: { type: Object },
  u: { type: Object },
});

module.exports = mongoose.model('Forecast', ForecastSchema);

Intuitive Connection: Spot & Forecast Models

The Spot model includes fundamental attributes like geolocation and spot name, along with an array of references to the Forecast Model. As you may observe, we have kept the model design simple and intuitive. The Models reference each other in a one-way manner: Spot -> Forecast -> ForecastInfo.

// src/models/spot.js
const mongoose = require('mongoose');

const { Schema } = mongoose;

const SpotSchema = new Schema({
  name: { type: String, required: true, maxLength: 100 },
  lat: { type: Number, required: true },
  lon: { type: Number, required: true },
  forecasts: [{ type: Schema.Types.ObjectId, ref: 'Forecast' }],
});

module.exports = mongoose.model('Spot', SpotSchema);

Efficient Model Handling in index.js

In the index.js file, we import each model and export them within a unified Object. This organizational approach proves to be highly convenient, particularly as the number of models increases.

// src/models/index.js
const Forecast = require('./forecast');
const ForecastInfo = require('./forecastinfo');
const Spot = require('./spot');

module.exports = {
  Forecast,
  ForecastInfo,
  Spot,
};

Conclusion

We have made significant progress in setting up our weather forecasting system with MongoDB. We utilize MongoDB Atlas for a secure cloud database, and the dotenv package ensures the protection of sensitive data in version control.

Our MongoDB models efficiently store forecast data for specific locations. In the next part, we will convert .grib files to JSON and look at the data we will receive.

As always, your comments and feedback are most welcome. Stay with me as we continue this journey to build a weather forecasting app based on .grib files.

-Tobias