The Ultimate List of Mongoose Advance Filtration Implementations

If you’re looking for the ultimate list of Mongoose advanced filtration implementations, look no further! This comprehensive list includes everything from simple text filters to more complex image filters. Filtration and pagination have a significant role in API design. A good API design creates a great developer experience (DX). However, there’s no specific standard guideline for API design. RESTful is just an architecture for serving data to front-end applications from web servers. But to cater to data serving efficiently our APIs need to be efficient to reduce the over-fetching and under-fetching of data.

In this tutorial, we are going to build an API with advanced filtration and pagination with the help of Node.js, Express.js & mongoose and will be adding in-route validations for API using Joi validations and express-validation.

1. Create a folder with the name adv-filtration or go with your choice of name.
`mkdir adv-filtration`

2. Initiate and project with the command `yarn init –y`

3. Create an src directory where we will write all of our source code.

4. Add following dependencies

Express – For server initialization and routing.
Mongoose – To work with mongo-DB.
Dotenv – To load our environment variable from the .env file.
Joi – for creating validation schema for routes.
Express-validation – To validate in request before it reaches the main controller.

By `yarn add express mongoose dotenv joi express-validation`

5. Add nodemon as dev-dependency `yarn add –D nodemon’
6. Add the following command inside the `package.json` file in the scripts block.

package.json

"scripts": {
		"start": "node src/index.js",
		"dev": "nodemon src/index.js"
		},

7. Now we must create a database schema to work with mongo-DB using mongoose.
Create a new folder by the name `models` inside the src directory and create a file with the name
`product.model.js`.

Add the following code to it.

src/models/product.model.js

const mongoose = require('mongoose');
		const { Schema } = mongoose;
		
		const ProductSchema = new Schema({
		title : { type: String, required: true },
		type : { type: String, required: true },
		description : { type: String, required: true },
		price : { type: Number, required: true },
		rating : { type: Number, required: true, max: 5 }
		}); 
		
		module.exports = mongoose.model('product', ProductSchema, 'products');

8. Now we have to create validations for in-route validations for validating the queryString which is responsible for querying the data. And we have to make sure we protect our API from misbehaving because of the wrong conditions in the query.

Note. Whatever we will receive in query will be determining the querying data directly so we have to make our validation as flexible and strong as required.

src/models/product.validation.js

const Joi = require('joi');
		
		const numberValidations = [
		Joi.number().optional(),
		Joi.object({
		gt: Joi.number().min(1).optional(),
		gte: Joi.number().min(1).optional(),
		lt: Joi.number().min(1).optional(),
		lte: Joi.number().min(1).optional()
		}).optional().not({})
		]
		
		const validTypes = ["bakery", "dairy", "fruit", "meat", "vegan", "vegetable"];
		
		exports.getAllProducts = {
		query: Joi.object({
		title: Joi.string().optional(),
		type: Joi.string().valid(...validTypes).optional(),
		page: Joi.number().min(1).optional(),
		limit: Joi.number().min(1).optional(),
		price: numberValidations,
		rating: numberValidations,
		sort: Joi.string().optional(),
		fields: Joi.string().optional()
		})
		}

In the above validations, we are making every field optional, but if they are provided we need to make sure to validate them, like `numberValidation` here we are supposed to expect a single number of an object having fields [ gt, gte, lt, lte ] only. Type should be one of [“bakery”, “dairy”, “fruit”, “meat”, “vegan”, “vegetable”] etc.

9. Now we will create our controller in the src directory with the name product.controller.js which is responsible for serving our data. Our controller will be flexible enough to fetch data from the database with given conditions in query, sort records according to the given field and more than one field and orders of given fields in a query, paginate it, and limit the number of fields requested by the client.

src/controllers/product.controller.js
		
		const getAllProducts = async (req, res, next) => {
		try {
		
		let queryObject = { ...req.query };
		
		/* Basic Filtration */
		const excludeFields = ['page', 'sort', 'limit', 'fields'];
		excludeFields.forEach(item => delete queryObject[item]);
		
		/* Advance Filtering */
		let queryString = JSON.stringify(queryObject);
		queryString = queryString.replace(/\b(gte|gt|lte|lt)\b/g, match => `$${match}`);
		queryObject = JSON.parse(queryString);
		
		/* Search on the basis of title of product */
		if(queryObject.title) {
		queryObject.title = new RegExp(queryObject.title, 'i');
		}
		
		let query = Product.find({ ...queryObject });
		const countQuery = Product.find({ ...queryObject });
		
		/* Sorting */
		if(req.query.sort) {
		const sortBy = req.query.sort.split(',').join(' ');
		query = query.sort(sortBy);
		} else {
		query = query.sort({ createdAt: - 1 })
		}
		
		/* Limiting the field ( projection ) */
		if (req.query.fields) {
		const fields = req.query.fields.split(',').join(' ');
		query = query.select(fields);
		} else {
		query = query.select('-__v');
		}
		
		/* Pagination */
		const page = req.query.page * 1 || 1;
		const limit = req.query.limit * 1 || 10;
		const skip = (page - 1) * limit;
		
		const totalRecords = await countQuery.countDocuments();
		
		query = query.skip(skip).limit(limit);
		
		let numOfRecords = 0;
		if (req.query.page) {
		numOfRecords = await Product.countDocuments({ ...queryObject });
		if (skip > numOfRecords) {
		return res.status(404).json({ status: false, message: 'Page does not exists!' });
		}
		}
		
		let products = await query;
		return res.status(200).json({
		status: true,
		data: {
		products,
		totalPages: Math.ceil(totalRecords / limit),
		page,
		limit,
		totalRecords
		}
		});
		} catch (error) {
		next(error)
		}
		}

In the above controller, we will be having the below flow.
a) Copy everything from req. query to queryObject using spread operator so modification in queryObject doesn’t affect the req. query;
b) Basic Filtration
c) Remove these [‘page’,’sort’,’limit’,’fields’] from queryObject.
d) Advance Filtration
e) Convert our queryObject to JSON string using JSON.stringify to queryString and use regex to replace exact gt/gte/lt/lte words with $ prepend with it so they can act as MongoDB operators.
i) e.g gt to $gt, lt to $lt.
ii) and parse back that queryString to queryObject again using JSON.parse()
g) If we are given the facility to search products by titles. Then convert that title string to regex to search globally and case-insensitively.
Till now we have created the condition of our query now we will create a query so we can perform additional operations on it for reference of query you can read it here: https://mongoosejs.com/docs/queries.html

Along with that, we will fire a query with the same query object in order to get the total count of records with the given filtration.

h) Now it’s time to sort. If we have got a sort request in the req. the query then we will split the sort value and join them to create a comma-separated string having fields separated by a comma. And will use the query.sort() method and pass that string to it.
What if we didn’t receive a sort request we will sort the record by its creation date by default.

i) Now it’s time to limit the number of fields to be sent in response, If we have got fields in the req. the query then we will split and fields value and join them to create a comma, separated string and will use query.select() method and pass that string to it. So the mentioned fields are only the object we will be getting from the response.
a) What if we didn’t receive the fields in request them we will remove the __v field by default.

b) Note: To remove field use – before that field and that field will be removed from response.
c) Or mention those fields only those fields which you expect from the response.

j) Now its turns for pagination
If page and limit are not provided in req. the query then by default the page will be 1 having a limit of 10 records per page.
For the custom pages, you can pass pages and limit values in the request.

k) Then comes the last condition to check if the page is requested is present or not.
If not then we have to throw an error with an appropriate message.

l) Till now we have built our query with all features, it’s time to fire that query now, we will execute that query and will send a response with that data and pagination-related data like currentPage, limit, total pages, and total records available.

m) Send the gathered data.

10) Now create the route for the product inside the routes directory with name product.route.js
And add the following code to it.

const router = require('express').Router();
		const { validate } = require('express-validation');
		const { getAllProducts } = require("../validations/product.validation");
		const ProductController = require('../controllers/product.controller');
		
		router.get('/', (req, res, next) => res.redirect('/products'));
		router.use('/products', validate(getAllProducts), ProductController.getAllProducts);
		
		module.exports = router;

Note: Here we have added the in-route validation of query using the assurance we have created using joi and validating it using validate function imported from express-validation.

11) Create the .env file on the root level of the project and add the following code to it.

PORT=4000
DB_URI=mongodb://localhost:27017/test-DB

12) Now final thing to do add create an index.js file inside the src directory and add the below code.

require('dotenv').config();
		const express = require('express');
		const { ValidationError } = require('express-validation');
		const mongoose = require('mongoose');
		const indexRouter = require('./routes/product.route');
		const port = process.env.PORT || 4000;
		const app = express();
		
		app.use(indexRouter);
		
		app.use((req, res, next) => {
		const err = new Error('Resource not found');
		err.status(404);
		next(err);
		});
		
		app.use((err, req, res, next) => {
		let status;
		let message;
		if (err instanceof ValidationError) {
		status = 422;
		message = err.details.query[0].message;
		} else {
		status = err.status || 500;
		message = err.message || 'Something went wrong.';
		}
		return res.status(status).json({ status: false, message });
		})
		
		mongoose.connect(process.env.DB_URI).then(() => {
		console.log("Database is Connected.");
		app.listen(4000, () => console.log(`Server is up 🚀 and running at ${port}`));
		});

We have done with our server part now you can call our API with filtration.
e.g

server part