Advanced MongoDB Operations and Mongoose Integration

Learning Objectives

• Understand how to insert and manage nested data in MongoDB.

• Learn data filtering techniques using comparison, logical, and element operators, including filtering nested documents.

• Use MongoDB cursor methods like count, sort, limit, and skip for advanced querying.

• Implement Regex for pattern matching in MongoDB.

• Understand the Need for Schema Validation: Explain its role in ensuring data integrity and enforcing business rules in web applications.

• Apply Key Mongoose Validation Terminologies: Use required, default, min, max, unique, enum, validate, and other built-in options.

• Design Comprehensive Mongoose Schemas: Build schemas with advanced validations and custom logic to enhance data accuracy and formatting.

• Apply these concepts in a movie API project using Mongoose, with data filtering, sorting, and pagination.

1. Inserting Nested Data in MongoDB

In MongoDB, documents can contain nested data, meaning that one document can include embedded documents or arrays. Nested data is useful when storing related information directly within a single document, such as movie details and reviews.

Example: Inserting Nested Data for a Movie

To insert a movie document with nested reviews in MongoDB:

db.movies.insertOne({
    title: "Inception",
    genre: "Sci-Fi",
    year: 2010,
    director: {
        name: "Christopher Nolan",
        nationality: "British"
    },
    reviews: [
        { reviewer: "Alice", rating: 5, comment: "Fantastic movie!" },
        { reviewer: "Bob", rating: 4, comment: "Great plot and visuals" }
    ]
});

In this example:

• The director field is a nested document.

• The reviews field is an array of documents.

2. Data Filtering in MongoDB

MongoDB provides a range of operators to filter data based on different conditions. Here are some commonly used operators:

Simple Filtering with find()

The find() method retrieves documents that match specified criteria:

db.movies.find({ genre: "Sci-Fi" });

Comparison Operators

• $eq: Matches values equal to a specified value.

• $ne: Matches values not equal to a specified value.

• $gt, $gte, $lt, $lte: Match values greater/less than or equal to a specified value.

Example:

db.movies.find({ year: { $gte: 2010 } });

Logical Operators

• $and: Matches documents that satisfy all specified conditions.

• $or: Matches documents that satisfy any of the specified conditions.

• $not: Matches documents that do not meet a specified condition.

Example:

db.movies.find({
    $or: [{ genre: "Sci-Fi" }, { year: { $gte: 2000 } }]
});

Element Operators

• $exists: Checks if a field exists in documents.

• $type: Matches documents where the field is a specified BSON type.

Example:

db.movies.find({ "director.name": { $exists: true } });

Filtering Nested Documents

To query nested documents, use dot notation.

Example:

db.movies.find({ "reviews.reviewer": "Alice" });

3. Cursor Methods in MongoDB

MongoDB’s cursor methods allow you to manage query results more flexibly. Common cursor methods include:

count()

Returns the number of documents matching the query:

db.movies.find({ genre: "Sci-Fi" }).count();

sort()

Sorts documents based on field values in ascending (1) or descending (-1) order:

db.movies.find().sort({ year: -1 });

limit()

Limits the number of documents returned:

db.movies.find().limit(5);

skip()

Skips a specified number of documents, useful for pagination:

db.movies.find().skip(10).limit(5);

4. Using Regex in MongoDB

MongoDB allows pattern matching with regular expressions (Regex) for flexible querying. For example, finding movies with titles starting with “The”:

db.movies.find({ title: { $regex: /^The/, $options: 'i' } });

• ^The: Matches titles starting with “The.”

• $options: 'i': Case-insensitive matching.

Schema Validation in MongoDB with Mongoose

Schema validation is an essential process that ensures data entered into a database adheres to the expected format, type, and business rules. In MongoDB, Mongoose (a popular ODM for Node.js) provides a way to define schema validations to enforce data integrity at the application level.

---

Why Schema Validation is Needed

1. Data Consistency: Ensures that the data stored in the database is consistent and follows the same format and rules.

2. Prevents Invalid Data: Stops users or applications from inserting incorrect or incomplete data.

3. Reduces Errors: Minimizes bugs caused by incorrect data inputs.

4. Enforces Business Rules: Validates data according to business requirements, such as required fields, minimum/maximum values, or specific formats like email.

5. Improves Data Quality: Helps maintain high data accuracy and reliability.

---

Common Mongoose Validation Terminologies

1. required:

   • Ensures that the field must be provided.

   • If not present, Mongoose will throw a validation error.

   • Example:

       required: true
     

2. default:

   • Specifies a default value if the field is not provided.

   • Example:

       default: "No description available"
     

3. type:

   • Defines the type of the field (e.g., String, Number, Boolean, Date).

   • Example:

       type: String
     

4. enum:

   • Restricts the field’s value to a predefined set of values.

   • Example:

       enum: ["male", "female", "non-binary"]
     

5. min:

   • Ensures the field's value is greater than or equal to the specified minimum.

   • Applies to Number or Date types.

   • Example:

       min: 18
     

6. max:

   • Ensures the field's value is less than or equal to the specified maximum.

   • Applies to Number or Date types.

   • Example:

       max: 65
     

7. match:

   • Validates the field against a regular expression.

   • Example (email validation):

       match: /^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$/
     

8. unique:

   • Ensures that the field value is unique across all documents in the collection.

   • Example:

       unique: true
     

9. validate (Custom Validator):

   • Allows you to define custom validation logic using a function.

   • Example:

       validate: {
         validator: function(v) {
           return v.length >= 3;
         },
         message: "Name must be at least 3 characters long"
       }
     

10. trim:

    • Removes leading and trailing whitespaces from a string.

    • Example:

        trim: true
      

11. lowercase / uppercase:

    • Converts the string to lowercase or uppercase before storing it in the database.

    • Example:

        lowercase: true
      

12. immutable:

    • Prevents the field from being changed after the document is created.

    • Example:

        immutable: true
      

---

Comprehensive Schema with All Fields and Validations

Below is an example Mongoose schema for a User collection, demonstrating all the commonly used validation options:

const mongoose = require('mongoose');

const userSchema = new mongoose.Schema({
  username: {
    type: String,
    required: true,
    unique: true,
    trim: true,
    minlength: 3,
    maxlength: 20
  },
  email: {
    type: String,
    required: true,
    unique: true,
    match: /^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$/
  },
  password: {
    type: String,
    required: true,
    minlength: 8
  },
  age: {
    type: Number,
    min: 18,
    max: 65,
    default: 18
  },
  gender: {
    type: String,
    enum: ["male", "female", "non-binary"],
    default: "non-binary"
  },
  bio: {
    type: String,
    trim: true,
    default: "No bio provided",
    maxlength: 500
  },
  isActive: {
    type: Boolean,
    default: true
  },
  createdAt: {
    type: Date,
    default: Date.now,
    immutable: true
  },
  preferences: {
    theme: {
      type: String,
      enum: ["dark", "light"],
      default: "light"
    }
  },
  customField: {
    type: String,
    validate: {
      validator: function (v) {
        return v !== "forbidden"; // Custom validation rule
      },
      message: "CustomField cannot have the value 'forbidden'"
    }
  }
});

module.exports = mongoose.model('User', userSchema);

---

Explanation of the Schema

1. username:

   • Required, unique, trimmed, and must be between 3 and 20 characters long.

2. email:

   • Required, unique, and validated using a regular expression for proper email formatting.

3. password:

   • Required and must be at least 8 characters long.

4. age:

   • Defaults to 18 if not provided, and must be between 18 and 65.

5. gender:

   • Must be one of the specified values in the enum.

6. bio:

   • Trimmed, has a maximum length of 500 characters, and defaults to “No bio provided” if not set.

7. isActive:

   • Boolean field that defaults to true.

8. createdAt:

   • A date field set to the current date by default and marked as immutable.

9. preferences.theme:

   • Nested field for storing user preferences with a default theme value of "light."

10. customField:

    • Custom validation logic that ensures the field value is not "forbidden."

---

Benefits of Using Mongoose Validation

1. Prevents Invalid Data: Ensures only valid data gets stored in the database.

2. Centralized Rules: Keeps all validation rules in one place, making the schema easy to understand and maintain.

3. Error Handling: Automatically throws descriptive validation errors, making it easier to debug and provide user-friendly feedback.

4. Flexible: Supports both simple validations (e.g., required) and complex custom validations.

By leveraging Mongoose validations, developers can enforce data integrity and build robust, error-resistant applications!

Project: Advanced Movie API with Mongoose

In this project, we’ll enhance our movie API to support nested data, filtering, sorting, and pagination.

Step 1: Define Mongoose Schema with Nested Data

1. Movie Schema (models/Movie.js):

    const mongoose = require('mongoose');
   
    const reviewSchema = new mongoose.Schema({
        reviewer: String,
        rating: Number,
        comment: String
    });
   
    const movieSchema = new mongoose.Schema({
        title: { type: String, required: true },
        genre: String,
        year: Number,
        director: {
            name: String,
            nationality: String
        },
        reviews: [reviewSchema]
    });
   
    module.exports = mongoose.model('Movie', movieSchema);
   

Step 2: Adding Nested Data (POST Endpoint)

1. Create a New Movie with Reviews:

    app.post('/movies', async (req, res) => {
        try {
            const movie = new Movie(req.body);
            await movie.save();
            res.status(201).send(movie);
        } catch (error) {
            res.status(400).send(error);
        }
    });
   

Step 3: Filtering with Mongoose

1. Filter by Genre and Year Range:

    app.get('/movies', async (req, res) => {
        try {
            const { genre, yearFrom, yearTo } = req.query;
            const filter = {};
   
            if (genre) filter.genre = genre;
            if (yearFrom || yearTo) filter.year = {};
            if (yearFrom) filter.year.$gte = parseInt(yearFrom);
            if (yearTo) filter.year.$lte = parseInt(yearTo);
   
            const movies = await Movie.find(filter);
            res.send(movies);
        } catch (error) {
            res.status(500).send(error);
        }
    });
   

2. Filtering by Reviewer:

    app.get('/movies/reviews/:reviewer', async (req, res) => {
        try {
            const movies = await Movie.find({ "reviews.reviewer": req.params.reviewer });
            res.send(movies);
        } catch (error) {
            res.status(500).send(error);
        }
    });
   

Step 4: Sorting, Pagination, and Regex Search

1. Sorting and Pagination:

    app.get('/movies', async (req, res) => {
        const { sortField = "year", sortOrder = "desc", page = 1, limit = 10 } = req.query;
   
        try {
            const movies = await Movie.find()
                .sort({ [sortField]: sortOrder === "asc" ? 1 : -1 })
                .skip((page - 1) * limit)
                .limit(parseInt(limit));
   
            res.send(movies);
        } catch (error) {
            res.status(500).send(error);
        }
    });
   

2. Regex Search for Titles:

    app.get('/movies/search', async (req, res) => {
        const { title } = req.query;
   
        try {
            const movies = await Movie.find({ title: { $regex: title, $options: 'i' } });
            res.send(movies);
        } catch (error) {
            res.status(500).send(error);
        }
    });
   

Summary

• Nested Data: We can store nested documents in MongoDB, ideal for embedding related information.

• Filtering: MongoDB offers operators for flexible filtering, including filtering nested fields.

• Cursor Methods: count, sort, limit, and skip enable advanced data management.

• Regex: Useful for pattern matching, e.g., for searching titles.

• Project: We created a robust movie API with Mongoose, including nested data handling, data filtering, sorting, and pagination.