top of page
Search

Guide to Using MongoDB Backup Tool and Integrating it into a Microservices System


1. The Problem

In system operations, data loss is inevitable. The main reasons include:

  • Conflicts during data migration

  • System hacks

  • Developers accidentally deleting data

Recovering data can be time-consuming, which is why daily backups and backups before version updates are essential.

There are various solutions for this, such as using server provider backup services, but they often come with a cost. How can we ensure data safety for free using existing resources?


MongoDB

2. The Solution MongoDB Backup Tool

Before proposing a solution, let's revisit the problem our company faced:

  • Our data volume is not large, mainly storing user and app information, not transaction data. Using external services would be wasteful.

  • The system is regularly maintained, and server migrations during major updates are inevitable.

  • We previously used shell scripts for data backup, but setting them up again during system migrations (across dev, staging, and production environments) was cumbersome.

  • We do not use cloud database services but prefer a homegrown solution - deployed as services.

=> We need a service that can access the MongoDB service to back up all databases and upload the backup files to a storage repository (I chose Google Drive because it is free).


3. Features

  • Automatically backup MongoDB databases at specified times and upload to Google Drive

  • Allow forced backups of MongoDB databases

  • Enable automatic deletion of MongoDB backup files on Local and Drive after a specified time

  • Send MongoDB backup status notifications via Telegram


4. Integration

4.1. Retrieve Necessary Information from Google

Please refer to section 6. Demo for instructions on retrieving the required parameters.

  • Google Client Mail

  • Google Private Key

  • Google Folder ID


4.2. Local Testing

Create a .env file at the same level as the src directory and initialize the environment variables:

GOOGLE_CLIENT_MAIL=""
GOOGLE_PRIVATE_KEY=""
GOOGLE_FOLDER_ID=""
IS_FORCE_BACKUP=1

4.3. Production

Create a docker-compose.yml file as follows:

version: '3.8'
services:
  mongodb_backup:
    image: vtuanjs/mongodb_backup:1.0.0
    networks:
      - net
    environment:
      NODE_ENV: 'production'
      GOOGLE_CLIENT_MAIL: ${GOOGLE_CLIENT_MAIL}
      GOOGLE_PRIVATE_KEY: ${GOOGLE_PRIVATE_KEY}
      GOOGLE_FOLDER_ID: ${GOOGLE_FOLDER_ID}
      MONGO_ROOT_USER: ${MONGO_ROOT_USER}
      MONGO_ROOT_PASSWORD: ${MONGO_ROOT_PASSWORD}
      MONGO_HOST: ${MONGO_HOST}
      IS_FORCE_BACKUP: ${IS_FORCE_BACKUP}
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == worker
      resources:
        limits:
          cpus: '0.70'
          memory: '700M'
        reservations:
          memory: '4M'
networks:
  net:
    driver: overlay
    attachable: true

The above configuration is the minimum required for the app to function. You can customize additional features using the environment variables in section 5.

Note:

  • MONGO_ROOT_USER: A user with root or read-all-database permissions.

  • The folder storing backup files must be shared with the Client Mail.


5. Environment Variables

  • MONGO_ROOT_USER: User

  • MONGO_ROOT_PASSWORD: Password

  • MONGO_HOST: IP address / Container name / Service name of MongoDB

  • MONGO_PORT: Port

  • IS_AUTO_BACKUP: Enable automatic backup or not, Default: 1

  • CRON_JOB_TIME: Backup schedule in GMT+7. Default: '00 00 *' (0:00 AM)

  • IS_FORCE_BACKUP: Force backup on app start. Default: 0

  • IS_REMOVE_OLD_LOCAL_BACKUP: Allow deletion of expired backups on the server. Default: 1

  • KEEP_LAST_DAYS_OF_LOCAL_BACKUP: Retention period for local backups. Default: 2 days

  • IS_REMOVE_OLD_DRIVE_BACKUP: Allow deletion of expired backups on Drive. Default: 1

  • KEEP_LAST_DAYS_OF_DRIVE_BACKUP: Retention period for Drive backups. Default: 7 days

  • GOOGLE_CLIENT_MAIL: Mail with API usage rights (generated by Google when creating Google User Service)

  • GOOGLE_PRIVATE_KEY: Key generated when creating Google User Service

  • GOOGLE_FOLDER_ID: Folder to store backup files. Note: This folder must be shared with the Google Client Email.

  • IS_ALLOW_SEND_TELEGRAM_MESSAGE: Enable sending messages via Telegram. Default: 1

  • TELEGRAM_CHANEL_ID: Configure to send backup notifications via Telegram

  • TELEGRAM_BOT_TOKEN

  • TELEGRAM_MESSAGE_LEVELS: Configure message types to be sent via Telegram. Default: "info error"

  • TELEGRAM_PREFIX: Custom prefix for messages sent via Telegram. Default: MongoDB Backup


Sample Message:

21/11/2020, 6:43:34 PM, Vietnam
✅ MongoDB Backup
Backup database to GG Drive with file name: 2020-11-21.zip successfully!

Note: Telegram messages only work in three environments: Development, Staging, Production. You need to provide both TELEGRAM_CHANEL_ID and TELEGRAM_BOT_TOKEN for the system to send messages via Telegram.

Comments

bottom of page