CouchBase Mirror Data Records

Hi everyone, recently, I have been worked on the couchbase with a request of data mirror from one couchbase database to another.

Basically, what I have done can be summarised as: we try to export original data and use a NodeJS script to generate a query for all the original data and finally run the query to inject all data from original data (One word: Using query to copy and paste data)

Here is the query generation script:

// This script can only be suitable for the data records which is less than 2000;

const fs = require("fs");

function convert(collectionName, sourceFile, query, resultFile) {
  fs.readFile(sourceFile, "utf8", (err, data) => {
    if (err) {
      console.log("Source file read failed:", err);
      return;
    }

    const json = JSON.parse(data);

    const response = json.reduce((acc, cur, index) => {
      const key = cur[collectionName].id.toString();
      const value = cur[collectionName];
      const valueString = JSON.stringify(value);

      cur = ('("' + key + '", ' + valueString + ")").replace(/\\/g, "");

      comma = index === json.length - 1 ? "" : ", ";

      acc += cur + comma;

      return acc;
    }, "");

    if (response) {
      const final = `${query} ${response}`;
      fs.writeFile(resultFile, final, function (err) {
        if (err) return console.log(err);
        console.log(`Generated data have been saved into ${resultFile}`);
      });
    }
  });
}

Here is the example of how to do it:

Step 1: we need to go to the original data source we want to copy from (The couchbase bucket -> collection -> document).

Just provide an example here, if I want to mirror data for test-bucket (bucket name) -> test-collection (collection name) -> test-documents (document name)

Step 2: we click the Query tab on the left side menu bar

Step 3: we run a query: SELECT * FROM test-bucket.test-collection.test-documents

Step 4: we select EXPORT button on the top right corner and you will see a JSON file export modal window:

Step 5: we name the file name as test-documents

Then Click Save button to export data as JSON file

Step 6: You can open the convert.js script I have attached below and you will see this setup from line 39 to line 45 below:

convert(
  "test-documents", // the collection name
  "./test-documents.json", // the file you have exported from CouchBase (Step 5)
  "INSERT INTO `test-bucket`.`test-collection`.`test-documents`(KEY, VALUE) VALUES ", // the target bucket -> collection -> document 
  "test-documents-query.txt" // the final generated result query for data mirror / injection usage 

);

Step 7: Modify the script provided in above (Step 6) based on your specific requirements.

As an example here, we want to copy the one couchbase database test-documents (document name) data to another couchbase database test-documents (document name), we can run above script for generating the data injection /mirror query.

run

node convert.js

command in your terminal within the current folder path and you will be able to generate the new query text file called as test-documents-query.txt.

Step 8: We go to the destination / target document (eg: test-documents) and go to the Query tab (Same as Step 2) and paste the new generated query inside the Query Editor

Step 9 [Optional]: If you just created a new document for UK couchBase, you need to run this query first to avoid some index errors which will be reported by couchbase

CREATE PRIMARY INDEX ON `default`:`test-bucket`.`test-collection`.`test-documents`

*** If you already have document primary index setup already, you can skip Step 9. ***

Step 10: You will open the final generated data injection text file: test-documents-query.txt and copy all the contents and paste into the Query Editor

Step 11: Click the Execute button and starting injecting the data from one test-documents to another 🚀🚀

Some Additional Tips:

• Before mirror the data, better to try to use the script locally (localhost:3000), so you will be able to get familiar with and understand this script before actually use it

• Always ensure the bucket name, collection name and document name are entered correctly. Otherwise, you might inject the data in a wrong document

Finally, as a reference,

You might be able to find the work I have been working on:
https://github.com/DamengRandom/n1ql-query-generator

Thank you very much for reading, hopefully it could help a little with your couchbase data mirror process.