Predicting House Prices with XGBoost in Node.js

What Is XGBoost?

XGBoost is popular machine learning algorithm that regularly places high in Kaggle and other data science competitions. What sets XGBoost apart is its ability to combine multiple weak models (in this case, decision trees) into a strong one. This is done through a technique called gradient boosting, which helps make the algorithm robust and highly effective for a wide variety of predictive tasks.

How Does XGBoost Work?

XGBoost uses gradient boosting, which means it builds trees sequentially where each tree tries to correct the mistakes of the previous trees. Here's a simplified view of the process:

1. Make an initial prediction (could be the average of all target values)
2. Calculate how wrong this prediction was (the error)
3. Build a decision tree to predict this error
4. Add this tree's predictions to our running prediction total (but scaled down to prevent overconfidence)
5. Repeat steps 2-4 many times

For example, if we're predicting house prices:

• First tree might predict $200,000
• If actual price is $250,000, the error is $50,000
• Next tree focuses on predicting this $50,000 error
• Final prediction combines all trees' predictions

This process, combined with some clever mathematics and optimizations, makes XGBoost both accurate and fast.

Why XGBoost in Node.js?

While XGBoost is originally implemented as a C++ library, there are bindings available for languages like Python and R, making it accessible to a wide range of developers who typically specialize in data and machine learning.

I recently had a project that had a hard requirement for Node.js, so I saw an opportunity to bridge the gap by writing bindings for Node.js. I hope this helps open up the door to more ML for JavaScript developers.

In this article, we'll take a closer look at how to use XGBoost in your Node.js applications.

Prerequisites

Before getting started, ensure you have:

• Linux operating system (current requirement for xgboost_node)
• Node.js version 18.0.0 or higher
• Basic understanding of machine learning concepts

Installation

Install the XGBoost Node.js bindings using npm:

npm install xgboost_node

Understanding the Data

Before jumping into the code, let's understand what our features represent in the house price prediction example:

// Each feature array represents:
[square_feet, property_age, total_rooms, has_parking, neighborhood_type, is_furnished]

// Example:
[1200,       8,            10,           0,           1,                1        ]

Here's what each feature means:

• square_feet: The size of the property (e.g., 1200 sq ft)
• property_age: Age of the property in years (e.g., 8 years)
• total_rooms: Total number of rooms (e.g., 10 rooms)
• has_parking: Binary (0 = no parking, 1 = has parking)
• neighborhood_type: Category (1 = residential, 2 = commercial area)
• is_furnished: Binary (0 = unfurnished, 1 = furnished)

And the corresponding labels array contains house prices in thousands (e.g., 250 means $250,000).

Transforming Your Data

If you have raw data in a different format, here's how to transform it for XGBoost:

// Let's say you have data in this format:
const rawHouses = [
    {
        address: "123 Main St",
        sqft: 1200,
        yearBuilt: 2015,
        rooms: 10,
        parking: "Yes",
        neighborhood: "Residential",
        furnished: true,
        price: 250000
    },
    // ... more houses
];

// Transform it to XGBoost format:
const features = rawHouses.map(house => [
    house.sqft,
    new Date().getFullYear() - house.yearBuilt,  // Convert year built to age
    house.rooms,
    house.parking === "Yes" ? 1 : 0,             // Convert Yes/No to 1/0
    house.neighborhood === "Residential" ? 1 : 2, // Convert category to number
    house.furnished ? 1 : 0                       // Convert boolean to 1/0
]);

const labels = rawHouses.map(house => house.price / 1000); // Convert price to thousands

Training Your First Model

Here's a complete example that shows how to train a model and make predictions:

import xgboost from 'xgboost_node';

async function test() {
    const features = [
        [1200, 8, 10, 0, 1, 1],
        [800, 14, 15, 1, 2, 0],
        [1200, 8, 10, 0, 1, 1],
        [1200, 8, 10, 0, 1, 1],
        [1200, 8, 10, 0, 1, 1],
        [800, 14, 15, 1, 2, 0],
        [1200, 8, 10, 0, 1, 1],
        [1200, 8, 10, 0, 1, 1],
    ];
    const labels = [250, 180, 250, 180, 250, 180, 250, 180];

    const params = {
        max_depth: 3,
        eta: 0.3,
        objective: 'reg:squarederror',
        eval_metric: 'rmse',
        nthread: 4,
        num_round: 100,
        min_child_weight: 1,
        subsample: 0.8,
        colsample_bytree: 0.8,
    };

    try {
        await xgboost.train(features, labels, params);
        const predictions = await xgboost.predict([[1000, 0, 1, 0, 1, 1], [800, 0, 1, 0, 1, 1]]);
        console.log('Predicted value:', predictions[0]);
    } catch (error) {
        console.error('Error:', error);
    }
}

test();

The example above shows how to:

1. Set up training data with features and labels
2. Configure XGBoost parameters for training
3. Train the model
4. Make predictions on new data

Model Management

XGBoost provides straightforward methods for saving and loading models:

// Save model after training
await xgboost.saveModel('model.xgb');

// Load model for predictions
await xgboost.loadModel('model.xgb');

Further Considerations

You may have noticed there are parameters for this model. I would advise looking into XGBoost documentation to understand how to tune and choose your parameters. Here's what some of these parameters are trying to achieve:

const params = {
    max_depth: 3,              // Controls how deep each tree can grow
    eta: 0.3,                 // Learning rate - how much we adjust for each tree
    objective: 'reg:squarederror',  // For regression problems
    eval_metric: 'rmse',      // How we measure prediction errors
    nthread: 4,               // Number of parallel processing threads
    num_round: 100,           // Number of trees to build
    min_child_weight: 1,      // Minimum amount of data in a leaf
    subsample: 0.8,           // Fraction of data to use in each tree
    colsample_bytree: 0.8,    // Fraction of features to consider for each tree
};

These parameters significantly impact your model's performance and behavior. For example:

• Lower max_depth helps prevent overfitting but might underfit if too low
• Lower eta means slower learning but can lead to better generalization
• Higher num_round means more trees, which can improve accuracy but increases training time

Conclusion

This guide provides a starting point for using XGBoost in Node.js. For production use, I recommend:

1. Understanding and tuning the XGBoost parameters for your specific use case
2. Implementing proper cross-validation to evaluate your model
3. Testing with different data scenarios to ensure robustness
4. Monitoring model performance in production

Jonathan Farrow

@farrow_jonny