DEV Community

ardsh
ardsh

Posted on • Updated on

Solving the overfetching problem with tRPC when rendering tables (Pt. 1)

A common issue when using tRPC over GraphQL is the fact that overfetching is supposed to be solved by GraphQL, but it's not clear how you can do the same thing with tRPC.

I'm gonna try to show a pattern of how to create react tables with optional columns, when not all columns are supposed to be visible to the user.

In that case, we want to fetch only the data for the columns that we display, and not fetch all the data a tRPC query has to offer.

This is the first part in a series of articles for creating and querying tRPC queries, in efficient, type-safe methods.

Setup

I'll use React Table Library for the UI, but this pattern can work with most other tables or datagrids in a very similar way.

Solution

We can implement selective fetching in our tRPC APIs, using a select array that we pass as an argument. Doing so manually isn't too difficult, but you can also use a package like slonik-trpc that is designed to implement this kind of API automatically.

We're gonna abstract this handling of dependencies (and pagination) in its own generic function like below:

export const createTableLoader = <TPayload extends Record<string, any>>() => {
  // define initial state, context providers, and reducer
  // ...
  return {
    ContextProvider,
    useVariables,
    createColumn,
    useColumns
  }
}
Enter fullscreen mode Exit fullscreen mode

This function takes a generic parameter for the type of data we're trying to load, e.g. an Employee type, and will be responsible for storing the dependencies of all visible columns, as well as making the column declaration type-safe, and any other table-related data fetching responsibilities.

Example implementation

Let's say you're building a table that displays employee information, using this type:

type Employee = {
  id: number;
  firstName: string;
  lastName: string;
  email: string;
  jobTitle: string;
  company: {
    name: string;
  };
}
Enter fullscreen mode Exit fullscreen mode

Then you'd declare the columns like below:

const employeeColumns = employeeLoader.useColumns([{
  label: 'Name',
  dependencies: ["firstName", "lastName"],
  renderCell: (employee) => {
    return <div>{employee.firstName} {employee.lastName}</div>
  },
}, {
  label: 'Email',
  dependencies: ["email"],
  renderCell: (employee) => employee.email,
}, {
  label: 'Title',
  dependencies: ["jobTitle"],
  renderCell: (employee) => employee.jobTitle,
}, {
  label: 'Company',
  dependencies: ["company"],
  renderCell: (employee) => employee.company.name,
}]);
Enter fullscreen mode Exit fullscreen mode

This is enough for our implementation to work, however, if we want to make each renderCell function get an argument with the correct type, we can use the createColumn function to wrap each column, and make it type-safe.

Let's start by declaring the createColumn function

import type { Column } from '@table-library/react-table-library/types/compact';

type ColumnDefinitions<TPayload> = Omit<Column, "renderCell" | "dependencies"> & {
  renderCell: (item: TPayload) => React.ReactNode,
  dependencies?: readonly (Extract<keyof TPayload, string>)[],
};

export const createTableLoader = <TPayload extends Record<string, any>>() => {
  // ...
  return {
    createColumn: <TDependencies extends keyof TPayload=never>(column: Omit<ColumnDefinitions<TPayload>, "dependencies" | "renderCell"> & {
      dependencies?: TDependencies[],
      renderCell: (data: Pick<TPayload, TDependencies>) =>  React.ReactNode,
    }) => {
      return column;
    },
  // ...
  }
}
Enter fullscreen mode Exit fullscreen mode

Whoa, that looks like some complex typescript! What we're essentially doing here though, is restricting the data type, using the Pick utility.

So if you specify only the firstName field in a column, you won't be able to access other fields in the renderCellfunction. E.g. writing employee.email would result in a typescript error.

Now simply wrap all your columns in this helper function. The best part about this is, you can get easy type-safety AND composability, by declaring the columns in different places, then simply adding them to the array you need.

export const nameColumn = employeeLoader.createColumn({
  label: 'Name',
  dependencies: ["firstName", "lastName"],
  renderCell: (employee) => {
    return <div>{employee.firstName} {employee.lastName}</div>
  },
});

export const companyColumn = employeeLoader.createColumn({
  label: 'Company',
  dependencies: ["company"],
  renderCell: (employee) => employee.company.name,
});
// ...

const columns = employeeLoader.useColumns([
  nameColumn,
  employeeColumn,
]);
Enter fullscreen mode Exit fullscreen mode

The useColumns implementation

Now we need to actually implement the useColumns function, and the rest of the table data loader.

We start by saving the dependencies array in a reducer.

import React from 'react';

type Action = {
  type: "APPEND_FIELDS",
  dependencies: string[]
};

type State = {
  dependencies: string[]
}

const stateReducer = (state: State, action: Action) => {
  switch (action.type) {
    case 'APPEND_FIELDS':
      return {
        ...state,
        // Sort alphabetically to have a stable array
        dependencies: [... new Set(state.dependencies.concat(action.dependencies))].sort(),
      };
    default: return state;
  }
}
Enter fullscreen mode Exit fullscreen mode

This is a simple array of unique strings, that are sorted alphabetically (to prevent tRPC refetching queries if the dependent fields are attached out of order).

We're gonna pass this dependencies array down using the context provider.

export const createTableLoader = <TPayload extends Record<string, any>>() => {
  const initialState = {
    dependencies: [],
  };
  const DependenciesContext = React.createContext([] as (keyof TPayload)[]);
  const DispatchContext = React.createContext((() => {
    throw new Error("tableDataLoader Context provider not found!");
  }) as React.Dispatch<Action>);

  return {
    ContextProvider: ({ children }: { children: React.ReactNode }) => {
      const [state, dispatch] = React.useReducer(stateReducer, initialState);
      return (<DispatchContext.Provider value={dispatch}>
        <DependenciesContext.Provider value={state.dependencies}>
          {children}
        </DependenciesContext.Provider>
      </DispatchContext.Provider>)
    },
    // ...
}
Enter fullscreen mode Exit fullscreen mode

The context provider makes it possible to access the state in sub-components.
You simply need to provide it at the root of your page, before you use any loader hooks.

<employeeLoader.ContextProvider>
  <EmployeesTable />
</employeeLoader.ContextProvider>
Enter fullscreen mode Exit fullscreen mode

Finally, we implement the useColumns and useVariables functions

import  type { Column } from  '@table-library/react-table-library/types/compact';

const diff = (arr1: any[], arr2: any[]) => {
  return arr1.filter(x => !arr2.includes(x));
}

type ColumnDefinitions<TPayload> = Omit<Column, "renderCell"> & {
  renderCell: (item: TPayload) => React.ReactNode,
  dependencies?: readonly (Extract<keyof TPayload, string>)[],
};

export const  createTableLoader = <TPayload  extends  Record<string, any>>() => {
  // ... context state
  return {
    ContextProvider: //...
    useColumns: (columns: ColumnDefinitions<TPayload>[]) => {
      const dispatch = React.useContext(DispatchContext);
      const existingDeps = React.useContext(DependenciesContext);
      React.useEffect(() => {
        const select = columns.flatMap(column => {
          return (column.dependencies || [])
        }).filter(Boolean);
        if (diff(dependencies, existingDeps).length) {
          // We only add fields to dependencies array, without removing.
          dispatch({
            type: 'APPEND_FIELDS',
            dependencies: dependencies,
          });
        }
      }, [existingDeps, columns]);
      return columns;
    },
    useVariables: () => {
      const dependencies = React.useContext(DependenciesContext);
      return React.useMemo(() => ({
        select: dependencies,
      }), [dependencies]);
    },
  }
}
Enter fullscreen mode Exit fullscreen mode

You can use the useVariables hook to get the select array while fetching the data with trpc, and pass it as an argument.

const employeeLoader = createTableLoader<Employee>();

// ...

const { select } = employeeLoader.useVariables();
const { data, isLoading } = trpc.employees.getEmployees.useQuery({
  select,
});
// ...
return <Table data={data.nodes} />
Enter fullscreen mode Exit fullscreen mode

So the API can return just the fields that we actually need to display our columns.

The data loader can be extended to also handle pagination, and other API specifics. Stay tuned for the next articles in this series, about pagination and actually implementing such an API in a tRPC server!

The complete tableDataLoader implementation

Finally, here's the complete implementation of the table data loader function (so far in the series:

import type { Column } from '@table-library/react-table-library/types/compact';
import React from 'react';

const diff = (arr1: any[], arr2: any[]) => {
  return arr1.filter(x => !arr2.includes(x));
}

type Action = {
  type: "APPEND_FIELDS",
  dependencies: string[]
};

type ColumnDefinitions<TPayload> = Omit<Column, "renderCell" | "dependencies"> & {
  renderCell: (item: TPayload) => React.ReactNode,
  dependencies?: readonly (Extract<keyof TPayload, string>)[],
};

type State = {
  dependencies: string[]
}

const stateReducer = (state: State, action: Action) => {
  switch (action.type) {
    case 'APPEND_FIELDS':
      return {
        ...state,
        // Sort alphabetically to have a stable array
        dependencies: [... new Set(state.dependencies.concat(action.dependencies))].sort(),
      };
    default: return state;
  }
}

export const  createTableLoader = <TPayload  extends  Record<string, any>>() => {
  const initialState = {
    dependencies: [],
  };
  const DependenciesContext = React.createContext([] as (keyof TPayload)[]);
  const DispatchContext = React.createContext((() => {
    throw new Error("tableDataLoader Context provider not found!");
  }) as React.Dispatch<Action>);

  return {
    ContextProvider: ({ children }: { children: React.ReactNode }) => {
      const [state, dispatch] = React.useReducer(stateReducer, initialState);
      return (<DispatchContext.Provider value={dispatch}>
        <DependenciesContext.Provider value={state.dependencies}>
          {children}
        </DependenciesContext.Provider>
      </DispatchContext.Provider>)
    },
    useColumns: (columns: ColumnDefinitions<TPayload>[]) => {
      const dispatch = React.useContext(DispatchContext);
      const existingDeps = React.useContext(DependenciesContext);
      React.useEffect(() => {
        const dependencies = columns.flatMap(column => {
          return (column.dependencies || [])
        }).filter(Boolean);
        if (diff(dependencies, existingDeps).length) {
          // We only add fields to dependencies array, without removing.
          dispatch({
            type: 'APPEND_FIELDS',
            dependencies: dependencies,
          });
        }
      }, [existingDeps, columns]);
      return columns;
    },
    useVariables: () => {
      const dependencies = React.useContext(DependenciesContext);
      return React.useMemo(() => ({
        select: dependencies,
      }), [dependencies]);
    },
    createColumn: <TDependencies extends keyof  TPayload=never>(column: Omit<ColumnDefinitions<TPayload>, "dependencies" | "renderCell"> & {
      dependencies?: TDependencies[],
      renderCell: (data: Pick<TPayload, TDependencies>) =>  React.ReactNode,
    }) => {
      return column;
    },
  }
}
Enter fullscreen mode Exit fullscreen mode

And the table component

import React from 'react';
import { CompactTable } from '@table-library/react-table-library/compact';

import { trpc, type Employee } from '../../utils/trpc';

const employeeTableLoader = createTableLoader<Employee>();

export default function EmployeeList() {
  const employeeColumns = employeeTableLoader.useColumns([
    employeeTableLoader.createColumn({
        label: 'Name',
        dependencies: ["firstName", "lastName"],
        renderCell: (employee) => {
            return <div>{employee.firstName} {employee.lastName}</div>
        },
    }), employeeTableLoader.createColumn({
        label: 'Salary',
        dependencies: ["salary"],
        renderCell: (employee) => employee.salary,
    }), employeeTableLoader.createColumn({
        label: 'Start Date',
        dependencies: ["startDate"],
        renderCell: (employee) => employee.startDate,
    }), employeeTableLoader.createColumn({
        label: 'Company',
        dependencies: ["company"],
        renderCell: (employee) => employee.company,
    })]);

  const pagination = employeeTableLoader.useVariables();

    const { data, isLoading } = trpc.employees.getPaginated.useQuery(
        {
            take: 100,
            ...pagination,
        }
    );


  if (!data) return null;
  const { nodes, pageInfo } = data;

  return (
    <>
      <CompactTable columns={employeeColumns} data={{
        nodes: nodes || [],
        pageInfo: pageInfo,
      }} />
    </>
  );
}

Enter fullscreen mode Exit fullscreen mode

Oldest comments (0)