Overview
A Full-Stack Developer is someone who does both client-side and server-side development for mobile and web applications. In the data domain, a full-stack engineer performs similar kinds of job duties that involve full ownership, like ingestion, processing, and publishing data. Full stack data engineering would require data analysis, SQL scripting, ETL/ELT programming, data model design, workload orchestration, visualization, etc .
In some cases, a cloud full-stack data analytics engineer will have more responsibilities like provisioning data infrastructure, dataops, devops, etc. Full stack development and full stack data analytics are two different domains, and the only layer of intersection is the datastore. The backend of a full-stack web or mobile application can be a source or a consumer of the data analytics application. But there is another layer in the full-stack data analytics architecture, the modern visualisation stack, that needs full-stack dev engineering.
In this first part of the article, we will see the need for full stack development in the data analytics architecture and how it can be developed and owned within the data analytics application in a serverless manner.
Embedded Analytics
The industry's popular low code/no code visualisation SaaS products are built using full stack technologies like GraphQL, Java Script, etc. But these products still require training about their inbuilt modules and, often, they are not fully customizable. The metrics that are visualised in the dashboards are the output of the heavy lifting done in the data processing layer. GBs of data are processed and finally shared with the semantic layer for building self-service analytics and interactive dashboards. There are cases where visualisation teams need to embed the analytical metrics into the web application/apps for business users. These kinds of metrics are called embedded analytics and often come with minimal filters.
These kinds of implementations need full stack development that requires both client and server side programming using Angular and ReactJS frameworks. For mobile devices, it comes with another set of tech stacks like React Native, etc.
The modern data stack has evolved a lot to have embedded web/mobile analytics. AWS QuickSight is one such example that supports low-code embedded analytics via APIs. SaaS data platforms like Snowflake and Databricks have inbuilt dashboard features but cannot be embedded with mobile/web apps.
Embedded Analytics using AWS Amplify
AWS Amplify lets full-stack developers host their server-less backend for the front end. The Amplify Studio has features to build front-end UI, design app data models with minimal effort,etc. The JavaScript modules that support charts and graphs like D3.JS and Chart.JS can be used along with popular web frameworks like React to build custom dashboards using AWS Amplify.
The periodic data refresh for the back-end data stores can be performed by using continuous data engineering pipelines within the AWS cloud. The payload size plays a pivotal role in providing low latency custom dashboards. So one has to choose the right web framework and data model based on the traverse pattern of front-end analytics dashboards.
Architecture
The diagram below depicts the end-to-end flow for embedding analytics into any business application, product, website, or portal. There is no additional BI tool or licencing involved for an organisation that relies on static dashboards with basic filters.
The data engineering pipeline processes the raw data into aggregates that are suitable for business strategy decisions, measuring KPIs, etc. The processed data is made available for business users both in real time and at batch intervals. AppSync acts as an integration layer to share data with the front-end applications in a real-time manner.
Few of the advantages of using AWS Amplify in Embedded Analytics:
- Faster Time to Market
- Serverless
- Pay as per the usage
- UI based development to build analytics apps
- Option to have own Data Cache layer
- Build cloud native apps with the responsive design
- Support for additional JavaScript modules for the web frameworks
Conclusion
In this article, we have seen embedded analytics usage and how it can be achieved using AWS Amplify. Due to the complexity of the technical stack, full stack engineering is kept as a separate product and is often not owned by the data engineering or the analytics team. AWS Amplify has enough features to simplify the process of prototyping and development of embedded analytical dashboards. Going forward, one can expect wider adoption of AWS Amplify for data analytics use cases. In the next article, I will describe how raw data is enriched, stored, and visualised using AWS Amplify and AppSync.
Top comments (0)