There is something strange about December, I don't know why but I always seem to start personal projects this month.
This year is no different, my latest project is called micro-stat and I wanted to use this post to introduce it and talk about why I think it is needed.
Before I proceed, micro-stat is not a finished product, it is a prototype under active development so I am sure it will evolve over the coming weeks and months. It isn't ready to try out in your own projects yet.
What is micro-stat?
micro-stat is a project which aims to make collecting and using statistics far simpler. The "micro" in the name makes it obvious that these should be small / low level metrics.
https://www.micro-stat.com (Under development)
You can integrate micro-stat in a web based project using our NPM SDK. There is scope to add integrators for other technologies in future too.
With the NPM SDK it is really easy to collect custom statistics with just a few lines of code. There are a myriad of different statistic types; all which have a simple to use interface that allow you to work with them easily.
import { Counter } from 'micro-stat'
const basketItemsStatistic = new Counter('Basket Items Added')
basketItemsStatistic.increment()
basketItemsStatistic.publish()
The SDK publishes statistics in real time where they become viewable in the micro-stat portal with multi project/tenant support.
The portal shows all of your metrics and provides drill downs and comparisons to get insight into how your application is being used and how well it is functioning.
The aim is to create different projections (visualisations) for different types of base metrics. Be it a Timer, a HitCounter or more. The page for each metric can show relevant information in easily digestible ways and allow for direct comparisons.
It is also important to me that you can see all of the data on a mobile device through our installable PWA too - which isn't typically available with other similar platforms
Problem: Why is micro-stat needed?
Collecting statistics should be a core part of building a website or app; its how you know which features are used, how often and how they are performing.
Being able to track statistics over time shows if you are making your customer experiences better or worse, allowing you to make focused improvements to poorly performing code, or focus on adding features you know customers will actually use.
The problem with this is that it isn't easy to do, it can cost a lot of money and it can be very challenging to integrate with and scale.
Lets take a look at some of the options you might use today...
Vendor options
You may choose to use the tools available in your selected cloud like Application Insights in Azure or Cloud Watch in AWS. Both options are advanced but complex tools that often require vendor lock in.
They do a lot beyond simply statistics; so trying to use them for targeted use-cases is challenging. You also need to do a lot of leg work to visualise your metrics in a usable way, which requires a fair bit of practice and experience to do right.
They also collect way too much information, people are starting to demand privacy, so sending far more data than you need (particularly identifiable stuff) isn't going to be a popular approach
Open source options
You can also use options like Prometheus or StatsD, but again these are hugely complex and costly to operate. With these options you may host them yourself or pay to have them provided as a service.
Beyond just the complexity and cost though, they tend to be very painful to work with when building modern applications. For example in a PWA where most of the application functions on the client side and not on a web server it is very challenging to get your statistics ingested.
When using Prometheus for example you would need to add more infrastructure like Push Gateways to communicate from the edge, this adds even more cost and complexity.
Synthetic monitoring
Synthetic monitoring tools aren't really a direct comparison to the above options, but I have observed many people using synthetic monitoring to try and learn how their application is performing.
The problem is synthetic monitoring is not representative of what real users are experiencing. It has a place within operation stacks but statistics in this vein should be focused around real user metrics which synthetic tooling cannot capture.
Solution: What is the aim of micro-stat
The question I ask myself repeatedly is why is it so complicated to collect simple learnings and act on them, so just as Sentry has vastly improved the way we capture and visualise logs and errors for modern applications I think there is a place for somebody to simplify the collection of statistics too - maybe this, maybe not.
I want micro-stat to make statistics capture easier and cheaper, so you are encouraged to capture small, simple pieces of anonymous information that you can use to make your application better over time.
Simplicity really is at the core of this, there should be no need to do tons of up front configuration, instead when you want to capture a new metric you simply name it in your code and publish it through the integrator and within seconds you can see the metric in the portal.
micro-stat is not designed for complex holistic implementations that cover every facet of an application, but focus around the key pieces that can really impact the customers experience.
What is the technology stack of micro-stat
Simply put, the most ambitious to date for my personal projects. A project like this needs several pieces, so I have put them together in the best way to achieve low cost and high scalability. This is what it looks like right now
- Integrations - Like the NPM package, which allows easy incorporation into an application
- Azure Serverless Ingestion Function - An endpoint that ingests statistics into our data store. Written in NodeJS our integrations communicate with this over HTTPS.
- Azure Serverless Projection Workers - Workers that regularly output projections from raw statistics, to show graphs quickly and scalably over huge data sets. Written in NodeJS and run on a schedule.
- Data Storage - where we store raw statistics and projections, we use CosmosDB for great scalability and ease of querying.
- Portal Microsite - The authenticated part where you sign in to see your metrics, written in Nuxt, VueJS and hosted on Netlify as a JamStack app
- Portal API - A restful API responsible for serving data to the portal microsite - written in NodeJS using Express
- Marketing Microsite - The bit that you will see when you visit micro-stat for the first time to learn more about it. Written in Nuxt, VueJS and hosted on Netlify as a JamStack app
- Help Centre Microsite - Another Jamstack app to help with questions, feedback and more.
Example use cases
There are lots, but these are the ones I am focused around at the moment. They sometimes cross the boundary of a pure statistic into a form of analytics. But the key is making it possible to use it for any purpose.
- How often is a feature used - Count the frequency with which it is used and compare it to the number of total users
- How quickly does a feature take to load - Time how long it takes for a customer to interact with a feature and see if it truly performs as quickly as you would hope
- How fast are our API's - Time how long actual users spend waiting for your API's to respond
- How long does a customer spend using a feature - Can we reduce the time it takes to complete an activity?
- Business metrics - like how many items does your customer add to their basket, how many end up checking out?
What happens next?
I am working towards an MVP, when I get there I will open it up to see if anybody wants to try it out and provide me with feedback on their experience.
I think a project like this could be a long term project if it gains traction, constantly adding new integration points and clever new ways of projecting your statistics to make them easier to digest.
I have lots of ideas, now I just need to keep making progress on the build.
If you have any questions, comments, suggestions or damning indictments please post a comment below!
If you want to try out the early prototype so far to see how it looks, check out https://www.portal.micro-stat.com (No auth/login yet)
Top comments (0)