Hi, I normally contract in MSBI, Oracle, .Net/.Net Core, focusing on a property platform at the moment. Have also been working hard on upgrading my limited company website too.
Hmmm. I have to say, I don't like the sound of this approach unless you really need to use Kibana and Elastic Search? Unless everything needs to be real time. We already know the answer, it can't be when you are doing daily, monthly, and yearly reporting?
I happen to be writing a data publication solution for my property platform right now, which does the following;
Open a library namespace.
Iterate through reflection to return each classes' IEnumerable. (This is not expensive, so please don't say how terrible reflection is). This means I don't care what the data source is (SQL/Oracle/MongoDB)
Save them as JSON.
Publish via FTP.
Send a notification to an end point.
The end point will then invoke loading the publications into cache.
Application/Service end points then have that data available.
We could change these steps, not use FTP etc BUT...
I would rarely take the above approach if I was doing this in an enterprise. I would simply build reports in SSRS and call the report server web service with a request stream. In the above example, I am developing software for a much more specific requirement than described but it is too bespoke for standard staff to support.
I would build a proper data warehouse or reporting database.
It always concerning vendor data can change time to time. This provides a risk of changing grain, meaning the more code you write to produce reports, the more risk there is in terms of maintenance.
My feeling is you need to be very careful not to try and build the solution you are describing just because it satisfies a high degree of technical expertise. There needs to be a genuine reason. Now, nothing is stopping you building a data warehouse or using SSRS to publish to a location where the data can end up in Elastic Search but be careful.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Hmmm. I have to say, I don't like the sound of this approach unless you really need to use Kibana and Elastic Search? Unless everything needs to be real time. We already know the answer, it can't be when you are doing daily, monthly, and yearly reporting?
I happen to be writing a data publication solution for my property platform right now, which does the following;
We could change these steps, not use FTP etc BUT...
I would rarely take the above approach if I was doing this in an enterprise. I would simply build reports in SSRS and call the report server web service with a request stream. In the above example, I am developing software for a much more specific requirement than described but it is too bespoke for standard staff to support.
I would build a proper data warehouse or reporting database.
It always concerning vendor data can change time to time. This provides a risk of changing grain, meaning the more code you write to produce reports, the more risk there is in terms of maintenance.
My feeling is you need to be very careful not to try and build the solution you are describing just because it satisfies a high degree of technical expertise. There needs to be a genuine reason. Now, nothing is stopping you building a data warehouse or using SSRS to publish to a location where the data can end up in Elastic Search but be careful.