Kibana

Kibana

DevOps / Monitoring / Monitoring Tools
Data Scientist ·

I recently started a new position as a data scientist at an E-commerce company. The company is founded about 4-5 years ago and is new to many data-related areas. Specifically, I'm their first data science employee. So I have to take care of both data analysis tasks as well as bringing new technologies to the company.

  1. They have used Elasticsearch (and Kibana) to have reporting dashboards on their daily purchases and users interactions on their e-commerce website.

  2. They also use the Oracle database system to keep records of their daily turnovers and lists of their current products, clients, and sellers lists.

  3. They use Data-Warehouse with cockpit 10 for generating reports on different aspects of their business including number 2 in this list.

At the moment, I grab batches of data from their system to perform predictive analytics from data science perspectives. In some cases, I use a static form of data such as monthly turnover, client values, and high-demand products, and run my predictive analysis using Python (VS code). Also, I use Google Datastudio or Google Sheets to present my findings. In other cases, I try to do time-series analysis using offline batches of data extracted from Elastic Search to do user recommendations and user personalization.

I really want to use modern data science tools such as Apache Spark, Google BigQuery, AWS, Azure, or others where they really fit. I think these tools can improve my performance as a data scientist and can provide more continuous analytics of their business interactions. But honestly, I'm not sure where each tool is needed and what part of their system should be replaced by or combined with the current state of technology to improve productivity from the above perspectives.

READ MORE
5 upvotes·332.4K views
Replies (2)
Data Scientist / Data Engineer at The Prosperity Company AG·

It's hard to make a suggestion here as your use case isn't clear enough.

Use BigQuery if you want to replicate your probably on premise Oracle and Elasticsearch databases so you can profit from the speed of BigQuery. You can do the replication via Google Cloud Functions. Your Google Sheets can be connected to BigQuery and BigQuery can easily be connected to DataStudio.

If you do data science on the data there would be BigQuery ML and Google Colab that would fit into your stack.

In case you do BigData analysis you can go with Apache Spark if you have enough resources (on-prem or Cloud). I suggest you to use a Kubernetes backbone for this as you only reserve the resources when in use and the cluster can be used for other stuff as well.

For dashboarding find your preference and the preference of your audience with DataStudio, Tableau or Apache Superset

READ MORE
9 upvotes·2 comments·5.4K views
Bob hs
Bob hs
·
March 16th 2022 at 12:09PM

Thank you for the answer.

A few days ago the head of IT told me to try AWS if I need cloud resources. I cannot migrate everything from on-premise to cloud. But, I need to choose what data I need for my Data Science tasks on the cloud. For example, I need to extract their daily sale records stored in Oracle as well as their web usage from Elasticsearch.

My main tasks would be "sale/demand forecast", "user retention prediction", "recommendation systems", and "user activity analysis". So, BigData analysis would be part of the job.

I think BigQuery and Datastudio would be out of my options. I need to use resources offered by AWS or compatible with AWS. I'm not sure if I need to grab their web data directly from their web platform's server or from Elasticsearch.

Also, what dashboarding tool is better when I use AWS for my DS pipeline?

·
Reply
peol solutions
peol solutions
·
January 8th 2024 at 1:36PM

BigQuery and Datastudio would be out of my options. I need to use resources offered by AWS or compatible with AWS. I'm not sure if I need to grab their web data directly from their web platform's server or from Elasticsearch.

·
Reply

A trending feed showcases the most popular topics, hashtags, or posts on social media platforms in real time. These feeds are driven by algorithms that analyze user engagement, such as likes, shares, and comments, to identify what resonates with audiences. Trending feeds serve as a valuable tool for users and marketers, offering insights into current events, pop culture, and consumer interests. Engaging with trending topics can enhance visibility and drive traffic. For a deeper understanding, check resources from platforms like Hootsuite and Sprout Social, which provide insights into leveraging trending feeds effectively.

READ MORE
1 upvote·50 views
Needs advice
on
KibanaKibana
and
New RelicNew Relic

I need to choose a monitoring tool for my project, but currently, my application doesn't have much load or many users. My application is not generating GBs of data. We don't want to send the user information to New Relic because it's a 3rd party tool. And we can deploy Kibana locally on our server. What should I use, Kibana or New Relic?

READ MORE
6 upvotes·136.9K views
Replies (2)
Software Developer ·
Recommends
on
New Relic

Kibana and ELK stack is way far better in enterprise solution. But if you are going to deploy something smal, it does't worth the configuration and maintenance of the ELK stack. You'll have lots of challenges every day. If you have a small team, I do not recommend on-promiss ELK. You can also consider ELK hosted services which are very easier to use, like logz.io

READ MORE
3 upvotes·14.8K views
Recommends
on
New Relic

New Relic's value to me is the ability to see how end users perceive the application. Kibana is going to be limited to what is sent to it. The value to larger companies is paying New Relic to package up knowledge on what are typical trigger values. If you your scope is small, not a global website for example, and your key outage risks are local events then Kibana would be a low cost solution but you may be the sole provider of configuration logic.

READ MORE
2 upvotes·14.7K views