Top 10 Common Data Engineers and Scientists Pain Points in 2024

As we navigate through 2024, the landscape of data engineering and science continues to evolve at a breakneck pace. With advancements in AI technology come new challenges, and professionals in these fields are grappling with a unique set of challenges. Nowadays, the integration of AI and machine learning models into applications requires real-time data processing. Let's explore the top 10 challenges that data engineers and scientists face in their workflow with the integration of real-time data.

For Data Scientists

1. Java-based Tools

   Data scientists often prefer Python for its simplicity and powerful libraries like Pandas or SciPy. However, many real-time data processing tools are Java-based. Take the example of Kafka, Flink, or Spark streaming. While these tools have their Python API/wrapper libraries, they introduce increased latency, and data scientists need to manage dependencies for both Python and JVM environments. For example, implementing a real-time anomaly detection model in Kafka Streams would require translating Python code into Java, slowing down pipeline performance, and requiring a complex initial setup.

2. Data Integration

   Integrating data from multiple sources and formats for analysis is challenging. Think about combining streaming data from IoT devices with historical data stored in different formats (e.g., CSV, SQL databases like tabular format). This process complicates the workflow., requiring custom connectors or scripts to understand data sources through profiling, the creation of data mapping, and transformation rules.

3. Offline ML Pipeline

   Building an offline ML pipeline for experimentation, model reproduction, and local debugging presents significant struggles. Experimenting with different feature engineering techniques on a dataset stored in distributed file systems can be difficult to replicate locally.

4. Insight Delays

   Translating complex data transformations from Python to JVM languages for real-time processing can introduce latency. For instance, converting a Pandas DataFrame operation into a PyFlink Table operation might delay the delivery of insights.

5. Batch Processing Mindset

   Data scientists got used to defining and executing jobs all at once, like in batch processing. They struggle to adapt to event-driven models, where data is processed as it arrives. This shift requires rethinking data pipeline design, which can be challenging without proper tools or guidance.

6. Software Engineering Practices

   The unfamiliarity with software engineering best practices complicates the integration of ML models into application codebases. Integrating a machine learning model into a production-grade microservices architecture requires knowledge of containerization and orchestration tools like Docker and Kubernetes, which many data scientists find daunting.

7. Infrastructure Management

   Setting up and managing a Kubernetes cluster for deploying a TensorFlow model serving API requires operational knowledge that data scientists might not have, diverting their focus from data analysis.

8. Scalability Issues: Automatic scaling for data transformation with increasing volumes or complexity is not supported by the tools they currently use.

9. Prototype vs. Production

   Mirroring the production environment when building prototypes is challenging with the tools available to data scientists. For example, developing an ML model in a Jupyter Notebook with a big subset of data is not straightforward.

10. Evolving Data Patterns: Real-time data streams often exhibit non-stationary behavior, where data distributions and relationships between variables change over time. Models trained on a specific snapshot of data may perform well initially but can quickly become overfitted as they fail to generalize to new patterns, leading to decreased accuracy in predictions.

For Data Engineers

1. Dependency on Other Teams Data engineers often depend on other teams to maintain data infrastructures. Sometimes data engineers need to ask DevOps assistance to provision cloud resources for deploying a new data pipeline creating delays. For example, waiting for the necessary cloud permissions to launch an Apache Airflow instance can slow down project timelines.

2. Java-based Stateful Processing

   Implementing stateful computations in Kafka Streams for analysis requires Java expertise from engineering teams. As a result, analytics projects with short deadlines are often delayed.

3. Event-driven Architecture

   Transitioning from batch processing to event-driven architecture means rearchitecting the entire data pipeline, which comes with high costs, complexity, and maintenance challenges.

4. Operational Overheads

   The need to hire Kafka specialists just to maintain the messaging infrastructure for a real-time logistics tracking system significantly increases budgets for data teams.

5. Access and Sharing Barriers

   Encountering barriers that prevent effective access to or sharing of data is a major concern. For example, data engineers facing restrictions in accessing sales data stored in Salesforce due to API rate limits or security policies can slow down the development of integrated analytics solutions.

6. Insufficient Resources

   Early startups or even midsize companies might lack sufficient resources, including infrastructure, tools, and support, which makes harder to design, build, and maintain effective data pipelines. Implementing a scalable data lake on AWS without adequate budget or expertise can lead to suboptimal configurations, affecting performance and cost.

7. Poor Data Quality

   Ensuring high data quality remains a persistent challenge. Upstream data quality issues prevent data engineers from efficiently and reliably delivering quality data to their consumers. For example, real-time ingestion of user-generated content into a data warehouse like Snowflake without proper validation or cleaning mechanisms can lead to inaccurate analytics.

8. Legacy Systems

   Migrating a legacy SQL-based reporting system to a modern, real-time dashboard requires overcoming significant technical debt and compatibility issues, limiting agility and innovation.

9. Batch and Stream processing separation

   Maintaining two separate pipelines for batch processing and real-time streaming. Separate teams might develop different conventions and standards for handling data, leading to inconsistencies that can affect data quality and complicate data integration efforts.

10. Querying real-time data with SQL: Engineers and scientists must navigate these hurdles to extract timely insights from continuously updating data sources, often requiring advanced techniques or additional tools like streaming databases to bridge the gap effectively.

There are more common challenges for both data engineers and scientists in building and maintaining streaming data pipelines. One common pain point for many organizations is being slow to discover any upstream data issues flowing through their data warehouse. Another common issue is that many real-time data transformation tools require you to create and keep a self-hosted CI/CD (Continuous Integration/Continuous Deployment) pipeline. It’s difficult to develop and test data pipelines locally, deploy them, and keep them updated over time when technological changes frequently introduce complications.

How GlassFlow helps?

GlassFlow offers serverless real-time data transformation in Python and addresses several of these pain points by simplifying data processing workflows and reducing operational overhead. With the serverless infrastructure—everything is configured in GlassFlow and you run data transformation logic in your data warehouse without moving your data.

GlassFlow can connect with whatever real-time data platform or database you’re using, and it provides a framework to develop data pipelines, test them, and then deploy them in minutes so that the resulting data is useful to the organization for decision-making. By staying ahead of these challenges, data professionals can unlock the full potential of their data, driving innovation and creating value for their organizations.

Next

Read more about what is GlassFlow for and use cases.

About the author

Visit my blog: www.iambobur.com