Companies are struggling to move operational data into data warehouses and lakes, as the proliferation of streaming systems and tools creates operational bottlenecks, according to a new study.
The research, commissioned by Conduktor, a provider of intelligent data hubs for streaming and AI, surveyed 200 senior IT and data executives at large companies with revenues of $50 million or more. Respondents identified the top challenges as maintaining reliable infrastructure, protecting sensitive information, and synchronising multiple data sources.
Many organisations now rely on data streaming to support both operational systems and artificial intelligence, but the variety of tools used to ingest data is creating significant headaches. Among those surveyed: 73% reported building custom pipelines with Spark or Flink; 69% used Kafka Connect or similar platforms; half relied on fully managed services such as Firehose or Snowpipe; and 28% used ELT or ETL tools such as Fivetran or Airbyte.
Executives cited time efficiency, data complexity caused by schema changes, and managing parallel architectures as the biggest operational pain points.
Respondents reported using a wide array of data lakes and warehouses, notably Amazon S3 or Lake Formation, Databricks Delta Lake, Google Cloud Platform, Google BigQuery, Amazon Redshift, Azure Synapse Analytics, and IBM Db2 Warehouse.
Nicolas Orban, CEO of Conduktor, said: “As data streaming adoption grows, especially for AI, organisations need to prioritise governance. Using multiple lakes and tools with different governance models and schema formats can create chaos, leading to missed signals, duplicated work, and poor decisions.”
The market for streaming data processing software is expanding rapidly. Dataintelo estimates it was valued at $9.5 billion in 2023 and projects growth to $23.8 billion by 2032, driven by the surge in real-time data from social media, IoT devices, and enterprise systems.
Conduktor says its platform helps organisations unify operational data, improving IT productivity while ensuring visibility, governance, and control.
