Business · Companies · 5 hrs ago
Optus speeds data preparation from days to hours with streaming technology
Australian carrier Optus is pulling data from 400 Oracle tables into Databricks using Confluent and Kafka streaming technology.
The data comes from older customer-facing applications and feeds Optus's analytics and reporting work.
Preparing accurate data for leaders now takes hours instead of days, helping them assess how the business performed over the past 24 hours.
The faster data flow also supports Optus's regulatory and compliance obligations, where accuracy is important.
Optus can also track customer behaviour in its stores near real-time, which could help it offer products based on purchases.
The work is part of a unified data platform built with Microsoft, Databricks and Tech Mahindra, bringing business data into a lakehouse on Azure.
Optus plans to keep developing the platform, while some data will remain outside it and be added when needed.
Optus has cut the time needed to prepare accurate data for its leadership from days to hours using streaming technology.
The carrier uses Confluent’s Oracle CDC and LogMiner with Kafka to move data from Oracle into its Databricks analytics platform in near real time.
The system pulls 400 data tables and supports regulatory reporting and compliance obligations.
Optus also uses the technology to track customer behaviour in its stores in near real time.
The streaming system forms part of a unified data platform built with Microsoft, Databricks and Tech Mahindra.
- Who
- Optus, including its associate director of design and delivery, Anupam Das.
- What
- Optus is using streaming technology to move Oracle data into Databricks, reducing data preparation time from days to hours.
- When
- The technology is in use now. The sources do not state when it began.
- Where
- At Optus, with data moved from Oracle into its Databricks analytics platform.
- Why
- To provide leadership with accurate recent business data and support regulatory and compliance obligations.
This story does not have two clearly opposing sides.
Das said the current speed of data preparation was not possible, and Optus had to wait two or three days.
Optus completed a unified data platform built with Microsoft, Databricks and Tech Mahindra.
Optus is preparing data for leadership within hours using streaming technology.
- Data tables
- 400 Oracle tables are fed into Databricks.
- Preparation time
- Reduced from two or three days to within hours.
- Streaming technology
- Confluent Oracle CDC and LogMiner with Kafka.
- Unified data platform
- Built by Optus with Microsoft, Databricks and Tech Mahindra.
- Destination
- Databricks analytics platform on an Azure data lakehouse.
What people are saying
Now it is within hours
The most important [driver] is the accuracy of the data, so that we can give it to our leadership and say, ‘OK, this is how Optus worked in the last 24 hours’.
We’re driving a high degree of data consolidation, but we're not executing a strategy of one data mart for the entire organisation