International Business Data & Insights Organization
Our client, a global financial information company, was operating outdated mainframe technology at the heart of its enterprise data supply chain. Onboarding data sources took months rather than days or weeks, and left the company unable to rapidly respond to changes in existing data sources, nor with the agility needed to fulfil new client requirements. And the problem was only getting worse – personnel and processing capabilities simply could not cope as data volume and complexity increased. Expertise in original sources was lost, with no clear understanding of the data path from existing systems to the consumer. This resulted in an inconsistent response to client queries.
A rapidly developed proof of concept demonstrated that a significant number of source records were not making it into the supply chain, and that the number of attributes being propagated for each record was limited.
The solution was to build a cloud native data supply chain platform on AWS to support multiple business lines across the organization. A multi-tier design that would allow for the core framework to support solutions by either business operations, or by engineering teams, according to the complexity of the data and the associated processing requirements. This accelerated the time to market for the majority of new data products.
Technology stack includes AWS, Snowflake, Trifacta, Databricks, EMR,
Airflow, Kubernetes with a custom micro services component-based UI
- New data and therefore new revenue sources were in production and into the hands of the end consumer within six months.
- The flexible, scalable architecture allows for ingestion of hundreds of millions of records at the same time and the ability to share resulting data sets with the rest of the organization without custom integration.
Deep dive experience and expertise knowledge transfer to client technology and operations teams, including significant pairing and scaffolding, has empowered them with the skills and insight necessary to sustain the operation of the system as well as implementing continual enhancements.