Talend, a global leader in data integration and data management, recently hosted its annual user conference Talend Connect with the theme of “Go from data saturated to data driven”.
The event revealed the latest market trends around data from IDC Research and Lopez Research, industry insights from Amazon and Snowflake and customer stories from eBay, Lenovo, and Covanta.
The event kicked off with a session from IDC analyst Tim Crawford, examining data priorities for enterprises. They were revealed as reducing uncertainty by improving data agility; prioritising adaptability with advanced analytics, and enabling efficiency and optimisation.
The three major contributors to a data driven organisation’s data health were identified as access, trust and monitoring. Finding and managing data efficiently, data literacy and governed access underpin widespread trust in data, which is still in many cases lacking. Sadly, there is still no universal standard for measuring data quality. The Talend Trust Score is an industry first and looks to provide data professionals confidence in the data used. To underpin the trust score, 24/7 monitoring of changes made to data is necessary.
Currently, enterprises are focusing on investments in data across two main areas – customer engagement and experience. Challenges abound, such as understanding what the changing customer looks like; the factors driving their decisions; and understanding this detail from the moment a consumer starts thinking about becoming a customer to post sale. CIOs are engaging directly with customers to understand pain points around data culture and literacy, and relevancy throughout the business value chain. In the past, data could be siloed. That really isn’t the case today.
In a session featuring global marketplace eBay, Head of Enterprise Data Services & Business Intelligence, Parani Ghandhi, revealed how the firm had undertaken data fabric modernisation. As a result of acquisitions and divestments over 20 years, and a major data framework update in 2018, there was a need to modernise, especially as there were 600 integrations on three platforms. With a culture of “getting it done”, eBay was able to undertake the entire migration in eight months. Supported by Talend’s data fabric and consultancy services, eBay tackled simpler use cases first, and once consistency and governance was established, migration was rolled out at scale. The new data landscape means that transaction volumes are up, failure rates are down, and $1 million was saved. Without the change, the legacy architecture would become a huge overhead huge, delaying business initiatives.
Cameron Davie, Principal Solutions Engineer at Talend hosted a joint seminar with AWS and Snowflake on maximising cloud modernisation. With barriers to cloud migration still complex, especially around data silos in the cloud, it was welcoming to hear how insurer and underwriter Tokio Marine Kiln undertook a new organisation wide data strategy, with smoother workflows, simplified regulatory compliance and reporting time reduction from 14 to 4 hours as a result.
Agility is key to business success for financial firms, especially with constantly changing rates. With data integration everything is connected and silos smashed. Tamara Ashkatova, Sr Partner Solutions Architect, AWS explained that the new data set up promoted a culture of accessibility and centrally managed data integrity.
Challenges to data modernisation are often around scaling, connectivity, and moving clean and validated data. Ripu Jain, Sr Partner Engineer at Snowflake, reported that cost and lock in are often worrisome. However, data platforms should not be treated as commodities. For instance, AWS collects, transforms and cleans data; while Talend makes it easy to collect, govern, transform, and share the data in a Snowflake Data Cloud. This triad accelerates and maximises data integration, analytics and visualisation, ultimately leading to better decisions, increased efficiencies and driving innovation.
Tech industry analyst Maribel Lopez, the founder of Lopez Research, interviewed Susie Wolff, one of the first women in F1 and founder of Dare to Be Different about how data is being used in racing. Typically, 800 sensors collect information from the track and simulator, and teams of up to 2,000 need to understand which data is most important for performance. Motorsport comes down to the results and numbers, and this is where a deep dive into the data helps to support even the tiniest of amends to gain important seconds in a race.
There is a fine balance, whereby data is used but teams do not get lost in the data. There is where data trust is critical.
Every element should be available to compare against teammates, for example, break pressure during turns. Data in Formula 1 is about identifying marginal gains for the ultimate performance.
Julia Fryk, Data Engineer at Waterstone Mortgage, participated in the “Data chaos into data excellence” session. Despite the value data provides, it can be hard to manage and protect and wreak havoc through data breaches and privacy issues– data has to be protected and well managed.
With data volumes exploding, some firms provide employees with more access than necessary. This is where a balance is necessary to deliver trusted data which is consistent, accurate and high quality, while subject to internal data policies around self-service data access, management and usage. To comply with standards such as GDPR and CCPA, there need to be data quality caps, semantic profiling, a Trust Score, and masking and obfuscation to protect sensitive data.
Waterstone Mortgage is like many financial firms subject to huge data growth from multiple sources. Moving to the cloud eliminated many of the challenges faced by the firm. Initially buy in from stakeholders was necessary and pilots with Talend and Snowflake revealed insights into what was necessary to deliver data excellence. Data governance was a huge success factor. Common terminology and the use of metadata was identified to ensure data is accurate during self-service access.
Veteran analyst Stuart long from IDC spoke about future enterprise resilience with a particular focus on data architectures and data leaders. In surveys IDC has conducted, 95% of organisations have integrated data across hybrid, multi-cloud environments studies data.
73% of respondents stated that there are an increasing diversity of data types and they need to adopt new methods of working with data in data intelligence, data governance and data engineer and 77% of organisations are integrating up with five different types of data pipelines and up to up to 10 different types of data management technology.
There are a lot of different approaches that are being used to resolve these issues. There is a big focus on centralising data into lakes warehouses. Businesses are making the shift towards data democratisation and becoming more data driven. Obviously, it’s important that there is access to data, but that can be a double-edged sword.
Given this trend of democratisation, how do individuals and teams know what data is trustworthy? And what data can be relied on?
Data intelligence answers who, what, where, when, why and how about data. The answers to these questions should reveal how confident data producers and data consumers are when making any decisions, whether individuals or teams. In IDC surveys, organisations with centralised data teams are more successful data engineering and analytics. However, there are still slightly more organisations with a distributed structure than those with a centralised structure. Regardless of organisational reporting structure, there’s still a need to centralise standards, policies, technologies and processes and they need to be enforced.
Mandates need to be enterprise-wide not siloed or departmental, because the number of different roles that work with data is continuously increasing. That’s why IDC has coined the term generation data or Gen D for a vocational generation that represents the roles of people who work with data. Data is what they work and complete their tasks with, whether that be tactical or strategic. They are expected to make data driven decisions and have a sense of what information is true or false.
And as a result, that enforces the evolution around data quality, trustworthiness, and literacy.
In another session, Andrew Bates, CTO at online lender Harmoney, revealed how the company had evolved and data was at the heart of understanding the customer in depth, and differentiating the company. In a sector, where some financial organisations still use file based transfer protocols, Talend technology has enabled the firm to slice and dice data to produce complex file structures. For example, Kafka enables Harmoney to stream complex credit files and bank statement data through clean procedures to deliver insight in real time.
To end Talend Connect, Sam Pierson, the new Chief Technology Officer at Talend, explained how the firm was looking to accelerate operational excellence, and brought out new features to Talend API Services. Now teams can operationalize data via REST and Open Data APIs in minutes, making sharing data faster than ever.
Talend has also added improved the integration of applications with Talend Management Console allowing new service account capabilities, including Run profiles, and support for driving data services and routes.
Major security advances have been made to the Talend product portfolio. A single sign on support for Studio and we added SSO for google workspace in Stitch.
Talend’s ML teams have been hard at work building advanced, one-of-a-kind capabilities to save time across the entire lifecycle of development and improve operational metrics when it comes to data health and your data operations.
To watch sessions from Talend Connect, please visit https://www.talend.com/uk/connect-on-demand/