I simply spoke at Zero Gravity by Incorta, a contemporary cloud information pipeline occasion. I’ve been working with information pipelines for years, so it was wonderful sharing about my experiences taming the chaos in at this time’s fashionable information stack!
In case you missed my speak or need to dive into the small print, right here’s my slide deck from this speak. In case you have any questions, please drop a remark or attain out to me on Twitter or LinkedIn!
As we speak, metadata is all over the place. Each part of the trendy information stack and each person interplay on it generates metadata. Aside from conventional varieties like technical metadata (e.g. schemas) and enterprise metadata (e.g. taxonomy, glossary), our information techniques now create fully new types of metadata.
All these new types of metadata are being created by dwelling information techniques, generally in real-time. This has led to an explosion within the measurement and scale of metadata. Nevertheless, this has made it troublesome for centralized information engineering and reporting groups to totally leverage metadata that holds the important thing to the elusive promised land — a single supply of fact.
Groups at this time wrestle to maintain tempo with “service requests” and to ramp up on area experience wanted. The web result’s misplaced alternative. However by successfully accumulating metadata, a group can unify context about all their instruments, processes, and information.
This session covers how can information groups leverage metadata to energy higher collaboration, enabling organizations to speed up onboarding and construct information merchandise outdoors the central group. I’ll cowl lively metadata and the way its utility by means of embedded collaboration, open API interface, powered by information graphs, programmable-intelligence bots, information course of automation, reverse metadata, can assist meet enterprise governance requirements.
Curious to study extra? Right here’s extra info on every part I referenced throughout my speak: