By Brendan Siebecker, Director of Alliances – Atlan
By Gaurav Malhotra, Options Architect – AWS
Extra persons are utilizing knowledge at this time than ever earlier than, but it surely’s getting tougher and tougher for everybody to collaborate on the identical knowledge. This consists of knowledge engineers and analysts, product managers, entrepreneurs, researchers, and extra.
Atlan is an AWS Companion with the Amazon Redshift Prepared designation. It has pioneered a collaborative workspace serving to fashionable knowledge groups work collectively higher. Atlan is a collaboration and orchestration layer—the glue that brings collectively your workforce, the instruments you like, and the information you want.
With deep integrations throughout the trendy knowledge stack, Atlan helps groups create a single supply of fact for all of their knowledge property. Atlan is extending its suite of integrations via in depth collaboration with Amazon Net Companies (AWS), and now you can discover Atlan on AWS Market.
On this publish, we are going to share how corporations use Atlan and AWS to democratize their knowledge, collaborate extra successfully, and unify all of their information and context in a single place. We’ll additionally present learn how to combine Atlan and Amazon Redshift with a step-by-step walkthrough.
How Corporations Use Atlan and AWS
Clients within the AWS ecosystem can profit from Atlan’s seamless integration with a collection of AWS companies—together with analytics tooling reminiscent of Amazon Redshift, Amazon Athena, and AWS Glue—in addition to in style instruments within the fashionable knowledge stack reminiscent of Tableau, Apache Airflow, and dbt.
For instance, Postman (an API platform utilized by greater than 500,000 corporations worldwide) makes use of AWS and Atlan to open up their knowledge, construct belief, and turn out to be extra data-driven. That is vital as a result of Postman’s leaders staunchly consider that everybody within the firm ought to have the ability to entry knowledge and acquire insights from it. Nonetheless, earlier than Atlan, their knowledge was usually a thriller and context lived within the heads of early workforce members.
Prudhvi Vasa, Analytics Chief at Postman, defined the worth of democratizing and documenting knowledge with AWS and Atlan: “We’ve been in a position to catalog and doc all of our knowledge, which acts as a single supply of fact for our knowledge. The consequence? Everybody is ready to discover the best knowledge for his or her use case, and the information is constant throughout the board for all accessing it,” says Vasa.
“Having a dependable knowledge basis, the place folks can discover and perceive all our knowledge opens the opportunity of having everybody take part in analyzing knowledge. This allows our complete firm to turn out to be extra data-aware and data-driven, which is the objective for any main firm at this time,” Vasa provides.
Atlan, AWS, and the Trendy Information Stack
Atlan acts as a virtualized layer throughout a wide range of instruments within the fashionable knowledge stack. Its push- and pull-based metadata crawlers deliver metadata from completely different instruments within the knowledge platform to construct a unified collaboration platform.
First, Atlan creates a robust search and discovery layer. It acts as a Google-like search engine for all your knowledge, the place you possibly can rapidly uncover and entry any knowledge asset together with all of its related context and documentation. This search helps clever key phrase recognition, highly effective search filters, sorting by relevance or recognition, and even a Cmd+Ok shortcut.
This search doesn’t simply floor knowledge tables—it surfaces the whole lot about a company’s knowledge. In at this time’s day and age, knowledge property should not simply tables. That’s why Atlan lets folks search throughout each kind of information asset—enterprise intelligence (BI) dashboards, pipelines, code, fashions, queries, metrics, directed acrylic graphs (DAGs), and extra.
Second, Atlan unifies context from all of the completely different instruments in your knowledge stack in a single place. The place does this knowledge come from? Who makes use of it? Can I belief it? The “asset profile” in Atlan solutions questions like these with data like an information asset’s description, certification (verified, WIP, or deprecated), column previews, pattern knowledge, and Readme. This makes it simpler to know every knowledge asset (like lineage, documentation, and possession) in a single view.
Embedded Collaboration Integrations
Atlan is constructed on the premise of embedded collaboration, borrowing ideas from GitHub, Figma, Superhuman, and different fashionable future-of-work instruments.
Embedded collaboration is about work taking place the place you might be, with the least quantity of friction. What when you might request entry to an information asset whenever you get a hyperlink, and the proprietor might get the request on Slack and approve or reject it proper there?
What if, whenever you’re inspecting an information asset and have to report a difficulty, you would instantly set off a help request that’s completely built-in along with your engineering workforce’s Jira workflow?
Embedded collaboration unifies these micro-workflows that waste time, trigger frustration, and result in device fatigue, turning time-consuming duties throughout a number of instruments into a number of clicks in whichever device you’re already utilizing.
Getting Began with Atlan and AWS
Atlan is constructed on high of AWS’s highly effective companies. By immediately integrating with AWS companies like Amazon Redshift, Atlan helps knowledge groups accomplish extra by making collaboration a seamless a part of their course of.
Within the following step-by-step information, we’ll present you learn how to rapidly combine Atlan with Redshift to open up a brand new world of collaboration, readability, and belief for contemporary knowledge groups.
Comply with the steps under to ascertain a connection and combine Atlan with a Redshift database.
Step 1: Choose the Supply
- Log into your Atlan workspace.
- Click on on the Workflow button within the left sidebar.
- You’ll see the Market web page with the listing of sources obtainable in your workspace. Click on on New Workflow on the high proper.
- Choose Redshift from the listing of choices within the integrations tab, and click on Setup Workflow.
Step 2: Present Credentials
- To arrange a brand new connection, fill in your Redshift credentials on the Credential web page. Beneath is an instance:
- Hostname: examplecluster.abc123xyz789.us-west-2.redshift.amazonaws.com
- Port: 5439
- Username: myusername
- Password: xxxxxx
- Default Database: dev
- Choose the right authentication technique (fundamental authentication, or an IAM consumer or IAM function).
- After getting stuffed within the particulars, click on on Check Authentication after which Subsequent.
Step 3: Set Up Your Configuration
- On the Connection web page, title your connection and choose the customers or teams who ought to have the ability to entry it.
- On the Metadata web page, specify any metadata you need to embody or exclude from crawling.
- Click on Run to run the crawler as soon as, or click on on Schedule & Run to schedule it for a every day, weekly, or month-to-month run.
- When you click on Run or set a schedule, the workflow will begin working.
Atlan will crawl your Amazon Redshift occasion and ingest the entire metadata into Atlan. This course of varies relying on the scale of the warehouse you’re seeking to crawl, but it surely usually takes lower than half-hour.
Step 4: Uncover Your Belongings
Now that you simply’ve efficiently linked Atlan to Amazon Redshift and the Atlan crawler has ingested the metadata, you can begin discovering your property inside Atlan.
Within the instance under, an organization has 362 Redshift property inside Atlan. These are seen on the Discovery web page, filtered by the Amazon Redshift integration.
The information workforce can click on via to any asset to see related context, such because the column names, glossary phrases, classifications, standing, associated queries, and Readme.
They will additionally see the information asset’s lineage, which is auto-generated on the column stage for each Redshift asset. This helps knowledge groups see the place every asset comes from and which dashboards use it.
Organising Atlan is simple for analysts and enterprise customers alike. From connecting your tables and dashboards inside Amazon Redshift to enriching imported knowledge property, the method doesn’t require main knowledge engineering sources or time. Crawling occurs seamlessly and stays in sync along with your AWS situations.
For extra data, take a look at these hyperlinks:
This text was initially printed on the AWS Companion Community (APN) weblog.