Data sourced from Dakota Marketplace, the global LP and GP intelligence platform trusted by thousands of investment professionals. Learn More | Book a Demo
Databricks was built on Apache Spark to run machine learning directly on large, messy datasets. That flexibility is why data science and engineering teams at investment firms choose it, and it’s also why a Databricks deployment without discipline turns into a data swamp and a surprise invoice.
We work with investment firms connecting Dakota data to Databricks every day, and the same patterns keep showing up. The teams that get the most from the platform decide early what it’s for, govern everything in one place, and stop building pipelines for data they could simply receive.
In this article, we’ll cover eight Databricks best practices for investment firms, from data routing and governance to bringing in external data and keeping costs in check.
8 Databricks Best Practices for Investment Firms
1. Decide What Databricks Is For Before You Build
Databricks is strongest in the hands of data science and engineering teams. Our shorthand: Snowflake is for the analytics team, and Databricks is for the data science and engineering team.
That doesn’t make Databricks the wrong choice for analysts. It means you should be clear about the core workloads before you design around them. Databricks tends to win on:
- Risk and model governance, where you need to trace exactly what data fed a model
- Real-time work that runs continuously, not on a reporting cycle
- Heavier ML research, where quant teams go beyond SQL-only analysis
Many of the largest firms run both platforms, with engineering in Databricks and analytics in Snowflake. If that’s your setup, decide which team owns which data so the two environments don’t drift apart.
2. Route Data by Its Shape
Not all data belongs in the same place. Our rule of thumb: data that is already structured and ready to use can go straight to a warehouse, while data that needs more cleaning and structuring goes to a lake first.
Databricks is a lakehouse, which means it can handle both. The risk is treating that flexibility as permission to dump everything in raw. A lake applies structure only when someone queries it, so without clear conventions for what lands where, teams end up with duplicate tables, conflicting versions, and no one sure which dataset is the source of truth.
Set naming and storage conventions early, and separate raw, cleaned, and production-ready data so analysts know what they can trust.
3. Govern Everything in Unity Catalog
Unity Catalog is where Databricks governance lives, and we recommend investment teams use it from day one. Its biggest strength is lineage: it traces what data was fed into a model and what came out.
For investment firms, that matters for model risk and auditing. When a compliance team asks what fed an allocation model or a risk calculation, the answer should already be recorded, not reconstructed after the fact.
Unity Catalog is also a prerequisite for receiving shared data through Delta Sharing. If your workspace isn’t enabled for it yet, that’s the first fix.
4. Bring External Data In Through Shares, Not Pipelines
Third-party data usually arrives the hard way: file transfers, ETL jobs, and custom code that someone has to maintain. Delta Sharing, Databricks’ native protocol for sharing data across organizations, skips all of that by mounting a provider’s data directly in your workspace as a catalog.
When a provider offers a private share, ask for one. A standard marketplace listing comes with a predefined set of records that may or may not fit your firm. A private share is scoped to what you actually need. For example, a firm that only cares about RIAs can receive just RIA accounts and contacts, and nothing else.
An API still makes sense when you’re pushing data into an application your team is building. But if your work already runs in Databricks, a share drops the data into the environment you’ve built instead of asking you to build around the data.
Want to see what a tailored Dakota share would look like in your workspace? Book a demo to scope it.
5. Keep External Data Read-Only and Write to Your Own Catalogs
Shared data from a provider should stay exactly as the provider delivers it. Our Databricks share, for example, is read-only, which protects your team from accidentally overwriting the source.
The best practice is to query the shared catalog, join it to your own tables, and write any enriched or transformed results to catalogs your firm owns. That keeps a clean line between what came from the provider and what your team built on top of it, which also makes lineage in Unity Catalog easier to follow.
6. Grant Access by Group, Not by Individual
When a new dataset lands, the fastest way to create a governance problem is to hand out access one person at a time. Instead, have your Databricks administrator grant access to the right internal groups, such as research, data science, or business development.
This also simplifies onboarding external data. When you send a provider your Sharing ID, tell them which users or groups should have access up front, so the share is ready for the right people the day it goes live.
7. Give Analysts a Way In
Databricks used to require Spark and Python skills, which kept analysts on the sidelines. That changed with Databricks SQL and better BI tooling, which let analysts query data directly without engineering support.
If your analysts are still filing requests with the data team for every question, set them up on Databricks SQL. Shared datasets like allocator accounts and contacts are a natural starting point, since they’re already structured and ready to query. Your team can join them directly to your CRM records and fundraising history.
8. Manage the Workload, Not the Price Sheet
Cost comparisons between Databricks and Snowflake are everywhere, and we’d be cautious about taking either vendor’s numbers at face value. What drives the bill is how much you use the platform and how well the deployment is optimized.
Databricks can be more cost efficient for large, complex workloads at scale. But in both platforms, an unoptimized deployment gets expensive quickly. Review which jobs run, how often, and on what compute, and retire the ones no one uses.
Frequently Asked Questions
Is Databricks or Snowflake better for investment firms?
It depends on who will use it. Databricks fits data science and engineering teams running machine learning on large datasets. Snowflake fits analytics teams that want clean, structured data for reporting. Many of the largest firms use both.
What is Unity Catalog used for?
Unity Catalog is Databricks’ governance layer. It controls who can access data and traces what data fed into a model and what came out, which supports model risk management and auditing.
What’s the best way to get third-party data into Databricks?
Through Delta Sharing, when the provider supports it. A share mounts the provider’s data in your workspace as a catalog, with no ETL pipeline or custom code.
Is Dakota Marketplace data available in Databricks?
Yes. We deliver Dakota’s allocator account and contact data to Databricks as a private share tailored to your firm. Setup takes two steps: send us your Sharing ID, then mount the DAKOTA share as a catalog.
Bring Dakota Data Into Your Databricks Workspace
Dakota Marketplace’s 250,000+ allocator accounts and 386,000+ verified contacts can be live in your Databricks workspace as a private share, governed in Unity Catalog and ready to query. Scope it to the allocator types, geographies, and contacts your team targets, then build on it with the same tools you already use.
Want to see Dakota data running in Databricks? Book a demo!
