Consuming AWS MSK from Azure Databricks over mTLS

Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does not cover the one that keeps showing up in large enterprises: the Kafka cluster is in AWS, the compute is Azure Databricks, IAM is off the table because the identity system is a corporate PKI, and the traffic never touches the public internet. This walks through that setup end to end. Certificates from a private CA, a private network path between the two clouds, and a Structured Streaming job that actually keeps running on a multi-node cluster instead of only on the driver. Why mTLS instead of IAM MSK offers four authentication modes: plaintext, TLS with client certificates, SASL/SCRAM, and IAM. IAM is the easiest and the best choice when your consumers run in AWS. Cross-cloud, IAM stops being convenient. Azure Databricks executors have no AWS identity. You can bolt one on with OIDC federation and assumed roles, but in most regulated enterprises the decision has already been made elsewhere: there is a corporate PKI, every service-to-service hop uses client certificates issued from it, and the security architecture review is going to ask why this one connection is different. mTLS is the path of least resistance, not the clever choice. One important constraint before you start. MSK will only accept client certificates issued by an AWS Private CA (ACM PCA) that is associated with the cluster. If your corporate PKI is not that CA, you have two options: stand up an ACM PCA subordinate signed by your corporate root, or issue the Databricks client certificate from a dedicated ACM PCA and treat it as a separate trust domain. The subordinate route is usually what security wants, and it takes longer to get approved than everything else in this article combined. Start that conversation first. Network path Three ways to get private connectivity between an Azure VNet and an AWS VPC: Site-to-site VPN. IPsec tunnel between an Azure VPN