Indeed, when the Debezium Kafka Connector starts, it checks whether such a publication exists, and if not, it tries to create one tracking all the tables (create publication dbz_publication for all tables;), but that requires superuser privileges on the PostgreSQL database, which Azure does not provide (the database admin roles are limited to pg_admin only, not superuser, which Microsoft reserve to their own administrators).
Thank you sharing Denis! I used wal2json in the blog, but this is relevant if using pgoutput with managed offerings (such as Azure PostgreSQL). I'll try to capture this information in the form of another blog post... thanks again :)
Abhishek,
We use Postgresql as a service on Azure. Is it mandatory to install Kafka? Can we just use Azure EventHub as a replacement? If Kafka is really required then what is the best approach to run Kafka in Azure?
the "best" approach is purely based on requirements, so the answer is "it depends". there are multiple options including HD Insight, Confluent Cloud via Azure Marketplace, Kafka on AKS, Event Hubs.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Many thanks, Abhishek!
A small (potential) missing step: at least for the
pgoutputplugin (which Debezium kind of recommend for PostgreSQL 10+), a publication, named specificallydbz_publication, must be created on the PostgreSQL database for all the tables, which are to be tracked, with a SQL command likecreate publication dbz_publication for table table1,...,tableN;.Indeed, when the Debezium Kafka Connector starts, it checks whether such a publication exists, and if not, it tries to create one tracking all the tables (
create publication dbz_publication for all tables;), but that requiressuperuserprivileges on the PostgreSQL database, which Azure does not provide (the database admin roles are limited topg_adminonly, notsuperuser, which Microsoft reserve to their own administrators).Thank you sharing Denis! I used
wal2jsonin the blog, but this is relevant if usingpgoutputwith managed offerings (such as Azure PostgreSQL). I'll try to capture this information in the form of another blog post... thanks again :)You might also want to take a look at the publication.autocreate.mode property
It is now documented - debezium.io/documentation/referenc...
Abhishek,
We use Postgresql as a service on Azure. Is it mandatory to install Kafka? Can we just use Azure EventHub as a replacement? If Kafka is really required then what is the best approach to run Kafka in Azure?
the "best" approach is purely based on requirements, so the answer is "it depends". there are multiple options including HD Insight, Confluent Cloud via Azure Marketplace, Kafka on AKS, Event Hubs.