I have a system built of microservices communicating using Kafka. How should the topics be created during deployment?
I see two options:
I. Centralized topics creation. Have a central place (repository) where teams add the topics required by their microservices. In this approach, the deployment will look as follows:
1. Deploy kafka 2. Deploy kafka topics artifact that contains the required topics for all microservices 3. Deploy microservices
II. Every service deploys its own topics. The deployment looks as follows:
1. Deploy Kafka 2. Deploy microservices. Within each microservice deployment the required topics are created.
I see the value in option I. I could see all the topics that are deployed, partitions, retention and compaction policies. This would help be understand kafka resource allocation and its configuration. The downside of this option is a potential coupling, where I would need to deploy a single microservice on a running system. In this scenario I would need to deploy a new version of Kafka topics artifact and the microservice itself.
What are the best practices here?