RELATED IDEAS
Allow users to set much larger retention times on topics to support more use cases, including kappa architectures
Some use cases are not possible with Message Hub because the topic retention times are too small. One example is kappa architecture where you would like to reprocess your data after changing your stream processing logic.
By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.
This would have been useful with a recent RFP
Some more information on my use case ...
I would like to load changes in reference data for a customer’s product data into kafka. I’m currently loading the purchase transactions and would like to join the product data to the transactions to enrich the transaction data. The product data is going to be in the region of MB rather than GB. However, because I can only keep 30 days of state for the product data in the normal topics, because this data changes slowly, it would eventually all be deleted. Therefore, I would need to (a) either rely on the ktable internal topic to save my product data, or (b) save the state in a db and somehow synchronise that with my streams app. IMHO, neither of these options are any good.