Can yugabyteDB be used for storage & streaming?

Design Discussions
1

We have GoLang backend service:

  1. Storing data in yugabyte DB
  2. Publishing the same data to Kafka

Step 2 was necessary so that consumers can stream through kafka

Can yugabyteDB help stream data, once a new row created in a table? to avoid maintaining kafka…

2

Hi @sham_yuga ,

You will be able to do this with YugabyteDB CDC steams as soon as it is available. This work is in progress and can be tracked here.

3

@Marko_Rajcevic
Does streaming responsibility(in addition to storage), comes with a compromise that there could be noisy neighbours or bad consumers to knock out a database?

So, how does authorization/authentication designed to have right neighbours as consumers?

Does yugabyte db support streaming with push model?

4

I believe so. But assuming the CDC get_changes() will be efficient, probably not.

I don’t believe at first. You can ask on the github issue [Master]Add Change Data Capture (CDC) APIs to stream data out of YugabyteDB · Issue #9019 · yugabyte/yugabyte-db · GitHub for more clarity.

5

@dorian_yugabyte

Raised a query but no response

6

@sham_yuga replied on the github thread

7

@dorian_yugabyte

I did not get this:
Can you ask on #11287 ?

You are linking to my own query

8

@sham_yuga edited. Meant on [Master]Add Change Data Capture (CDC) APIs to stream data out of YugabyteDB · Issue #9019 · yugabyte/yugabyte-db · GitHub