In-memory DB support for Java

Design Discussions
1

Our apps are predominantly written in java talking to mongoDB/cassandra/redis. All three DB have in-memory java implementations that can be spun-up in a Java process making development & testing much easier. This makes development & CI much less brittle because there are no network dependencies from the application to the DB, preventing external parties from modifying the data. There is no need to deploy an external DB, which is very useful.

If we could develop against mongoDB/cassandra/redis in-memory doubles and simply switch to yugabyteDB later on in the software lifecycle there would not be an issue, but mongoDB support is not implemented; we would need to convert our existing applications to use Document Data Modeling (YugabyteDB 1.1 New Feature: Document Data Modeling with the JSON Data Type | Yugabyte)

Are there any plans to release in-memory test doubles to cover the Document Data Modeling gap? Are there any workarounds?

2

Simple switch is never simple. You have to check the exact features that you’re using. I very much doubt your memory-java-redis has all redis features (like lua support/transactions).

It’s always better to test/develop on the same db, using some type of memory config or removing transaction log or not persisting data.

@kannan can answer for mongoql plans

3

Hi @gapmeister66

Apologies for missing to respond earlier.

YugaByteDB is implemented in C++. While it doesn’t have a JNI plug-in for embedded testing from Java applications, it is possible to easily spawn a local light-weight cluster using external process(es) for testing automation. We follow a similar model for our own unit tests. There should be no network dependencies in this model as the cluster will be running on the same host.

For testing you can use a replication-factor 1 cluster, where the test setup phase starts 1 yb-master & 1 yb-tserver process and you can then run a whole array of functional tests, and finally tear down the yb-master & yb-tserver processes.

The only prerequisite would be that you provide access to temp directory that YugaByte can use as its working directory for its data/metadata files.

Will send you a bit more detailed notes shortly.

regards,
Kannan

4

Hi @gapmeister66

Following up on my earlier note, here are a few more details.

  1. One time software install on the node:
export VERSION=1.1.4.0
wget https://downloads.yugabyte.com/yugabyte-ce-${VERSION}-linux.tar.gz
tar xvfz yugabyte-ce-${VERSION}-linux.tar.gz
cd yugabyte-${VERSION}
./bin/post_install.sh
  1. Then to start a local cluster

The setup phase of your Java test would start the cluster with some steps along these lines.

What follows is in bash, but you will have to this programmatically from Java (using something like a system exec call).

Create the data directory to be used by YugaByte.

export DATA_DIR=$HOME/yb-data
mkdir $DATA_DIR

Destroy old cluster if any

pkill yb-master
pkill yb-tserver
rm -rf $DATA_DIR/*

Start Cluster:

# create a RF=1 (replication factor) local cluster and use $DATA_DIR as the storage directory.
bin/yb-master --fs_data_dirs $DATA_DIR --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --replication_factor=1 --master_addresses 127.0.0.1:7100 >& $DATA_DIR/yb-master.out &
sleep 1
bin/yb-tserver --fs_data_dirs $DATA_DIR --webserver_interface 127.0.0.1 --rpc_bind_addresses 127.0.0.1 --tserver_master_addrs 127.0.0.1:7100 --memory_limit_hard_bytes 1000000000 --redis_proxy_bind_address 127.0.0.1 --cql_proxy_bind_address 127.0.0.1 --pgsql_proxy_bind_address 127.0.0.1 --local_ip_for_outbound_sockets 127.0.0.1 >& $DATA_DIR/yb-tserver.out &
sleep 1

[Optional Step] if you plan to use the YEDIS (redis-compatible) interface:

bin/yb-admin --master_addresses 127.0.0.1:7100 setup_redis_table

At this point, you can use the 127.0.0.1:9042 endpoint to interact with this local cluster for Cassandra style opertions and 127.0.0.1:6379 endpoint to interact with this local cluster for Redis style operations.

[You can also visit the master web ui at 127.0.0.1:7000 to confirm if the cluster looks to have come up correctly.]

  1. After running your array of tests, the tear-down phase of your test would then simply cleanup the processes and the data directory.
pkill yb-master
pkill yb-tserver
rm -rf $DATA_DIR/*

Hope this helps, but feel free to let us know if you have further questions.

5

Hi @gapmeister66

Here’s a pointer to the MiniYBCluster framework we use for unit testing. (https://github.com/YugaByte/yugabyte-db/blob/00a6c82531a2ce1c3747914d3fd862efdbc2b11f/java/yb-client/src/test/java/org/yb/minicluster/MiniYBCluster.java#L156 )

It uses Java ProcessBuilder() to spawn yb-master, yb-tserver processes (https://github.com/YugaByte/yugabyte-db/blob/00a6c82531a2ce1c3747914d3fd862efdbc2b11f/java/yb-client/src/test/java/org/yb/minicluster/MiniYBDaemon.java#L266).

This a bit more general in that it handles different replication factors (i.e. can be used to test multi-node like clusters using multiple yb-master/yb-tserver processes on a single node), picks non-conflicting ports dynamically, etc.

But catch is that, today it is not packaged standalone utility and is something that is integrated into our test framework. It’ll probably take us a little bit of work to extract this into its own too. But we are also happy to get your/community help in making such a thing available as a standalone utility.

Let us know if you have additional questions.

regards,
Kannan