OBJECT_NOT_FOUND Error when restarting cluster

The issue:

Hello there, good day to you!
I’ve a self manage cluster with podman compose I was running studies in (mainly about performance and storage size of my current dataset) because we are planning a migration to yugabytedb from mongodb. And one of the times I restarted the server it returned this error when connecting:

Table with identifier 000000010000300080000000000004ec not found: OBJECT_NOT_FOUND

The server has an attached volume, and it’s the host is Fedora 41 with ext4 file system.
I’ve removed the volume and started the server again, which solved the issue, however If I were to run it in production, this cannot happen. Can you explain why it happen? If it’s a configuration issue? What are the steps to prevent it happening again? What can I do to restore the data If it were to happened?

In the real deployment we are going to use podman volume export to backup the data from the server. Would this be a problem? Is there a better way to back it up (please keep in mind, I need to run a backup every 6 hours)?

Configuration:

compose.yml:

version: "3"

volumes:
    yugabyte-data:
        driver: local

networks:
    default: {}

services:
    yugabyte:
        image: docker.io/yugabytedb/yugabyte:latest
        tty: true
        container_name: yugabytedb
        environment:
            YSQL_USER: ${yugabyte_user}
            YSQL_DB: fancyfone
            YSQL_PASSWORD: ${yugabyte_password}
        networks:
            default:
                aliases:
                    - yugabytedb
        command: ./bin/yugabyted start --advertise_address=yugabytedb --cloud_location=aws.us-east-1.us-east-1a --fault_tolerance=zone --background=false --base_dir=/home/yugabyte/yb_data
        ports:
            - 7100:7100 # YB-Master
            - 9100:9100 # YB-TServer
            - 15433:15433 # YB-UI
            - 5433:5433 # YSQL
            - 9042:9042 # YCQL
        volumes:
            - yugabyte-data:/home/yugabyte/yb_data

I run this cluster on Hetzner, because of the size of the data, it’s not feasible to run multiple nodes right now, but we may look into adding extra nodes in the future, once it becomes feasible.

Hi @lmtr0 ,

The error indicates it is something to do with initdb initialization. It’s difficult to say why the error occurred without complete logs.

  • Can you please upload the logs from all the containers by running -
./bin/yugabyted collect_logs --base_dir <base_dir_path>
  • Also, please provide all the steps you performed before the error appeared.

For production based deployments, we recommend using the images of the stable branch. Current latest stable release is 2024.2.1.

Funny, I am unable to reproduce the problem locally. One think I did and I didn’t mentioned is:
This server has been running for 3 months, I ran podman compose pull and restarted the services. which means, I probably upgraded the container from the latest version of 3 months ago to the current version (I know I should be upgrading it more frequently, but the study was running for 3 months straight).

Would that be a factor? Or is yugabytedb smart enough to migrate the data without human intervention?

For production based deployments, we recommend using the images of the stable branch. Current latest stable release is 2024.2.1.

is the latest tag, not a stable branch?

I’m going to test creating a cluster and populating it with data on 2.20.7.2-b1-x86_64 then upgrading it to 2024.2.1.0-b185 and reporting on my findings

1 Like

The latest tag usually means the latest preview release. The stable branch release will have their own tag. Generally, we don’t recommend using preview release in prod and upgrade is not supported from stable to preview release.

The latest tag is 2.25 https://hub.docker.com/layers/yugabytedb/yugabyte/latest/images/sha256-a44ab2d5b5b85ae42f86f0365d7b39ee1e36288282d232ccb8cf47fe5dfebe95 which is not stable but preview release.

Upgrading to 2.25 has still issues, probably why you had an issue.

The latest tag is 2.25 https://hub.docker.com/layers/yugabytedb/yugabyte/latest/images/sha256-a44ab2d5b5b85ae42f86f0365d7b39ee1e36288282d232ccb8cf47fe5dfebe95 which is not stable but preview release.

Upgrading to 2.25 has still issues, probably why you had an issue.

Nice to know :slight_smile: I found the What's new in the YugabyteDB v2024.2 LTS release series | YugabyteDB Docs and the relevant documentation about my error.

I will say I’m only ever commiting this error once LoL.

I was able to reproduce the error (After upgrading from the LTS branch to the preview branch) and here are the logs about it:

And I’ve also tested migrating the container volume from 2.23.1.0-b220 (Nov 12, 2024) to 2024.2.1.0-b185 (Jan 30, 2025) and it worked without issue.

What does that mean?

  • Create a yugabyte container with a data volume in image version 2.23.1.0-b220
  • Run some random migration to insert data into the database
  • Recreate the yugabyte container with the same data volume and image version 2024.2.1.0-b185

Here is the example data I populated the database with:

create table example
(
    id                     varchar(32) primary key,
    "namespace"            varchar(255) not null,
    "comment"              text not null
);

insert into example ("id", "namespace", "comment")
values
    ('e_01', 'namespace_01', 'comment 01'),
    ('e_02', 'namespace_02', 'comment 02'),
    ('e_03', 'namespace_03', 'comment 03'),
    ('e_04', 'namespace_04', 'comment 04'),
    ('e_05', 'namespace_05', 'comment 05'),
    ('e_06', 'namespace_06', 'comment 06'),
    ('e_07', 'namespace_07', 'comment 07'),
    ('e_08', 'namespace_08', 'comment 08'),
    ('e_09', 'namespace_09', 'comment 09'),
    ('e_10', 'namespace_10', 'comment 10'),
    ('e_11', 'namespace_11', 'comment 11'),
    ('e_12', 'namespace_12', 'comment 12'),
    ('e_13', 'namespace_13', 'comment 13'),
    ('e_14', 'namespace_14', 'comment 14'),
    ('e_15', 'namespace_15', 'comment 15'),
    ('e_16', 'namespace_16', 'comment 16'),
    ('e_17', 'namespace_17', 'comment 17'),
    ('e_18', 'namespace_18', 'comment 18'),
    ('e_19', 'namespace_19', 'comment 19'),
    ('e_20', 'namespace_20', 'comment 20')
-- .... I made it until 5000 rows
;

Test connecting & retrieving data. No errors.

TD;DR: The problem was: “I didn’t read the documentation”. I should have read the documentation and notice it isn’t latest tag that is presented as the installation method to containers, but a tagged version.

Thank you for your time, have a great day! :slight_smile:

1 Like

Upgrading from preview to stable is also not supported, see Upgrade YugabyteDB | YugabyteDB Docs :

Upgrades are not supported between preview and stable versions.

1 Like