I have a requirement where I have two independent processes running on an embedded linux.

One of the processes takes CAN bus messages every 10 – 25ms and writes them to a BerkeleyDB.
(NOTE: the process is configured such that it can skip every n-th message (resolution control), so the DB writes don’t necessarily have to execute at the speed of the messages.)

The other process reads the CAN messages from the DB and does further processing – if “further processing” is successful, the processed message is deleted from the DB.

Currently, my open_db function is as follows:

int open_db(DB **dbpp,   /* The DB handle that we are opening */
    const char *file_name,     /* The file in which the db lives */
    const char *program_name,  /* Name of the program calling this function */
    FILE *error_file_pointer)  /* File where we want error messages sent */
{
    DB *dbp;    /* For convenience */
    u_int32_t open_flags;
    int ret;
    /* Initialize the DB handle */
    ret = db_create(&dbp, NULL, 0);
    if (ret != 0) {
        fprintf(error_file_pointer, "%s: %s\n", program_name,
                db_strerror(ret));
        return(ret);
    }

    /* Point to the memory malloc'd by db_create() */
    *dbpp = dbp;

    /* Set up error handling for this database */
    dbp->set_errfile(dbp, error_file_pointer);
    dbp->set_errpfx(dbp, program_name);

    /* Set the open flags */
    open_flags = DB_CREATE;

    /* open the database */
    ret = dbp->open(dbp,        /* Pointer to the database */
                    NULL,     
                    file_name,  /* File name */
                    NULL,      
                    DB_BTREE,   /* Database type (using btree) */
                    open_flags, /* Open flags */
                    0);     

    if (ret != 0) {
        dbp->err(dbp, ret, "Database '%s' open failed.", file_name);
        return(ret);
    }

    return (0); 
}

My reader process (written in Python) is as follows:

...
    try:
        
        cursor = self._db.cursor()
        record = cursor.first()

        
        if record is None:
            raise EmptyCanMessages('No CAN messages in DB.')


        while record:
            (id, data) = record
            
            decoded_id = str(id, 'utf-8')
            decoded_data = str(data, 'utf-8')

            try:
                # processing here

                del self._dbhttps://stackoverflow.com/q/72484056

                try:
                    self._db.sync()
                    self.log(f"Removed {decoded_id} record and synced DB", log_level=LogLevel.DEBUG)

                except Exception as e:
                    self.log(e, log_level=LogLevel.ERROR)
                
            except Exception as e:
                self.log(e, log_level=LogLevel.ERROR)
                pass

            record = cursor.next()

Currently, I am experiencing a few intermittent DB issues:

can_database: BDB0689 /var/log/can.db page 12 is on free list with type 5
can_database: BDB0061 PANIC: Invalid argument
Unable to insert record bf4307a9-b0c2-42f8-b230-65aed5db75ed, err: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
can_database: BDB0060 PANIC: fatal region error detected; run recovery
Unable to sync db, err: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
...
...

Some additional details, questions and guidance on optimal configuration:

After every write, I am doing a DB->sync()

CAN_RECORD record;
memset(&record, 0, sizeof(CAN_RECORD));

asprintf(&record.can_data, "{\"t\": \"%s\", \"b\": \"%s\" \"}", U_UID, thing_name);
insert_record(&can_db.can_db, &record);

sync_db(&can_db.can_db);

free(record.can_data);

After every read (in the Python app), after I del self._dbhttps://stackoverflow.com/q/72484056 I self._db.sync()

• What is the best database type? (currently, using the standard DB_BTREE) – would DB_QUEUE or DB_RECNO be better here?

I want to minimize in-memory, and just write all to disk (since I am working on a memory-constrained device).

in my db->open() flags, there is a range of flags, addressed in more detail here: https://docs.oracle.com/cd/E17076_05/html/api_reference/C/frame_main.html

• Should I be using transactions?

• Should I be syncing DB every write and every cursor read + delete? What is the better way to do this?

• Is there a way to circumvent db->sync() all together and automatically commit to disk?

My main requirement is the ability of two independent processes to a) write to, and read from + delete record(s) as they are processed simultaneously.

The read process can continue to process the incoming messages anytime it comes back online.

(NOTE: I will keep the growth management of the DB out of this discussion, and purely focus on the optimal way of configuring for write and read with modify from two independent processes on a single DB.)