shelve module in Python - Persistence in Python objects

The shelve module in the standard library offers an efficient way of  storing and retrieving persistent objects in a dictionary-like format.

shelve allows you to store multiple objects in a single file with a key-value system. It uses the underlying dbm module (database manager) to handle the storage of data, making it highly efficient and fast for both read and write operations.

The shelve module can be very useful when we want objects to persist across different executions.

To use the module, we first need to import it in our program:

import shelve

print(shelve)

Basic Usage

Basically, to create a shelf, we use the open() helper function in the module, the function has the following basic syntax:

shelve.open(filename, flag = 'c')

The filename parameter specifies the name of the database file. Consider the following example: 

import shelve

s = shelve.open('cities.db')

#write to the file
s['Asia'] = ['Tokyo', 'Manilla']
s['Europe'] = ['Paris', 'Berlin']
s['N.America'] = ['Ottawa', 'Washington']
s['Africa'] = ['Kigali', 'Nairobi']

#close the connection
s.close()

In the above example, a file named cities.db will be created if not already existing  and it will be populated with the data we have provided. To use the stored data, re-open the file and use it as if it was a dictionary. The following example show how to do this.

import shelve

s = shelve.open('cities.db', flag = 'r')

print(list(s.keys()))

print(s['Africa'])
print(s['Europe'])

s.close()

Specifying the 'r' flag opens the file in read-only mode. While this is not necessary for performing read operations, it is particularly useful when multiple processes or threads are accessing the same database, as it prevents any conflicts or errors from occurring due to simultaneous write operations.

Note that calling the close() method is necessary to ensure that the shelf and associated  files are released. We can automate calling of the close() method by using the contextlib.closing function which automatically closes the shelve before exiting the with block. This way, the previous example will look as shown below.

import shelve
from contextlib import closing

with closing(shelve.open('cities.db', flag = 'r')) as s:
    print(list(s.keys()))

    print(s['Africa'])
    print(s['Europe'])

#s is now closed

Enable writeback

By default a value in the shelve will not be changeable after it has been stored.  For example, if we modify a list through the shelve the changes will not be committed to the stored object. Consider the following example:

import shelve
from contextlib import closing

with closing(shelve.open('cities.db')) as s:
    s['Europe'].append('Moscow') #this will not be committed to the shelf

with closing(shelve.open('cities.db', flag = 'r')) as s:
    print(s['Europe'])

As you can see from the above example, appending 'Moscow' to the list whose key is 'Europe' did not get propagated to the stored list. 

We can set the writeback parameter to True to allow changes on the stored object to be performed through the opened shelve.

import shelve
from contextlib import closing

with closing(shelve.open('cities.db', writeback = True)) as s:
    s['Europe'].append('Moscow') #this will not be committed to the shelf

with closing(shelve.open('cities.db', flag = 'r')) as s:
    print(s['Europe'])

Note that enabling writeback can be inefficient because a cache of the existing data has to be stored and updated regularly.  So instead of enabling writeback, it would be more efficient to entirely re-store the updated object again. This way, our previous example, would look as follows:

import shelve
from contextlib import closing

with closing(shelve.open('cities.db')) as s:
    europe_cities = s['Europe']
    europe_cities.append('Moscow')
    
    s['Europe'] = europe_list

with closing(shelve.open('cities.db', flag = 'r')) as s:
    print(s['Europe'])