python3-memoizedb

2023-05-10 21:51 UTC
  • Xyne

Metadata

Description: Generic data retrieval memoizer that uses an sqlite database to cache data.
Latest Version: 2021
Source Code: src/
Architecture:
  • any
Dependencies:
  • python3
Arch Repositories:
  • [xyne-any]
  • [xyne-i686]
  • [xyne-x86_64]
AUR Page: python3-memoizedb
Tags:

About

MemoizeDB provides a way to cache results that should not be regenerated or reretrieved every time they are accessed. It was originally written to cache JSON data retrieved from a web interface to avoid placing unnecessary load on the server for results that were unlikely to change between each query.

You can determine how long results should be saved in the database before they are refreshed.

Example

The following script will retreive package data from the AUR. You will notice that it is slower the first time it displays information for a given package name but subsequent queries within the set caching period are much faster.

#!/usr/bin/env python3

import json
import MemoizeDB
import sqlite3
import sys
import urllib.parse
import urllib.request


# The retriever function.
def get_pkginfo(names):
  for name in names:
    encoded_name = urllib.parse.quote(name)
    url = 'https://aur.archlinux.org/rpc.php?type=info&arg={}'.format(encoded_name)
    with urllib.request.urlopen(url) as f:
      yield name, (f.read().decode(),)

# The glue for handling the database. 300: cache the data for 5 minutes.
glue = {
  'pkginfo' : (get_pkginfo, (('json', 'TEXT'),), 300)
}

# The connection.
conn = sqlite3.connect(
  'test.sqlite3',
  detect_types=(sqlite3.PARSE_DECLTYPES | sqlite3.PARSE_COLNAMES),
  isolation_level=None
)
mdb = MemoizeDB.MemoizeDB(conn, glue)
mdb.db_initialize()

# A wrapper function to convert the JSON text to an object.
def get_obj(name):
  txt = mdb.get_one('pkginfo', name)
  return json.loads(txt)

if sys.argv[1:]:
  names = sys.argv[1:]
else:
  names = ('python3-memoizedb', 'python3-aur', 'powerpill', 'bauerbill')

# Display the data.
for name in names:
  obj = get_obj(name)
  print(json.dumps(obj, indent=2, sort_keys=True))

CHANGELOG

2017-03-11

  • MDBError now requires the database connection as its first argument in addition to the previous arguments.

2016-04-03

  • Glue function now takes multiple keys and yields 2-tuples of keys and values.
  • Multiple functions have have “*_many” variants which support multiple keys.

2012-11-03

  • Fixed documentation for glue function in MemoizeDB’s help text.
  • Added null_ttl parameter for temporarily caching empty data.
  • Removed support for non-tuple values to db_insert method.
Contact
echo xyne.archlinux.org | sed 's/\./@/'
Validation
XHTML 1.0 Strict CSS level 3 Atom 1.0