Description: | Generic data retrieval memoizer that uses an sqlite database to cache data. |
Latest Version: | 2021 |
Source Code: | src/ |
Architecture: |
|
Dependencies: |
|
Build Dependencies: |
|
Arch Repositories: |
|
AUR Page: | python3-memoizedb |
Tags: |
MemoizeDB provides a way to cache results that should not be regenerated or reretrieved every time they are accessed. It was originally written to cache JSON data retrieved from a web interface to avoid placing unnecessary load on the server for results that were unlikely to change between each query.
You can determine how long results should be saved in the database before they are refreshed.
The following script will retreive package data from the AUR. You will notice that it is slower the first time it displays information for a given package name but subsequent queries within the set caching period are much faster.
#!/usr/bin/env python3 import json import MemoizeDB import sqlite3 import sys import urllib.parse import urllib.request # The retriever function. def get_pkginfo(names): for name in names: encoded_name = urllib.parse.quote(name) url = 'https://aur.archlinux.org/rpc.php?type=info&arg={}'.format(encoded_name) with urllib.request.urlopen(url) as f: yield name, (f.read().decode(),) # The glue for handling the database. 300: cache the data for 5 minutes. glue = { 'pkginfo' : (get_pkginfo, (('json', 'TEXT'),), 300) } # The connection. conn = sqlite3.connect( 'test.sqlite3', detect_types=(sqlite3.PARSE_DECLTYPES | sqlite3.PARSE_COLNAMES), isolation_level=None ) mdb = MemoizeDB.MemoizeDB(conn, glue) mdb.db_initialize() # A wrapper function to convert the JSON text to an object. def get_obj(name): txt = mdb.get_one('pkginfo', name) return json.loads(txt) if sys.argv[1:]: names = sys.argv[1:] else: names = ('python3-memoizedb', 'python3-aur', 'powerpill', 'bauerbill') # Display the data. for name in names: obj = get_obj(name) print(json.dumps(obj, indent=2, sort_keys=True))