When using adbapi with a sqlite backend, it is demonstrated that the memory footprint depends on the number of simultaneous connections running at a given moment. This memory is not reclaimed, but it is reused. On an amd64 processor it approximately amounts to 2KB per simultaneous connection.
The test script runs 50,000 simultaneous connections to a virgin database with a single table. The request being done is a noop. Once all requests have been served, the amount of RSS memory is reported together with the amount of VIRT memory. The RSS memory is what we are interested in : it is the memory that is actually used and cannot be shared. The VIRT memory difference is either shared with other processes (it happens when pages from mmap(3)ed files are used) or not yet used.
from twisted.internet import defer, reactor
from twisted.enterprise import adbapi
import os
import psutil
import sqlite3
database='t.sqlite'
try:
os.unlink(database)
except:
pass
db = sqlite3.connect(database)
c = db.cursor()
c.execute("CREATE TABLE games ( id INTEGER )")
c.close()
db.close()
db = adbapi.ConnectionPool("sqlite3", database=database, cp_min=1, cp_max=1)
def g():
l = []
for i in range(1,50000):
l.append(db.runOperation("DELETE FROM games"))
return defer.DeferredList(l)
@defer.inlineCallbacks
def f():
m = psutil.Process(os.getpid()).get_memory_info()
print "***************** RSS %dM VIRT %dM" % ( m[0] / ( 1024 * 1024 ), m[1] / ( 1024 * 1024 ))
yield g()
reactor.callLater(0.1, f)
reactor.callLater(0.1, f)
reactor.run()
The result of the above benchmark script is as follows:
***************** RSS 11M VIRT 63M ***************** RSS 107M VIRT 203M ***************** RSS 117M VIRT 203M ***************** RSS 110M VIRT 195M ***************** RSS 110M VIRT 195M ***************** RSS 109M VIRT 195M ***************** RSS 109M VIRT 195M ***************** RSS 109M VIRT 195M ***************** RSS 110M VIRT 195M ***************** RSS 110M VIRT 195M ***************** RSS 110M VIRT 195M ***************** RSS 110M VIRT 195M ***************** RSS 110M VIRT 195M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 110M VIRT 194M ***************** RSS 108M VIRT 193M ***************** RSS 108M VIRT 192M ***************** RSS 108M VIRT 192M
The first time around, the memory grows by 100MB that is an about 2KB per simultaneous connection initiated. The amount of RSS memory stays the same after each iteration, showing that the objects allocated during the first round are being re-used.