是否有一种方法可以让Python程序确定它当前使用了多少内存?我看到过关于单个对象的内存使用情况的讨论,但我需要的是进程的总内存使用情况,这样我就可以确定何时需要开始丢弃缓存的数据。


当前回答

对于Python 3.6和psutil 5.4.5,使用这里列出的memory_percent()函数更容易。

import os
import psutil
process = psutil.Process(os.getpid())
print(process.memory_percent())

其他回答

在Windows上,你可以使用WMI(主页,cheeseshop):

def memory():
    import os
    from wmi import WMI
    w = WMI('.')
    result = w.query("SELECT WorkingSet FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=%d" % os.getpid())
    return int(result[0].WorkingSet)

在Linux上(来自python烹饪书http://code.activestate.com/recipes/286222/:

import os
_proc_status = '/proc/%d/status' % os.getpid()

_scale = {'kB': 1024.0, 'mB': 1024.0*1024.0, 'KB': 1024.0, 'MB': 1024.0*1024.0}

def _VmB(VmKey):
    '''Private.'''
    global _proc_status, _scale
     # get pseudo file  /proc/<pid>/status
    try:
        t = open(_proc_status)
        v = t.read()
        t.close()
    except:
        return 0.0  # non-Linux?
     # get VmKey line e.g. 'VmRSS:  9999  kB\n ...'
    i = v.index(VmKey)
    v = v[i:].split(None, 3)  # whitespace
    if len(v) < 3:
        return 0.0  # invalid format?
     # convert Vm value to bytes
    return float(v[1]) * _scale[v[2]]

def memory(since=0.0):
    '''Return memory usage in bytes.'''
    return _VmB('VmSize:') - since

def resident(since=0.0):
    '''Return resident memory usage in bytes.'''
    return _VmB('VmRSS:') - since

def stacksize(since=0.0):
    '''Return stack size in bytes.'''
    return _VmB('VmStk:') - since

使用sh和os进入python拜耳的答案。

float(sh.awk(sh.ps('u','-p',os.getpid()),'{sum=sum+$6}; END {print sum/1024}'))

答案是以兆字节为单位。

甚至比/proc/self/status更容易使用:/proc/self/statm.它只是一个由几个统计数据以空格分隔的列表。我无法判断这两个文件是否始终存在。

/proc/[pid]/statm Provides information about memory usage, measured in pages. The columns are: size (1) total program size (same as VmSize in /proc/[pid]/status) resident (2) resident set size (same as VmRSS in /proc/[pid]/status) shared (3) number of resident shared pages (i.e., backed by a file) (same as RssFile+RssShmem in /proc/[pid]/status) text (4) text (code) lib (5) library (unused since Linux 2.6; always 0) data (6) data + stack dt (7) dirty pages (unused since Linux 2.6; always 0)

这里有一个简单的例子:

from pathlib import Path
from resource import getpagesize

PAGESIZE = getpagesize()
PATH = Path('/proc/self/statm')


def get_resident_set_size() -> int:
    """Return the current resident set size in bytes."""
    # statm columns are: size resident shared text lib data dt
    statm = PATH.read_text()
    fields = statm.split()
    return int(fields[1]) * PAGESIZE


data = []
start_memory = get_resident_set_size()
for _ in range(10):
    data.append('X' * 100000)
    print(get_resident_set_size() - start_memory)

生成的列表如下所示:

0
0
368640
368640
368640
638976
638976
909312
909312
909312

你可以看到,在大约分配了3次10万字节后,它增加了大约30万字节。

下面是我的函数装饰器,它可以跟踪这个进程在函数调用之前消耗了多少内存,在函数调用之后使用了多少内存,以及函数执行了多长时间。

import time
import os
import psutil


def elapsed_since(start):
    return time.strftime("%H:%M:%S", time.gmtime(time.time() - start))


def get_process_memory():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss


def track(func):
    def wrapper(*args, **kwargs):
        mem_before = get_process_memory()
        start = time.time()
        result = func(*args, **kwargs)
        elapsed_time = elapsed_since(start)
        mem_after = get_process_memory()
        print("{}: memory before: {:,}, after: {:,}, consumed: {:,}; exec time: {}".format(
            func.__name__,
            mem_before, mem_after, mem_after - mem_before,
            elapsed_time))
        return result
    return wrapper

当你用它来装饰某个函数时

from utils import track

@track
def list_create(n):
    print("inside list create")
    return [1] * n

你将会看到这样的输出:

inside list create
list_create: memory before: 45,928,448, after: 46,211,072, consumed: 282,624; exec time: 00:00:00

对于Unix系统,如果您传递-v,命令time (/usr/bin/time)将提供该信息。参见下面的最大驻留集大小,这是程序执行期间使用的最大(峰值)真实(而不是虚拟)内存:

$ /usr/bin/time -v ls /

    Command being timed: "ls /"
    User time (seconds): 0.00
    System time (seconds): 0.01
    Percent of CPU this job got: 250%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 0
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 315
    Voluntary context switches: 2
    Involuntary context switches: 0
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 0