我有一个Python脚本,它把一个整数列表作为输入,我需要一次处理四个整数。不幸的是,我无法控制输入,否则我将它作为一个四元素元组列表传入。目前,我以这种方式迭代它:

for i in range(0, len(ints), 4):
    # dummy op for example code
    foo += ints[i] * ints[i + 1] + ints[i + 2] * ints[i + 3]

不过,它看起来很像“C-think”,这让我怀疑有一种更python的方式来处理这种情况。该列表在迭代后被丢弃,因此不需要保留。也许这样会更好?

while ints:
    foo += ints[0] * ints[1] + ints[2] * ints[3]
    ints[0:4] = []

不过,感觉还是不太对。: - /

相关问题:在Python中如何将列表分割成大小均匀的块?


当前回答

要避免所有到列表的转换,请导入itertools和:

>>> for k, g in itertools.groupby(xrange(35), lambda x: x/10):
...     list(g)

生产:

... 
0 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1 [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
2 [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
3 [30, 31, 32, 33, 34]
>>> 

我检查了groupby,它不转换为列表或使用len,所以我(认为)这将延迟每个值的解析,直到它实际使用。不幸的是,没有一个现成的答案(在这个时候)似乎提供了这种变化。

显然,如果你需要依次处理每一项,在g上嵌套一个for循环:

for k,g in itertools.groupby(xrange(35), lambda x: x/10):
    for i in g:
       # do what you need to do with individual items
    # now do what you need to do with the whole group

我对此特别感兴趣的是需要消耗一个生成器,以批量提交最多1000个更改到gmail API:

    messages = a_generator_which_would_not_be_smart_as_a_list
    for idx, batch in groupby(messages, lambda x: x/1000):
        batch_request = BatchHttpRequest()
        for message in batch:
            batch_request.add(self.service.users().messages().modify(userId='me', id=message['id'], body=msg_labels))
        http = httplib2.Http()
        self.credentials.authorize(http)
        batch_request.execute(http=http)

其他回答

我从来不想填充我的块,所以这个要求是必要的。我发现在任何可迭代对象上工作的能力也是必需的。鉴于此,我决定扩展公认的答案,https://stackoverflow.com/a/434411/1074659。

如果由于需要比较和筛选填充值而不需要填充,则这种方法的性能会受到轻微的影响。然而,对于大块大小,这个实用程序是非常高性能的。

#!/usr/bin/env python3
from itertools import zip_longest


_UNDEFINED = object()


def chunker(iterable, chunksize, fillvalue=_UNDEFINED):
    """
    Collect data into chunks and optionally pad it.

    Performance worsens as `chunksize` approaches 1.

    Inspired by:
        https://docs.python.org/3/library/itertools.html#itertools-recipes

    """
    args = [iter(iterable)] * chunksize
    chunks = zip_longest(*args, fillvalue=fillvalue)
    yield from (
        filter(lambda val: val is not _UNDEFINED, chunk)
        if chunk[-1] is _UNDEFINED
        else chunk
        for chunk in chunks
    ) if fillvalue is _UNDEFINED else chunks

还有另一个答案,它的优点是:

1)容易理解 2)适用于任何可迭代对象,而不仅仅是序列(上面的一些答案会阻塞文件句柄) 3)不立即将数据块加载到内存 4)不会在内存中生成对同一迭代器的块长的引用列表 5)在列表的末尾没有填充填充值

话虽如此,我还没有计算它的时间,所以它可能比一些更聪明的方法慢,而且考虑到用例,一些优势可能是无关紧要的。

def chunkiter(iterable, size):
  def inneriter(first, iterator, size):
    yield first
    for _ in xrange(size - 1): 
      yield iterator.next()
  it = iter(iterable)
  while True:
    yield inneriter(it.next(), it, size)

In [2]: i = chunkiter('abcdefgh', 3)
In [3]: for ii in i:                                                
          for c in ii:
            print c,
          print ''
        ...:     
        a b c 
        d e f 
        g h 

Update: A couple of drawbacks due to the fact the inner and outer loops are pulling values from the same iterator: 1) continue doesn't work as expected in the outer loop - it just continues on to the next item rather than skipping a chunk. However, this doesn't seem like a problem as there's nothing to test in the outer loop. 2) break doesn't work as expected in the inner loop - control will wind up in the inner loop again with the next item in the iterator. To skip whole chunks, either wrap the inner iterator (ii above) in a tuple, e.g. for c in tuple(ii), or set a flag and exhaust the iterator.

除非我遗漏了一些内容,否则没有提到以下使用生成器表达式的简单解决方案。它假设块的大小和数量都是已知的(通常情况下),并且不需要填充:

def chunks(it, n, m):
    """Make an iterator over m first chunks of size n.
    """
    it = iter(it)
    # Chunks are presented as tuples.
    return (tuple(next(it) for _ in range(n)) for _ in range(m))

我喜欢这种方法。它感觉简单而不神奇,支持所有可迭代类型,并且不需要导入。

def chunk_iter(iterable, chunk_size):
it = iter(iterable)
while True:
    chunk = tuple(next(it) for _ in range(chunk_size))
    if not chunk:
        break
    yield chunk

使用map()而不是zip()修复填充问题在J.F.塞巴斯蒂安的回答:

>>> def chunker(iterable, chunksize):
...   return map(None,*[iter(iterable)]*chunksize)

例子:

>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9'), ('0', None, None)]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8'), ('9', '0', None, None)]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]