Using Iterators and Generators in Python I

When I see patterns in my programs, I consider it a sign of trouble. The shape of a program should reflect only the problem it needs to solve.

Python has a unique way of expressing iterators, so that it can be easily used in programming. While other languages have a strict procedure of iterating through any data-structures, Python has a user friendly interface, using which we can implement iteration of all type of data-structures. Before going through the iterators and of course the special Python iterator : Generators, let us be clear of the technical jargons, we would use throughout this article.

This article would deal with iterators, and next one would deal with generators.

We roughly have the following distinction, while programming in Python -

1. Containers
2. Iterators and Iterables.

Let us go through it one at a time -

Containers

Containers, as I referred through this blog post, can be thought of as a data-structure, which store data in different types and way, and return data, in different way as possible, in which some might be memory efficient, while some might be time efficient. Whatever be the approach, when we think of containers in an abstract manner, these are magic boxes, which take in data, and return data, as and when required.

Iterables and Iterators

Let us not go through the fundamental difference between them ( I would discuss them later), but consider them as a Pythonic approach to access the data from the containers, and represent them in a repetitive manner. We often use iteration for simplifying, or rather looping out the same task over and over again. What makes Pythonic approach different, is that we can have the same interface of looping through, without having different ways of looping through different data-structures. This is what makes Python, so Pythonic in nature !

Every collection in Python is iterable, and iterators are used internally to support:

• for loops;
• collection types construction and extension;
• looping over text files line by line;
• list, dict and set comprehensions;
• tuple unpacking;
• unpacking actual parameters with * in function calls.

Using Iterables and Iterators

In Python, we often follow this premise -

If it looks like a duck and walks like a duck, it is a duck

Similarly, as all iteration methods are same for different data-structures, all Python data-structures must implement some certain behaviours, using which Python can extract data using iteration philosophy.

Most Containers are iterable. But there are other things, which might not be a data-structure, but would still be iterable, like open files, open sockets, etc.

An iterable is any object, not necessarily a data structure, that can return an iterator (with the purpose of returning all of its elements).

So most containers are iterable, because they provide material for iteration, however, we can also have iterables, which might not be a data-structure.

Difference between Iterables and Iterator

There is unique difference between Iterables and Iterators, and this would be evident with the following snippet -

>>> x = [1,2,3]
>>> a = iter(x)
>>> b = iter(x)
>>> type(x)
<class ‘list’>
>>> type(a)
<class ‘list_iterator’>

Here we see that the object x is list object( container object), while a is an iterator object. I would explain the logic, then the technical difference between them-

Logically, iterables are object( which may or may not be container), which provide the necessary ‘raw material’ to iterate through. Iterators are object, which provide necessary methods(or rather behaviours) using which we can iterate through each item.

Technically, in Python, any object which implement __iter__ method, is said to be iterable, and this method should return the object which is in fact the iterable( or container object, in this case the object x). Iterators are those which implement both __iter__ and __next__ method, and the __next__ method should return each element of the iterable, and must return StopIteration exception when there are no elements left. So,all iterators are iterable, but not vice versa.

This must raise a question, as to why iterators, implement __iter__ method, even though the same purpose is fulfilled through iterables ?( i.e. we can pass an iterable to an iterator, and then iterate through the iterator?). There is a reason behind this. for loops in Python, first initialises the object, or it requires the iterable, which is provided through __iter__ method, while once initialized, it implements __next__ method to loop through the elements. There is one more reason, that this property helps in maintaining the state of the iterator, that we can resume a partially consumed(or used) iterator. If this was not the case, then everytime, we would call the __next__ method, we would get only first method as a result.

So to summarise, in order to iterate through the containers, we should first get an iterable, and then iterate through the elements using __next__ function.

Example of using Iterators

Here is an example showing the implementation of iterator protocol -

>>> class Even():
...      def __init__(self, start=0):
...              self.start = start
...      def __iter__(self):
...              return self
...      def __next__(self):
...              if self.start % 2 == 0:
...                      self.start = self.start + 2
...                      return self.start
...              else:
...                      raise("Not an even starting number!")

This snippet prints out even numbers, starting from a given number. Below we implement some results -

>>> c = Even()
>>> d = c.__iter__()
>>> d
<__main__.Even object at 0x7ff37e7896d8>
>>> d.__next__()
2
>>> d.__next__()
4
>>> d.__next__()
6
>>> d.__next__()
8

Here we observe, that we create a Even number object, and then create an iterator, and call the __next__ method to get the resultant element of the iterator.

Next article would discuss a special kind of iterators : Generators. Feel free to comment down your queries and suggestions, and I would be happy to address them down.