Python Generators Tutorial with Examples

Generators in Python are a powerful way to create iterators that allow you to iterate over data without storing the entire sequence in memory at once.

They are particularly useful for handling large datasets or streams of data where holding everything in memory would be inefficient or impractical.

In this tutorial, you’ll learn:

What are generators?

Differences between functions, iterators, and generators
Creating generators using generator functions
Creating generators using generator expressions
Working with yield and controlling generator state
Practical examples of generators

1. What Are Generators?

Generators are special iterators in Python that yield items one at a time and only compute the next item when requested. This behavior makes them memory-efficient, especially when dealing with large datasets. Instead of returning all values at once, a generator generates values on the fly.

Generators are typically created using functions with the yield keyword or generator expressions.

2. Difference Between Functions, Iterators, and Generators

Regular Functions

A regular function computes all results at once and returns them using return. Once the function finishes, its local variables are destroyed, and calling the function again will start from the beginning.

Example:

def generate_numbers():
    return [1, 2, 3, 4, 5]

print(generate_numbers())  # Returns the entire list at once

Output:

[1, 2, 3, 4, 5]

Iterators

An iterator in Python is an object that can be iterated over (i.e., it returns data one element at a time).

The iter() function can convert an iterable (like a list) into an iterator, and the next() function is used to retrieve elements from it.

Example:

numbers = [1, 2, 3, 4, 5]
iterator = iter(numbers)

print(next(iterator))  #

Output:
1
print(next(iterator)) #

Output:
2

Generators

Generators are a type of iterator, but instead of holding all the values in memory, they generate each value when needed using yield.

3. Creating Generators Using Generator Functions

A generator function is defined just like a normal function, but instead of using return, it uses the yield keyword.

When the function is called, it returns a generator object but does not start executing the function. The function executes only when you iterate over the generator or call next() on it.

Example 1: Simple Generator

def generate_numbers():
    for i in range(1, 6):
        yield i  # Generates one value at a time

# Create a generator object
numbers_gen = generate_numbers()

# Iterating through the generator
for number in numbers_gen:
    print(number)

Output:

1
2
3
4
5

In this example, each call to yield returns the next value, and the function’s state is preserved between successive calls.

Example 2: Using next() to Access Generator Items

You can manually access each item from the generator using the next() function.

numbers_gen = generate_numbers()

print(next(numbers_gen))  #

Output:
1
print(next(numbers_gen)) #

Output:
2

Output:

1
2

Calling next() beyond the available values will raise a StopIteration exception.

4. Creating Generators Using Generator Expressions

Generator expressions provide a concise way to create generators using a syntax similar to list comprehensions.

The key difference is that generator expressions use parentheses () instead of square brackets [].

Example: Generator Expression

# A generator that produces squares of numbers from 1 to 5
squares_gen = (x ** 2 for x in range(1, 6))

# Iterating through the generator
for square in squares_gen:
    print(square)

Output:

1
4
9
16
25

This expression generates each square value lazily, without creating the entire list in memory.

5. Working with yield and Controlling Generator State

Example: yield in Action

A generator can use multiple yield statements to produce different values in successive iterations.

def count_up_to(max_value):
    count = 1
    while count <= max_value:
        yield count
        count += 1

# Create a generator object
counter = count_up_to(3)

print(next(counter))  #

Output:
1
print(next(counter)) #

Output:
2
print(next(counter)) #

Output:
3

Output:

1
2
3

Each call to next() resumes the generator from where it left off, preserving the state of local variables (like count in this case).

Example: Using yield with return

Generators can also have a return statement, which terminates the generator and raises the StopIteration exception.

def generate_numbers():
    yield 1
    yield 2
    return "Done"
    yield 3  # This line is never reached

# Create a generator object
gen = generate_numbers()

print(next(gen))  #

Output:
1
print(next(gen)) #

Output:
2
try:
print(next(gen)) #

Output:
StopIteration with value “Done”
except StopIteration as e:
print(e.value)

Output:

1
2
Done

6. Practical Examples of Generators

Example 1: Reading Large Files Line by Line

Generators are perfect for working with large files, as they allow you to process one line at a time without loading the entire file into memory.

def read_large_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line.strip()

# Using the generator to read a large file
for line in read_large_file("large_file.txt"):
    print(line)

This is memory-efficient, especially for large files that could be gigabytes in size.

Example 2: Generating an Infinite Sequence

Generators can also be used to create infinite sequences since they compute values lazily. Here’s an example of a generator that produces an infinite series of Fibonacci numbers.

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Create a Fibonacci generator
fib_gen = fibonacci()

# Print the first 10 Fibonacci numbers
for _ in range(10):
    print(next(fib_gen))

Output:

0
1
1
2
3
5
8
13
21
34

The generator will continue producing values indefinitely, but you can control how many values to retrieve by limiting the iteration.

Example 3: Filtering Data with Generators

You can use generators to lazily filter data as it is being generated.

# Generator to filter even numbers
def even_numbers(max_value):
    for i in range(max_value):
        if i % 2 == 0:
            yield i

# Using the generator to get even numbers up to 10
for even in even_numbers(10):
    print(even)

Output:

0
2
4
6
8

This is particularly useful when working with large datasets, as you can filter values without storing the entire dataset in memory.

Conclusion

Generators are an efficient way to handle data streams and large datasets by generating values lazily. Here's a summary of what you’ve learned:

Generators are iterators that yield values one at a time.
Generator functions use yield to return values lazily, while generator expressions provide a shorthand way to create generators.
next() retrieves the next value from a generator, and when there are no more values, StopIteration is raised.
Generators are memory-efficient and can handle large data or infinite sequences efficiently.

Generators are a powerful tool in Python and can help you write more efficient and readable code, especially when dealing with data streams or large datasets.