• Create Account

\$8

Like
7Likes
Dislike

Advanced Python Part One: List Comprehensions

By Thomas Hobohm | Published Aug 16 2013 01:41 PM in General Programming

advanced python python python2 python3 list comprehensions lists programming

This is an extension of my blog article about list comprehensions and the first part of an advanced python series. We will cover List Comprehensions, Lambdas, Functional Programming, Map / Reduce / Filter, Decorators, and Descriptors.

Python List Comprehensions

List comprehensions are Python's solution to the large loops required to do tasks which should be easy. They transform the simple task of getting all the even numbers between zero and one-thousand from:

my_list = []
for index in range(1000):
if index % 2 == 0:
my_list.append(index)


to

my_list = [x for x in range(1000) if x % 2 == 0]


The use for list comprehensions

List comprehensions are essentially:

list_name = [expression values filter]


Values - A for loop of values to check against the filter
Filter - All values go through this. It is an if statement, and the expression is appended to the end of the list when it is true.
Expression - Data to append onto the list if Filter is true

It's important to note here that the expression could be anything. We could get the squares of all even numbers between zero and one-thousand just as easily:

my_list = [x**2 for x in range(1000) if x % 2 == 0]


Interesting Points

List comprehensions work on strings, and the for loop can loop through any list. The expression can be as complex as you want. The expression is what gets appended, and it can even be another list. Look at this piece of code:

my_list = [[x*y for y in range(1,11) if y % 2 == 0] for x in range(1,11) if x % 2 == 0]


This generates a list of lists (which is stored in my_list) where every even number between one and ten is multiplied by every other even number between one and ten.

Conclusion

To practice list comprehensions you can try out Code Academy's interactive course. You can also learn more by looking at the Python Docs List Comprehensions are a very useful tool which can replace many loops if you look out for it.

Cheers :)!

Article Update Log

June 20 2013 - Released Article

I'm a programmer who worked with C++ for two years, followed by one and a half years using SFML 1.6 / 2. For the past few months I've used Python and am just starting Pygame.

In my_list = [x**2 for x in range(1000) if x % 2 == 0]

What does x**2 mean?

In my_list = [x**2 for x in range(1000) if x % 2 == 0]

What does x**2 mean?

To the Power: 2 ** 2 = 4

In my_list = [x**2 for x in range(1000) if x % 2 == 0]

What does x**2 mean?

To the Power: 2 ** 2 = 4

Oh. That's a nice operator. Neither C++ nor Java has this type of operator

I'm really starting to like Python!

This is a nice article but it is a bit short.

Here is some code that uses list comprehensions to filter all of the jpg files in the current directory:

import re
import os

filtered_list = [f for f in os.listdir(os.getcwd()) if re.match(r".*\.jpg", f, re.IGNORECASE)]

for filename in filtered_list:
print(filename)

Without list comprehensions it would look like this:

import re
import os

filtered_list = []

for f in os.listdir(os.getcwd()):
if re.match(r".*\.jpg", f, re.IGNORECASE):
filtered_list.append(f)

for filename in filtered_list:
print(filename)

and yes, I know I could compile the regular expression but this is just an example .

Such things like list comprehensions, decorators, ect can be useful in the right context, but they can also be misused. I will even go so far as to say that in a good majority of situations you should avoid using such things unless you understand enough of what you are doing and have a well thought out reason to introduce them.

Keep in mind that one of the wonderful things about Python is the readability. You can do some neat things with Python but it can make others on your team or other people not as able to understand the code that you have written. This may not be a concern for everyone, but it is a concern that I deal with when working with Python.

For instance in the example I posted previously the second code is really in my mind easier to understand even though it is more lines of Python. We saved some line count but now we have to spend more time to read and understand the code.

Even with the ** operator it is not immediately clear that this raises something to a power, where pow(x, 2) can be more clear (pow is similar to ** but it is NOT the same as math.pow... I know that is confusing).

Just remember that the brilliant code that you write today will be an annoying puzzle to you in the future .

python is awsome!

Shadowisadog, I would be very surprised if someone didn't know what the ** operator did. It is one of the operators taught in all the beginning courses I've taken, and if someone hasn't studied python enough to know what ** means (or doesn't know how to google it) then I shouldn't be importing the math module and using an extra function just in case.

List comprehensions are almost always better than the longer and harder to read solutions. I haven't ever looked at a list comprehensions and not known what it did, and that's because I've studied them.

As another example, in the pygame library here's how you get events:

import pygame

# List to hold keydown events
keydown_events = []

# Game Loop
while True:
# Get all the keydown events
for event in pygame.event.get():
if event.type == pygame.KEYDOWN:
keydown_events.append(event)


it could be easily simplified to:

import pygame

keydown_events = [event for event in pygame.event.get() if event.type == pygame.KEYDOWN]


Once someone learns about list comprehensions I doubt that there are ones that they can't understand just because they're too complicated.

Cheers !

(Also, sorry for downvoting you. It was spur of the moment and unfortunately gamedev.net won't let me take it back :0!)

You should mention:

• that there are memory-friendly generator expressions, which can replace many list uses
• that there are dict and set comprehensions with a very similar syntax
• that multiple for and if clauses are allowed, and obviously very useful to navigate and join data structures
• that such nested clauses are evaluated left to right (the most important non-obvious detail)

Clumsy style:

my_list = [[x*y for y in range(1,11) if y % 2 == 0] for x in range(1,11) if x % 2 == 0]

Almost normal style:

my_list = [x*y for x in range(1,11) if x % 2 == 0 for y in range(1,11) if y % 2 == 0]
my_list = [x*y for x in range(1,11) for y in range(1,11) if if (x % 2 == 0 and y % 2 == 0)]

Good style:

my_list = [x*y for x in range(2,11,2) for y in range(2,11,2)]
• More practical and developed examples would also be an improvement.

Shadowisadog, I would be very surprised if someone didn't know what the ** operator did. It is one of the operators taught in all the beginning courses I've taken, and if someone hasn't studied python enough to know what ** means (or doesn't know how to google it) then I shouldn't be importing the math module and using an extra function just in case.

pow is not part of math. I mentioned that there is pow and math.pow. I think using pow is clearer because it is more explicit and it is can prevent typos where you accidentally type * instead of **. Someone on these very comments specifically asked what ** as well so I guess feel free to be surprised?

List comprehensions are almost always better than the longer and harder to read solutions. I haven't ever looked at a list comprehensions and not known what it did, and that's because I've studied them.

I disagree from the standpoint that I think the longer solution is easier to read. The list comprehension approach may save line count and avoid "extra" code, but I do not think it is easier to read. List comprehensions may not be hard for you to read but that is precisely because you have studied them! This is an "advanced python" article after all?

As another example, in the pygame library here's how you get events:

import pygame

# List to hold keydown events
keydown_events = []

# Game Loop
while True:
# Get all the keydown events
for event in pygame.event.get():
if event.type == pygame.KEYDOWN:
keydown_events.append(event)


it could be easily simplified to:

import pygame

keydown_events = [event for event in pygame.event.get() if event.type == pygame.KEYDOWN]


Once someone learns about list comprehensions I doubt that there are ones that they can't understand just because they're too complicated.

I am not sure that is a great simplification from the stand point that I may want to add extra event types (and probably would in context). I think the code would be simpler if I did not use a list comprehension in this context... because then I might repeat it for the KEYUP, ect and then I am repeating the event for event in pygame.event.get() ?

Cheers !

(Also, sorry for downvoting you. It was spur of the moment and unfortunately gamedev.net won't let me take it back :0!)

No worries.

Shadowisadog, I would be very surprised if someone didn't know what the ** operator did. It is one of the operators taught in all the beginning courses I've taken, and if someone hasn't studied python enough to know what ** means (or doesn't know how to google it) then I shouldn't be importing the math module and using an extra function just in case.

pow is not part of math. I mentioned that there is pow and math.pow. I think using pow is clearer because it is more explicit and it is can prevent typos where you accidentally type * instead of **. Someone on these very comments specifically asked what ** as well so I guess feel free to be surprised?

List comprehensions are almost always better than the longer and harder to read solutions. I haven't ever looked at a list comprehensions and not known what it did, and that's because I've studied them.

I disagree from the standpoint that I think the longer solution is easier to read. The list comprehension approach may save line count and avoid "extra" code, but I do not think it is easier to read. List comprehensions may not be hard for you to read but that is precisely because you have studied them! This is an "advanced python" article after all?

As another example, in the pygame library here's how you get events:

import pygame

# List to hold keydown events
keydown_events = []

# Game Loop
while True:
# Get all the keydown events
for event in pygame.event.get():
if event.type == pygame.KEYDOWN:
keydown_events.append(event)


it could be easily simplified to:

import pygame

keydown_events = [event for event in pygame.event.get() if event.type == pygame.KEYDOWN]


Once someone learns about list comprehensions I doubt that there are ones that they can't understand just because they're too complicated.

I am not sure that is a great simplification from the stand point that I may want to add extra event types (and probably would in context). I think the code would be simpler if I did not use a list comprehension in this context... because then I might repeat it for the KEYUP, ect and then I am repeating the event for event in pygame.event.get() ?

Cheers !

(Also, sorry for downvoting you. It was spur of the moment and unfortunately gamedev.net won't let me take it back :0!)

No worries.

Well, for getting various types of events:

import pygame

keyboard_events = [event for event in pygame.event.get() if event.type == pygame.KEYDOWN or event.type == pygame.KEYUP]


List comprehensions are also rapidly replacing things like Map, Replace, and Filter. I honestly would be very confused as to why someone wouldn't understand list comprehensions unless they were learning from an outdates source, also, considering almost every tutorial I've looked at (including Code Academy's) has included sections on it.

Also, the whole entire power argument doesn't make sense. All pow does is call **, and it is one of the basic operators of the language. Even if someone didn't know what it did, a quick google of "Python operators" would help.

Everything LorenzoGatti said.

A treatment of list comprehensions that is named "advanced" really needs to explain generator expressions, the relation of the one to the other, and the performance tradeoffs of the two approaches.

I'd also expect to see some discussion of iterators and generators, and how they relate to the above.

(Or, potentially, renaming this this to "Beginners Python")

This really was meant to be an introduction, not to cover every single aspect of list comprehensions.

Nevertheless, I'm going to edit in a link to the python Documentation on the subject.