with statement
Here is the syntax of the with statement:
with expression as var:
statements
The expression should be a context manager object, or it should produce a context manager object. When this with statement is executed, the first thing that happens is that the expression is evaluated and it gives an object which is a context manager. Now let us see what is a context manager.
A context manager is an object that follows the Context Management Protocol. This protocol states that an object should support __enter__ and __exit__ methods to be qualified as a context manager.
Any class that implements a context manager should have these two magic methods defined. When we instantiate such a class, the objects that we get are context managers.
class CM(): …………… def __enter__(self): pass def __exit__(self, exc_type, exc_value, traceback): pass
The __enter__ method takes only a single argument, which is self, and the __exit__ method takes three more arguments in addition to self.
So, instances of any class that defines __enter__ and __exit__ methods conform to the Context Management Protocol and thus can be used in the with statement.
Now, let us see the flow of control when the with statement executes. As we have seen, first of all the expression written after the with keyword is evaluated and we get a context manager object. After this the __enter__ method of this context manager is called. The value returned by the __enter__ method is assigned to the variable that is specified after the as keyword.
The value that is returned by __enter__ is something that we would like to use inside the with code block, this is generally the context manager itself, but it can be anything else also. So, mostly the __enter__ method returns self but it can return something else, too. The as keyword and the variable written after it are optional. If they are not present, the value returned by __enter__ is just discarded. This is why it is not necessary for __enter__ to return any value.
After the __enter__ method has finished executing and its return value has been assigned to var, the body of the with statement executes, so all the statements inside the with code block are executed.
After all the statements have been executed, the __exit__ method of the context manager is called and executed. So, this is how the with statement works. Here is a review of the control flow:
- Expression is evaluated to get a context manager object
- The __enter__ method of the context manager is executed
- Value returned by the __enter__ method is assigned to var
- Statements inside the with block are executed
- The __exit__ method of the context manager is executed
If an exception occurs during the execution of statements inside the with code block, then the rest of the statements in the with code block are skipped, and the control is transferred to __exit__. So, the __exit__ method is always called when the with code block is exited, no matter how the block is exited whether it is due to the end of the block, a return statement or an exception.
Like the finally block of the try statement, the context managers’s __exit__ method is guaranteed to be always called, and so you can place any cleanup code in this method.
Why we need with statement and context managers
Suppose in your program, you have different pieces of code where you have to interact with a database. Before executing the code that communicates with the database, you need to execute some setup code that connects to the database and, after you have finished working with the database, you need to execute some teardown code that disconnects the database and performs any cleanup actions.
You want to ensure that the cleanup code is executed no matter what happens, even if an exception occurs while working on the database. We have seen earlier that the finally blocks guarantee the execution of our cleanup code. So, we can put the teardown code in the finally block of the try statement.
The with statement provides a better alternative to the try...finally approach. The setup and teardown code will be the same every time you interact with the database. So, when you make your context manager class, you can place the repetitive setup and teardown code in it, and then there will be no need to repeat the code every time. The setup code goes in the __enter__ method, and the teardown code goes in the __exit__ method.
When the with statement is executed, we will get a context manager object which will be an instance of class CManager. First, the __enter__ method of this object will be executed, and so the setup code will execute. Then the statements inside the with code block will execute and after that the __exit__ will execute, so the teardown code will execute. Even if any exception occurs while interacting with the database in the with code block, the __exit__ method will be executed, and the database will be properly disconnected.
We can see that the code has become cleaner, less verbose, and more readable, and we also get the guarantee of execution of our cleanup code. The setup and teardown code can be lengthy and complex, and writing it every time you use the resource is not desirable; context managers help you avoid repeating the same code at many places, and at the same time, they give you the guarantee that the teardown code will definitely be executed.
We have moved the boilerplate entry and exit code in the context manager class, so we do not have to repeat it every time, and we can focus on the main task that we have to perform. The details of the setup and teardown code are hidden inside the context manager, and in your main program, only the database processing code is seen. So, we can abstract away most of the resource management logic by using a context manager.
The try...finally approach is more explicit; you can see the full code there, but that is why it is also more verbose. If you have to repeat it at many places in your program, then it increases the code size.
It is a very common thing to acquire a resource and then release it when we are done with it. We saw the example of connecting and disconnecting a database; the resource could be a network connection, file, lock, web transaction, or logged-in session, or we could temporarily change a setting in the program and then restore it back to the original, or we could start something like a timer and stop it automatically. In these types of scenarios, when there is some setup code and some teardown code that needs to be executed multiple times, you can create a context manager class and write the with statement. They provide us with a mechanism for automatic setup and teardown of resources.
So, they make our code more readable by simplifying the common resource management patterns. Of course, they help us avoid any resource leaks as they ensure that the resources are deallocated and default settings are restored in any case.
Here are some examples of cases where you need to execute some setup code and teardown code:
So, when you want to ensure execution of some special code before or after a piece of code, and you want to do this multiple times in your program, you can use a with statement. Context managers are in a way like decorators, they are used to surround code with some special code, but the difference is that decorators are used to wrap defined blocks of code like functions or classes, while with context managers, you can surround any arbitrary piece of code.
with statements are generally used when you want to temporarily acquire a resource, work with it, and release it, or when you have to restore some previous state that was temporarily changed for some time. However, with statements can be used only for objects that follow the context management protocol, while the try...finally approach can be used to perform cleanup actions for any sort of object.
Implementing a context manager by using a decorator on a generator
We can use either predefined context managers (like the file object) or create our own context managers. There are two ways to create our own context managers. They can be implemented either by writing classes or by writing decorated generator functions.
We have already seen how to write classes that can be instantiated to give context managers; in this section, we will see how to get context managers by using the second approach. This approach is simpler, but you need to have a basic knowledge of decorators and generators to implement it.
The contextlib module of the standard Python library contains a decorator called contextmanager. This decorator can be used on a generator function to create a factory of context managers. Let us see how this works.
We have defined a function; it contains the yield keyword, which makes it a generator function.
>>> def manager():
... print('Entering')
... yieldd
... print('Exiting')
When we call this generator function, we get a generator object.
>>> x = manager()
>>> type(x)
<class 'generator'>
>>> dir(x)
[…………, '__iter__', ………………, '__next__', ……………… ]
x is a generator object, as we can see the __iter__ and __next__ methods when dir function is used on it.
If we apply the contextmanager decorator to the generator function, then it will give us a context manager that has the __enter__ and __exit__ methods.
>>> from contextlib import contextmanager
>>> @contextmanager
... def manager():
... print('Entering')
... yield
>>> x = manager()
>>> type(x)
<class 'contextlib._GeneratorContextManager'>
>>> dir(x)
[………………, '__enter__', …………… , '__exit__', ………………]
Now the object that we get does not support the __iter__ and __next__ methods, but we can see the __enter__ and __exit__ methods, so it can be used in a with statement. In the following code, we have written two simple with statements that make use of the context manager returned by this function:
from contextlib import contextmanager @contextmanager def manager(): print('Entering') yield print('Exiting') with manager(): print('xxx') print('yyy') print() with manager(): print('Python') print(4 + 15 / 3) print('Runtime') print('Context')
Output-
Entering
xxx
yyy
Exiting
Entering
Python
9.0
Runtime
Context
Exiting
Let us see how this works. When the with statement is executed, first the code before the yield statement is executed, then when the yield statement is encountered, the execution of the function is stopped temporarily, and the control is transferred to the with code block. The with code block is executed, and when it finishes executing, the control returns to the function and whatever is there after the yield, is executed.
Whatever we write before the yield statement executes before the execution of with code block, so we can write our setup code there. Whatever we write after the yield statement executes after the execution of with code block, so the teardown code can be written after yield. So before yield, we will write the code that would have gone in the __enter__ method had we written a class, and after the yield we would write the code that would have gone in the __exit__ method.
We have seen that we can have an optional as keyword in the with statement. So, suppose we have the as keyword and a variable after it in our with statement.
with manager() as var: print('xxx') print('yyy') print(var)
We know that we can use the variable var in the with code block to interact with the context. We have printed it inside the with code block. When we run our code with this with statement, we will see that None is printed for the variable var.
Now, let us change our manager function. If we write the yield keyword with a value after it, then that value will be bound to the variable placed after the as keyword in the with statement. This is equivalent to returning a value from the __enter__ method, if you were writing a class.
@contextmanager def manager(): print('Entering') x = 5 yield x print('Exiting')
Here, we have specified variable x in the yield statement, so now the variable var in our with statement will be bound to this variable x. Now when we run our previous with statement, we get 5 printed for variable var.
So, if you want your function to give out a value that can be assigned to the target variable in the as clause, then instead of writing a plain yield, make your yield produce a value.
Now, let us see what happens when an exception is raised inside the with code block.
from contextlib import contextmanager @contextmanager def manager(): print('Entering') yield print('Exiting') with manager(): print('Python') print(4 + 15 / 0) print('Runtime') print('Context')
Output-
Entering
Python
Traceback (most recent call last):
File "E:\Deepali\Programs\21_ContextManager\P21_20.py", line 11, in <module>
print(4 + 15 / 0)
ZeroDivisionError: division by zero
A ZeroDivisionError will be raised and since it is not handled, it is passed to the yield expression. The yield statement reraises the exception, but there is no error handling, so we will have abnormal termination and the teardown code will not be executed. The program is abnormally terminated.
Thus, the proper way to write the generator function is to enclose the yield statement in a try block and write the teardown code in the finally block. This ensures the execution of teardown code even when an exception occurs.
from contextlib import contextmanager @contextmanager def manager(): print('Entering') x = 5 try: yield x finally: print('Exiting') with manager(): print('Python') print(4 + 15 / 0) print('Runtime') print('Context')
Output-
Entering
Python
Exiting
Traceback (most recent call last):
File "E:\Deepali\Programs\21_ContextManager\P21_21.py", line 14, in <module>
print(4 + 15 / 0)
ZeroDivisionError: division by zero
The teardown code is now inside finally block so it is executed even when an exception occurred in the with block. The program was terminated because the exception could not find an appropriate handler.
Whenever an unhandled exception occurs in the with code block, it is reraised by yield inside the generator. If we want to handle the exception, we can write except blocks inside the generators which can handle the exception and suppress it, or we can just partially handle the exception and then reraise it again. If the exception is totally handled and suppressed inside the generator, then the execution will resume with the statement that immediately follows the with statement in which the exception occurred.
Here is the proper format for writing a decorated generator function that gives us a context manager:
from contextlib import contextmanager @contextmanager def manager(): SET UP CODE try: yield x finally: TEARDOWN CODE
In the beginning of the function, we write the setup code. Inside try block we write the yield expression, the value produced by yield will be assigned to the target variable in the as clause. In the finally block, we write the teardown code. There should be only one yield statement inside the function and at this point the with code block execution starts.
