Galaxyblog224

2019-04-30

Code coverage is a simple tool for checking which lines of your application code are run by your test suite.100% coverage is a laudable goal, as it means every line is run at least once.

Coverage.py is the Python tool for measuring code coverage.Ned Batchelder has maintained it for an incredible 14 years!

I like adding Coverage.py to my Django projects, like fellow Django Software Foundation member Sasha Romijn.

Let’s look at how we can integrate it with a Django project, and how to get that golden 100% (even if it means ignoring some lines).

Source code for django.test.runner. Import logging import os import unittest from importlib import importmodule from unittest import TestSuite, defaultTestLoader from django.conf import settings from django.core.exceptions import ImproperlyConfigured from django.test import SimpleTestCase, TestCase from django.test.utils import setuptestenvironment, teardowntestenvironment from django. Adds support for running Django tests in Visual Studio Code. Provides shortcuts to run closest method, class, file, app and previous tests. Provides support for Django-Nose in settings. Draws inspiration from vscode-django-tests and vim-python-test-runner. In order to guarantee that all TestCase code starts with a clean database, the Django test runner reorders tests in the following way: All TestCase subclasses are run first. Then, all other Django-based tests (test cases based on SimpleTestCase, including TransactionTestCase ) are run with no particular ordering guaranteed nor enforced among them. The first place to look is the test management command, which Django finds and executes when we run manage.py test. This lives in django.core.management.commands.test. As management commands go, it’s quite short - under 100 lines. Its handle method is mostly concerned with handing off to a a “Test Runner”. This VS Code extension gives quick access to running Django tests by invoking python manage.py test with the VS Code action Django Test Runner: Run Tests or a keyboard shortcut. This will run tests in a VS Code terminal. You can optionally display the status of tests by configuring an XMLRunner test report.

Configuring Coverage.py¶

Install coverage with pip install coverage.It includes a C extension for speed-up, it’s worth checking that this installs properly - see the installation docs for information.

Then set up a configuration file for your project.The default file name is .coveragerc, but since that’s a hidden file I prefer to use the option to store the configuration in setup.cfg.

This INI file was originally used only by setuptools but now many tools have the option to read their configuration from it.For Coverage.py, we put our settings there in sections prefixed with coverage:.

The Run Section¶

This is where we tell Coverage.py what coverage data to gather.

We tell Coverage.py which files to check with the source option.In a typical Django project this is as easy as specifying the current directory (source = .) or the app directory (source = myapp/*).Add it like so:

(Remove the coverage: if you’re using .coveragerc.).

An issue I’ve seen on a Django project is Coverage.py finding Python files from a nested node_modules.It seems Python is so great even JavaScript projects have a hard time resisting it!We can tell coverage to ignore these files by adding omit = */node_modules/*.

When you come to a fork in the road, take it.

—Yogi Berra

An extra I like to add is branch coverage.This ensures that your code runs through both the True and False paths of each conditional statement.You can set this up by adding branch = True in your run section.

As an example, take this code:

With branch coverage off, we can get away with tests that pass in a red widget.Really, we should be testing with both red and non-red widgets.Branch coverage enforces this, by counting both paths from the if.

The Report Section¶

This is where we tell Coverage.py how to report the coverage data back to us.

I like to add three settings here.

1. fail_under = 100 requires us to reach that sweet 100% goal to pass.If we’re under our target, the report command fails.
2. show_missing = True adds a column to the report with a summary of which lines (and branches) the tests missed.This makes it easy to go from a failure to fixing it, rather than using the HTML report.
3. skip_covered = True avoids outputting file names with 100% coverage.This makes the report a lot shorter, especially if you have a lot of files and are getting to 100% coverage.

Add them like so:

(Again, remove the coverage: prefix if you’re using .coveragerc.)

Template Coverage¶

Your Django project probably has a lot of template code.It’s a great idea to test its coverage too.This can help you find blocks or whole template files that the tests didn’t run.

Lucky for us, the primary plugin listed on the Coverage.py plugins page is the Django template plugin.

See the django_coverage_plugin PyPI page for its installation instructions.It just needs a pip install and activation in [coverage:run].

Git Ignore¶

If your project is using Git, you’ll want to ignore the files that Coverage.py generates.GitHub’s default Python .gitignore already ignores Coverage’s file.If your project isn’t using this, add these lines in your .gitignore:

Using Coverage in Tests¶

This bit depends on how you run your tests.I prefer using pytest with pytest-django.However many, projects use the default Django test runner, so I’ll describe that first.

With Django’s Test Runner¶

Django Test Case

If you’re using manage.py test, you need to change the way you run it.You need to wrap it with three coverage commands like so:

99% - looks like I have a little bit of work to do on my test application!

Having to run three commands sucks.That’s three times as many commands as before!

We could wrap the tests with a shell script.You could add a shell script with this code:

Update (2020-01-06):Previously the below section recommended a custom test management command.However, since this will only be run after some imports, it's not possible to record 100% coverage this way.Thanks to Hervé Le Roy for reporting this.

However, there’s a more integrated way of achieving this inside Django.We can patch manage.py to call Coverage.py’s API to measure when we run the test command.Here’s how, based on the default manage.py in Django 3.0:

Notes:

1. The two customizations are the blocks before and after the execute_from_command_line block, guarded with if running_tests:.

2. You need to add manage.py to omit in the configuration file, since it runs before coverage starts.For example:

   (It's fine, and good, to put them on multiple lines.Ignore the furious red from my blog's syntax highlighter.)

3. The .report() method doesn’t exit for us like the commandline method does.Instead we do our own test on the returned covered amount.This means we can remove fail_under from the [coverage:report] section in our configuration file.

Run the tests again and you'll see it in use:

Yay!

(Okay, it’s still 99%.Spoiler: I’m actually not going to fix that in this post because I’m lazy.)

With pytest¶

It’s less work to set up Coverage testing in the magical land of pytest.Simply install the pytest-cov plugin and follow its configuration guide.

The plugin will ignore the [coverage:report] section and source setting in the configuration, in favour of its own pytest arguments.We can set these in our pytest configuration’s addopts setting.For example in our pytest.ini we might have:

(Ignore the angry red from my blog’s syntax highlighter.)

Run pytest again and you’ll see the coverage report at the end of the pytest report:

Hooray!

(Yup, still 99%.)

Browsing the Coverage HTML Report¶

The terminal report is great but it can be hard to join this data back with your code.Looking at uncovered lines requires:

1. Remembering the file name and line numbers from the terminal report
2. Opening the file in your text editor
3. Navigating to those lines
4. Repeat for each set of lines in each file

This gets tiring quickly!

Coverage.py has a very useful feature to automate this merging, the HTML report.

After running coverage run, the coverage data is stored in the .coverage file.Run this command to generate an HTML report from this file:

This creates a folder called htmlcov.Open up htmlcov/index.html and you’ll see something like this:

Click on an individual file to see line by line coverage information:

The highlighted red lines are not covered and need work.

Django itself uses this on its Jenkins test server.See the “HTML Coverage Report” on the djangoci.com project django-coverage.

With PyCharm¶

Coverage.py is built-in to this editor, in the “Run <name> with coverage” feature.

This is great for individual development but less so for a team as other developers may not use PyCharm.Also it won’t be automatically run in your tests or your Continuous Integration pipeline.

See more in this Jetbrains feature spotlight blog post.

Is 100% (Branch) Coverage Too Much?¶

Some advocate for 100% branch coverage on every project.Others are skeptical, and even believe it to be a waste of time.

For examples of this debate, see this Stack Overflow question and this one.

Like most things, it depends.

First, it depends on your project’s maturity.If you’re writing an MVP and moving fast with few tests, coverage will definitely slow you down.But if your project is supporting anything of value, it’s an investment for quality.

Second, it depends on your tests.If your tests are low quality, Coverage won’t magically improve them.That said, it can be a tool to help you work towards smaller, better targeted tests.

100% coverage certainly does not mean your tests cover all scenarios.Indeed, it’s impossible to cover all scenarios, due to the combinatorial explosion from multiplying branches.(See all-pairs testing for one way of tackling this explosion.)

Third, it depends on your code.Certain types of code are harder to test, for example branches dealing with concurrent conditions.

IF YOU’RE HAVING CONCURRENCY PROBLEMS I FEEL BAD FOR YOU SON

99 AIN’T GOT I BUT PROBLEMS CONCURRENCY ONE

—[@quinnypig on Twitter](https://twitter.com/QuinnyPig/status/1110567694837800961)

Some tools, such as unittest.mock, help us reach those hard branches.However, it might be a lot of work to cover them all, taking time away from other means of verification.

Fourth, it depends on your other tooling.If you have good code review, quality tests, fast deploys, and detailed monitoring, you already have many defences against bugs.Perhaps 100% coverage won’t add much, but normally these areas are all a bit lacking or not possible.For example, if you’re working a solo project, you don’t have code review, so 100% coverage can be a great boon.

To conclude, I think that coverage is a great addition to any project, but it shouldn’t be the only priority.A pragmatic balance is to set up Coverage for 100% branch coverage, but to be unafraid of adding # pragma: no cover.These comments may be ugly, but at least they mark untested sections intentionally.If no cover code crashes in production, you should be less surprised.

Also, review these comments periodically with a simple search.You might learn more and change your mind about how easy it is to test those sections.

Fin¶

Go forth and cover your tests!

If you used this post to improve your test suite, I’d love to hear your story.Tell me via Twitter or email - contact details are on the front page.

—Adam

Thanks to Aidas Bendoraitis for reviewing this post.

🎉 My book Speed Up Your Django Tests is now up to date for Django 3.2. 🎉
Buy now on Gumroad

One summary email a week, no spam, I pinky promise.

Related posts:

Tags:django

© 2019 All rights reserved.

Testing is an important but often neglected part of any Django project. In this tutorial we'll review testing best practices and example code that can be applied to any Django app.

Broadly speaking there are two types of tests you need to run:

• Unit Tests are small, isolated, and focus on one specific function.
• Integration Tests are aimed at mimicking user behavior and combine multiple pieces of code and functionality.

While we might we use a unit test to confirm that the homepage returns an HTTP status code of 200, an integration test might mimic the entire registration flow of a user.

For all tests the expectation is that the result is either expected, unexpected, or an error. An expected result would be a 200 response on the homepage, but we can--and should--also test that the homepage does not return something unexpected, like a 404 response. Anything else would be an error requiring further debugging.

The main focus of testing should be unit tests. You can't write too many of them. They are far easier to write, read, and debug than integration tests. They are also quite fast to run.

Complete source code is available on Github.

When to run tests

The short answer is all the time! Practically speaking, whenever code is pushed or pulled from a repo to a staging environment is ideal. A continuous integration service can perform this automatically. You should also re-run all tests when upgrading software packages, especially Django itself.

Layout

By default all new apps in a Django project come with a tests.py file. Any test within this file that starts with test_ will be run by Django's test runner. Make sure all test files start with test_.

As projects grow in complexity, it's recommended to delete this initial tests.py file and replace it with an app-level tests folder that contains individual tests files for each area of functionality.

For example:

Sample Project

Let's create a small Django project from scratch and thoroughly test it. It will mimic the message board app from Chapter 4 of Django for Beginners.

On the command line run the following commands to start our new project. We'll place the code in a folder called testy on the Desktop, but you can locate the code anywhere you choose.

Now update settings.py to add our new pages app and configure Django to look for a project-level templates folder.

Create our two templates to test for a homepage and about page.

Populate the templates with the following simple code.

Update the project-level urls.py file to point to the pages app.

Create a urls.py file within the pages app.

Then update it as follows:

And as a final step add our views.

Start up the local Django server.

Then navigate to the homepage at http://127.0.0.1:8000/ and about page at http://127.0.0.1:8000/about to confirm everything is working.

Time for tests.

Django Test Suite Runner

SimpleTestCase

Our Django application only has two static pages at the moment. There's no database involved which means we should use SimpleTestCase.

We can use the existing pages/tests.py file for our tests for now. Take a look at the code below which adds five tests for our homepage. First we test that it exists and returns a 200 HTTP status code. Then we confirm that it uses the url named home. We check that the template used is home.html, the HTML matches what we've typed so far, and even test that it does not contain incorrect HTML. It's always good to test both expected and unexpected behavior.

Now run the tests.

They should all pass.

As an exercise, see if you can add a class for AboutPageTests in this same file. It should have the same five tests but will need to be updated slightly. Run the test runner once complete. The correct code is below so try not to peak...

Message Board app

Now let's create our message board app so we can try testing out database queries. First create another app called posts.

Add it to our settings.py file.

Then run migrate to create our initial database.

Now add a basic model.

Create a database migration file and activate it.

For simplicity we can just a post via the Django admin. So first create a superuser account and fill in all prompts.

Update our admin.py file so the posts app is active in the Django admin.

Then restart the Django server with python manage.py runserver and login to the Django admin at http://127.0.0.1:8000/admin/. You should see the admin’s login screen:

Click on the link for + Add next to Posts. Enter in the simple text Hello world!.

Django Nose Test Runner

On 'save' you'll see the following page.

Now add our views file.

Create a posts.html template file.

And add the code below to simply output all posts in the database.

Finally, we need to update our urls.py files. Start with the project-level one located at myproject/urls.py.

Then create a urls.py file in the posts app.

And populate it as follows.

Django Run Tests

Okay, phew! We're done. Start up the local server python manage.py runserver and navigate to our new message board page at http://127.0.0.1:8000/posts.

It simply displays our single post entry. Time for tests!

TestCase

TestCase is the most common class for writing tests in Django. It allows us to mock queries to the database.

Let's test out our Post database model.

Django Test Runner Review

With TestCase the Django test runner will create a sample test database just for our tests. Here we've populated it with the text 'just a test'.

In the first test we confirm that the test entry has the primary id of 1 and the content matches. Then in the second test on the view we confirm that that it uses the url name posts, has a 200 HTTP response status code, contains the correct text, and uses the correct template.

Vscode Django Test Runner

Run the new test to confirm everything works.

Next Steps

There is far more testing-wise that can be added to a Django project. A short list includes:

• Continuous Integration: automatically run all tests whenever a new commit is made, which can be done using Github Actions or a service like Travis CI.
• pytest: pytest is the most popular enhancement to Django and Python's built-in testing tools, allowing for more repeatable tests and a heavy use of fixtures.
• coverage: With coverage.py you can have a rough overview of a project's total test coverage.
• Integration tests: useful for testing the flow of a website, such as authentication or perhaps even payments that rely on a 3rd party.

I'm working on a future course on advanced Django testing so make sure to sign up for the LearnDjango newsletter below to be notified when it's ready.

If you need more testing help now, an excellent course is Test-Driven Development with Django, Django REST Framework, and Docker by my friend Michael Herman.