
Instead of rolling my own progress bars on new batch management commands, I've been using `tqdm` lately. Here's a quick snippet for multiprocessing a batch of items in a queryset.

You'll need tqdm to run this: `pip install tqdm`

**Note:** Since the function that I'm calling is using django's ORM, I close all db connections. Things tend to go wrong when forked processes are all using the same connection.

```python
from django.db import connection
from django.core.management.base import BaseCommand

from multiprocessing import Pool
from tqdm import tqdm

from project.models import Item

def func(item):
    connection.close()
    # Do some process
    return True


class Command(BaseCommand):

    def add_arguments(self, parser):
        parser.add_argument(
            '--processes',
            dest='processes',
            default=2,
            type=int,
            help='Number of processes to use to make run command on queryset',
        )

    def handle(self, *args, **options):
        items = Item.objects.all()

        with tqdm(total=items.count()) as bar, Pool(processes=options['processes']) as pool:
            for _ in pool.imap_unordered(func, items):
                bar.update()
```


Django Mulitprocessing with Progress Bar


In Django, fields with `auto_now=True` are not always updated correctly. A common practice to work around this is to override the `save()` method on the model, or to use a pre-save hook. However, this doesn't work when calling `Model.objects.update()` since `save()` is not called on all instances in the queryset.

A common way to fix this would be to alter the table, for example, in MySQL it may look like this:

```sql
ALTER TABLE app_model
MODIFY updated_at
DEFAULT CURRENT_TIMESTAMP
ON UPDATED CURRENT_TIMESTAMP
```

But, this doesn't translate well into a migration and doesn't work with sqlite3 and other db backends.

I have found a simple way to fix this which will work with any database backend, so you don't need to write migrations or alter your database tables. We can do this with a simple queryset method override. All we are doing here is adding the current timestamp to the update kwargs, so all instances in the queryset will get that update!

```python
from django.db.models import query, manager
from django.utils import timezone

class BaseQuerySet(query.QuerySet):

    def update(self, **kwargs):
        kwargs['updated_at'] = timezone.now()
        super().update(**kwargs)

class Manager(manager.BaseManager.from_queryset(BaseQuerySet)):
    pass
```

Then to use this, just add it to your model like so.

```python
from django.db import models
from .manager import Manager

class BaseModel(models.Model):
    objects = Manager()
    updated_at = models.DateTimeField()
```

Hope this helps, and thanks for reading!


Auto-updated timestamp fields in Django bulk querysets


Django querysets are really powerful. They can often be used to reduce expensive object iterations down to the database level. Let's take the following models and figure out which product has the least in stock.

```python
from django.db import models

class Product(models.Model):
inventory = models.IntegerField()

class Package(models.Model):
items = models.ManyToMany(Product)

class Order(models.Model):
customer = models.ForeignKey(Customer)
package = models.ForeignKey(Package)
```

The naiive implementation may look something like this. The loops and temp variables probably remind you of something you wrote in CS 101.

```python
def least_stock():
  # Which product has the least in stock?
  least = None
  least_stock = 100000

  for product in Product.objects.all():
      ordered = sum([p.orders.count() for p in product.packages.all()])
      in_stock = product.inventory - ordered
      if in_stock < least_stock:
          least = product
          least_stock = in_stock
  return least
```

This method is extremely expensive. Notice how it first queries for all of the packages that contain that product. Then for each of those packages, it runs a count to get the number of orders. This is `n + 1` database queries, where `n` is the number of packages that a product is in. I would run some benchmarking on this, but it is obvious that the running time would be horrible.

We can definitely do better, using a powerful queryset function built into django: `annotate`. Annotating allows you to add fields to the query using various functions that translate down to SQL. Using queryset functions, the sum becomes a count, the loop becomes an annotation, and the comparison beecomes an order_by. All of which is done at that database level.

```python
from django.db.models import Count, F

def least_stock():
    # Which product has the least in stock?
    return Product.objects.annotate(
        ordered=Count('packages__orders')
    ).annotate(
        in_stock=F('inventory') - F('ordered')
    ).order_by('in_stock').first()
```

This implementation uses annotate twice to significantly reduce the number of queries on the database. In fact, this results in a single query!

The [Django Docs page on queries](https://docs.djangoproject.com/en/1.11/topics/db/queries/) has many more examples. Using annotations and aggregates can siginificantly improve the performance of your django site.


Django Queryset Annotations for Increased Performance


A problem that many applications (hopefully) eventually run into is scalability. In a project I'm working on, we have a django management command that makes a time consuming API call to update a model. Let's take a look at the original code:

```python
class Command(BaseCommand):
    help = 'Create Shipping labels for upcoming shipments'

    def handle(self, *args, **options):
        start = datetime.date.today()
        end = start + datetime.timedelta(14)

        for shipment in Shipment.objects.filter(shipping_date__range=[start, end]):
            create_shipping_label(shipment)
```

At first, this seems like an innocent little script. It does exactly what it is supposed to do. For every shipment that is upcoming, let's create the shipping label.

Notice anything wrong with it though? What if we now have 20,000 shipments? Even if the API call takes only 1 second, that's still about 6 hours of processing time, for one seemingly innocent command.

Enter multiprocessing. As you may know, computers these days have anywhere from 2 to 16 cores in the CPU. When this script above is running, it's really only using one! By using the stdlib package for multiprocessing, we can run this script up to 16x faster. (8x faster on my i7, 4x faster on many common CPUs).

Here's the updated command to take advantage of multiprocessing:

from multiprocessing import cpu_count, Pool

```python
class Command(BaseCommand):
    help = 'Create Shipping labels for upcoming shipments'

    def handle(self, *args, **options):
        start = datetime.date.today()
        end = start + datetime.timedelta(14)

        shipments = Shipment.objects.filter(shipping_date__range=[start, end]):

        pool = Pool(processes=cpu_count())
        pool.map(create_shipping_label, shipments)
```

Those two lines of code create a thread pool of as many CPUs as possible, and simply map the iterator onto the function. Now that script can run 4 or more of those API calls at the same time, reducing the running time of the script significantly


Mutiprocessing in Python Django Management Commands


A problem I recently ran into was allowing for multiple themes to be used on the same project. I decided to set a django setting called `BRAND` in order to control the theme through separate css files.

For this example, we will assume that your static directory has sub-directories for each brand, with a file `site.css` in each.

Now, in order to inject our brand setting into the context, we need to add a context processor. Create a file called `context.py` in your project.

```python
from django.conf import settings

def brand_details(request):
    return {'BRAND': settings.BRAND}
```

This simple function injects the `BRAND` setting into the template context. Next we need to add the context processor to our template settings.

```python
BRAND = 'brand1'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['your-template-dir'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                ...
                'project.context.brand_details',
            ],
        },
    },
]
```

Lastly, we need to use the injected variable to pull out the correct theme. In the head of your base.html file, we will use the variable to get the correct brand css file.

```html
{% load staticfiles %} {% with 'css/'|add:BRAND|add:'/site.scss' as brand_style
%}
<link rel="stylesheet" type="text/x-scss" href="{% static brand_style %}" />
{% endwith %}
```

There you have it! Django themes with a simple site setting. If you wanted to take this a step further, you could use the variable to pull in base templates. Also, that context processor function could run a query to get the theme, allowing you to edit the theme from the admin instead of the settings.py.
