Processing large files with django celery tasks

Processing large files with django celery tasks

My goal it to process a large CSV file using Celery that is uploaded
through a Django form. When the file's size is less than
SETTINGS.FILE_UPLOAD_MAX_MEMORY_SIZE, I can pass the form's cleaned_data
variable to a celery task and read the file with:
@task
def taskFunction(cleaned_data):
for line in csv.reader(cleaned_data['upload_file']):
MyModel.objects.create(field=line[0])
However, when the file's size is greater than the above setting, I get the
following error:
expected string or Unicode object, NoneType found
Where the stack trace shows the error occurring during pickle:
return dumper(obj, protocol=pickle_protocol)
It appears that when the uploaded file is read from a temporary file,
pickle fails.
The simple solution to this problem is to increase the
FILE_UPLOAD_MAX_MEMORY_SIZE. However, I am curious if there is a better
way to manage this issue?