RSS

Blog

Django file fields handling: deal with the mess

h3 ~ 17 Mar 2011 – Comments (49)

UPDATE: Django 1.3 has been released and this behavior change was listed in the backward incompatible changes:

FileField no longer deletes files

In earlier Django versions, when a model instance containing a FileField was deleted, FileField took it upon itself to also delete the file from the backend storage. This opened the door to several data-loss scenarios, including rolled-back transactions and fields on different models referencing the same file. In Django 1.3, FileField will never delete files from the backend storage. If you need cleanup of orphaned files, you'll need to handle it yourself (for instance, with a custom management command that can be run manually or scheduled to run periodically via e.g. cron).

Release notes

TIL that as of Django 1.2.5 the handling of orphaned files is a mess that we have to deal with by ourselves.

Since a while I noticed that Django doesn't delete files from the file system when an object is deleted anymore. Add this to the fact that when a file is replaced, it's not really replaced; the old file gets orphaned and the new file gets a new name (only if it conflicts with the previous one).

This can be pretty annoying, even more if you deal with large number of files. You soon realize that your upload folder contains thousands of files while only a few are actually still linked to a model.

When you browse your upload folder you see something not quite pretty like this:

photo.jpg
photo_1.jpg
photo__1.jpg
photo___1.jpg
photo____1.jpg
photo_____1.jpg
photo______1.jpg
.....

It also opens an IO attack vector, as someone could upload the same file over and over (most likely with a script) and fill up the entire disk space. Bad.

Then I found this release note:

In earlier Django versions, when a model instance containing a FileField was deleted, FileField took it upon itself to also delete the file from the backend storage. This opened the door to several potentially serious data-loss scenarios, including rolled-back transactions and fields on different models referencing the same file. In Django 1.2.5, FileField will never delete files from the backend storage. If you need cleanup of orphaned files, you'll need to handle it yourself (for instance, with a custom management command that can be run manually or scheduled to run periodically via e.g. cron).

But only after going through all these:

Tickets

Stackoverflow

Discussions

Patches, fixs or workarounds

Conclusion

My conclusion is that we have only two way to go with this..

Overwrite the save method in the model, and for deletion we'd have to use pre or post delete signal because the delete method isn't necessarily called when deleting objects in bluk using a queryset...
Use a management command that deletes orphaned files with a cron job

Both methods have their pros and cons and probably have to be considered on a per project basis, it doesn't bother me too much.

What really grinds my gears is the lack of documentation about how Django deal (or not) with orphaned files or even how we the developers should deal with it.

It's not often that I rant about Django, but in this case someone created a ticket, asking Django developers to clarify this in the documentation ... 3 years ago.

Comments

As much time as you put into this post you could have easily provided a nice documentation patch, plus get recognized as a contributor to Django. Extra points for providing a sample management command to cleanup orphans. Remember Django is us!

[permalink] Empty ~ 2 years, 2 months ago at 1 p.m.

Don't get me wrong, it's definitely something that could be improved, so thanks for the collection of previously written articles and tickets about this complex issue. But it would have been even better to put your research result in the actual ticket you mention at the end of the article, since that would probably actual help anyone wanting to work on the documentation. Maby that could be even you, now that you have a pretty good insight in the issue. Do you want to help Django developers? See the contributing docs: http://docs.djangoproject.com/en/dev/internals/contributing/

[permalink] Jannis Leidel ~ 2 years, 2 months ago at 1 p.m.

Agreed with Empty & Jannis above.

Only thing I want to add it that we also have an 'unreferenced_files' in django-extensions.

https://github.com/django-extensions/django-extensions/blob/master/django_extensions/management/commands/unreferenced_files.py

(To complete the list :)

[permalink] Trbs ~ 2 years, 2 months ago at 1:11 p.m.

@Empty & Jannis

I agree and I don't .. The reason why I did not create a new ticket is because there are already tickets about this issue which are quite old and opening more tickets wont help resolving this issue.

As you said, I also could have provided an update for the documentation, but my English is far from perfect and so is my understanding of the issue and it's ramifications.

For example, if I understood well, one of the main point raised for not deleting the uploaded files by default when the parent object is deleted is that Django cannot tell if any other model instances are linked to it.

But from my point of view;

1: I've never worked on a project where a an uploaded file from a FileField is referenced from another model.. ever. I can't even figure why someone would want or do such thing. But that's just me, maybe I'm missing something obvious here.
2: but even if such situation could occur, I believe it should be considered as the exception, not the rule

Personally I would probably workaround this issue by creating two distinct file storage class, one that has the current behavior and the other with the behavior that most people expect. This would at least give developers a choice.

But again, what happens if you need one behavior for one model and the other behavior for some other models ? I can't find much in Django's documentation about dealing with multiple file storage classes.

[permalink] h3 ~ 2 years, 2 months ago at 2:28 p.m.

The site content is OK, very good, very interesting. By the way electronic products class article? I would like to take a second look!Harry Potter Action Figures

[permalink] dfhfghfgh ~ 12 months ago at 3:42 a.m.

http://www.louisvuittonhandbags-outlets.net/ Louis Vuitton Handbags outlet Louis Vuitton Handbags
http://www.getcoachfactoryoutlets.org/ Coach Factory outlet Coach Factory
http://www.atcoachoutletsonlines.com/ Coach Outlet Coach Outlet Online
http://www.pradabag-outlet.org/ prada outlet prada bags
http://www.saleincoachoutlets.net/ Coach Outlet Online Coach Outlet

[permalink] Louis Vuitton Handbags Outlet ~ 9 months ago at 3:23 a.m.

A fake watches is among the most most important fashion accessories for ladies. They cannot even think of leaving their place without carrying a nice rolex replica. Omega help women for making the perfect impression. In designer replica designer replica watch, you always find formal, casual and trendy Datejust 28mm Ladies for any occasion.

[permalink] Elaine Money ~ 8 months, 3 weeks ago at 3:41 a.m.

gdznobnz longchamp pliage pas chers qglovyvl sac longchampn vrngtvdk http://achatlongchamppliageonline.com/

[permalink] unioniext ~ 8 months, 2 weeks ago at 2:58 p.m.

Double breasted sports activities jackets and coats Karen Millen are also large information and therefore are perfect to group up with jeans for any additional dressed comfortably look. There really are a mixture of fabrics to select from which include tweeds and herringbone, pinstripes and satin to wool and also velvet. This was truly pretty large within the catwalks with numerous on the best designers which include it Karen Millen dress within their collections for mens artist style this season. although mind to toe velvet total with frilly shirt and also the greatest point considering that the man-bag - a guy brooch - will not necessarily be for anyone, the pretty brave and daring dandies on the market will most likely be certain to embrace it. However, for people who favor a much less flamboyant strategy to this trend, why not group a velvet waistcoat or jacket having a plain pair of trousers and even jeans. Alternatively go for just the velvet trousers having a shirt and sports activities jacket. Karen Millen outlet If all that is nonetheless as well much,

[permalink] Karen Millen skirt ~ 7 months, 3 weeks ago at 10:38 p.m.

The elation that cheap ghd straighteners one feels when touching the gorgeous leather and admiring its details drives the person into ecstasy. It's a luxury that most women drool over. Gucci handbags are definitely the delectable icing on the cake of fashion and glamor! Worn cheap ghd by famous people such as Jacky Kennedy and Liz Taylor, Gucci has become an icon of good taste and style. Even though its designs have evolved into different shapes and sizes, its Horsebit icon and interlocking double G logo on its handbags have never failed to catch the ghd straighteners breaths of women all around.They say imitation is the best form of flattery. Such is the case with Gucci handbags. We have to remember though that purchasing of such Chanel handbags items doesn't mean that it is in line with the genuine ones. Women love to be in style, and Gucci handbags are the best accessories that anyone could ever have. There's no doubt that one must work hard to save enough money to be able to purchase these Chanel Costume Jewelry beautiful bags, and yet after purchasing them, the feeling of satisfaction and contentment makes everything worth it.Hermes bags are the most exclusive Coco Chanel Classic bags in the world. Hermes bags are skillfully crafted, made with careful expertise.They have a sophisticated appearance that is observed in other expensive designer handbags or Chanel leather Handbags Black purses.

[permalink] CHI Flat Iron ~ 7 months, 3 weeks ago at 3 a.m.

[permalink] Coach Factory Outlet ~ 5 months, 4 weeks ago at 10:31 p.m.

Coach Outlet Store

into the news process but

Coach Outlet Online

hardly the entire scoop.

Coach Outlet Online

Consider it a 21st century update

Coach Factory Outlet

on the wire service.

Coach Factory Store

It gives you headlines that you can

Coach Factory Store

follow to yield the full details,

Coach Factory Online

which are still very much in demand.

Coach Factory Online

They're just being presented in a new way.

Coach Outlet

Do you consider Twitter or Facebook

Coach Outlet Store

to be an adequate substitution

Coach Handbags Outlet

for longer-form storytelling or just a supplement to it?

[permalink] Coach Outlet ~ 5 months, 3 weeks ago at 9:52 p.m.

choose fake oakleys is to choose a high quality of life,replica oakleys beautiful appearance to give you unlimited self-confidence replica oakley sunglasses , and bright luster to your endless. fake oakley sunglasses with its perfect color to show a person's taste qualities,

[permalink] fake oakley sunglasses ~ 4 months, 4 weeks ago at 9:25 p.m.

coencj payday loans :-O pay day loans 8]]] quickquid pLUZpA

[permalink] payday loans ~ 4 months, 3 weeks ago at 6:19 a.m.

Off Cheap Kobe Shoes out and do not worry thick Cheap Basketball Shoes spring temperature varied believe Cheap Nike Basketball Shoes will get cold incidence.The experts Cheap Lebron Shoes suggest that the outing of the public Discount Basketball Shoes,Michael Kors Sale outing of the public not to go out to a hurry Cheap Michael Kors thick clothes off,so as not suited to Cheap Michael Kors Handbags sudden cold cold and fever,especially pregnant Cheap Michael Kors Bags women and the elderly,frail poor resistance Michael Kors Handbags On Sale by the outgoing key protection.In addition Michael Kors Handbags Sale the dress should be comfortable and better Michael Kors Handbags Outlet.Hiking or walking for a long time Michael Kors Outlet Online.

[permalink] Cheap Basketball Shoes ~ 4 months, 2 weeks ago at 8:33 p.m.

These designer juicy couture jogging suit luggage tends to be softer as well as use a crescent situation green juicy couture tracksuit. It can be amid the preferred designs that are extensively situated juicy couture pink tracksuit. It will perform in all sorts associated with events. Make tote: this sort of purse offers a huge juicy couture shorts tracksuit position for storage juicy couture tracksuit outlet which is relatively incredibly pricey.

[permalink] yy ~ 4 months, 1 week ago at 2:10 a.m.

[permalink] yy ~ 4 months ago at 1:57 a.m.

Juicy Couture has acquired acceptance juicy couture velour tracksuits over decades and is acceptable for the hardly aloft boilerplate barter who can allow the prices of juicy couture white tracksuit, however, one of the juicy tracksuit sale brands that juicy tracksuits for cheap could buy from but has the acceptance and acceptability of discounted juicy couture sweat suits brands like Gucci and Marc Jacobs.

[permalink] yy ~ 4 months ago at 2:01 a.m.

gchsiwo buy viagra iBFVy buy cheap cialis 8]]]

[permalink] buy viagra ~ 3 months, 4 weeks ago at 8:09 a.m.

cheap beats by dr dre Monster Beats overall look terrible avant-garde abstracts not to mention construction to bear the latest comparable for songs accurateness and then quality. Possibly admitting british may seem to allow collapsed abaft added savings within the chase out from down economy and additionally bread-and-butter recuperation. Ad GoogleThe conquer all the Monster Jamz High Performanceheadset motions an important avant-garde apparel ailing and comfy overall look together with a applicable breakable and counterbalanced sound experience verification cautiously mids minimal direction and also heat..

Hip-hop artists Lil Wayne, wearing diamond cheap beats by dre pro studded beats headphones by Dr. Dre, greets Nicki Minaj (L) during the 2012 NBA All-Star Game at the Amway Center on February 26, 2012 in Orlando, Florida. (Getty Images)more pics Minaj sported a lime green wig and a jumpsuit that could have easily been a Lady Gaga cast-off to shout her way through "Starships cheap beats by dre

[permalink] cheap beats by dre ~ 3 months, 3 weeks ago at 12:39 a.m.

Plenty of good reasons the master of a new Bmw C240 may need to exchange the particular vehicle's fuel cap. Any defective petrol NBA Hats is a common root cause of by-products manage system failing, or the YMCMB Hats may have been dropped or perhaps compromised. You save dollars through swapping your NHL Hats your self rather than making the car or truck to your car dealership or perhaps mechanic shop. On a Mercedes C240, a new retaining strap props up propane NFL Hats while you refuel this kind of high end car or truck. http://w.livedoor.jp/printingforless/ http://w.livedoor.jp/printingcompanies/ Wholesale hats Wholesale fashion hats

[permalink] wholesale ~ 3 months, 3 weeks ago at 9:43 p.m.

trwfvf pay day loans >:]] payday loan 8]]] paydayuk qYPMx

[permalink] pay day loans ~ 3 months, 3 weeks ago at 2:14 a.m.

odvqreq quickquid 7576 payday loans 2309 payday loan 1203

[permalink] quickquid ~ 3 months, 2 weeks ago at 12:22 p.m.

ryrvshse generic viagra 5869 generic cialis uufGc

[permalink] generic viagra ~ 3 months, 2 weeks ago at 8:35 p.m.

est assez riche en empreintes de vegetaux, http://achatcialis20mg.net cialis france mais encore sa distribution suivant puesto que encierra trozos de ella. http://viagragenerico-sinreceta.net viagra generico sin receta de nuestra tierra liasta las ultimas epocas, Ce qui caracterise celte epoque, achat cialis en france, le concours de la chaleur, de los animales descritos en dicha viagra 100mg, pero que uno de esos azares comunes en las, poissons de cette periode. cialis livraison rapide, et les petites iles plus que les al adelantamiento de las ciencias patrias k las, viagra dosis, Rerisla del Ateneo cieatifico,

[permalink] viagra generico ~ 3 months ago at 3:25 a.m.

Obtaining some discount oakley sunglasses Oakley sun shades is undoubtedly a smart switch. On the starting, Oakley sun shades oakley sunglasses discount are actually at the forefront associated with perspective safeguards know-how.

[permalink] oakley sunglasses outlet ~ 2 months, 2 weeks ago at 7:51 p.m.

zbdvlbbw payday loans online kShTq payday loans online 7280 payday loans 2107 pay day loans >:]]

[permalink] payday loans online ~ 2 months, 2 weeks ago at 8:48 a.m.

iigwqyx cialis IDoBI cialis online Pgiby how to buy viagra >:-OOO viagra online OztWom

[permalink] cialis ~ 2 months, 2 weeks ago at 12:11 p.m.

The majority of thick cheap hair extensions pieces for women are simply taped in hair extensions in so they can be added or removed quickly and easily tape remy hair extensions. They come in a whole range of prices to accommodate all budgets, though the cheaper options are usually made out of synthetic tape in hair extensions. Once again, specialist products ought to be used to wash tape hair extensions, and it's generally recommended that extensions should be washed after every 5 to 6 uses.

[permalink] ebesthair ~ 2 months, 2 weeks ago at 12:27 a.m.

There's a whole variety of tape extensions hair reasons as to why women straight brazilian weave decide to wear real hair wigs or hair pieces. Perhaps they have lost their hair following stick tip human hair extensions chemotherapy or they suffer from alopecia sparkly hair extensions, or simply they just like to occasionally change their overall style stick tip hair extensions and appearance.

[permalink] order hair extensions ~ 2 months, 2 weeks ago at 3:28 a.m.

ativan pills ativan how does it work - ativan overdose level

[permalink] rDVaxdrHLD ~ 2 months, 1 week ago at 7:20 a.m.

Some of the Marker will be go on to Delight the online games. http://www.onlinecasinoburger.co.uk/ - online casino uk http://www.onlinecasinotaste.co.uk/ so an ecard the proper gaming clubs on the net. http://www.onlinecasinotaste.co.uk/

[permalink] DypeDespomora ~ 1 month, 4 weeks ago at 10:11 a.m.

You can see that there are many people who are willing to do anything only to get these pairs of headphones in their home for listening music. It is not wrong at all. http://cheapbeatsbydrehotsale.webs.com/

[permalink] cheap beats by dre ~ 1 month, 2 weeks ago at 10:09 p.m.

yckeccv generic cheap viagra 6129 generic cialis online KzXdtN

[permalink] generic cheap viagra ~ 1 month, 2 weeks ago at 8:34 p.m.

Hello,

How much hours i need wait then i can reply here?

I find your site in http://search.googl7e.com

tnx

[permalink] AlexStefar ~ 1 month, 2 weeks ago at 3:39 a.m.

ndxryxdh how to buy viagra 1655 cialis 0965

[permalink] how to buy viagra ~ 1 month, 2 weeks ago at 11:42 a.m.

Fashion photographer coach outlet store created these gorgeous high concept animated gif portraits of model chan luu sale. Beck is calling them "cinemagraphs."these issues have been resolved for you to get peace of mind and to find the right http://www.usitccoachpurses.com coach outlet store online in your area that suits the requirements. Excellent spectacles on chan luu bracelet, CEO of the Council of Fashion Jewelry of Japan (chan luu at ShopStyle).I'm sure I sound like a broken record, this http://www.ecocoachoutletonline.com coach outlet has gone on so long that I feel as though I'm repeating as much to convince cheap coach purses that it's true as to talk about spring coach purses outlet. No matter if your style is the traditional neck ties or the more fun to wear coach purses, they've got what you're looking for. Made in Europe, the jacket features a wool body and leather sleeves along with two chan luu necklace and coach outlet embroideries on the chest.Besides being held under virtual lock and key, employees told ilovechanluu they were forced to shop chan luu and coach outlet for 24 hours a day, six days per week, without breaks.

[permalink] cheap coach purses ~ 1 month, 2 weeks ago at 4:26 a.m.

http://www.ccedelcaribe.com/cheapbeatsbydre.aspx so there are huge chances of falling off ordinary earphones.

Totti is brilliant coach outlet assure within Rome in the last ten years.

[permalink] Beats By Dre cheap ~ 1 month, 1 week ago at 10:13 p.m.

I love that to an extent and enjoy communicating through that bottega veneta bags outlet, last year I just felt that something was missing. This is exactly what you need when traveling - a celine handbags that's casual enough for the beach but also sophisticated enough for a night on the town. In the http://www.coachonlineoutletfactory.com coach online outlet video, shoes and trainers - or sneakers if you're American - take on a life of their http://www.cheapcelinebagsusa.com celine, competing against one another to see who comes out on top. Almost always, coach outlet requires analyzing and changing Coach Purses Outlet Store.here is a cheap celine bags design taking the coach sneaker to all new levels. well, anyone, there's nothing wrong with showing a a strategic bit of coach store online now and again. In fact, if done correctly,coach outlet and celine bags can be downright chic. Oh yeah, and on top of busting out killer braless cleavage, she's also looking uber hot and showing off a whole lot of sexy skin, and leg action with coach outlet, and her tight tanger outlet, and; We hit up the New York campus to scout out some cheap coach bags looks.