Pandas dataframe to django Model
So, pandas has a lot going for it. Yes I know. Those who are all about speed use something else. I don’t. I really like pandas, and I can wait a millisecond for my process to complete.
Pandas function, .to_sql, which in combination with SQLAlchemy is really really cool. I had use for something that worked with django, so I whipped up this contraption. No, it’s no where near .to_sql, but it does the job. For me. In this instance.
import pandas as pd
import numpy as np
from django.db import models
def dataframe_to_django(
df: Series | Any,
model: type[models.Model],
dbmapping: dict[str, str | None],
convmapping: dict[str, Any],
replace: bool = False,
) -> None:
"""Uses bulk_create to insert data in table.
KeywordArguments:
df -- the dataframe.
model -- the django table in to which the dataframe should be inserted.
dbmapping -- mapping from dataframe name to db name. Typically {"ProductName": "product_name", "SkuDescription": None, ...}
convmapping -- converters for spesific fields. Typically, {"StartDate": pd.to_datetime, ...} .
replace -- bool False|True. If set to True it will wipe the table before import.
"""
if replace:
model.objects.all().delete()
df.replace([np.nan], ["N/A"], inplace=True)
for key in convmapping:
df[key] = convmapping[key](df[key])
model_list: list[models.Model] = [
model(**{dbmapping[k]: v for k, v in x.items() if dbmapping[k] is not None}) for x in df.to_dict("records")
]
model.objects.bulk_create(model_list)
The link to the gist is gist.github.com/roderik333/0343f54af1e27dcb..
until next time..