For some reason I felt an urge to move from envsubst to a python based templating system.
Enter pydantic. A fantastic package, which incidentally also can be used for settings management.
I wanted to split my configs into logical chunks but still have one monolithic .env file. This is a no-brainer using pydantic
from pydantic import BaseSettings
class GlobalSettings(BaseSettings):
class Config():
env_file: str = ".env"
class DbSettings(GlobalSettings):
user: str
password:str
database: str
.
.
.
class Settings(GlobalSettings):
something: str
other: str
dbsettings: DbSettings = DbSettings()
Imagine my surprise when I ran this code in ipython
Python 3.10.6 (main, Aug 2 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from settings import Settings
In [2]: s = Settings()
In [3]: s.user
Out[3]: '👨 #(whoami)'
My surprise subsided quickly when I realized that I could have read the documentation
Even when using a dotenv file, pydantic will still read environment variables as well as the dotenv file, environment variables will always take priority over values loaded from a dotenv file..
It became adamant to find a way to not "pollute" my settings with all the weirdness of my environment. Turns out it's a pretty easy fix. It might even be a fix that won't break pydantic on the next upgrade [¹]
from pydantic.env_settings import EnvSettingsSource, SettingsSourceCallable
class IgnoreLocalEnvSettingsSource(EnvSettingsSource):
"""Exists so that we can ignore local environment."""
def __call__(self, settings: BaseSettings) -> Dict[str, Any]: # noqa C901
"""
Build environment variables suitable for passing to the Model.
"""
os.environ.clear()
return super().__call__(settings)
class GlobalSettings(BaseSettings):
"""Set the base config .env file."""
class Config:
"""Base config."""
env_file: str = ".env"
env_file_encoding: str = "utf-8"
@classmethod
def customise_sources(
cls,
init_settings: SettingsSourceCallable,
env_settings: SettingsSourceCallable,
file_secret_settings: SettingsSourceCallable,
) -> Tuple[SettingsSourceCallable, ...]:
"""Ignore __init__."""
return (
init_settings,
IgnoreLocalEnvSettingsSource(
env_file=cls.env_file, env_file_encoding=cls.env_file_encoding
),
file_secret_settings,
)
How useful was this exercise? In a production environment, probably not that much. Yes, there are (numerous) attack vectors that rely on replacing and overwriting environment variables. It's not inconceivable that an attacker might be able to exploit that.
However, in a development environment, like my local machine, there's are a plethora of weird and forgotten environment variables. Knowing that the local env is not overwriting the variables in your .env file is one less thing to worry about.
Will this have adverse side effects? That remains to be seen, but if I understand the (pydantic) code correctly it shouldn't.
As an added bonus I got to clean up my local environment. My installation is only 6 months old, and still it was quite messy.
[¹]: I was looking at the github code thinking about implementing some really fancy stuff (I usually traverse down that particular rabbit hole before coming to my senses) when I realized that the pydantic I was looking at was nothing like the one on pypi.. which forced me to implement a much simpler solution.