Data management with Python, Pandas, Geopandas, Matplotlib and Openpyxl - Migrations

Migrations

Some tips to switch from Pandas 1.x to 2.1.1, with SqlAlchemy 2.0.21 and Python 3.9 (and maybe Python 3.11)

ImportError: cannot import name SettingWithCopyWarning from pandas.core.common

It can happens if you use:

[code]import warnings
from pandas.core.common import SettingWithCopyWarning
warnings.simplefilter(action='ignore', category=SettingWithCopyWarning)[/code]

Now prefer:

[code]import pandas as pd
...
pd.set_option('mode.chained_assignment', None)[/code]

And if you script is loaded from several Python versions, you can do:

[code]import pandas as pd
...
pandas_version = pd.__version__

if pandas_version >= '2.1.1':
    pd.set_option('mode.chained_assignment', None)

else:
    import warnings
    from pandas.core.common import SettingWithCopyWarning
    warnings.simplefilter(action='ignore', category=SettingWithCopyWarning)[/code]

sqlalchemy.exc.ObjectNotExecutableError: Not an executable object

It can happens when you execute a query from a string, example:

[code]from sqlalchemy import create_engine
from _params import my_username, my_password, my_host, my_port, my_database

my_engine = create_engine('mysql+mysqldb://%s:%s@%s:%i/%s?charset=utf8mb4' % (my_username, my_password, my_host, my_port, my_database))

my_connection = my_engine.connect()

my_query = my_connection.execute('SELECT 1 FROM mytable ;')[/code]

Just put your SQL query in text:

[code]from sqlalchemy import create_engine, text
from _params import my_username, my_password, my_host, my_port, my_database

my_engine = create_engine('mysql+mysqldb://%s:%s@%s:%i/%s?charset=utf8mb4' % (my_username, my_password, my_host, my_port, my_database))

my_connection = my_engine.connect()

my_query = my_connection.execute(text('SELECT 1 FROM mytable ;'))[/code]

Specify dtype option on import or set low_memory=False

Specify low_memory when you read your file:

df = pd.read_csv('your file.csv', low_memory=False)

FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated

It can happens if you do a pd.concat, something like:

[code]df_1 = ...
df_2 = ...
df_3 = ...

df = pd.concat([df_1, df_2, df_3], axis=0)[/code]

Prefer that:

[code]ListMyDfs = ['df_1', 'df_2', 'df_3']

df = pd.DataFrame()

for MyDf in ListMyDfs:
   if not globals()[MyDf].empty:
        df = pd.concat([df, globals()[MyDf]], axis=0)[/code]