Unpack dictionary from Pandas Column

All we need is an easy explanation of the problem, so here it is.

I have a dataframe that has one of the columns as a dictionary. I want to unpack it into multiple columns (i.e. code, amount are separate columns in the below Raw column format). The following code used to work with pandas v0.22, now (0.23) giving an index error:

pd.DataFrame.from_records(df.col_name.fillna(pd.Series([{'code':'not applicable'}], index=df.index)).values.tolist())

ValueError: Length of passed values is 1, index implies x

I searched google/stack overflow for hours and none of the other solutions previously presented work anymore.

Raw column format:

     dict_codes
0   {'code': 'xx', 'amount': '10.00',...
1   {'code': 'yy', 'amount': '20.00'...
2   {'code': 'bb', 'amount': '30.00'...
3   {'code': 'aa', 'amount': '40.00'...
10  {'code': 'zz', 'amount': '50.00'...
11                            NaN
12                            NaN
13                            NaN

Does anyone have any suggestions?

Thanks

How to solve :

I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.

Method 1

Setup

df = pd.DataFrame(dict(
    codes=[
        {'amount': 12, 'code': 'a'},
        {'amount': 19, 'code': 'x'},
        {'amount': 37, 'code': 'm'},
        np.nan,
        np.nan,
        np.nan,
    ]
))

df

                         codes
0  {'amount': 12, 'code': 'a'}
1  {'amount': 19, 'code': 'x'}
2  {'amount': 37, 'code': 'm'}
3                          NaN
4                          NaN
5                          NaN

apply with pd.Series

Make sure to dropna first

df.codes.dropna().apply(pd.Series)

   amount code
0      12    a
1      19    x
2      37    m

df.drop('codes', 1).assign(**df.codes.dropna().apply(pd.Series))

   amount code
0    12.0    a
1    19.0    x
2    37.0    m
3     NaN  NaN
4     NaN  NaN
5     NaN  NaN

tolist and from_records

Same idea but skip the apply

pd.DataFrame.from_records(df.codes.dropna().tolist())

   amount code
0      12    a
1      19    x
2      37    m

df.drop('codes', 1).assign(**pd.DataFrame.from_records(df.codes.dropna().tolist()))

   amount code
0    12.0    a
1    19.0    x
2    37.0    m
3     NaN  NaN
4     NaN  NaN
5     NaN  NaN

Method 2

Setup

                        codes
0  {'amount': 12, 'code': 10}
1    {'amount': 3, 'code': 3}

apply with pd.Series

df.codes.apply(pd.Series)

   amount  code
0      12    10
1       3     3

Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply