My data frame looks like -
no city amount
1 Kenora 56%
2 Sudbury 23%
3 Kenora 71%
4 Sudbury 41%
5 Kenora 33%
6 Niagara 22%
7 Hamilton 88%
It consist of 92M records. I want my data frame looks like -
no city amount new_city
1 Kenora 56% X
2 Niagara 23% X
3 Kenora 71% X
4 Sudbury 41% Sudbury
5 Ottawa 33% Ottawa
6 Niagara 22% X
7 Hamilton 88% Hamilton
Using python I can manage it(using np.where
) but not getting any results in pyspark. Any help?
I have done so far -
#create dictionary
city_dict = {'Kenora':'X','Niagara':'X'}
mapping_expr = create_map([lit(x) for x in chain(*city_dict .items())])
#lookup and replace
df= df.withColumn('new_city', mapping_expr[df['city']])
#But it gives me wrong results.
df.groupBy('new_city').count().show()
new_city count
X 2
null 3
Why gives me null values?
See Question&Answers more detail:os