Normalise between 0 and 1 ignoring NaN

Question!

For a list of numbers ranging from x to y that may contain NaN, how can I normalise between 0 and 1, ignoring the NaN values (they stay as NaN).

Typically I would use MinMaxScaler (ref page) from sklearn.preprocessing, but this cannot handle NaN and recommends imputing the values based on mean or median etc. it doesn't offer the option to ignore all the NaN values.



Answers

consider pd.Series s

s = pd.Series(np.random.choice([3, 4, 5, 6, np.nan], 100))
s.hist()

enter image description here


Option 1
Min Max Scaling

new = s.sub(s.min()).div((s.max() - s.min()))
new.hist()

enter image description here


NOT WHAT OP ASKED FOR
I put these in because I wanted to

Option 2
sigmoid

sigmoid = lambda x: 1 / (1 + np.exp(-x))

new = sigmoid(s.sub(s.mean()))
new.hist()

enter image description here


Option 3
tanh (hyperbolic tangent)

new = np.tanh(s.sub(s.mean())).add(1).div(2)
new.hist()

enter image description here



You could try this - it finds the appropriate index via sys.key_constraints and uses sys.index_columns to identify the column in the index

SELECT 
    so.name AS TableName,
    sc.colid AS Id,
    sc.name AS Name,
    sc.xtype AS TypeId,
    sc.prec AS Precision,
    sc.scale AS Scale,
    sc.iscomputed AS IsComputed,
    sc.isnullable AS IsNullable,
    sc.collation AS Collation,
    (   select count(*) from sys.key_constraints kc
        inner join sys.indexes i on i.name = kc.name
        inner join sys.index_columns ic on ic.object_id = kc.parent_object_id and ic.index_id = i.index_id
        where kc.type = 'PK' and kc.parent_object_id = so.id and ic.column_id = sc.colid ) as IsPrimaryKey

FROM syscolumns sc
JOIN sysobjects so ON sc.id = so.id
WHERE so.xtype = 'U'
AND so.name = 'Person'

The output is:

TableName   Id  Name    TypeId  Precision   Scale   IsComputed  IsNullable  Collation               IsPrimaryKey
Person      1   Id          56      10          0       0               0    NULL                   1
Person      2   Name        231     50          NULL    0               0    Latin1_General_CI_AS   0
Person      3   Age         56      10          0       0               0    NULL                   0
By : John D


in the constructor, you just need to do this.state = { video: {} } not this.setState

constructor(){
    super();
    this.state = {
        video:{}
    };
}

this.setState can be used anywhere else but the constructor. non es6 way of doing this was doing:

getInitialState: function() {
  return { 
    video: {}
  }
}

which is equivalent to just doing this.state = {} in the constructor

By : finalfreq


This video can help you solving your question :)
By: admin