how to use output of sklearn pipeline elements

Question

I have three features:

feature_one -> number of tokens in the given sentence.  
feature_two -> number of verbs in the given sentence.  
feature_three -> number of tokens - number of verbs in the given sentence.  
(feature_one - feature_two)

I have written custom transformers for feature_one and feature_two and want to written custom transformer for feature_three such that I can use result of feature_one and feature_two by running pipeline as:

Pipeline([
        #input to feature_one and feature_two is list of sentences.
        ("feature", FeatureUnion([
            ("feature_one", feature_one_transformer()),
            ("feature_two", feature_two_transformer())          
        ])),

        ("feature_three", feature_three_transformer())    
])

feature_one_transformer:

class feature_one_transformer(BaseEstimator, TransformerMixin):

    def __init__(self):
        pass

    def fit(self, x, y):
        return self

    def transform(self, sentence_list):
        number_of_tokens_in_sentence_list = list()

        for sentence in sentence_list:
            number_of_tokens = compute_number_of_tokens

            number_of_tokens_in_sentence_lista.append(number_of_tokens)

        return pandas.DataFrame(number_of_tokens_in_sentence_list)

feature_two_transformer:

class feature_two_transformer(BaseEstimator, TransformerMixin):
    def __init__(self):
        pass

    def fit(self, x, y):
        return self

    def transform(self, sentence_list):
        number_of_verbs_in_sentence_list = list()

        for sentence in sentence_list:
            number_of_verbs = compute_number_of_verbs_in_sentence

            number_of_verbs_in_sentence_lista.append(number_of_verbs)

        return pandas.DataFrame(number_of_verbs_in_sentence_list)

Can somebody tell me how should I write custom transformer for feature_three and how to use in pipeline so that I can use result of feature_one and feature_two transformers. Thank you.

Oriol Mirosa · Accepted Answer · 2017-07-27 17:09:53Z

1

It's not clear to me why you would want to make this so complicated. I would just use one transformer that does everything you want. Something like this:

class features_transformer(BaseEstimator, TransformerMixin):

    def __init__(self, variable):
        self.variable = variable

    def fit(self, X):
        return self

    def transform(self, X):
        X['number_of_tokens'] = X[self.variable].apply(lambda cell: compute_number_of_tokens(cell))
        X['number_of_verbs'] = X[self.variable].apply(lambda cell: compute_number_of_verbs(cell))
        X['tokens_minus_verbs'] = X['number_of_tokens'] - X['number_of_verbs']

        return X

new_X = features_transformer('sentences').fit_transform(X)

answered Jul 27, 2017 at 17:09

Oriol Mirosa

2,8361 gold badge16 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Saurabh Jain Over a year ago

Thanks you. I am learning how to use Pipeline, so this question hit me and I tried to figure out a solution.

Oriol Mirosa Over a year ago

No problem! We're here to help. Did the answer work for you?

Oriol Mirosa Over a year ago

Good! Don't forget to mark the answer as correct so others can find it!

Collectives™ on Stack Overflow

how to use output of sklearn pipeline elements

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related