Welcome to the Permutation Feature Importance section. This section will describe how to use Permutation Feature Importance, also known as PFI or Feature Importance to help screen indicator or other custom data (the features) that are used in creating a machine learning model.
Permutation feature importance is a method used to evaluate the importance of features in a machine learning model. This can help to show how much a particular feature contributes to the performance of a model. It works by taking a labeled dataset such as the profit target reached and profit target not reached datasets, choosing a feature, and permuting the values for that feature across all the examples, so that each example now has a random value for the feature and the original values for all other features.
The evaluation metric (e.g. R-squared) is then calculated for this modified dataset, and the change in the evaluation metric from the original dataset is computed. The larger the change in the evaluation metric, the more important the feature is to the model. Feature Importance works by performing this permutation analysis across all the features of a model, one after another.
In order to use the Feature Importance, we need to create a machine learning model. Please see the Creating A Machine Learning Model section for instructions on how to create a machine learning model.
In the Strategy Analyzer dialog, select the DSTCreateModelPFITest.
Please note, calculating feature importance can be a time consuming operation. How much time it takes to calculate is proportional to the number of features you have. The more features, the longer it will take to run.
In this example, we are using Regression with RSquared as the optimizing metric.
After clicking on Run and waiting for the Create Model Progress dialog to finish, the Feature Importance button should be ungrayed.
At this point, we can click on Ok, View Full Results or Feature Importance. If we click on Feature Importance, the Feature Importance dialog will be displayed. The Feature Importance dialog was expanded by dragging the lower right hand corner to show all of the short and long features. In this example, the Permutation count has been changed to 2 for faster results.
The Permutation Count is the number of iterations that the feature importance algorithm will use to calculate feature importance for each feature. The Use Feature Weight Filter will use the features weight to pre-filter features.
Click on Run Feature Importance to start calculating the feature importance for each dataset.
After running Feature Importance and sorting the RSquared column by clicking on the top of the column, the RSquared column shows the ATR(20) feature is -0.011358 for Short trades and -0.011259 for Long. This shows that the ATR(20) feature decreased our R Squared value by roughly 0.01 for both Long and Short datasets. Other features slightly negatively impacted our model but did not contribute substantially to the R Squared metric. Although the model is not significantly impacted by ATR, we may choose to remove the ATR feature from our model to help improve performance.
Futures, foreign currency and options trading contains substantial risk and is not for every investor. An investor could potentially lose all or more than the initial investment. Risk capital is money that can be lost without jeopardizing ones financial security or lifestyle. Only risk capital should be used for trading and only those with sufficient risk capital should consider trading. Past performance is not necessarily indicative of future results.