Otdfbt

Let's assume that we have a binary classification problem, and we built a decision tree on our data set.

Assuming that we have 5 features, then the decision tree, in the first step, will choose the best feature of the 5, and on this feature it will choose the best threshold in order to split the data set, and then continue to make the tree deeper etc. The definition of best is the lowest classification error.

My question is: Since the decision tree, on each step, chooses the best feature to split on, and the best threshold to split on, then why random forest (which is many decision trees), is an improvement of decision trees? Shouldn't a decision tree be sufficient?

UPDATE

I more mean it like: If you have a decision tree classifier, and a random forest classifier with the same parameters, when possible, (max_depth, number of children etc), will the decision tree classifier score the same on the training set, with the random forest classifier?

It comes down to overfitting as you scale. Decision trees tend to overfit as they grow deep. After every split there will be fewer and fewer samples for the next split to work with. Fewer samples means that risk of splitting on noise increases.

Random forest avoids the overfitting problem of decision trees by instead scaling by adding more trees instead of building one big tree. Averaging the outputs of the trees in the forest means that it does not matter as much if the individual trees are overfitting.

Regarding your update. No, they will not score the same. Random forest will not have just one decision tree. It has several and divides the features into random subsets for each tree to be trained on. So even if the size of the decision trees in random forest would be the same as a single decision tree, the features they are trained on would not be.

But if you ask what happens if we take a random forest, only use one tree and train it on the same features as a single decision tree of the same size, then yes they would be one and the same.

This is an interesting question to answer as there are multiple reasons why random forests work better than a decision tree. I'll compare how each of classifier/regressor work in each of the below cases

1. So, We have a dataset with 5 features as you said. Let's consider our decision tree classifier is overfitted to that data. Since the model is overfitted, Any small change in data will cause a huge change in classification (Variance problem). But in RF, Since we are using multiple decision trees in a random forest, Any small change in data will not cause dramatic changes in classification as we take a majority vote of all the trees to take a decision. Hence reducing the overfitting (variance) problem.


2. If you notice, We do not feed in the entire dataset at once in a random forest. We perform row sampling with replacement column sampling without replacement at every data feeding step and so your model will be able to generalize much better than a decision tree.


3. Random forests are made up of Decision trees with large depth which has a lot of variance at the start and has reduced variance at the end of learning while. But decision trees you hyperparameter tune them, you don't fix their depths(i.e you don't say whether they are shallow or deep).

Hope this helps!