Feature reduction is widely used in machine learning based phishing URL detection to lower model complexity while retaining detection performance. However, conclusions about how much reduction is acceptable may depend on how training and testing data are partitioned. This study examines whether conclusions about feature reduction remain stable when evaluation changes from random stratified to domain disjoint cross validation. Experiments were conducted on URL-Phish Version 2 using Mutual Information ranking within each training fold and four predefined feature budgets, from 22 to 5 features. Logistic Regression and Random Forest were evaluated under both protocols, with PR AUC as the primary metric. PR AUC decreased as the feature budget was reduced for both classifiers under both evaluation protocols, so the qualitative conclusion about feature reduction remained consistent within this experimental setting. However, the size of the protocol gap varied across classifiers and feature budgets: it was clearest for Random Forest under the smallest feature budget, whereas the Logistic Regression gaps were less clearly separated from fold-level variability. Domain disjoint evaluation also produced greater fold-level variability. The results show that claims about compact phishing URL representations should be interpreted together with the evaluation protocol used. Future work should examine whether this pattern persists across additional datasets, classifiers, and domain grouping rules............... Keywords:............ Phishing URL Detection, Feature Reduction, Domain Disjoint Evaluation, Machine Learning, Mutual Information, Generalization