Drought Prediction using ML and Ensemble Models along with Explainable AI using SHAP and LIME: v1.0.0 - Initial Release
Fuente:
Zenodo
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Pubblicazione: |
Zenodo
2025
|
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866901716769701888 |
|---|---|
| author | EHTESHAM ASHRAF |
| author_facet | EHTESHAM ASHRAF |
| contents | <h2> Initial Release - Drought Prediction System</h2> <p>We're excited to release the first version of our comprehensive <strong>Drought Prediction System</strong> using Machine Learning and Explainable AI. This release includes a complete end-to-end pipeline for predicting drought conditions across Indian districts with state-of-the-art model performance and full interpretability.</p> <h3>Core Features</h3> <ul> <li><strong>Complete ML Pipeline</strong>: End-to-end workflow from data preprocessing to model deployment</li> <li><strong>Multi-Model Comparison</strong>: Evaluation of Random Forest, SVM, KNN, and XGBoost models</li> <li><strong>Advanced Feature Engineering</strong>: Time-lagged features (t-1 to t-6 months) capturing temporal dependencies</li> <li><strong>Class Imbalance Handling</strong>: Comparison of SMOTE and weighted loss approaches</li> <li><strong>Hyperparameter Optimization</strong>: Automated tuning using RandomizedSearchCV</li> <li><strong>Explainable AI</strong>: Full interpretability with SHAP and LIME explanations</li> <li><strong>Pre-trained Model</strong>: Ready-to-use XGBoost model with 97.2% accuracy</li> </ul> <h3>Performance Highlights</h3> <p>| Metric | Best Model (XGBoost) |</p> <p>|--------------------|-------------------------|</p> <p>| <strong>Accuracy</strong> | <strong>97.2%</strong> |</p> <p>| <strong>F1 Score</strong> | <strong>80.4% </strong> |</p> <p>| <strong>Macro Precision</strong> | 88.6% |</p> <p>| <strong>Macro Recall </strong> | 90.3% |</p> <p>| <strong>PR-AUC</strong> | 0.881 |</p> <h3> Explainability Features</h3> <ul> <li><strong>SHAP Analysis</strong>: Global and local feature importance with beeswarm plots</li> <li><strong>LIME Explanations</strong>: Case studies for drought and non-drought scenarios</li> <li><strong>State-specific Insights</strong>: Top 5 important features per state</li> <li><strong>Probabilistic Policy Insights</strong>: Actionable recommendations for policymakers</li> </ul> <h3>Dataset Coverage</h3> <ul> <li><strong>49 Districts</strong> across <strong>5 Indian States</strong></li> <li><strong>29,988 Observations</strong> of historical meteorological data</li> <li><strong>42 Engineered Features</strong> including time-lagged variables</li> <li><strong>5 Decades</strong> of temporal coverage</li> </ul> <h2>What's Included</h2> <h3>Code & Models</h3> <ul> <li>✅ Complete Jupyter notebook with all analysis steps</li> <li>✅ Pre-trained XGBoost model (<code>xgbmodel_files.pkl</code>)</li> <li>✅ StandardScaler and LabelEncoders for preprocessing</li> <li>✅ Requirements file with all dependencies</li> </ul> <h3>Visualizations</h3> <ul> <li>✅ Time series plots for all meteorological variables</li> <li>✅ Model performance comparison charts</li> <li>✅ SHAP beeswarm and decision plots</li> <li>✅ LIME explanation plots and HTML files</li> <li>✅ Interactive Folium maps for predictions</li> </ul> <h3>Documentation</h3> <ul> <li>✅ Comprehensive README with installation instructions</li> <li>✅ Methodology documentation</li> <li>✅ Usage examples and code snippets</li> </ul> <h2>Key Capabilities</h2> <h3>1. <strong>Data Preprocessing</strong></h3> <ul> <li>Automatic handling of duplicate district names</li> <li>Label encoding for categorical variables</li> <li>Cyclical encoding for temporal features (sine/cosine)</li> <li>Standard scaling for numerical features</li> </ul> <h3>2. <strong>Feature Engineering</strong></h3> <ul> <li>Time-lagged features (1-6 months prior)</li> <li>36 additional lagged features</li> <li>Total of 42 engineered features</li> </ul> <h3>3. <strong>Model Training</strong></h3> <ul> <li>Multiple ML algorithms comparison</li> <li>Automated hyperparameter tuning</li> <li>Cross-validation with StratifiedKFold</li> <li>Class imbalance handling strategies</li> </ul> <h3>4. <strong>Model Interpretability</strong></h3> <ul> <li>SHAP values for global feature importance</li> <li>LIME explanations for individual predictions</li> <li>Uncertainty quantification</li> <li>State-specific feature importance analysis</li> </ul> <h3>5. <strong>Predictions & Insights</strong></h3> <ul> <li>January 2023 drought predictions for 11 districts</li> <li>Probabilistic policy insights</li> <li>Feature sensitivity analysis</li> <li>Geographic visualization of predictions</li> </ul> <h2>Quick Start</h2> <h3>Installation</h3> <pre><code># Clone the repository git clone <repository-url> cd "Drought Prediction Final" # Create virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install -r requirements.txt </code></pre> <h3>Load Pre-trained Model</h3> <pre><code>import pickle with open('Code/xgbmodel_files.pkl', 'rb') as f: xgbmodel_files = pickle.load(f) best_xgb_model = xgbmodel_files[0] # Trained XGBoost model sc = xgbmodel_files[1] # StandardScaler le1 = xgbmodel_files[2] # LabelEncoder for DISTRICT le2 = xgbmodel_files[3] # LabelEncoder for STATE </code></pre> <h2>Technical Details</h2> <h3>Models Evaluated</h3> <ul> <li><strong>Random Forest</strong>: 400 trees, 96.28% accuracy</li> <li><strong>SVM</strong>: RBF kernel, 95.57% accuracy</li> <li><strong>KNN</strong>: K-Nearest Neighbors, 95.72% accuracy</li> <li><strong>XGBoost</strong>: Gradient boosting (best performer), 97.2% accuracy</li> </ul> <h3>Best Model Configuration</h3> <ul> <li><strong>Algorithm</strong>: XGBoost with weighted loss</li> <li><strong>Hyperparameters</strong>: Optimized via RandomizedSearchCV</li> <li><strong>Class Imbalance</strong>: Handled with <code>scale_pos_weight=12.77</code></li> <li><strong>Cross-Validation</strong>: 5-fold StratifiedKFold</li> </ul> <h3>Top Important Features</h3> <ol> <li><code>SPI_t-1</code> - Standardized Precipitation Index (1 month prior)</li> <li><code>Precipitation_t-6</code> - Precipitation (6 months prior)</li> <li><code>SPI_t-2</code> - SPI (2 months prior)</li> <li><code>VapourPres_t-1</code> - Vapour Pressure (1 month prior)</li> <li><code>WetDayFreq_t-1</code> - Wet Day Frequency (1 month prior)</li> </ol> <h2>Prediction Results</h2> <h3>January 2023 Drought Predictions</h3> <p>The model identified <strong>11 districts</strong> at risk of drought:</p> <ul> <li>2 districts in Andhra Pradesh</li> <li>3 districts in Karnataka</li> <li>6 districts in Maharashtra</li> </ul> <p>All predictions include geographic coordinates for mapping and visualization.</p> <h2>Explainability Highlights</h2> <h3>SHAP Analysis</h3> <ul> <li>Global feature importance rankings</li> <li>Feature impact direction (positive/negative)</li> <li>State-specific feature importance variations</li> </ul> <h3>LIME Case Studies</h3> <ul> <li><strong>Gujarat Drought Case</strong>: Patan district, December 2022</li> <li><strong>Bihar Non-Drought Case</strong>: Siwan district, October 2022</li> <li>Detailed feature contribution analysis for each prediction</li> </ul> <h2>Dependencies</h2> <p>All required packages are listed in <code>requirements.txt</code>:</p> <ul> <li>Core: numpy, pandas, scipy</li> <li>ML: scikit-learn, xgboost, imbalanced-learn</li> <li>Visualization: matplotlib, seaborn, folium</li> <li>Explainability: shap, lime</li> <li>Utilities: tqdm, cloudpickle</li> </ul> <h2>Research Applications</h2> <p>This release is suitable for:</p> <ul> <li><strong>Academic Research</strong>: Drought prediction and climate modeling</li> <li><strong>Policy Making</strong>: Early warning systems and resource allocation</li> <li><strong>Agricultural Planning</strong>: Crop planning and water management</li> <li><strong>Climate Studies</strong>: Understanding drought patterns and trends</li> </ul> <h2>Notes</h2> <ul> <li>The notebook was originally developed in Google Colab - update file paths for local execution</li> <li>Pre-trained model is optimized for the included dataset</li> <li>For new data, retrain the model with the provided pipeline</li> <li>All visualizations are saved in the <code>Figures/</code> directory</li> </ul> <p><strong>Release Date</strong>: December 25, 2025<br><strong>Version</strong>: 1.0.0<br><strong>Status</strong>: Stable</p> <p><em>For questions or suggestions, please open an issue.</em></p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18053879 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Drought Prediction using ML and Ensemble Models along with Explainable AI using SHAP and LIME: v1.0.0 - Initial Release EHTESHAM ASHRAF <h2> Initial Release - Drought Prediction System</h2> <p>We're excited to release the first version of our comprehensive <strong>Drought Prediction System</strong> using Machine Learning and Explainable AI. This release includes a complete end-to-end pipeline for predicting drought conditions across Indian districts with state-of-the-art model performance and full interpretability.</p> <h3>Core Features</h3> <ul> <li><strong>Complete ML Pipeline</strong>: End-to-end workflow from data preprocessing to model deployment</li> <li><strong>Multi-Model Comparison</strong>: Evaluation of Random Forest, SVM, KNN, and XGBoost models</li> <li><strong>Advanced Feature Engineering</strong>: Time-lagged features (t-1 to t-6 months) capturing temporal dependencies</li> <li><strong>Class Imbalance Handling</strong>: Comparison of SMOTE and weighted loss approaches</li> <li><strong>Hyperparameter Optimization</strong>: Automated tuning using RandomizedSearchCV</li> <li><strong>Explainable AI</strong>: Full interpretability with SHAP and LIME explanations</li> <li><strong>Pre-trained Model</strong>: Ready-to-use XGBoost model with 97.2% accuracy</li> </ul> <h3>Performance Highlights</h3> <p>| Metric | Best Model (XGBoost) |</p> <p>|--------------------|-------------------------|</p> <p>| <strong>Accuracy</strong> | <strong>97.2%</strong> |</p> <p>| <strong>F1 Score</strong> | <strong>80.4% </strong> |</p> <p>| <strong>Macro Precision</strong> | 88.6% |</p> <p>| <strong>Macro Recall </strong> | 90.3% |</p> <p>| <strong>PR-AUC</strong> | 0.881 |</p> <h3> Explainability Features</h3> <ul> <li><strong>SHAP Analysis</strong>: Global and local feature importance with beeswarm plots</li> <li><strong>LIME Explanations</strong>: Case studies for drought and non-drought scenarios</li> <li><strong>State-specific Insights</strong>: Top 5 important features per state</li> <li><strong>Probabilistic Policy Insights</strong>: Actionable recommendations for policymakers</li> </ul> <h3>Dataset Coverage</h3> <ul> <li><strong>49 Districts</strong> across <strong>5 Indian States</strong></li> <li><strong>29,988 Observations</strong> of historical meteorological data</li> <li><strong>42 Engineered Features</strong> including time-lagged variables</li> <li><strong>5 Decades</strong> of temporal coverage</li> </ul> <h2>What's Included</h2> <h3>Code & Models</h3> <ul> <li>✅ Complete Jupyter notebook with all analysis steps</li> <li>✅ Pre-trained XGBoost model (<code>xgbmodel_files.pkl</code>)</li> <li>✅ StandardScaler and LabelEncoders for preprocessing</li> <li>✅ Requirements file with all dependencies</li> </ul> <h3>Visualizations</h3> <ul> <li>✅ Time series plots for all meteorological variables</li> <li>✅ Model performance comparison charts</li> <li>✅ SHAP beeswarm and decision plots</li> <li>✅ LIME explanation plots and HTML files</li> <li>✅ Interactive Folium maps for predictions</li> </ul> <h3>Documentation</h3> <ul> <li>✅ Comprehensive README with installation instructions</li> <li>✅ Methodology documentation</li> <li>✅ Usage examples and code snippets</li> </ul> <h2>Key Capabilities</h2> <h3>1. <strong>Data Preprocessing</strong></h3> <ul> <li>Automatic handling of duplicate district names</li> <li>Label encoding for categorical variables</li> <li>Cyclical encoding for temporal features (sine/cosine)</li> <li>Standard scaling for numerical features</li> </ul> <h3>2. <strong>Feature Engineering</strong></h3> <ul> <li>Time-lagged features (1-6 months prior)</li> <li>36 additional lagged features</li> <li>Total of 42 engineered features</li> </ul> <h3>3. <strong>Model Training</strong></h3> <ul> <li>Multiple ML algorithms comparison</li> <li>Automated hyperparameter tuning</li> <li>Cross-validation with StratifiedKFold</li> <li>Class imbalance handling strategies</li> </ul> <h3>4. <strong>Model Interpretability</strong></h3> <ul> <li>SHAP values for global feature importance</li> <li>LIME explanations for individual predictions</li> <li>Uncertainty quantification</li> <li>State-specific feature importance analysis</li> </ul> <h3>5. <strong>Predictions & Insights</strong></h3> <ul> <li>January 2023 drought predictions for 11 districts</li> <li>Probabilistic policy insights</li> <li>Feature sensitivity analysis</li> <li>Geographic visualization of predictions</li> </ul> <h2>Quick Start</h2> <h3>Installation</h3> <pre><code># Clone the repository git clone <repository-url> cd "Drought Prediction Final" # Create virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install -r requirements.txt </code></pre> <h3>Load Pre-trained Model</h3> <pre><code>import pickle with open('Code/xgbmodel_files.pkl', 'rb') as f: xgbmodel_files = pickle.load(f) best_xgb_model = xgbmodel_files[0] # Trained XGBoost model sc = xgbmodel_files[1] # StandardScaler le1 = xgbmodel_files[2] # LabelEncoder for DISTRICT le2 = xgbmodel_files[3] # LabelEncoder for STATE </code></pre> <h2>Technical Details</h2> <h3>Models Evaluated</h3> <ul> <li><strong>Random Forest</strong>: 400 trees, 96.28% accuracy</li> <li><strong>SVM</strong>: RBF kernel, 95.57% accuracy</li> <li><strong>KNN</strong>: K-Nearest Neighbors, 95.72% accuracy</li> <li><strong>XGBoost</strong>: Gradient boosting (best performer), 97.2% accuracy</li> </ul> <h3>Best Model Configuration</h3> <ul> <li><strong>Algorithm</strong>: XGBoost with weighted loss</li> <li><strong>Hyperparameters</strong>: Optimized via RandomizedSearchCV</li> <li><strong>Class Imbalance</strong>: Handled with <code>scale_pos_weight=12.77</code></li> <li><strong>Cross-Validation</strong>: 5-fold StratifiedKFold</li> </ul> <h3>Top Important Features</h3> <ol> <li><code>SPI_t-1</code> - Standardized Precipitation Index (1 month prior)</li> <li><code>Precipitation_t-6</code> - Precipitation (6 months prior)</li> <li><code>SPI_t-2</code> - SPI (2 months prior)</li> <li><code>VapourPres_t-1</code> - Vapour Pressure (1 month prior)</li> <li><code>WetDayFreq_t-1</code> - Wet Day Frequency (1 month prior)</li> </ol> <h2>Prediction Results</h2> <h3>January 2023 Drought Predictions</h3> <p>The model identified <strong>11 districts</strong> at risk of drought:</p> <ul> <li>2 districts in Andhra Pradesh</li> <li>3 districts in Karnataka</li> <li>6 districts in Maharashtra</li> </ul> <p>All predictions include geographic coordinates for mapping and visualization.</p> <h2>Explainability Highlights</h2> <h3>SHAP Analysis</h3> <ul> <li>Global feature importance rankings</li> <li>Feature impact direction (positive/negative)</li> <li>State-specific feature importance variations</li> </ul> <h3>LIME Case Studies</h3> <ul> <li><strong>Gujarat Drought Case</strong>: Patan district, December 2022</li> <li><strong>Bihar Non-Drought Case</strong>: Siwan district, October 2022</li> <li>Detailed feature contribution analysis for each prediction</li> </ul> <h2>Dependencies</h2> <p>All required packages are listed in <code>requirements.txt</code>:</p> <ul> <li>Core: numpy, pandas, scipy</li> <li>ML: scikit-learn, xgboost, imbalanced-learn</li> <li>Visualization: matplotlib, seaborn, folium</li> <li>Explainability: shap, lime</li> <li>Utilities: tqdm, cloudpickle</li> </ul> <h2>Research Applications</h2> <p>This release is suitable for:</p> <ul> <li><strong>Academic Research</strong>: Drought prediction and climate modeling</li> <li><strong>Policy Making</strong>: Early warning systems and resource allocation</li> <li><strong>Agricultural Planning</strong>: Crop planning and water management</li> <li><strong>Climate Studies</strong>: Understanding drought patterns and trends</li> </ul> <h2>Notes</h2> <ul> <li>The notebook was originally developed in Google Colab - update file paths for local execution</li> <li>Pre-trained model is optimized for the included dataset</li> <li>For new data, retrain the model with the provided pipeline</li> <li>All visualizations are saved in the <code>Figures/</code> directory</li> </ul> <p><strong>Release Date</strong>: December 25, 2025<br><strong>Version</strong>: 1.0.0<br><strong>Status</strong>: Stable</p> <p><em>For questions or suggestions, please open an issue.</em></p> |
| title | Drought Prediction using ML and Ensemble Models along with Explainable AI using SHAP and LIME: v1.0.0 - Initial Release |
| url | https://doi.org/10.5281/zenodo.18053879 |