andwati/student-retention
1
0
Fork 0
mirror of https://github.com/andwati/student-retention.git synced 2026-10-18 08:52:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

yolo

Browse Source
This commit is contained in:
andwati
2025-09-28 20:02:12 +03:00
parent 9f2087f6d2
commit a8eba6d56e
7 changed files with 94 additions and 148 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/copilot-instructions.md
+94 -99
View File
@@ -1,112 +1,107 @@
# Predictive Analysis - Student Success Classification
# Student Retention Predictor - AI Developer Guide
## Project Overview
This is a comprehensive data science project analyzing student academic success using a dataset with 37 features and 4,424 instances. The goal is to predict whether students will **Dropout**, **Graduate**, or remain **Enrolled** based on academic, demographic, and economic factors.
## Architecture Overview
## Data Architecture
- **Source**: `./data/data.csv` - Student academic success dataset from UCI ML Repository
- **Target Variable**: 3-class categorical (`Dropout`, `Graduate`, `Enrolled`) with class imbalance (~50% Graduate, 32% Dropout, 17% Enrolled)
- **Features**: 36 predictors including academic performance, demographics, socioeconomic factors, and macroeconomic indicators
- **Data Quality**: No missing values, no duplicates, clean dataset
This is a **Streamlit-based ML application** that predicts student academic outcomes (Graduate/Dropout/Enrolled) using a Random Forest model trained on 36 features. The app uses role-based authentication and a clean modular architecture.
## Key Data Processing Patterns
### Column Naming Conventions
- Remove apostrophes from column names: `Mother's qualification` → `Mothers qualification`
- Fix typos consistently: `Nacionality` → `Nationality`
- Use snake_case or maintain original spacing for display purposes
### Feature Engineering Workflow
1. **Categorical Encoding**: Label encode target variable (Dropout=0, Enrolled=1, Graduate=2)
2. **High Cardinality Handling**: One-hot encode categorical features but consider binning for `Course` and `Application order` columns
3. **Outlier Detection**: Pay attention to curricular units and age at enrollment for extreme values
4. **Class Imbalance**: Address using resampling techniques before modeling
## Notebook Structure & Workflow
The main analysis follows this sequential pattern:
### 1. Data Loading & Initial EDA
```python
# Standard data loading pattern
dataFileNamePath = "./data/data.csv"
masterRawData = pd.read_csv(dataFileNamePath)
data = masterRawData.copy() # Always work with a copy
```
### 2. Data Cleaning & Preprocessing
- Column name standardization (remove apostrophes, fix typos)
- Null/duplicate checks (this dataset has none)
- Feature type identification and conversion
### 3. Exploratory Data Analysis
- Distribution analysis with histograms for numeric features
- Categorical variable analysis with count plots
- Target variable distribution and class balance assessment
- Feature correlation analysis and outlier detection
### 4. Feature Engineering
- One-hot encoding for categorical variables (results in ~105 features after encoding)
- Label encoding for target variable
- Feature scaling considerations for ML models
### 5. Modeling Approach
- Train/test split (75/25 split standard)
- Baseline logistic regression model
- Handle class imbalance before final modeling
- Multiple algorithm comparison
### Core Components
- **`app.py`**: Main Streamlit application with auth decorators and component orchestration
- **`src/components/`**: UI components split into form and prediction modules
- **`src/config/`**: Theme, mappings, and configuration constants
- **`src/utils/`**: Authentication, model utilities, and business logic
- **`models/`**: ML model training and serialized model files
- **`data/`**: CSV dataset and preprocessing pipeline
## Development Patterns
### Display & Visualization Setup
Always include this setup for proper notebook display:
### Authentication System
Uses **decorator-based auth** with role hierarchy (admin > educator > user):
```python
# Expand jupyter display width
from IPython.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
# Standard display options
pd.set_option('display.max_columns', None)
pd.set_option('display.width', 150)
# Standard imports
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
@require_auth
def main():
# Auth required for entire app
# Role-based features
if check_role_permission("educator"):
render_export_section()
```
### Data Exploration Best Practices
- Use `data.nunique().sort_values()` to identify categorical vs numeric features
- Apply `.describe().T` for transposed statistical summaries
- Use `.filter(like='pattern')` to examine related columns (e.g., `Curricular`)
- Print statements with controlled display width for viewing all columns
User credentials stored in `users.json` with SHA-256 hashing. Default admin: `admin`/`admin123`.
### Feature Analysis Patterns
- **High Cardinality Features**: `Course` (51 unique), `Application order` (9 unique) - consider binning
- **Binary Features**: Most encoded as 0/1 (Gender, Scholarship holder, etc.)
- **Continuous Features**: Academic grades, age, curricular units, economic indicators
- **Ordinal Features**: Application order, qualification grades
### Component Architecture
Components follow **separation of concerns**:
- `form_components.py`: All input forms and sidebar widgets
- `prediction_components.py`: Results display and model interaction
- All components accept preprocessed data and return structured dictionaries
## Code Quality Standards
- Always create a copy of raw data before manipulation
- Use descriptive variable names: `masterRawData`, `dataFileNamePath`
- Include shape and info statements after major transformations
- Comment data quality observations directly in markdown cells
- Use `.sample()` for quick data inspection rather than `.head()` when exploring patterns
### Configuration Pattern
**Centralized config** in `src/config/`:
- `mappings.py`: Feature mappings, presets, validation rules
- `theme.py`: Rose Pine color scheme with CSS-in-Python
- Use `create_feature_mappings()` for all dropdowns/selects
## Performance Considerations
- Dataset size is manageable (4,424 rows) - no special memory handling needed
- One-hot encoding creates ~105 features - monitor for overfitting
- Class imbalance requires resampling strategies
- Consider feature selection techniques for high-dimensional encoded data
### Model Integration
**Cached model loading** with preprocessing pipeline:
```python
@st.cache_data
def load_model_and_info():
# Returns model, preprocessing_info, feature_names
```
## Domain Knowledge
This is an **educational/academic success prediction** problem where:
- Students are classified into 3 outcomes based on their academic journey
- Economic factors (GDP, inflation, unemployment) are included as external influences
- Academic performance metrics are key predictors
- Demographic and socioeconomic factors provide additional context
- The prediction can help institutions identify at-risk students early for intervention
Features undergo **one-hot encoding** to match training format. Missing columns auto-filled with zeros.
## Development Workflows
### Running the Application
```bash
streamlit run app.py
```
### Model Training
```bash
cd models
python train_and_save_model.py
```
Generates: `random_forest_model.pkl`, `preprocessing_info.pkl`, `feature_names.pkl`
### Data Pipeline
1. CSV data in `data/data.csv`
2. Notebook analysis in `notebook/predictive_analysis.ipynb`
3. Model training script recreates notebook preprocessing
4. Streamlit app loads serialized models
## Project-Specific Conventions
### Form State Management
Use **preset configurations** from `mappings.py`:
- `PRESET_CONFIGURATIONS`: Default values for different student types
- `ACADEMIC_DEFAULTS`: Semester performance presets
- All forms accept `preset` parameter for consistent state
### Validation & Recommendations
- `validate_input()`: Business rule validation (age limits, unit consistency)
- `generate_recommendations()`: Personalized advice based on prediction + input
- Validation messages use emoji prefixes: ⚠️ 🚫 ✅
### Styling System
**Rose Pine theme** with CSS variables defined in `theme.py`. Prediction boxes use gradient styling with hover effects. All UI follows dark theme with custom CSS injected via `st.markdown()`.
### Error Handling
- Model files missing → `st.error()` + `st.stop()`
- Authentication failure → login form re-render
- Invalid input → warning boxes with specific guidance
## Key Files for Modifications
- **Feature changes**: Update `mappings.py` constants and `train_and_save_model.py`
- **UI/UX changes**: Modify components in `src/components/`
- **Authentication**: Extend `src/utils/auth.py` role system
- **Model updates**: Retrain via `models/train_and_save_model.py`
- **Styling**: Adjust `theme.py` CSS variables
## Data Flow
1. User input collected via sidebar forms
2. Input validated and preprocessed to match training features
3. Model prediction with confidence scores
4. Results displayed with personalized recommendations
5. Role-based export functionality for educators/admins
.idea/.gitignore
Generated
-5
View File
@@ -1,5 +0,0 @@
# Default ignored files
/shelf/
/workspace.xml
# Editor-based HTTP Client requests
/httpRequests/
.idea/inspectionProfiles/profiles_settings.xml
Generated
-6
View File
@@ -1,6 +0,0 @@
<component name="InspectionProjectProfileManager">
<settings>
<option name="USE_PROJECT_PROFILE" value="false" />
<version value="1.0" />
</settings>
</component>
.idea/misc.xml
Generated
-7
View File
@@ -1,7 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="Black">
<option name="sdkName" value="Python 3.13 (student-retention)" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.13 (student-retention)" project-jdk-type="Python SDK" />
</project>
.idea/modules.xml
Generated
-8
View File
@@ -1,8 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ProjectModuleManager">
<modules>
<module fileurl="file://$PROJECT_DIR$/.idea/student-retention.iml" filepath="$PROJECT_DIR$/.idea/student-retention.iml" />
</modules>
</component>
</project>
.idea/student-retention.iml
Generated
-17
View File
@@ -1,17 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<module type="PYTHON_MODULE" version="4">
<component name="NewModuleRootManager">
<content url="file://$MODULE_DIR$">
<excludeFolder url="file://$MODULE_DIR$/.venv" />
</content>
<orderEntry type="jdk" jdkName="Python 3.13 (student-retention)" jdkType="Python SDK" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
<component name="PyDocumentationSettings">
<option name="format" value="PLAIN" />
<option name="myDocStringFormat" value="Plain" />
</component>
<component name="TestRunnerService">
<option name="PROJECT_TEST_RUNNER" value="py.test" />
</component>
</module>
.idea/vcs.xml
Generated
-6
View File
@@ -1,6 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="VcsDirectoryMappings">
<mapping directory="" vcs="Git" />
</component>
</project>