diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 53d9c55..1941e8c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,112 +1,107 @@ -# Predictive Analysis - Student Success Classification +# Student Retention Predictor - AI Developer Guide -## Project Overview -This is a comprehensive data science project analyzing student academic success using a dataset with 37 features and 4,424 instances. The goal is to predict whether students will **Dropout**, **Graduate**, or remain **Enrolled** based on academic, demographic, and economic factors. +## Architecture Overview -## Data Architecture -- **Source**: `./data/data.csv` - Student academic success dataset from UCI ML Repository -- **Target Variable**: 3-class categorical (`Dropout`, `Graduate`, `Enrolled`) with class imbalance (~50% Graduate, 32% Dropout, 17% Enrolled) -- **Features**: 36 predictors including academic performance, demographics, socioeconomic factors, and macroeconomic indicators -- **Data Quality**: No missing values, no duplicates, clean dataset +This is a **Streamlit-based ML application** that predicts student academic outcomes (Graduate/Dropout/Enrolled) using a Random Forest model trained on 36 features. The app uses role-based authentication and a clean modular architecture. -## Key Data Processing Patterns - -### Column Naming Conventions -- Remove apostrophes from column names: `Mother's qualification` → `Mothers qualification` -- Fix typos consistently: `Nacionality` → `Nationality` -- Use snake_case or maintain original spacing for display purposes - -### Feature Engineering Workflow -1. **Categorical Encoding**: Label encode target variable (Dropout=0, Enrolled=1, Graduate=2) -2. **High Cardinality Handling**: One-hot encode categorical features but consider binning for `Course` and `Application order` columns -3. **Outlier Detection**: Pay attention to curricular units and age at enrollment for extreme values -4. **Class Imbalance**: Address using resampling techniques before modeling - -## Notebook Structure & Workflow -The main analysis follows this sequential pattern: - -### 1. Data Loading & Initial EDA -```python -# Standard data loading pattern -dataFileNamePath = "./data/data.csv" -masterRawData = pd.read_csv(dataFileNamePath) -data = masterRawData.copy() # Always work with a copy -``` - -### 2. Data Cleaning & Preprocessing -- Column name standardization (remove apostrophes, fix typos) -- Null/duplicate checks (this dataset has none) -- Feature type identification and conversion - -### 3. Exploratory Data Analysis -- Distribution analysis with histograms for numeric features -- Categorical variable analysis with count plots -- Target variable distribution and class balance assessment -- Feature correlation analysis and outlier detection - -### 4. Feature Engineering -- One-hot encoding for categorical variables (results in ~105 features after encoding) -- Label encoding for target variable -- Feature scaling considerations for ML models - -### 5. Modeling Approach -- Train/test split (75/25 split standard) -- Baseline logistic regression model -- Handle class imbalance before final modeling -- Multiple algorithm comparison +### Core Components +- **`app.py`**: Main Streamlit application with auth decorators and component orchestration +- **`src/components/`**: UI components split into form and prediction modules +- **`src/config/`**: Theme, mappings, and configuration constants +- **`src/utils/`**: Authentication, model utilities, and business logic +- **`models/`**: ML model training and serialized model files +- **`data/`**: CSV dataset and preprocessing pipeline ## Development Patterns -### Display & Visualization Setup -Always include this setup for proper notebook display: +### Authentication System +Uses **decorator-based auth** with role hierarchy (admin > educator > user): ```python -# Expand jupyter display width -from IPython.display import display, HTML -display(HTML("")) - -# Standard display options -pd.set_option('display.max_columns', None) -pd.set_option('display.width', 150) - -# Standard imports -import pandas as pd -import numpy as np -import matplotlib.pyplot as plt -import seaborn as sns -import warnings -warnings.filterwarnings('ignore') -%matplotlib inline +@require_auth +def main(): + # Auth required for entire app + +# Role-based features +if check_role_permission("educator"): + render_export_section() ``` -### Data Exploration Best Practices -- Use `data.nunique().sort_values()` to identify categorical vs numeric features -- Apply `.describe().T` for transposed statistical summaries -- Use `.filter(like='pattern')` to examine related columns (e.g., `Curricular`) -- Print statements with controlled display width for viewing all columns +User credentials stored in `users.json` with SHA-256 hashing. Default admin: `admin`/`admin123`. -### Feature Analysis Patterns -- **High Cardinality Features**: `Course` (51 unique), `Application order` (9 unique) - consider binning -- **Binary Features**: Most encoded as 0/1 (Gender, Scholarship holder, etc.) -- **Continuous Features**: Academic grades, age, curricular units, economic indicators -- **Ordinal Features**: Application order, qualification grades +### Component Architecture +Components follow **separation of concerns**: +- `form_components.py`: All input forms and sidebar widgets +- `prediction_components.py`: Results display and model interaction +- All components accept preprocessed data and return structured dictionaries -## Code Quality Standards -- Always create a copy of raw data before manipulation -- Use descriptive variable names: `masterRawData`, `dataFileNamePath` -- Include shape and info statements after major transformations -- Comment data quality observations directly in markdown cells -- Use `.sample()` for quick data inspection rather than `.head()` when exploring patterns +### Configuration Pattern +**Centralized config** in `src/config/`: +- `mappings.py`: Feature mappings, presets, validation rules +- `theme.py`: Rose Pine color scheme with CSS-in-Python +- Use `create_feature_mappings()` for all dropdowns/selects -## Performance Considerations -- Dataset size is manageable (4,424 rows) - no special memory handling needed -- One-hot encoding creates ~105 features - monitor for overfitting -- Class imbalance requires resampling strategies -- Consider feature selection techniques for high-dimensional encoded data +### Model Integration +**Cached model loading** with preprocessing pipeline: +```python +@st.cache_data +def load_model_and_info(): + # Returns model, preprocessing_info, feature_names +``` -## Domain Knowledge -This is an **educational/academic success prediction** problem where: -- Students are classified into 3 outcomes based on their academic journey -- Economic factors (GDP, inflation, unemployment) are included as external influences -- Academic performance metrics are key predictors -- Demographic and socioeconomic factors provide additional context -- The prediction can help institutions identify at-risk students early for intervention \ No newline at end of file +Features undergo **one-hot encoding** to match training format. Missing columns auto-filled with zeros. + +## Development Workflows + +### Running the Application +```bash +streamlit run app.py +``` + +### Model Training +```bash +cd models +python train_and_save_model.py +``` +Generates: `random_forest_model.pkl`, `preprocessing_info.pkl`, `feature_names.pkl` + +### Data Pipeline +1. CSV data in `data/data.csv` +2. Notebook analysis in `notebook/predictive_analysis.ipynb` +3. Model training script recreates notebook preprocessing +4. Streamlit app loads serialized models + +## Project-Specific Conventions + +### Form State Management +Use **preset configurations** from `mappings.py`: +- `PRESET_CONFIGURATIONS`: Default values for different student types +- `ACADEMIC_DEFAULTS`: Semester performance presets +- All forms accept `preset` parameter for consistent state + +### Validation & Recommendations +- `validate_input()`: Business rule validation (age limits, unit consistency) +- `generate_recommendations()`: Personalized advice based on prediction + input +- Validation messages use emoji prefixes: ⚠️ 🚫 ✅ + +### Styling System +**Rose Pine theme** with CSS variables defined in `theme.py`. Prediction boxes use gradient styling with hover effects. All UI follows dark theme with custom CSS injected via `st.markdown()`. + +### Error Handling +- Model files missing → `st.error()` + `st.stop()` +- Authentication failure → login form re-render +- Invalid input → warning boxes with specific guidance + +## Key Files for Modifications + +- **Feature changes**: Update `mappings.py` constants and `train_and_save_model.py` +- **UI/UX changes**: Modify components in `src/components/` +- **Authentication**: Extend `src/utils/auth.py` role system +- **Model updates**: Retrain via `models/train_and_save_model.py` +- **Styling**: Adjust `theme.py` CSS variables + +## Data Flow +1. User input collected via sidebar forms +2. Input validated and preprocessed to match training features +3. Model prediction with confidence scores +4. Results displayed with personalized recommendations +5. Role-based export functionality for educators/admins \ No newline at end of file diff --git a/.idea/.gitignore b/.idea/.gitignore deleted file mode 100644 index b58b603..0000000 --- a/.idea/.gitignore +++ /dev/null @@ -1,5 +0,0 @@ -# Default ignored files -/shelf/ -/workspace.xml -# Editor-based HTTP Client requests -/httpRequests/ diff --git a/.idea/inspectionProfiles/profiles_settings.xml b/.idea/inspectionProfiles/profiles_settings.xml deleted file mode 100644 index 105ce2d..0000000 --- a/.idea/inspectionProfiles/profiles_settings.xml +++ /dev/null @@ -1,6 +0,0 @@ - - - - \ No newline at end of file diff --git a/.idea/misc.xml b/.idea/misc.xml deleted file mode 100644 index bbaca11..0000000 --- a/.idea/misc.xml +++ /dev/null @@ -1,7 +0,0 @@ - - - - - - \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml deleted file mode 100644 index e1b5bf3..0000000 --- a/.idea/modules.xml +++ /dev/null @@ -1,8 +0,0 @@ - - - - - - - - \ No newline at end of file diff --git a/.idea/student-retention.iml b/.idea/student-retention.iml deleted file mode 100644 index 42b6342..0000000 --- a/.idea/student-retention.iml +++ /dev/null @@ -1,17 +0,0 @@ - - - - - - - - - - - - - - \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml deleted file mode 100644 index 35eb1dd..0000000 --- a/.idea/vcs.xml +++ /dev/null @@ -1,6 +0,0 @@ - - - - - - \ No newline at end of file