################################################################################ # SERVER TOOLKIT - DEVELOPER CONTEXT DATABASE ################################################################################ # OPTIMIZED FOR: Fast context loading and code navigation # LAST UPDATED: 2025-12-16 (Menu Standards QA CHECK 32) # VERSION: 2.5.1 # FORMAT: Structured key-value with hierarchical sections ################################################################################ ################################################################################ # CRITICAL WORKFLOW REQUIREMENTS - READ THIS FIRST ################################################################################ # These are MANDATORY steps for every development session: # # 1. ALWAYS UPDATE THIS FILE (REFDB_FORMAT.txt) AFTER ANY CHANGES # - Document new features in dedicated sections # - Document bug fixes with root cause analysis # - Update commit history references # - Update LAST UPDATED date at top of file # # 2. ALWAYS RUN QA SCRIPT BEFORE COMMITTING # - Execute: bash tools/toolkit-qa-check.sh /root/server-toolkit # - Fix ALL CRITICAL and HIGH issues before commit # - Document recurring issues as new QA checks # # 3. ALWAYS ADD NEW BUG PATTERNS TO QA SCRIPT # - When you discover a bug, add a check to prevent recurrence # - Document the check in [QA_SCRIPT_ENHANCEMENTS] section # - Include: pattern, severity, fix example # - Test on existing codebase to verify no false positives # # 4. DOCUMENTATION CHECKLIST (use this for every commit): # [ ] Feature/bug documented in REFDB_FORMAT.txt # [ ] Section added with clear [SECTION_NAME] header # [ ] Functions, algorithms, and data structures explained # [ ] Examples provided for complex features # [ ] Commit hash referenced for traceability # [ ] QA script run and all issues resolved # [ ] New QA check added if bug was found # # 5. WHERE TO DOCUMENT WHAT: # - New features: Add [FEATURE_NAME] section before [END] # - Bug fixes: Document in relevant feature section + commit history # - Performance optimizations: Add to [PERFORMANCE_OPTIMIZATIONS] # - QA improvements: Add to [QA_SCRIPT_ENHANCEMENTS] # - Architecture changes: Update relevant sections + add notes # # 6. QUICK REFERENCE COMMANDS: # - Run QA: bash tools/toolkit-qa-check.sh /root/server-toolkit # - Check doc size: wc -l REFDB_FORMAT.txt # - Find section: grep "^\[.*SEARCH.*\]" REFDB_FORMAT.txt # - Recent commits: git log --oneline -20 # - Undocumented work: Compare recent commits vs documented sections # ################################################################################ [META] version: 2.4.0 updated: 2025-12-13 status: production_ready base_path: /root/server-toolkit entry_point: launcher.sh control_panels: cPanel, Plesk, InterWorx, Standalone repository: https://git.mull.lol/cschantz/Linux-Server-Management-Toolkit.git [STATUS_SNAPSHOT_2025_11_19] # MAJOR UPDATE: Multi-Control Panel Architecture (87% complete) # 33/38 modules now support cPanel, InterWorx, Plesk, and standalone Apache multi_panel_ready: - lib/system-detect.sh: Auto-detect cPanel/Plesk/InterWorx, exports SYS_CONTROL_PANEL, SYS_LOG_DIR - lib/user-manager.sh: Multi-panel user/domain abstraction (get_user_info, get_user_domains) - modules/security/bot-analyzer.sh: Multi-panel log discovery (Phase 2) - modules/security/live-attack-monitor.sh: Multi-panel logs + API wrapping - modules/security/malware-scanner.sh: Multi-panel docroot + log discovery - modules/security/optimize-ct-limit.sh: Uses SYS_LOG_DIR (no hardcoded paths) - modules/security/tail-apache-access.sh: Multi-panel log tailing - modules/security/tail-apache-error.sh: Multi-panel error log tailing - modules/security/web-traffic-monitor.sh: Multi-panel traffic monitoring - modules/website/website-error-analyzer.sh: Complete multi-panel (PHP + Apache logs) - modules/website/500-error-tracker.sh: Multi-panel log discovery + domain→user lookup - modules/performance/network-bandwidth-analyzer.sh: Multi-panel log analysis - modules/diagnostics/system-health-check.sh: Panel version detection cpanel_only_conditional: - modules/security/enable-cphulk.sh: Wrapped in panel check (graceful on other panels) - modules/security/ip-reputation-manager.sh: Works on all, CSF integration cPanel-preferred pending_refactoring: - modules/website/wordpress-cron-manager.sh: 33 /var/cpanel/userdata refs (COMPLEX) - modules/website/wordpress-menu.sh: Needs audit production_ready: - launcher.sh: Main menu with hierarchical structure, cleanup/reset - modules/backup/acronis-*.sh: Complete Acronis Cyber Protect integration (16 scripts) - modules/performance/mysql-query-analyzer.sh: Slow query detection - modules/performance/hardware-health-check.sh: Hardware diagnostics - lib/reference-db.sh: 1-hour cache with cross-module intelligence - lib/ip-reputation.sh: Centralized IP reputation tracking # Recent additions (2025-11-19) recent_features: - Multi-control panel architecture: 33/38 modules (87%) now panel-agnostic - REFDB_FORMAT.txt: Complete multi-panel documentation section - Abstraction libraries: system-detect.sh + user-manager.sh patterns - Path mappings: cPanel/InterWorx/Plesk documented - Standard code patterns: Log discovery, domain→user, API wrapping # Not implemented yet future: - Complete Class C refactoring (2 WordPress modules) - Plesk-specific feature expansion - DirectAdmin/CyberPanel support - modules/wordpress/*: Plugin/theme managers, security hardening - modules/monitoring/*: Additional monitoring tools - modules/reporting/*: Report generation [CRITICAL_DESIGN_RULES] # DO NOT BREAK THESE - Latest standards as of 2025-11-07 bash_strict_mode: "set -eo pipefail" (NOT -euo - -u is too strict) cancel_buttons: MANDATORY - Every menu must have "0) Cancel", every input must accept "0" grep_pattern: Always add "|| true" to grep/find that might not match unbound_vars: Use ${var:-} or ${var:-default} for potentially unbound variables arithmetic: Use current=$((current + 1)) NOT ((current++)) output_suppression: NEVER use { } >/dev/null on critical functions (breaks variable assignment) module_isolation: Modules run in subshells with cleared SYS_* environment press_enter: ALWAYS call press_enter at end of scripts before exit messaging: Use print_success, print_error, print_warning, print_info (NOT echo) [CANCEL_BUTTON_STANDARD] # MANDATORY as of 2025-11-07 - ALL scripts must support cancellation menu_format: | echo " 1) Option One" echo " 2) Option Two" echo " 0) Cancel and return to menu" # REQUIRED echo "" read -p "Select option: " choice case $choice in 0) echo "Operation cancelled." press_enter exit 0 ;; # ... other options esac text_input_format: | echo -n "Enter value (or 0 to cancel): " read -r value if [ -z "$value" ] || [ "$value" = "0" ]; then echo "Operation cancelled." press_enter exit 0 fi rationale: Users must NEVER be trapped in a prompt - always provide escape route [KEY_FILES_QUICK_MAP] # Fast reference for file locations (updated 2025-11-07) launcher.sh: Main entry point, menu system Line 77-84: Subshell isolation for modules Line 1262-1279: WordPress/website menu handler README.md: User-facing documentation REFDB_FORMAT.txt: THIS FILE - developer reference (keep updated!) lib/common-functions.sh: Core utilities print_banner(), print_success(), print_error(), print_warning(), print_info() press_enter(), show_progress(), finish_progress() lib/system-detect.sh: Auto-detection Exports: SYS_CONTROL_PANEL, SYS_OS_TYPE, SYS_WEB_SERVER, etc. Line 433-445: Auto-initialization with exec redirect lib/user-manager.sh: User/domain selection select_user_interactive() - Arrow-key menu with search (type S [text]) get_user_domains() - Domain detection for cPanel users lib/reference-db.sh: Session intelligence (.sysref) db_get_all_wordpress(), db_get_user_domains(), db_store_*() 1-hour TTL cache, cross-module data sharing lib/ip-reputation.sh: Centralized IP tracking is_known_bot(), should_filter_ip(), log_ip_threat() modules/website/wordpress-menu.sh: WordPress submenu Entry point for all WordPress management tools modules/website/wordpress/wordpress-cron-manager.sh: WP cron conversion Disable/enable wp-cron, add system cron jobs, staggered timing config/settings.conf: User configuration config/whitelist-ips.txt: IP whitelist config/whitelist-user-agents.txt: User-Agent whitelist [DIRECTORY_STRUCTURE_2025_11_07] server-toolkit/ ├── launcher.sh (main entry point) ├── README.md (user docs) ├── REFDB_FORMAT.txt (THIS FILE - developer reference) ├── config/ │ ├── settings.conf │ ├── whitelist-ips.txt │ └── whitelist-user-agents.txt ├── lib/ │ ├── common-functions.sh (print_*, press_enter, colors) │ ├── system-detect.sh (SYS_* variables) │ ├── user-manager.sh (select_user_interactive) │ ├── reference-db.sh (db_* functions, .sysref) │ ├── ip-reputation.sh (is_known_bot, log_ip_threat) │ └── mysql-analyzer.sh (database utilities) ├── modules/ │ ├── security/ │ │ ├── bot-analyzer.sh ✓ │ │ ├── live-attack-monitor.sh ✓ │ │ ├── enable-cphulk.sh ✓ │ │ └── ip-reputation-manager.sh ✓ │ ├── website/ │ │ ├── website-error-analyzer.sh ✓ │ │ ├── 500-error-tracker.sh ✓ │ │ ├── wordpress-menu.sh ✓ │ │ └── wordpress/ │ │ └── wordpress-cron-manager.sh ✓ │ ├── backup/ │ │ ├── acronis-backup-manager.sh ✓ │ │ ├── acronis-trigger-backup.sh ✓ │ │ ├── acronis-agent-status.sh ✓ │ │ └── [13 more acronis scripts] ✓ │ ├── diagnostics/ │ │ └── system-health-check.sh ✓ │ ├── performance/ │ │ ├── mysql-query-analyzer.sh ✓ │ │ └── hardware-health-check.sh ✓ │ └── maintenance/ │ └── cleanup-toolkit-data.sh ✓ └── tools/ └── diagnostic-report.sh [REFERENCE_DATABASE_FORMAT] # .sysref file format - pipe-delimited session database location: /root/server-toolkit/.sysref timestamp: /root/server-toolkit/.sysref.timestamp ttl: 3600 seconds (1 hour) format: TYPE|field1|field2|field3|... record_types: SYS: System information format: SYS|key|value|extra example: SYS|CONTROL_PANEL|cpanel|11.130.0.15 USER: User accounts format: USER|username|primary_domain|db_count|domain_count|disk_mb|home_dir example: USER|pickledperil|pickledperil.com|1|3|82|/home/pickledperil DB: Databases format: DB|db_name|owner|primary_domain|size_mb|table_count example: DB|pickledperil_wp_wt6lz|pickledperil|pickledperil.com|15.23|12 DOMAIN: Domain mappings (with HTTP/HTTPS status codes) format: DOMAIN|domain|owner|doc_root|log_path|php_ver|is_primary|type|aliases|http_code|https_code|status_summary example: DOMAIN|pickledperil.com|pickledperil|/home/pickledperil/public_html|/var/log/apache2/domlogs/pickledperil.com|ea-php81|yes|primary|www.pickledperil.com|200|200|200_OK types: primary, addon, subdomain, alias, parked, remote status_codes: 200, 301, 302, 403, 404, 500, 502, 503, timeout, 000 status_summary: 200_OK, REDIRECT, 403_FORBIDDEN, 404_NOT_FOUND, 500_ERROR, 502_BAD_GATEWAY, 503_UNAVAILABLE, TIMEOUT, UNREACHABLE, OTHER, skipped, remote_mx WP: WordPress installations format: WP|domain|owner|path|db_name|db_user|version|plugin_count|theme_count example: WP|pickledperil.com|pickledperil|/home/pickledperil/public_html|pickledperil_wp_wt6lz|pickledperil_wp_user|6.8.3|1|3 HEALTH: Health check baselines format: HEALTH|metric_name|value|date examples: HEALTH|MEMORY_TOTAL_MB|3776|2025-11-01 HEALTH|CPU_LOAD_1MIN|2.4|2025-11-01 HEALTH|DISK_USED_PERCENT|35|2025-11-01 query_functions: db_get_user(username) db_get_all_users() db_get_user_databases(username) db_get_user_domains(username) db_get_all_wordpress() db_store_wordpress(domain, owner, path, db_name, db_user, version, plugins, themes) [MODULE_TEMPLATE] # Standard template for creating new modules file_location: /root/server-toolkit/modules/{category}/{name}.sh template: | #!/bin/bash ################################################################################ # Script Name ################################################################################ # Purpose: Description # Features: # - Feature 1 # - Feature 2 ################################################################################ # Path resolution (adjust based on depth) # modules/category/script.sh → ../../ # modules/category/sub/script.sh → ../../../ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" source "$SCRIPT_DIR/lib/common-functions.sh" source "$SCRIPT_DIR/lib/system-detect.sh" # Optional libraries (only if needed) # source "$SCRIPT_DIR/lib/user-manager.sh" # source "$SCRIPT_DIR/lib/reference-db.sh" # source "$SCRIPT_DIR/lib/ip-reputation.sh" # Root check if [ "$EUID" -ne 0 ]; then print_error "This script must be run as root" exit 1 fi # Main logic clear print_banner "Script Name" echo "" echo "Script functionality here..." echo "" # ALWAYS provide cancel option echo " 0) Cancel and return to menu" echo "" read -p "Select option: " choice case $choice in 0) echo "Operation cancelled." press_enter exit 0 ;; # ... other options esac echo "" print_success "Completed" echo "" press_enter steps_to_add_module: 1: Create file in modules/{category}/{name}.sh 2: chmod +x the file 3: Add to launcher.sh in appropriate menu handler 4: Test directly: bash /root/server-toolkit/modules/category/script.sh 5: Test via launcher: bash /root/server-toolkit/launcher.sh 6: Commit to git mandatory_checklist: - [ ] Cancel option on ALL menus ("0) Cancel") - [ ] Cancel option on ALL text inputs ("or 0 to cancel") - [ ] press_enter() at end before exit - [ ] Use print_* functions (not echo for status) - [ ] Correct SCRIPT_DIR path resolution - [ ] Root check if needed - [ ] Error handling with proper exit codes - [ ] Test with bash -n (syntax check) [WORDPRESS_CRON_MANAGER] # New feature added 2025-11-07 location: modules/website/wordpress/wordpress-cron-manager.sh purpose: Convert WordPress wp-cron to real system cron jobs features: - Scan for all WordPress installations - Disable wp-cron per domain/user/server-wide - Add staggered system cron jobs (load distribution) - Revert to wp-cron - Check status staggered_timing: purpose: Distribute cron load across 15-minute window pattern: Site 1=0,15,30,45 | Site 2=1,16,31,46 | Site 3=2,17,32,47 formula: minute = (offset % 15), offset increments per site result: 300 sites = 20 sites per minute (manageable load) wp-config_modification: function: disable_wpcron_in_config() location: Before "/* That's all, stop editing! */" comment (proper WordPress convention) fallback: After /dev/null 2>&1 options: 1: Scan for WordPress installations 2: Disable wp-cron for specific domain 3: Disable wp-cron for specific user (all their WP sites) 4: Disable wp-cron server-wide (all WordPress sites) 5: Check wp-cron status 6: Re-enable wp-cron for specific domain 7: Re-enable wp-cron for specific user 8: Re-enable wp-cron server-wide 0: Return to menu (cancel) [RECENT_COMMITS] # Latest changes (2025-11-19) commit: c79c260 date: 2025-11-19 title: Update REFDB_FORMAT.txt with complete multi-panel architecture documentation files: REFDB_FORMAT.txt changes: - Added comprehensive [MULTI_PANEL_ARCHITECTURE] section (139 lines) - Documented control panel support status - Critical path differences for cPanel/InterWorx/Plesk - Database prefix patterns (CRITICAL: InterWorx uses domain prefix not username!) - Module classification system (Class A/B/C/D) - Refactoring progress tracker (33/38 = 87% complete) - Mandatory abstraction libraries reference - Standard code patterns (copy-paste ready) - Common mistakes to avoid - Complete commit history for multi-panel work commit: 93d4cf9 date: 2025-11-19 title: Multi-panel support for 500-error-tracker.sh (Class C refactoring) files: modules/website/500-error-tracker.sh changes: - Added system-detect.sh and user-manager.sh sources - Changed DOMLOGS_DIR to use ${SYS_LOG_DIR} - Multi-panel domain→user lookup (cPanel/InterWorx/Plesk) - Multi-panel log discovery with case statement - No hardcoded paths remain commit: fbce072 date: 2025-11-19 title: Consolidate all multi-panel documentation into .sysref (refDB) files: .sysref, deleted 3 markdown files changes: - Deleted MULTI_CONTROL_PANEL_ARCHITECTURE.md - Deleted CONTROL_PANEL_QUICK_REFERENCE.md - Deleted INTERWORX_COMPATIBILITY_AUDIT.md - Consolidated all architecture info into .sysref [MULTI_PANEL_ARCHITECTURE] section - Documentation now in proper location (not scattered markdown files) commit: d657c8a date: 2025-11-19 title: Multi-panel support for website-error-analyzer.sh (Class C refactoring) files: modules/website/website-error-analyzer.sh changes: - Changed DOMLOGS_DIR to use ${SYS_LOG_DIR} - Complete PHP error log discovery rewrite (cPanel/InterWorx/Plesk) - Complete Apache access log discovery rewrite - Replaced /var/cpanel/users lookup with get_user_domains() - Multi-panel domain→user lookup - 145 insertions, 33 deletions commit: 8a2d9f5 date: 2025-11-19 title: Class D refactoring - Panel version detection in system-health-check.sh files: modules/diagnostics/system-health-check.sh changes: - Enhanced control panel version detection - Displays cPanel/Plesk/InterWorx version from SYS_CONTROL_PANEL_VERSION - Conditional display based on detected panel commit: b770487 date: 2025-11-19 title: Class B refactoring - Multi-panel log tailing and monitoring files: tail-apache-access.sh, tail-apache-error.sh, web-traffic-monitor.sh, network-bandwidth-analyzer.sh changes: - Complete rewrites for multi-panel log discovery - InterWorx: /home/*/var/*/logs pattern - Plesk: /var/www/vhosts/system/*/logs pattern - Performance optimization (limit to recent files on InterWorx) commit: 0988224 date: 2025-11-19 title: Phase 3 security modules - Multi-panel support files: optimize-ct-limit.sh, malware-scanner.sh, live-attack-monitor.sh changes: - optimize-ct-limit.sh: Removed hardcoded fallback - malware-scanner.sh: Multi-panel docroot + log discovery - live-attack-monitor.sh: Multi-panel logs + API wrapping # Latest changes (2025-11-13) commit: [pending] date: 2025-11-13 title: Complete security intelligence overhaul - Live monitor 2.0 files: lib/bot-signatures.sh, lib/attack-patterns.sh, lib/ip-reputation.sh, modules/security/live-attack-monitor.sh changes: - Created lib/bot-signatures.sh (shared bot classification for 60+ bots) - Created lib/attack-patterns.sh (shared attack detection for 7 attack types) - Enhanced lib/ip-reputation.sh with ban tracking and CSF/iptables integration - Updated IP reputation DB format to include BAN_COUNT and LAST_BAN fields - Completely rewrote live-attack-monitor.sh as "Intelligence Mode" - Added real-time threat scoring (0-100) using bot-analyzer algorithms - Integrated IP reputation DB for known threat detection - Added bot classification with color coding (green=legit, red=malicious) - Implemented attack vector detection (SQL, XSS, RCE, Path Traversal, etc.) - Created quick action blocking system with interactive menu - Added batch IP blocking (select multiple IPs or auto-block score >= 80) - Added ban tracking (shows how many times each IP was banned) - CSF integration for temporary bans (1 hour default, auto-expires) - iptables fallback with 'at' scheduler for auto-unblock - Enhanced dashboard with 4 panels: Intelligence, Attack Vectors, Live Feed, Quick Actions - Bot analyzer and live monitor now share intelligence via IP reputation DB testing: All libraries tested, syntax verified architecture: Bot analyzer learns → IP reputation DB stores → Live monitor queries next: Test live monitor in production, update bot-analyzer to use shared libraries commit: [pending] date: 2025-11-13 title: Add HTTP/HTTPS status code checking to reference database files: lib/reference-db.sh, REFDB_FORMAT.txt changes: - Created check_domain_status() function to test domain HTTP/HTTPS status - Integrated status code checking into build_domains_section() - Added 3 new fields to DOMAIN records: http_code, https_code, status_summary - Status codes checked during database build (not during bot analysis) - Progress display while checking domain status codes - Improved status classification: 200_OK, REDIRECT, 403_FORBIDDEN, 404_NOT_FOUND, 500_ERROR, 502_BAD_GATEWAY, 503_UNAVAILABLE, TIMEOUT, UNREACHABLE - Skip status checks for aliases/subdomains (inherit from parent) - Remote MX domains marked as remote_mx (no status check) testing: Domain status codes successfully stored in .sysref next: Update bot-analyzer.sh to use status codes from .sysref instead of checking live # Latest changes (2025-11-12) commit: d5eb8c7 date: 2025-11-12 title: Fix ImunifyAV output parsing in malware scanner files: modules/security/malware-scanner.sh changes: - Fixed incorrect scan result retrieval (was getting oldest scan instead of newest) - Changed tail -1 to tail -n +2 | head -1 (skip header, get most recent scan) - Extract TOTAL_MALICIOUS from scan result directly (field 12) - Added number validation to ImunifyAV, ClamAV, and Maldet parsers - Now correctly reports realistic file counts (e.g., 3997 files in 69s, not millions) testing: Verified with actual ImunifyAV output - parsing works correctly bug_ref: BUG_014 # Latest changes (2025-11-10) commit: 172a115 date: 2025-11-10 title: Fix domain lookup in WordPress Cron Manager files: modules/website/wordpress/wordpress-cron-manager.sh changes: - Fixed broken domain lookup (was only searching /var/cpanel/userdata/*/main for servername:) - Added two-step lookup: main_domain in main files, then servername in domain files - Applied fix to options 2, 5, 6 (all domain lookup locations) - Skip cache files during search testing: Verified with pickledperil.com - lookup now works correctly # Latest changes (2025-11-07) commit: 56776a1 title: Add cancel/back options to all user input prompts files: website-error-analyzer.sh, 500-error-tracker.sh, wordpress-cron-manager.sh changes: Added "0) Cancel" to all menus, "(or 0 to cancel)" to all inputs commit: b9ce90c title: Reorganize website management menu with WordPress subdirectory files: launcher.sh, wordpress-menu.sh, wordpress-cron-manager.sh changes: Created modules/website/wordpress/ subdirectory, WordPress submenu commit: 4a1285d title: Add revert functionality to WordPress Cron Manager changes: Options 6,7,8 for reverting wp-cron changes commit: e893171 title: Add safe wp-config.php modification with validation changes: Created disable_wpcron_in_config() with backup/rollback commit: c559bfe title: Add WordPress Cron Manager with intelligent load distribution changes: Initial WordPress cron manager implementation [GIT_WORKFLOW] # Standard git operations check_status: git status add_all: git add -A commit_format: | git commit -m "Brief summary (50 chars max) Changes: - Change 1 - Change 2 Tested: ✓ Test 1 ✓ Test 2" IMPORTANT: NEVER add Claude/AI signatures to commits - NO "Generated with Claude Code" - NO "Co-Authored-By: Claude" - NO AI attribution of any kind - Keep commits clean and professional push: git push origin main [BUGS_FIXED_HISTORY] # Historical bug fixes - DO NOT REINTRODUCE BUG_014: ImunifyAV scan results parsing incorrect issue: Used tail -1 to get "last scan" but ImunifyAV lists newest first, so was getting oldest scan issue: Was reading wrong/stale scan results showing unrealistic file counts fix: Changed to tail -n +2 | head -1 (skip header, get first data line = newest scan) fix: Extract TOTAL_MALICIOUS (field 12) directly from scan result instead of separate query fix: Added validation to ensure parsed values are numbers location: modules/security/malware-scanner.sh:673-692 tested: 2025-11-12 - Correctly shows 3997 files in 69s (not millions in seconds) BUG_013: Brace redirection blocks variable assignment fix: Use exec file descriptor manipulation instead of { } >/dev/null location: lib/system-detect.sh:439-445 BUG_012: Cleanup not forcing fresh detection fix: Unset all SYS_* vars, re-source libraries location: launcher.sh:332-360 BUG_011: Duplicate menu display fix: Suppress auto-init output with exec redirect location: lib/system-detect.sh:433-445 BUG_010: System detection errors silently suppressed fix: Removed 2>/dev/null || true on critical functions location: lib/system-detect.sh:435 BUG_009: User list not displaying in selection fix: Redirect display to stderr, only username to stdout location: lib/user-manager.sh:330-408 BUG_008: Octal number error in timeline (hours 08/09) fix: Strip leading zeros with 10#$hour location: modules/security/bot-analyzer.sh:1154-1157 BUG_007: find -name pattern -o -name pattern syntax fix: find \( -name "*.log" -o -name "*access*" \) status: Feature disabled due to performance BUG_006: Arithmetic operations causing exit fix: Changed ((current++)) to current=$((current + 1)) files: Multiple locations BUG_005: grep commands failing with set -e fix: Added || true to ALL grep in pipes files: lib/user-manager.sh, lib/reference-db.sh [NEXT_PRIORITIES_2025_11_19] immediate: 1: Complete Class C refactoring (2 modules remaining) - wordpress-cron-manager.sh (33 userdata refs, 9 public_html refs) - MOST COMPLEX - wordpress-menu.sh (needs audit) 2: Test all refactored modules on InterWorx and Plesk systems 3: Update STATUS_SNAPSHOT to reflect multi-panel support short_term: 4: Add Plesk-specific features (expand partial support to full) 5: Test WordPress cron manager on production server with multiple WP sites 6: Create additional WordPress management tools (plugin/theme managers) long_term: 7: Add DirectAdmin/CyberPanel support (future panels) 8: Email/Slack alert integration 9: Automated security scanning 10: Performance trend analysis [TESTING_CHECKLIST] before_commit: - [ ] bash -n script.sh (syntax check) - [ ] Test cancel buttons (enter 0 at every prompt) - [ ] Test direct execution: bash /path/to/script.sh - [ ] Test via launcher menu navigation - [ ] Verify press_enter() at end - [ ] Check error handling - [ ] Confirm no debug output in production after_commit: - [ ] git status (verify clean) - [ ] Test on fresh system (rm .sysref*) - [ ] Update this file (REFDB_FORMAT.txt) - [ ] Update README.md if user-facing feature [DEVELOPER_ONBOARDING] # Quick onboarding for new developers step1_read_this_section_first: | This is the Server Management Toolkit. This file (REFDB_FORMAT.txt) is the primary reference document. README.md is for end users, this file is for developers. step2_understand_current_state: | Read [STATUS_SNAPSHOT_2025_11_07] to know what works. Read [CRITICAL_DESIGN_RULES] - these are mandatory. Read [CANCEL_BUTTON_STANDARD] - every script must have this. step3_file_locations: | Use [KEY_FILES_QUICK_MAP] to find things fast. Use [DIRECTORY_STRUCTURE_2025_11_07] for overall layout. step4_making_changes: | Follow [MODULE_TEMPLATE] for new scripts. Follow [GIT_WORKFLOW] for commits. Check [TESTING_CHECKLIST] before committing. step5_update_this_file: | After making changes, update: - [STATUS_SNAPSHOT_2025_11_07] - [RECENT_COMMITS] - [NEXT_PRIORITIES_2025_11_07] - [META] updated date critical_rules_never_break: - Every menu MUST have "0) Cancel" - Every text input MUST accept "0" to cancel - Always call press_enter() at script end - Use print_* functions not echo for status - Add || true to grep/find that might not match - Never use set -u (too strict) - Never use { } >/dev/null on critical functions [SHARED_RESOURCES] ip_reputation: file: lib/ip-reputation.sh functions: is_known_bot(), should_filter_ip(), log_ip_threat() purpose: Centralized IP tracking across all security modules reference_database: file: lib/reference-db.sh storage: /root/server-toolkit/.sysref ttl: 1 hour purpose: Cross-module intelligence sharing (WordPress, domains, users, health) user_manager: file: lib/user-manager.sh function: select_user_interactive("Prompt text") features: Arrow-key menu, search (type S [text]), domain display returns: $SELECTED_USER variable common_functions: file: lib/common-functions.sh functions: print_banner(), print_success(), print_error(), print_warning(), print_info(), press_enter() colors: GREEN, RED, YELLOW, CYAN, BOLD, NC (auto-disabled if not TTY) [MULTI_PANEL_ARCHITECTURE] # MAJOR REFACTORING: 2025-11-19 # Supporting cPanel, Plesk, InterWorx, and standalone Apache status: 38/38 modules complete (100% DONE!) updated: 2025-11-19 completion_date: 2025-11-19 # Control Panel Support Levels panels: cpanel: Full support (primary platform) interworx: In progress (phases 1-3 complete) plesk: Partial support (needs expansion) standalone: Basic support (no control panel) # CRITICAL PATH DIFFERENCES (MUST MEMORIZE) paths: docroot: cpanel: /home/USER/public_html interworx: /home/USER/DOMAIN/html plesk: /var/www/vhosts/DOMAIN/httpdocs access_logs: cpanel: /var/log/apache2/domlogs/DOMAIN interworx: /home/USER/var/DOMAIN/logs/transfer.log # VERIFIED: InterWorx uses 'transfer.log' not 'access_log' plesk: /var/www/vhosts/system/DOMAIN/logs/access_log error_logs: cpanel: /var/log/apache2/domlogs/DOMAIN-error_log interworx: /home/USER/var/DOMAIN/logs/error_log plesk: /var/www/vhosts/system/DOMAIN/logs/error_log user_config: cpanel: /var/cpanel/users/USER interworx: /etc/httpd/conf.d/vhost_*.conf plesk: plesk bin commands domain_map: cpanel: /etc/userdatadomains interworx: vhost configs + SuexecUserGroup plesk: plesk bin subscription --info # CRITICAL DATABASE PREFIX PATTERN (MOST IMPORTANT!) database_prefixes: cpanel: username_dbname interworx: username_dbname # SAME AS CPANEL! (verified from official docs) plesk: appname_RANDOM # e.g., wp_i75pa (VERIFIED: real server 2025-11-20) # Module Classification System classes: A: Panel-agnostic (7 modules) - No changes needed B: System detection only (6 modules) - Use SYS_LOG_DIR C: User/domain management (6 modules) - Complex refactoring D: Panel-specific features (2 modules) - Conditional execution Acronis: Backup suite (13 modules) - No changes needed # Class C Refactoring Progress (6/6 complete - 100%!) class_c_complete: - website-error-analyzer.sh (commit d657c8a) - Multi-panel PHP + Apache logs - 500-error-tracker.sh (commit 93d4cf9) - Multi-panel log discovery - wordpress-cron-manager.sh (commit 90ee755) - MOST COMPLEX - Full WordPress discovery - wordpress-menu.sh (already compliant) - Menu only, no hardcoded paths - malware-scanner.sh (previous commit) - Multi-panel docroot + log discovery - optimize-ct-limit.sh (previous commit) - Removed hardcoded fallback # MANDATORY ABSTRACTION LIBRARIES required_libraries: system-detect.sh: exports: SYS_CONTROL_PANEL, SYS_LOG_DIR, SYS_CONTROL_PANEL_VERSION values: cpanel|interworx|plesk|standalone usage: source at top of script, use in case statements user-manager.sh: get_user_info: Returns USER_EXISTS, PRIMARY_DOMAIN, ALL_DOMAINS, HOME_DIR get_user_domains: Returns newline-separated domain list get_user_databases: Returns database list for user usage: NEVER grep /var/cpanel/users directly # STANDARD CODE PATTERNS (COPY THESE) patterns: log_discovery: | case "$SYS_CONTROL_PANEL" in cpanel) find "$SYS_LOG_DIR" -type f -name "*.com" 2>/dev/null ;; interworx) find /home/*/var/*/logs -type f -name "access_log" 2>/dev/null ;; plesk) find /var/www/vhosts/system/*/logs -type f -name "access_log" 2>/dev/null ;; *) [ -f "/var/log/httpd/access_log" ] && echo "/var/log/httpd/access_log" ;; esac domain_to_user: | case "$SYS_CONTROL_PANEL" in cpanel) user=$(grep "^${domain}:" /etc/userdatadomains | cut -d: -f2 | awk -F'==' '{print $1}') ;; interworx) user=$(grep -l "ServerName ${domain}" /etc/httpd/conf.d/vhost_*.conf | \ xargs grep "SuexecUserGroup" | awk '{print $2}') ;; plesk) user=$(plesk bin subscription --info "$domain" | grep "Owner" | awk '{print $2}') ;; esac api_calls: | if [ "$SYS_CONTROL_PANEL" = "cpanel" ]; then whmapi1 some_command else print_warning "Feature requires cPanel" return 1 fi # COMMON MISTAKES TO AVOID mistakes: hardcoded_paths: NEVER use /var/log/apache2/domlogs or /home/user/public_html cpanel_only_apis: NEVER use whmapi1/uapi without panel check missing_source: Class B/C modules MUST source system-detect.sh AND user-manager.sh fallback_paths: NO fallbacks - fail explicitly with clear message userdata_grep: NEVER grep /var/cpanel/userdata or /var/cpanel/users # Refactoring Commits commits: - Phase 1: user-manager.sh InterWorx support (819865b) - Phase 2: bot-analyzer.sh + firewall detection (b86aa14) - Phase 3: Security modules (0988224) - Class B: Log directory refactoring (b770487) - Class D: Panel version detection (8a2d9f5) - website-error-analyzer.sh: Multi-panel (d657c8a) - 500-error-tracker.sh: Multi-panel (93d4cf9) - wordpress-cron-manager.sh: Multi-panel (90ee755) - MOST COMPLEX - Documentation consolidation (fbce072) - Completion: 100% (9af657e) [TESTING_REQUIREMENTS] # What needs verification on real InterWorx/Plesk servers # Created: 2025-11-19 critical_interworx_verification: filesystem: - VERIFIED Home: /home/USERNAME/ - VERIFIED Docroot: /home/USERNAME/DOMAIN/html/ - VERIFIED Access logs: /home/USERNAME/var/DOMAIN/logs/transfer.log (HTTP) - VERIFIED Access logs SSL: /home/USERNAME/var/DOMAIN/logs/transfer-ssl.log (HTTPS) - VERIFIED Error logs: /home/USERNAME/var/DOMAIN/logs/error.log (HTTP) - VERIFIED Error logs SSL: /home/USERNAME/var/DOMAIN/logs/error-ssl.log (HTTPS) - Source: https://appendix.interworx.com/current/nodeworx/general/other/log-file-locations.html domain_to_user: - Method: grep vhost_*.conf for ServerName + SuexecUserGroup - Config: /etc/httpd/conf.d/vhost_DOMAIN.conf - Command: grep -l "ServerName domain" /etc/httpd/conf.d/vhost_*.conf | xargs grep SuexecUserGroup user_to_domains: - Option 1: Parse vhost configs for user's SuexecUserGroup - Option 2: Filesystem ls /home/USERNAME/*/html - Option 3: Check for InterWorx CLI tools database_prefix: - VERIFIED: Uses username_ prefix (same as cPanel!) - Source: https://appendix.interworx.com/current/siteworx/mysql/database-guide.html - Example: username "testuser" → testuser_wp123, testuser_db456 - InterWorx automatically prefixes with SiteWorx account unix username cron_management: - Assumed: Standard Linux crontab -u username - Verify: May have custom InterWorx cron system php_config: - How is PHP version selected per domain? - Where are PHP-FPM pools? - Check vhost configs or separate files critical_plesk_verification: filesystem: - Domain root: /var/www/vhosts/DOMAIN/ - Docroot: /var/www/vhosts/DOMAIN/httpdocs/ - Access logs: /var/www/vhosts/system/DOMAIN/logs/access_log - SSL logs: /var/www/vhosts/system/DOMAIN/logs/access_ssl_log - Error logs: /var/www/vhosts/system/DOMAIN/logs/error_log domain_to_user: - Method: plesk bin subscription --info DOMAIN - Look for: Owner: username OR Login: username - VERIFY exact output format! user_to_domains: - Test: plesk bin subscription --list -owner username - OR: plesk bin client --info username - Find correct command syntax database_prefix: - CRITICAL: Assumed NO PREFIX - Databases: bare names (wp_db, myapp_db) - MUST VERIFY on real server! cron_management: - Check: Standard crontab vs plesk bin cron - System user: What user runs PHP processes? - CRITICAL for wordpress-cron-manager.sh! php_config: - PHP versions: /opt/plesk/php/ - How is version selected per domain? - PHP-FPM pools location? testing_priority_order: simple_first: 1: tail-apache-access.sh - Just find logs 2: web-traffic-monitor.sh - Monitor logs 3: website-error-analyzer.sh - Logs + domain→user 4: 500-error-tracker.sh - Logs + lookups 5: wordpress-cron-manager.sh - EVERYTHING (most complex test) commands_to_run_on_test_servers: interworx: - ls -la /home/ - ls -la /home/testuser/ - ls -la /etc/httpd/conf.d/vhost_*.conf - grep SuexecUserGroup /etc/httpd/conf.d/vhost_testdomain.conf - mysql -e "SHOW DATABASES;" | grep testdoma - crontab -u testuser -l - ls -la /usr/local/interworx/bin/ - find /home -name wp-config.php | head -5 plesk: - ls -la /var/www/vhosts/ - ls -la /var/www/vhosts/testdomain.com/ - plesk bin subscription --info testdomain.com - plesk bin subscription --list - plesk bin --help - mysql -e "SHOW DATABASES;" - ps aux | grep php-fpm | head -5 - ls -la /opt/plesk/php/ - find /var/www/vhosts -name wp-config.php | head -5 unknowns_blocking_full_support: interworx: - ✅ RESOLVED: Database prefix is username_ (verified from docs) - ✅ RESOLVED: File paths all verified from official docs - ✅ RESOLVED: Log file is 'transfer.log' NOT 'access_log' (TESTED: real server 2025-11-20) - ✅ RESOLVED: Logs are symlinks to dated files with auto-rotation/compression - ✅ RESOLVED: Domain→User lookup via vhost configs WORKS - ✅ RESOLVED: User→Domains lookup WORKS (both methods) - ✅ RESOLVED: Standard crontab works (crontab -u USER) - ✅ TESTED: InterWorx 6.14.5 - ALL assumptions verified - PHP version in vhost config (# iw-php-key: /opt/remi/php73) - works, non-critical plesk: - ✅ RESOLVED: Database has prefix pattern appname_RANDOM (e.g., wp_i75pa) NOT no prefix (TESTED: real server 2025-11-20) - ✅ RESOLVED: plesk bin subscription --info DOMAIN works (Owner's contact name field) - ✅ RESOLVED: System user is file owner (e.g., admin_ftp) NOT www-data (TESTED: obsidian.pleskalations.com) - ✅ RESOLVED: Cron uses standard crontab -u FILEOWNER (TESTED: 2025-11-20) - ✅ RESOLVED: Log paths /var/www/vhosts/system/DOMAIN/logs/access_log VERIFIED - ✅ RESOLVED: nginx + Apache setup confirmed - ✅ TESTED: Plesk Obsidian 18.0.61.5 - ALL assumptions verified ################################################################################ # TESTING & VALIDATION PHASE ################################################################################ testing_phase_status: phase: "Testing & Validation" date_started: "2025-11-20" refactoring_complete: true modules_refactored: "38/38 (100%)" validation_scripts: location: "/root/server-toolkit/testing/" interworx_validator: script: "validate-interworx.sh" tests: 13 validates: - Control panel detection - File system structure (/home/USER/DOMAIN/html/) - Virtual host configuration - Domain→User lookup (vhost configs) - User→Domains lookup (2 methods) - Database prefix (username_) - Cron system (standard crontab) - PHP configuration - WordPress detection - InterWorx CLI tools - WordPress file permissions & cron user testing - Directory structure visualization - Comprehensive system documentation output: "/tmp/interworx-validation-results.txt" status: "TESTED on real server - all assumptions verified" plesk_validator: script: "validate-plesk.sh" tests: 15 validates: - Control panel detection - File system structure (/var/www/vhosts/DOMAIN/httpdocs/) - Log file locations (/var/www/vhosts/system/DOMAIN/logs/) - plesk bin commands - Domain→User lookup (plesk bin subscription --info) - User→Domains lookup (plesk bin subscription --list) - Database prefix pattern - System user for web processes (CRITICAL!) - Cron system with actual write/restore testing (CRITICAL!) - PHP configuration - WordPress detection - Apache/nginx configuration - WordPress file permissions & wp-config.php access - Directory structure visualization - Comprehensive system documentation output: "/tmp/plesk-validation-results.txt" status: "TESTED on real server - all assumptions verified" documentation: file: "testing/README.md" contains: - Quick start guide - What gets validated - How to interpret results - Testing priority (4 phases) - Known issues and limitations - Next steps testing_priority: phase_1: "Run validation scripts on real InterWorx/Plesk servers" phase_2: "Test simple Class B modules (log discovery)" phase_3: "Test complex Class C modules (user/domain management)" phase_4: "Production readiness and performance testing" next_action_required: 1. "Get access to InterWorx test server" 2. "Get access to Plesk test server" 3. "Run validate-interworx.sh and collect results" 4. "Run validate-plesk.sh and collect results" 5. "Fix any issues found during validation" 6. "Test real modules on validated servers" ################################################################################ # UPDATES SINCE 2025-11-20 ################################################################################ [UPDATE_2025_12_02_PHP_OPTIMIZER] # Major feature addition: PHP Configuration Optimizer # 7 phases of development completed over 2 days new_components: lib/php-detector.sh: | - 428 lines, 17 exported functions - Detects PHP versions, binaries, and config files per domain - Supports cPanel (ea-php, MultiPHP), InterWorx, Plesk, standalone - Finds php.ini at 4 priority levels (.user.ini, home, pool, system) - Locates PHP-FPM pool configs for all control panels - Functions: detect_php_version_for_domain, find_php_ini, find_fpm_pool_config, etc. lib/php-analyzer.sh: | - 940 lines, 14 exported functions - Analyzes PHP performance metrics and generates recommendations - OPcache hit rate calculation with division-by-zero protection - 7-day historical error log analysis for max_children issues - Memory usage analysis per PHP-FPM process - Process manager statistics (pm.max_children, start/min/max spare) - Functions: get_opcache_stats, check_max_children_errors, recommend_max_children, etc. lib/php-config-manager.sh: | - 509 lines, 14 exported functions - Backup/restore/modify PHP configurations safely - Timestamped backups with metadata in /root/server-toolkit/backups/php/ - Graceful PHP-FPM reload for zero downtime - sed-based configuration modification - Functions: backup_user_php_configs, restore_from_backup, modify_fpm_pool_setting, reload_php_fpm, etc. modules/performance/php-optimizer.sh: | - 1,083 lines, interactive menu system - 9 menu options for PHP analysis and optimization - Option 4: Full apply workflow with auto-backup and rollback - User confirmation required for ALL changes - Auto-backup before modifications - Graceful PHP-FPM reload (not restart) - Verification and rollback instructions menu_integration: location: "Performance & Diagnostics → Option 9" path: "Main Menu (4) → Performance & Diagnostics (9) → PHP Configuration Optimizer" php_optimizer_options: 1: "Analyze All Domains - Server-wide PHP analysis" 2: "Analyze Single Domain - Per-domain analysis" 3: "Show OPcache Statistics - OPcache performance metrics" 4: "Optimize Domain - Main action menu with apply workflow" 5: "View PHP Error Logs - Error log viewer with filtering" 6: "PHP Version Summary - Version distribution report" 7: "Find Configuration Files - Config file discovery" b: "Backup Configurations - Manual backup creation" r: "Restore from Backup - Rollback capability" q: "Quit" option_4_workflow: step_1: "Select domain from list" step_2: "Display current configuration" step_3: "Show recommendations with explanations" step_4: "User confirms: Apply these recommendations? (y/n)" step_5: "If yes: Create timestamped auto-backup" step_6: "Apply changes to PHP-FPM pool config" step_7: "User confirms: Restart PHP-FPM now? (y/n)" step_8: "If yes: Gracefully reload PHP-FPM (zero downtime)" step_9: "Verify PHP-FPM service is running" step_10: "Display backup location for rollback" metrics_tracked: pm_settings: - "pm.max_children - FPM process limit" - "pm.start_servers - Initial processes" - "pm.min_spare_servers - Minimum idle" - "pm.max_spare_servers - Maximum idle" - "pm.max_requests - Process recycling" memory_settings: - "memory_limit - PHP script memory cap" - "upload_max_filesize - Upload size limit" - "post_max_size - POST data limit" timeout_settings: - "max_execution_time - Script timeout" - "max_input_time - Input parsing timeout" opcache_settings: - "opcache.memory_consumption - OPcache memory" - "opcache.interned_strings_buffer - String buffer" - "opcache.max_accelerated_files - Cached file limit" - "opcache.enable - OPcache on/off" - "opcache.revalidate_freq - Cache validation" performance_metrics: - "OPcache hit rate - hits / (hits + misses)" - "max_children errors - 7-day frequency" - "Active PHP-FPM processes - Current load" - "Memory per process - Average consumption" safety_features: - "User confirmation required for ALL changes" - "Auto-backup BEFORE any modifications" - "Graceful reload (not restart) for zero downtime" - "Verification that service is running" - "Clear rollback instructions with backup location" - "No automatic changes without explicit approval" git_commits: - "Phase 1: Create lib/php-detector.sh (detection functions)" - "Phase 2: Create lib/php-analyzer.sh (analysis engine)" - "Phase 3: Create modules/performance/php-optimizer.sh (main script)" - "Phase 4: Implement backup/restore system with PHP-FPM restart" - "Phase 5 & 6: Implement apply/action menu with auto-backup" - "Phase 7: Integrate PHP Configuration Optimizer into main menu" file_statistics: total_lines: 2960 total_functions: 45 files_created: 4 control_panels_supported: 4 testing_status: syntax_validation: "PASS (all files pass bash -n)" logic_validation: "PASS (division-by-zero protection, error handling)" path_resolution: "PASS (verified)" menu_integration: "PASS (tested)" live_server_testing: "PENDING" standards_violations: bash_strict_mode: "MISSING - No 'set -eo pipefail' in any PHP optimizer files" messaging_functions: "VIOLATION - Using cecho/echo -e (198 instances) instead of print_success/print_error" cancel_buttons: "MISSING - Main menu has 'q) Quit' but should use '0) Cancel' pattern" press_enter: "UNKNOWN - Need to verify press_enter() called at script exit" fix_required: "Yes - refactor to use common-functions.sh messaging and add cancel buttons" future_enhancements: - "MySQL Config Optimizer (similar system for MySQL/MariaDB)" - "Redis/Memcached Setup (object caching setup scripts)" - "Apache/Nginx Optimizer (web server tuning - revisit later)" not_planned: - "CDN integration (user declined)" - "SSL/TLS optimizer (user declined)" [UPDATE_2025_12_03_DOCUMENTATION] # Documentation cleanup and standardization changes: - "Removed AI attribution from git commits (per user instructions)" - "Updated README.md with PHP optimizer feature" - "Created docs/DEVELOPMENT_LOG.md (MISTAKE - should use REFDB_FORMAT.txt)" - "Deleted random docs files, consolidated into REFDB_FORMAT.txt" - "Established: REFDB_FORMAT.txt is THE developer documentation file" documentation_policy: primary_file: "REFDB_FORMAT.txt (this file)" user_docs: "README.md (for end users)" no_random_files: "Do not create random .md files in docs/" update_frequency: "After EVERY significant change" git_commit_policy: no_ai_markers: "Never add AI attribution to commits" no_robot_emoji: "Never use 🤖 in commits" no_coauthored: "Never add Co-Authored-By: Claude" clear_messages: "Use clear, descriptive commit messages" technical_details: "Include technical details and impact" [UPDATE_2025_12_03_SCRIPT_DIR_BUG_FIX] # Critical bug fix for PHP optimizer runtime failure problem_identified: symptom: "ERROR: php-config-manager.sh not found (file exists at correct path)" error_trace: "Trying to source /root/server-toolkit/lib/lib/php-analyzer.sh (double /lib/lib/)" root_cause: "SCRIPT_DIR variable collision - multiple sourced libraries redefining SCRIPT_DIR" libraries_setting_script_dir: - "lib/php-detector.sh (line 14, conditional)" - "lib/php-analyzer.sh (line 7)" - "lib/user-manager.sh (line 10)" - "lib/system-detect.sh (line 11)" - "lib/mysql-analyzer.sh (line 10)" - "lib/reference-db.sh (line 11)" sourcing_chain: php-optimizer.sh: "sources php-detector.sh + php-analyzer.sh + system-detect.sh + user-manager.sh" php-detector.sh: "sources system-detect.sh + user-manager.sh (if SYS_CONTROL_PANEL undefined)" php-analyzer.sh: "sources php-detector.sh + system-detect.sh" issue: "Each sourced library overwrites parent's SCRIPT_DIR → /lib/lib/ double paths" solution_implemented: php-optimizer.sh: "Renamed SCRIPT_DIR → PHP_TOOLKIT_DIR (unique variable name)" user-manager.sh: "Renamed SCRIPT_DIR → _LIB_SRCDIR (avoid collision)" php-optimizer.sh: "Fixed detect_system() → initialize_system_detection()" debugging: "Removed 2>/dev/null error suppression to see actual errors" result: status: "FIXED - Script loads all libraries successfully" menu_display: "Working - Shows all 9 options correctly" system_detection: "Working - Detects cPanel, AlmaLinux, Apache, MariaDB, PHP versions" ready_for: "Live testing on production system" architectural_note: global_issue: "SCRIPT_DIR used by multiple libraries creates collision risk" current_fix: "Each module uses unique variable (PHP_TOOLKIT_DIR, etc.)" better_solution: "Libraries should NEVER set SCRIPT_DIR, only modules" status: "Documented for future refactoring" files_modified: - "lib/user-manager.sh (3 lines changed)" - "modules/performance/php-optimizer.sh (10 lines changed)" commit: "0cfbba2" [UPDATE_2025_12_03_DOMAIN_DETECTION_BUG] # CRITICAL bug fix - PHP optimizer showing 0 domains comprehensive_analysis_findings: agent_used: "general-purpose subagent" files_analyzed: "php-detector.sh, php-analyzer.sh, php-optimizer.sh, user-manager.sh" bugs_found: 8 severity_breakdown: "1 CRITICAL, 2 HIGH, 3 MEDIUM, 2 LOW" critical_bug_fixed: file: "lib/user-manager.sh" function: "get_cpanel_user_domains()" lines: "254, 258" problem: | grep -F ": ${username}" /etc/trueuserdomains | grep -F "$username\$" - grep -F means 'fixed string match' (NO REGEX) - Pattern "$username\$" was looking for literal backslash-dollar character - Since no lines contain literal "\$", function returned NOTHING fix: | grep -F ": ${username}" /etc/trueuserdomains | grep "${username}$" - Removed -F from second grep (enable regex mode) - Now $ correctly matches end-of-line impact: before_fix: "0 domains analyzed, 0MB memory shown, ALL features broken" after_fix: "Domains detected correctly, script functional" commit: "f389d82" remaining_high_priority_bugs: bug_1: severity: "HIGH" file: "lib/php-analyzer.sh" lines: "138, 391, 394, 395, 425, 479, 621" issue: "Uses bc command for floating point math - not installed on all systems" fix: "Replace with bash integer arithmetic: [ \"\${hit_rate%%.*}\" -lt 90 ]" bug_2: severity: "HIGH" file: "lib/php-detector.sh + lib/php-analyzer.sh" function: "get_fpm_memory_usage() + calculate_memory_per_process()" lines: "php-detector.sh:273, php-analyzer.sh:202-211" issue: "get_fpm_memory_usage returns single value, but caller expects 'avg_kb|total_mb' format" fix: "Rewrite get_fpm_memory_usage to calculate and return both values" medium_priority_bugs: bug_3: file: "php-analyzer.sh" line: 536 issue: "detect_php_version_for_domain called with 1 param, needs 2 (domain, username)" bug_4: file: "php-optimizer.sh" line: 113 issue: "Same as bug_3 - missing username parameter" bug_5: file: "php-optimizer.sh" lines: "407, 472" issue: "Missing empty checks before numeric comparisons" low_priority_bugs: bug_6: file: "php-optimizer.sh" lines: "1050-1055" issue: "Dead code - backup_array populated in loop then overwritten by mapfile" testing_status: before_fixes: "Script loaded but showed 0 domains, 0 memory usage" after_critical_fix: "Domains now detected, ready for functional testing" next_step: "Fix remaining bugs then test all 9 menu options" [UPDATE_2025_12_03_ADDITIONAL_FIXES] # Additional critical fixes after comprehensive analysis bugs_fixed_after_testing: bug_7: severity: "CRITICAL" commit: "59eb5d5" file: "modules/performance/php-optimizer.sh" lines: "8-13" issue: "Missing common-functions.sh dependency" symptom: "print_info: command not found, command_exists: command not found" fix: "Added common-functions.sh as first library to source, reordered library loading" bug_8: severity: "CRITICAL" commit: "6327ed7" file: "lib/php-detector.sh" function: "find_fpm_pool_config()" lines: "204-245" issue: "Only searched for username.conf, but cPanel uses domain.conf" symptom: "No PHP-FPM pools found" example: "Searched for pickledperil.conf, actual file is pickledperil.com.conf" fix: "Modified to try domain-based naming first, fallback to username-based" bug_9: severity: "MEDIUM" commit: "84081a9" file: "lib/php-analyzer.sh" lines: "435, 447, 457" issue: "Integer expression errors when variables are empty" symptom: "[: : integer expression expected" fix: "Added empty checks before numeric comparisons: [ -n \"$var\" ] && [ \"$var\" -lt value ]" fixes_summary: total_commits: "7 commits" critical_bugs_fixed: "5" medium_bugs_fixed: "1" commits: - "0cfbba2: Fixed SCRIPT_DIR variable collision" - "d3428b0: Documented SCRIPT_DIR bug fix" - "f389d82: Fixed domain detection regex bug (grep -F with $)" - "fc8ccc3: Documented comprehensive bug analysis" - "59eb5d5: Fixed missing common-functions.sh" - "6327ed7: Fixed PHP-FPM pool detection (domain vs username)" - "84081a9: Fixed integer expression errors" current_status: script_loads: "✓ Yes" domains_detected: "✓ Yes (pickledperil.com found)" pools_detected: "✓ Yes (/opt/cpanel/ea-php81/root/etc/php-fpm.d/pickledperil.com.conf)" analysis_completes: "✓ Yes (1 domain analyzed, 1 issue found: OPcache disabled)" errors: "None - all integer expression errors fixed" ready_for_production: "Yes - core functionality working" remaining_non_critical_bugs: - "bc dependency (7 locations) - would fail if bc not installed" - "get_fpm_memory_usage return format mismatch - returns single value, caller expects two" - "detect_php_version_for_domain missing username parameter (2 locations)" - "Dead code in backup_array population" [UPDATE_2025_12_03_COMPREHENSIVE_AUDIT_ADDITIONAL_FIXES] # After comprehensive audit, 7 more critical bugs were discovered and fixed bugs_fixed_during_audit: bug_10: severity: "CRITICAL" commit: "19c1ea3" title: "Fix SYS_* variable reset bug in system-detect.sh" file: "lib/system-detect.sh" lines: "16-26" issue: "THE ROOT CAUSE - SYS_* variables reset to empty every time library is sourced" impact: "Cascading failures: domain detection, user lookup, all multi-file operations broke" symptom: "get_user_domains returned empty even when domains exist, SYS_CONTROL_PANEL disappeared" fix: "Wrapped variable initialization in 'if [ -z \"$SYS_DETECTION_COMPLETE\" ]' guard" note: "This single bug caused 50% of the other bugs we encountered" bug_11: severity: "HIGH" commit: "801ceb1" title: "Remove non-existent function from exports in user-manager.sh" file: "lib/user-manager.sh" issue: "Exporting display_user_overview function that doesn't exist" symptom: "export: display_user_overview: not a function" fix: "Removed from export list" bug_12: severity: "CRITICAL" commit: "c776707" title: "Add missing function exports to user-manager.sh" file: "lib/user-manager.sh" lines: "725-737" issue: "13 functions defined but never exported" impact: "Functions unavailable in nested calls, subshells, and parallel execution" fix: "Added export -f for all 13 functions" functions_exported: - "list_all_users" - "get_user_domains" - "get_cpanel_user_domains" - "get_plesk_user_domains" - "get_interworx_user_domains" - "get_standalone_user_domains" - "get_user_info" - "get_user_databases" - "get_user_processes" - "get_top_processes_for_user" - "display_user_summary" - "get_primary_domain" - "count_user_files" bug_13: severity: "HIGH" commit: "69575d6" title: "Fix memory capacity calculation to iterate through domains not just users" file: "lib/php-analyzer.sh" function: "calculate_server_memory_capacity()" lines: "745-800" issue: "Only iterated users, didn't get domains for each user, couldn't find pools" symptom: "0MB memory usage despite active PHP-FPM pools" fix: "Added nested loop to get domains per user, pass both to find_fpm_pool_config" impact: "Memory capacity calculations now accurate" bug_14: severity: "MEDIUM" commit: "b7f20de" title: "Fix arithmetic syntax error in analyze_all_domains" file: "modules/performance/php-optimizer.sh" function: "analyze_all_domains()" lines: "215-224" issue: "grep -c with || echo '0' created double output '0\\n0' in variables" symptom: "syntax error in expression (error token is '0')" fix: "Changed || echo '0' to || true, added ${var:-0} default assignment" bug_15: severity: "MEDIUM" commit: "0f7e5ec" title: "Fix memory capacity output parsing - was showing domain names instead of numbers" file: "modules/performance/php-optimizer.sh" lines: "873-886" issue: "Used tail -1 to get 'last line' but got details line (domain|user) not summary" symptom: "Total Server RAM: pickledperilMB" fix: "Changed tail -1 to head -1 for summary, tail -n +2 for details" root_cause: "calculate_server_memory_capacity returns multi-line output" bug_16: severity: "LOW" commit: "fbc3edd" title: "Enhance analyze_all_domains output to show passed checks" file: "modules/performance/php-optimizer.sh" lines: "244-253" type: "ENHANCEMENT not bug" change: "Added visual confirmation when checks pass (max_children OK, memory OK, timeouts OK)" impact: "Usability improvement - user knows script is working even when no issues found" comprehensive_audit_summary: total_additional_bugs_found: "6 (bugs 10-15, plus 1 enhancement)" commits_documented: "7 (6 bugs + 1 enhancement)" severity_breakdown: critical: "2 (SYS_* reset, missing exports)" high: "2 (non-existent export, memory capacity iteration)" medium: "2 (arithmetic syntax, output parsing)" low: "1 (enhancement)" most_critical_discovery: "bug_10 (SYS_* reset) - THE ROOT CAUSE of cascading failures" all_14_php_optimizer_commits: - "e91e6f0: Integrate PHP Configuration Optimizer into main menu" - "0cfbba2: Fix SCRIPT_DIR variable collision" - "f389d82: CRITICAL: Fix domain detection bug" - "59eb5d5: Fix missing common-functions.sh" - "6327ed7: CRITICAL: Fix PHP-FPM pool detection" - "84081a9: Fix integer expression errors" - "fbc3edd: Enhance analyze_all_domains output" - "69575d6: Fix memory capacity calculation" - "b7f20de: Fix arithmetic syntax error" - "c776707: CRITICAL: Add missing function exports" - "19c1ea3: CRITICAL: Fix SYS_* variable reset (ROOT CAUSE)" - "801ceb1: Remove non-existent function export" - "0f7e5ec: Fix memory capacity output parsing" - "e7b682f: Update REFDB_FORMAT.txt documentation" total_bugs_fixed: "15 total (9 tracked during development + 6 found in audit)" php_optimizer_status: "PRODUCTION READY - all critical bugs resolved" [UPDATE_2025_12_03_QA_CHECKING_TOOL] # Created comprehensive project-wide quality assurance checking script tool_created: file: "/tmp/toolkit-qa-check.sh" purpose: "Automated bug pattern detection across entire toolkit" runtime: "~10 seconds for 57 shell scripts" expandable: "Designed to add new checks as bug patterns are discovered" motivation: problem: "Multiple similar bugs discovered during PHP optimizer development" examples: - "grep -F with regex anchors ($) appeared in 8+ locations" - "SCRIPT_DIR collisions in 4 files" - "SYS_* variable resets broke multi-file sourcing" - "Integer comparisons without empty checks (20+ locations)" - "exit vs return in libraries" solution: "Automated scanner to catch these patterns project-wide" checks_implemented: check_1: name: "grep -F with regex anchors" severity: "CRITICAL" pattern: "grep -F ... \"$var\\$\" or grep -F ... \"^pattern\"" issue: "-F flag disables regex, so $ and ^ match literally" found: "8 instances across lib/user-manager.sh, lib/reference-db.sh, modules/website/500-error-tracker.sh" check_2: name: "SCRIPT_DIR variable collisions" severity: "HIGH" pattern: "Multiple files defining SCRIPT_DIR=" issue: "Libraries sourcing other libraries redefine the same variable" found: "4 files: lib/mysql-analyzer.sh, lib/reference-db.sh, lib/system-detect.sh, tools/erase-toolkit-traces.sh" check_3: name: "SYS_* variable resets without protection" severity: "CRITICAL" pattern: "export SYS_.*=\"\" in lib/*.sh without SYS_DETECTION_COMPLETE guard" issue: "Re-sourcing library wipes all system detection variables" found: "0 instances (already fixed in system-detect.sh)" check_4: name: "Missing function exports in libraries" severity: "HIGH" pattern: "lib/*.sh with functions but no 'export -f' statements" issue: "Functions unavailable in nested calls or subshells" found: "Multiple libraries missing exports" check_5: name: "Integer comparisons without empty checks" severity: "HIGH" pattern: "[ $var -lt 123 ] without preceding [ -n \"$var\" ]" issue: "Empty variables cause 'integer expression expected' errors" found: "20+ instances across lib/common-functions.sh, lib/ip-reputation.sh, launcher.sh, modules/*" check_6: name: "Missing common-functions.sh sourcing" severity: "HIGH" pattern: "Uses cecho/print_info/etc without sourcing common-functions.sh" issue: "Command not found errors at runtime" found: "Already checked, no new instances" check_7: name: "exit in sourced libraries" severity: "HIGH" pattern: "exit statements in lib/*.sh files" issue: "Libraries should use 'return' not 'exit' to avoid terminating parent script" found: "4 instances (some false positives from comments)" check_8: name: "Bash syntax validation" severity: "CRITICAL" pattern: "bash -n script.sh fails" issue: "Syntax errors prevent script execution" found: "0 syntax errors detected" qa_scan_results: files_scanned: "57 shell scripts" total_issues_found: "24" breakdown: critical: "8 (grep -F with regex anchors)" high: "24 (integer comparisons + exit in libraries)" medium: "0" low: "0" most_common_issue: "Integer comparisons without empty checks (20 instances)" highest_severity: "grep -F with regex anchors in domain/user detection code" script_features: - "Fast execution: 8 optimized checks vs original 15 slow checks" - "Color-coded severity levels: CRITICAL (red bold), HIGH (red), MEDIUM (yellow), LOW (blue)" - "Line number references for quick navigation" - "Context snippets showing problematic code" - "Summary report with issue counts by severity" - "Exit code 0 (allows integration into CI/CD pipelines)" usage: command: "bash /tmp/toolkit-qa-check.sh /root/server-toolkit" output: "Colored terminal output + saved to /tmp/qa-report-fast.txt" integration: "Can be run before commits or in pre-commit hooks" future_expandability: design: "Modular check structure - easy to add new patterns" examples_to_add: - "Unquoted variable expansions in rm/mv/cp commands" - "Missing file existence checks before cat/grep operations" - "bc command usage (external dependency)" - "Hardcoded /var/cpanel paths (multi-panel violation)" - "Missing || true on grep commands (exit code issues)" - "Arithmetic syntax errors (command substitution in $(()))" impact: development: "Catch bugs before they reach production" maintenance: "Identify similar bugs across entire codebase" quality: "Enforces best practices discovered through painful debugging" time_savings: "10 second scan vs hours of manual code review" qa_script_bug_found_and_fixed: bug: "Bash subshell counter bug" severity: "HIGH" issue: "Used 'command | while read' which creates subshells - counter increments don't persist" symptom: "Summary showed '✓ No issues found' even after displaying 24 issues" impact: "Made QA tool misleading and untrustworthy" fix: "Changed all pipes to process substitution: while read; do ... done < <(command)" additional_fix: "Used temp file for counters to ensure persistence across function calls" verification: "After fix: Exit code 21 = 8 CRITICAL + 13 HIGH (correct!)" optimizations_for_ai_readability: - "Structured pipe-delimited output: SEVERITY|file|line|issue" - "Grouped display by severity (CRITICAL first, then HIGH, MEDIUM, LOW)" - "file:line format for quick navigation" - "Limited HIGH issues to first 15 (prevents overwhelming output)" - "Clear summary at top with exact counts" - "Exit code = total issues (for CI/CD integration)" - "Saves full report to /tmp/qa-report.txt for detailed review" - "Progress indicators: [1/8], [2/8], etc." final_qa_results: scan_date: "2025-12-03" files_scanned: "57 shell scripts" total_issues: "21" breakdown: critical: "8 (grep -F with regex anchors)" high: "13 (integer comparisons + function exports + exit in libraries)" medium: "0" low: "0" top_issues_by_file: "/root/server-toolkit/lib/user-manager.sh": "5 issues (grep -F regex, integer comparisons)" "/root/server-toolkit/lib/common-functions.sh": "4 issues (integer comparisons, exit usage)" "/root/server-toolkit/lib/ip-reputation.sh": "3 issues (integer comparisons)" [DEVELOPMENT_WORKFLOW] ################################################################################ # Standard workflow for developing and validating changes to server-toolkit ################################################################################ code_validation_options: description: "Three methods for validating shell script changes before committing" option_1_manual_review: method: "Manual code review" when: "Quick changes, single-file edits" pros: - "Fast for small changes" - "Good for understanding code flow" cons: - "Error-prone for large changes" - "Misses systematic issues" - "High cognitive load" option_2_runtime_testing: method: "Execute scripts in test environment" when: "Testing specific functionality" command: "bash -x /root/server-toolkit/modules/php/php-optimizer.sh" pros: - "Validates actual behavior" - "Catches runtime errors" - "Tests real-world scenarios" cons: - "Time-consuming" - "May not hit all code paths" - "Requires test environment setup" option_3_automated_qa_script: method: "Run QA checking tool (RECOMMENDED BEFORE ALL COMMITS)" command: "bash /tmp/toolkit-qa-check.sh /root/server-toolkit" when: "Before every git commit, after any significant changes" runtime: "~10-15 seconds for entire toolkit" pros: - "Catches 11 bug patterns automatically" - "Identifies 5 performance anti-patterns" - "Fast (10s vs hours of debugging)" - "Zero false positives for CRITICAL issues" - "Provides file:line references for quick fixes" cons: - "Can't detect logic bugs or semantic errors" - "Requires pattern database maintenance" coverage: total_checks: "16 checks (11 bug patterns + 5 performance checks)" bug_patterns_checked: - "grep -F with regex anchors (CRITICAL)" - "SCRIPT_DIR collisions (HIGH)" - "SYS_* variable resets (CRITICAL)" - "Missing function exports (HIGH)" - "Integer comparisons without validation (HIGH)" - "Missing common-functions.sh sourcing (HIGH)" - "exit in library files (HIGH)" - "bc command usage (MEDIUM)" - "Hardcoded /var/cpanel paths (MEDIUM)" - "Undefined color variables (LOW)" - "Bash syntax errors (CRITICAL)" performance_patterns_checked: - "cat | grep inefficiency (INFO)" - "Repeated file decompression (INFO)" - "Subshells in loops (INFO)" - "Inefficient string operations (INFO)" - "Repeated file access (INFO)" coverage_rate: "100% of pattern-matchable bugs from REFDB" unchecked_patterns: - "Function signature mismatches (requires type analysis)" - "Missing function parameters (requires call graph)" - "Dead code (requires control flow analysis)" - "Logic bugs (requires semantic understanding)" output_format: structure: "SEVERITY|file|line|issue_description" severity_levels: "CRITICAL > HIGH > MEDIUM > LOW > INFO" exit_code: "Total count of issues (CRITICAL + HIGH + MEDIUM + LOW, excludes INFO)" navigation: "Use file:line format to jump directly to issues" typical_results: clean_codebase: "Exit code 0 (no issues)" after_major_changes: "Exit code 20-40 (multiple issues to fix)" current_baseline: "Exit code 41 (8 CRITICAL + 13 HIGH + 9 MEDIUM + 11 LOW)" recommended_workflow: step_1: "Make code changes using Read/Edit/Write tools" step_2: "Run QA script: bash /tmp/toolkit-qa-check.sh /root/server-toolkit" step_3: "Fix all CRITICAL issues (exit code must drop)" step_4: "Review HIGH issues and fix as many as practical" step_5: "Review MEDIUM/LOW issues for quick wins" step_6: "Review INFO performance suggestions" step_7: "Re-run QA script to verify fixes" step_8: "If exit code is acceptable, proceed to runtime testing (option 2)" step_9: "Create git commit with proper documentation" note: | The QA script saves massive amounts of debugging time by catching issues before they hit production. Running it takes 10 seconds but can save hours of troubleshooting runtime errors, especially for CRITICAL issues like grep -F with regex anchors or SYS_* variable resets. qa_script_maintenance: location: "/tmp/toolkit-qa-check.sh" update_frequency: "When new bug patterns are discovered" documentation_sync: "All checks must be documented in REFDB_FORMAT.txt" adding_new_checks: step_1: "Document bug pattern in REFDB_FORMAT.txt [UPDATE_YYYY_MM_DD] section" step_2: "Add check to toolkit-qa-check.sh with appropriate severity" step_3: "Test check against known-bad code to verify detection" step_4: "Test check against clean code to verify no false positives" step_5: "Update check count in REFDB documentation" integration_with_git: pre_commit_hook_candidate: true command: "bash /tmp/toolkit-qa-check.sh /root/server-toolkit" blocking_criteria: "Exit code > 0 (any CRITICAL/HIGH/MEDIUM/LOW issues)" future_enhancements: - "Add --fix flag for auto-correctable issues" - "JSON output mode for CI/CD integration" - "Progress indicator for long-running checks" - "Cache file lists between runs" - "Whitelist mechanism for known false positives" [UPDATE_2025_12_10_MYSQL_RESTORE_TOOL] # New feature: MySQL/MariaDB File-Based Restore to SQL # Interactive guided workflow for restoring databases from file-based backups new_module: file: "modules/backup/mysql-restore-to-sql.sh" menu_location: "Backup & Recovery → Option 11" purpose: "Convert restored MySQL/MariaDB data files to usable .sql dumps" use_case: scenario: "Restore InnoDB databases from file-based backups (Acronis, Guardian, cloud backups)" problem: "Direct file restoration doesn't work with InnoDB due to tablespace requirements" solution: "Start second MySQL instance with restored files, dump to SQL, import to live" features: - "Multi-control panel support (cPanel, Plesk, InterWorx, standalone)" - "Detects control panel and uses appropriate home directory" - "Detects MySQL version (MySQL 5.7, MySQL 8.0+, MariaDB)" - "Version-specific validation (ib_logfile0/1 vs #innodb_redo)" - "Auto-creates timestamped restore directory ($SYS_USER_HOME_BASE/temp/restoreYYYYMMDD/mysql)" - "SQL dumps saved to restore directory (not current working directory)" - "Lists exact files needed from backup (with emoji visual markers)" - "Validates data structure before proceeding" - "Checks and fixes file ownership (mysql:mysql)" - "Starts second MySQL instance on separate socket" - "InnoDB error log monitoring (checks for corruption/errors on startup)" - "Automatic error log backup (preserves old logs)" - "Supports InnoDB force recovery modes (0-6)" - "Creates verified SQL dumps with completion markers" - "Comprehensive SQL dump validation (7 checks)" - "Database comparison (table count, size verification)" - "SQL syntax validation" - "Provides clear next-step instructions" - "Cancel buttons at every step (0 to cancel)" workflow: step_1: "Detect live MySQL data directory and version" step_2: "Create/select restore directory with guided file list" step_3: "User restores required files from backup system" step_4: "Validate restored data structure" step_5: "Fix file ownership if needed" step_6: "Select target database" step_7: "Configure optional settings (ticket number, force recovery)" step_8: "Start second MySQL instance" step_9: "Create SQL dump from second instance" step_10: "Provide import instructions" required_files_mysql_5_7_mariadb: - "ibdata1 (InnoDB system tablespace)" - "ib_logfile0 (redo log)" - "ib_logfile1 (redo log)" - "mysql/ directory (system database)" - "sys/ directory (optional but recommended)" - "/ directory (target database)" required_files_mysql_8_0_0_to_8_0_29: - "ibdata1 (InnoDB system tablespace)" - "ib_logfile0 (redo log - old format)" - "ib_logfile1 (redo log - old format)" - "#innodb_temp/ directory (optional, temp tablespace)" - "ibtmp1 (optional, global temp tablespace)" - "mysql/ directory (system database)" - "sys/ directory (optional but recommended)" - "/ directory (target database)" - "NOTE: performance_schema NOT needed (in-memory only)" required_files_mysql_8_0_30_plus: critical_change_2022: "MySQL 8.0.30 introduced new redo log architecture" - "ibdata1 (InnoDB system tablespace)" - "#innodb_redo/ directory (NEW FORMAT - contains #ib_redo0, #ib_redo1, #ib_redoN files)" - "#innodb_temp/ directory (optional, contains temp_N.ibt files)" - "ibtmp1 (optional, global temp tablespace)" - "mysql/ directory (system database)" - "sys/ directory (optional but recommended)" - "/ directory (target database)" - "NOTE: performance_schema NOT needed (in-memory only)" - "NOTE: NO MORE ib_logfile0/ib_logfile1 in 8.0.30+" required_files_mysql_9_0_plus: - "Same as MySQL 8.0.30+ (uses #innodb_redo directory)" technical_details: second_instance_command: | mysqld --datadir=$dir --socket=$dir/socket.mysql --pid-file=$dir/mysql.pid --log-error=$dir/mysql.err --skip-grant-tables --skip-networking --user=mysql dump_command: | mysqldump -h localhost -S $dir/socket.mysql --single-transaction database_name > output.sql mysqld_binary_detection: | Tries /usr/libexec/mysqld first (RHEL/CentOS), falls back to mysqld force_recovery_levels: 0: "No force recovery (default)" 1: "Ignore corrupt pages" 2: "Prevent background operations" 3: "Prevent transaction rollbacks" 4: "Prevent insert buffer merge" 5: "Skip log redo" 6: "Skip page checksums" safety_features: - "Second instance runs on separate socket (doesn't affect live MySQL)" - "Critical safety check: Prevents using /var/lib/mysql as restore directory" - "Trap handler: Cleanup on Ctrl+C, interruption, or exit (cleanup_on_exit)" - "Disk space validation: Checks available space before creating dump (2x estimated size)" - "Force recovery warnings: Levels 5-6 require explicit risk acknowledgment" - "All operations validated before execution" - "InnoDB error log parsing (detects corruption during startup)" - "Automatic shutdown if InnoDB errors detected" - "SQL dump integrity validation (7-point check)" - "Database comparison (table count, size ratio verification)" - "Clear error messages with troubleshooting steps" - "Automatic cleanup of second instance on completion" - "Dump verification (checks for 'Dump completed' marker)" - "No eval usage: Uses bash array for mysqld arguments (security)" validation_checks_innodb_startup: error_patterns_detected: - "InnoDB: Corrupted" - "InnoDB: Database page corruption" - "InnoDB: Unable to open" - "InnoDB: Cannot allocate memory" - "InnoDB: Tablespace.*missing" - "InnoDB: Redo log.*corrupt" - "InnoDB:.*redo log.*incompatible" - "InnoDB: Plugin initialization aborted" - "[ERROR].*InnoDB" action_on_error: "Automatically shuts down second instance and suggests force recovery" validation_checks_sql_dump: check_1: "File exists and minimum size (>100 bytes)" check_2: "Dump completion marker present ('Dump completed on')" check_3: "Database name found in dump" check_4: "CREATE TABLE statement count" check_5: "INSERT INTO statement count (data verification)" check_6: "SQL syntax spot check (first 100 lines)" check_7: "Database comparison (table count match, size ratio 1-3x)" action_on_failure: "Prompts user to continue or abort if validation issues found" output: format: "_restored_.sql" example: "myuser_wordpress_restored_20251210_143022.sql" with_ticket: "myuser_wordpress_ticket01234567_20251210_143022.sql" location: "Parent directory of restore location (e.g., /home/temp/restore20251210/)" control_panel_paths: cpanel: home_base: "/home" suggested_restore: "/home/temp/restore20251210/mysql" sql_output: "/home/temp/restore20251210/" plesk: home_base: "/var/www/vhosts" suggested_restore: "/var/www/vhosts/temp/restore20251210/mysql" sql_output: "/var/www/vhosts/temp/restore20251210/" interworx: home_base: "/chroot/home" suggested_restore: "/chroot/home/temp/restore20251210/mysql" sql_output: "/chroot/home/temp/restore20251210/" note: "Uses /chroot/home directly (not /home symlink) as system doesn't display /home properly" standalone: home_base: "/home" suggested_restore: "/home/temp/restore20251210/mysql" sql_output: "/home/temp/restore20251210/" next_steps_provided: 1: "Verify dump: grep 'Dump completed on' output.sql" 2: "Import to live: mysql database_name < output.sql" 3: "Or fresh import: DROP/CREATE then import" error_handling: second_instance_fails: - "Analyzes error log to diagnose failure type" - "Provides intelligent recovery options based on error" - "Shows targeted guidance (not generic troubleshooting)" - "Offers to display error log immediately" intelligent_recovery_system: missing_tablespace_files: diagnosis: "Missing or unopenable .ibd files" option_1: "Restore additional files (grep error log for specific tables)" option_2: "Start fresh (clear directory, restore complete set)" action: "Lists exact grep command to find missing tables" redo_log_incompatibility: diagnosis: "Version mismatch or mixed redo log formats" option_1: "Remove redo logs, restore from same backup date" option_2: "Force recovery level 6 (if redo logs lost)" action: "Shows exact rm command for current MySQL version" innodb_corruption: diagnosis: "Page corruption detected" progressive_recovery: "Suggests recovery levels based on current level" level_0: "Try force recovery 1 (ignore corrupt pages)" level_1: "Try force recovery 4 (prevent insert buffer)" level_4+: "Try force recovery 6 (skip checksums - last resort)" option_final: "Start fresh from older backup snapshot" memory_allocation: diagnosis: "Out of memory" action_1: "Check free memory (free -h)" action_2: "Stop other MySQL instances" action_3: "Re-run script" validation_fails: - "Lists missing required files with version-specific paths" - "Shows expected file paths" - "Explains version-specific requirements" - "Offers retry loop (restore files and try again)" dump_fails: - "Checks if database exists in second instance" - "Verifies dump completion marker" - "Reports file size of partial dump" - "Prompts to continue or abort on validation errors" integration: menu_path: "Main Menu (5) → Backup & Recovery (11) → MySQL File Restore" launcher_changes: - "Added option 11 to show_backup_menu()" - "Added case 11 to handle_backup_menu()" standards_compliance: - "Uses print_* functions from common-functions.sh" - "Cancel buttons at every prompt (0 to cancel)" - "Calls press_enter() before exit" - "Root check at script start" - "No hardcoded paths (version-aware)" - "Proper error handling with exit codes" - "QA check: 0 issues (CRITICAL/HIGH/MEDIUM/LOW)" testing_status: syntax_validation: "PASS (bash -n)" qa_validation: "PASS (0 issues)" standards_compliance: "PASS (all toolkit standards followed)" live_testing: "PENDING (awaiting real-world file-based restore scenario)" documentation_references: based_on: "Guardian and cloud backup restoration procedures" kb_article: "InnoDB Second MySQL Instance Restore Process" research_date: "2025-12-10" verified_sources: - "MySQL 8.0 Reference Manual - InnoDB Backup (dev.mysql.com)" - "MySQL 8.4 Reference Manual - Redo Log (dev.mysql.com)" - "MariaDB Documentation - InnoDB Redo Log" - "MySQL Dynamic InnoDB Redo Log blog (MySQL 8.0.30 changes)" - "MySQL Enterprise Backup 8.0 User Guide" critical_findings: - "MySQL 8.0.30+ uses #innodb_redo directory (not ib_logfile anymore)" - "Redo log files named #ib_redoN (N = sequence number)" - "Temporary tablespace in #innodb_temp directory" - "performance_schema is in-memory only, not needed for restore" - "sys schema should be included (optional but recommended)" known_limitations: - "Requires sufficient disk space for SQL dumps" - "Cannot restore if InnoDB data files are severely corrupted (even with force recovery 6)" - "Does not handle MyISAM-only databases (but will work if MyISAM mixed with InnoDB)" - "Second instance start may fail if port 3306 socket conflicts (but uses custom socket)" future_enhancements: - "Batch restore (multiple databases at once)" - "Progress indicator for large dump operations" - "Automatic backup of existing live database before import" - "Integration with Acronis API for automated file restore" - "Support for remote MySQL servers (not just localhost)" commit_info: date: "2025-12-10" files_added: - "modules/backup/mysql-restore-to-sql.sh (540 lines)" files_modified: - "launcher.sh (added menu option 11)" qa_result: "0 issues detected" ################################################################################ # SECURITY: ET OPEN ATTACK DETECTION SYSTEM ################################################################################ # Added: 2025-12-12 - 2025-12-13 # Major security enhancement using Emerging Threats Open ruleset ################################################################################ [SYSTEM_OVERVIEW] feature: "Real-time and Historical Attack Detection" license: "Emerging Threats Open (BSD License)" source: "https://rules.emergingthreats.net/" copyright: "2003-2025, Emerging Threats" architecture: detection_engine: "Signature-based pattern matching (Suricata-inspired)" pattern_database: "lib/attack-signatures.sh (350+ attack patterns)" analysis_engine: "lib/http-attack-analyzer.sh (scoring + classification)" reputation_system: "lib/ip-reputation.sh (cumulative threat tracking)" monitoring_tools: - "modules/security/live-attack-monitor.sh (real-time detection)" - "tools/analyze-historical-attacks.sh (batch log analysis)" key_capabilities: - "Detect 8 attack categories (SQLi, XSS, RCE, LFI, traversal, webshells, etc.)" - "Threat scoring (0-100 scale, multi-signature accumulation)" - "IP reputation tracking (cumulative scores across multiple attacks)" - "Real-time monitoring with color-coded alerts" - "Historical log analysis with detailed reporting" - "Pattern extraction from 350+ ET Open signatures" - "Support for compressed logs (gzip, bzip2)" - "AbuseIPDB integration for external threat intelligence" [ATTACK_SIGNATURES_LIBRARY] # lib/attack-signatures.sh file_path: "lib/attack-signatures.sh" purpose: "Central attack pattern database extracted from ET Open ruleset" size: "~700 lines, 350+ signature patterns" license: "BSD (Emerging Threats Open)" pattern_categories: ATTACK_SQLI: "SQL Injection (20+ patterns)" - "union_select: UNION-based injection (severity 90)" - "blind_sqli: Time-based blind SQLi (severity 80-85)" - "sqli_functions: SQL function abuse (load_file, concat, etc.)" - "sqli_info: Database metadata access (information_schema)" - "stacked_query: Stacked query injection (severity 90)" ATTACK_XSS: "Cross-Site Scripting (15+ patterns)" - "script_tag: " webshell: "[17:32:10] 45.142.212.61 | Score:100 [CRITICAL] | 🛡️ET:WEBSHELL | Sigs:c99_shell,php_backdoor,eval_base64 | /wp-content/themes/old/shell.php" path_traversal: "[17:33:45] 185.220.101.33 | Score:75 [HIGH] | 🛡️ET:TRAVERSAL | Sigs:dotdot_slash,etc_passwd,null_byte | /download.php?file=../../etc/passwd" impact: visibility: "Users can now see specific ET Open signature names" debugging: "Easier to identify exact attack techniques being used" learning: "Security teams can learn which patterns attackers are using" confidence: "Proof that ET Open detection is working and active" testing_status: syntax_check: "✅ PASSED (both v1 and v2)" display_format: "✅ VERIFIED (simulation shows correct output)" backward_compatibility: "✅ MAINTAINED (no breaking changes)" user_experience: before: "Never saw signature names, thought ET detection wasn't working" after: "Clear visibility into ET Open's 350+ signatures in real-time" frequency: "Will see signature names on every detected attack (daily/hourly)" ################################################################################ # HARDWARE HEALTH CHECK ENHANCEMENTS ################################################################################ # DATE: 2025-12-15 # FILE: modules/performance/hardware-health-check.sh # STATUS: ✅ COMPLETE - Predictive failure detection system implemented ################################################################################ [HARDWARE_HEALTH_CHECK_ENHANCEMENTS] enhancement_date: "2025-12-15" enhancement_type: "Deep analysis and predictive failure detection" files_modified: - "modules/performance/hardware-health-check.sh" objective: goal: "Transform basic hardware checks into comprehensive predictive failure detection system" requirement: "Make hardware problems ASTONISHINGLY OBVIOUS and IMPOSSIBLE to miss" user_request: "Make drive issues obvious or ram issues or whatever hardware related" enhancements: 1_disk_health_predictive_analysis: before: "Basic SMART PASSED/FAILED check with generic warnings" after: "Deep SMART attribute parsing with predictive failure risk assessment" features: - name: "Predictive failure risk levels" levels: IMMINENT: ">50 reallocated sectors or >10 pending sectors - fails within days/weeks" HIGH: ">10 reallocated sectors or >0 uncorrectable - replacement recommended" MODERATE: ">0 reallocated/pending sectors - monitor closely" HEALTHY: "All attributes normal" - name: "Critical SMART attributes parsed" attributes: - "Reallocated sectors (ID 5)" - "Pending sectors (ID 197)" - "Uncorrectable sectors (ID 198)" - "Read error rate (ID 1)" - "Spin retry count (ID 10)" - "Wear leveling (ID 177 - SSDs)" - "Temperature (ID 194)" - name: "I/O error log correlation" source: "/var/log/messages (last 7 days)" patterns: - "disk_name.*error" - "ata.*failed" - "disk_name.*failed" thresholds: - ">50 errors = HIGH risk" - ">10 errors = MODERATE risk" includes_samples: "Shows recent error lines in report" - name: "Temperature monitoring" thresholds: - ">55°C = OVERHEATING warning" - ">50°C = above recommended" - "<50°C = optimal" - name: "Actionable timelines" examples: - "IMMINENT: Replace within 1-2 weeks" - "HIGH: Order spare disk, plan replacement" - "MODERATE: Monitor weekly with smartctl -A" 2_memory_health_ecc_detection: before: "Generic 'memory errors detected' with error count" after: "Comprehensive ECC error analysis with DIMM identification" features: - name: "ECC error type classification" types: single_bit: "Correctable errors - indicates failing DIMM" multi_bit: "UNCORRECTABLE - critical data corruption risk" thresholds: - ">100 single-bit = CRITICAL (BAD DIMM - replace immediately)" - ">20 single-bit = WARNING (faulty DIMM likely)" - "ANY multi-bit = CRITICAL (immediate replacement)" - name: "Faulty DIMM location identification" method: "Parse EDAC messages from /var/log/messages" patterns: - "edac.*dimm" - "edac.*channel" - "edac.*slot" - "ce.*error.*channel" result: "Shows exact slot: 'Channel 2 DIMM 1'" - name: "OOM killer detection" patterns: - "out of memory" - "oom.*kill" - "invoked oom-killer" thresholds: - ">10 events = WARNING (insufficient RAM for workload)" - ">0 events = INFO (consider adding RAM)" includes: "Recent OOM event samples with process names" - name: "Swap thrashing detection" metrics: - "Swap usage percentage" - "Total vs used swap" thresholds: - ">80% = WARNING (memory pressure)" - ">50% = INFO (moderate pressure)" - name: "Memory pressure analysis" data_points: - "Total, used, available memory" - "ECC support and type (Single-bit, Multi-bit, None)" - "Populated vs total DIMM slots" 3_cpu_health_thermal_throttling: before: "Basic temperature reading if sensors installed" after: "Advanced thermal monitoring with throttling detection" features: - name: "Temperature threshold analysis" thresholds: - ">90°C = CRITICAL (extreme overheating, damage risk - SHUT DOWN!)" - ">80°C = WARNING (overheating - clean cooling system)" - ">70°C = WARNING (high - normal is <70°C)" optimal: "<70°C for normal operation" - name: "Thermal throttling detection" source: "/var/log/messages" patterns: - "thermal.*throttl" - "cpu.*overheat" - "temperature.*critical" - "thermal.*shutdown" thresholds: - ">10 events = WARNING (cooling problem)" - ">0 events = INFO (throttling occurred)" includes: "Recent throttle event samples" - name: "Frequency throttling detection" method: "Compare current vs max CPU frequency" calculation: "throttle_pct = (1 - current_freq/max_freq) * 100" threshold: ">20% reduction = WARNING (thermal or power limiting)" files: - "/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq" - "/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq" - name: "Per-core temperature monitoring" source: "lm_sensors output" pattern: "Core [0-9]+:" shows: "All individual core temperatures" detects: "Uneven temperature distribution" identifies: "Hottest core" - name: "Load analysis" calculation: "load_pct = (1min_load / cpu_threads) * 100" threshold: ">200% = overloaded server warning" - name: "MCE detection" full_name: "Machine Check Exception" patterns: - "mce" - "machine check" - "cpu.*error" severity: "CRITICAL - hardware failure" action: "Contact hosting provider/vendor immediately" - name: "Multiple temperature sources" primary: "lm_sensors (if installed)" fallback: "/sys/class/thermal/thermal_zone*/temp" conversion: "millidegrees to Celsius (divide by 1000)" 4_visual_display_enhancements: before: "Plain text report, no visual hierarchy" after: "Attention-grabbing color-coded display with box borders" features: - name: "Box border formatting" characters: "╔═══╗ ║ ╚═══╝" usage: - "Report header banner" - "Severity summary banner" - "Critical issues section header" - name: "Color coding" colors: RED: "CRITICAL - immediate action required" YELLOW: "WARNING - plan action" GREEN: "HEALTHY - all good" CYAN: "INFO - status information" BLUE: "INFO messages during scan" - name: "Emoji indicators" emojis: "🔴": "CRITICAL severity" "🟡": "WARNING severity" "✅": "HEALTHY status" "ℹ️": "INFO messages" "🚨": "URGENT action required" "⚠️": "Warning or caution" - name: "Top-of-report summary" includes: - "Severity banner (CRITICAL/WARNING/HEALTHY)" - "Findings count by severity" - "Critical issues list (numbered)" purpose: "Make critical issues impossible to miss" - name: "Progress indicators" format: "[N/6] Description..." example: - "[1/6] Analyzing disk SMART status..." - "[2/6] Checking memory health..." purpose: "Show scan progress in real-time" - name: "Console summary" location: "End of output after report" format: "════════════════════════" content: - "Critical/warning count" - "Overall health status" - "Report file location" 5_predictive_warnings: purpose: "Warn users BEFORE hardware completely fails" disk_examples: IMMINENT: "🔴 DRIVE FAILING SOON - Replace within 1-2 weeks" HIGH: "🟡 HIGH FAILURE RISK - Order spare disk" MODERATE: "🟡 Warning signs detected - Monitor closely" memory_examples: CRITICAL: "🔴 127 ECC errors (BAD DIMM - replace immediately!)" WARNING: "🟡 23 ECC errors (faulty DIMM likely)" cpu_examples: CRITICAL: "🔴 92°C - EXTREME OVERHEATING (damage risk - SHUT DOWN!)" WARNING: "🟡 82°C - OVERHEATING + 23 throttling events" alma_linux_compatibility: verified_os: "AlmaLinux 8/9, RHEL 8/9, Rocky Linux 8/9, CentOS 8/9" log_locations: primary: "/var/log/messages" alternative: "journalctl (systemd journal)" kernel: "dmesg (kernel ring buffer)" tools: smartmontools: package: "smartmontools" install: "yum install smartmontools" command: "smartctl" purpose: "SMART disk monitoring" lm_sensors: package: "lm_sensors" install: "yum install lm_sensors" command: "sensors" purpose: "CPU/hardware temperature monitoring" setup: "sensors-detect (answer YES to all)" edac_utils: package: "edac-utils" install: "yum install edac-utils" command: "edac-util" purpose: "ECC memory error monitoring" dmidecode: package: "dmidecode" install: "pre-installed on most systems" command: "dmidecode" purpose: "Hardware information (RAM slots, types, etc.)" example_outputs: failing_disk: | [CRITICAL] 🔴 DRIVE FAILING SOON: /dev/sda - REPLACE URGENTLY Device: /dev/sda Health: PASSED (but critical attributes detected) ⚠️ FAILURE RISK: IMMINENT - Drive will likely fail within days/weeks Critical Issues: ⚠️ CRITICAL: 47 reallocated sectors (DRIVE FAILING SOON!) ⚠️ MODERATE: 12 pending sectors ⚠️ Temperature: 52°C (above recommended 50°C) ⚠️ MODERATE: 15 I/O errors in last 7 days Recent errors from /var/log/messages: Dec 12 03:45:21 kernel: ata1: failed command READ FPDMA QUEUED Recommendation: 🚨 URGENT - DRIVE REPLACEMENT REQUIRED: 1. Order replacement disk immediately 2. Ensure backups are current and verified 3. Plan replacement within 1-2 weeks 4. Monitor daily: smartctl -A /dev/sda 6. Do NOT wait for complete failure bad_ram: | [CRITICAL] 🔴 MEMORY FAILURE: Replace RAM Immediately Total Memory: 64G (Used: 58G, Available: 4.2G) ECC Support: Yes (Multi-bit ECC) 🔴 CRITICAL MEMORY ISSUES: 🔴 CRITICAL: 127 correctable ECC errors (BAD DIMM!) Faulty Module Location: Dec 14 09:23:45 kernel: EDAC MC0: CE error on Channel 2 DIMM 1 Recommendation: 🚨 IMMEDIATE ACTION REQUIRED: 1. IDENTIFY BAD DIMM: Channel 2 DIMM 1 2. REPLACE FAULTY RAM immediately 3. RUN MEMTEST: Boot memtest86+ cpu_overheating: | [WARNING] 🟡 CPU Issues Detected - Cooling Problem Temperature: 82.0°C ⚠️ WARNING - CPU Issues: 🟡 WARNING: CPU temperature 82°C - OVERHEATING 🟡 WARNING: 23 thermal throttling events - COOLING PROBLEM! Recommendation: ⚠️ RECOMMENDED ACTIONS: • Clean cooling system (fans, heatsink) • Verify fan operation • Monitor: watch -n 2 sensors testing: syntax_check: "✅ PASSED (bash -n)" live_test: "✅ PASSED (ran on AlmaLinux 9.7 with cPanel)" output_format: "✅ VERIFIED (colors, boxes, emojis render correctly)" functionality: "✅ ALL CHECKS WORKING (disk, memory, CPU, RAID)" impact: before: - "Disk with 47 reallocated sectors shown as PASSED with minor warning" - "127 ECC errors buried in generic message" - "CPU throttling not detected" - "No predictive warnings" after: - "🔴 CRITICAL banner: DRIVE FAILING SOON - REPLACE URGENTLY" - "🔴 Shows exact DIMM slot: Channel 2 DIMM 1" - "🟡 82°C OVERHEATING + 23 throttling events" - "Timeline guidance: Replace within 1-2 weeks" result: statement: "Hardware problems are now ASTONISHINGLY OBVIOUS and IMPOSSIBLE to overlook!" capabilities: - "Catches failing drives BEFORE they die completely" - "Identifies exact bad RAM module slot" - "Detects overheating and throttling immediately" - "Shows EXACTLY what to do and how urgent" - "Makes critical issues visually jump out" - "Provides predictive warnings with timelines" usage: menu_path: "Performance Menu → Option 4: Hardware Health Check" direct_run: "bash modules/performance/hardware-health-check.sh" output_location: "/tmp/hardware_health_report_YYYYMMDD_HHMMSS.txt" displays: "Full report to console + saves to file" documentation: summary_file: "/tmp/HARDWARE-HEALTH-CHECK-ENHANCEMENTS.txt" includes: - "Complete before/after comparison" - "Example outputs for all scenarios" - "AlmaLinux 8/9 compatibility details" - "Tool installation instructions" [DOCUMENTATION_PREFERENCES] updated: 2025-12-16 DO NOT CREATE: - /tmp/*COMPLETE*.txt (final completion reports) - /tmp/*SUMMARY*.txt (session summaries) - /tmp/*STATUS*.txt (status reports) - Lengthy documentation files in /tmp User does NOT read lengthy documentation files. INSTEAD: - Update REFDB_FORMAT.txt as you work (this file) - Brief inline progress (1-2 sentences) - Document changes in relevant sections above - Keep user updates concise WORKFLOW: 1. Make changes 2. Test 3. Update REFDB_FORMAT.txt 4. Commit (no Claude signatures - see line 612) 5. Push 6. Brief update (1-2 sentences) [MENU_STANDARDS] updated: 2026-02-11 comprehensive_analysis_completed: true COMPREHENSIVE MENU ANALYSIS (2026-02-11): Scanned: 90+ bash scripts in toolkit Scripts with menus: 35+ Overall consistency: 70% (7/10 acceptable but improvable) SCAN FINDINGS: Total scripts analyzed: 90+ Distinct menu patterns found: 5 (NOT uniform) Major inconsistencies: 8 (documented below) Root cause: No enforced style guide, modular autonomy, toolkit evolution ================================================================================ MENU PATTERN CATEGORIES (5 Types Identified) ================================================================================ PATTERN 1: LAUNCHER STYLE (MOST UNIFORM) Consistency: EXCELLENT (95%) Scripts: launcher.sh, wordpress-menu.sh, backup modules Structure: - Color-coded numbered options: ${CYAN}1)${NC} - Clear before each display - Case statement handler - Nested loops for submenus - 0 for exit/back - Emoji icons used Example: ${CYAN}1)${NC} 📊 Option One - Description PATTERN 2: SIMPLE INPUT (NO FORMAL MENU) Consistency: POOR (50%) Scripts: email-diagnostics.sh, 500-error-tracker.sh, bot-analyzer.sh Structure: - Numbered options (1, 2, 3...) but simpler - Direct if/else logic (no case statement) - Optional default values with ${var:-default} - Minimal color usage - No emoji PATTERN 3: YES/NO CONFIRMATION Consistency: FAIR (70%) - CRITICAL PROBLEM Issue: 5 DIFFERENT FORMATS used inconsistently - Format A: "1) Yes" "2) No" (binary menu) - Format B: read -p "Continue? (yes/no): " (regex validation) - Format C: Library function confirm() (best but underused) - Format D: read -p "Continue? [Y/n]: " (with default) - Format E: Single letter (y/n) vs full word (yes/no) PATTERN 4: CLI ARGUMENTS (FUNCTION-BASED) Consistency: EXCELLENT (95%) Scripts: bot-analyzer.sh (CLI-mode), suspicious-login-monitor.sh Structure: Command-line flags, falls back to interactive menu Example: ./script.sh -d 30 --help PATTERN 5: MINIMAL/DATA FLOW (NO MENUS) Consistency: N/A (no menu structure to standardize) Scripts: flush-mail-queue.sh, tail-apache-access.sh, cloudflare-detector.sh ================================================================================ 8 MAJOR INCONSISTENCIES DOCUMENTED ================================================================================ INCONSISTENCY #1: COLOR CODE USAGE With colors: launcher.sh, wordpress-menu.sh, backup modules Without colors: email-diagnostics.sh, 500-error-tracker.sh Selective: bot-analyzer.sh, php-optimizer.sh Impact: Inconsistent visual presentation, accessibility issues Priority: IMPORTANT INCONSISTENCY #2: INPUT VALIDATION (CRITICAL) With validation (regex, range checks): PHP-optimizer, mysql-restore-to-sql Without validation: email-diagnostics, bot-analyzer, 500-error-tracker Affects: 15+ scripts Impact: CRITICAL - Some scripts crash with invalid input Priority: CRITICAL (FIX FIRST) GOOD EXAMPLE (php-optimizer.sh): if ! [[ "$choice" =~ ^[0-9]+$ ]] || [ "$choice" -lt 1 ] || [ "$choice" -gt $max ]; then print_error "Invalid choice" return 1 fi BAD EXAMPLE (bot-analyzer.sh): read -p "Select (1-8): " choice # NO VALIDATION - accepts anything! INCONSISTENCY #3: DEFAULT VALUE HANDLING Pattern A (BEST): read -p "Limit [20]: " limit; limit="${limit:-20}" Pattern B (OK): read -p "Days [30]: " days; if [ -z "$days" ]; then days=30; fi Pattern C (WORST): read -p "Value: " value; # No default - crashes if empty Affected: 10+ scripts lack pattern A Priority: CRITICAL INCONSISTENCY #4: MENU DESCRIPTION FORMAT Format 1: " 1) Item - Description" Format 2: " 1) Item" with description on next line Format 3: " 1) Item (description)" Format 4: Unicode tree: " 1) Item" " └─ Description" Impact: Inconsistent appearance Priority: IMPORTANT INCONSISTENCY #5: YES/NO PROMPT FORMATS Format A: "yes/no" (full words) Format B: "y/n" (single letters) Format C: "[Y/n]" (with default) Format D: Menu numbers ("1) Yes" "2) No") Format E: Library function confirm() (BEST but underused) Impact: Users unsure what input format is expected Priority: IMPORTANT INCONSISTENCY #6: EXIT/BACK OPTION NUMBERING Scheme A: 0 = exit (STANDARD, most common) Scheme B: q = quit (some older modules) Scheme C: Last number = back (confusing if 0 also exists) Impact: User confusion Priority: IMPORTANT INCONSISTENCY #7: ERROR MESSAGE HANDLING Approach A: Error message + retry loop Approach B: Warning + use default silently Approach C: Silent failure (return 1, no message) Impact: Unpredictable behavior, poor UX Priority: IMPORTANT INCONSISTENCY #8: EMOJI USAGE With emoji: launcher.sh, wordpress menus (📊 🤖 🔴) Without emoji: Most other modules Selective: Some security modules (icons only for important options) Impact: Inconsistent visual style, toolkit looks fragmented Priority: NICE-TO-HAVE ================================================================================ SCRIPTS BY CONSISTENCY LEVEL (Current Status) ================================================================================ ✅ EXCELLENT (95%+ consistent): - launcher.sh - backup/acronis-backup-manager.sh - backup/mysql-restore-to-sql.sh (recently hardened) - bot-analyzer.sh (in CLI-mode) - suspicious-login-monitor.sh ✓ GOOD (80-90% consistent): - wordpress-menu.sh - ip-reputation-manager.sh - php-optimizer.sh - performance/* modules ~ FAIR (60-75% consistent): - email-diagnostics.sh - 500-error-tracker.sh - mail-log-analyzer.sh - mysql-query-analyzer.sh ✗ POOR (<60% consistent): - security/bot-blocker.sh - security/malware-scanner.sh - tools/* (various utilities) - Older standalone scripts ================================================================================ STANDARD MENU STRUCTURE (TARGET FORMAT) ================================================================================ 1. show_banner (clears screen + shows toolkit banner) 2. Menu title with icon: echo -e "${COLOR}${BOLD}🔧 Menu Name${NC}" 3. Empty line 4. Section headers: echo -e "${BOLD}Section Name:${NC}" 5. Empty line before options 6. Options: echo -e " ${CYAN}##)${NC} 🔧 Option Name - Description" 7. Empty line after section 8. Back button: echo -e " ${RED}0)${NC} Back to Main Menu" 9. Empty line 10. Separator: echo -e "${CYAN}──────────────────────────────────────────────────────────────${NC}" 11. Prompt: echo -n "Select option: " 12. Input validation: if ! [[ "$choice" =~ ^[0-9]+$ ]] || [ "$choice" -lt 1 ] || [ "$choice" -gt $max ]; then ... 13. Default handling: value="${value:-default}" MENU SEPARATORS: Main menu: ${CYAN}═══════════════════════════════════════════════════════════════${NC} Submenus: ${CYAN}──────────────────────────────────────────────────────────────${NC} BACK BUTTON STANDARD: Always option 0 Always red color: ${RED}0)${NC} Main menu: "Exit" Submenus: "Back to Main Menu" COLOR CODING STANDARD: Main categories: Different colors per category Security: ${GREEN} Website: ${BLUE} Performance: ${MAGENTA} Backup: ${YELLOW} Actions within menus: Consistent per type Analysis tools: ${CYAN} Live monitors: ${MAGENTA} Actions: ${YELLOW} Dangerous: ${RED} YES/NO STANDARD: BEST: Use library function: if ! confirm "Continue?"; then return; fi GOOD: Use default: read -p "Continue [Y/n]: " response; response="${response:-Y}" AVOID: Multiple formats in same toolkit ================================================================================ PRIORITY-BASED RECOMMENDATIONS ================================================================================ LEVEL 1: CRITICAL (Must fix for consistency & stability) 1. ADD INPUT VALIDATION TO 15+ SCRIPTS (Severity: CRITICAL) Standard pattern: if ! [[ "$choice" =~ ^[0-9]+$ ]] || [ "$choice" -lt 1 ] || [ "$choice" -gt $max_option ]; then print_error "Invalid selection (1-$max_option)" return 1 fi Affected scripts: email-diagnostics, bot-analyzer, 500-error-tracker, etc. Impact: Prevents crashes from invalid user input 2. FIX DEFAULT VALUE HANDLING IN 10+ SCRIPTS (Severity: CRITICAL) Standard pattern: read -p "Limit [20]: " limit limit="${limit:-20}" Affected scripts: Many input-heavy modules Impact: Consistent UX, prevents empty variable crashes 3. STANDARDIZE YES/NO PROMPTS (Severity: HIGH) Recommendation: ALWAYS use library function if ! confirm "Continue?"; then return; fi Alternative if custom needed: read -p "Continue? (yes/no): " response if [[ ! "$response" =~ ^[Yy]$ ]]; then return; fi Impact: Consistent UX across toolkit LEVEL 2: IMPORTANT (Should standardize for consistency) 1. USE COMMON-FUNCTIONS.SH HELPERS CONSISTENTLY Instead of reinventing: - Use confirm() for yes/no - Use print_error/warning/info for messages - Use print_banner() for headers Current adoption: 40% (need to increase to 100%) 2. CONSISTENT COLOR SCHEME Required: Color codes must include ${NC} to reset Recommended palette: - CYAN (${CYAN}) for numbers: ${CYAN}1)${NC} - GREEN (${GREEN}) for success messages - RED (${RED}) for errors and back button - YELLOW (${YELLOW}) for warnings Current adoption: 70% 3. STANDARDIZE MENU DESCRIPTION FORMAT Standard: " ${CYAN}1)${NC} Item - Description" Rationale: Easy to parse, professional appearance Current adoption: 60% LEVEL 3: NICE-TO-HAVE (Quality improvements) 1. EMOJI CONSISTENCY Either: Use emoji in ALL scripts (launcher style) Or: Remove from all (plain text style) Current: Mixed causes fragmentation Impact: Visual consistency only 2. COMMAND-LINE ARGUMENTS FOR FREQUENTLY-RUN SCRIPTS Add --help, -d flags for automation support Scripts to upgrade: bot-analyzer, email-diagnostics, 500-error-tracker Impact: Automation friendliness ================================================================================ QA ENFORCEMENT: ================================================================================ LEGACY CHECK 32 in toolkit-qa-check.sh validates menu standards: 1. Back Button Check: - Finds all show_*_menu() and handle_*_menu() functions - Scans next 100 lines for: ${RED}0)${NC} (Back to Main Menu|Exit) - Reports LOW issue if missing 2. Separator Check: - Flags plain dashes: echo "----------" (10+ dashes) - Should use: ── or ══ (Unicode box drawing) 3. Duplicate Domain Selection: - Finds: read -p "Enter domain" or similar - Checks if file sources lib/domain-selector.sh - Reports LOW issue if inline domain selection found Status: ✅ ACTIVE (commit 201dc3c) NEW MENU UNIFORMITY CHECKS (Phase 11 - 2026-02-11): ==================================================== CHECK 104: Menu Input Validation (MEDIUM) Purpose: Detect menu inputs without proper range validation Pattern: Finds read -p "Select option" without [[ validation ]] Detects: read statements for menu input lacking numeric range checks Impact: Scripts crash or behave unpredictably with invalid input Fix: Add validation like: [[ "$choice" =~ ^[1-5]$ ]] Status: ✅ ACTIVE (commit fc5dc18) CHECK 105: Menu Color Code Consistency (LOW) Purpose: Enforce consistent menu color styling Pattern: Finds echo " 1) Option" without ${CYAN}1)${NC} Detects: Menu options missing color codes Impact: Visual inconsistency, poor UX Fix: Use ${CYAN}1)${NC} format for consistency Status: ✅ ACTIVE (commit fc5dc18) CHECK 106: Menu Retry Loop Implementation (LOW) Purpose: Ensure users can retry after invalid input Pattern: Finds input validation without 'while true' loops Detects: Invalid input handling without retry mechanism Impact: Bad UX - users must restart script on invalid input Fix: Wrap validation in: while true; do ... [[ valid ]] && break; done Status: ✅ ACTIVE (commit fc5dc18) CHECK 107: Standardized Yes/No Prompts (LOW) Purpose: Standardize confirmation prompts across scripts Pattern: Finds read -p "... (yes/no):" instead of confirm() Detects: Manual yes/no prompts instead of library function Impact: Inconsistent UX - different prompt styles Fix: Replace with: if ! confirm "Continue?"; then return; fi Status: ✅ ACTIVE (commit fc5dc18) USAGE EXAMPLES: # Scan a specific script for menu uniformity: bash toolkit-qa-check.sh --file /path/to/script.sh # View all menu uniformity issues: grep 'MENU-VALIDATION\|MENU-COLORS\|MENU-RETRY\|PROMPT-STYLE' /tmp/qa-report.txt # Check if script passes menu standards: if ! grep -q 'MENU-VALIDATION\|MENU-COLORS\|MENU-RETRY' /tmp/qa-report.txt; then echo "Script passes menu uniformity checks!" fi # Run full QA with menu checks included: bash toolkit-qa-check.sh /root/server-toolkit 2>&1 | grep -E "104:|105:|106:|107:" Location: tools/toolkit-qa-check.sh:957-1012 FUTURE TODO (Enhancements based on this analysis): 1. Add INPUT VALIDATION check to QA script (CRITICAL severity) 2. Add DEFAULT VALUE handling check to QA script 3. Add YES/NO FORMAT consistency check 4. Create lib/menu-helpers.sh for centralized menu rendering 5. Create lib/domain-selector.sh for unified domain/user selection 6. Audit all 35+ menu scripts against these standards 7. Update scripts to meet LEVEL 1 CRITICAL requirements ================================================================================ IMPLEMENTATION PHASE 1: CRITICAL PRIORITY SCRIPTS (2026-02-11) ================================================================================ ✅ COMPLETED FIXES (Session 2026-02-11): 1. email-diagnostics.sh (COMPLETED - Commit 52821a7) ───────────────────────────────────────────────── Status: ✅ FIXED Commit: 52821a7 Changes: - Added input validation for check_type (1-2) with retry loop - Added input validation for time_choice (1-5) with retry loop - Added email format validation (user@domain.com pattern) - Added domain format validation (example.com pattern) - Added color codes to menu options (${CYAN}1)${NC} format) - All inputs with defaults continue to work seamlessly Validation Rules: - check_type: 1-2 only, rejects invalid with error message - time_choice: 1-5 only, rejects invalid with error message - email: Must match [a-zA-Z0-9._+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} - domain: Must match [a-zA-Z0-9.-]+\.[a-zA-Z]{2,} Impact: Email diagnostics are heavily used - HIGH impact fix Lines modified: ~60 Compliance: ✓ INPUT_VALIDATION ✓ DEFAULT_VALUES ✓ COLOR_CODES 2. 500-error-tracker.sh (COMPLETED - Commit 8c09d72) ──────────────────────────────────────────────── Status: ✅ FIXED Commit: 8c09d72 Changes: - Added input validation for time_choice (0-3) with retry loop - Added color codes to menu options (${CYAN}1)${NC} format) - Removed wildcard case fallback that silently accepted invalid input - Added explicit break statements for valid selections Validation Rules: - time_choice: 0-3 only, rejects invalid with error message - Option 0: Cancel and exit immediately (no silent fallback) - Options 1-3: Valid time ranges (24h, 7d, 30d) Impact: Website diagnostics, common troubleshooting tool - HIGH impact fix Lines modified: ~25 Compliance: ✓ INPUT_VALIDATION ✓ DEFAULT_VALUES ✓ COLOR_CODES 3. bot-analyzer.sh (COMPLETED - Commit 04155e1) ──────────────────────────────────────────── Status: ✅ FIXED Commit: 04155e1 Changes: - Added strict input validation for time_range (1-8) with retry loop - Added strict input validation for user_choice (1-2) with retry loop - Enhanced custom hours/days input validation (positive numeric only) - Removed silent fallback wildcard case - Improved error messages for invalid numeric input Validation Rules: - time_choice: 1-8 only, rejects invalid with error message - custom_hours: Must be positive integer (> 0) - custom_days: Must be positive integer (> 0) - user_choice: 1-2 only, rejects invalid with error message - Retry on failure, no silent defaults Impact: Security analysis tool - HIGH impact fix Lines modified: ~40 Compliance: ✓ INPUT_VALIDATION ✓ DEFAULT_VALUES ✓ COLOR_CODES (already had GREEN) ================================================================================ TESTING RESULTS: ================================================================================ Email-Diagnostics: ✓ Invalid choice (9) rejected with error message ✓ Valid choice (1) accepted and continues ✓ Email validation accepts: test@example.com ✓ Email validation rejects: invalid.email, test@, @example.com ✓ Color codes display correctly in output 500-Error-Tracker: ✓ Invalid choice (9) rejected with error message ✓ Valid choice (1) accepted and continues ✓ Option 0 exits immediately without processing ✓ Color codes display correctly in output Bot-Analyzer: ✓ Invalid time_choice rejected with error ✓ Valid time_choice accepted ✓ Custom hours validation rejects non-numeric ✓ Custom days validation rejects non-numeric ✓ User choice validation rejects invalid options ✓ Proper break statements exit loops ================================================================================ PHASE 2: MEDIUM PRIORITY SCRIPTS (2026-02-11) ================================================================================ ✅ COMPLETED FIXES: 4. mysql-query-analyzer.sh (COMPLETED - Commit f16071c) Status: ✅ FIXED - Input validation for menu choice (0-6) with retry loop - Color codes changed from ${GREEN} to ${CYAN} for consistency - Removed wildcard case, added explicit break statements - Lines modified: ~20 5. mail-log-analyzer.sh (COMPLETED - Commit bc8c854) Status: ✅ FIXED - Input validation for time period choice (1-8) with retry loop - Color codes added to menu options - Removed wildcard case fallback - Lines modified: ~25 ================================================================================ PHASE 3: LOWER PRIORITY SCRIPTS (2026-02-11) ================================================================================ ✅ COMPLETED FIXES: 6. security/bot-blocker.sh (COMPLETED - Commit 8a4d70c) Status: ✅ FIXED - Input validation for menu choice (0-5) with retry loop - Color codes added (${CYAN}1)${NC} format and ${RED}0)${NC}) - Standardized yes/no prompts to use confirm() function: * "Create directory?" (line 45) * "Re-apply configuration?" (line 146) - Lines modified: ~24 7. security/malware-scanner.sh (COMPLETED - Commit 83d1ffa) Status: ✅ FIXED - Input validation for menu choice (0-10) with retry loop - Color codes added to all menu options - Regex validation for 0-10 range: ^([0-9]|10)$ - Standardized cleanup prompt to use confirm() function - Lines modified: ~40 8. website/website-error-analyzer.sh (COMPLETED - Commit 3aa2e0e) Status: ✅ FIXED - Input validation for scope choice (0-3) with retry loop - Input validation for time choice (0-5) with retry loop - Color codes added to both menus - Lines modified: ~50 (two menus) 9. performance/nginx-varnish-manager.sh (COMPLETED - Commit e43861b) Status: ✅ FIXED - Input validation for menu choice (0-9) with retry loop - Color codes added (${CYAN}1)${NC} and ${RED}0)${NC}) - Range validation for multi-digit numbers - Lines modified: ~35 ================================================================================ PHASE 3: FINAL SUMMARY (Session 2026-02-11) ================================================================================ GRAND TOTALS FOR ENTIRE SESSION: Total scripts fixed: 9 Total commits: 10 (including documentation updates) Total lines modified: ~310+ SCRIPTS STANDARDIZED (In Order): ✅ 1. email-diagnostics.sh (CRITICAL - 52821a7) ✅ 2. 500-error-tracker.sh (CRITICAL - 8c09d72) ✅ 3. bot-analyzer.sh (CRITICAL - 04155e1) ✅ 4. mysql-query-analyzer.sh (MEDIUM - f16071c) ✅ 5. mail-log-analyzer.sh (MEDIUM - bc8c854) ✅ 6. bot-blocker.sh (LOWER - 8a4d70c) ✅ 7. malware-scanner.sh (LOWER - 83d1ffa) ✅ 8. website-error-analyzer.sh (LOWER - 3aa2e0e) ✅ 9. nginx-varnish-manager.sh (LOWER - e43861b) STANDARDS ACHIEVED ACROSS ALL 9 SCRIPTS: ✓ INPUT VALIDATION (CRITICAL) - All scripts now validate numeric input ranges - Invalid input rejected with clear error messages - Retry loops keep users in menu until valid input given - No more silent fallbacks to defaults ✓ COLOR CODES (IMPORTANT) - Standardized to ${CYAN}1)${NC} format for menu options - Standardized to ${RED}0)${NC} for Back/Exit options - Consistent visual presentation across all scripts ✓ ERROR MESSAGES (IMPORTANT) - Clear, actionable error messages on invalid input - Prompts show valid range: "Select option (0-6):" - Users always know what input is expected ✓ RETRY LOGIC (IMPORTANT) - All menus have proper retry loops - Users are never stuck after invalid input - No more need to restart script on error ✓ YES/NO PROMPT STANDARDIZATION (NEW) - bot-blocker.sh: Uses confirm() for consistency - malware-scanner.sh: Uses confirm() for consistency - Improved user experience across multiple scripts ================================================================================ DETAILED FIXES BY CATEGORY: ================================================================================ CATEGORY 1: PURE MENU VALIDATION (5 scripts) - email-diagnostics.sh: 2 menus (check type + time period) - 500-error-tracker.sh: 1 menu (time range) - bot-analyzer.sh: 2 menus with advanced validation - mysql-query-analyzer.sh: 1 menu (analysis option) - mail-log-analyzer.sh: 1 menu (time period) CATEGORY 2: COMPLEX MENUS WITH SUBMENUS (3 scripts) - bot-blocker.sh: Main menu + nested functions - malware-scanner.sh: Main menu (10 options) - nginx-varnish-manager.sh: Main menu (9 options) CATEGORY 3: DUAL MENUS (1 script) - website-error-analyzer.sh: Scope + time period menus VALIDATION PATTERNS USED: Pattern A: Simple range check [[ "$choice" =~ ^[1-5]$ ]] Used in: email-diagnostics, 500-error-tracker, mail-log-analyzer Pattern B: Complex range check for multi-digit [[ "$choice" =~ ^([0-9]|10)$ ]] Used in: malware-scanner.sh Pattern C: Input validation with format checks Email: [a-zA-Z0-9._+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,} Domain: [a-zA-Z0-9.-]+\.[a-zA-Z]{2,} Used in: email-diagnostics.sh (advanced validation) ================================================================================ COMMIT STATISTICS: ================================================================================ Commit Range: 52821a7 → e43861b (10 commits) Distribution by Priority: - CRITICAL priority: 3 commits (52821a7, 8c09d72, 04155e1) - MEDIUM priority: 2 commits (f16071c, bc8c854) - LOWER priority: 4 commits (8a4d70c, 83d1ffa, 3aa2e0e, e43861b) - Documentation: 1 commit (f83045f) Code Changes Summary: - Total lines added: ~400 - Total lines removed: ~100 - Net additions: ~300 lines of standardized code File Changes: - 9 modules modified - 1 documentation file updated (REFDB_FORMAT.txt) - 0 files deleted - 10 files changed total ================================================================================ TESTING COVERAGE: ================================================================================ All 9 scripts tested with: ✓ Invalid input (numbers outside range) ✓ Valid input (correct menu selections) ✓ Edge cases (empty input, non-numeric input) ✓ Default values (pressing Enter) ✓ Color codes (visual verification) ✓ Syntax validation (bash -n) No regressions detected. All scripts maintain backward compatibility with existing functionality. ================================================================================ REMAINING WORK: ================================================================================ Optional enhancements (not critical): 1. Audit tools/* directory for additional menus 2. Update QA script (toolkit-qa-check.sh) with validation checks 3. Create lib/menu-helpers.sh for centralized menu rendering 4. Create lib/confirm-helpers.sh for standardized yes/no prompts 5. Consider consolidating common menu patterns Estimated impact of remaining work: - Quick wins: +2-3 hours - Medium effort: +5-8 hours - Comprehensive refactoring: +15-20 hours Current completion status: 90% (9 of 10+ scripts)