Managing an AWS EC2 server means monitoring system resources, Docker containers, disk usage, running processes, and IAM security. Without a proper audit, unused resources can consume storage, and outdated access keys or missing MFA can create security risks.

HOW I DEBUGGED IT

● I built a Python-based audit tool to investigate the server step by step.

●  Checked CPU, RAM, and disk usage. Reviewed Docker containers, images, networks, and volumes.

●   Identified large files and reviewed long-running or idle processes.Audited IAM users, MFA status, access keys, and administrative permissions

THE SOLUTION    EC2 → Audit Checks → Safe Cleanup → Reports

I built a Python audit tool with Dry-Run mode, audit logs, and JSON reports to identify issues and quickly review cleanup actions.

Has anyone else faced this problem?

How did you fix it?

HashnodProfile:https://hashnode.com/@vishnusaiK
GitHub : https://github.com/ksaivishnusaikvs/DevOps-Engineering/tree/main/Linkedin/Post

Step1: System Monitoring: Displays server details, hostname, IP address, and logged-in user. Monitors CPU, RAM, and disk usage while running in Dry-Run mode without making changes.

Step2: The system resource screen provides a live view of CPU, memory utilization, and disk usage across mounted partitions. This helps identify servers or resources that are consuming high CPU, RAM, or disk capacity.

Step3: The document clearly shows Docker containers, images, networks, and volumes. However, the production cleanup is highly destructive because it can remove all containers, images, networks, and volumes. AWSMONITORING AUDIT(2)Add a clear warning before the production screenshot: “Production cleanup should be performed only after validating that the resources are unused.” sime and sho

Step4: Good coverage of partitions, total/used/free space, and large files. The /var, /tmp, and /home analysis is useful for identifying storage-consuming files. AWS MONITORING AUDIT(2): Add the actual finding/result below the screenshot instead of only showing the command output.

Step5: The script checks idle and long-running processes and has a protected-process list. AWS MONITORING AUDIT(2): This is good from a safety perspective, but the document should explain why a process was identified as a cleanup candidate before showing termination.

Step6: This is one of the strongest sections. It checks MFA, inactive users, access-key age, admin access, policies and groups, and generates an IAM report. AWS MONITORING AUDIT(2): The recommendations such as MFA enablement, key rotation, and admin-access review are clearly presented.

Step7: IAM Security & Audit Logs: The IAM audit reviews users, MFA, access keys, and flagged accounts, while recording audit activities.The summary provides the security status and generates a detailed JSON audit report.

======================================================================
ubuntu@ip-172-31-44-197:~$ cat awsaudit.py 
#!/usr/bin/env python3
"""
╔══════════════════════════════════════════════════════════════════╗
║        ENTERPRISE PRODUCTION CLEANUP SCRIPT  v4.0               ║
║  Docker · Processes · Disk · IAM User Tracker                   ║
║  Dry-Run + Production · Audit Trail · NDJSON Logs               ║
╚══════════════════════════════════════════════════════════════════╝
"""

import os
import sys
import json
import signal
import socket
import datetime
import platform
import argparse
import subprocess
import textwrap
from pathlib import Path
from typing import List, Dict, Tuple, Optional

# ── Auto-install dependencies ──────────────────────────────────────
def _pip(pkg):
    subprocess.check_call([sys.executable, "-m", "pip", "install", pkg, "--quiet",
                           "--break-system-packages"], stderr=subprocess.DEVNULL)

try:
    from rich.console import Console
    from rich.table import Table
    from rich.panel import Panel
    from rich.prompt import Prompt, Confirm
    from rich.progress import Progress, SpinnerColumn, BarColumn, TextColumn, TimeElapsedColumn
    from rich import box
    from rich.rule import Rule
except ImportError:
    _pip("rich")
    from rich.console import Console
    from rich.table import Table
    from rich.panel import Panel
    from rich.prompt import Prompt, Confirm
    from rich.progress import Progress, SpinnerColumn, BarColumn, TextColumn, TimeElapsedColumn
    from rich import box
    from rich.rule import Rule

try:
    import psutil
except ImportError:
    _pip("psutil"); import psutil

try:
    import boto3
    from botocore.exceptions import ClientError, NoCredentialsError
except ImportError:
    _pip("boto3")
    import boto3
    from botocore.exceptions import ClientError, NoCredentialsError

# ══════════════════════════════════════════════════════════════════
# GLOBALS
# ══════════════════════════════════════════════════════════════════
console      = Console()
SCRIPT_START = datetime.datetime.now(datetime.timezone.utc)
LOG_DIR      = Path.home() / "logs" / "prod_cleanup"
RUN_TAG      = SCRIPT_START.strftime("%Y%m%dT%H%M%SZ")
LOG_DIR.mkdir(parents=True, exist_ok=True)

RUN_LOG       = LOG_DIR / f"cleanup_{RUN_TAG}.ndjson"
AUDIT_LOG     = LOG_DIR / "audit_master.ndjson"
DELETIONS_LOG = LOG_DIR / f"deletions_{RUN_TAG}.json"
UNPROC_LOG    = LOG_DIR / f"unprocessed_{RUN_TAG}.json"
IAM_REPORT    = LOG_DIR / f"iam_report_{RUN_TAG}.json"

DELETIONS  : List[dict] = []
UNPROCESSED: List[dict] = []
STATS: Dict = {
    "docker_containers_killed": 0,
    "docker_images_removed":    0,
    "docker_networks_removed":  0,
    "docker_volumes_removed":   0,
    "processes_killed":         0,
    "iam_users_scanned":        0,
    "iam_keys_flagged":         0,
    "iam_inactive_users":       0,
    "iam_no_mfa":               0,
}

PROTECTED_PROCS = {
    "init","systemd","kthreadd","kworker","ksoftirqd","migration","rcu_sched","rcu_bh",
    "watchdog","sshd","bash","python3","python","journald","dbus-daemon","rsyslogd",
    "chronyd","cron","atd","auditd","containerd","dockerd","docker","kubelet","etcd",
    "aws","amazon-ssm-agent","amazon-cloudwatch-agent","prod.py","prod_cleanup.py","cl.py",
}

IDLE_THRESHOLD_HOURS = 2
LONG_RUNNING_HOURS   = 24
KEY_WARN_DAYS        = 90
KEY_CRIT_DAYS        = 180
INACTIVE_DAYS        = 90

# ══════════════════════════════════════════════════════════════════
# LOGGING HELPERS
# ══════════════════════════════════════════════════════════════════
def _now_iso():
    return datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")

def log_event(event_type, payload):
    entry = {"ts": _now_iso(), "type": event_type, **payload}
    for f in (RUN_LOG, AUDIT_LOG):
        with open(f, "a") as fp:
            fp.write(json.dumps(entry) + "\n")

def record_deletion(kind, ident, details=None):
    rec = {"ts": _now_iso(), "kind": kind, "id": ident, "details": details or {}}
    DELETIONS.append(rec); log_event("DELETE", rec)

def record_unprocessed(kind, ident, reason):
    rec = {"ts": _now_iso(), "kind": kind, "id": ident, "reason": reason}
    UNPROCESSED.append(rec); log_event("SKIP", rec)

def flush_json_logs():
    with open(DELETIONS_LOG, "w") as f: json.dump(DELETIONS, f, indent=2)
    with open(UNPROC_LOG,    "w") as f: json.dump(UNPROCESSED, f, indent=2)

# ══════════════════════════════════════════════════════════════════
# UTILS
# ══════════════════════════════════════════════════════════════════
def run(cmd, capture=True):
    r = subprocess.run(cmd, shell=True, capture_output=capture, text=True)
    return r.returncode, r.stdout.strip(), r.stderr.strip()

def docker_available():
    return run("docker info")[0] == 0

def bytes_to_mb(b):
    return round(b / 1_048_576, 2)

def bar(pct, width=30):
    f = int(pct / 100 * width)
    return "[green]" + chr(9608) * f + "[/green]" + chr(9617) * (width - f)

def days_ago(dt):
    if dt is None: return None
    if dt.tzinfo is None: dt = dt.replace(tzinfo=datetime.timezone.utc)
    return (datetime.datetime.now(datetime.timezone.utc) - dt).days

# ══════════════════════════════════════════════════════════════════
# SECTION 1 — RUNNER IDENTITY
# ══════════════════════════════════════════════════════════════════
def show_identity():
    hostname = socket.gethostname()
    uname    = platform.uname()
    try:    local_ip = socket.gethostbyname(hostname)
    except: local_ip = "unknown"
    rc, pub_ip, _ = run("curl -s --max-time 5 https://api.ipify.org")
    pub_ip = pub_ip if rc == 0 else "unavailable"
    try:
        import pwd
        pw = pwd.getpwuid(os.getuid())
        username, home, shell = pw.pw_name, pw.pw_dir, pw.pw_shell
    except:
        username = os.environ.get("USER","unknown")
        home, shell = os.path.expanduser("~"), "unknown"

    lines = [
        f"[bold cyan]Hostname:[/bold cyan]   {hostname}",
        f"[bold cyan]OS/Kernel:[/bold cyan]  {uname.system} {uname.release} ({uname.machine})",
        f"[bold cyan]Started:[/bold cyan]    {SCRIPT_START.strftime('%Y-%m-%dT%H:%M:%SZ')}",
        "",
        "[bold yellow]-- Network ----------------------------------------[/bold yellow]",
        f"[bold cyan]Local IP:[/bold cyan]   {local_ip}",
        f"[bold cyan]Public IP:[/bold cyan]  {pub_ip}",
        "",
        "[bold yellow]-- Linux User -------------------------------------[/bold yellow]",
        f"[bold cyan]Username:[/bold cyan]   {username} (uid={os.getuid()}, euid={os.geteuid()})",
        f"[bold cyan]Root?:[/bold cyan]      {'[bold red]Yes[/bold red]' if os.geteuid()==0 else 'No'}",
        f"[bold cyan]Home:[/bold cyan]       {home} | {shell}",
        "",
        "[bold yellow]-- Log Folder -------------------------------------[/bold yellow]",
        f"[bold cyan]Log Dir:[/bold cyan]    {LOG_DIR}",
        f"[bold cyan]IAM Report:[/bold cyan] {IAM_REPORT.name}",
    ]
    console.print(Panel("\n".join(lines), title="[bold]RUNNER IDENTITY[/bold]", border_style="blue"))
    log_event("IDENTITY", {"hostname": hostname, "public_ip": pub_ip, "user": username})

# ══════════════════════════════════════════════════════════════════
# SECTION 2 — RESOURCE USAGE
# ══════════════════════════════════════════════════════════════════
def show_resource_usage():
    cpu = psutil.cpu_percent(interval=1)
    mem = psutil.virtual_memory()
    lines = [
        f"[bold]CPU[/bold]  {bar(cpu)}  {cpu:.1f}%\n",
        f"[bold]RAM[/bold]  {bar(mem.percent)}  {mem.percent:.1f}%  "
        f"{bytes_to_mb(mem.used):.0f} MB / {bytes_to_mb(mem.total):.0f} MB\n",
        "[bold]Disk[/bold]",
    ]
    for p in psutil.disk_partitions():
        try:
            u = psutil.disk_usage(p.mountpoint)
            lines.append(f"  {p.mountpoint:<20} {bar(u.percent, 20)}  "
                         f"{u.percent:.0f}%  {u.used/1e9:.1f}G / {u.total/1e9:.1f}G")
        except PermissionError:
            pass
    console.print(Panel("\n".join(lines), title="[bold]RESOURCE USAGE[/bold]", border_style="cyan"))

# ══════════════════════════════════════════════════════════════════
# SECTION 3 — DOCKER CLEANUP
# ══════════════════════════════════════════════════════════════════
DOCKER_ONE_LINER = (
    "docker ps -aq | xargs -r docker stop && "
    "docker ps -aq | xargs -r docker rm -f && "
    "docker images -q | xargs -r docker rmi -f && "
    "docker network ls --filter type=custom -q | xargs -r docker network rm && "
    "docker volume ls -q | xargs -r docker volume rm && "
    "docker system prune -af --volumes"
)

def docker_cleanup(dry_run, interactive):
    if not docker_available():
        console.print("[yellow]  Docker daemon not reachable -- skipping.[/yellow]"); return
    console.print(Rule("[bold red]DOCKER FULL CLEANUP[/bold red]"))

    _, cr, _ = run("docker ps -a --format '{{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Image}}'")
    _, ir, _ = run("docker images --format '{{.ID}}\t{{.Repository}}\t{{.Tag}}\t{{.Size}}'")
    _, nr, _ = run("docker network ls --filter type=custom --format '{{.ID}}\t{{.Name}}\t{{.Driver}}'")
    _, vr, _ = run("docker volume ls --format '{{.Name}}\t{{.Driver}}'")

    containers = [l.split("\t") for l in cr.splitlines() if l]
    images     = [l.split("\t") for l in ir.splitlines() if l]
    networks   = [l.split("\t") for l in nr.splitlines() if l]
    volumes    = [l.split("\t") for l in vr.splitlines() if l]

    def make_table(title, headers, rows, style="cyan"):
        t = Table(title=title, box=box.ROUNDED, border_style=style, show_lines=True)
        for h in headers: t.add_column(h, style="white")
        for r in rows:    t.add_row(*r[:len(headers)])
        return t

    console.print(make_table("Containers", ["ID","Name","Status","Image"], containers)
                  if containers else "[green]  No containers.[/green]")
    console.print(make_table("Images",    ["ID","Repository","Tag","Size"], images, "yellow")
                  if images     else "[green]  No images.[/green]")
    console.print(make_table("Networks",  ["ID","Name","Driver"], networks, "magenta")
                  if networks   else "[green]  No custom networks.[/green]")
    if volumes:
        vt = Table(title="Volumes", box=box.ROUNDED, border_style="red")
        vt.add_column("Name"); vt.add_column("Driver")
        for v in volumes: vt.add_row(*v[:2])
        console.print(vt)
    else:
        console.print("[green]  No volumes.[/green]")

    console.print(Panel(
        f"[bold yellow]{DOCKER_ONE_LINER}{'  [DRY-RUN]' if dry_run else ''}[/bold yellow]",
        title="DOCKER NUKE ONE-LINER", border_style="red"))

    if not (containers or images or networks or volumes):
        console.print("[green]  Docker already clean.[/green]"); return

    if dry_run:
        console.print("[bold yellow]  [DRY-RUN] No changes made.[/bold yellow]")
        for c in containers: record_unprocessed("docker_container", c[0], "dry-run")
        for i in images:     record_unprocessed("docker_image",     i[0], "dry-run")
        return

    if interactive and not Confirm.ask(
        f"[bold red]Remove {len(containers)} containers, {len(images)} images, "
        f"{len(networks)} networks, {len(volumes)} volumes?[/bold red]", default=False):
        console.print("[yellow]  Skipped.[/yellow]"); return

    steps = [
        ("Stop all containers",    "docker ps -aq | xargs -r docker stop"),
        ("Remove all containers",  "docker ps -aq | xargs -r docker rm -f"),
        ("Remove ALL images",      "docker images -q | xargs -r docker rmi -f"),
        ("Remove custom networks", "docker network ls --filter type=custom -q | xargs -r docker network rm"),
        ("Remove all volumes",     "docker volume ls -q | xargs -r docker volume rm"),
        ("System prune -af",       "docker system prune -af --volumes"),
    ]
    with Progress(SpinnerColumn(), TextColumn("[progress.description]{task.description}"),
                  BarColumn(), TimeElapsedColumn(), console=console) as prog:
        task = prog.add_task("Docker cleanup...", total=len(steps))
        for label, cmd in steps:
            prog.update(task, description=f"[cyan]{label}[/cyan]")
            rc, out, err = run(cmd)
            log_event("DOCKER_STEP", {"cmd": cmd, "rc": rc})
            prog.advance(task)

    console.print("[bold green]  Docker cleanup complete.[/bold green]")
    for c in containers:
        record_deletion("docker_container", c[0], {"name": c[1] if len(c)>1 else ""})
        STATS["docker_containers_killed"] += 1
    for i in images:
        record_deletion("docker_image", i[0]); STATS["docker_images_removed"] += 1
    for n in networks:
        record_deletion("docker_network", n[0]); STATS["docker_networks_removed"] += 1
    for v in volumes:
        record_deletion("docker_volume", v[0]); STATS["docker_volumes_removed"] += 1

# ══════════════════════════════════════════════════════════════════
# SECTION 4 — PROCESS CLEANUP
# ══════════════════════════════════════════════════════════════════
def _is_protected(name):
    nl = name.lower()
    return any(p in nl for p in PROTECTED_PROCS)

def _proc_age_h(proc):
    try: return (datetime.datetime.now().timestamp() - proc.create_time()) / 3600
    except: return 0.0

def _idle_h(proc):
    try:
        ct = proc.cpu_times()
        age = _proc_age_h(proc)
        return age if age > 0 and (ct.user + ct.system) < 0.5 else 0.0
    except: return 0.0

def collect_suspicious_processes():
    idle, long_r = [], []
    for proc in psutil.process_iter(["pid","name","username","status",
                                      "cpu_percent","memory_info","create_time","cmdline"]):
        try:
            info = proc.info
            name = info["name"] or ""
            if _is_protected(name) or info["pid"] in (0,1): continue
            if info["username"] in ("root", None) and name.startswith("k"): continue
            age_h = round(_proc_age_h(proc), 1)
            ih    = round(_idle_h(proc), 1)
            mem   = bytes_to_mb(info["memory_info"].rss) if info["memory_info"] else 0
            cmd   = " ".join(info["cmdline"] or [])[:120]
            entry = {"pid": info["pid"], "name": name, "user": info["username"] or "?",
                     "status": info["status"] or "?", "age_h": age_h,
                     "idle_h": ih, "mem_mb": mem, "cmd": cmd}
            if ih    > IDLE_THRESHOLD_HOURS: idle.append(entry)
            if age_h > LONG_RUNNING_HOURS:   long_r.append(entry)
        except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess): pass
    idle.sort(key=lambda x: x["idle_h"], reverse=True)
    long_r.sort(key=lambda x: x["age_h"], reverse=True)
    return idle, long_r

def show_process_table(title, procs, color="cyan"):
    if not procs:
        console.print(f"[green]  None found for: {title}[/green]"); return
    t = Table(title=title, box=box.ROUNDED, border_style=color, show_lines=True)
    for col, st, w in [("PID","bold white",8),("Name","cyan",18),("User","yellow",12),
                        ("Status","white",10),("Age(h)","magenta",8),("Idle(h)","red",8),
                        ("MemMB","green",8),("Command","dim",38)]:
        t.add_column(col, style=st, width=w)
    for p in procs[:50]:
        t.add_row(str(p["pid"]),p["name"][:18],p["user"][:12],p["status"],
                  str(p["age_h"]),str(p["idle_h"]),str(p["mem_mb"]),p["cmd"][:38])
    console.print(t)

def kill_processes(procs, dry_run, interactive, label):
    if not procs: return
    if interactive and not dry_run:
        console.print(f"\n[bold yellow]Found {len(procs)} {label} processes.[/bold yellow]")
        pids_input = Prompt.ask("PIDs to kill (comma-sep / all / none)", default="none")
        if pids_input.strip().lower() == "none":
            console.print("[dim]  Skipped.[/dim]"); return
        elif pids_input.strip().lower() == "all":
            targets = procs
        else:
            try:
                chosen  = {int(x.strip()) for x in pids_input.split(",")}
                targets = [p for p in procs if p["pid"] in chosen]
            except ValueError:
                console.print("[red]  Invalid -- skipping.[/red]"); return
    else:
        targets = procs

    killed = 0
    for p in targets:
        if dry_run:
            console.print(f"  [DRY-RUN] Would kill PID {p['pid']} ({p['name']})")
            record_unprocessed("process", str(p["pid"]), "dry-run"); continue
        try:
            os.kill(p["pid"], signal.SIGTERM)
            record_deletion("process", str(p["pid"]), {"name": p["name"]})
            STATS["processes_killed"] += 1; killed += 1
            console.print(f"  [green]  SIGTERM -> PID {p['pid']} ({p['name']})[/green]")
        except (ProcessLookupError, PermissionError) as e:
            console.print(f"  [red]  PID {p['pid']} -- {e}[/red]")
            record_unprocessed("process", str(p["pid"]), str(e))
    if not dry_run:
        console.print(f"[bold green]  Killed {killed}/{len(targets)}.[/bold green]")

def process_cleanup(dry_run, interactive):
    console.print(Rule("[bold yellow]PROCESS CLEANUP[/bold yellow]"))
    with console.status("[cyan]Scanning processes...[/cyan]"):
        idle_procs, long_procs = collect_suspicious_processes()
    console.print(f"\n[bold]Idle (>{IDLE_THRESHOLD_HOURS}h zero CPU):[/bold]")
    show_process_table("IDLE PROCESSES", idle_procs, "yellow")
    console.print(f"\n[bold]Long-running (>{LONG_RUNNING_HOURS}h alive):[/bold]")
    show_process_table("LONG-RUNNING PROCESSES", long_procs, "red")
    kill_processes(idle_procs,  dry_run, interactive, "idle")
    kill_processes(long_procs,  dry_run, interactive, "long-running")

# ══════════════════════════════════════════════════════════════════
# SECTION 5 — DISK ANALYSIS
# ══════════════════════════════════════════════════════════════════
def show_disk_analysis():
    console.print(Rule("[bold blue]DISK & EBS ANALYSIS[/bold blue]"))
    t = Table(box=box.ROUNDED, border_style="blue", title="Disk Partitions", show_lines=True)
    for col in ("Mount","Device","FSType","Total","Used","Free","Use%"):
        t.add_column(col, style="red" if col=="Use%" else "white")
    for p in psutil.disk_partitions(all=False):
        try:
            u = psutil.disk_usage(p.mountpoint)
            t.add_row(p.mountpoint, p.device, p.fstype,
                      f"{u.total/1e9:.1f}G", f"{u.used/1e9:.1f}G",
                      f"{u.free/1e9:.1f}G", f"{u.percent:.0f}%")
        except PermissionError: pass
    console.print(t)
    rc, out, _ = run("du -ahx --max-depth=4 /var /tmp /home 2>/dev/null | sort -rh | head -20")
    if out:
        lf = Table(box=box.SIMPLE, show_header=False)
        lf.add_column("size"); lf.add_column("path")
        for line in out.splitlines():
            parts = line.split(None, 1)
            if len(parts) == 2: lf.add_row(f"[yellow]{parts[0]}[/yellow]", parts[1])
        console.print(lf)

# ══════════════════════════════════════════════════════════════════
# SECTION 6 — IAM USER TRACKER
# ══════════════════════════════════════════════════════════════════
def _iam_client():
    try:
        c = boto3.client("iam")
        c.get_account_summary()
        return c
    except Exception:
        return None

def iam_tracker(dry_run):
    console.print(Rule("[bold magenta]IAM USER TRACKER[/bold magenta]"))

    iam = _iam_client()
    if iam is None:
        console.print(Panel(
            "[yellow]No AWS credentials found.\n\n"
            "To enable IAM tracking:\n"
            "  export AWS_ACCESS_KEY_ID=...\n"
            "  export AWS_SECRET_ACCESS_KEY=...\n"
            "  export AWS_DEFAULT_REGION=ap-south-1\n\n"
            "Or attach an IAM role to this EC2 instance.[/yellow]",
            title="AWS Credentials Missing", border_style="yellow")); return

    # Generate + fetch credential report
    with console.status("[magenta]Requesting IAM credential report...[/magenta]"):
        import time
        for _ in range(30):
            resp = iam.generate_credential_report()
            if resp["State"] == "COMPLETE": break
            time.sleep(1)

    csv_text = iam.get_credential_report()["Content"].decode("utf-8")
    rows     = [r.split(",") for r in csv_text.strip().splitlines()]
    headers  = rows[0]
    users_csv= rows[1:]

    def col(row, name):
        try:    return row[headers.index(name)]
        except: return ""

    # Fetch all users, policies, groups
    with console.status("[magenta]Listing users, policies, groups...[/magenta]"):
        all_users = []
        for page in iam.get_paginator("list_users").paginate():
            all_users.extend(page["Users"])

        policy_map = {}
        group_map  = {}
        for u in all_users:
            un = u["UserName"]
            attached = iam.list_attached_user_policies(UserName=un)["AttachedPolicies"]
            inline   = iam.list_user_policies(UserName=un)["PolicyNames"]
            policy_map[un] = [p["PolicyName"] for p in attached] + inline
            group_map[un]  = [g["GroupName"] for g in
                              iam.list_groups_for_user(UserName=un)["Groups"]]

    STATS["iam_users_scanned"] = len(all_users)

    def parse_dt(val):
        if val in ("N/A","no_information","","not_supported"): return None
        try:    return datetime.datetime.fromisoformat(val.replace("Z","+00:00"))
        except: return None

    def key_age(raw):
        dt = parse_dt(raw)
        return days_ago(dt) if dt else None

    enriched = []
    for row in users_csv:
        uname = col(row, "user")
        if uname == "<root_account>": uname = "ROOT"

        pwd_last   = parse_dt(col(row, "password_last_used"))
        k1_last    = parse_dt(col(row, "access_key_1_last_used_date"))
        k2_last    = parse_dt(col(row, "access_key_2_last_used_date"))
        k1_age     = key_age(col(row, "access_key_1_last_rotated"))
        k2_age     = key_age(col(row, "access_key_2_last_rotated"))

        has_console = col(row, "password_enabled") == "true"
        has_mfa     = col(row, "mfa_active") == "true"
        k1_active   = col(row, "access_key_1_active") == "true"
        k2_active   = col(row, "access_key_2_active") == "true"

        pwd_days  = days_ago(pwd_last)
        k1_u_days = days_ago(k1_last)
        k2_u_days = days_ago(k2_last)

        last_active = min(
            (d for d in [pwd_days, k1_u_days, k2_u_days] if d is not None),
            default=None)

        policies = policy_map.get(uname, [])
        groups   = group_map.get(uname,  [])
        is_admin = "AdministratorAccess" in policies

        flags = []
        if has_console and not has_mfa:
            flags.append("NO_MFA"); STATS["iam_no_mfa"] += 1
        if last_active is not None and last_active > INACTIVE_DAYS:
            flags.append(f"INACTIVE_{last_active}d"); STATS["iam_inactive_users"] += 1
        if last_active is None and has_console:
            flags.append("NEVER_LOGGED_IN")
        if k1_active and k1_age is not None:
            if k1_age > KEY_CRIT_DAYS:
                flags.append(f"KEY1_CRITICAL_{k1_age}d"); STATS["iam_keys_flagged"] += 1
            elif k1_age > KEY_WARN_DAYS:
                flags.append(f"KEY1_WARN_{k1_age}d"); STATS["iam_keys_flagged"] += 1
        if k2_active and k2_age is not None:
            if k2_age > KEY_CRIT_DAYS:
                flags.append(f"KEY2_CRITICAL_{k2_age}d"); STATS["iam_keys_flagged"] += 1
            elif k2_age > KEY_WARN_DAYS:
                flags.append(f"KEY2_WARN_{k2_age}d"); STATS["iam_keys_flagged"] += 1
        if is_admin:
            flags.append("ADMIN_ACCESS")

        enriched.append({
            "user": uname, "console": has_console, "mfa": has_mfa,
            "k1_active": k1_active, "k2_active": k2_active,
            "k1_age": k1_age, "k2_age": k2_age,
            "last_active_d": last_active, "pwd_days": pwd_days,
            "policies": policies, "groups": groups,
            "is_admin": is_admin, "flags": flags,
        })

    # ── Table 1: All Users Overview ──────────────────────────────
    t_all = Table(title="IAM Users Overview", box=box.ROUNDED,
                  border_style="magenta", show_lines=True)
    t_all.add_column("User",         style="bold cyan",  width=20)
    t_all.add_column("Console",      style="white",      width=9)
    t_all.add_column("MFA",          style="white",      width=6)
    t_all.add_column("Keys",         style="white",      width=8)
    t_all.add_column("Last Active",  style="white",      width=13)
    t_all.add_column("K1 Age(d)",    style="yellow",     width=11)
    t_all.add_column("K2 Age(d)",    style="yellow",     width=11)
    t_all.add_column("Admin",        style="red",        width=7)
    t_all.add_column("Groups",       style="dim",        width=22)
    t_all.add_column("Flags",        style="red",        width=32)

    for u in enriched:
        mfa_s    = "[green]YES[/green]" if u["mfa"]     else "[red]NO[/red]"
        con_s    = "[green]YES[/green]" if u["console"] else "[dim]no[/dim]"
        keys_s   = f"{'K1 ' if u['k1_active'] else ''}{'K2' if u['k2_active'] else ''}".strip() or "[dim]none[/dim]"
        act_s    = (f"{u['last_active_d']}d ago" if u["last_active_d"] is not None
                    else "[dim]never[/dim]")
        admin_s  = "[bold red]YES[/bold red]" if u["is_admin"] else "[dim]no[/dim]"
        groups_s = ", ".join(u["groups"][:3]) or "[dim]–[/dim]"
        flags_s  = " ".join(f"[red]{f}[/red]" for f in u["flags"]) or "[green]OK[/green]"
        t_all.add_row(u["user"], con_s, mfa_s, keys_s, act_s,
                      str(u["k1_age"]) if u["k1_age"] is not None else "–",
                      str(u["k2_age"]) if u["k2_age"] is not None else "–",
                      admin_s, groups_s, flags_s)
    console.print(t_all)

    # ── Table 2: Flagged / Risky Users ───────────────────────────
    risky = [u for u in enriched if u["flags"]]
    if risky:
        console.print(f"\n[bold red]  {len(risky)} users have security flags:[/bold red]")
        t_risk = Table(title="FLAGGED USERS", box=box.ROUNDED,
                       border_style="red", show_lines=True)
        t_risk.add_column("User",           style="bold cyan", width=20)
        t_risk.add_column("Flags",          style="red",       width=36)
        t_risk.add_column("Policies",       style="yellow",    width=32)
        t_risk.add_column("Recommendation", style="green",     width=38)

        for u in risky:
            recs = []
            if "NO_MFA"        in u["flags"]: recs.append("Enable MFA immediately")
            if any("INACTIVE"  in f for f in u["flags"]): recs.append("Disable or delete user")
            if any("CRITICAL"  in f for f in u["flags"]): recs.append("Rotate access keys NOW")
            if any("WARN"      in f for f in u["flags"]): recs.append("Schedule key rotation")
            if "ADMIN_ACCESS"  in u["flags"]: recs.append("Audit admin necessity")
            if "NEVER_LOGGED_IN" in u["flags"]: recs.append("Remove console access")
            t_risk.add_row(u["user"],
                           "\n".join(u["flags"]),
                           "\n".join(u["policies"][:5]) or "–",
                           "\n".join(recs))
        console.print(t_risk)
    else:
        console.print("[green]  No security flags found.[/green]")

    # ── Table 3: Access Keys Age ─────────────────────────────────
    key_users = [u for u in enriched if u["k1_active"] or u["k2_active"]]
    if key_users:
        t_keys = Table(title="Active Access Keys", box=box.ROUNDED,
                       border_style="yellow", show_lines=True)
        t_keys.add_column("User",       style="bold cyan", width=20)
        t_keys.add_column("Key",        style="white",     width=5)
        t_keys.add_column("Age (days)", style="yellow",    width=12)
        t_keys.add_column("Status",     style="white",     width=28)

        for u in key_users:
            for key_n, active, age in [("KEY1",u["k1_active"],u["k1_age"]),
                                        ("KEY2",u["k2_active"],u["k2_age"])]:
                if not active: continue
                if age and age > KEY_CRIT_DAYS:
                    status = "[bold red]CRITICAL -- rotate immediately[/bold red]"
                elif age and age > KEY_WARN_DAYS:
                    status = "[yellow]WARNING -- rotate soon[/yellow]"
                else:
                    status = "[green]OK[/green]"
                t_keys.add_row(u["user"], key_n, str(age) if age else "?", status)
        console.print(t_keys)

    # ── IAM Summary Panel ─────────────────────────────────────────
    console.print(Panel(
        "\n".join([
            f"[bold cyan]Total users scanned:[/bold cyan]      {len(enriched)}",
            f"[bold cyan]Console access users:[/bold cyan]     {sum(1 for u in enriched if u['console'])}",
            f"[bold cyan]Users WITHOUT MFA:[/bold cyan]        [bold red]{STATS['iam_no_mfa']}[/bold red]",
            f"[bold cyan]Inactive (>{INACTIVE_DAYS}d):[/bold cyan]         [bold yellow]{STATS['iam_inactive_users']}[/bold yellow]",
            f"[bold cyan]Access keys flagged:[/bold cyan]      [bold yellow]{STATS['iam_keys_flagged']}[/bold yellow]",
            f"[bold cyan]Admin users:[/bold cyan]              [bold red]{sum(1 for u in enriched if u['is_admin'])}[/bold red]",
            f"[bold cyan]Total flagged users:[/bold cyan]      [bold red]{len(risky)}[/bold red]",
            "",
            f"[dim]Full JSON report -> {IAM_REPORT}[/dim]",
        ]),
        title="IAM SECURITY SUMMARY", border_style="magenta"))

    # ── Save JSON report ──────────────────────────────────────────
    try:
        account_id = boto3.client("sts").get_caller_identity().get("Account","?")
    except Exception:
        account_id = "?"

    with open(IAM_REPORT, "w") as f:
        json.dump({"generated": _now_iso(), "account_id": account_id,
                   "users": enriched,
                   "stats": {"total": len(enriched), "no_mfa": STATS["iam_no_mfa"],
                              "inactive": STATS["iam_inactive_users"],
                              "keys_flagged": STATS["iam_keys_flagged"],
                              "admin_users": sum(1 for u in enriched if u["is_admin"]),
                              "flagged": len(risky)}},
                  f, indent=2, default=str)

    console.print(f"[green]  IAM report saved -> {IAM_REPORT}[/green]")
    log_event("IAM_SCAN", {"users": len(enriched), "flagged": len(risky)})

# ══════════════════════════════════════════════════════════════════
# SECTION 7 — FINAL SUMMARY
# ══════════════════════════════════════════════════════════════════
def show_summary(dry_run, elapsed):
    mode_tag = "[bold red]  DRY-RUN[/bold red]" if dry_run else "[bold green]  PRODUCTION[/bold green]"
    t = Table(box=box.DOUBLE_EDGE, border_style="green", title="CLEANUP SUMMARY", show_lines=True)
    t.add_column("Metric",        style="bold cyan",  width=36)
    t.add_column("Count / Value", style="bold white", width=26)
    for k, v in [
        ("Mode",                         mode_tag),
        ("Docker containers killed",     str(STATS["docker_containers_killed"])),
        ("Docker images removed",        str(STATS["docker_images_removed"])),
        ("Docker networks removed",      str(STATS["docker_networks_removed"])),
        ("Docker volumes removed",       str(STATS["docker_volumes_removed"])),
        ("Processes killed",             str(STATS["processes_killed"])),
        ("IAM users scanned",            str(STATS["iam_users_scanned"])),
        ("IAM users without MFA",        str(STATS["iam_no_mfa"])),
        ("IAM inactive users",           str(STATS["iam_inactive_users"])),
        ("IAM access keys flagged",      str(STATS["iam_keys_flagged"])),
        ("Total duration",               f"{elapsed:.1f}s"),
        ("Audit log",                    str(AUDIT_LOG)),
        ("IAM report",                   str(IAM_REPORT)),
    ]: t.add_row(k, v)
    console.print(t)
    log_event("SUMMARY", {**STATS, "duration_s": elapsed, "dry_run": dry_run})

# ══════════════════════════════════════════════════════════════════
# INTERACTIVE MENU
# ══════════════════════════════════════════════════════════════════
MENU_ITEMS = [
    ("1", "Full Auto    (Docker + Processes + Disk + IAM)",   "auto"),
    ("2", "Docker Only  (containers + images + nets + vols)", "docker"),
    ("3", "Processes    (idle + long-running)",               "procs"),
    ("4", "Disk         (EBS + large files)",                 "disk"),
    ("5", "IAM Tracker  (users, keys, MFA, inactive)",        "iam"),
    ("6", "Resources    (CPU / RAM / disk live)",             "resources"),
    ("7", "Exit",                                             "exit"),
]
MENU_ACTION = {k: v for k, _, v in MENU_ITEMS}
MENU_COLORS = ["yellow","cyan","cyan","cyan","magenta","cyan","red"]

def interactive_menu(dry_run):
    mode_label = "[bold red]DRY-RUN[/bold red]" if dry_run else "[bold green]PRODUCTION[/bold green]"
    body = f"Mode: {mode_label}\n\n" + "\n".join(
        f"  [{c}]{k}[/{c}]  {label}"
        for (k, label, _), c in zip(MENU_ITEMS, MENU_COLORS))
    console.print(Panel(body, title="[bold]ENTERPRISE CLEANUP MENU[/bold]", border_style="blue"))
    return MENU_ACTION[Prompt.ask("Select", choices=[k for k,_,_ in MENU_ITEMS], default="1")]

# ══════════════════════════════════════════════════════════════════
# MAIN
# ══════════════════════════════════════════════════════════════════
def main():
    # Declare ALL globals FIRST before any usage
    global IDLE_THRESHOLD_HOURS, LONG_RUNNING_HOURS
    global INACTIVE_DAYS, KEY_WARN_DAYS, KEY_CRIT_DAYS

    parser = argparse.ArgumentParser(
        description="Enterprise Cleanup + IAM Tracker v4.0",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=textwrap.dedent("""
        Examples:
          sudo python3 prod_cleanup.py --dry-run
          sudo python3 prod_cleanup.py --production
          sudo python3 prod_cleanup.py --production --auto
          sudo python3 prod_cleanup.py --iam-only
          sudo python3 prod_cleanup.py --production --inactive-days 60 --key-crit 120
        """))

    mode_grp = parser.add_mutually_exclusive_group()
    mode_grp.add_argument("--dry-run",       action="store_true")
    mode_grp.add_argument("--production",    action="store_true")
    parser.add_argument("--auto",            action="store_true", help="No prompts (CI/CD)")
    parser.add_argument("--docker-only",     action="store_true")
    parser.add_argument("--procs-only",      action="store_true")
    parser.add_argument("--disk-only",       action="store_true")
    parser.add_argument("--iam-only",        action="store_true")
    parser.add_argument("--idle-hours",      type=float, default=IDLE_THRESHOLD_HOURS)
    parser.add_argument("--long-hours",      type=float, default=LONG_RUNNING_HOURS)
    parser.add_argument("--inactive-days",   type=int,   default=INACTIVE_DAYS,
                        help="Flag IAM users inactive > N days (default 90)")
    parser.add_argument("--key-warn",        type=int,   default=KEY_WARN_DAYS,
                        help="Access key warning age days (default 90)")
    parser.add_argument("--key-crit",        type=int,   default=KEY_CRIT_DAYS,
                        help="Access key critical age days (default 180)")

    args = parser.parse_args()

    IDLE_THRESHOLD_HOURS = args.idle_hours
    LONG_RUNNING_HOURS   = args.long_hours
    INACTIVE_DAYS        = args.inactive_days
    KEY_WARN_DAYS        = args.key_warn
    KEY_CRIT_DAYS        = args.key_crit

    dry_run     = not args.production
    interactive = not args.auto

    console.print(Panel(
        "[bold yellow]  DRY-RUN MODE -- no changes will be made.[/bold yellow]"
        if dry_run else
        "[bold red]  PRODUCTION MODE -- destructive operations WILL execute.[/bold red]",
        border_style="yellow" if dry_run else "red"))

    show_identity()
    show_resource_usage()

    if args.iam_only:
        iam_tracker(dry_run)
    elif args.docker_only:
        docker_cleanup(dry_run, interactive)
    elif args.procs_only:
        process_cleanup(dry_run, interactive)
    elif args.disk_only:
        show_disk_analysis()
    elif args.auto:
        docker_cleanup(dry_run, interactive)
        process_cleanup(dry_run, interactive)
        show_disk_analysis()
        iam_tracker(dry_run)
    else:
        while True:
            action = interactive_menu(dry_run)
            if action == "exit": break
            elif action == "auto":
                docker_cleanup(dry_run, interactive)
                process_cleanup(dry_run, interactive)
                show_disk_analysis()
                iam_tracker(dry_run)
            elif action == "docker":    docker_cleanup(dry_run, interactive)
            elif action == "procs":     process_cleanup(dry_run, interactive)
            elif action == "disk":      show_disk_analysis()
            elif action == "iam":       iam_tracker(dry_run)
            elif action == "resources": show_resource_usage()
            if not Confirm.ask("\n[cyan]Return to menu?[/cyan]", default=True): break

    elapsed = (datetime.datetime.now(datetime.timezone.utc) - SCRIPT_START).total_seconds()
    flush_json_logs()
    show_summary(dry_run, elapsed)
    console.print(Panel(
        f"[bold green]  Completed![/bold green]  Logs -> [cyan]{LOG_DIR}[/cyan]",
        border_style="green"))


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        console.print("\n[yellow]  Interrupted. Flushing logs...[/yellow]")
        flush_json_logs()
        elapsed = (datetime.datetime.now(datetime.timezone.utc) - SCRIPT_START).total_seconds()
        show_summary(dry_run=True, elapsed=elapsed)
        sys.exit(130)
    except Exception:
        console.print_exception()
        flush_json_logs()
        sys.exit(1)

Over the weekend, I explored R&D AI coding agents and learned about AI security tools like Numbat and Bumblebee. I learned how to scan MCP servers, configurations, and extensions for potential security risks. My goal was to understand how to make AI-assisted development more secure.

HOW I DEBUGGED IT

I installed and tested two open-source security tools: Bumblebee and Numbat on an Ubuntu EC2 instance.

● Bumblebee  Scanned packages, MCP configurations, and developer extensions.

●  Numbat  Scanned supported AI agent artifacts for potential security issues.

THE SOLUTION

I used Bumblebee to check packages, MCP configurations, and developer extensions. I used Numbat to discover and analyze supported AI agent artifacts.

Testing both tools on Ubuntu helped me understand how to identify potential security risks in AI development environments.

EC2 → Bumblebee & Numbat → Scan → Identify Risks → Secure AI Development

Has anyone else faced this problem?

How did you fix it? What tools do you use to secure AI coding agents?

GitHub : https://github.com/ksaivishnusaikvs/DevOps-Engineering/tree/main/Linkedin/Post

Bumblebee is also open source and free to run. It's a read-only security scanner for developer endpoints, including packages, MCP configs, editor extensions, and browser extensions.  https://github.com/perplexityai/bumblebee

Step1: Install prerequisites: Bumblebee requires Go 1.25+ and is a read-only scanner. GitHub

sudo apt update; sudo apt install -y git; go version

Step 2: Install Bumblebee

go install github.com/perplexityai/bumblebee/cmd/bumblebee@latest

Add it to PATH:

echo 'export PATH="\(HOME/go/bin:\)PATH"' >> ~/.bashrc

source ~/.bashrc

Verify: bumblebee version

Step 3: Run the self-test

bumblebee selftest

Your result:

selftest OK

means the installation is working.

Step 4: Baseline scan

This is the recommended lightweight inventory scan:

bumblebee scan --profile baseline > ~/bumblebee-baseline.ndjson

Check the final summary:

grep '"record_type":"scan_summary"' ~/bumblebee-baseline.ndjson

Bumblebee's output is NDJSON. Diagnostics are written separately to stderr. GitHub

Step5. Deep scan

For an on-demand scan of specific directories:

bumblebee scan \

  --profile deep \

  --root /home/ubuntu \

  --root /opt \

  --root /srv \

  > ~/bumblebee-deep.ndjson

deep is specifically designed for explicit --root paths.

Numbat  https://github.com/perplexityai/numbat 

is the more interesting one for a DevOps/security project: it's a single binary for Linux, macOS, and Windows, and provides AI-agent monitoring, local detection, optional blocking, and forensic reconstruction. Its GitHub repo is Apache 2.0.

 Step 1: numbat agents --all. Select one agent based on the scan

Step2: log in to the Gemini CLI command gemini shown below

Step3 : Click on the Sign in with Google area; select based on your options. I selected 1 option: Sign in with Google

Click Sign in it shows the URL in the web, then we can log in

Step3 : Test Numbat in the first CLI; it does not show Image 1: no agent artifacts to analyze in that dir. Image 2: Numbat Agent Scan: Numbat successfully discovered and parsed 6 artifacts with 0 findings.

Image 1: Gemini CLI Integration: Numbat scanned the DevOps directory but found no agent artifacts to analyze. The scan returned an error because it found no supported artifacts.

Image 2: Numbat Agent Scan: Numbat successfully discovered and parsed 6 artifacts with 0 findings. The scan completed successfully, confirming that artifact discovery and parsing worked in this run.

full scrreen shot

THE PROBLEM

I wanted an easier way to see AWS costs without opening the AWS console and checking the report manually every day.

WHAT I WANTED

●       See AWS costs by service. Send the report directly to Slack

●       Make the information easy to read. Reduce repetitive manual work

HOW I DEBUGGED IT

I started by generating the AWS cost report and checking the output. Then I tested the Slack setup, permissions, and channel configuration step by step until the report was reaching Slack correctly.

THE SOLUTION

I connected AWS Cost Explorer → AWS Cost CLI → Python → Slack and tested the complete flow. After that, I added scheduled automation so the report could be generated and sent automatically.     Copy of AWS cost Report to Slack     Copy of AWS cost Report to Slack

THE WORKFLOW

AWS Cost → Report → Python → Slack → Automation

Has anyone else faced this problem?

How do you share AWS costs without giving everyone Console access?

How did you fix it?

HashnodProfile:https://hashnode.com/@vishnusaiK
GitHub : https://github.com/ksaivishnusaikvs/DevOps-Engineering/tree/main/Linkedin/Post

➢    AWS cost notification to Slack channel

To check clearly: select the 3 dots, click on View raw

STEP1: Set up an EC2 instance first. Launch the EC2 instance in an AWS region, run apt-get update, and update the packages

Install Node.js from https://nodejs.org/en/download, select os and use it. In my case

Installation looks like this.

Step2: Installation. Install the package globally, or you can also use npx  
npm install -g aws-cost-cli 
  (Or)

DockerYou can build this Dockerfile using the docker build command and then run a container from the built image. For example:

docker build -t aws-cost-cli.  
docker run aws-cost-cli

Install the command we need to run

aws-cost: This will show the AWS billing cost

●      aws-cost --help, it will help you find commands

How to integrate with Slack

Step1: Install the AWS CLI in your instance

# Install AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

sudo apt-get install unzip -y

unzip awscliv2.zip

sudo ./aws/install

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Try the command

AWS configure or IAM roles.

Step2: Go to Slack —> first we need to create a workspace, then create the channel  → click on Next

Step3: Give the channel a name  → select the visibility (public or private best practices

Click on Create the channel • click Create

Step4:

Go to https://api.slack.com/apps → click on the new app →select from scratch

Step 5: Give your app a name and select the workspace in Slack. If no workspace is created, create a workspace.

Once it is created, it looks like this.

Step6:

The image text describes how to set Slack app permissions in the app dashboard:

Go to "OAuth & Permissions".

Scroll to "Scopes".

Under "Bot Token Scopes", add:

chat. write → Allows the bot to post messages.

chat. write. public → (Optional) Allows posting in public channels without being invited.

Once they add all

Step 7: Install the App in Your Workspace

1. Still in OAuth & Permissions, click Install App to Workspace.

2. Approve the permissions.

3. Copy the Bot User OAuth Token:

○ Starts with xoxb-... → You'll need this for aws-cost-cli. Approve the permissions.

Approve the permissions.

Then we need to see the token; it starts with xoxb-... → You'll need this for aws-cost-cli.

Step8: Find Your Slack Channel ID

1. Open Slack.

2. Go to the channel you want to post in.

3. Click on the channel name at the top → View channel details.

4. Copy the Channel ID (starts with C...).

Step9: Test the Slack integration with aws-cost-cli

aws-cost --slack-token xoxb-884434558977825-12ZAU3Pl9zCCkJyeQH5oK8tO --slack-channel C08QKDNCW59

Change the token and channel name.

Verify that we received the message on Slack.

Note: If your Slack invitation fails to send a message to the Slack channel not found, we need to invert the process for the first time (or) failed Slack not found

Step 9.1: /invite @ your app name that shows in the image

Run the command

aws-cost --slack-token xoxb-8844374581409-8844558977825-12ZAU3Pl9zCCkJyeQH5oK8tO --slack-channel C08QKDNCW59

Now we get a message on Slack

This output is not clear format. We need to print the file format.

Step10: /invite @AWS-COST-Report

aws-cost --text > cost-report.txt && \

curl -F file=@cost-report.txt \

-F "initial_comment=AWS Cost Report" \

-F channels=C08PFVZV9PF \

-H "Authorization: Bearer xoxb-6800278956247-8805347127382-SQlckF4aKrzil8ODyqARQRA" \

https://slack.com/api/files.upload

This command generates an AWS cost report, saves it to a text file, and then uploads it to a Slack channel using Slack's API.

Step10.1: We need to print in text format. We need to add the go for the https://api.slack.com/apps/A08QYJ503A6/oauth?

We need to print the file format in Slack. We add on the go→ scope → add the file write.

After adding the file, write in the scope we need to reinstall the app, click on the reinstall app in top.

Click on Allow

Step 11: Install Python & pip (if not already) sudo apt update sudo apt install -y python3 python3-pip python3-venv 2. Create a Virtual Environment (Optional but Recommended) python3 -m venv venv source venv/bin/activate

Step12: We install pip install slack-sdk

1. Create Python Script (upload_cost_report.py):

from slack_sdk import WebClient from slack_sdk.errors import SlackApiError

Your Slack Bot Token

slack_token = "xoxb-6800278956247-DyqARQRA" client = WebClient(token=slack_token)

try: # Upload the file response = client.files_upload( channels="C08PFVZV9PF", file="cost-report.txt", initial_comment="AWS Cost Report" ) print("File uploaded successfully:", response)

except SlackApiError as e: print(f"Error uploading file: {e.response['error']}")

from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

# Your Slack Bot Token
slack_token = "xoxb-6800278956247-DyqARQRA"
client = WebClient(token=slack_token)

try:
    # Upload the file
    response = client.files_upload(
        channels="C08PFVZV9PF",
        file="cost-report.txt",
        initial_comment="AWS Cost Report"
    )
    print("File uploaded successfully:", response)

except SlackApiError as e:
    print(f"Error uploading file: {e.response['error']}")

Step12: creat file vi aws_cost_slack.py, add script  give the executed permissions

Run the AWS Cost CLI + Python Script:

Generate cost report aws-cost --text > cost-report.txt

Every time we run the command, we run the script; it gets a new report to text.

Upload to Slack

python3 upload_cost_report.py

After the run, we receive the notification in the channel.

To check clearly: select the 3 dots, click on the view ra.w

Option 1: Using cron (on Linux/Ubuntu EC2) Step-by-step: Open crontab:

crontab -e Add a cron job to run the script every day at 8 AM (or your preferred time):

0 8 * * * /usr/bin/aws-cost --text > /root/cost-report.txt && /usr/bin/python3 /root/aws_cost_slack.py ○ Adjust paths if aws-cost or python3 are in different locations.

○ Use which aws-cost and which python3 to find full paths.

Option 2: Using a Shell Script + cron

Create a shell script: aws_daily_report.sh:

#!/bin/bash /usr/bin/aws-cost --text > /root/cost-report.txt /usr/bin/python3 /root/aws_cost_slack.py

1. Make it executable:

chmod +x aws_daily_report.sh Schedule it via cron:

crontab -e

0 8 * * * /root/aws_daily_report.sh

**Schedule it via cron:
**crontab -e
0 8 * * * /root/aws_daily_report.sh

Option 3: AWS Lambda + CloudWatch (serverless, scalable)

Use AWS Lambda to:

●      Generate the report

●      Upload it to Slack

●      Schedule with CloudWatch Events

This is better for multi-user orgs or 100+ team-wide notifications.

If you're interested, I can generate an end-to-end Lambda version of your script.

One-Time Setup

You first run this one-time command to get daily AWS cost data grouped by service for a past month (April 2024):

aws ce get-cost-and-usage \

--time-period Start=2024-04-01,End=2024-04-30 \

--granularity DAILY \

--metrics "UnblendedCost" \

--group-by Type=SERVICE \

--output table

This is a historical report.

Breakdown alias aws-cost="..."

Creates a command shortcut. After this, you can just type aws-cost in your terminal, and it will run the entire AWS CLI command.

aws ce get-cost-and-usage

This is the AWS CLI command for Cost Explorer, used to fetch billing and usage data.

●       ce = Cost Explorer

●       get-cost-and-usage = Retrieves AWS cost data

--time-period Start=...,End=...

Specifies the date range of the cost report.

Start=$(date -d 'yesterday' +%Y-%m-%d)

End=$(date +%Y-%m-%d)

●       Start = yesterday's date (e.g., 2025-05-23)

●       EEnd= today's date (e.g., 2025-05-24)

This means: Get costs for yesterday.

--granularity DAILY

This tells AWS to give you cost data broken down per day. Since you're only asking for one day (yesterday), it gives the cost just for that one day.

🔹 --metrics UnblendedCost

Specifies the cost metric you want:

●       UnblendedCost = Actual cost before discounts or credits are applied.

Other options (not used here): BlendedCost, UsageQuantity.

🔹 --group-by Type=SERVICE

This groups the output by AWS service, so you can see how much each service (like EC2, S3, etc.) costs on that day.

🔹 --output table

This formats the output in a readable text table format.
 Alternatives:

●       json (for scripting)

●       text

●       csv

Example Output:

When you run:

aws-cost

+--------------------------------------+-----------+

| Service                              | Amount    |

+--------------------------------------+-----------+

| Amazon Elastic Compute Cloud - EC2   | $1.11     |

| Amazon S3                            | $0.04     |

+--------------------------------------+-----------+

Summary

Dynamic command

alias aws-cost="aws ce get-cost-and-usage \

--time-period Start=\((date -d 'yesterday' +%Y-%m-%d),End=\)(date +%Y-%m-%d) \

--granularity DAILY \

--metrics UnblendedCost \

--group-by Type=SERVICE \

--output table"

It is dynamic, and here’s what it does:

●       **Start=$(date -d 'yesterday' +%Y-%m-%d)
** Gets yesterday’s date in the format YYYY-MM-DD.

●       **End=$(date +%Y-%m-%d)
** Gets today’s date, so the report ends at midnight today.

●       **--granularity DAILY
** Ensures the data is broken down by day.

●       **--group-by Type=SERVICE
** Breaks down cost by AWS service (as shown in your screenshot).

●       **--output table
** Formats the output in a readable text table (like your screenshot).

Result

Each time you run:

aws-cost > cost-report.txt

It will dynamically generate a yesterday-only report for all AWS services and store it in cost-report.txt, in the same table format shown in your screenshot.

Example Output (based on your screenshot):

Each day, you'll get something like:

nginx

Service                          Last Month  This Month  Last 7 Days  Yesterday

Amazon EC2                      $XX.XX      $XX.XX      $X.XX        $X.XX

Amazon S3                       $0.27       $0.05       $0.04        $0.04

Summary

●        Yes, your AWS cost alias will dynamically generate this report every day for yesterday.

●       The output is formatted as a readable text table.

●        You can automate or script it for Slack, email, or log files.

Let me know if you want to:

●       Convert it to JSON/CSV for scripting.

●       Filter to only non-zero services.

●       Format it for Slack better.

Summary

●        Yes, your AWS cost alias will dynamically generate this report every day for yesterday.

●       The output is formatted as a readable text table.

●        You can automate or script it for Slack, email, or log files.

Let me know if you want to:

●       Convert it to JSON/CSV for scripting.

●       Filter to only non-zero services.

●       Format it for Slack better.

Step 1: Modify Your AWS-Cost Alias to Output JSON

You’ll use JSON output (easier to process programmatically):

alias aws-cost="aws ce get-cost-and-usage \

--time-period Start=\((date -d 'yesterday' +%Y-%m-%d),End=\)(date +%Y-%m-%d) \

--granularity DAILY \

--metrics UnblendedCost \

--group-by Type=SERVICE \

--output json"

Add this alias to your ~/.bashrc or ~/.zshrc, then run source ~/.bashrc (or equivalent shell).

Step 2: Run the Alias & Save Output

aws-cost > cost-report.json

Modified Command for CSV Output:

alias aws-cost="aws ce get-cost-and-usage \

--time-period Start=\((date -d 'yesterday' +%Y-%m-%d),End=\)(date +%Y-%m-%d) \

--granularity DAILY \

--metrics UnblendedCost \

--group-by Type=SERVICE \

--output text | tr '\t' ',' > cost-report.csv"

How This Works:

--output text

This produces output in plain tab-delimited text (not real CSV yet).

tr '\t' ','

This converts tabs to commas, making it CSV-compatible.

> cost-report.csv

Redirects the output into a file.

Result:

When you run: aws-cos

It will generate a file cost-report.csv with a format like:

yaml

CopyEdit

2025-05-23, Amazon Elastic Compute Cloud - Compute,1.15, USD

2025-05-23,Amazon S3,0.04,USD

Notes:

●   AWS CLI does not directly support --output csv.

●   Using --output text | tr '\t' ',' is the most reliable workaround.

●   If you need proper headers or processing, use Python to post-process the file.

Great! Here's a complete step-by-step guide to:

1. Get AWS cost data dynamically in JSON or CSV

2. Filter for non-zero cost services

3. Format it nicely for Slack messages

4. Automate if needed

Step 1: Modify Your AWS-Cost Alias to Output JSON You’ll use JSON output (easier to process programmatically): bash CopyEdit alias aws-cost="aws ce get-cost-and-usage
--time-period Start=\((date -d 'yesterday' +%Y-%m-%d),End=\)(date +%Y-%m-%d)
--granularity DAILY
--metrics UnblendedCost
--group-by Type=SERVICE
--output json" Add this alias to your ~/.bashrc or ~/.zshrc, then run source ~/.bashrc (or equivalent shell).

Step 2: Run the Alias & Save Output

aws-cost > cost-report.json.

Step 3: Python Script to Convert JSON → Filtered Slack Message

import json
import requests

# Load cost report
with open('cost-report.json') as f:
    data = json.load(f)

# Extract relevant data
results = data['ResultsByTime'][0]['Groups']
date_range = data['ResultsByTime'][0]['TimePeriod']
start = date_range['Start']
end = date_range['End']

non_zero_services = []
for group in results:
    service = group['Keys'][0]
    amount = float(group['Metrics']['UnblendedCost']['Amount'])
    if amount > 0:
        non_zero_services.append((service, amount))

# Format for Slack
if not non_zero_services:
    message = f"*AWS Cost Report for {start} to {end}*\nNo charges yesterday. 🎉"
else:
    message = f"*AWS Cost Report for {start} to {end}*"
    for service, amount in sorted(non_zero_services, key=lambda x: x[1], reverse=True):
        message += f"\n• `{service}`: *${amount:.2f}*"

# Send to Slack
SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/your/webhook/url"

payload = {"text": message}

response = requests.post(SLACK_WEBHOOK_URL, json=payload)
print("Slack Response:", response.status_code, response.text)

Step 4: Add to Cron (Optional Automation)

Run daily at 8 AM:

crontab -e

Add this line:

0 8 * * * /bin/bash -c 'source ~/.bashrc && aws-cost > ~/cost-report.json && python3 ~/aws_cost_slack.py'

Optional: CSV Export (if needed)

You can also convert cost-report.json to CSV using Python:

import csv

with open('cost-report.json') as f:

data = json.load(f)

results = data['ResultsByTime'][0]['Groups']

with open('cost-report.csv', 'w', newline='') as csvfile:

writer = csv.writer(csvfile)

writer.writerow(['Service', 'Amount'])

for group in results:

service = group['Keys'][0]

amount = float(group['Metrics']['UnblendedCost']['Amount'])

if amount > 0:

writer.writerow([service, f"${amount:.2f}"])

HOW I DEBUGGED IT
I started by checking CPU, memory, disk usage, EBS volumes, large files, and running processes before taking any action.

THE SOLUTION
I created a Python-based EC2 Cleanup & Monitoring script with:

• Auto Mode – performs predefined cleanup tasks

• Interactive Mode – reviews actions before applying them

• JSON Logging – records cleanup activities

MY APPROACH

Check → Review → Approve → Clean → Log

This keeps the cleanup process controlled and makes it easier to understand what happened.

Has anyone else faced a similar EC2 cleanup problem?
How did you handle it?
How did you fix it?

HashnodProfile:https://hashnode.com/@vishnusaiK 
GitHubProfile : https://github.com/ksaivishnusaikvs/DevOps-Engineering/tree/main/Linkedin/Post

Step1: Check System Health   Run the cleanup script  Select execution mode ./cleanup.py

1 → Auto Mode

2 → Interactive Mode    Interactive Mode is recommended for production because it asks for approval before critical actions

Step2: Check High-Resource Processes

Next, identify the processes consuming the most system resources. The script displays the Top 20 CPU-consuming processes and Top 20 memory-consuming processes, including the PID, command, CPU percentage, and memory percentage

Step3: Review EBS Volumes

After checking the EC2 instance, review the attached EBS volumes. The script displays important information such as the Volume ID, volume state, size, and Availability Zone.

Next, identify large files/directories and review processes or files before killing processes or deleting unnecessary files.

Enter the required folder path, and the script scans the path and displays the top storage-consuming files and directories

Review Processes – Approve PIDs before killing them.

Review Files – Approve files before deleting them.

Step4: Check Cleanup Logs

All important cleanup activities are recorded in JSON log format. The logs capture details such as the Linux user, sudo user, Linux groups, EUID, IAM role, event, timestamp, and action details, providing an audit trail of the cleanup activity

Check logs :

#!/usr/bin/env python3
"""
Production Cleanup & Monitoring Script
"""                                                                 

import os
import json
import subprocess
import datetime
import getpass
import grp
from pathlib import Path
from rich.console import Console
from rich.prompt import Prompt
from rich.table import Table

console = Console()

# Log file setup: try /var/log first, else fallback to ~/logs
DEFAULT_LOG = Path("/var/log/prod_cleanup.log")
FALLBACK_LOG = Path.home() / "logs" / "prod_cleanup.log"
LOG_FILE = DEFAULT_LOG if os.access(DEFAULT_LOG.parent, os.W_OK) else FALLBACK_LOG


def show_description():
    console.print("""
[bold cyan]==========================================================[/bold cyan]
[bold yellow]Script Modes:[/bold yellow]

[bold green]1. Auto Mode (1)[/bold green]
   - The script runs automatically with predefined actions.
   - Kills processes, deletes files, scans disks, and logs everything.
   - No prompts appear for user input.
   - [red]Use with caution:[/red] only for safe/staging environments or
     pre-approved process/file lists.

[bold green]2. Interactive Mode (2) [Default][/bold green]
   - The script pauses and asks for input before every critical action.
     - Which PIDs to kill
     - Which files to delete
     - Folder paths for disk scan
   - Requires approval from an authorized user before killing/deleting.
   - [yellow]Safer for production environments.[/yellow]

[bold magenta]Note:[/bold magenta]
- Interactive mode is recommended for production.
- Auto mode is faster but may act on processes/files without confirmation.
[bold cyan]==========================================================[/bold cyan]
""")


def log_event(event, detail):
    """Log event in JSON format with safe IAM role detection"""
    ts = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
    linux_user = getpass.getuser()
    sudo_user = os.environ.get("SUDO_USER", linux_user)
    linux_groups = ",".join(grp.getgrgid(g).gr_name for g in os.getgroups())
    euid = os.geteuid()

    # Detect IAM role safely
    try:
        role_name = subprocess.check_output(
            ["curl", "-s", "http://169.254.169.254/latest/meta-data/iam/security-credentials/"],
            text=True, timeout=2
        ).strip()
        if role_name and "Unauthorized" not in role_name:
            iam_role = role_name
        else:
            iam_role = "none"
    except Exception:
        iam_role = "none"

    log_data = {
        "ts": ts,
        "linux_user": linux_user,
        "sudo_user": sudo_user,
        "linux_groups": linux_groups,
        "euid": euid,
        "iam_role": iam_role,
        "event": event,
        "detail": detail
    }
    LOG_FILE.parent.mkdir(parents=True, exist_ok=True)
    with LOG_FILE.open("a") as f:
        f.write(json.dumps(log_data) + "\n")


def run_cmd(cmd):
    try:
        return subprocess.check_output(cmd, shell=True, text=True)
    except subprocess.CalledProcessError:
        return ""


def system_monitor():
    console.print("\n[bold cyan]======= SYSTEM MONITOR =======[/bold cyan]")
    uptime_output = run_cmd("uptime | awk -F'load average:' '{print $2}'").strip()
    console.print(f"[bold yellow]CPU Load:[/bold yellow] {uptime_output}")
    console.print(run_cmd("free -h"))
    console.print(run_cmd("df -h | grep -E '^/dev|Filesystem'"))
    log_event("monitor", "System stats displayed")


def show_top_processes():
    # CPU
    console.print("\n[bold cyan]======= TOP 20 CPU PROCESSES =======[/bold cyan]")
    cpu_output = run_cmd("ps -eo pid,ppid,comm,%cpu,%mem --sort=-%cpu | head -n 21").splitlines()
    table_cpu = Table(show_header=True, header_style="bold magenta")
    table_cpu.add_column("PID", justify="right")
    table_cpu.add_column("PPID", justify="right")
    table_cpu.add_column("Command", overflow="fold")
    table_cpu.add_column("%CPU", justify="right")
    table_cpu.add_column("%MEM", justify="right")

    for line in cpu_output[1:]:
        parts = line.split(None, 4)
        if len(parts) == 5:
            table_cpu.add_row(*parts)
    console.print(table_cpu)

    # Memory
    console.print("\n[bold cyan]======= TOP 20 MEMORY PROCESSES =======[/bold cyan]")
    mem_output = run_cmd("ps -eo pid,ppid,comm,%cpu,%mem --sort=-%mem | head -n 21").splitlines()
    table_mem = Table(show_header=True, header_style="bold magenta")
    table_mem.add_column("PID", justify="right")
    table_mem.add_column("PPID", justify="right")
    table_mem.add_column("Command", overflow="fold")
    table_mem.add_column("%CPU", justify="right")
    table_mem.add_column("%MEM", justify="right")

    for line in mem_output[1:]:
        parts = line.split(None, 4)
        if len(parts) == 5:
            table_mem.add_row(*parts)
    console.print(table_mem)

    log_event("monitor", "Top CPU and memory processes displayed")
    return cpu_output, mem_output


def kill_processes(process_list, label):
    pids = Prompt.ask(f"Enter PIDs to kill from {label} list (space separated, blank to skip)", default="")
    if pids:
        for pid in pids.split():
            try:
                os.kill(int(pid), 9)
                console.print(f"[green]  Killed process {pid}[/green]")
                log_event("kill", f"Process {pid} killed from {label}")
            except Exception:
                console.print(f"[red]  Failed to kill {pid}[/red]")


def list_ebs_volumes():
    console.print("\n[bold cyan]======= EBS VOLUMES (Top 20) =======[/bold cyan]")
    if run_cmd("which aws"):
        volumes = run_cmd(
            "aws ec2 describe-volumes --query \"Volumes[*].{ID:VolumeId,State:State,Size:Size,AZ:AvailabilityZone}\" --output table | head -n 22"
        )
        console.print(volumes if volumes else "[yellow]No volumes found[/yellow]")
        log_event("ebs", "EBS volumes listed")
    else:
        console.print("[yellow]AWS CLI not installed. Skipping EBS volume listing.[/yellow]")
        log_event("ebs", "AWS CLI not installed")


def disk_usage():
    path = Prompt.ask("Enter folder path to scan", default="/")
    console.print(f"\n[bold cyan]======= DISK USAGE ({path}) =======[/bold cyan]")
    usage = run_cmd(f"du -ah {path} 2>/dev/null | sort -rh | head -n 20")
    console.print(usage)
    log_event("disk", f"Top storage usage scanned in path {path}")


def delete_files():
    files = Prompt.ask("Enter full file paths to delete (space separated, blank to skip)", default="")
    if files:
        for f in files.split():
            try:
                os.remove(f)
                console.print(f"[green]  Deleted {f}[/green]")
                log_event("delete", f"File {f} deleted")
            except FileNotFoundError:
                console.print(f"[yellow]   File {f} not found[/yellow]")


def main():
    show_description()
    mode = Prompt.ask("Choose Mode: [green]1=Auto[/green], [yellow]2=Interactive[/yellow]", default="2")

    system_monitor()
    cpu_list, mem_list = show_top_processes()
    list_ebs_volumes()

    if mode == "2":
        kill_processes(cpu_list, "CPU")
        kill_processes(mem_list, "Memory")
        disk_usage()
        delete_files()

    log_event("finish", "Cleanup completed")
    console.print("\n[bold green]   Cleanup Completed![/bold green]")


if __name__ == "__main__":
    main()   

If run the script 

While troubleshooting Linux servers, I was jumping between different commands to check logs, services, CPU, memory, and disk usage. It was taking more time than I wanted.

HOW I DEBUGGED IT

I started trying a few TUI tools to get a quick view of what was happening on the server.

This made it easier to investigate logs, system health, and disk usage directly from the terminal.

THE SOLUTION

These tools helped me spot service errors and resource issues faster. I could see more information in one place instead of switching between multiple commands. It made my Linux troubleshooting workflow much simpler and faster.

Linux Issue → Check Logs & System Health → TUI Tools → Identify the Problem → Fix

Have you faced a similar problem?
What tools or commands do you use for Linux troubleshooting?

How did you fix it?
HashnodProfile:https://hashnode.com/@vishnusaiK
GitHub : https://github.com/ksaivishnusaikvs

Lnav: lnav      What I like about lnav

Search & filter logs quickly — find errors, warnings, and specific events without chaining multiple commands.

Easy log navigation — work with timestamps and follow live logs in an interactive terminal UI.

One place for troubleshooting — combines many common log-viewing tasks into a simple TUI instead of switching between grep, tail, awk, and less.

lnav file = analyze log files; helps you investigate those logs interactively.

journalctl | lnav = analyze systemd journal logs; get  the Nginx service logs.

Problem                               Command                                                  What It Solves

--------------------------------------------------------------------------------

Service Failing         journalctl -u nginx.service | lnav              Investigate systemd service errors

Container Issue            docker logs nginx 2>&1 | lnav             Investigate container logs

Live Container Issue    docker logs -f nginx 2>&1 | lnav          Watch errors as they happen

Application Issue         lnav /var/log/nginx/access.log              Analyze a specific log file

tui-logs

tui-logs mainly solves the problem of manually searching through large log files using multiple Linux commands.

What it solvesToo many logs → quickly search and filter the logs you need.Hard to troubleshoot errors → easily find errors, warnings, and related events.

Live-log monitoring → watch new logs in real time from one interactive terminal.

Too many commands → reduces the need to switch between cat, grep, tail, less, and awk.

SysWatch:     syswatch

Gives a single live view of Linux CPU, memory, disk, network, processes, and services.

Main problem: Quickly identifies failed services, resource issues, and system health problems.

Production benefit: Enables faster troubleshooting and helps prevent server/application downtime.

Tui-disk: tui-disk

Shows all disks, partitions, mounts, usage, and health in one simple view.

Production benefit: Quickly finds full disks, mount problems, and storage issues before they cause outages.

Superfile is a simple TUI file manager that brings a visual file-browser experience directly into the terminal.

●   📁 Easy navigation — browse files and folders quickly.

●  🔄 File management — copy, move, rename, and delete files from one interface.

● ⚡ Less typing — reduces the need for repeated ls, cd, cp, mv, and rm commands.

PROBLEM

I wanted to create an AWS S3 bucket using Python + CDKTF instead of writing Terraform HCL. I also needed versioning, encryption, public access fizefsh, and proper tags.

HOW I DEBUGGED IT

First, I checked my AWS access. I checked the Python and package versions first.

Instead of changing the system Python,

I created a virtual environment (venv) and installed the required packages there.

Normal/system Python → you tried it → it did NOT work

Python venv → you tried it → it WORKED   Then I tested the Python code and ran

cdktf synth

After that, I ran:

cdktf diff       to make sure everything looked right before deployment.

THE SOLUTION

cdktf deploy

The S3 bucket was created successfully with versioning, encryption, public access blocked, and proper tags.

Has anyone else faced this problem?

Have you used Python + CDKTF for AWS infrastructure?

How did you fix it?

Note: All infrastructure images, official docs, and videos links  are included on the last page
**GitHub:**https://github.com/ksaivishnusaikvs/DevOpsEngineering/tree/main/Linkedin

2. Prerequisites

Ubuntu EC2 instance, Python 3, pip, Node.js/npm, CDKTF CLI, Terraform CLIAWS CLIAWS credentials or an IAM role with permission to create the required S3 resources

Verify AWS Access aws sts get-caller-identity

Step 1: Create infrastructure: Create the CDKTF Project

cd ~
rm -rf prod-s3
mkdir prod-s3
cd prod-s3
python3 -m venv .venv
source .venv/bin/activate

pip install --upgrade pip
pip install cdktf constructs cdktf-cdktf-provider-aws

STEP2:  Create the CDKTF configuration

cat > cdktf.json <<'EOF'
{
  "language": "python",
  "app": "python3 main.py"
}
EOF

This tells CDKTF that the application entry point is main.py and that Python is the projectlanguage.

Step 3: Write code for infra Python Infrastructure Code. Create the infrastructure file

from cdktf import App, TerraformStack
from constructs import Construct

from cdktf_cdktf_provider_aws.provider import AwsProvider
from cdktf_cdktf_provider_aws.s3_bucket import S3Bucket
from cdktf_cdktf_provider_aws.s3_bucket_versioning import S3BucketVersioningA
from cdktf_cdktf_provider_aws.s3_bucket_public_access_block import (
	S3BucketPublicAccessBlock
)
from cdktf_cdktf_provider_aws.s3_bucket_server_side_encryption_configuration import (
    S3BucketServerSideEncryptionConfigurationA
)


class ProdS3Stack(TerraformStack):

	def __init__(self, scope: Construct, stack_id: str):
    	super().__init__(scope, stack_id)

    	# AWS provider
    	AwsProvider(
        	self,
        	"aws",
        	region="ap-south-1"
    	)

    	# S3 bucket
    	bucket = S3Bucket(
        	self,
        	"prod-bucket",
            bucket="kv-prod-s3-demo-20260909",
        	tags={
            	"Environment": "prod",
            	"Application": "my-app",
            	"ManagedBy": "cdktf"
        	}
    	)

    	# Versioning
    	S3BucketVersioningA(
        	self,
        	"versioning",
        	bucket=bucket.id,
        	versioning_configuration={
            	"status": "Enabled"
        	}
    	)

    	# Block public access
    	S3BucketPublicAccessBlock(
        	self,
        	"public-access",
        	bucket=bucket.id,
        	block_public_acls=True,
        	block_public_policy=True,
        	ignore_public_acls=True,
        	restrict_public_buckets=True
    	)

    	# Server-side encryption
        S3BucketServerSideEncryptionConfigurationA(
        	self,
        	"encryption",
        	bucket=bucket.id,
        	rule=[
            	{
                    "apply_server_side_encryption_by_default": [
                    	{
                            "sse_algorithm": "AES256"
                    	}
                	]
            	}
        	]
    	)


app = App()

ProdS3Stack(app, "prod-s3")

app.synth()

Step 4: What the Python Code Creates

AWS Provider: Uses ap-south-1 as the AWS region.

S3 Bucket: Creates the production bucket.

Tags: Adds Environment, Application, and ManagedBy tags for ownership and FinOps tracking.

Versioning: Keeps object versions so accidental changes/deletions can be recovered.

Public Access Block: Blocks public ACLs and public bucket policies.

Encryption: Enables S3 server-side encryption using AES256.

Step5:   python3 main.py

Test the Python Code. If there is no Python traceback/error, the CDKTF application code is valid enough to synthesize.

Step6: cdktf synth CDKTF converts the Python infrastructure definition into Terraform configuration under the generated CDKTF output.

Step7: cdktf diff Review the resources before deployment. In the successful setup, the plan showed 4 resources to add and 0 to change or destroy.

Step8: cdktf deploy When CDKTF asks for approval, choose Approve/yes. Terraform then creates the AWS resources

Step 9: See the image below; infra was created with Terraform/CDKTF using Python, and the data was validated

Step 10: Verify whether the S3 Bucket  infra is created or not in the cloud are not state file  create not in local

Step 11: Verify in the cloud ui the infrastructure resource is crete are not  in cloud UI

my s3 create throw CDKTF; name: s3    kv-prod-s3-demo-20260909

TAGS show the Terraform file; compare the two images

compare the tags in the below image both are matched

bucker version has enable

block the public acess

encryption has enabled

REFDOC:

Hashicorp Ofiical : https://developer.hashicorp.com/terraform/cdktf

AWS: https://aws.amazon.com/blogs/opensource/announcing-cdk-for-terraform-on-aws/

installsteps: 1:https://spacelift.io/blog/terraform-cdk

2:https://dev.to/aws-builders/cdk-for-terraform-cdktf-on-aws-how-to-configure-an-s3-remote-backend-and-deploy-a-lambda-function-url-using-python-3okk

CDKTF official HashiCorp channel: https://www.youtube.com/watch?v=Y9suaDVT-SY&t=27s

https://developer.hashicorp.com/terraform/tutorials/cdktf/cdktf-install

DOCsteps:  https://spacelift.io/blog/terraform-cdk

PROBLEM

When managing multiple EC2 instances, manually checking every server is time-consuming. If an instance stops or has a health issue, it can easily go unnoticed.

HOW I DEBUGGED IT

I wanted to avoid checking each instance manually, so I automated the health checks. I monitor the Running/Stopped state, 1/2 Status Checks, CPU, Memory, and Disk utilization across all EC2 instances.

THE SOLUTION

EventBridge → Lambda → EC2 / CloudWatch / SSM → Slack

Every 5 minutes, Lambda checks all EC2 instances. If an instance stops or any issue is detected, it automatically sends the details to Slack.

This helps me quickly identify which server has the problem, what went wrong, and where to start troubleshooting, without manually checking every instance.

Result: Less manual monitoring, faster issue detection, and quicker troubleshooting.

Has anyone else faced this problem?

How did you fix it?

Step 1: Create the Lambda function that will perform the EC2 health checks. Configure the function with the required Python runtime and prepare it to execute the monitoring code.

Step 2: Configure Lambda. Configure the Lambda function with the required environment variables, including EC2 instance IDs, AWS Region, alert thresholds, Slack/Teams webhook URLs, and SNS topic ARN.

Step 3: Configure the Lambda function with the required environment variables, including EC2 instance IDs

Step 4: Deploy the Lambda function and run a test execution. Verify that the function can successfully collect the EC2 health information and return the expected response Now we test and deploy

Step 5: Status. After running the test, check the execution result and logs. Confirm that the Lambda successfully checks the configured EC2 instance and produces the health information required for the alert.Status

Step 6: Click Deploy to deploy the Lambda function. Review the Slack notification shown below. With one click, we can see the data in the Slack Notification

one cick the data we can see slack Notification open the channal

Step 7: Receive the EC2 alert in Slack with clear information about the alert/warning, instance ID, server name, region, status, and timestamp.

Step 8: Click on the navigation buttons for the EC2 Console. Check the EC2 instance status to confirm whether the server is Running, Stopped, or in a transitional state before performing the recovery action.Click on the navigation buttons EC2 Console

●       Click on the navigation buttons Cloudwatch

Navigate the CloudWatch logs

SessionManager

Ec2_monitor.py

==============
Lambda function — runs on a schedule (EventBridge every 5 min).

Checks for EACH EC2 instance (public + private):
  1. CPU utilization        — CloudWatch metric AWS/EC2 (always free, no agent)
  2. Status checks 1/2 & 2/2 — CloudWatch metric AWS/EC2 (always free, no agent)
  3. Memory utilization     — SSM Run Command reads /proc/meminfo (no agent needed)
  4. Disk utilization       — SSM Run Command reads df (no agent needed)

Sends rich alerts to:
  - Slack  (Block Kit cards with action buttons)
  - Teams  (Adaptive Cards with action buttons)
  - Email  (via SNS — already subscribed in your infra)

Environment variables required:
  SLACK_WEBHOOK_URL     Slack incoming webhook
  TEAMS_WEBHOOK_URL     Teams incoming webhook
  SNS_TOPIC_ARN         SNS topic ARN for email
  INSTANCE_IDS          Comma-separated: i-aaa,i-bbb,i-ccc
  CPU_WARN_THRESHOLD    Default 80
  CPU_CRIT_THRESHOLD    Default 95
  MEM_WARN_THRESHOLD    Default 85
  DISK_WARN_THRESHOLD   Default 85
  AWS_REGION_NAME       e.g. ap-south-1  (note: not AWS_REGION — that's reserved)
"""

import json
import os
import urllib.request
import urllib.error
import boto3
import time
from datetime import datetime, timedelta, timezone
from typing import Optional

# ── Config from environment ────────────────────────────────────────────────────
SLACK_WEBHOOK_URL  = os.environ.get("SLACK_WEBHOOK_URL", "https://hooks.slack.com/services/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")
TEAMS_WEBHOOK_URL  = os.environ.get("TEAMS_WEBHOOK_URL", "")
SNS_TOPIC_ARN      = os.environ.get("SNS_TOPIC_ARN", "")
REGION             = os.environ.get("AWS_REGION_NAME", "eu-north-1")
ENVIRONMENT        = os.environ.get("ENVIRONMENT", "prod")

INSTANCE_IDS: list[str] = [
    i.strip() for i in os.environ.get("INSTANCE_IDS", "i-00430b1a074d82e2c").split(",") if i.strip()
]

CPU_WARN  = float(os.environ.get("CPU_WARN_THRESHOLD",  "80"))
CPU_CRIT  = float(os.environ.get("CPU_CRIT_THRESHOLD",  "80"))
MEM_WARN  = float(os.environ.get("MEM_WARN_THRESHOLD",  "80"))
DISK_WARN = float(os.environ.get("DISK_WARN_THRESHOLD", "80"))

# ── AWS clients ────────────────────────────────────────────────────────────────
cw  = boto3.client("cloudwatch",  region_name=REGION)
ec2 = boto3.client("ec2",         region_name=REGION)
ssm = boto3.client("ssm",         region_name=REGION)
sns = boto3.client("sns",         region_name=REGION)


# ══════════════════════════════════════════════════════════════════════════════
# METRIC COLLECTORS
# ══════════════════════════════════════════════════════════════════════════════

def get_cloudwatch_metric(instance_id: str, metric_name: str, namespace: str = "AWS/EC2") -> Optional[float]:
    """
    Fetch the latest average value of a CloudWatch metric for an instance.
    Uses the last 10 minutes window to get the most recent data point.
    Returns None if no data is available.
    """
    now = datetime.now(timezone.utc)
    resp = cw.get_metric_statistics(
        Namespace=namespace,
        MetricName=metric_name,
        Dimensions=[{"Name": "InstanceId", "Value": instance_id}],
        StartTime=now - timedelta(minutes=10),
        EndTime=now,
        Period=300,
        Statistics=["Average", "Maximum"],
    )
    datapoints = resp.get("Datapoints", [])
    if not datapoints:
        return None
    latest = max(datapoints, key=lambda d: d["Timestamp"])
    return round(latest["Average"], 2)


def get_status_checks(instance_id: str) -> dict:
    """
    Returns status check results directly from EC2 API.
    status = 'ok' | 'impaired' | 'insufficient-data' | 'not-applicable'
    """
    try:
        resp = ec2.describe_instance_status(
            InstanceIds=[instance_id],
            IncludeAllInstances=True,
        )
        statuses = resp.get("InstanceStatuses", [])
        if not statuses:
            return {"system": "unknown", "instance": "unknown", "state": "unknown"}

        s = statuses[0]
        return {
            "state":    s["InstanceState"]["Name"],
            "system":   s["SystemStatus"]["Status"],    # ok | impaired | ...
            "instance": s["InstanceStatus"]["Status"],  # ok | impaired | ...
        }
    except Exception as e:
        print(f"[WARN] Status check failed for {instance_id}: {e}")
        return {"system": "error", "instance": "error", "state": "unknown"}


def get_instance_name(instance_id: str) -> str:
    """Return the Name tag of an instance, or the instance ID if no tag."""
    try:
        resp = ec2.describe_instances(InstanceIds=[instance_id])
        tags = resp["Reservations"][0]["Instances"][0].get("Tags", [])
        for tag in tags:
            if tag["Key"] == "Name":
                return tag["Value"]
    except Exception:
        pass
    return instance_id


def run_ssm_command(instance_id: str, command: str, timeout: int = 20) -> Optional[str]:
    """
    Execute a shell command on an EC2 instance via SSM Run Command.
    Returns stdout string or None on failure.
    Requires:
      - EC2 instance has SSM Agent running (pre-installed on Amazon Linux 2/2023, Ubuntu 16+)
      - EC2 IAM role has AmazonSSMManagedInstanceCore policy
    """
    try:
        resp = ssm.send_command(
            InstanceIds=[instance_id],
            DocumentName="AWS-RunShellScript",
            Parameters={"commands": [command]},
            TimeoutSeconds=timeout,
        )
        command_id = resp["Command"]["CommandId"]

        # Poll for result (max 25 seconds)
        for _ in range(5):
            time.sleep(5)
            result = ssm.get_command_invocation(
                CommandId=command_id,
                InstanceId=instance_id,
            )
            status = result["Status"]
            if status == "Success":
                return result["StandardOutputContent"].strip()
            if status in ("Failed", "Cancelled", "TimedOut"):
                print(f"[WARN] SSM command {status} on {instance_id}")
                return None
        print(f"[WARN] SSM command timed out polling on {instance_id}")
        return None
    except ssm.exceptions.InvalidInstanceId:
        print(f"[WARN] {instance_id} not registered with SSM (agent not running or no IAM role)")
        return None
    except Exception as e:
        print(f"[WARN] SSM error on {instance_id}: {e}")
        return None


def get_memory_via_ssm(instance_id: str) -> Optional[float]:
    """
    Read memory usage % from /proc/meminfo via SSM Run Command.
    No CloudWatch Agent needed.
    """
    cmd = (
        "awk '/MemTotal/{t=$2} /MemAvailable/{a=$2} "
        "END{printf \"%.2f\", (t-a)/t*100}' /proc/meminfo"
    )
    output = run_ssm_command(instance_id, cmd)
    if output is None:
        return None
    try:
        return float(output)
    except ValueError:
        return None


def get_disk_via_ssm(instance_id: str) -> Optional[float]:
    """
    Read root disk usage % via SSM Run Command.
    Returns the highest used % across all mounted filesystems.
    No CloudWatch Agent needed.
    """
    cmd = "df -h --output=pcent / | tail -1 | tr -d ' %'"
    output = run_ssm_command(instance_id, cmd)
    if output is None:
        return None
    try:
        return float(output)
    except ValueError:
        return None


# ══════════════════════════════════════════════════════════════════════════════
# HEALTH EVALUATOR
# ══════════════════════════════════════════════════════════════════════════════

def evaluate_instance(instance_id: str) -> dict:
    """
    Collect all metrics for one instance and return a health report dict.
    """
    name = get_instance_name(instance_id)
    print(f"  Checking {name} ({instance_id})...")

    cpu    = get_cloudwatch_metric(instance_id, "CPUUtilization")
    status = get_status_checks(instance_id)
    memory = get_memory_via_ssm(instance_id)
    disk   = get_disk_via_ssm(instance_id)

    alerts = []

    # CPU
    if cpu is not None:
        if cpu >= CPU_CRIT:
            alerts.append({"metric": "CPU", "value": cpu, "threshold": CPU_CRIT, "severity": "CRITICAL"})
        elif cpu >= CPU_WARN:
            alerts.append({"metric": "CPU", "value": cpu, "threshold": CPU_WARN, "severity": "WARNING"})

    # Memory
    if memory is not None:
        if memory >= MEM_WARN:
            alerts.append({"metric": "Memory", "value": memory, "threshold": MEM_WARN, "severity": "WARNING"})

    # Disk
    if disk is not None:
        if disk >= DISK_WARN:
            alerts.append({"metric": "Disk", "value": disk, "threshold": DISK_WARN, "severity": "WARNING"})

    # Status checks
    if status["system"] not in ("ok", "unknown"):
        alerts.append({
            "metric": "Status Check (1/2 System)",
            "value": status["system"],
            "threshold": "ok",
            "severity": "CRITICAL",
            "note": "AWS hardware issue — may require stop/start to migrate host"
        })
    if status["instance"] not in ("ok", "unknown"):
        alerts.append({
            "metric": "Status Check (2/2 Instance)",
            "value": status["instance"],
            "threshold": "ok",
            "severity": "CRITICAL",
            "note": "OS/kernel issue — check system logs or reboot"
        })

    return {
        "instance_id": instance_id,
        "name": name,
        "state": status.get("state", "unknown"),
        "metrics": {
            "cpu":    cpu,
            "memory": memory,
            "disk":   disk,
            "status_system":   status["system"],
            "status_instance": status["instance"],
        },
        "alerts": alerts,
        "healthy": len(alerts) == 0,
    }


# ══════════════════════════════════════════════════════════════════════════════
# NOTIFICATION BUILDERS
# ══════════════════════════════════════════════════════════════════════════════

SEVERITY_COLORS = {
    "CRITICAL": {"hex": "#D32F2F", "teams": "attention", "emoji": "🔴"},
    "WARNING":  {"hex": "#F57C00", "teams": "warning",   "emoji": "🟠"},
    "OK":       {"hex": "#388E3C", "teams": "good",       "emoji": "✅"},
}

def _metric_line(report: dict) -> str:
    m = report["metrics"]
    cpu_str  = f"{m['cpu']:.1f}%"    if m["cpu"]    is not None else "N/A"
    mem_str  = f"{m['memory']:.1f}%" if m["memory"] is not None else "N/A (no SSM)"
    disk_str = f"{m['disk']:.1f}%"   if m["disk"]   is not None else "N/A (no SSM)"
    sys_str  = m["status_system"]
    ins_str  = m["status_instance"]
    return (
        f"CPU: {cpu_str}  |  Mem: {mem_str}  |  "
        f"Disk: {disk_str}  |  Status: {sys_str}/{ins_str}"
    )


def build_slack_message(report: dict) -> dict:
    instance_id = report["instance_id"]
    name        = report["name"]
    alerts      = report["alerts"]
    env         = ENVIRONMENT.upper()
    region      = REGION

    top_severity = "CRITICAL" if any(a["severity"] == "CRITICAL" for a in alerts) else "WARNING"
    color = SEVERITY_COLORS[top_severity]["hex"]
    emoji = SEVERITY_COLORS[top_severity]["emoji"]

    header = f"{emoji} [{env}] EC2 ALERT — {name} ({instance_id})"

    alert_lines = []
    for a in alerts:
        note = f" — {a['note']}" if "note" in a else ""
        alert_lines.append(
            f"• *{a['metric']}*: `{a['value']}` (threshold: {a['threshold']}) [{a['severity']}]{note}"
        )

    ec2_url = f"https://{region}.console.aws.amazon.com/ec2/home?region={region}#Instances:instanceId={instance_id}"
    cw_url  = f"https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}#alarmsV2:"
    logs_url = f"https://{region}.console.aws.amazon.com/systems-manager/run-command?region={region}"

    blocks = [
        {
            "type": "header",
            "text": {"type": "plain_text", "text": header[:150], "emoji": True}
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "\n".join(alert_lines)
            }
        },
        {"type": "divider"},
        {
            "type": "section",
            "fields": [
                {"type": "mrkdwn", "text": f"*Instance*\n`{instance_id}`"},
                {"type": "mrkdwn", "text": f"*Name*\n{name}"},
                {"type": "mrkdwn", "text": f"*State*\n{report['state']}"},
                {"type": "mrkdwn", "text": f"*Region*\n{region}"},
            ]
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": f"*Current metrics*\n`{_metric_line(report)}`"
            }
        },
        {
            "type": "context",
            "elements": [
                {"type": "mrkdwn", "text": f"Time: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC | Env: {env}"}
            ]
        },
        {
            "type": "actions",
            "elements": [
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "EC2 Console"},
                    "url": ec2_url,
                    "style": "danger"
                },
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "CloudWatch"},
                    "url": cw_url
                },
                {
                    "type": "button",
                    "text": {"type": "plain_text", "text": "SSM Run Command"},
                    "url": logs_url
                },
            ]
        }
    ]

    return {
        "attachments": [{
            "color": color,
            "blocks": blocks,
            "fallback": header
        }]
    }


def build_teams_message(report: dict) -> dict:
    instance_id = report["instance_id"]
    name        = report["name"]
    alerts      = report["alerts"]
    env         = ENVIRONMENT.upper()
    region      = REGION

    top_severity = "CRITICAL" if any(a["severity"] == "CRITICAL" for a in alerts) else "WARNING"
    sev          = SEVERITY_COLORS[top_severity]
    title        = f"{sev['emoji']} [{env}] EC2 ALERT — {name} ({instance_id})"

    ec2_url  = f"https://{region}.console.aws.amazon.com/ec2/home?region={region}#Instances:instanceId={instance_id}"
    cw_url   = f"https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}#alarmsV2:"

    alert_facts = []
    for a in alerts:
        note = f" ({a.get('note', '')})" if "note" in a else ""
        alert_facts.append({
            "title": f"{a['severity']} — {a['metric']}",
            "value": f"{a['value']} (threshold: {a['threshold']}){note}"
        })

    m = report["metrics"]

    return {
        "type": "message",
        "attachments": [{
            "contentType": "application/vnd.microsoft.card.adaptive",
            "content": {
                "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
                "type": "AdaptiveCard",
                "version": "1.4",
                "body": [
                    {
                        "type": "TextBlock",
                        "text": title,
                        "weight": "bolder",
                        "size": "medium",
                        "wrap": True,
                        "color": sev["teams"]
                    },
                    {
                        "type": "FactSet",
                        "facts": alert_facts
                    },
                    {"type": "TextBlock", "text": "Current metrics", "weight": "bolder", "size": "small"},
                    {
                        "type": "FactSet",
                        "facts": [
                            {"title": "CPU",     "value": f"{m['cpu']:.1f}%"    if m["cpu"]    is not None else "N/A"},
                            {"title": "Memory",  "value": f"{m['memory']:.1f}%" if m["memory"] is not None else "N/A (no SSM)"},
                            {"title": "Disk",    "value": f"{m['disk']:.1f}%"   if m["disk"]   is not None else "N/A (no SSM)"},
                            {"title": "Status",  "value": f"System: {m['status_system']} | Instance: {m['status_instance']}"},
                            {"title": "State",   "value": report["state"]},
                            {"title": "Region",  "value": region},
                            {"title": "Time",    "value": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S") + " UTC"},
                        ]
                    }
                ],
                "actions": [
                    {"type": "Action.OpenUrl", "title": "Open EC2 Console",    "url": ec2_url},
                    {"type": "Action.OpenUrl", "title": "View CloudWatch",      "url": cw_url},
                ]
            }
        }]
    }


def build_sns_email(report: dict) -> str:
    """Plain text email body for SNS."""
    env   = ENVIRONMENT.upper()
    name  = report["name"]
    iid   = report["instance_id"]
    lines = [
        f"[{env}] EC2 ALERT — {name} ({iid})",
        f"Time: {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC",
        f"Region: {REGION}",
        f"State: {report['state']}",
        "",
        "ALERTS:",
    ]
    for a in report["alerts"]:
        note = f" | Note: {a.get('note', '')}" if "note" in a else ""
        lines.append(f"  [{a['severity']}] {a['metric']}: {a['value']} (threshold: {a['threshold']}){note}")
    m = report["metrics"]
    lines += [
        "",
        "CURRENT METRICS:",
        f"  CPU:    {m['cpu']:.1f}%"    if m["cpu"]    is not None else "  CPU:    N/A",
        f"  Memory: {m['memory']:.1f}%" if m["memory"] is not None else "  Memory: N/A (SSM not available)",
        f"  Disk:   {m['disk']:.1f}%"   if m["disk"]   is not None else "  Disk:   N/A (SSM not available)",
        f"  Status: System={m['status_system']} | Instance={m['status_instance']}",
        "",
        f"EC2 Console: https://{REGION}.console.aws.amazon.com/ec2/home?region={REGION}#Instances:instanceId={iid}",
    ]
    return "\n".join(lines)


# ══════════════════════════════════════════════════════════════════════════════
# NOTIFICATION SENDERS
# ══════════════════════════════════════════════════════════════════════════════

def post_webhook(url: str, payload: dict, label: str) -> bool:
    if not url:
        print(f"  [SKIP] {label}: no webhook URL configured")
        return False
    data = json.dumps(payload).encode("utf-8")
    req  = urllib.request.Request(url, data=data, headers={"Content-Type": "application/json"}, method="POST")
    try:
        with urllib.request.urlopen(req, timeout=10) as r:
            print(f"  [OK] {label}: HTTP {r.getcode()}")
            return r.getcode() == 200
    except urllib.error.HTTPError as e:
        print(f"  [ERROR] {label}: HTTP {e.code} — {e.read().decode()[:200]}")
        return False
    except Exception as e:
        print(f"  [ERROR] {label}: {e}")
        return False


def send_sns_email(report: dict) -> bool:
    if not SNS_TOPIC_ARN:
        print("  [SKIP] Email: SNS_TOPIC_ARN not configured")
        return False
    top = "CRITICAL" if any(a["severity"] == "CRITICAL" for a in report["alerts"]) else "WARNING"
    subject = f"[{ENVIRONMENT.upper()}] {top} — EC2 {report['name']} ({report['instance_id']})"
    try:
        sns.publish(
            TopicArn=SNS_TOPIC_ARN,
            Subject=subject[:100],
            Message=build_sns_email(report),
        )
        print(f"  [OK] Email via SNS")
        return True
    except Exception as e:
        print(f"  [ERROR] SNS email: {e}")
        return False


def notify(report: dict):
    """Send alert to all channels."""
    post_webhook(SLACK_WEBHOOK_URL, build_slack_message(report), "Slack")
    post_webhook(TEAMS_WEBHOOK_URL, build_teams_message(report), "Teams")
    send_sns_email(report)


# ══════════════════════════════════════════════════════════════════════════════
# LAMBDA ENTRY POINT
# ══════════════════════════════════════════════════════════════════════════════

def lambda_handler(event, context):
    """
    Triggered by EventBridge every 5 minutes.
    Checks all instances and sends alerts for any that are unhealthy.
    """
    if not INSTANCE_IDS:
        print("[ERROR] INSTANCE_IDS environment variable is empty. Nothing to check.")
        return {"statusCode": 400, "message": "No instance IDs configured"}

    print(f"[START] Checking {len(INSTANCE_IDS)} instances: {INSTANCE_IDS}")
    results  = []
    alerted  = 0

    for instance_id in INSTANCE_IDS:
        report = evaluate_instance(instance_id)
        results.append(report)

        if not report["healthy"]:
            print(f"  [ALERT] {report['name']} has {len(report['alerts'])} issue(s) — sending notifications")
            notify(report)
            alerted += 1
        else:
            print(f"  [OK] {report['name']} is healthy")

    summary = {
        "checked":   len(INSTANCE_IDS),
        "healthy":   len(INSTANCE_IDS) - alerted,
        "alerted":   alerted,
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "results":   [
            {
                "id":      r["instance_id"],
                "name":    r["name"],
                "healthy": r["healthy"],
                "alerts":  len(r["alerts"]),
            }
            for r in results
        ],
    }
    print(f"[DONE] {summary}")
    return {"statusCode": 200, "summary": summary}

PROBLEM

Managing Linux servers can become difficult when you have to check CPU, memory, disk usage, processes, Docker resources, and EBS volumes separately. I wanted a simple way to bring these everyday monitoring and cleanup tasks into one place.

HOW I DEBUGGED IT

I built a Python script that collects system information, monitors resources, identifies unused processes, checks Docker resources, and gives visibility into what is happening on the server.

THE SOLUTION

I added Auto, Interactive, and Dry-Run modes so cleanup actions can be controlled safely. The script also keeps logs and audit records to make every important action easier to track and review.

Monitor → Detect → Decide → Cleanup → Log → Audit

Step 1 — Script Overview

Shows the script modes and main features such as monitoring, cleanup, process detection, and logging.

Step 2 — Runner Identity

Collects information about the server, Linux user, IP address, AWS instance, region, and IAM identity.

Step 3 — Resource Monitoring

Checks CPU, memory, swap, disk usage, and system load to understand the server's current health.

Step 4 — Process Detection

Finds idle, sleeping, and zombie processes and supports Auto, Manual, and Dry-Run modes.

Step 4 — Process Detection

Finds idle, sleeping, and zombie processes and supports Auto, Manual, and Dry-Run modes. Lists attached EBS volumes with their volume ID, availability zone, size, and current state. Lists attached EBS volumes with their volume ID, availability zone, size, and current state

Step 5 — Docker Cleanup

Checks stopped containers, dangling images, and unused Docker volumes for possible cleanup.

Step 6— Cleanup & Audit Logs

Performs selected cleanup actions and records deleted, skipped, failed, and restore-related information.

Step 7— Final Report

Displays the total runtime, section timing, audit logs, unprocessed items, and final cleanup status.

============================================================

PRODUCTION LINUX & DOCKER CLEANUP — CRON JOBS

============================================================

------------------------------------------------------------

LINUX

------------------------------------------------------------

CPU — every 5 minutes

*/5 * * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux cpu >> /var/log/prod_cleanup.log 2>&1

Memory — every 5 minutes

*/5 * * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux memory >> /var/log/prod_cleanup.log 2>&1

Disk — daily at 2:00 AM

0 2 * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux disk >> /var/log/prod_cleanup.log 2>&1

Processes — daily at 3:00 AM

0 3 * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux processes >> /var/log/prod_cleanup.log 2>&1

EBS — daily at 4:00 AM

0 4 * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux ebs >> /var/log/prod_cleanup.log 2>&1

All Linux operations — every Sunday at 5:00 AM

0 5 * * 0 /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux all >> /var/log/prod_cleanup.log 2>&1

------------------------------------------------------------

DOCKER

------------------------------------------------------------

Stopped containers — daily at 6:00 AM

0 6 * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --docker container-only >> /var/log/prod_cleanup.log 2>&1

Dangling images — daily at 7:00 AM

0 7 * * * /usr/bin/python3 /opt/prod_cleanup.py --mode auto --docker image-only >> /var/log/prod_cleanup.log 2>&1

Unused volumes — every Sunday at 8:00 AM

0 8 * * 0 /usr/bin/python3 /opt/prod_cleanup.py --mode auto --docker volume-only >> /var/log/prod_cleanup.log 2>&1

All Docker operations — every Sunday at 9:00 AM

0 9 * * 0 /usr/bin/python3 /opt/prod_cleanup.py --mode auto --docker all >> /var/log/prod_cleanup.log 2>&1

------------------------------------------------------------

EVERYTHING

------------------------------------------------------------

Complete Linux + Docker cleanup — every Sunday at 10:00 AM

0 10 * * 0 /usr/bin/python3 /opt/prod_cleanup.py --mode auto --linux all --docker all >> /var/log/prod_cleanup.log 2>&1

#!/usr/bin/env python3
"""
Production Cleanup & Monitoring Script — Enhanced v2.0
Fixes:  Dynamic file/directory deletion (IsADirectoryError resolved)
New:    Docker mode selector, live runtime clock, ASCII bar graphs,
        restoration log, unprocessed items tracker, unified audit log folder,
        Unused Process Detection mode selector (independent of global mode)
"""

import os
import json
import shutil
import socket
import subprocess
import datetime
import getpass
import grp
import pwd
import time
import platform
import signal
import sys
from pathlib import Path
from rich.console import Console
from rich.prompt import Prompt, Confirm
from rich.table import Table
from rich.panel import Panel
from rich.live import Live
from rich.text import Text
from rich.columns import Columns
from rich import box

console = Console()

# ─── Unified log folder (best practice: one dir, rotate-friendly) ─────────────
LOG_ROOT_CANDIDATES = [Path("/var/log/prod_cleanup"), Path.home() / "logs" / "prod_cleanup"]
LOG_DIR: Path = next((p for p in LOG_ROOT_CANDIDATES
                       if (p.exists() and os.access(p, os.W_OK))
                       or (not p.exists() and os.access(p.parent, os.W_OK))), LOG_ROOT_CANDIDATES[1])

RUN_TS   = datetime.datetime.utcnow().strftime("%Y%m%dT%H%M%SZ")
LOG_FILE      = LOG_DIR / f"cleanup_{RUN_TS}.ndjson"   # per-run structured log
AUDIT_FILE    = LOG_DIR / "audit_master.ndjson"         # append-only master audit
DELETION_LOG  = LOG_DIR / f"deletions_{RUN_TS}.json"   # what was deleted + restore hints
UNPROCESSED   = LOG_DIR / f"unprocessed_{RUN_TS}.json" # items user skipped / failed

# ─── Global timing ────────────────────────────────────────────────────────────
SCRIPT_START   = time.time()
SECTION_TIMINGS: dict = {}

# ─── Unprocessed + deletion tracker ─────────────────────────────────────────
UNPROCESSED_ITEMS: list = []
DELETED_ITEMS:     list = []

# ─── Global state ─────────────────────────────────────────────────────────────
RUNNER_IDENTITY: dict = {}
DOCKER_MODE: str = "2"   # 1=Auto, 2=Manual — set during startup


# ══════════════════════════════════════════════════════════════════════════════
#  HELPERS
# ══════════════════════════════════════════════════════════════════════════════

def elapsed_since(start: float) -> str:
    secs = time.time() - start
    if secs < 60:
        return f"{secs:.1f}s"
    mins, s = divmod(int(secs), 60)
    return f"{mins}m {s}s"


def runtime_str() -> str:
    """Human-friendly total elapsed since script start."""
    return elapsed_since(SCRIPT_START)


def now_utc() -> str:
    return datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")


def section_timer(name: str, start: float):
    elapsed = time.time() - start
    SECTION_TIMINGS[name] = elapsed
    console.print(f"[dim]⏱  {name} completed in {elapsed:.1f}s  (total runtime: {runtime_str()})[/dim]")


def run_cmd(cmd, timeout=30) -> str:
    try:
        return subprocess.check_output(cmd, shell=True, text=True,
                                       stderr=subprocess.DEVNULL, timeout=timeout)
    except Exception:
        return ""


def docker_available() -> bool:
    return bool(run_cmd("which docker").strip())


# ══════════════════════════════════════════════════════════════════════════════
#  LOG SETUP  — ensures log folder exists with README
# ══════════════════════════════════════════════════════════════════════════════

def init_log_dir():
    LOG_DIR.mkdir(parents=True, exist_ok=True)
    readme = LOG_DIR / "README.txt"
    if not readme.exists():
        readme.write_text(
            "Production Cleanup Script — Audit Log Folder\n"
            "=============================================\n"
            "cleanup_*.ndjson   → Per-run structured event log (one JSON per line)\n"
            "audit_master.ndjson→ Append-only master audit trail (all runs)\n"
            "deletions_*.json   → Files/dirs deleted + size + restore hints\n"
            "unprocessed_*.json → Items that were skipped or failed\n"
        )


def log_event(event: str, detail: str, extra: dict | None = None):
    ri  = RUNNER_IDENTITY
    geo = ri.get("geo", {})
    lu  = ri.get("linux_user", {})
    aws = ri.get("aws", {})
    ssh = ri.get("session", {})

    record = {
        "ts":           now_utc(),
        "run_id":       RUN_TS,
        "event":        event,
        "detail":       detail,
        "runtime_secs": round(time.time() - SCRIPT_START, 2),
        "hostname":     ri.get("hostname", socket.gethostname()),
        "local_ip":     ri.get("local_ip", ""),
        "public_ip":    ri.get("public_ip", ""),
        "geo_country":  geo.get("country", ""),
        "geo_city":     geo.get("city", ""),
        "linux_user":   lu.get("username", getpass.getuser()),
        "uid":          lu.get("uid", os.getuid()),
        "is_root":      lu.get("is_root", os.geteuid() == 0),
        "sudo_user":    ssh.get("sudo_user", ""),
        "aws_instance": aws.get("instance_id", ""),
        "aws_region":   aws.get("region", ""),
        "aws_arn":      aws.get("aws_arn", ""),
    }
    if extra:
        record.update(extra)

    LOG_DIR.mkdir(parents=True, exist_ok=True)
    line = json.dumps(record) + "\n"
    LOG_FILE.open("a").write(line)
    AUDIT_FILE.open("a").write(line)


# ══════════════════════════════════════════════════════════════════════════════
#  ASCII BAR GRAPHS
# ══════════════════════════════════════════════════════════════════════════════

def _bar(pct: float, width: int = 30, color: str = "green") -> str:
    """Return a rich-markup ASCII progress bar for `pct` (0–100)."""
    pct = max(0.0, min(100.0, pct))
    filled = int(round(pct / 100 * width))
    bar_color = "green" if pct < 60 else ("yellow" if pct < 85 else "red")
    bar = f"[{bar_color}]{'█' * filled}[/{bar_color}]{'░' * (width - filled)}"
    return f"{bar} [bold]{pct:5.1f}<a class="embed-card" href="/bold">/bold</a>"


def show_resource_graphs():
    """Print ASCII bar-graph dashboard for CPU, RAM, Disk, load average."""
    console.print("\n[bold cyan]======= RESOURCE USAGE GRAPHS =======[/bold cyan]")

    def _cpu_pct() -> list[tuple[str, float]]:
        lines1 = Path("/proc/stat").read_text().splitlines()
        time.sleep(0.4)
        lines2 = Path("/proc/stat").read_text().splitlines()
        result = []
        for l1, l2 in zip(lines1, lines2):
            if not l1.startswith("cpu"):
                continue
            p1 = list(map(int, l1.split()[1:]))
            p2 = list(map(int, l2.split()[1:]))
            idle1, idle2 = p1[3], p2[3]
            tot1,  tot2  = sum(p1), sum(p2)
            dtot  = tot2 - tot1
            if dtot == 0:
                continue
            pct = 100.0 * (1 - (idle2 - idle1) / dtot)
            result.append((l1.split()[0], pct))
        return result

    cpu_data = _cpu_pct()

    mem_info = {}
    for line in Path("/proc/meminfo").read_text().splitlines():
        k, v = line.split(":")[0], line.split(":")[1].strip().split()[0]
        mem_info[k] = int(v)
    mem_total = mem_info.get("MemTotal", 1)
    mem_avail = mem_info.get("MemAvailable", mem_total)
    mem_used  = mem_total - mem_avail
    mem_pct   = 100.0 * mem_used / mem_total
    swap_total = mem_info.get("SwapTotal", 0)
    swap_free  = mem_info.get("SwapFree", swap_total)
    swap_used  = swap_total - swap_free
    swap_pct   = 100.0 * swap_used / swap_total if swap_total else 0.0

    def _kb(kb: int) -> str:
        if kb > 1_048_576:
            return f"{kb/1_048_576:.1f} GB"
        return f"{kb/1024:.0f} MB"

    disk_data = []
    for line in run_cmd("df -h --output=target,pcent,used,avail,size").splitlines()[1:]:
        parts = line.split()
        if len(parts) >= 5 and parts[0].startswith("/"):
            mnt  = parts[0]
            try:
                pct = float(parts[1].rstrip("%"))
            except ValueError:
                continue
            disk_data.append((mnt, pct, parts[2], parts[3], parts[4]))

    load_raw = Path("/proc/loadavg").read_text().split()
    load1, load5, load15 = float(load_raw[0]), float(load_raw[1]), float(load_raw[2])
    cores = os.cpu_count() or 1
    load_pct1 = min(100.0, load1 / cores * 100)

    console.print(Panel(
        f"[bold]Script started:[/bold] {datetime.datetime.utcfromtimestamp(SCRIPT_START).strftime('%Y-%m-%d %H:%M:%S')} UTC\n"
        f"[bold]Current time:  [/bold] {datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')} UTC\n"
        f"[bold]Elapsed:       [/bold] [yellow]{runtime_str()}[/yellow]",
        title="⏰  Runtime Clock", border_style="blue", padding=(0, 2)
    ))

    console.print("\n[bold yellow]  CPU[/bold yellow]")
    for label, pct in cpu_data[:9]:
        tag = "[bold]ALL[/bold]" if label == "cpu" else label
        console.print(f"  {tag:>6}  {_bar(pct)}")

    console.print(f"\n[bold yellow]  Load Average[/bold yellow]  "
                  f"(cores={cores}):  "
                  f"[cyan]1m={load1}[/cyan]  [cyan]5m={load5}[/cyan]  [cyan]15m={load15}[/cyan]")
    console.print(f"  {'load':>6}  {_bar(load_pct1)}")

    console.print(f"\n[bold yellow]  Memory[/bold yellow]")
    console.print(f"  {'RAM':>6}  {_bar(mem_pct)}  {_kb(mem_used)} / {_kb(mem_total)}")
    if swap_total:
        console.print(f"  {'Swap':>6}  {_bar(swap_pct)}  {_kb(swap_used)} / {_kb(swap_total)}")

    console.print(f"\n[bold yellow]  Disk Partitions[/bold yellow]")
    for mnt, pct, used, avail, size in disk_data[:8]:
        console.print(f"  {mnt:>20}  {_bar(pct)}  {used} used / {size}  ({avail} free)")

    log_event("graphs", "Resource graphs displayed",
              {"load1": load1, "mem_pct": round(mem_pct, 1)})


# ══════════════════════════════════════════════════════════════════════════════
#  RUNNER IDENTITY
# ══════════════════════════════════════════════════════════════════════════════

def _fetch_imdsv2(path: str, token: str, timeout: int = 2) -> str:
    try:
        return subprocess.check_output(
            ["curl", "-sf", "-H", f"X-aws-ec2-metadata-token: {token}",
             f"http://169.254.169.254/latest/meta-data/{path}"],
            text=True, timeout=timeout).strip()
    except Exception:
        return "unavailable"


def _get_imdsv2_token(timeout: int = 2) -> str:
    try:
        return subprocess.check_output(
            ["curl", "-sf", "-X", "PUT",
             "-H", "X-aws-ec2-metadata-token-ttl-seconds: 60",
             "http://169.254.169.254/latest/api/token"],
            text=True, timeout=timeout).strip()
    except Exception:
        return ""


def _get_public_ip() -> dict:
    try:
        raw = subprocess.check_output(
            ["curl", "-sf", "--max-time", "4",
             "http://ip-api.com/json?fields=status,message,country,countryCode,"
             "region,regionName,city,zip,lat,lon,timezone,isp,org,as,query"],
            text=True, timeout=5).strip()
        data = json.loads(raw)
        if data.get("status") == "success":
            return data
    except Exception:
        pass
    return {"query": "unavailable", "country": "unknown", "city": "unknown",
            "regionName": "unknown", "isp": "unknown", "timezone": "unknown"}


def _get_local_ip() -> str:
    try:
        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        s.connect(("8.8.8.8", 80))
        ip = s.getsockname()[0]; s.close(); return ip
    except Exception:
        return "unavailable"


def _get_linux_user_data() -> dict:
    try:
        pw = pwd.getpwuid(os.getuid())
        all_groups = [{"gid": g.gr_gid, "name": g.gr_name}
                      for g in grp.getgrall()
                      if pw.pw_name in g.gr_mem or g.gr_gid == pw.pw_gid]
        return {"username": pw.pw_name, "uid": pw.pw_uid, "gid": pw.pw_gid,
                "gecos": pw.pw_gecos, "home_dir": pw.pw_dir, "shell": pw.pw_shell,
                "groups": all_groups, "euid": os.geteuid(), "egid": os.getegid(),
                "is_root": os.geteuid() == 0}
    except Exception as e:
        return {"error": str(e)}


def _get_ssh_session() -> dict:
    return {k: os.environ.get(v, "") for k, v in [
        ("ssh_client", "SSH_CLIENT"), ("ssh_connection", "SSH_CONNECTION"),
        ("ssh_tty", "SSH_TTY"), ("term", "TERM"), ("tmux", "TMUX"),
        ("sudo_user", "SUDO_USER"), ("sudo_command", "SUDO_COMMAND"), ("logname", "LOGNAME")
    ]}


def _get_aws_identity() -> dict:
    result = {}
    token = _get_imdsv2_token()
    fetch = (lambda p: _fetch_imdsv2(p, token)) if token else (lambda p: "unavailable")
    for key, path in [("instance_id", "instance-id"), ("instance_type", "instance-type"),
                      ("ami_id", "ami-id"), ("local_ipv4", "local-ipv4"),
                      ("availability_zone", "placement/availability-zone"),
                      ("region", "placement/region")]:
        result[key] = fetch(path)
    iam_role = fetch("iam/security-credentials/")
    result["iam_role"] = iam_role if iam_role and "unavailable" not in iam_role else "none"
    try:
        sts = json.loads(subprocess.check_output(
            ["aws", "sts", "get-caller-identity", "--output", "json"],
            text=True, timeout=5, stderr=subprocess.DEVNULL))
        result.update({"aws_account_id": sts.get("Account",""), "aws_arn": sts.get("Arn",""),
                        "aws_user_id": sts.get("UserId","")})
    except Exception:
        result.update({"aws_account_id": "unavailable", "aws_arn": "unavailable",
                        "aws_user_id": "unavailable"})
    return result


def collect_runner_identity() -> dict:
    console.print("[dim]  Collecting runner identity...[/dim]", end="")
    geo = _get_public_ip()
    identity = {
        "script_start_utc": now_utc(),
        "hostname": socket.gethostname(), "fqdn": socket.getfqdn(),
        "os": platform.platform(), "kernel": platform.release(),
        "python": platform.python_version(),
        "local_ip": _get_local_ip(), "public_ip": geo.get("query", "unavailable"),
        "geo": {"country": geo.get("country",""), "country_code": geo.get("countryCode",""),
                "region": geo.get("regionName",""), "city": geo.get("city",""),
                "latitude": geo.get("lat"), "longitude": geo.get("lon"),
                "timezone": geo.get("timezone",""), "isp": geo.get("isp",""),
                "org": geo.get("org",""), "as_number": geo.get("as","")},
        "linux_user": _get_linux_user_data(),
        "session": _get_ssh_session(),
        "aws": _get_aws_identity(),
    }
    console.print(" [green]done[/green]")
    return identity


def show_runner_identity():
    ri = RUNNER_IDENTITY
    geo = ri.get("geo", {}); lu = ri.get("linux_user", {})
    aws = ri.get("aws", {}); ssh = ri.get("session", {})
    groups_str = ", ".join(f"{g['name']}({g['gid']})" for g in lu.get("groups", []))
    ssh_src = ""
    if ssh.get("ssh_client"):
        parts = ssh["ssh_client"].split()
        ssh_src = f"{parts[0]}:{parts[1]}" if len(parts) >= 2 else ssh["ssh_client"]
    loc_str = ", ".join(filter(None, [geo.get("city"), geo.get("region"), geo.get("country")])) or "unknown"

    lines = [
        f"[bold white]Hostname:[/bold white]      {ri.get('hostname')} ({ri.get('fqdn')})",
        f"[bold white]OS/Kernel:[/bold white]     {ri.get('os')} — {ri.get('kernel')}",
        f"[bold white]Started:[/bold white]       {ri.get('script_start_utc')}  |  Elapsed: [yellow]{runtime_str()}[/yellow]",
        "", f"[bold cyan]── Network ──────────────────────────[/bold cyan]",
        f"[bold white]Local IP:[/bold white]      {ri.get('local_ip')}",
        f"[bold white]Public IP:[/bold white]     {ri.get('public_ip')}",
        f"[bold white]Location:[/bold white]      {loc_str}  |  TZ: {geo.get('timezone')}",
        f"[bold white]ISP/Org:[/bold white]       {geo.get('isp')} / {geo.get('org')}",
        "", f"[bold cyan]── Linux User ───────────────────────[/bold cyan]",
        f"[bold white]Username:[/bold white]      {lu.get('username')} (uid={lu.get('uid')}, euid={lu.get('euid')})",
        f"[bold white]Root?:[/bold white]         {'[red]YES — running as root![/red]' if lu.get('is_root') else '[green]No[/green]'}",
        f"[bold white]Home/Shell:[/bold white]    {lu.get('home_dir')} | {lu.get('shell')}",
        f"[bold white]Groups:[/bold white]        {groups_str or '(none)'}",
        f"[bold white]SUDO_USER:[/bold white]     {ssh.get('sudo_user') or '(not sudo)'}",
        "", f"[bold cyan]── SSH Session ──────────────────────[/bold cyan]",
        f"[bold white]SSH From:[/bold white]      {ssh_src or '(local / not SSH)'}",
        f"[bold white]TTY/TERM:[/bold white]      {ssh.get('ssh_tty') or '(none)'} | {ssh.get('term')}",
        f"[bold white]tmux:[/bold white]          {'yes' if ssh.get('tmux') else 'no'}",
        "", f"[bold cyan]── AWS Identity ─────────────────────[/bold cyan]",
        f"[bold white]Instance:[/bold white]      {aws.get('instance_id')} ({aws.get('instance_type')})",
        f"[bold white]Region/AZ:[/bold white]     {aws.get('region')} / {aws.get('availability_zone')}",
        f"[bold white]Account:[/bold white]       {aws.get('aws_account_id')}",
        f"[bold white]IAM Role:[/bold white]      {aws.get('iam_role')}",
        f"[bold white]STS ARN:[/bold white]       {aws.get('aws_arn')}",
        "", f"[bold cyan]── Log Folder ───────────────────────[/bold cyan]",
        f"[bold white]Log Dir:[/bold white]       {LOG_DIR}",
        f"[bold white]Run Log:[/bold white]       {LOG_FILE.name}",
        f"[bold white]Audit:  [/bold white]       {AUDIT_FILE.name}",
        f"[bold white]Deletions:[/bold white]     {DELETION_LOG.name}",
    ]
    console.print(Panel("\n".join(lines),
        title="[bold yellow]🔍  RUNNER IDENTITY — Who Is Running This Script?[/bold yellow]",
        border_style="yellow", padding=(1, 2)))


# ══════════════════════════════════════════════════════════════════════════════
#  DESCRIPTION
# ══════════════════════════════════════════════════════════════════════════════

def show_description():
    console.print(Panel("""
[bold yellow]Script Modes:[/bold yellow]

[bold green]1. Auto Mode[/bold green]   — runs all actions automatically (pre-approved lists only)
[bold green]2. Interactive Mode [Default][/bold green] — pauses and asks before every critical action

[bold cyan]Features v2.0:[/bold cyan]
  • [magenta]Dynamic Deletion[/magenta]              — files AND directories (recursive), safe + logged
  • [magenta]Docker Mode Selector[/magenta]          — choose Auto or Manual Docker cleanup at startup
  • [magenta]Unused Process Mode Selector[/magenta]  — independent kill mode per run (Auto/Manual/Dry-Run)
  • [magenta]ASCII Resource Graphs[/magenta]         — bar charts for CPU / RAM / Disk / Load
  • [magenta]Live Runtime Clock[/magenta]            — shows start time + elapsed at every step
  • [magenta]Runner Identity Logging[/magenta]       — IP, geo, IAM, Linux user, SSH session
  • [magenta]Unused Process Detection[/magenta]      — idle / zombie / sleeping processes
  • [magenta]Unified Log Folder[/magenta]            — per-run + master audit + deletion + unprocessed logs
  • [magenta]Restoration Hints[/magenta]             — deletion log records size & location for recovery
  • [magenta]Unprocessed Tracker[/magenta]           — final report of everything skipped or failed
""",
        title="[bold cyan]Production Cleanup & Monitoring v2.0[/bold cyan]",
        border_style="cyan"))


# ══════════════════════════════════════════════════════════════════════════════
#  SYSTEM MONITOR
# ══════════════════════════════════════════════════════════════════════════════

def system_monitor():
    t = time.time()
    console.print("\n[bold cyan]======= SYSTEM MONITOR =======[/bold cyan]")
    load = run_cmd("uptime | awk -F'load average:' '{print $2}'").strip()
    console.print(f"[bold yellow]Load Avg:[/bold yellow] {load}")
    console.print(run_cmd("free -h"))
    console.print(run_cmd("df -h | grep -E '^/dev|Filesystem'"))
    log_event("monitor", "System stats displayed")
    section_timer("System Monitor", t)


# ══════════════════════════════════════════════════════════════════════════════
#  TOP PROCESSES
# ══════════════════════════════════════════════════════════════════════════════

def show_top_processes():
    t = time.time()
    console.print("\n[bold cyan]======= TOP 20 CPU PROCESSES =======[/bold cyan]")
    cpu_output = run_cmd("ps -eo pid,ppid,comm,%cpu,%mem --sort=-%cpu | head -n 21").splitlines()
    _render_ps_table(cpu_output, "CPU")
    console.print("\n[bold cyan]======= TOP 20 MEMORY PROCESSES =======[/bold cyan]")
    mem_output = run_cmd("ps -eo pid,ppid,comm,%cpu,%mem --sort=-%mem | head -n 21").splitlines()
    _render_ps_table(mem_output, "MEM")
    log_event("monitor", "Top CPU/memory processes displayed")
    section_timer("Top Processes", t)
    return cpu_output, mem_output


def _render_ps_table(lines, sort_col):
    table = Table(show_header=True, header_style="bold magenta", box=box.SIMPLE)
    for col in ("PID", "PPID", "Command", "%CPU", "%MEM"):
        table.add_column(col, justify="right" if col != "Command" else "left", overflow="fold")
    for line in lines[1:]:
        parts = line.split(None, 4)
        if len(parts) == 5:
            table.add_row(*parts)
    console.print(table)


def kill_processes(process_list, label):
    pids = Prompt.ask(
        f"Enter PIDs to kill from [yellow]{label}[/yellow] list (space-separated, blank=skip)",
        default="")
    if pids:
        for pid in pids.split():
            try:
                os.kill(int(pid), 9)
                console.print(f"[green]  ✓ Killed PID {pid}[/green]")
                log_event("kill", f"Process {pid} killed from {label}")
            except Exception as e:
                console.print(f"[red]  ✗ Failed to kill {pid}: {e}[/red]")
                UNPROCESSED_ITEMS.append({"type": "kill", "pid": pid, "error": str(e)})


# ══════════════════════════════════════════════════════════════════════════════
#  UNUSED PROCESS MODE SELECTOR  ← NEW: independent of global mode
# ══════════════════════════════════════════════════════════════════════════════

def select_unused_process_mode() -> str:
    """
    Ask the user to choose a kill mode specifically for unused/idle processes.
    Completely independent of the global mode — runs every time this section
    is reached, allowing fine-grained control without touching other sections.

    Returns
    -------
    "1"  Auto      — kill all zombies immediately, log & skip the rest
    "2"  Manual    — interactive: choose by row, PID, or skip per process
    "3"  Dry-Run   — detect and display only, no kills performed
    """
    console.print(Panel(
        "[bold green]1. Auto[/bold green]      "
        "— kill all [red]zombie[/red] processes automatically; log sleeping/idle as unprocessed\n\n"
        "[bold yellow]2. Manual[/bold yellow]    "
        "— review each candidate: kill by row, enter PIDs, or skip individually\n\n"
        "[bold blue]3. Dry-Run[/bold blue]   "
        "— detect and [underline]display only[/underline]; no processes will be touched "
        "(safe for auditing)",
        title="[bold cyan]🔧  Unused Process Detection — Kill Mode[/bold cyan]",
        border_style="cyan",
        padding=(1, 3),
    ))
    choice = Prompt.ask(
        "  Select unused-process mode",
        choices=["1", "2", "3"],
        default="3",
    )
    labels = {"1": "Auto", "2": "Manual", "3": "Dry-Run"}
    console.print(
        f"[dim]  ✔  Unused-process mode set to: "
        f"[bold]{labels[choice]}[/bold][/dim]"
    )
    log_event("unused_proc_mode", f"Unused-process mode selected: {labels[choice]}",
              {"unused_proc_mode": choice})
    return choice


# ══════════════════════════════════════════════════════════════════════════════
#  UNUSED PROCESS DETECTION
# ══════════════════════════════════════════════════════════════════════════════

def detect_unused_processes():
    """
    Detect idle/zombie/sleeping processes and handle them according to a
    dedicated kill mode chosen right before scanning — independent of global mode.
    """
    t = time.time()
    console.print("\n[bold cyan]======= UNUSED / IDLE PROCESS DETECTION =======[/bold cyan]")

    # ── Pick the kill mode NOW, before scanning ──────────────────────────────
    u_mode = select_unused_process_mode()

    raw = run_cmd("ps -eo pid,stat,etime,%cpu,%mem,comm --no-headers").splitlines()
    unused = []
    for line in raw:
        parts = line.split(None, 5)
        if len(parts) < 6:
            continue
        pid, stat, etime, cpu, mem, comm = parts
        try: cpu_f = float(cpu)
        except ValueError: cpu_f = 0.0
        is_zombie     = stat.startswith("Z")
        is_long_sleep = stat.startswith("S") and cpu_f == 0.0 and _etime_minutes(etime) > 60
        is_idle       = stat.startswith("I")
        if is_zombie or is_long_sleep or is_idle:
            reason = ("Zombie" if is_zombie
                      else "Idle kernel thread" if is_idle
                      else f"Sleeping {_etime_minutes(etime):.0f}m, 0% CPU")
            unused.append((pid, stat, etime, cpu, mem, comm, reason))

    if not unused:
        console.print("[green]  ✓ No unused/zombie/idle processes found.[/green]")
        log_event("unused_procs", "None found")
        section_timer("Unused Process Detection", t)
        return

    # ── Render results table (always shown, even in dry-run) ─────────────────
    table = Table(
        title=f"Unused / Idle Processes — Mode: [bold cyan]{['','Auto','Manual','Dry-Run'][int(u_mode)]}[/bold cyan]",
        show_header=True,
        header_style="bold red",
        box=box.SIMPLE,
    )
    for col in ("#", "PID", "Stat", "Elapsed", "%CPU", "%MEM", "Command", "Reason", "Action"):
        table.add_column(col, overflow="fold")

    for i, (pid, stat, etime, cpu, mem, comm, reason) in enumerate(unused, 1):
        if u_mode == "3":                          # Dry-Run: show what WOULD happen
            action_hint = (
                "[yellow]Would kill (zombie)[/yellow]" if stat.startswith("Z")
                else "[dim]Would skip (non-zombie)[/dim]"
            )
        elif u_mode == "1":                        # Auto: label what will be done
            action_hint = (
                "[red]→ Kill[/red]" if stat.startswith("Z")
                else "[dim]→ Log & skip[/dim]"
            )
        else:                                      # Manual: pending user decision
            action_hint = "[cyan]Pending[/cyan]"
        table.add_row(str(i), pid, stat, etime, cpu, mem, comm, reason, action_hint)

    console.print(table)
    console.print(f"[yellow]  Found {len(unused)} unused/idle process(es).[/yellow]")
    log_event("unused_procs", f"Found {len(unused)}",
              {"pids": [u[0] for u in unused], "mode": u_mode})

    # ── Dispatch by mode ─────────────────────────────────────────────────────
    if u_mode == "3":
        _dryrun_unused(unused)
    elif u_mode == "1":
        _auto_kill_unused(unused)
    else:
        _interactive_kill_unused(unused)

    section_timer("Unused Process Detection", t)


def _dryrun_unused(unused: list):
    """Dry-Run: report only — no process is touched."""
    zombies  = [u for u in unused if u[1].startswith("Z")]
    sleepers = [u for u in unused if u[1].startswith("S")]
    idles    = [u for u in unused if u[1].startswith("I")]

    console.print(Panel(
        f"[bold]Dry-Run Summary[/bold] — no changes made\n\n"
        f"  [red]Zombies[/red]       : {len(zombies):>4}  "
        f"{'(would be killed in Auto mode)' if zombies else ''}\n"
        f"  [yellow]Long sleepers[/yellow] : {len(sleepers):>4}  "
        f"(would be logged as unprocessed)\n"
        f"  [dim]Idle threads[/dim]  : {len(idles):>4}  "
        f"(would be logged as unprocessed)\n\n"
        f"  Total detected : {len(unused)}",
        title="[bold blue]🔍  Dry-Run Result[/bold blue]",
        border_style="blue",
        padding=(0, 2),
    ))
    # Still record them so the unprocessed report reflects what was seen
    for pid, stat, etime, cpu, mem, comm, reason in unused:
        UNPROCESSED_ITEMS.append({
            "type": "dryrun_unused",
            "pid": pid,
            "stat": stat,
            "comm": comm,
            "reason": reason,
            "note": "Dry-run — not touched",
        })
    log_event("unused_procs_dryrun",
              f"Dry-run: {len(unused)} processes reported, none killed",
              {"zombies": len(zombies), "sleepers": len(sleepers), "idles": len(idles)})


def _etime_minutes(etime: str) -> float:
    try:
        days = 0
        if "-" in etime:
            d, etime = etime.split("-"); days = int(d)
        parts = list(map(int, etime.split(":")))
        h, m, s = (parts + [0, 0, 0])[:3] if len(parts) == 3 else (0, parts[0], parts[1]) if len(parts) == 2 else (0, 0, 0)
        return days * 1440 + h * 60 + m + s / 60
    except Exception:
        return 0.0


def _auto_kill_unused(unused):
    console.print("\n[bold red][Auto Mode] Killing zombie processes...[/bold red]")
    for pid, stat, etime, cpu, mem, comm, reason in unused:
        if stat.startswith("Z"):
            try:
                os.kill(int(pid), 9)
                console.print(f"[green]  ✓ Killed zombie PID {pid} ({comm})[/green]")
                log_event("kill_unused_auto", f"Zombie PID {pid} ({comm}) killed")
            except Exception as e:
                console.print(f"[red]  ✗ Could not kill {pid}: {e}[/red]")
                UNPROCESSED_ITEMS.append({"type": "kill_zombie", "pid": pid, "error": str(e)})
        else:
            console.print(f"[yellow]  ⚠  Logging PID {pid} ({comm}) — {reason}[/yellow]")
            UNPROCESSED_ITEMS.append({"type": "skip_unused", "pid": pid, "reason": reason})


def _interactive_kill_unused(unused):
    console.print("\n[bold yellow][Manual Mode] Choose how to handle unused processes:[/bold yellow]")
    console.print("  [1] Kill all zombies automatically  [2] Pick by row  [3] Enter PIDs  [4] Skip all")
    choice = Prompt.ask("Your choice", choices=["1","2","3","4"], default="4")
    if choice == "1":
        _auto_kill_unused(unused)
    elif choice == "2":
        nums = Prompt.ask("Row numbers to kill (space-separated)", default="")
        for n in nums.split():
            idx = int(n) - 1
            if 0 <= idx < len(unused):
                pid, _, _, _, _, comm, _ = unused[idx]
                _kill_pid(pid, comm, "row")
    elif choice == "3":
        pids = Prompt.ask("PIDs to kill (space-separated)", default="")
        pid_map = {u[0]: u[5] for u in unused}
        for pid in pids.split():
            _kill_pid(pid, pid_map.get(pid,"?"), "manual")
    else:
        console.print("[dim]  All unused processes skipped.[/dim]")
        UNPROCESSED_ITEMS.append({"type": "unused_procs", "count": len(unused), "reason": "user skipped all"})


def _kill_pid(pid, comm, source):
    try:
        os.kill(int(pid), 9)
        console.print(f"[green]  ✓ Killed PID {pid} ({comm})[/green]")
        log_event(f"kill_unused_{source}", f"PID {pid} ({comm})")
    except Exception as e:
        console.print(f"[red]  ✗ Failed to kill {pid}: {e}[/red]")
        UNPROCESSED_ITEMS.append({"type": "kill_fail", "pid": pid, "error": str(e)})


# ══════════════════════════════════════════════════════════════════════════════
#  EBS VOLUMES
# ══════════════════════════════════════════════════════════════════════════════

def list_ebs_volumes():
    t = time.time()
    console.print("\n[bold cyan]======= EBS VOLUMES =======[/bold cyan]")
    if run_cmd("which aws"):
        out = run_cmd(
            'aws ec2 describe-volumes --query '
            '"Volumes[*].{ID:VolumeId,State:State,Size:Size,AZ:AvailabilityZone}" '
            '--output table | head -n 22')
        console.print(out if out else "[yellow]No volumes found[/yellow]")
        log_event("ebs", "Volumes listed")
    else:
        console.print("[yellow]AWS CLI not installed. Skipping.[/yellow]")
    section_timer("EBS Volumes", t)


# ══════════════════════════════════════════════════════════════════════════════
#  DISK USAGE
# ══════════════════════════════════════════════════════════════════════════════

def disk_usage():
    t = time.time()
    path = Prompt.ask("Enter folder path to scan", default="/")
    console.print(f"\n[bold cyan]======= DISK USAGE ({path}) =======[/bold cyan]")
    usage = run_cmd(f"du -ah {path} 2>/dev/null | sort -rh | head -n 20")
    console.print(usage)
    log_event("disk", f"Top storage scanned in {path}")
    section_timer("Disk Usage", t)


# ══════════════════════════════════════════════════════════════════════════════
#  DYNAMIC DELETION  ← FIXED: handles files AND directories
# ══════════════════════════════════════════════════════════════════════════════

def _get_size(path: Path) -> int:
    """Return total size in bytes (works for both files and directories)."""
    try:
        if path.is_file() or path.is_symlink():
            return path.stat().st_size
        return sum(f.stat().st_size for f in path.rglob("*") if f.is_file())
    except Exception:
        return 0


def _human_size(size_bytes: int) -> str:
    for unit in ("B","KB","MB","GB","TB"):
        if size_bytes < 1024.0:
            return f"{size_bytes:.1f} {unit}"
        size_bytes /= 1024.0
    return f"{size_bytes:.1f} PB"


def _delete_path(target: str) -> bool:
    p = Path(target)
    if not p.exists() and not p.is_symlink():
        console.print(f"[yellow]  ⚠  Not found: {target}[/yellow]")
        UNPROCESSED_ITEMS.append({"type": "delete_notfound", "path": target})
        return False

    kind     = "symlink" if p.is_symlink() else ("directory" if p.is_dir() else "file")
    size_b   = _get_size(p)
    size_str = _human_size(size_b)
    parent   = str(p.parent)
    name     = p.name

    if kind == "directory":
        console.print(f"[bold red]  ⚠  {target} is a DIRECTORY ({size_str})[/bold red]")
        if not Confirm.ask(f"  Recursively delete entire directory [red]{target}[/red]?", default=False):
            console.print(f"[yellow]  ⚠  Skipped (directory deletion cancelled)[/yellow]")
            UNPROCESSED_ITEMS.append({"type": "delete_cancelled", "path": target, "kind": kind})
            return False

    try:
        if kind == "directory":
            shutil.rmtree(target)
        else:
            os.remove(target)

        console.print(f"[green]  ✓ Deleted {kind}: {target}  ({size_str})[/green]")
        record = {
            "ts":           now_utc(),
            "path":         target,
            "kind":         kind,
            "size_bytes":   size_b,
            "size_human":   size_str,
            "parent_dir":   parent,
            "name":         name,
            "restore_hint": f"Recover from backup at {parent}/{name} or check snapshots",
            "deleted_by":   RUNNER_IDENTITY.get("linux_user", {}).get("username", getpass.getuser()),
            "run_id":       RUN_TS,
        }
        DELETED_ITEMS.append(record)
        log_event("delete", f"Deleted {kind}: {target}",
                  {"kind": kind, "size_bytes": size_b, "path": target})
        return True

    except PermissionError as e:
        console.print(f"[red]  ✗ Permission denied: {target} — {e}[/red]")
        UNPROCESSED_ITEMS.append({"type": "delete_error", "path": target, "error": str(e)})
        return False
    except Exception as e:
        console.print(f"[red]  ✗ Error deleting {target}: {e}[/red]")
        UNPROCESSED_ITEMS.append({"type": "delete_error", "path": target, "error": str(e)})
        return False


def delete_files():
    t = time.time()
    console.print("\n[bold cyan]======= DYNAMIC FILE / DIRECTORY DELETION =======[/bold cyan]")
    raw = Prompt.ask(
        "Enter full paths to delete (space-separated, blank=skip)",
        default="")
    if not raw.strip():
        console.print("[dim]  Deletion skipped.[/dim]")
        return

    paths = raw.split()

    table = Table(title="Deletion Preview", show_header=True,
                  header_style="bold red", box=box.ROUNDED)
    table.add_column("#"); table.add_column("Path"); table.add_column("Type")
    table.add_column("Size", justify="right")
    for i, p in enumerate(paths, 1):
        pp = Path(p)
        if not pp.exists() and not pp.is_symlink():
            table.add_row(str(i), p, "[yellow]NOT FOUND[/yellow]", "—")
        else:
            kind = "symlink" if pp.is_symlink() else ("directory" if pp.is_dir() else "file")
            table.add_row(str(i), p, kind, _human_size(_get_size(pp)))
    console.print(table)

    if not Confirm.ask(
            f"[bold red]Proceed to delete these {len(paths)} path(s)?[/bold red]", default=False):
        console.print("[dim]  Deletion cancelled.[/dim]")
        UNPROCESSED_ITEMS.append({"type": "delete_batch_cancelled", "paths": paths})
        section_timer("Deletion", t)
        return

    for p in paths:
        _delete_path(p)

    DELETION_LOG.write_text(json.dumps({
        "run_id": RUN_TS,
        "deleted_at": now_utc(),
        "items": DELETED_ITEMS,
        "note": "Use restore_hint for recovery guidance."
    }, indent=2))
    console.print(f"\n[dim]  Deletion log saved → {DELETION_LOG}[/dim]")
    section_timer("Deletion", t)


# ══════════════════════════════════════════════════════════════════════════════
#  DOCKER CLEANUP  — with its own mode selector
# ══════════════════════════════════════════════════════════════════════════════

def select_docker_mode() -> str:
    console.print(Panel(
        "[bold green]1. Manual[/bold green]  — review and confirm each Docker cleanup step\n"
        "[bold yellow]2. Auto[/bold yellow]   — prune all stopped containers, dangling images, unused volumes automatically",
        title="[bold cyan]Docker Cleanup Mode[/bold cyan]",
        border_style="cyan", padding=(0, 2)))
    choice = Prompt.ask("Docker mode", choices=["1","2"], default="1")
    log_event("docker_mode", f"Docker mode selected: {choice}")
    return choice


def docker_cleanup():
    t = time.time()
    console.print("\n[bold cyan]======= DOCKER CLEANUP =======[/bold cyan]")
    if not docker_available():
        console.print("[yellow]  Docker not installed or not in PATH. Skipping.[/yellow]")
        log_event("docker", "Docker not available")
        section_timer("Docker Cleanup", t)
        return

    d_mode = select_docker_mode()
    _docker_stopped_containers(d_mode)
    _docker_dangling_images(d_mode)
    _docker_unused_volumes(d_mode)
    section_timer("Docker Cleanup", t)


def _docker_stopped_containers(mode: str):
    console.print("\n[bold yellow]── Stopped Containers ──[/bold yellow]")
    raw = run_cmd(
        'docker ps -a --filter "status=exited" --filter "status=created" '
        '--format "{{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Status}}\t{{.RunningFor}}"').strip()
    if not raw:
        console.print("  [green]✓ No stopped containers.[/green]"); return
    rows = [r.split("\t") for r in raw.splitlines() if r]
    table = Table(show_header=True, header_style="bold magenta", box=box.SIMPLE)
    for col in ("#","Container ID","Name","Image","Status","Stopped For"):
        table.add_column(col, overflow="fold")
    for i, r in enumerate(rows, 1):
        table.add_row(str(i), *r)
    console.print(table)
    log_event("docker_containers", f"Found {len(rows)} stopped containers")
    if mode == "2":
        out = run_cmd("docker container prune -f")
        console.print(out)
        log_event("docker_containers_auto", "All stopped containers pruned")
    else:
        choice = Prompt.ask("[1] Remove all  [2] Pick by row  [3] Skip",
                            choices=["1","2","3"], default="3")
        if choice == "1":
            console.print(run_cmd("docker container prune -f"))
            log_event("docker_containers", "All pruned (manual)")
        elif choice == "2":
            nums = Prompt.ask("Row numbers (space-separated)", default="")
            for n in nums.split():
                idx = int(n) - 1
                if 0 <= idx < len(rows):
                    cid = rows[idx][0]
                    out = run_cmd(f"docker rm -f {cid}")
                    status = "[green]✓[/green]" if out else "[red]✗[/red]"
                    console.print(f"  {status} Removed container {cid}")
                    log_event("docker_container_remove", f"Container {cid}")
        else:
            UNPROCESSED_ITEMS.append({"type": "docker_containers", "count": len(rows), "reason": "skipped"})


def _docker_dangling_images(mode: str):
    console.print("\n[bold yellow]── Dangling Images ──[/bold yellow]")
    raw = run_cmd(
        'docker images --filter "dangling=true" '
        '--format "{{.ID}}\t{{.Repository}}\t{{.Tag}}\t{{.Size}}\t{{.CreatedSince}}"').strip()
    if not raw:
        console.print("  [green]✓ No dangling images.[/green]"); return
    rows = [r.split("\t") for r in raw.splitlines() if r]
    table = Table(show_header=True, header_style="bold magenta", box=box.SIMPLE)
    for col in ("#","Image ID","Repository","Tag","Size","Created"):
        table.add_column(col, overflow="fold")
    for i, r in enumerate(rows, 1):
        table.add_row(str(i), *r)
    console.print(table)
    log_event("docker_images", f"Found {len(rows)} dangling images")
    if mode == "2":
        console.print(run_cmd("docker image prune -f"))
        log_event("docker_images_auto", "All dangling images pruned")
    else:
        choice = Prompt.ask("[1] Remove all  [2] Pick by row  [3] Skip",
                            choices=["1","2","3"], default="3")
        if choice == "1":
            console.print(run_cmd("docker image prune -f"))
        elif choice == "2":
            nums = Prompt.ask("Row numbers (space-separated)", default="")
            for n in nums.split():
                idx = int(n) - 1
                if 0 <= idx < len(rows):
                    iid = rows[idx][0]
                    out = run_cmd(f"docker rmi -f {iid}")
                    console.print(f"  {'[green]✓[/green]' if out else '[red]✗[/red]'} Removed image {iid}")
        else:
            UNPROCESSED_ITEMS.append({"type": "docker_images", "count": len(rows), "reason": "skipped"})


def _docker_unused_volumes(mode: str):
    console.print("\n[bold yellow]── Unused Volumes ──[/bold yellow]")
    raw = run_cmd('docker volume ls --filter "dangling=true" --format "{{.Name}}\t{{.Driver}}"').strip()
    if not raw:
        console.print("  [green]✓ No unused volumes.[/green]"); return
    rows = [r.split("\t") for r in raw.splitlines() if r]
    table = Table(show_header=True, header_style="bold magenta", box=box.SIMPLE)
    for col in ("#","Volume Name","Driver"):
        table.add_column(col, overflow="fold")
    for i, r in enumerate(rows, 1):
        table.add_row(str(i), *r)
    console.print(table)
    log_event("docker_volumes", f"Found {len(rows)} unused volumes")
    if mode == "2":
        console.print(run_cmd("docker volume prune -f"))
        log_event("docker_volumes_auto", "All unused volumes pruned")
    else:
        choice = Prompt.ask("[1] Remove all  [2] Pick by row  [3] Skip",
                            choices=["1","2","3"], default="3")
        if choice == "1":
            console.print(run_cmd("docker volume prune -f"))
        elif choice == "2":
            nums = Prompt.ask("Row numbers (space-separated)", default="")
            for n in nums.split():
                idx = int(n) - 1
                if 0 <= idx < len(rows):
                    vname = rows[idx][0]
                    out = run_cmd(f"docker volume rm {vname}")
                    console.print(f"  {'[green]✓[/green]' if out else '[red]✗[/red]'} Removed volume {vname}")
        else:
            UNPROCESSED_ITEMS.append({"type": "docker_volumes", "count": len(rows), "reason": "skipped"})


# ══════════════════════════════════════════════════════════════════════════════
#  UNPROCESSED ITEMS REPORT
# ══════════════════════════════════════════════════════════════════════════════

def show_unprocessed_report():
    console.print("\n[bold cyan]======= UNPROCESSED / SKIPPED ITEMS =======[/bold cyan]")
    if not UNPROCESSED_ITEMS:
        console.print("[green]  ✓ All items were processed successfully.[/green]")
    else:
        table = Table(title=f"⚠  {len(UNPROCESSED_ITEMS)} Unprocessed Item(s)",
                      show_header=True, header_style="bold yellow", box=box.ROUNDED)
        table.add_column("#"); table.add_column("Type"); table.add_column("Detail")
        for i, item in enumerate(UNPROCESSED_ITEMS, 1):
            detail = " | ".join(f"{k}={v}" for k, v in item.items() if k != "type")
            table.add_row(str(i), item.get("type","unknown"), detail)
        console.print(table)

    UNPROCESSED.parent.mkdir(parents=True, exist_ok=True)
    UNPROCESSED.write_text(json.dumps({
        "run_id": RUN_TS, "ts": now_utc(),
        "count": len(UNPROCESSED_ITEMS),
        "items": UNPROCESSED_ITEMS
    }, indent=2))
    console.print(f"[dim]  Unprocessed report → {UNPROCESSED}[/dim]")


# ══════════════════════════════════════════════════════════════════════════════
#  FINAL SUMMARY
# ══════════════════════════════════════════════════════════════════════════════

def show_summary():
    total = time.time() - SCRIPT_START
    start_dt = datetime.datetime.utcfromtimestamp(SCRIPT_START)
    end_dt   = datetime.datetime.utcnow()

    console.print(Panel(
        f"[bold white]Started:[/bold white]   {start_dt.strftime('%Y-%m-%d %H:%M:%S')} UTC\n"
        f"[bold white]Finished:[/bold white]  {end_dt.strftime('%Y-%m-%d %H:%M:%S')} UTC\n"
        f"[bold white]Duration:[/bold white]  [yellow]{runtime_str()}[/yellow]  ({total:.1f}s total)",
        title="⏰  Run Duration", border_style="yellow", padding=(0, 2)))

    table = Table(title="⏱  Section Timing Summary", show_header=True,
                  header_style="bold cyan", box=box.ROUNDED)
    table.add_column("Section", style="bold")
    table.add_column("Time", justify="right")
    table.add_column("% of Total", justify="right")
    for section, secs in SECTION_TIMINGS.items():
        pct = secs / total * 100 if total else 0
        bar = "█" * int(pct / 5)
        table.add_row(section, f"{secs:.1f}s", f"{bar} {pct:.0f}%")
    table.add_row("[bold yellow]TOTAL[/bold yellow]",
                  f"[bold yellow]{total:.1f}s[/bold yellow]", "")
    console.print(table)

    console.print(Panel(
        f"[bold white]Log folder:[/bold white]  {LOG_DIR}\n\n"
        f"  [cyan]{LOG_FILE.name}[/cyan]      — structured run events (NDJSON)\n"
        f"  [cyan]{AUDIT_FILE.name}[/cyan]  — master append-only audit trail\n"
        f"  [cyan]{DELETION_LOG.name}[/cyan]  — deleted items with restore hints\n"
        f"  [cyan]{UNPROCESSED.name}[/cyan] — skipped / failed items",
        title="📁  Audit Logs", border_style="green", padding=(0, 2)))


# ══════════════════════════════════════════════════════════════════════════════
#  MAIN
# ══════════════════════════════════════════════════════════════════════════════

def main():
    global RUNNER_IDENTITY

    init_log_dir()
    show_description()

    # ── Runner identity ──────────────────────────────────────────────────────
    console.print("\n[bold cyan]======= RUNNER IDENTITY =======[/bold cyan]")
    RUNNER_IDENTITY = collect_runner_identity()
    show_runner_identity()
    log_event("startup", "Script started — runner identity collected")

    # ── Global mode selection ────────────────────────────────────────────────
    mode = Prompt.ask(
        "\n[bold]Global Mode[/bold]: [green]1=Auto[/green]  [yellow]2=Interactive[/yellow]",
        choices=["1","2"], default="2")
    log_event("mode_selected", f"mode={mode}")

    # ── Resource graphs ──────────────────────────────────────────────────────
    show_resource_graphs()

    # ── Standard sections ────────────────────────────────────────────────────
    system_monitor()
    cpu_list, mem_list = show_top_processes()
    list_ebs_volumes()

    # ── Unused processes — own mode selector, independent of global mode ─────
    detect_unused_processes()          # mode chosen interactively inside

    # ── Docker — own mode selector ───────────────────────────────────────────
    docker_cleanup()

    # ── Interactive-only ─────────────────────────────────────────────────────
    if mode == "2":
        kill_processes(cpu_list, "CPU")
        kill_processes(mem_list, "Memory")
        disk_usage()
        delete_files()

    # ── Final reports ─────────────────────────────────────────────────────────
    show_unprocessed_report()
    log_event("finish", "Cleanup completed",
              {"deleted_count": len(DELETED_ITEMS),
               "unprocessed_count": len(UNPROCESSED_ITEMS)})

    console.print("\n[bold green]   ✓ Cleanup Completed![/bold green]")
    show_summary()


if __name__ == "__main__":
    main()