# Bitbucket SonarQube Daily Scan Automation

Automated system for monitoring Bitbucket repositories, detecting changes, running SonarQube scans, and generating PDF reports.

## Features

- **Automatic Repository Monitoring**: Fetches all repositories from a Bitbucket workspace using Bitbucket Cloud REST API 2.0
- **Read-Only API Access**: Uses read-only API calls (GET requests only) for security
- **Change Detection**: Only scans repositories with new commits since last scan
- **SonarQube Integration**: Runs SonarQube scans using existing `sonar-project.properties` files
- **PDF Report Generation**: Creates detailed PDF reports with scan results and metrics
- **History Tracking**: Tracks scan history to avoid redundant scans
- **Rate Limiting**: Respects Bitbucket API rate limits with automatic retry
- **Error Handling**: Continues processing other repositories if one fails

## Prerequisites

- Python 3.8 or higher
- SonarQube Scanner CLI installed and accessible in PATH (or provide full path)
- Git installed
- Bitbucket App Password
- SonarQube authentication token
- Access to SonarQube server at https://scan.appmonitor.co.za/

## Installation

1. **Clone or download this repository**

2. **Install Python dependencies**:
   ```bash
   pip install -r requirements.txt
   ```

3. **Install SonarQube Scanner**:
   - Download from [SonarQube Scanner](https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/)
   - Add to PATH or note the full path for configuration

4. **Configure environment variables**:
   - Copy `.env.example` to `.env`
   - Fill in your credentials:
     ```
     BITBUCKET_WORKSPACE=your_workspace_name
     BITBUCKET_USERNAME=your_email@example.com
     
     # Use either API Token OR App Password (not both)
     BITBUCKET_API_TOKEN=your_api_token
     # OR
     # BITBUCKET_APP_PASSWORD=your_app_password
     
     SONARQUBE_URL=https://scan.appmonitor.co.za/
     SONARQUBE_TOKEN=your_sonarqube_token
     ```
   - **Quick Setup**: Run `powershell -ExecutionPolicy Bypass -File setup_token.ps1` to configure interactively
   - **Authentication Options**:
     - **API Token**: Used for Bitbucket REST API calls (repository listing, commit checking)
     - **App Password**: **REQUIRED** for Git clone operations. API tokens don't work for Git.
       - When creating the Bitbucket App Password, select **read-only** scopes:
         - `repository:read` - Read access to repositories
         - `pullrequest:read` - Read access to pull requests (optional)
     - **Note**: Both can be configured - API token for API calls, App Password for Git operations

5. **Create configuration file** (optional):
   - Copy `config/config.example.yaml` to `config/config.yaml`
   - Or let the script prompt you during first run

## Usage

### Repository Verification

To verify all Bitbucket repositories are present and up-to-date:

```bash
python src/verify_repos.py
```

This script will:
- Fetch all repositories from Bitbucket (including additional repos from config)
- Compare with local repositories
- Clone missing repositories
- Update existing repositories
- Generate a summary report

### Pull all repos (git pull)

To run `git pull` in every repository under `repos/` (only in directories that are git repos):

```powershell
# From project root
.\scripts\git_pull_all_repos.ps1
```

Or double-click `scripts\git_pull_all_repos.bat`. Output is logged to `logs\git_pull_YYYYMMDD_HHmmss.log`.

### Manual Execution

Run the main script:

```bash
python src/main.py
```

The script will:
1. Load configuration from environment variables and config file
2. Prompt for any missing required values
3. Fetch all repositories from your Bitbucket workspace
4. Check each repository for changes since last scan
5. Clone/update repositories with changes
6. Run SonarQube scans
7. Generate PDF reports in the `reports/` directory
8. Update scan history

### First Run

On first run, the script will prompt you for:
- Bitbucket workspace name
- Bitbucket username
- Bitbucket app password
- SonarQube token

These values can also be provided via environment variables (recommended).

## Configuration

### Environment Variables

- `BITBUCKET_WORKSPACE`: Your Bitbucket workspace name
- `BITBUCKET_USERNAME`: Your Bitbucket username
- `BITBUCKET_APP_PASSWORD`: Your Bitbucket app password
- `SONARQUBE_URL`: SonarQube server URL (default: https://scan.appmonitor.co.za/)
- `SONARQUBE_TOKEN`: SonarQube authentication token
- `REPOS_DIR`: Directory for cloned repositories (default: ./repos)
- `REPORTS_DIR`: Directory for generated reports (default: ./reports)
- `HISTORY_FILE`: Path to scan history JSON file (default: ./scan_history.json)
- `SONAR_SCANNER_PATH`: Path to sonar-scanner command (default: sonar-scanner)

### Config File

You can also use `config/config.yaml` for non-sensitive configuration:

```yaml
bitbucket:
  workspace: "your_workspace"
  
sonarqube:
  url: "https://scan.appmonitor.co.za/"
  
scanning:
  repos_dir: "./repos"
  reports_dir: "./reports"
  history_file: "./scan_history.json"
  sonar_scanner_path: "sonar-scanner"
```

**Note**: Sensitive credentials (passwords, tokens) should always be in environment variables, not in config files.

## Project Structure

```
sonar_scan/
├── src/
│   ├── __init__.py
│   ├── main.py                 # Main orchestration script
│   ├── bitbucket_client.py     # Bitbucket API integration
│   ├── git_manager.py          # Git clone/update operations
│   ├── sonarqube_scanner.py    # SonarQube scanning logic
│   ├── report_generator.py     # PDF report generation
│   └── config_manager.py       # Configuration management
├── config/
│   └── config.example.yaml     # Configuration template
├── reports/                    # Generated PDF reports
├── repos/                      # Cloned repositories
├── requirements.txt            # Python dependencies
├── README.md                   # This file
└── .env.example                # Environment variables template
```

## How It Works

1. **Repository Discovery**: Fetches all repositories from Bitbucket workspace using the API
2. **Change Detection**: Compares last commit date with last scan timestamp from history file
3. **Repository Management**: Clones new repositories or updates existing ones
4. **SonarQube Scanning**: 
   - Verifies `sonar-project.properties` exists
   - Updates it with SonarQube server URL
   - Runs `sonar-scanner` command
   - Fetches results from SonarQube API
5. **Report Generation**: Creates PDF report with:
   - Scan information
   - Quality gate status
   - Code metrics
   - Error details (if any)
6. **History Update**: Records scan timestamp and results

## Requirements for Repositories

Each Bitbucket repository must have a `sonar-project.properties` file in the root directory with at minimum:

```properties
sonar.projectKey=your_project_key
sonar.sources=.
```

The script will automatically add:
```properties
sonar.host.url=https://scan.appmonitor.co.za/
```

## Scheduling (Future Enhancement)

For daily automated runs, you can:

1. **Windows Task Scheduler**: Create a scheduled task to run `python src/main.py` daily
2. **Python Schedule Library**: Modify the script to run continuously with scheduled checks
3. **Cron Job** (Linux/Mac): Add a cron entry for daily execution

## Troubleshooting

### SonarQube Scanner Not Found

If you get "sonar-scanner: command not found":
- Install SonarQube Scanner and add to PATH, or
- Set `SONAR_SCANNER_PATH` environment variable to full path

### Authentication Errors

- Verify Bitbucket app password has repository read permissions
- Verify SonarQube token is valid and has project analysis permissions

### No Changes Detected

- Check scan history file (`scan_history.json`) to see last scan dates
- Delete repository entry from history to force re-scan

### Missing sonar-project.properties

- Ensure all repositories have `sonar-project.properties` in root directory
- Repositories without this file will be skipped

### Plugin Download Failures

If you encounter errors like "Fail to download plugin [kotlin]":

- **Solution**: The batch files have been updated to include:
  - Trailing slash on SonarQube URL (`https://scan.appmonitor.co.za/`)
  - Increased download timeout (5 minutes: `-Dsonar.scanner.download.timeout=300000`)
  - Global scanner configuration updated in `sonar-scanner/conf/sonar-scanner.properties`

- **Regenerate batch files**: Run `python create_scan_batch_files.py` to update all batch files with the latest fixes

- **Manual fix**: Add `-Dsonar.scanner.download.timeout=300000` and ensure URL has trailing slash

## Logging

Logs are written to:
- Console (stdout)
- File: `sonar_scan.log`

Log level can be adjusted in `src/main.py` if needed.

## License

This project is provided as-is for internal use.