_indZddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z  ddl m Z ddlmZddlmZdd lmZdd lmZej.ej0d ej2d ej4ej6gej8eZGddZdZ edk(re yy#e$r!d dl m Z d dlmZd dlmZd d lmZd d lmZYwxYw)z Quick scan orchestration script for Bitbucket SonarQube automation. Scans only the top 20 repositories with the most recent commits. N)Path)datetime)DictOptionalListTuple) ConfigManager)BitbucketClient) GitManager)SonarQubeScanner)ReportGeneratorz4%(asctime)s - %(name)s - %(levelname)s - %(message)szsonar_scan.log)levelformathandlersc eZdZdZdefdZdefdZddZde de e fd Z de de fd Zdde d e d e e ddfd ZdefdZddeededeefdZddZy)ScanOrchestratorzIOrchestrates the entire scanning workflow for quick scans (top 20 repos).configc*||_|jd}|jd}t|jd|jd|s|nd||_t |jd|_t |jd |jd |jd  |_t|jd |_ t|jd|_ |j|_ y)zt Initialize orchestrator. Args: config: Configuration manager instance bitbucket.api_tokenbitbucket.app_passwordbitbucket.workspacebitbucket.usernameN) workspaceusername app_password api_tokenscanning.repos_dir) repos_dirz sonarqube.urlsonarqube.tokenzscanning.sonar_scanner_path) sonar_url sonar_token scanner_pathscanning.reports_dir) reports_dirzscanning.history_file)rgetr bitbucketr git_managerr sonar_scannerr report_generatorr history_file _load_history scan_history)selfrrrs 2/home/batmin/projects/sonar_scan/src/main_quick.py__init__zScanOrchestrator.__init__+s JJ45 zz":; (jj!67ZZ 45-6D  &jj!56 .jj1 #45$AB  !0 #9:!  !,C!DE ..0returnc|jjr6 t|jd5}tj|cdddSiS#1swYiSxYw#t $r#}t jd|Yd}~iSd}~wwxYw)z!Load scan history from JSON file.rNzFailed to load scan history: )r+existsopenjsonload Exceptionloggerwarningr.fes r/r,zScanOrchestrator._load_historyRs    # # % D$++S1(Q99Q<(( (  D!>qcBCC  Ds4A A A AA A B )BB Nc t|jd5}tj|j|ddddy#1swYyxYw#t $r"}t jd|Yd}~yd}~wwxYw)zSave scan history to JSON file.w)indentNzFailed to save scan history: )r6r+r7dumpr-r9r:errorr<s r/ _save_historyzScanOrchestrator._save_history\sj >d''- : $++Qq9 : : : > LL8< = = >s3A#AAA  A A A;A66A; repo_slugc||jvr6|j|jd}|r tj|Sy#t$rYywxYw)z Get last scan date for a repository. Args: repo_slug: Repository slug/name Returns: Last scan datetime or None last_scanN)r-r&r fromisoformatr9)r.rFdate_strs r/_get_last_scan_datez$ScanOrchestrator._get_last_scan_datedsb )) )((377 DH#11(;;!sA AAct|jjd}|jsyt j j d}|d|d}t|j|}t|dkDS)z Check if a report exists for today's date for the repository. Args: repo_slug: Repository slug/name Returns: True if a report exists for today, False otherwise r$Fz%Y%m%d_z_*.pdfr) rrr&r5rnowstrftimelistgloblen)r.rFr% today_strpatternmatching_filess r/_has_report_for_todayz&ScanOrchestrator._has_report_for_todayws~4;;??+ABC !!#LLN++H5 Kq 62k..w78>"Q&&r1success report_pathctjj||r t|ndd|j|<|j y)z Update scan history for a repository. Args: repo_slug: Repository slug/name success: Whether scan was successful report_path: Path to generated report N)rHrWrX)rrN isoformatstrr-rE)r.rFrWrXs r/_update_scan_historyz%ScanOrchestrator._update_scan_historysE"113/:3{+( )$ r1c |jjdg}|sgSg}|D]N}t|trH|j drf|j ddj dd}|j d}t|dk(sh|\}}|j||d|idd |d gid |j d rk|j d dj dd}|j d}t|dk(s|\}}|j||d|idd |d d |d gid d|vs|j d}t|dk(s/|\}}d|d|d} |j||d|idd | d gid ]t|tso|jdxs|jd}|jd}|jdxs|jd} |s| st|trd|i}nJ|sHd| vrD| j ddj dd}d|vr|j ddnd} | rd| ini}|j|||xsidd | d gid Q|S)z Get additional repositories from configuration. These can be from different workspaces. Returns: List of repository info dictionaries z scanning.additional_repositorieszgit@bitbucket.org:.git/rAslugclonessh)namehref)rdrarlinkszhttps://github.com/httpsrdrssh_url clone_urlrN) rr& isinstancer[ startswithreplacesplitrRappenddict) r.additional_repos_configrepos repo_configpathpartsr repo_nameorgrhws_names r/_get_additional_repositoriesz-ScanOrchestrator._get_additional_repositoriess#'++//2TVX"Y&I2M K+s+))*>?&../CRHPPQWY[\D JJsOE5zQ/4, 9 $-$-*0))< '-2K$H*"& & !++,AB&../DbIQQRXZ\]D JJsOE5zQ).Y $-$-*0# '-2K$H-4k$J*"& & K''--c2E5zQ/4, 9$6yk9+T"R $-$-*0))< '-2G$D*"& & K.'OOF3N{v7N 'OOK8 %//)4T  8T!)S1%+Y$7 &/7:#*??3G#L#T#TU[]_#`DzAScanOrchestrator._get_repos_with_recent_commits..6s AaDr1T)keyreversez Selected z& repositories with most recent commitsrz z. z ... and z morezAdding z$ additional repositories from config)r:inforRr&rrnr'get_latest_commitrrIrlr}debugr9r;minsort enumerateextend)r.rqryrepos_with_datesrepos_without_dates repo_inforFworkspace_slug latest_commitcommit_date_str commit_dater>rMselected_reposis r/_get_repos_with_recent_commitsz/ScanOrchestrator._get_repos_with_recent_commitss/  0U  K*7*;*;F*C&.&<&<_=T=TUXZb=c&d &--*5*=*=T*=*JK(//i0HI '=i[;-%XY NN%Ei[P`#ab$++X\\9,EFE& CR .$?9I%8PQ 9)QQ iN 344Z[\  ).!*"Q& j^)|d0n3tj d?|d@ntjId5|dA|j4jK| |}|j4j;|}d}|rA|jdBr0d dl&}|jOdC|j4jQ|} |jRjU||||jdDE}tjdF||j#||jdBd||jdBr|dz }tjdH|n/|dz }tj!dI|d3|jdJCtjdMtjdNtjdOt |tjdP|tjdQ|tjdR|tjdMy#t8$r+}tj!d2|d3||dz }Yd}~d}~wwxYw#1swYxYw#t8$rYwxYw#t8$r(}tj!dG|d3|d}Yd}~d}~wwxYw#t8$r?}tj!dK|d3|dL|dz }|j#|dYd}~d}~wwxYw)Sz4Run the complete scanning workflow for top 20 repos.zKStarting SonarQube quick scan automation (top 20 repos with recent commits)z"No repositories found in workspaceNzFound z repositories in workspace)ryz%No repositories selected for scanningz Processing z) repositories (top 20 + additional repos)rrdraz'Skipping repository with no name/slug: zProcessing repository: rr_z*Repository folder is empty or missing for z , cloning...rc)protocolzNo SSH clone URL found for z , skippingr)rFrizFailed to clone FTzNo report found for today for z , will scanrrzUpdating repository: z Updating additional repository: zNo changes detected for z$, skipping (report exists for today)zChanges detected for z , updating...zFailed to update rM- z [^a-z0-9_] descriptionr^z(Generating sonar-project.properties for z (project key: )) repo_path project_key project_namez0Failed to generate sonar-project.properties for r~z@Could not extract project key from sonar-project.properties for z Project 'z-' does not exist in SonarQube, creating it...r4zutf-8)encodingzsonar.projectName=)rrrz(Successfully created SonarQube project: z (z$Failed to create SonarQube project 'z', but continuing with scanz' already exists in SonarQuberWroutput)ru scan_results sonar_results scan_outputzReport generated: zFailed to generate report for zSuccessfully scanned zScan failed for rDzUnexpected error processing exc_infoz<============================================================zQuick Scan Summary:z Total repositories processed: z Scanned: z Skipped: z Failed: )+r:rr'get_all_repositoriesr;rRrr&rrr5 get_clone_urlr(clone_or_updaterjtuplerDr\rVrKhas_changes_sinceget_sonar_properties_pathlowerrlresubr)generate_sonar_propertiesr9get_project_keycheck_project_existsr6striprkrmcreate_projectrrun_scantimesleepget_scan_resultsr*generate_report) r. all_reposrq scanned_count skipped_count failed_countrrFrris_empty_or_missingri clone_resultis_first_clonehas_today_reportris_additional_repolast_scan_date has_changessonar_props_pathrrrrrr>r=linerrrrXs r/rzScanOrchestrator.run_scanKs;  abNN779?R  NN? @  fS^,,FGH33IR3H NNB C  k#e*-VWX   S K < 1E!FG % 1 *3*:*:*<&<&aYQWEWD_D_Da@a#'KK"LYKWc de!% < $0 )-(,'A'A)'L$+ &DYK{$[\)2{B)G)K)KF)S-;t{{Od?e-e*1"KK*? {(KL"KK*J9+(VW*3{B)G)K)KF)S-;t{{Od?e-e*1-1-E-Ei-PN*...*J*J9Vd*eK#. & .FykQu,v w - 2 ("KK*? {-(XY"KK*J9+(VW!% < $0 ).$(#3#3#M#Mi#X ' $ 0E FI%.Kq "<"B"B"D"L"LSRU"V"^"^_bdg"hK"$&&["IK$-==#8#EIL"+-- r"B"HbKKK"J9+Udepdqqr st !+/+=+=+W+W&/(3)5,X,(#00@@AQR "NN%efoep#qr!Q&M))>>{KKK)K=8e fg$-==#8#EIL"+-- r"B"HbK!"2C'J*a()*#'::<#:#:;N#O37::c13Ea3H3N3N3PL$)**))88$/%1$/9  &N{m[]^j]kkl$mn)Mk]Zu'vwLL9[M9V!WX $11::9FVW #00@@AQR $ <#3#3I#>JJqM$($6$6$G$G $TM '"&"7"7"G"G"+%1&3$0$4$4X$> #H#K KK"4[M BC )) $$Y6  ##I.!Q&MKK"7 { CD A%LLL#3I;bAQAQRYAZ@[!\]]S %c'$&c' c4C*d8=AdBd80 c$9 cd8c$$d8'c1 ,c44 d=d8dd8 d5 d0*d80d55d88 f4e;;f)r2N)N)r)__name__ __module__ __qualname____doc__r r0rr,rEr[rrrKboolrVr\rPrxrintrrrr1r/rr(sS%1}%1Nt>SXh5G&'s't'0cDxX[}hl \d\|LDJLsLTXY]T^L\wr1rc t}|jds*tjdt j dt |jd}t |jd}|jds*tjdt j d|s,|s*tjdt j d|jd s*tjd t j d|jt|}|jy#t$r-tjd t j dYyt$r9}tjd |d t j dYd}~yd}~wwxYw)zMain entry point.rz"Bitbucket workspace not configuredrrrrz'Bitbucket username/email not configuredzOBitbucket authentication not configured (need either API token or app password)r zSonarQube token not configuredzScan interrupted by userz Fatal error: TrN) r r&r:rDsysexitr save_configrrKeyboardInterruptrr9)r has_api_tokenhas_app_password orchestratorr>s r/mainrEsB%zz/0 LL= > HHQKVZZ(=>?  +C DEzz./ LLB C HHQK%5 LLj k HHQKzz+, LL9 : HHQK (/   ./   }QC(4 8  sEE 3GG/F<<G__main__)!rrr7loggingpathlibrrtypingrrrrconfig_managerr bitbucket_clientr r(r sonarqube_scannerr r*r ImportError basicConfigINFO FileHandler StreamHandlerstdout getLoggerrr:rrrr1r/rs .. 2,0&20 ,, A,-cjj)   8 $ZZz'T zFy2-1'31 2sB>>#C$#C$