From 2ff4d593f83474357ff4d3b222c7843c91d25656 Mon Sep 17 00:00:00 2001 From: Luciabrightcode Date: Tue, 23 Dec 2025 12:18:11 +0800 Subject: [PATCH] Updated implementation_plan.md and walkthrough.md for approval before kicking off Phase4 --- implementation_plan.md | 73 ++++++++++++------ .../__pycache__/ghost_cursor.cpython-310.pyc | Bin 0 -> 5301 bytes src/extractor/client.py | 3 +- ..._ghost_cursor.cpython-310-pytest-9.0.2.pyc | Bin 0 -> 4627 bytes tests/unit/test_ghost_cursor.py | 59 ++++++++++++++ walkthrough.md | 17 +++- 6 files changed, 125 insertions(+), 27 deletions(-) create mode 100644 src/browser/__pycache__/ghost_cursor.cpython-310.pyc create mode 100644 tests/unit/__pycache__/test_ghost_cursor.cpython-310-pytest-9.0.2.pyc create mode 100644 tests/unit/test_ghost_cursor.py diff --git a/implementation_plan.md b/implementation_plan.md index 3ea6240..934f24c 100644 --- a/implementation_plan.md +++ b/implementation_plan.md @@ -1,29 +1,58 @@ -# Phase 3: Evasion & Resilience Implementation Plan (COMPLETED) +# Phase 4: Deployment & Optimization Implementation Plan ## Goal Description -Implement the "Human" behavior layer to defeat behavioral biometrics and temporal analysis. +Transition the system from a functional prototype to a scalable, production-ready extraction grid. This involves: +1. **Scaling**: Configuring Docker Compose for high concurrency (5 Browsers, 20 Extractors). +2. **Resilience**: Implementing persistent task queues and auto-recovery logic. +3. **Observability**: Integrating Prometheus metrics for monitoring health and success rates. -## Completed Changes +## User Review Required +> [!NOTE] +> **Monitoring**: We will add `prometheus` and `grafana` containers to `docker-compose.yml` to support the metrics collected by `src/core/monitoring.py`. +> **Task Loops**: We will introduce a new entry point `src/orchestrator/worker.py` to act as the persistent long-running process consuming from Redis. -### Browser Tier (Human Mimesis) -- **GhostCursorEngine** (`src/browser/ghost_cursor.py`): - - Implemented composite cubic Bezier curves. - - Implemented Fitts's Law velocity profiles. - - Added random micro-movements for human drift simulation. +## Proposed Changes -### Core Tier (Temporal & Network Entropy) -- **EntropyScheduler** (`src/core/scheduler.py`): - - Implemented Gaussian noise injection ($\sigma=5.0$). - - Implemented Phase shift rotation to prevent harmonic detection. -- **MobileProxyRotator** (`src/core/proxy.py`): - - Implemented Sticky Session logic. - - Implemented Cooldown management. +### Infrastructure +#### [UPDATE] [docker-compose.yml](file:///home/kasm-user/workspace/FAEA/docker-compose.yml) +- **Services**: + - `camoufox`: Scale to 5 replicas. Set `shm_size: 2gb`. Limit CPU/Mem. + - `extractor`: Scale to 20 replicas. Limit resources. + - `prometheus`: Add service for metrics collection. + - `grafana`: Add service for visualization. + - `redis`: Optimize config. -### Remediation: TLS Fingerprint Alignment -- **Tuned** `src/browser/manager.py`: Updated to trigger `Chrome/124`. -- **Tuned** `src/extractor/client.py`: Updated to use `chrome124` impersonation verify consistency. -- **Verified**: Static alignment achieved. Dynamic verification (`tests/manual/verify_tls.py`) confirms logic but faced prompt-specific network blocks. +### Core Tier (Orchestration & Monitoring) +#### [NEW] [src/core/monitoring.py](file:///home/kasm-user/workspace/FAEA/src/core/monitoring.py) +- **Class**: `MetricsCollector` +- **Metrics**: + - `auth_attempts` (Counter) + - `session_duration` (Histogram) + - `extraction_throughput` (Counter) -## Verification Status -- **Functional**: Components implemented and unit-testable. -- **TLS**: Aligned to Chrome 124 standard. +#### [NEW] [src/orchestrator/worker.py](file:///home/kasm-user/workspace/FAEA/src/orchestrator/worker.py) +- **Class**: `TaskWorker` +- **Features**: + - Infinite loop consuming tasks from Redis lists (`BLPOP`). + - Dispatch logic: `auth` -> `CamoufoxManager`, `extract` -> `CurlClient`. + - Integration with `SessionRecoveryManager` for handling failures. + +#### [NEW] [src/core/recovery.py](file:///home/kasm-user/workspace/FAEA/src/core/recovery.py) +- **Class**: `SessionRecoveryManager` +- **Features**: + - Handle `cf_clearance_expired`, `ip_reputation_drop`, etc. + +### Documentation +#### [UPDATE] [README.md](file:///home/kasm-user/workspace/FAEA/README.md) +- Add "Production Usage" section. +- Document how to scale and monitor. + +## Verification Plan + +### Automated Tests +- **Integration**: Verify Worker picks up task from Redis. +- **Metrics**: Verify `/metrics` endpoint is exposed and scraping. + +### Manual Verification +- `docker-compose up --scale camoufox=5 --scale extractor=20` to verify stability. +- Check Grafana dashboard for metric data flow. diff --git a/src/browser/__pycache__/ghost_cursor.cpython-310.pyc b/src/browser/__pycache__/ghost_cursor.cpython-310.pyc new file mode 100644 index 0000000000000000000000000000000000000000..76e722164aa7695d9e329703035344095b56f7d8 GIT binary patch literal 5301 zcmaJ_&2JmW72nw}E@f&wKB=2 zXP1)5ECHu-+f#dQivkkROM(6a1q!4^PC56srvfcn_}XKDHvPR>a!JW{mzcNn`DS+J z_kQmU!)mpp;rZo>_ky2a)U>}-VfJxQcoQkwhTs}!ZEYiavkj)o`i8FF#)g5n-Zned zhSjk*Y^Dtz)EMnT$Jub0_Mz4-o0=#NE;EgOoNYKCYhAXh-PO3st;ZU-_VtYtXZN+b z{RH!A>vcV~e%SA|MQXkuL~)&^<~_e98q_%Z)KGX6DZ0TlO=z?PaORNMaHufW z|Mlx%(euZ()~o9Iat%jVXa;c{dHy7`N#_`ecJud|B=~*(+Ui!=5vvdUsB@(s3Awr( z$_G);Z-~`9H*eoujbvkWQ--_L)S@+d4YkOX-XL{6FX#rb=grZwObFX3vEkXLzGY?7 zF*SR@Q0m0KY>C+0%X5PU4WB%;GJ^k?NYNi4K4%Hztgp#pjE~-DKf&i8pze^pi>-J^ z`$Qj^BP-Sqj6_e&Gl2d{Ein=+%bnH^*~pH8#2847?F2|Xr|q+SeP16HpsA3UiPh9G z)_^P=8Y3rm;$q_LGY%d5>}~DSg;5bbO9@Myq`0J!B~zn2<8H%?DtK(-0qu0QC7sEbld`dt9QXp1g1iapWj$HJTWsCr2j zu?ATsK^!9qWN8r-OMEk+lV<@$4u`1!rcjc_r3dW(516Jf`|{YLyue^M-FSh0w^{f5Co%!40^E}hwi3ueGJ`!n1U`fu%XBBA{U|HEu@GLsXx+7^X7tCA%uK0 zeEDf)JkwMxK$WaSba`quLfF<3?jv?DAl;&A{sED&78{x1J(d{zI+$p4WF=N&oYO`& zSP}j+v4I_WtI`>5Q9G@H*T9(LHhQblxXn4*AIt`iJ?%3p#q2;&N=flhAC;4G%(mgl z&ym(jY#vw~NX2Sw(uN+@q>xlG!cHoOOwC;bJ5o&{ubE5C11G5^b4~q_fhiqTR{6w) z`?7}p{A6>^1XBpv5j-FQ&CM8)03F`-`%x75UAHGBfU@b+`Q<8r<0$C#+cdBff@;CB z-B^ZgfHjE36qIrSJDlNyHvS2Do_bN9qgo<`)bc?r*HgV0z`lK8w3AvbnmaWbAp{6K zwf$ZXr&*uR&~upvFO#@J;v$J}L8L5B_4XdpAk{hEd;s;`7DB8NzM|)4`8l;juj9Ai zTl-wdmr21ZBxrS0_L65&{0J$!0AX7--C^hT^K9OnXBAdsi^hVnz~=RmZnEKbo{7%m zynFj@XEAwQLt#ZB2L4qiPdnx>uvk{3Jg%9T+<2hL3mhj5->LlTp$=bd zjcjg_=iN4tfb9bd9=LE0ew#)r3^_5pG(4~lUyQghhL7e>=Cy%WcwgYZ6g%92AI2zU z88-;6s-=SYib(|?-GX%(hZbh7j%o>$orGZf9N$$fGko;-QataFAu#4A28Uv#~|& zdxwIwS0eX)e|H9&imeEh3Z&~#u2a?~u|nbm#Mh^e@uT~?OHBB+rKm7vqxzh}u{?#| z>C)5{Z>mVc8=FNWiE<>Jc51f6)^&M?%1b237SgIW9-#bpYAK$E*GJ%x7W`Yg{0d@n9w% zRVaIsH%Ob_gC+F_$8F<9wCy29v>N@+u%^POn0xKB{rlT*wT23+3T%=m#0@ELv z|9&QiKO;D47*INS$Q4k`G$rh#WQveazuWPOG z73K>C^H#RQfBRyRnY4ek)@rKfpKEn<3U7H01JXjj8z83b)N9At_`7J96oaKE#fzyO zA!_k?YAf>H8@~r>4FMJ6;V4FQ)uO1jcMaht$pPebmA~#ig*b`}euos1=@18>0u$SY z2_8OgoM)8g_2K3J$HT{sat!dR0QfZ|azn2+hyf-K87y6#6!!j%4BRpsld=jbfft}r z)tM*Zt`m@JM%Q%ylQyd7Svyd(qyh*+1E?Li%*vqbS=|U1o}*i}JV*DqE!RCR1Lt;9 zP`I*&7jDhKCA$i_<5-P?Sw)6V$N;BqTN!`d$qt-0P5y|)CnP>4@naIt^`md2^)E;f zk>rYg7SUV{vHmpY=);$uac;AecqHnlVI}b zy+4VLngPds4o+y5%b*!g&2v55Uh*<^CVzmJTqE%&iJK60TZO~Q>ZI~bx2fq43FQLD z=b>(J6rsqMAku=G2Szl`45qicV;r8okOf>`CuqpfoA`yVpe3R}9CtMn)}>eU3Uk2b zXN-$%_{!7g^h_=5^?CaLqU(2r=cOeN>UqCS`HJT~?ECGkMOJ9xWMY{!{5Iw2f|pv& zcIe017AQ1QG$zUINMZs>ppu&;rcs;9(Zy4e!yv~I5&t6LBk6Mv~2c{?Zvx0%bnQ- zn;9XRJXJ!9`cU=34>k`-kr2;);H|2J_yc$tkU*jmvcxkF5u*IgnGa^&1hrMfT65;k zoiq3R?(x09-??7BUQzJt9s6eguB9k{qle;8!^1UP{u?kt5$Zr$SNX0Dbxl!~p|NhN z$s23k;y3oXjk`7|4a@6gywip8Kv}N{Q&_lHg)K_B*Mxdoamt4nSGnb=@$Bnco*%qA zk-jHyjyC%vw@0tCUkwc#*Q>bv_W&zU?kf=hMXJyuEmZc^Pz|*M?FU+t8XH0kr`A&{k|F@1yL!q!_u}EmBX_C8~D`( zS{tgUQ2&N@V2ElSFT{I0s3P7kCXYpLqTm5W_Pvg`>qTYUOGG8Iq&$DpQVk3E8Nv`{Ik^4cKmbP zvF!LxC5kJ3zd!PW?x^R+TCW|O!DKve9V6DqJ1sJ8GRtJnAJUF~q(0WJTyfOh77e$O zMMC4_kq}_vN)rEq-R--w)AL4w^aj2J^(eALPPJ>}oz9&}e-L0wu{H2|-GLuh&>OgZ(CNzfWXIp~ zCIiut?pSs<2i~3TpflyQLcVYwGWszg6E=f2Yc?pmU0K=g~p` z7G~XF=|@>t=nPW!PQS)nP+Q(zNeg6%BgH83*q-FJi~?3^iD zwA+|VXy0XAV)I@DcOKny{9Fc?mG(^VS0&}IN`b$ul*39%Z%J%a#nBFlpAU(#(Ec{H*I}eVhV*CmjcLM8 zIZ5Uena`2=JelXoya40O%P-&-YvUHKHm+q{E3qkEztbvGeOsQUkG@Ra1);bo&rlEh zw_}?=8}>!fqa|OW55IKe!wW@$SWEOVk(+jl#Ue2defC0;_Ib7>ZoSEX_3k;i_%x%0 z_lf)G^taTv*gk}PTYX2BH{j62mH%eY$~B<&L=DKQZ&aO0?M>C031zUL2i6JNon~mG z)-8o4|8Ah~quNG|s|qLt?NQXac3|#XT%WJveZs-7pp|p5DCHinkq1|n-y)B+HY;(P zs_$~{`F-}J3iC(&HmNYn+=jA<3M)t%m2@pE^S&oF(NVJ()x^Y}FX6H<4@1V_G z3G<=T#e49l`IwkbgZgPc908RjD#mPH2k5gy6fcYz!{*$qz^Q68O5e$J<5LhMSSLH%&I)t6rlnS z4}SO&esY0aaez2Y2Z(&E)5#G5V`a2EpOu_$C&$inNtl*cwZOhlAYPDVceLp`Gfce< zk%P;x)2FYKA+nU$VTwe$mJliPEgvURx@^m9xo6TfRf%7S)&5aJJ;j8Y!_Q`fBKl