mirror of
https://github.com/clash-verge-rev/clash-verge-rev.git
synced 2026-10-14 21:20:48 +08:00
test(perf): add macOS Tauri WebView performance harness
Drive the production home traffic graph through embedded WebDriver in an isolated optimized measurement build. Record attributed WebContent and main process CPU/RSS distributions across foreground, hidden and restored phases. Keep replay, raw reports, independent-run comparison, cleanup checks and focused validation available through project commands. Normal release builds exclude the automation service and measurement entry points.
This commit is contained in:
Tunglies
@@ -0,0 +1,140 @@
|
||||
# macOS Traffic Performance Harness
|
||||
|
||||
This guide is the operating and maintenance contract for humans and coding agents.
|
||||
The only scenario is the production home traffic graph in a dedicated, optimized
|
||||
Tauri WKWebView. It measures foreground updates, native hide, then restoration.
|
||||
It does not initialize the complete homepage or run Mihomo, Service or networking.
|
||||
|
||||
## Build And Run
|
||||
|
||||
Use this repository's pnpm, Rust and Xcode Command Line Tools on macOS. No global
|
||||
WebDriver installation is needed. Keep each binary beside its generated manifest.
|
||||
|
||||
```sh
|
||||
pnpm install --frozen-lockfile
|
||||
pnpm perf:build target/perf/baseline-build
|
||||
caffeinate -di pnpm perf:run -- --binary target/perf/baseline-build/clash-verge --output target/perf/baseline --repeat 3 --seconds 20 --warmup 10
|
||||
pnpm perf:compare -- target/perf/baseline target/perf/baseline
|
||||
pnpm perf:test
|
||||
```
|
||||
|
||||
`perf:build [directory]` uses release optimization, Vite `--mode perf` and the
|
||||
explicit Cargo `perf-harness` feature. It preserves the binary and records HEAD,
|
||||
dirty state, compiled source, lockfile, frontend, build-settings and binary hashes.
|
||||
Source changes during compilation reject the manifest. Each run also hashes the
|
||||
external collector and runner. Ordinary builds exclude the driver and measurement
|
||||
entry; debug assertions are not the gate. Do not edit compiled inputs during builds.
|
||||
|
||||
Output directories must be new and Git ignored; use `target/perf/`. JSON/CSV keep
|
||||
raw process samples, observations, phase markers, attribution and cleanup evidence.
|
||||
`summary.json` and `report.md` provide distributions by independent run, phase and
|
||||
process role. Failed runs exit nonzero and retain their evidence. The explicit
|
||||
`traffic-webview-v3` schema rejects older reports instead of silently migrating them.
|
||||
|
||||
## Isolation And Visibility
|
||||
|
||||
The dedicated measurement main bypasses normal initialization and shutdown. It
|
||||
does not read or write daily configuration, subscriptions, proxy, TUN or Service.
|
||||
It uses a distinct app identity and incognito WebView. Both automatic services
|
||||
bind loopback and are owned by the current run; never kill processes by name.
|
||||
|
||||
The measurement window stays above ordinary windows and covers part of the screen,
|
||||
without activation or mouse capture: clicks pass through to the application behind it.
|
||||
Native topmost and click-through states are verified, while native focus and application
|
||||
activation must remain false. WebDriver uses the embedded WebView directly; DOM focus
|
||||
is recorded but not required. Daily windows are unchanged; the topmost window exits
|
||||
with its owned measurement instance. You may keep using another application.
|
||||
The console must be unlocked. Lock state is checked before startup and recorded
|
||||
at phase boundaries; unknown state fails and observed drift rejects comparison.
|
||||
Boundary checks do not provide continuous lock monitoring or unlock the computer.
|
||||
|
||||
No focus does not mean no visibility: shown stages require `document.hidden=false`,
|
||||
fresh Worker data, advancing Canvas draw timestamps and changed Canvas pixels.
|
||||
Topmost placement avoids ordinary-window occlusion; system overlays or another Space
|
||||
can still pause WebKit. Such a run fails; do not weaken Canvas assertions to make it
|
||||
pass. `caffeinate -di` lasts only for the command and cannot prevent deliberate lock.
|
||||
Hidden stages still require continuous input and Worker data.
|
||||
|
||||
## Fixed Workload
|
||||
|
||||
The native window is 1000 x 700 logical pixels. Verify driver dimensions against
|
||||
DPR and record the actual CSS viewport; titlebar space is excluded. The graph is
|
||||
920 x 360 with a ten-minute range, default bezier style and real data retention.
|
||||
The measurement preload sets blur pause false without changing production defaults.
|
||||
|
||||
Deterministic loopback WebSocket replay enters the public traffic append boundary,
|
||||
then the production Worker and Canvas. It does not cover Mihomo or network traffic.
|
||||
There is no history prefill. Absolute 200 ms deadlines sustain 5 Hz without cumulative
|
||||
timer drift or catch-up bursts; missing a complete period fails the source.
|
||||
Warmup is at least the requested duration with a five-second readiness allowance.
|
||||
Fixed input anchors reserve two seconds for each native transition; measured stages
|
||||
retain their full duration. Readiness uses observable state and deadlines, not sleep.
|
||||
Before the first show, wait for actual replay input and Worker data so the graph's
|
||||
visibility listeners are installed; then require fresh Canvas output after showing.
|
||||
Total replay, including allowances, must remain below ten minutes for raw history.
|
||||
Two-input endpoint variation is 400 ms; larger differences reject comparison.
|
||||
|
||||
Worker snapshots are asynchronous. Count/latest-value checks verify deterministic
|
||||
upload/download values at the retained point count, with a joint period of 60 and
|
||||
received-minus-retained count below 60. This detects net missing/duplicate points,
|
||||
not every historical element or exactly cancelling losses and duplicates.
|
||||
Source age must be below 2500 ms, Worker age below 3500 ms and visible draw age below
|
||||
2500 ms. Future timestamps, lost sequence, process changes and failed restore fail.
|
||||
|
||||
## Metrics And Comparison
|
||||
|
||||
The native command obtains the actual WKWebView process ID using the private
|
||||
`_webProcessIdentifier` selector, with availability checking. External collection
|
||||
verifies the canonical system WebContent executable and process start identity.
|
||||
It handles PID reuse and exits; unsupported ownership fails closed. Main and WebView
|
||||
remain separate. Mihomo is not started; shared GPU/network attribution is unavailable.
|
||||
|
||||
CPU is cumulative process time converted from Mach ticks; interval percentages use
|
||||
adjacent valid samples, with one occupied core equal to 100%. The first difference
|
||||
is unavailable. Memory is RSS bytes, not physical footprint or exclusive memory.
|
||||
Reports include CPU mean/P50/P95/peak/seconds, RSS median/P95/peak/delta, sample count
|
||||
and time coverage. Memory slope requires at least 60 seconds of continuous samples.
|
||||
RSS growth is not proof of a leak; process statistics are not function profiles.
|
||||
The embedded driver adds a 20 Hz macOS runloop timer and observation overhead.
|
||||
|
||||
For A/B, build the candidate separately and alternate independent instances:
|
||||
|
||||
```sh
|
||||
pnpm perf:build target/perf/candidate-build
|
||||
for i in 1 2 3; do
|
||||
caffeinate -di pnpm perf:run -- --binary target/perf/baseline-build/clash-verge --repeat 1 --output target/perf/ab-baseline/$i
|
||||
caffeinate -di pnpm perf:run -- --binary target/perf/candidate-build/clash-verge --repeat 1 --output target/perf/ab-candidate/$i
|
||||
done
|
||||
pnpm perf:compare -- target/perf/ab-baseline target/perf/ab-candidate
|
||||
pnpm perf:run -- --binary target/perf/candidate-build/clash-verge --repeat 1 --seconds 5 --warmup 2 --fail-phase hidden --output target/perf/failed
|
||||
```
|
||||
|
||||
Comparison accepts summary files, run directories or groups of run directories.
|
||||
Build mode, toolchain, platform, host, power, WebKit, viewport and workload must match.
|
||||
Invalid and incomparable reports exit 2. A/A remains inconclusive. A/B compares
|
||||
independent-run summaries, not adjacent time samples. An optional third numeric
|
||||
argument is a percentage floor justified by baseline noise; there is no universal
|
||||
pass threshold. Fewer than three runs per group, overlapping ranges or differences
|
||||
within either group's range remain inconclusive. Background load limits inference.
|
||||
|
||||
## Automation Delivery Boundary
|
||||
|
||||
When the operator is a coding agent, the final change keeps only the
|
||||
performance-optimization code. Harness additions introduced during the
|
||||
work — measurement entries, configs, scripts, reports, manifests — must
|
||||
stay untracked by Git and out of every commit; keep them on ignored
|
||||
paths or remove them before delivery. Commits remain minimal and
|
||||
necessary for the optimization itself. Track or commit harness files
|
||||
only when a human explicitly requests it at the end.
|
||||
|
||||
## Maintenance And Acceptance
|
||||
|
||||
Keep measurement code behind the explicit build gates. Reuse the existing driver,
|
||||
Worker, graph, attribution and cleanup; do not add mock IPC or replace real WebView
|
||||
acceptance with Chrome/Vite. Preserve source/binary/report hashes and every failed run.
|
||||
After changes, run scoped ESLint, both perf TypeScript projects, `pnpm perf:test`,
|
||||
format checks and relevant production checks. Verify a real three-phase smoke,
|
||||
three independent runs, A/A, invalid/incomparable reports and injected-failure cleanup.
|
||||
Verify ordinary builds exclude measurement code and daily instance state is intact.
|
||||
Changes to contracts require a schema bump and new reports. Use bounded WebKit CPU
|
||||
or Allocation/Heap captures for deeper diagnosis; do not infer hotspots from counters.
|
||||
@@ -0,0 +1,84 @@
|
||||
use objc2::{class, msg_send, runtime::AnyObject, sel};
|
||||
use tauri::WebviewWindow;
|
||||
|
||||
#[tauri::command]
|
||||
async fn perf_state(window: WebviewWindow, action: String) -> Result<serde_json::Value, String> {
|
||||
if !matches!(action.as_str(), "hide" | "show" | "state") {
|
||||
return Err("unknown measurement action".into());
|
||||
}
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
window
|
||||
.with_webview(move |view| unsafe {
|
||||
let native = &*(view.ns_window() as *const AnyObject);
|
||||
let sender = std::ptr::null::<AnyObject>();
|
||||
if action == "show" {
|
||||
let _: () = msg_send![native, orderBack: sender];
|
||||
} else if action == "hide" {
|
||||
let _: () = msg_send![native, orderOut: sender];
|
||||
}
|
||||
let visible: bool = msg_send![native, isVisible];
|
||||
let focused: bool = msg_send![native, isKeyWindow];
|
||||
let click_through: bool = msg_send![native, ignoresMouseEvents];
|
||||
let app: *const AnyObject = msg_send![class!(NSApplication), sharedApplication];
|
||||
let active: bool = msg_send![app, isActive];
|
||||
let webview = &*(view.inner() as *const AnyObject);
|
||||
let supported: bool = msg_send![webview, respondsToSelector: sel!(_webProcessIdentifier)];
|
||||
let pid: i32 = if supported {
|
||||
msg_send![webview, _webProcessIdentifier]
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let _ = tx.send((pid, visible, focused, active, click_through));
|
||||
})
|
||||
.map_err(|e| e.to_string())?;
|
||||
let (pid, visible, focused, active, click_through) = rx
|
||||
.recv_timeout(std::time::Duration::from_secs(5))
|
||||
.map_err(|e| e.to_string())?;
|
||||
if pid <= 0 {
|
||||
return Err("WKWebView process ownership unavailable".into());
|
||||
}
|
||||
let top = window.is_always_on_top().map_err(|e| e.to_string())?;
|
||||
Ok(serde_json::json!({"main": std::process::id(), "webview": pid,
|
||||
"visible": visible, "nativeFocused": focused, "appActive": active,
|
||||
"top": top, "clickThrough": click_through,
|
||||
"token": std::env::var("PERF_TOKEN").map_err(|e| e.to_string())?}))
|
||||
}
|
||||
pub fn run() -> std::process::ExitCode {
|
||||
#[cfg(feature = "clippy")]
|
||||
let context = tauri::test::mock_context(tauri::test::noop_assets());
|
||||
#[cfg(not(feature = "clippy"))]
|
||||
let context = tauri::generate_context!();
|
||||
let result = tauri::Builder::default()
|
||||
.plugin(tauri_plugin_wdio_webdriver::init())
|
||||
.invoke_handler(tauri::generate_handler![perf_state])
|
||||
.setup(|app| {
|
||||
app.handle().set_activation_policy(tauri::ActivationPolicy::Accessory)?;
|
||||
let port: u16 = std::env::var("PERF_REPLAY_PORT")?.parse()?;
|
||||
tauri::WebviewWindowBuilder::new(
|
||||
app,
|
||||
"perf",
|
||||
tauri::WebviewUrl::App(format!("index.html?port={port}").into()),
|
||||
)
|
||||
.title("Clash Verge - performance replay")
|
||||
.inner_size(1000., 700.)
|
||||
.visible(false)
|
||||
.focused(false)
|
||||
.always_on_top(true)
|
||||
.incognito(true)
|
||||
.build()?
|
||||
.set_ignore_cursor_events(true)?;
|
||||
Ok(())
|
||||
})
|
||||
.build(context);
|
||||
match result {
|
||||
Ok(mut app) => {
|
||||
app.set_activation_policy(tauri::ActivationPolicy::Prohibited);
|
||||
app.run(|_, _| {});
|
||||
}
|
||||
Err(error) => {
|
||||
eprintln!("{error}");
|
||||
return std::process::ExitCode::FAILURE;
|
||||
}
|
||||
}
|
||||
std::process::ExitCode::SUCCESS
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { createHash } from 'node:crypto'
|
||||
import {
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
} from 'node:fs'
|
||||
import { resolve } from 'node:path'
|
||||
const output = resolve(process.argv[2] ?? 'target/perf/build')
|
||||
const hash = (bytes) => createHash('sha256').update(bytes).digest('hex')
|
||||
const command = (cmd, args) =>
|
||||
execFileSync(cmd, args, { encoding: 'utf8' }).trim()
|
||||
const source = () =>
|
||||
command('git', ['ls-files', '-co', '--exclude-standard'])
|
||||
.split('\n')
|
||||
.filter(existsSync)
|
||||
.filter(
|
||||
(path) =>
|
||||
!path.startsWith('scripts/perf/') ||
|
||||
/\.(rs|tsx|html|mts|json)$/.test(path),
|
||||
)
|
||||
.sort()
|
||||
.map((path) => `${path}\0${hash(readFileSync(path))}`)
|
||||
.join('\n')
|
||||
mkdirSync(output, { recursive: true })
|
||||
const before = hash(source())
|
||||
const buildSettingsSha256 = hash(
|
||||
['Cargo.toml', '.cargo/config.toml']
|
||||
.filter(existsSync)
|
||||
.map((path) => `${path}\0${hash(readFileSync(path))}`)
|
||||
.join('\n'),
|
||||
)
|
||||
execFileSync(
|
||||
'pnpm',
|
||||
[
|
||||
'exec',
|
||||
'vite',
|
||||
'build',
|
||||
'--mode',
|
||||
'perf',
|
||||
'--config',
|
||||
'scripts/perf/vite.mts',
|
||||
],
|
||||
{ stdio: 'inherit' },
|
||||
)
|
||||
const mode = 'release + perf-harness (build settings hash recorded)'
|
||||
const buildCommand =
|
||||
'TAURI_CONFIG=<scripts/perf/tauri.json> cargo build -p clash-verge --release --features perf-harness --locked --target-dir target/perf/cargo'
|
||||
execFileSync(
|
||||
'cargo',
|
||||
[
|
||||
'build',
|
||||
'-p',
|
||||
'clash-verge',
|
||||
'--release',
|
||||
'--features',
|
||||
'perf-harness',
|
||||
'--locked',
|
||||
'--target-dir',
|
||||
'target/perf/cargo',
|
||||
],
|
||||
{
|
||||
stdio: 'inherit',
|
||||
env: {
|
||||
...process.env,
|
||||
TAURI_CONFIG: readFileSync('scripts/perf/tauri.json', 'utf8'),
|
||||
},
|
||||
},
|
||||
)
|
||||
if (before !== hash(source()))
|
||||
throw new Error('Source changed during build; refuse manifest')
|
||||
const binary = `${output}/clash-verge`
|
||||
copyFileSync('target/perf/cargo/release/clash-verge', binary)
|
||||
const files = command('find', ['target/perf/dist', '-type', 'f'])
|
||||
.split('\n')
|
||||
.sort()
|
||||
writeFileSync(
|
||||
`${binary}.json`,
|
||||
JSON.stringify(
|
||||
{
|
||||
head: command('git', ['rev-parse', 'HEAD']),
|
||||
dirty: command('git', ['status', '--porcelain']),
|
||||
sourceSha256: before,
|
||||
binarySha256: hash(readFileSync(binary)),
|
||||
frontendSha256: hash(
|
||||
files.map((path) => `${path}\0${hash(readFileSync(path))}`).join('\n'),
|
||||
),
|
||||
mode,
|
||||
buildSettingsSha256,
|
||||
buildEnvironment: Object.fromEntries(
|
||||
Object.entries(process.env).filter(([key]) =>
|
||||
/^(RUSTFLAGS|CARGO_ENCODED_RUSTFLAGS|CARGO_PROFILE_|RUSTC_WRAPPER)/.test(
|
||||
key,
|
||||
),
|
||||
),
|
||||
),
|
||||
buildCommand,
|
||||
binary,
|
||||
toolchain: `${command('rustc', ['--version'])}; node ${process.version}; pnpm ${command('pnpm', ['--version'])}`,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
)
|
||||
console.log(`Measurement binary and manifest: ${binary}`)
|
||||
@@ -0,0 +1,41 @@
|
||||
import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
|
||||
import { compare } from './report.mjs'
|
||||
const args = process.argv.slice(2).filter((v) => v !== '--')
|
||||
if (args.length < 2)
|
||||
throw new Error(
|
||||
'Usage: pnpm perf:compare -- baseline/summary.json candidate/summary.json [noise-threshold-percent]',
|
||||
)
|
||||
const threshold = args[2] === undefined ? null : Number(args[2])
|
||||
if (threshold !== null && (!Number.isFinite(threshold) || threshold < 0))
|
||||
throw new Error(
|
||||
'threshold must be a nonnegative percentage justified by baseline noise',
|
||||
)
|
||||
function load(path) {
|
||||
if (statSync(path).isFile()) return JSON.parse(readFileSync(path))
|
||||
if (existsSync(join(path, 'summary.json')))
|
||||
return load(join(path, 'summary.json'))
|
||||
const reports = readdirSync(path)
|
||||
.sort()
|
||||
.map((name) => load(join(path, name)))
|
||||
const first = reports[0]
|
||||
if (
|
||||
!first ||
|
||||
reports.some(
|
||||
(r) =>
|
||||
r.build?.binarySha256 !== first.build?.binarySha256 ||
|
||||
['invalid', 'incomparable'].includes(compare(first, r).verdict),
|
||||
)
|
||||
)
|
||||
throw new Error('group contains mixed builds, contracts, or invalid runs')
|
||||
return { ...first, runs: reports.flatMap((r) => r.runs) }
|
||||
}
|
||||
let result
|
||||
try {
|
||||
result = compare(...args.slice(0, 2).map(load), threshold)
|
||||
} catch (error) {
|
||||
result = { verdict: 'invalid', reason: String(error) }
|
||||
}
|
||||
console.log(JSON.stringify(result, null, 2))
|
||||
if (['invalid', 'incomparable'].includes(result.verdict)) process.exitCode = 2
|
||||
@@ -0,0 +1,52 @@
|
||||
import { ThemeProvider, createTheme } from '@mui/material'
|
||||
import { useEffect } from 'react'
|
||||
import { createRoot } from 'react-dom/client'
|
||||
import { SWRConfig } from 'swr'
|
||||
|
||||
import { EnhancedCanvasTrafficGraph } from '../../src/components/home/enhanced-canvas-traffic-graph'
|
||||
import { useTrafficMonitorEnhanced } from '../../src/hooks/use-traffic-monitor'
|
||||
import { initializeLanguage } from '../../src/services/i18n'
|
||||
import { setPreloadConfig } from '../../src/services/preload'
|
||||
|
||||
setPreloadConfig({
|
||||
pause_render_traffic_stats_on_blur: false,
|
||||
} as IVergeConfig)
|
||||
await initializeLanguage('en')
|
||||
export function Replay() {
|
||||
const {
|
||||
graphData: { appendData },
|
||||
} = useTrafficMonitorEnhanced({ subscribe: false })
|
||||
useEffect(() => {
|
||||
let received = 0,
|
||||
previous = 0,
|
||||
gaps = 0
|
||||
const socket = new WebSocket(
|
||||
`ws://127.0.0.1:${new URLSearchParams(location.search).get('port')}`,
|
||||
)
|
||||
socket.onmessage = ({ data }) => {
|
||||
const point = JSON.parse(data)
|
||||
appendData(point)
|
||||
if (previous && point.sequence !== previous + 1) gaps++
|
||||
previous = point.sequence
|
||||
document.documentElement.dataset.perfReceived = String(++received)
|
||||
document.documentElement.dataset.perfGaps = String(gaps)
|
||||
document.documentElement.dataset.perfSourceTime = String(point.timestamp)
|
||||
}
|
||||
return () => socket.close()
|
||||
}, [appendData])
|
||||
return (
|
||||
<main
|
||||
aria-label="Home traffic graph replay"
|
||||
style={{ width: 920, height: 360, margin: 32 }}
|
||||
>
|
||||
<EnhancedCanvasTrafficGraph />
|
||||
</main>
|
||||
)
|
||||
}
|
||||
createRoot(document.getElementById('root')!).render(
|
||||
<SWRConfig value={{ revalidateOnFocus: false, revalidateOnReconnect: false }}>
|
||||
<ThemeProvider theme={createTheme()}>
|
||||
<Replay />
|
||||
</ThemeProvider>
|
||||
</SWRConfig>,
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
<!doctype html><html><head><meta charset="utf-8"><title>Home traffic graph replay</title></head><body><div id="root"></div><script type="module" src="./entry.tsx"></script></body></html>
|
||||
@@ -0,0 +1,443 @@
|
||||
import { isDeepStrictEqual } from 'node:util'
|
||||
|
||||
export const SCHEMA = 'traffic-webview-v3'
|
||||
export const phases = ['foreground', 'hidden', 'restored']
|
||||
export const roles = ['webview', 'main']
|
||||
export const quantile = (values, p) => {
|
||||
const v = values.filter(Number.isFinite).sort((a, b) => a - b)
|
||||
if (!v.length) return null
|
||||
const index = (v.length - 1) * p
|
||||
return (
|
||||
v[Math.floor(index)] +
|
||||
(v[Math.ceil(index)] - v[Math.floor(index)]) * (index % 1)
|
||||
)
|
||||
}
|
||||
export function delta(previous, next) {
|
||||
if (
|
||||
!previous ||
|
||||
previous.pid !== next.pid ||
|
||||
previous.start !== next.start ||
|
||||
![previous.time, next.time, previous.cpu, next.cpu].every(
|
||||
Number.isFinite,
|
||||
) ||
|
||||
next.time <= previous.time ||
|
||||
next.cpu < previous.cpu
|
||||
)
|
||||
return null
|
||||
return ((next.cpu - previous.cpu) / (next.time - previous.time)) * 100000
|
||||
}
|
||||
export function summarize(rows, seconds) {
|
||||
const output = {}
|
||||
for (const phase of phases)
|
||||
for (const role of roles) {
|
||||
const selected = rows.filter((r) => r.phase === phase && r.role === role)
|
||||
const samples = selected.map((r, i) => ({
|
||||
...r,
|
||||
percent: delta(selected[i - 1], r),
|
||||
cpuDelta: i ? r.cpu - selected[i - 1].cpu : null,
|
||||
}))
|
||||
const valid = samples.filter((r) => Number.isFinite(r.percent))
|
||||
const memory = samples.map((r) => r.rss)
|
||||
const span =
|
||||
samples.length > 1 ? samples.at(-1).time - samples[0].time : 0
|
||||
const continuous =
|
||||
samples.length > 1 &&
|
||||
samples.every(
|
||||
(r) =>
|
||||
[r.time, r.cpu, r.rss].every(Number.isFinite) &&
|
||||
Number.isInteger(r.pid) &&
|
||||
r.pid > 0 &&
|
||||
/^\d+$/.test(r.start) &&
|
||||
r.time >= 0 &&
|
||||
r.cpu >= 0 &&
|
||||
r.rss > 0,
|
||||
) &&
|
||||
samples.every(
|
||||
(r, i) =>
|
||||
!i ||
|
||||
(r.time > samples[i - 1].time &&
|
||||
r.time - samples[i - 1].time < 2500 &&
|
||||
r.cpu >= samples[i - 1].cpu &&
|
||||
r.pid === samples[i - 1].pid &&
|
||||
r.start === samples[i - 1].start),
|
||||
)
|
||||
output[`${phase}/${role}`] = {
|
||||
count: samples.length,
|
||||
cpuCount: valid.length,
|
||||
missing: Math.max(0, Math.floor(seconds) - samples.length),
|
||||
coverage: Math.min(1, samples.length / seconds),
|
||||
continuous,
|
||||
observedSeconds: span / 1000,
|
||||
intervalCoverage: Math.min(1, span / ((seconds - 1) * 1000)),
|
||||
cpuMean: null,
|
||||
cpuP50: quantile(
|
||||
valid.map((r) => r.percent),
|
||||
0.5,
|
||||
),
|
||||
cpuP95: quantile(
|
||||
valid.map((r) => r.percent),
|
||||
0.95,
|
||||
),
|
||||
cpuPeak: quantile(
|
||||
valid.map((r) => r.percent),
|
||||
1,
|
||||
),
|
||||
cpuSeconds: valid.length
|
||||
? valid.reduce((sum, r) => sum + r.cpuDelta, 0)
|
||||
: null,
|
||||
rssP50: quantile(memory, 0.5),
|
||||
rssP95: quantile(memory, 0.95),
|
||||
rssPeak: quantile(memory, 1),
|
||||
rssDelta: continuous ? samples.at(-1).rss - samples[0].rss : null,
|
||||
rssSlopeBytesPerSecond:
|
||||
span >= 60000 && continuous
|
||||
? (samples.at(-1).rss - samples[0].rss) / (span / 1000)
|
||||
: null,
|
||||
}
|
||||
output[`${phase}/${role}`].cpuMean =
|
||||
valid.length && span > 0
|
||||
? (output[`${phase}/${role}`].cpuSeconds / span) * 100000
|
||||
: null
|
||||
}
|
||||
return output
|
||||
}
|
||||
export function replayValues(sequence) {
|
||||
return {
|
||||
up: 100000 + (sequence % 20) * 10000,
|
||||
down: 500000 + (sequence % 30) * 20000,
|
||||
}
|
||||
}
|
||||
export const validVisibility = (s) =>
|
||||
typeof s.hidden === 'boolean' &&
|
||||
(s.phase === 'hidden'
|
||||
? s.visible === false && s.hidden
|
||||
: phases.includes(s.phase) && s.visible === true && !s.hidden)
|
||||
export const validObservation = (s) =>
|
||||
validWorkload(s) &&
|
||||
validVisibility(s) &&
|
||||
s.nativeFocused === false &&
|
||||
s.appActive === false &&
|
||||
s.top === true &&
|
||||
s.clickThrough === true &&
|
||||
[s.now, s.source, s.worker, s.draw].every(
|
||||
(v) => Number.isFinite(v) && v > 0,
|
||||
) &&
|
||||
s.now >= s.source &&
|
||||
s.now - s.source < 2500 &&
|
||||
s.now >= s.worker &&
|
||||
s.now - s.worker < 3500 &&
|
||||
s.now >= s.draw &&
|
||||
(s.phase === 'hidden' || s.now - s.draw < 2500)
|
||||
export function validWorkload(s) {
|
||||
if (!s) return false
|
||||
const expected = replayValues(s.points)
|
||||
return (
|
||||
Number.isInteger(s.received) &&
|
||||
Number.isInteger(s.points) &&
|
||||
s.points > 0 &&
|
||||
s.received >= s.points &&
|
||||
s.received - s.points < 60 &&
|
||||
s.lastUp === expected.up &&
|
||||
s.lastDown === expected.down &&
|
||||
s.gaps === 0
|
||||
)
|
||||
}
|
||||
export function compare(a, b, threshold = null) {
|
||||
const stats = new Map()
|
||||
const text = (value) => typeof value === 'string' && value.length > 0
|
||||
const object = (value) =>
|
||||
value && typeof value === 'object' && !Array.isArray(value)
|
||||
const expectedMarks = [
|
||||
'ready',
|
||||
...phases.flatMap((phase) => [phase, `${phase}:end`]),
|
||||
]
|
||||
const valid = (x) =>
|
||||
x?.schema === SCHEMA &&
|
||||
text(x?.harnessSha256) &&
|
||||
[
|
||||
'head',
|
||||
'sourceSha256',
|
||||
'binarySha256',
|
||||
'frontendSha256',
|
||||
'buildSettingsSha256',
|
||||
'mode',
|
||||
'buildCommand',
|
||||
'binary',
|
||||
'toolchain',
|
||||
].every((key) => text(x.build?.[key])) &&
|
||||
typeof x.build.dirty === 'string' &&
|
||||
object(x.build.buildEnvironment) &&
|
||||
[
|
||||
'method',
|
||||
'mode',
|
||||
'toolchain',
|
||||
'buildSettingsSha256',
|
||||
'platform',
|
||||
'machine',
|
||||
'arch',
|
||||
'host',
|
||||
'hardware',
|
||||
'power',
|
||||
'powerSource',
|
||||
'webkit',
|
||||
'replay',
|
||||
'automation',
|
||||
].every((key) => text(x.contract?.[key])) &&
|
||||
object(x.contract.buildEnvironment) &&
|
||||
x.build.buildSettingsSha256 === x.contract.buildSettingsSha256 &&
|
||||
[
|
||||
'warmup',
|
||||
'intervalMs',
|
||||
'transitionSeconds',
|
||||
'readinessAllowanceSeconds',
|
||||
].every((key) => Number.isFinite(x.contract[key]) && x.contract[key] > 0) &&
|
||||
x.contract.scenario === 'traffic' &&
|
||||
typeof x.contract.consoleLocked === 'boolean' &&
|
||||
x.contract.windowPolicy === 'nonactivating-top-click-through-v1' &&
|
||||
x.contract.replayMs === 200 &&
|
||||
x.contract.rangeMinutes === 10 &&
|
||||
['window', 'viewport'].every(
|
||||
(key) =>
|
||||
Array.isArray(x.contract[key]) &&
|
||||
x.contract[key].length === (key === 'window' ? 2 : 3) &&
|
||||
x.contract[key].every((v) => Number.isFinite(v) && v > 0),
|
||||
) &&
|
||||
Number.isFinite(x.contract?.seconds) &&
|
||||
x.contract.seconds >= 5 &&
|
||||
Array.isArray(x.contract.phaseTargets) &&
|
||||
x.contract.phaseTargets.length === phases.length &&
|
||||
x.contract.phaseTargets.every((n) => Number.isInteger(n) && n > 0) &&
|
||||
Array.isArray(x.runs) &&
|
||||
x.runs.length &&
|
||||
x.runs.every((r) => {
|
||||
if (
|
||||
!r?.ok ||
|
||||
!Array.isArray(r.samples) ||
|
||||
!r.samples.every(
|
||||
(s) => s && roles.includes(s.role) && phases.includes(s.phase),
|
||||
) ||
|
||||
!Array.isArray(r.marks) ||
|
||||
!r.marks.every((m) => m && typeof m.name === 'string')
|
||||
)
|
||||
return false
|
||||
stats.set(r, summarize(r.samples, x.contract.seconds))
|
||||
return (
|
||||
r.cleanup?.mainExited &&
|
||||
r.cleanup?.driverClosed &&
|
||||
r.cleanup?.webviewExited &&
|
||||
Array.isArray(r.consoleStates) &&
|
||||
r.consoleStates.length === expectedMarks.length &&
|
||||
r.consoleStates.every(
|
||||
(s, i) =>
|
||||
s &&
|
||||
s.name === (i ? expectedMarks[i] : 'initial') &&
|
||||
typeof s.locked === 'boolean' &&
|
||||
Number.isFinite(s.time) &&
|
||||
s.time > 0 &&
|
||||
(!i || s.time >= r.consoleStates[i - 1].time),
|
||||
) &&
|
||||
Number.isInteger(r.replay?.sent) &&
|
||||
r.replay.sent > 0 &&
|
||||
Number.isFinite(r.replay.maxLateMs) &&
|
||||
r.replay.maxLateMs >= 0 &&
|
||||
r.replay.maxLateMs < x.contract.replayMs &&
|
||||
!r.replay.error &&
|
||||
r.actualWarmupMs >= x.contract.warmup * 1000 &&
|
||||
Array.isArray(r.observations) &&
|
||||
r.observations.length > 0 &&
|
||||
r.observations.every(
|
||||
(s, i) =>
|
||||
validObservation(s) && (!i || s.now > r.observations[i - 1].now),
|
||||
) &&
|
||||
r.marks.length === expectedMarks.length &&
|
||||
r.marks.every(
|
||||
(m, i) =>
|
||||
m.name === expectedMarks[i] &&
|
||||
Number.isFinite(m.time) &&
|
||||
m.time > 0 &&
|
||||
(!i ||
|
||||
(validObservation({ ...m, phase: m.name.split(':')[0] }) &&
|
||||
/^[a-f0-9]{64}$/.test(m.pixelsSha256))) &&
|
||||
(!phases.includes(m.name) ||
|
||||
(m.target === x.contract.phaseTargets[phases.indexOf(m.name)] &&
|
||||
m.received === m.target)) &&
|
||||
(!i || m.time > r.marks[i - 1].time),
|
||||
) &&
|
||||
roles.every((role) =>
|
||||
r.samples
|
||||
.filter((s) => s.role === role)
|
||||
.every(
|
||||
(s, i, rows) =>
|
||||
!i ||
|
||||
(s.pid === rows[i - 1].pid &&
|
||||
s.start === rows[i - 1].start &&
|
||||
s.time > rows[i - 1].time &&
|
||||
s.cpu >= rows[i - 1].cpu &&
|
||||
phases.indexOf(s.phase) >= phases.indexOf(rows[i - 1].phase)),
|
||||
),
|
||||
) &&
|
||||
validStats(stats.get(r)) &&
|
||||
phases.every((phase) => {
|
||||
const start = r.marks.find((m) => m.name === phase)
|
||||
const end = r.marks.find((m) => m.name === `${phase}:end`)
|
||||
const observed = r.observations.filter((s) => s.phase === phase)
|
||||
return (
|
||||
start &&
|
||||
end &&
|
||||
observed.length > 0 &&
|
||||
[
|
||||
start.deadlineIndex,
|
||||
end.deadlineIndex,
|
||||
start.senderSent,
|
||||
end.senderSent,
|
||||
].every((n) => Number.isInteger(n) && n >= 0) &&
|
||||
end.deadlineIndex > start.deadlineIndex &&
|
||||
end.senderSent >= start.senderSent &&
|
||||
end.senderSent <= r.replay.sent &&
|
||||
start.senderSent >= start.received &&
|
||||
end.senderSent >= end.received &&
|
||||
Math.abs(
|
||||
end.received -
|
||||
start.received -
|
||||
(end.deadlineIndex - start.deadlineIndex),
|
||||
) <= 2 &&
|
||||
observed.every(
|
||||
(s) =>
|
||||
s.now >= start.now &&
|
||||
s.now <= end.now &&
|
||||
s.received >= start.received &&
|
||||
s.received <= end.received,
|
||||
) &&
|
||||
end.worker > start.worker &&
|
||||
(phase === 'hidden' ||
|
||||
(end.draw > start.draw &&
|
||||
end.pixelsSha256 !== start.pixelsSha256))
|
||||
)
|
||||
})
|
||||
)
|
||||
})
|
||||
if (![a, b].every(valid))
|
||||
return { verdict: 'invalid', reason: 'failed, malformed or missing runs' }
|
||||
if (
|
||||
[a, b].some((x) =>
|
||||
x.runs.some((r) =>
|
||||
r.consoleStates.some((s) => s.locked !== x.contract.consoleLocked),
|
||||
),
|
||||
)
|
||||
)
|
||||
return {
|
||||
verdict: 'incomparable',
|
||||
reason: 'console lock state changed at observed boundaries',
|
||||
}
|
||||
if (
|
||||
!isDeepStrictEqual(a.contract, b.contract) ||
|
||||
['mode', 'buildCommand', 'toolchain', 'buildEnvironment'].some(
|
||||
(key) => !isDeepStrictEqual(a.build[key], b.build[key]),
|
||||
) ||
|
||||
a.harnessSha256 !== b.harnessSha256
|
||||
)
|
||||
return {
|
||||
verdict: 'incomparable',
|
||||
reason: 'harness, platform, build mode, workload or environment differ',
|
||||
}
|
||||
for (const phase of phases) {
|
||||
for (const name of [phase, `${phase}:end`]) {
|
||||
const sizes = [a, b].map((x) =>
|
||||
x.runs.map((r) => r.marks.find((m) => m.name === name)?.received),
|
||||
)
|
||||
if (!sizes.flat().every(Number.isFinite))
|
||||
return { verdict: 'invalid', reason: 'missing observed workload size' }
|
||||
if (
|
||||
Math.abs(quantile(sizes[0], 0.5) - quantile(sizes[1], 0.5)) > 2 ||
|
||||
sizes.some((s) => Math.max(...s) - Math.min(...s) > 2)
|
||||
)
|
||||
return {
|
||||
verdict: 'incomparable',
|
||||
reason: `${name}: input size varies by more than two replay points`,
|
||||
}
|
||||
}
|
||||
}
|
||||
const same = a.build.binarySha256 === b.build.binarySha256
|
||||
const enough = a.runs.length >= 3 && b.runs.length >= 3
|
||||
const metrics = {}
|
||||
for (const phase of phases)
|
||||
for (const role of roles)
|
||||
for (const metric of ['cpuMean', 'rssP50']) {
|
||||
const key = `${phase}/${role}`
|
||||
const av = a.runs.map((r) => stats.get(r)[key][metric])
|
||||
const bv = b.runs.map((r) => stats.get(r)[key][metric])
|
||||
const before = quantile(av, 0.5),
|
||||
after = quantile(bv, 0.5)
|
||||
const noise = Math.max(...av) - Math.min(...av)
|
||||
const candidateRange = Math.max(...bv) - Math.min(...bv)
|
||||
const overlap =
|
||||
Math.max(...av) >= Math.min(...bv) &&
|
||||
Math.max(...bv) >= Math.min(...av)
|
||||
metrics[`${key}/${metric}`] = {
|
||||
before,
|
||||
after,
|
||||
delta: after - before,
|
||||
baselineRange: noise,
|
||||
candidateRange,
|
||||
verdict:
|
||||
same ||
|
||||
!enough ||
|
||||
threshold === null ||
|
||||
!av.every(Number.isFinite) ||
|
||||
!bv.every(Number.isFinite) ||
|
||||
overlap ||
|
||||
Math.abs(after - before) <=
|
||||
Math.max(
|
||||
noise,
|
||||
candidateRange,
|
||||
(Math.abs(before) * threshold) / 100,
|
||||
)
|
||||
? 'inconclusive'
|
||||
: after < before
|
||||
? 'lower'
|
||||
: 'higher',
|
||||
}
|
||||
}
|
||||
return {
|
||||
verdict: 'inconclusive',
|
||||
kind: same ? 'A/A' : 'A/B',
|
||||
reason: same
|
||||
? 'same binary; natural variation only'
|
||||
: 'inspect independent-run metrics; no universal pass threshold',
|
||||
metrics,
|
||||
}
|
||||
}
|
||||
export function markdown(report) {
|
||||
const lines = [
|
||||
'# Home traffic graph - replay measurement',
|
||||
'',
|
||||
`Success: ${report.runs.filter((r) => r.ok).length}/${report.runs.length}`,
|
||||
'',
|
||||
'CPU: one fully occupied core = 100%. Memory: process resident size bytes, not exclusive memory. Null = unavailable; first CPU observation has no delta. Slope requires 60 seconds of continuous observations.',
|
||||
'',
|
||||
'| Run / stage / role | CPU mean / P50 / P95 / peak (%) | CPU s | RSS P50 / P95 / peak / delta (bytes) | Samples / coverage |',
|
||||
'|---|---|---|---|---|',
|
||||
]
|
||||
for (const [i, run] of report.runs.entries()) {
|
||||
if (run.error) lines.push(`\nRun ${i + 1} failed: ${run.error}\n`)
|
||||
for (const [key, s] of Object.entries(run.stats))
|
||||
lines.push(
|
||||
`| ${i + 1}/${key} | ${[s.cpuMean, s.cpuP50, s.cpuP95, s.cpuPeak].map((v) => v?.toFixed(2) ?? 'unavailable').join(' / ')} | ${s.cpuSeconds?.toFixed(3) ?? 'unavailable'} | ${[s.rssP50, s.rssP95, s.rssPeak, s.rssDelta].map((v) => v ?? 'unavailable').join(' / ')} | ${s.count} / ${(s.coverage * 100).toFixed(1)}% |`,
|
||||
)
|
||||
}
|
||||
lines.push(
|
||||
'',
|
||||
'Scope: loopback replay → public traffic append boundary → production Worker → homepage Canvas component. Mihomo and its network/subscription chain are not running. Shared GPU/network processes are unavailable and excluded. Process statistics do not identify function hotspots, component CPU or retained objects; memory growth is not proof of a leak.',
|
||||
'',
|
||||
'Next measurement: use WebKit CPU sampling for the highest-CPU foreground/restored phase; use Allocation/Heap captures for repeatable RSS growth. React Compiler is an untested local A/B candidate only.',
|
||||
)
|
||||
return lines.join('\n') + '\n'
|
||||
}
|
||||
export const validStats = (stats) =>
|
||||
Object.values(stats).every(
|
||||
(s) =>
|
||||
s.continuous &&
|
||||
s.coverage >= 0.8 &&
|
||||
s.intervalCoverage >= 0.8 &&
|
||||
s.cpuCount >= 3,
|
||||
)
|
||||
@@ -0,0 +1,340 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { mkdirSync } from 'node:fs'
|
||||
import { test } from 'node:test'
|
||||
|
||||
import {
|
||||
compare,
|
||||
delta,
|
||||
phases,
|
||||
roles,
|
||||
quantile,
|
||||
summarize,
|
||||
validWorkload,
|
||||
replayValues,
|
||||
SCHEMA,
|
||||
validVisibility,
|
||||
} from './report.mjs'
|
||||
const latest = (sequence) => {
|
||||
const { up, down } = replayValues(sequence)
|
||||
return { lastUp: up, lastDown: down }
|
||||
}
|
||||
const observation = (phase, now) => ({
|
||||
phase,
|
||||
now,
|
||||
source: now - 100,
|
||||
worker: now - 100,
|
||||
draw: now - 100,
|
||||
visible: phase !== 'hidden',
|
||||
hidden: phase === 'hidden',
|
||||
focused: false,
|
||||
nativeFocused: false,
|
||||
appActive: false,
|
||||
top: true,
|
||||
clickThrough: true,
|
||||
})
|
||||
const rows = phases.flatMap((phase, phaseIndex) =>
|
||||
roles.flatMap((role) =>
|
||||
Array.from({ length: 5 }, (_, i) => ({
|
||||
phase,
|
||||
role,
|
||||
pid: role === 'main' ? 1 : 2,
|
||||
start: '1',
|
||||
time: (phaseIndex * 5 + i) * 1000,
|
||||
cpu: (phaseIndex * 5 + i) / 10,
|
||||
rss: 100 + i,
|
||||
})),
|
||||
),
|
||||
)
|
||||
const run = {
|
||||
ok: true,
|
||||
samples: rows,
|
||||
cleanup: { mainExited: true, driverClosed: true, webviewExited: true },
|
||||
replay: { sent: 130, maxLateMs: 10 },
|
||||
actualWarmupMs: 3000,
|
||||
observations: phases.map((phase, i) => ({
|
||||
...observation(phase, 2000 + i * 10000),
|
||||
received: 35 + i * 35,
|
||||
points: 34 + i * 35,
|
||||
gaps: 0,
|
||||
...latest(34 + i * 35),
|
||||
})),
|
||||
marks: [
|
||||
{ name: 'ready', time: 1 },
|
||||
...phases.flatMap((name, i) => [
|
||||
{
|
||||
name,
|
||||
target: 35 + i * 35,
|
||||
deadlineIndex: 35 + i * 35,
|
||||
senderSent: 35 + i * 35,
|
||||
received: 35 + i * 35,
|
||||
points: 34 + i * 35,
|
||||
...latest(34 + i * 35),
|
||||
gaps: 0,
|
||||
time: 1000 + i * 10000,
|
||||
...observation(name, 1000 + i * 10000),
|
||||
pixelsSha256: 'a'.repeat(64),
|
||||
},
|
||||
{
|
||||
name: `${name}:end`,
|
||||
deadlineIndex: 60 + i * 35,
|
||||
senderSent: 60 + i * 35,
|
||||
received: 60 + i * 35,
|
||||
points: 59 + i * 35,
|
||||
...latest(59 + i * 35),
|
||||
gaps: 0,
|
||||
time: 6000 + i * 10000,
|
||||
...observation(name, 6000 + i * 10000),
|
||||
pixelsSha256: 'b'.repeat(64),
|
||||
},
|
||||
]),
|
||||
],
|
||||
}
|
||||
run.consoleStates = run.marks.map((m, i) => ({
|
||||
name: i ? m.name : 'initial',
|
||||
time: m.time,
|
||||
locked: false,
|
||||
}))
|
||||
const baseline = {
|
||||
schema: SCHEMA,
|
||||
harnessSha256: 'fixture-harness',
|
||||
build: {
|
||||
head: 'fixture-head',
|
||||
dirty: '',
|
||||
sourceSha256: 'fixture-source',
|
||||
binarySha256: 'same',
|
||||
frontendSha256: 'fixture-assets',
|
||||
buildSettingsSha256: 'fixture-settings',
|
||||
mode: 'release',
|
||||
buildCommand: 'fixture-build',
|
||||
binary: '/fixture/app',
|
||||
toolchain: 'fixture-tools',
|
||||
buildEnvironment: {},
|
||||
},
|
||||
contract: {
|
||||
scenario: 'traffic',
|
||||
replayMs: 200,
|
||||
consoleLocked: false,
|
||||
windowPolicy: 'nonactivating-top-click-through-v1',
|
||||
seconds: 5,
|
||||
warmup: 2,
|
||||
phaseTargets: [35, 70, 105],
|
||||
transitionSeconds: 2,
|
||||
readinessAllowanceSeconds: 5,
|
||||
intervalMs: 1000,
|
||||
rangeMinutes: 10,
|
||||
window: [1000, 700],
|
||||
viewport: [1000, 668, 2],
|
||||
method: 'fixture-method',
|
||||
mode: 'release',
|
||||
toolchain: 'fixture-tools',
|
||||
buildSettingsSha256: 'fixture-settings',
|
||||
platform: 'macOS',
|
||||
machine: 'fixture-model',
|
||||
arch: 'arm64',
|
||||
host: 'fixture-host',
|
||||
hardware: 'fixture-hardware',
|
||||
power: 'fixture-power',
|
||||
powerSource: 'AC',
|
||||
webkit: 'fixture-webkit',
|
||||
replay: 'fixture-5Hz',
|
||||
automation: 'fixture-wdio',
|
||||
buildEnvironment: {},
|
||||
},
|
||||
runs: [run, run, run],
|
||||
}
|
||||
test('quantiles, CPU units, first observation and PID reuse', () => {
|
||||
assert.equal(quantile([0, 10, 20], 0.95), 19)
|
||||
assert.equal(delta(undefined, rows[0]), null)
|
||||
assert.equal(delta(rows[0], rows[1]), 10)
|
||||
assert.equal(delta(rows[0], { ...rows[1], start: '2' }), null)
|
||||
assert.equal(delta(rows[0], { ...rows[1], time: NaN }), null)
|
||||
assert.equal(summarize(rows, 5)['hidden/main'].cpuMean, 10)
|
||||
assert.equal(summarize([], 5)['hidden/main'].cpuSeconds, null)
|
||||
})
|
||||
test('report schema: A/A, failed, incomparable and corrupt samples', () => {
|
||||
assert.equal(compare(baseline, baseline).kind, 'A/A')
|
||||
assert.equal(
|
||||
compare(baseline, { ...baseline, schema: 'old' }).verdict,
|
||||
'invalid',
|
||||
)
|
||||
for (const change of [
|
||||
{ appActive: true },
|
||||
{ nativeFocused: true },
|
||||
{ top: false },
|
||||
{ top: undefined },
|
||||
{ clickThrough: false },
|
||||
{ clickThrough: undefined },
|
||||
{ hidden: true },
|
||||
{ worker: 3000 },
|
||||
{ now: 6000 },
|
||||
{ draw: 3000 },
|
||||
{ worker: null },
|
||||
]) {
|
||||
const invalid = structuredClone(baseline)
|
||||
Object.assign(invalid.runs[0].observations[0], change)
|
||||
assert.equal(compare(baseline, invalid).verdict, 'invalid')
|
||||
}
|
||||
for (const key of ['draw', 'pixelsSha256']) {
|
||||
const stalled = structuredClone(baseline)
|
||||
stalled.runs[0].marks[2][key] = stalled.runs[0].marks[1][key]
|
||||
assert.equal(compare(baseline, stalled).verdict, 'invalid')
|
||||
}
|
||||
const missingPhase = structuredClone(baseline)
|
||||
missingPhase.runs[0].observations.pop()
|
||||
assert.equal(compare(baseline, missingPhase).verdict, 'invalid')
|
||||
const consoleDrift = structuredClone(baseline)
|
||||
consoleDrift.runs[0].consoleStates[3].locked = true
|
||||
assert.equal(compare(baseline, consoleDrift).verdict, 'incomparable')
|
||||
consoleDrift.runs[0].consoleStates[3].locked = null
|
||||
assert.equal(compare(baseline, consoleDrift).verdict, 'invalid')
|
||||
assert.equal(
|
||||
validVisibility({ phase: 'restored', visible: true, hidden: true }),
|
||||
false,
|
||||
)
|
||||
const missedPeriod = structuredClone(baseline)
|
||||
missedPeriod.runs[0].replay.maxLateMs = 200
|
||||
assert.equal(compare(baseline, missedPeriod).verdict, 'invalid')
|
||||
const delayed = {
|
||||
received: 100,
|
||||
points: 97,
|
||||
gaps: 0,
|
||||
lastUp: 270000,
|
||||
lastDown: 640000,
|
||||
}
|
||||
assert.equal(validWorkload(delayed), true)
|
||||
assert.equal(validWorkload(null), false)
|
||||
assert.equal(validWorkload({ ...delayed, points: 96 }), false)
|
||||
assert.equal(validWorkload({ ...delayed, points: 98 }), false)
|
||||
assert.equal(validWorkload({ ...delayed, received: 157 }), false)
|
||||
assert.equal(validWorkload({ ...delayed, lastDown: 660000 }), false)
|
||||
const missedAnchor = structuredClone(baseline)
|
||||
missedAnchor.runs[0].marks[1].received++
|
||||
assert.equal(compare(baseline, missedAnchor).verdict, 'invalid')
|
||||
const droppedInputs = structuredClone(baseline)
|
||||
const end = droppedInputs.runs[0].marks[2]
|
||||
end.received -= 10
|
||||
end.points -= 10
|
||||
Object.assign(end, latest(end.points))
|
||||
assert.equal(compare(droppedInputs, droppedInputs).verdict, 'invalid')
|
||||
const workerLag = structuredClone(baseline)
|
||||
workerLag.runs[0].observations[0].points -= 3
|
||||
assert.equal(compare(baseline, workerLag).verdict, 'invalid')
|
||||
const inputDrift = structuredClone(baseline)
|
||||
inputDrift.runs = baseline.runs.map((r) => structuredClone(r))
|
||||
inputDrift.runs[0].marks[2].received += 3
|
||||
inputDrift.runs[0].marks[2].points += 3
|
||||
inputDrift.runs[0].marks[2].deadlineIndex += 3
|
||||
inputDrift.runs[0].marks[2].senderSent += 3
|
||||
Object.assign(
|
||||
inputDrift.runs[0].marks[2],
|
||||
latest(inputDrift.runs[0].marks[2].points),
|
||||
)
|
||||
assert.equal(compare(baseline, inputDrift).verdict, 'incomparable')
|
||||
const snapshotPhase = structuredClone(baseline)
|
||||
snapshotPhase.runs[0].marks[2].points++
|
||||
Object.assign(
|
||||
snapshotPhase.runs[0].marks[2],
|
||||
latest(snapshotPhase.runs[0].marks[2].points),
|
||||
)
|
||||
assert.equal(compare(baseline, snapshotPhase).kind, 'A/A')
|
||||
const changedSettings = structuredClone(baseline)
|
||||
changedSettings.build.buildSettingsSha256 =
|
||||
changedSettings.contract.buildSettingsSha256 = 'changed-settings'
|
||||
assert.equal(compare(baseline, changedSettings).verdict, 'incomparable')
|
||||
assert.equal(
|
||||
compare(baseline, { ...baseline, harnessSha256: 'changed' }).verdict,
|
||||
'incomparable',
|
||||
)
|
||||
assert.equal(
|
||||
compare(baseline, {
|
||||
...baseline,
|
||||
contract: { ...baseline.contract, seconds: 6 },
|
||||
}).verdict,
|
||||
'incomparable',
|
||||
)
|
||||
assert.equal(
|
||||
compare(baseline, { ...baseline, runs: [{ ...run, ok: false }] }).verdict,
|
||||
'invalid',
|
||||
)
|
||||
const corrupt = structuredClone(baseline)
|
||||
corrupt.runs[0].samples[1].time = 0
|
||||
assert.equal(compare(baseline, corrupt).verdict, 'invalid')
|
||||
const shifted = structuredClone(baseline)
|
||||
shifted.runs[0].samples
|
||||
.filter((s) => s.phase === 'hidden')
|
||||
.forEach((s) => {
|
||||
s.time -= 5000
|
||||
})
|
||||
assert.equal(compare(baseline, shifted).verdict, 'invalid')
|
||||
const badMarks = structuredClone(baseline)
|
||||
badMarks.runs[0].marks.forEach((m) => {
|
||||
m.time = -1
|
||||
})
|
||||
assert.equal(compare(baseline, badMarks).verdict, 'invalid')
|
||||
const clustered = structuredClone(baseline)
|
||||
clustered.runs[0].samples.forEach((s) => {
|
||||
s.time /= 1000
|
||||
})
|
||||
assert.equal(compare(baseline, clustered).verdict, 'invalid')
|
||||
for (const [section, key] of [
|
||||
['contract', 'platform'],
|
||||
['build', 'sourceSha256'],
|
||||
]) {
|
||||
const missing = structuredClone(baseline)
|
||||
delete missing[section][key]
|
||||
assert.equal(compare(missing, missing).verdict, 'invalid')
|
||||
}
|
||||
assert.equal(
|
||||
compare(baseline, {
|
||||
...baseline,
|
||||
contract: { ...baseline.contract, platform: 'other-macOS' },
|
||||
}).verdict,
|
||||
'incomparable',
|
||||
)
|
||||
for (const malformed of [
|
||||
null,
|
||||
{},
|
||||
{ runs: [null] },
|
||||
{ ...baseline, runs: [{ ...run, samples: null }] },
|
||||
])
|
||||
assert.equal(compare(baseline, malformed).verdict, 'invalid')
|
||||
const noisy = structuredClone(baseline)
|
||||
noisy.runs = baseline.runs.map((r) => structuredClone(r))
|
||||
noisy.build.binarySha256 = 'different'
|
||||
noisy.runs.forEach((r, i) =>
|
||||
r.samples.forEach((s) => {
|
||||
s.cpu *= [0.1, 0.2, 10][i]
|
||||
}),
|
||||
)
|
||||
assert.equal(
|
||||
compare(baseline, noisy, 5).metrics['foreground/webview/cpuMean'].verdict,
|
||||
'inconclusive',
|
||||
)
|
||||
})
|
||||
test('macOS collector agrees with process.cpuUsage on a busy owned process', {
|
||||
skip: process.platform !== 'darwin',
|
||||
}, () => {
|
||||
mkdirSync('target/perf', { recursive: true })
|
||||
execFileSync('cc', [
|
||||
'-Wall',
|
||||
'-Wextra',
|
||||
'-Werror',
|
||||
'scripts/perf/sample.c',
|
||||
'-o',
|
||||
'target/perf/test-sample',
|
||||
])
|
||||
const sample = () =>
|
||||
Number(
|
||||
execFileSync('target/perf/test-sample', [String(process.pid)], {
|
||||
encoding: 'utf8',
|
||||
}).split(' ')[2],
|
||||
)
|
||||
const first = sample(),
|
||||
cpu = process.cpuUsage(),
|
||||
start = performance.now()
|
||||
while (performance.now() - start < 500) Math.sqrt(Math.random())
|
||||
const used = process.cpuUsage(cpu),
|
||||
measured = sample() - first
|
||||
assert.ok(Math.abs(measured - (used.user + used.system) / 1e6) < 0.05)
|
||||
})
|
||||
@@ -0,0 +1,673 @@
|
||||
import { spawn, execFileSync } from 'node:child_process'
|
||||
import { createHash, randomUUID } from 'node:crypto'
|
||||
import {
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
openSync,
|
||||
closeSync,
|
||||
realpathSync,
|
||||
} from 'node:fs'
|
||||
import { createServer } from 'node:net'
|
||||
import { resolve, dirname } from 'node:path'
|
||||
import { setTimeout as sleep } from 'node:timers/promises'
|
||||
import { parseArgs } from 'node:util'
|
||||
|
||||
import { createTauriCapabilities } from '@wdio/tauri-service'
|
||||
import { remote } from 'webdriverio'
|
||||
import { WebSocketServer } from 'ws'
|
||||
|
||||
import {
|
||||
delta,
|
||||
summarize,
|
||||
markdown,
|
||||
phases,
|
||||
roles,
|
||||
validStats,
|
||||
validWorkload,
|
||||
replayValues,
|
||||
SCHEMA,
|
||||
validObservation,
|
||||
validVisibility,
|
||||
} from './report.mjs'
|
||||
|
||||
const { values } = parseArgs({
|
||||
args: process.argv.slice(2).filter((v) => v !== '--'),
|
||||
options: {
|
||||
output: { type: 'string', default: `target/perf/runs/${Date.now()}` },
|
||||
binary: { type: 'string', default: 'target/perf/build/clash-verge' },
|
||||
repeat: { type: 'string', default: '3' },
|
||||
seconds: { type: 'string', default: '20' },
|
||||
warmup: { type: 'string', default: '10' },
|
||||
'fail-phase': { type: 'string' },
|
||||
},
|
||||
})
|
||||
const seconds = Number(values.seconds),
|
||||
warmup = Number(values.warmup),
|
||||
repeat = Number(values.repeat)
|
||||
const replayMs = 200
|
||||
if (process.platform !== 'darwin') throw new Error('macOS only')
|
||||
function readConsole() {
|
||||
const consoleState = JSON.parse(
|
||||
execFileSync('/usr/bin/plutil', ['-convert', 'json', '-o', '-', '-'], {
|
||||
input: execFileSync('/usr/sbin/ioreg', ['-n', 'Root', '-d1', '-a']),
|
||||
encoding: 'utf8',
|
||||
}),
|
||||
)
|
||||
const locked = consoleState.IOConsoleLocked
|
||||
if (typeof locked !== 'boolean')
|
||||
throw new Error('Environment unavailable: unknown console lock state')
|
||||
return locked
|
||||
}
|
||||
function checkConsole() {
|
||||
const locked = readConsole()
|
||||
if (locked)
|
||||
throw new Error(
|
||||
'Environment unavailable: an unlocked graphical session is required; window focus is not required',
|
||||
)
|
||||
return locked
|
||||
}
|
||||
const consoleLocked = checkConsole()
|
||||
if (
|
||||
![seconds, warmup, repeat].every(Number.isFinite) ||
|
||||
seconds < 5 ||
|
||||
warmup < 2 ||
|
||||
repeat < 1 ||
|
||||
!Number.isInteger(repeat)
|
||||
)
|
||||
throw new Error('Require seconds >= 5, warmup >= 2, integer repeat >= 1')
|
||||
const phaseTargets = phases.map(
|
||||
(_, i) =>
|
||||
Math.ceil(((warmup + 5) * 1000) / replayMs) +
|
||||
i * Math.ceil(((seconds + 2) * 1000) / replayMs),
|
||||
)
|
||||
if ((phaseTargets[2] * replayMs) / 1000 + seconds + 5 >= 600)
|
||||
throw new Error(
|
||||
'Total replay including transition allowance must be < 10 minutes',
|
||||
)
|
||||
const binary = realpathSync(values.binary),
|
||||
output = resolve(values.output)
|
||||
const webcontentBinary = realpathSync(
|
||||
'/System/Library/Frameworks/WebKit.framework/Versions/A/XPCServices/com.apple.WebKit.WebContent.xpc/Contents/MacOS/com.apple.WebKit.WebContent',
|
||||
)
|
||||
const build = JSON.parse(readFileSync(`${binary}.json`))
|
||||
const hash = (data) => createHash('sha256').update(data).digest('hex')
|
||||
if (hash(readFileSync(binary)) !== build.binarySha256)
|
||||
throw new Error('Binary does not match build manifest')
|
||||
try {
|
||||
execFileSync('git', ['check-ignore', '-q', '--no-index', output])
|
||||
} catch {
|
||||
throw new Error('Output directory must be Git ignored')
|
||||
}
|
||||
mkdirSync(dirname(output), { recursive: true })
|
||||
mkdirSync(output)
|
||||
execFileSync('cc', ['-O2', 'scripts/perf/sample.c', '-o', `${output}/sample`])
|
||||
const command = (name, args) =>
|
||||
execFileSync(name, args, { encoding: 'utf8' }).trim()
|
||||
const report = {
|
||||
schema: SCHEMA,
|
||||
build,
|
||||
contract: {
|
||||
scenario: 'traffic',
|
||||
replayMs,
|
||||
consoleLocked,
|
||||
method: 'mach-rusage-v0-seconds/rss-bytes-v1',
|
||||
mode: build.mode,
|
||||
buildSettingsSha256: build.buildSettingsSha256,
|
||||
buildEnvironment: build.buildEnvironment,
|
||||
toolchain: build.toolchain,
|
||||
platform: command('sw_vers', []),
|
||||
machine: command('sysctl', ['-n', 'hw.model']),
|
||||
arch: process.arch,
|
||||
host: hash(command('scutil', ['--get', 'LocalHostName'])),
|
||||
hardware: command('sysctl', ['hw.ncpu', 'hw.memsize']),
|
||||
power: command('pmset', ['-g', 'custom']),
|
||||
powerSource: command('pmset', ['-g', 'batt']).split('\n')[0],
|
||||
webkit: command('defaults', [
|
||||
'read',
|
||||
'/System/Library/Frameworks/WebKit.framework/Resources/Info',
|
||||
'CFBundleVersion',
|
||||
]),
|
||||
seconds,
|
||||
warmup,
|
||||
phaseTargets,
|
||||
transitionSeconds: 2,
|
||||
readinessAllowanceSeconds: 5,
|
||||
intervalMs: 1000,
|
||||
window: [1000, 700],
|
||||
windowPolicy: 'nonactivating-top-click-through-v1',
|
||||
rangeMinutes: 10,
|
||||
replay:
|
||||
'deterministic-v1, 5Hz, no prefill, append boundary, default bezier, blur pause false',
|
||||
automation:
|
||||
'wdio plugin 1.3.0; 20Hz macOS runloop pump plus one WebDriver observation/second',
|
||||
},
|
||||
runs: [],
|
||||
}
|
||||
report.units = {
|
||||
time: 'monotonic milliseconds',
|
||||
cpu: 'cumulative CPU seconds',
|
||||
percent: 'one CPU core = 100%',
|
||||
rss: 'resident size bytes',
|
||||
start: 'Mach process start ticks',
|
||||
}
|
||||
report.monotonicOriginEpochMs = performance.timeOrigin
|
||||
report.harnessSha256 = hash(
|
||||
['run.mjs', 'report.mjs', 'sample.c']
|
||||
.map((p) => readFileSync(`scripts/perf/${p}`))
|
||||
.join('\n'),
|
||||
)
|
||||
let interrupted = false
|
||||
const abort = new AbortController()
|
||||
const pause = (ms) => sleep(ms, undefined, { signal: abort.signal })
|
||||
for (const signal of ['SIGINT', 'SIGTERM'])
|
||||
process.on(signal, () => {
|
||||
interrupted = true
|
||||
abort.abort()
|
||||
})
|
||||
const assert = (value, reason) => {
|
||||
if (!value) throw new Error(reason)
|
||||
}
|
||||
const drawn = (s) => s.now >= s.draw && s.now - s.draw < 2500
|
||||
const marker = (s) => ({
|
||||
...s,
|
||||
pixelsSha256: hash(s.pixels),
|
||||
pixels: undefined,
|
||||
ua: undefined,
|
||||
token: undefined,
|
||||
})
|
||||
async function freePort() {
|
||||
const server = createServer()
|
||||
await new Promise((done) => server.listen(0, '127.0.0.1', done))
|
||||
const port = server.address().port
|
||||
await new Promise((done) => server.close(done))
|
||||
return port
|
||||
}
|
||||
for (let iteration = 0; iteration < repeat && !interrupted; iteration++) {
|
||||
const initialLocked = checkConsole()
|
||||
const token = randomUUID(),
|
||||
port = await freePort()
|
||||
const run = {
|
||||
token,
|
||||
ok: false,
|
||||
samples: [],
|
||||
observations: [],
|
||||
consoleStates: [
|
||||
{ name: 'initial', locked: initialLocked, time: Date.now() },
|
||||
],
|
||||
marks: [],
|
||||
stats: {},
|
||||
cleanup: {},
|
||||
replay: { sent: 0, maxLateMs: 0 },
|
||||
}
|
||||
report.runs.push(run)
|
||||
const replay = new WebSocketServer({ host: '127.0.0.1', port: 0 })
|
||||
await new Promise((done) => replay.on('listening', done))
|
||||
let sequence = 0,
|
||||
replayStarted,
|
||||
timer
|
||||
replay.on('connection', () => {
|
||||
if (replayStarted !== undefined) {
|
||||
run.replay.error = 'unexpected replay reconnection'
|
||||
clearTimeout(timer)
|
||||
return
|
||||
}
|
||||
replayStarted = performance.now()
|
||||
timer = setTimeout(send, replayMs)
|
||||
})
|
||||
const deadlineIndex = () =>
|
||||
Math.floor((performance.now() - replayStarted) / replayMs)
|
||||
function send() {
|
||||
const deadline = replayStarted + (sequence + 1) * replayMs
|
||||
const late = performance.now() - deadline
|
||||
if (late < 0) {
|
||||
timer = setTimeout(send, Math.ceil(-late))
|
||||
return
|
||||
}
|
||||
run.replay.maxLateMs = Math.max(run.replay.maxLateMs, late)
|
||||
if (late >= replayMs) {
|
||||
run.replay.error = `replay source missed a complete ${replayMs} ms period`
|
||||
return
|
||||
}
|
||||
sequence++
|
||||
run.replay.sent = sequence
|
||||
const timestamp = Date.now()
|
||||
const point = JSON.stringify({
|
||||
sequence,
|
||||
timestamp,
|
||||
...replayValues(sequence),
|
||||
})
|
||||
for (const socket of replay.clients)
|
||||
if (socket.readyState === 1) socket.send(point)
|
||||
timer = setTimeout(
|
||||
send,
|
||||
Math.max(
|
||||
0,
|
||||
replayStarted + (sequence + 1) * replayMs - performance.now(),
|
||||
),
|
||||
)
|
||||
}
|
||||
const log = openSync(`${output}/${iteration + 1}.log`, 'w')
|
||||
const child = spawn(binary, [], {
|
||||
env: {
|
||||
...process.env,
|
||||
PERF_TOKEN: token,
|
||||
PERF_REPLAY_PORT: String(replay.address().port),
|
||||
TAURI_WEBDRIVER_PORT: String(port),
|
||||
},
|
||||
stdio: ['ignore', log, log],
|
||||
})
|
||||
closeSync(log)
|
||||
let browser, childError, previous, mainStart
|
||||
const identities = {}
|
||||
child.on('error', (error) => {
|
||||
childError = error
|
||||
})
|
||||
const exited = new Promise((done) => child.once('exit', done))
|
||||
const alive = () =>
|
||||
!childError && child.exitCode === null && child.signalCode === null
|
||||
const collect = (pids) =>
|
||||
command(`${output}/sample`, pids.map(String))
|
||||
.split('\n')
|
||||
.map((line) => {
|
||||
const [pid, start, cpu, rss, ...path] = line.split(' ')
|
||||
return {
|
||||
pid: Number(pid),
|
||||
start,
|
||||
cpu: Number(cpu),
|
||||
rss: Number(rss),
|
||||
path: path.join(' '),
|
||||
time: performance.now(),
|
||||
}
|
||||
})
|
||||
const identify = (role, row) => {
|
||||
assert(
|
||||
row.path === (role === 'main' ? binary : webcontentBinary),
|
||||
`unexpected ${role} process executable: ${row.path}`,
|
||||
)
|
||||
assert(
|
||||
BigInt(row.start) >= BigInt(mainStart),
|
||||
'process predates owned application',
|
||||
)
|
||||
const identity = `${row.pid}/${row.start}`
|
||||
assert(
|
||||
!identities[role] || identities[role] === identity,
|
||||
'process identity changed during scenario',
|
||||
)
|
||||
identities[role] = identity
|
||||
}
|
||||
const state = async (action = 'state', pixels = false) =>
|
||||
browser.execute(
|
||||
async (action, pixels) => {
|
||||
const native = await window.__TAURI_INTERNALS__.invoke('perf_state', {
|
||||
action,
|
||||
})
|
||||
const canvas = document.querySelector('canvas')
|
||||
return {
|
||||
...native,
|
||||
now: Date.now(),
|
||||
hidden: document.hidden,
|
||||
focused: document.hasFocus(),
|
||||
received: Number(document.documentElement.dataset.perfReceived),
|
||||
source: Number(document.documentElement.dataset.perfSourceTime),
|
||||
gaps: Number(document.documentElement.dataset.perfGaps),
|
||||
worker: Number(document.documentElement.dataset.perfWorker),
|
||||
points: Number(document.documentElement.dataset.perfPoints),
|
||||
lastUp: Number(document.documentElement.dataset.perfLastUp),
|
||||
lastDown: Number(document.documentElement.dataset.perfLastDown),
|
||||
draw: Number(canvas?.dataset.perfDraw),
|
||||
pixels: pixels ? canvas?.toDataURL() : null,
|
||||
ua: navigator.userAgent,
|
||||
}
|
||||
},
|
||||
action,
|
||||
pixels,
|
||||
)
|
||||
const checked = async (action, pixels) => {
|
||||
assert(!interrupted, 'interrupted')
|
||||
assert(!run.replay.error, run.replay.error)
|
||||
assert(
|
||||
alive(),
|
||||
`owned app exited: ${childError ?? child.exitCode ?? child.signalCode}`,
|
||||
)
|
||||
const value = await state(action, pixels)
|
||||
assert(!run.replay.error, run.replay.error)
|
||||
run.lastObservation = { ...value, pixels: undefined, ua: undefined }
|
||||
assert(
|
||||
value.token === token && value.main === child.pid,
|
||||
'instance identity mismatch',
|
||||
)
|
||||
if (!identities.webview) {
|
||||
run.initialAttribution = collect([value.webview])[0]
|
||||
identify('webview', run.initialAttribution)
|
||||
}
|
||||
assert(
|
||||
value.nativeFocused === false && value.appActive === false,
|
||||
'measurement window unexpectedly activated',
|
||||
)
|
||||
assert(
|
||||
value.top === true && value.clickThrough === true,
|
||||
'measurement window must be natively topmost and click-through',
|
||||
)
|
||||
return value
|
||||
}
|
||||
try {
|
||||
mainStart = collect([child.pid])[0].start
|
||||
const deadline = Date.now() + 30000
|
||||
while (true) {
|
||||
assert(
|
||||
Date.now() < deadline && !interrupted && alive(),
|
||||
'embedded WebDriver readiness deadline or app exit',
|
||||
)
|
||||
try {
|
||||
if (
|
||||
(
|
||||
await fetch(`http://127.0.0.1:${port}/status`, {
|
||||
signal: AbortSignal.timeout(500),
|
||||
})
|
||||
).ok
|
||||
)
|
||||
break
|
||||
} catch {}
|
||||
await pause(100)
|
||||
}
|
||||
const listener = command('lsof', [
|
||||
'-nP',
|
||||
'-a',
|
||||
'-p',
|
||||
String(child.pid),
|
||||
`-iTCP:${port}`,
|
||||
'-sTCP:LISTEN',
|
||||
])
|
||||
assert(
|
||||
listener.includes(`127.0.0.1:${port}`) && !listener.includes(`*:${port}`),
|
||||
'driver must belong to this app and bind loopback',
|
||||
)
|
||||
const capabilities = createTauriCapabilities(binary)
|
||||
delete capabilities['wdio:tauriServiceOptions']
|
||||
browser = await remote({
|
||||
hostname: '127.0.0.1',
|
||||
port,
|
||||
capabilities,
|
||||
logLevel: 'error',
|
||||
connectionRetryCount: 0,
|
||||
connectionRetryTimeout: 5000,
|
||||
})
|
||||
const handles = await browser.getWindowHandles()
|
||||
assert(handles.length === 1, 'expected exactly one measurement WebView')
|
||||
await browser.switchToWindow(handles[0])
|
||||
// Replay's parent effect starts the socket after the graph installs its listeners.
|
||||
await browser.waitUntil(
|
||||
async () => {
|
||||
const s = await checked()
|
||||
return (
|
||||
validWorkload(s) &&
|
||||
s.received >= 3 &&
|
||||
s.now >= s.worker &&
|
||||
s.now - s.worker < 2500
|
||||
)
|
||||
},
|
||||
{
|
||||
timeout: 15000,
|
||||
interval: 100,
|
||||
timeoutMsg: 'traffic effects/Worker not ready before show',
|
||||
},
|
||||
)
|
||||
run.initialData = run.lastObservation
|
||||
await checked('show')
|
||||
await browser.waitUntil(
|
||||
async () => {
|
||||
const s = await checked()
|
||||
return validObservation({ phase: 'foreground', ...s })
|
||||
},
|
||||
{
|
||||
timeout: 15000,
|
||||
interval: 250,
|
||||
timeoutMsg: 'traffic data path/page not ready',
|
||||
},
|
||||
)
|
||||
run.userAgent = (await checked()).ua
|
||||
run.backgroundLoad = command('sysctl', ['-n', 'vm.loadavg'])
|
||||
const viewport = await browser.execute(() => [
|
||||
innerWidth,
|
||||
innerHeight,
|
||||
devicePixelRatio,
|
||||
])
|
||||
run.viewport = { dom: viewport, driver: await browser.getWindowRect() }
|
||||
assert(
|
||||
run.viewport.driver.width / viewport[2] === 1000 &&
|
||||
run.viewport.driver.height / viewport[2] === 700 &&
|
||||
viewport[0] === 1000 &&
|
||||
viewport[1] > 0 &&
|
||||
viewport[1] <= 700,
|
||||
`unexpected window dimensions: ${JSON.stringify(run.viewport)}`,
|
||||
)
|
||||
assert(
|
||||
!report.contract.viewport ||
|
||||
JSON.stringify(report.contract.viewport) === JSON.stringify(viewport),
|
||||
'viewport changed between runs',
|
||||
)
|
||||
report.contract.viewport = viewport
|
||||
run.marks.push({ name: 'ready', time: Date.now() })
|
||||
const readyAt = performance.now()
|
||||
assert(
|
||||
phaseTargets[0] - (await checked()).received >=
|
||||
(warmup * 1000) / replayMs,
|
||||
'readiness exceeded fixed replay allowance',
|
||||
)
|
||||
for (const phase of phases) {
|
||||
run.consoleStates.push({
|
||||
name: phase,
|
||||
locked: readConsole(),
|
||||
time: Date.now(),
|
||||
})
|
||||
const operation = performance.now()
|
||||
let s = await checked(phase === 'hidden' ? 'hide' : 'show', true)
|
||||
assert(
|
||||
s.visible === (phase !== 'hidden'),
|
||||
`${phase}: native visibility mismatch`,
|
||||
)
|
||||
const beforeDraw = s.draw
|
||||
await browser.waitUntil(
|
||||
async () => {
|
||||
s = await checked('state', true)
|
||||
return (
|
||||
validVisibility({ phase, ...s }) &&
|
||||
(phase === 'hidden' || (s.draw > beforeDraw && drawn(s)))
|
||||
)
|
||||
},
|
||||
{
|
||||
timeout: 5000,
|
||||
interval: 100,
|
||||
timeoutMsg: `${phase}: window/Canvas readiness deadline`,
|
||||
},
|
||||
)
|
||||
const operationMs = performance.now() - operation
|
||||
const target = phaseTargets[phases.indexOf(phase)]
|
||||
await browser.waitUntil(
|
||||
async () => {
|
||||
s = await checked('state', true)
|
||||
assert(
|
||||
s.received <= target,
|
||||
`${phase}: missed replay anchor ${target}`,
|
||||
)
|
||||
return s.received === target
|
||||
},
|
||||
{
|
||||
timeout: Math.max(1000, (target - s.received) * replayMs + 1000),
|
||||
interval: 50,
|
||||
timeoutMsg: `${phase}: replay anchor deadline`,
|
||||
},
|
||||
)
|
||||
assert(
|
||||
validObservation({ phase, ...s }),
|
||||
`${phase}: invalid traffic observation`,
|
||||
)
|
||||
const first = s,
|
||||
start = performance.now(),
|
||||
firstIndex = deadlineIndex(),
|
||||
phaseEpoch = Date.now()
|
||||
if (phase === 'foreground') {
|
||||
run.actualWarmupMs = start - readyAt
|
||||
assert(
|
||||
run.actualWarmupMs >= warmup * 1000,
|
||||
'insufficient actual warmup',
|
||||
)
|
||||
}
|
||||
run.marks.push({
|
||||
name: phase,
|
||||
time: phaseEpoch,
|
||||
operationMs,
|
||||
transitionMs: start - operation,
|
||||
target,
|
||||
...marker(s),
|
||||
deadlineIndex: firstIndex,
|
||||
senderSent: run.replay.sent,
|
||||
})
|
||||
previous = {}
|
||||
if (values['fail-phase'] === phase)
|
||||
throw new Error(`injected failure at ${phase}`)
|
||||
while (performance.now() - start < seconds * 1000) {
|
||||
const tick = performance.now()
|
||||
s = await checked()
|
||||
run.observations.push({ phase, ...run.lastObservation })
|
||||
assert(
|
||||
validObservation({ phase, ...s }),
|
||||
`${phase}: invalid visibility, workload or freshness`,
|
||||
)
|
||||
assert(
|
||||
performance.now() - replayStarted < 600000,
|
||||
'replay exceeded raw history window',
|
||||
)
|
||||
const samples = collect(roles.map((role) => s[role]))
|
||||
run.attribution = {
|
||||
main: s.main,
|
||||
webview: s.webview,
|
||||
observed: samples,
|
||||
}
|
||||
for (let i = 0; i < roles.length; i++) {
|
||||
const row = { ...samples[i], phase, role: roles[i] }
|
||||
identify(row.role, row)
|
||||
row.percent = delta(previous[row.role], row)
|
||||
row.cpuDelta =
|
||||
row.percent === null ? null : row.cpu - previous[row.role].cpu
|
||||
run.samples.push(row)
|
||||
previous[row.role] = row
|
||||
}
|
||||
await pause(Math.max(0, 1000 - (performance.now() - tick)))
|
||||
}
|
||||
s = await checked('state', true)
|
||||
const endIndex = deadlineIndex()
|
||||
const expected = endIndex - firstIndex
|
||||
assert(
|
||||
validObservation({ phase, ...s }) &&
|
||||
s.worker > first.worker &&
|
||||
Math.abs(s.received - first.received - expected) <= 2,
|
||||
`${phase}: unstable or lost workload`,
|
||||
)
|
||||
if (phase !== 'hidden')
|
||||
assert(
|
||||
s.draw > first.draw && s.pixels !== first.pixels,
|
||||
`${phase}: Canvas did not render fresh data`,
|
||||
)
|
||||
run.marks.push({
|
||||
name: `${phase}:end`,
|
||||
time: s.now,
|
||||
...marker(s),
|
||||
deadlineIndex: endIndex,
|
||||
senderSent: run.replay.sent,
|
||||
})
|
||||
run.consoleStates.push({
|
||||
name: `${phase}:end`,
|
||||
locked: readConsole(),
|
||||
time: Date.now(),
|
||||
})
|
||||
}
|
||||
run.stats = summarize(run.samples, seconds)
|
||||
assert(
|
||||
validStats(run.stats),
|
||||
'insufficient coverage or process discontinuity',
|
||||
)
|
||||
run.ok = true
|
||||
} catch (error) {
|
||||
run.error = String(error)
|
||||
process.exitCode = 1
|
||||
} finally {
|
||||
await browser?.deleteSession().catch(() => {})
|
||||
try {
|
||||
if (alive() && collect([child.pid])[0].start === mainStart) {
|
||||
child.kill('SIGTERM')
|
||||
await Promise.race([exited, sleep(5000)])
|
||||
if (alive() && collect([child.pid])[0].start === mainStart) {
|
||||
child.kill('SIGKILL')
|
||||
await Promise.race([exited, sleep(2000)])
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
run.cleanup.error = String(error)
|
||||
}
|
||||
clearTimeout(timer)
|
||||
for (const socket of replay.clients) socket.terminate()
|
||||
await new Promise((done) => replay.close(done))
|
||||
run.cleanup.mainExited =
|
||||
child.exitCode !== null || child.signalCode !== null
|
||||
if (identities.webview) {
|
||||
const [pid, start] = identities.webview.split('/')
|
||||
const remains = () => {
|
||||
try {
|
||||
return collect([pid])[0].start === start
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
const deadline = Date.now() + 5000
|
||||
while (remains() && Date.now() < deadline) await sleep(100)
|
||||
run.cleanup.webviewExited = !remains()
|
||||
}
|
||||
try {
|
||||
await fetch(`http://127.0.0.1:${port}/status`, {
|
||||
signal: AbortSignal.timeout(500),
|
||||
})
|
||||
run.cleanup.driverClosed = false
|
||||
} catch {
|
||||
run.cleanup.driverClosed = true
|
||||
}
|
||||
if (
|
||||
!run.cleanup.mainExited ||
|
||||
!run.cleanup.driverClosed ||
|
||||
run.cleanup.webviewExited === false
|
||||
) {
|
||||
run.ok = false
|
||||
run.error = `${run.error ?? ''} cleanup failed`
|
||||
process.exitCode = 1
|
||||
}
|
||||
run.stats = summarize(run.samples, seconds)
|
||||
writeFileSync(
|
||||
`${output}/${iteration + 1}.json`,
|
||||
JSON.stringify(run, null, 2),
|
||||
)
|
||||
const columns = [
|
||||
'phase',
|
||||
'role',
|
||||
'time',
|
||||
'pid',
|
||||
'start',
|
||||
'cpu',
|
||||
'percent',
|
||||
'rss',
|
||||
]
|
||||
writeFileSync(
|
||||
`${output}/${iteration + 1}.csv`,
|
||||
[
|
||||
columns.join(','),
|
||||
...run.samples.map((r) =>
|
||||
columns.map((c) => r[c] ?? 'unavailable').join(','),
|
||||
),
|
||||
].join('\n'),
|
||||
)
|
||||
writeFileSync(`${output}/summary.json`, JSON.stringify(report, null, 2))
|
||||
writeFileSync(`${output}/report.md`, markdown(report))
|
||||
}
|
||||
console.log(`Run ${iteration + 1}: ${run.ok ? 'passed' : run.error}`)
|
||||
}
|
||||
console.log(`Reports: ${output}`)
|
||||
@@ -0,0 +1,20 @@
|
||||
#include <libproc.h>
|
||||
#include <mach/mach_time.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <sys/resource.h>
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
mach_timebase_info_data_t timebase;
|
||||
if (mach_timebase_info(&timebase) != KERN_SUCCESS) return 1;
|
||||
for (int i = 1; i < argc; i++) {
|
||||
int pid = atoi(argv[i]);
|
||||
struct rusage_info_v0 r;
|
||||
char path[PROC_PIDPATHINFO_MAXSIZE];
|
||||
if (pid <= 0 || proc_pid_rusage(pid, RUSAGE_INFO_V0, (rusage_info_t *)&r)) return 1;
|
||||
if (proc_pidpath(pid, path, sizeof(path)) <= 0) return 1;
|
||||
printf("%d %llu %.9f %llu %s\n", pid, r.ri_proc_start_abstime,
|
||||
((double)r.ri_user_time + r.ri_system_time) * timebase.numer / timebase.denom / 1e9, r.ri_resident_size, path);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"identifier": "io.github.clash-verge-rev.perf-harness",
|
||||
"build": {
|
||||
"frontendDist": "../target/perf/dist",
|
||||
"beforeBuildCommand": "",
|
||||
"removeUnusedCommands": false
|
||||
},
|
||||
"bundle": { "active": false, "externalBin": [], "resources": [] },
|
||||
"app": {
|
||||
"security": {
|
||||
"capabilities": [
|
||||
{
|
||||
"identifier": "perf",
|
||||
"windows": ["perf"],
|
||||
"permissions": ["core:default"]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"extends": "../../tsconfig.json",
|
||||
"include": ["../../src", "./*.tsx"]
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"extends": "../../tsconfig.json",
|
||||
"compilerOptions": {
|
||||
"allowJs": true,
|
||||
"checkJs": false,
|
||||
"types": ["node"],
|
||||
"allowImportingTsExtensions": true
|
||||
},
|
||||
"include": ["./*.mjs", "./*.mts"]
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import { defineConfig, mergeConfig } from 'vite'
|
||||
|
||||
import base from '../../vite.config.mts'
|
||||
export default mergeConfig(
|
||||
base,
|
||||
defineConfig({
|
||||
root: 'scripts/perf',
|
||||
build: { outDir: '../../target/perf/dist', target: 'safari16' },
|
||||
}),
|
||||
)
|
||||
Reference in New Issue
Block a user