Test bench #285
Test bench #285
Conversation
📝 WalkthroughWalkthroughThis pull request introduces a comprehensive CLI tool and automated benchmarking infrastructure across iOS and Android platforms. It adds a Kotlin-based CLI with subcommands for building, linting, and benchmarking, alongside a Swift CLI for iOS. Device management abstractions enable benchmark automation on physical and simulated iOS/Android devices, with result aggregation, persistence, and regression detection. The Android and iOS apps are extended with benchmarking UI components, state management, and auto-launch support from CLI invocations. Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as CLI Tool
participant DM as Device Manager
participant Dev as iOS/Android Device
participant App as Benchmark App
participant Service as Benchmark Service
participant Storage as File Storage
participant UI as Results Display
CLI->>DM: List connected devices
DM->>Dev: Query device info
Dev-->>DM: Device info
DM-->>CLI: Device list
CLI->>Dev: Launch benchmark (config, models)
Dev->>App: Intent/URL scheme with config
App->>Service: Start benchmark
Service->>Service: Load model, warmup, run iterations
Service->>Storage: Write benchmark_*.json
CLI->>Dev: Poll for completion
Dev->>App: Check result files
App-->>Dev: Completion status
Dev-->>CLI: Complete
CLI->>Dev: Pull results
Dev->>Storage: Read benchmark JSON
Storage-->>Dev: Result files
Dev-->>CLI: Results transferred
CLI->>CLI: Aggregate results
CLI->>UI: Display metrics & report
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ❌ 3❌ Failed checks (2 warnings, 1 inconclusive)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional Comments (1)
-
examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/presentation/navigation/AppNavigation.kt, line 32-34 (link)style: Comment claims 5 tabs but now has 6 tabs after adding Benchmark tab
Should the iOS app structure comment be updated to reflect the new 6-tab structure, or should the Benchmark tab positioning be reconsidered?
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
33 files reviewed, 32 comments
| fun toJsonArg(): String { | ||
| // Escape for shell usage | ||
| return Json.encodeToString(this) | ||
| .replace("\"", "\\\"") | ||
| } |
There was a problem hiding this comment.
logic: Shell escaping is incomplete - only handles quotes but misses other shell-sensitive characters like backticks, semicolons, pipes, and newlines
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli/src/main/kotlin/ai/runanywhere/cli/benchmark/BenchmarkConfig.kt
Line: 18:22
Comment:
**logic:** Shell escaping is incomplete - only handles quotes but misses other shell-sensitive characters like backticks, semicolons, pipes, and newlines
How can I resolve this? If you propose a fix, please make it concise.| } catch (e3: Exception) { | ||
| // Could not parse | ||
| } |
There was a problem hiding this comment.
style: The final catch block silently fails without logging. Consider adding error logging for debugging failed parse attempts.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli/src/main/kotlin/ai/runanywhere/cli/benchmark/ResultsAggregator.kt
Line: 57:59
Comment:
**style:** The final catch block silently fails without logging. Consider adding error logging for debugging failed parse attempts.
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.| if !launchHandler.isDownloadingModel { | ||
| ProgressView() | ||
| .scaleEffect(0.8) | ||
| } |
There was a problem hiding this comment.
logic: Progress indicator logic appears inverted - it shows when NOT downloading. Should the progress indicator show when isDownloadingModel is true instead of false?
Prompt To Fix With AI
This is a comment left during a code review.
Path: examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkView.swift
Line: 75:78
Comment:
**logic:** Progress indicator logic appears inverted - it shows when NOT downloading. Should the progress indicator show when `isDownloadingModel` is true instead of false?
How can I resolve this? If you propose a fix, please make it concise.| swift build -c release | ||
|
|
||
| # Install to PATH | ||
| cp .build/release/runanywhere-ios /usr/local/bin/ |
There was a problem hiding this comment.
style: Consider using sudo prefix or mention potential permission requirements for copying to /usr/local/bin/
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli-swift/README.md
Line: 188:188
Comment:
**style:** Consider using `sudo` prefix or mention potential permission requirements for copying to `/usr/local/bin/`
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.| } catch (e: Exception) { | ||
| // Error is handled by benchmarkService | ||
| } |
There was a problem hiding this comment.
style: Empty catch block with only a comment is concerning. Consider logging the exception or providing user feedback about benchmark failures.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkViewModel.kt
Line: 52:54
Comment:
**style:** Empty catch block with only a comment is concerning. Consider logging the exception or providing user feedback about benchmark failures.
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.| val documentsPath = Path.of(containerPath, "Documents") | ||
|
|
||
| if (Files.exists(documentsPath)) { | ||
| Files.list(documentsPath) |
There was a problem hiding this comment.
logic: potential resource leak - Files.list() should be used in a try-with-resources or closed explicitly
| Files.list(documentsPath) | |
| Files.list(documentsPath).use { stream -> | |
| stream.filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } | |
| .forEach { sourceFile -> | |
| val destFile = outputDir.resolve("ios_sim_${sourceFile.name}") | |
| Files.copy(sourceFile, destFile) | |
| pulledFiles.add(destFile) | |
| } | |
| } |
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli/src/main/kotlin/ai/runanywhere/cli/device/IOSDevice.kt
Line: 192:192
Comment:
**logic:** potential resource leak - Files.list() should be used in a try-with-resources or closed explicitly
```suggestion
Files.list(documentsPath).use { stream ->
stream.filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") }
.forEach { sourceFile ->
val destFile = outputDir.resolve("ios_sim_${sourceFile.name}")
Files.copy(sourceFile, destFile)
pulledFiles.add(destFile)
}
}
```
How can I resolve this? If you propose a fix, please make it concise.|
|
||
| /// Launch app with benchmark arguments | ||
| private func launchAppWithBenchmark(device: IOSDevice, config: String, models: [String]?) { | ||
| let bundleId = "com.runanywhere.RunAnywhere" |
There was a problem hiding this comment.
logic: Bundle ID mismatch between device detection and app management functions - may cause app installation/launch failures
| let bundleId = "com.runanywhere.RunAnywhere" | |
| let bundleId = "ai.runanywhere.RunAnywhereAI" |
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli-swift/Sources/RunAnywhereCLI.swift
Line: 245:245
Comment:
**logic:** Bundle ID mismatch between device detection and app management functions - may cause app installation/launch failures
```suggestion
let bundleId = "ai.runanywhere.RunAnywhereAI"
```
How can I resolve this? If you propose a fix, please make it concise.| let physicalOutput = shellOutput(["idevice_id", "-l"]) | ||
| let udids = physicalOutput.split(separator: "\n").map { String($0) } | ||
|
|
||
| for udid in udids where !udid.isEmpty { | ||
| let name = shellOutput(["idevicename", "-u", udid]).trimmingCharacters(in: CharacterSet.whitespacesAndNewlines) | ||
| let version = shellOutput(["ideviceinfo", "-u", udid, "-k", "ProductVersion"]).trimmingCharacters(in: CharacterSet.whitespacesAndNewlines) | ||
|
|
||
| devices.append(IOSDevice( | ||
| udid: udid, | ||
| name: name.isEmpty ? "iOS Device" : name, | ||
| osVersion: version.isEmpty ? "Unknown" : version, | ||
| isSimulator: false | ||
| )) | ||
| } |
There was a problem hiding this comment.
logic: Physical device detection relies on optional libimobiledevice tools without fallback handling
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli-swift/Sources/RunAnywhereCLI.swift
Line: 791:804
Comment:
**logic:** Physical device detection relies on optional libimobiledevice tools without fallback handling
How can I resolve this? If you propose a fix, please make it concise.| if let versionMatch = lineStr.range(of: #"\((\d+\.\d+)\)"#, options: .regularExpression) { | ||
| version = String(lineStr[versionMatch]).dropFirst().dropLast().description | ||
| } |
There was a problem hiding this comment.
style: Regex version extraction may fail for complex version formats and uses unnecessary .description
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli-swift/Sources/RunAnywhereCLI.swift
Line: 208:210
Comment:
**style:** Regex version extraction may fail for complex version formats and uses unnecessary `.description`
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.| dbPath.parentFile?.mkdirs() | ||
|
|
||
| // Connect to SQLite | ||
| Class.forName("org.xerial.JDBC") |
There was a problem hiding this comment.
syntax: org.xerial.JDBC is not a valid JDBC driver class name for SQLite - should be org.sqlite.JDBC
| Class.forName("org.xerial.JDBC") | |
| Class.forName("org.sqlite.JDBC") |
Prompt To Fix With AI
This is a comment left during a code review.
Path: cli/src/main/kotlin/ai/runanywhere/cli/benchmark/HistoryStore.kt
Line: 25:25
Comment:
**syntax:** `org.xerial.JDBC` is not a valid JDBC driver class name for SQLite - should be `org.sqlite.JDBC`
```suggestion
Class.forName("org.sqlite.JDBC")
```
How can I resolve this? If you propose a fix, please make it concise.![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed everything up to d3bdb1b in 12 minutes and 26 seconds. Click for details.
- Reviewed
7010lines of code in32files - Skipped
1files when reviewing. - Skipped posting
1draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. cli-swift/Sources/RunAnywhereCLI.swift:141
- Draft comment:
Typographical suggestion: The printed message "ON YOUR iPHONE:" appears in all caps inconsistent with the standard capitalization of "iPhone" elsewhere in the code. Consider changing it to "ON YOUR iPhone:" for consistency. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% This is a purely stylistic comment about capitalization in a printed message. While technically correct that "iPhone" is the proper capitalization (Apple's trademark), this is an extremely minor issue. The rules state that comments should not be "obvious or unimportant" and should require "clearly a code change." This is arguably both obvious (if someone cares about Apple trademark capitalization) and unimportant (it doesn't affect functionality at all). It's the kind of nitpicky detail that could be caught in a final polish pass but doesn't warrant blocking or slowing down a PR review. The comment is technically correct but falls into the "unimportant" category. While this is technically a valid observation about brand consistency, it's such a minor stylistic issue that it may not be worth the reviewer's or author's time. The all-caps usage might even be intentional for emphasis in the CLI output box. The all-caps usage could indeed be intentional for emphasis in the ASCII art box (lines 140-148), making this a stylistic choice rather than an error. Even if it's not intentional, this is too minor to be actionable in a code review - it doesn't affect functionality, readability, or maintainability in any meaningful way. This comment should be deleted. It's about an extremely minor typographical/stylistic issue that doesn't affect functionality. The rules explicitly state not to make comments that are "obvious or unimportant," and this falls squarely into that category.
Workflow ID: wflow_9zzwZo4EvH2FMVDN
You can customize 
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 17
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/presentation/navigation/AppNavigation.kt (1)
30-33: Update tab-count comments to include BenchmarkLines 30-33 and 81-87 still say 5 tabs and iOS parity. With Benchmark added, the comments are now inaccurate and likely to confuse future maintenance.
📝 Suggested update
- * 5 tabs: Chat, STT, TTS, Voice, Settings + * 6 tabs: Chat, STT, TTS, Voice, Benchmark, Settings ... - * iOS Reference: ContentView.swift - TabView with 5 tabs + * iOS Reference: ContentView.swift - TabView with 6 tabs (if parity is intended)Also applies to: 81-87
🤖 Fix all issues with AI agents
In `@cli/src/main/kotlin/ai/runanywhere/cli/benchmark/HistoryStore.kt`:
- Around line 32-170: Several JDBC Statements/PreparedStatements and ResultSets
are not closed, causing resource leaks; update createTables(), save(), query(),
and detectRegression() to wrap each connection.createStatement(),
connection.prepareStatement(), and the ResultSet returned by executeQuery() in
Kotlin's .use { } blocks so they are auto-closed (including the early-return
path in detectRegression()); specifically, change the createTables() calls that
call createStatement() for the three DDL/INDEX executes to use .use, wrap the
PreparedStatement in save() around stmt.use { it.executeUpdate() }, wrap the
PreparedStatement and the ResultSet in query() so both stmt.use { rs =
it.executeQuery(); rs.use { ... } } (ensuring rs is closed before returning),
and do the same for detectRegression() ensuring the PreparedStatement and its
ResultSet are closed even when returning null after rs.next() check.
- Around line 25-26: The Class.forName call is loading the wrong SQLite driver
class ("org.xerial.JDBC") which will cause ClassNotFoundException; update the
Class.forName(...) invocation in HistoryStore (the call that precedes creating
the JDBC connection) to use "org.sqlite.JDBC", or remove the Class.forName line
entirely and rely on JDBC4 auto-registration, then leave the
DriverManager.getConnection("jdbc:sqlite:${dbPath.absolutePath}") as-is so the
connection opens correctly.
In `@cli/src/main/kotlin/ai/runanywhere/cli/commands/BuildCommand.kt`:
- Around line 211-225: The runScript function currently logs non-zero exit codes
but does not propagate failures; update runScript to surface failures to callers
by either throwing an exception or returning the exit code. Specifically, modify
runScript (and callers if needed) so that when
ProcessBuilder(...).start().waitFor() yields a non-zero result, the function
throws an IOException/RuntimeException (including the exit code and script path)
or returns the non-zero integer instead of only printing via Terminal; ensure
callers of runScript (if expecting Unit) are updated to handle the thrown
exception or check the returned code so the CLI process can exit with non‑zero
on script failures.
- Around line 155-186: The install/launch steps spawned with
ProcessBuilder("./gradlew", "installDebug") and ProcessBuilder("adb", "shell",
"am", ...) currently ignore exit codes so the CLI can report success despite
failures; update the code to capture each ProcessBuilder.start().waitFor()
result into variables (e.g., installResult and adbResult), check if they are
non‑zero, print an error via terminal.println(red(...)) with context (install or
adb) and then return (or exit with a non‑zero status) to propagate failure;
ensure the green("✓ App launched") message is only printed when both
installResult and adbResult are zero.
- Around line 33-41: The buildAll function currently only prints progress and
never runs any build scripts; update buildAll to actually invoke each SDK's
build command (e.g., via ProcessBuilder or Runtime.exec) for the SDKs listed in
listOf("swift","kotlin","flutter","react-native") instead of only printing
gray(" Building $sdk..."), capture each process exit code and stdout/stderr,
print failures to Terminal and stop or aggregate errors, and only print green("✓
All SDKs built successfully") if all commands return successful exit codes;
reference the buildAll function and the printed messages (cyan/gray/green) to
locate where to run and handle the external processes.
In `@cli/src/main/kotlin/ai/runanywhere/cli/commands/LintCommand.kt`:
- Around line 19-69: The --fix flag is ignored because lintAll() never forwards
the fix option to runScript or the fallback linters; update lintAll() to build
args = if (fix) listOf("--fix") else emptyList() and pass that args to
runScript(scriptPath, args) and to the fallback calls (replace lintIOS() and
lintAndroid() with lintIOS(args) and lintAndroid(args)); then change
lintIOS(scriptArgs: List<String>) and lintAndroid(scriptArgs: List<String>) to
accept the args and forward them to runScript(scriptPath, scriptArgs). Ensure
all runScript calls in these methods use the new scriptArgs parameter.
In `@cli/src/main/kotlin/ai/runanywhere/cli/device/DeviceManager.kt`:
- Around line 92-107: The parseSimulators function contains dead JSON-parsing
variables (json parameter, devicesSection, bootedPattern, altPattern,
runtimePattern) that are never used because the function instead runs `xcrun
simctl list devices booted`; either remove the unused json parameter and all
parsing-related locals (devicesSection, bootedPattern, altPattern,
runtimePattern) to clean up the function, or replace the subprocess call with a
proper implementation that parses the provided json string using those patterns
and returns the parsed List<IOSDeviceInfo> (refer to parseSimulators,
devicesSection, bootedPattern, altPattern, runtimePattern to locate the code to
remove or complete).
- Around line 43-48: The ios-deploy output captured into the deployOutput
variable is never parsed, so physical devices are never detected; update the
DeviceManager code that calls runCommand("ios-deploy", "--detect", "--no-wifi")
to parse deployOutput (look for UDID/serial, model, or "Found" lines emitted by
ios-deploy) and convert those entries into your internal device representation
(e.g., add to the device list or return value used by DeviceManager); ensure you
use the existing runCommand result handling and add logging on parse failures,
and keep the existing catch(e2: Exception) block to fall back when ios-deploy is
unavailable.
- Around line 118-122: The current code calls runCommand("xcrun", "simctl",
"list", "devices", "-j") inside the loop for each booted device; move that
subprocess call out of the loop and reuse its output (or the existing json
parameter) to avoid repeated work. Replace the per-device call that produces
infoOutput with a single precomputed devicesJson (or use the passed json
parameter) and update where versionMatch is computed to run against that shared
JSON using the existing regex and udid variable so version is derived without
invoking runCommand repeatedly.
In `@cli/src/main/kotlin/ai/runanywhere/cli/device/IOSDevice.kt`:
- Around line 23-60: launchBenchmarkAuto computes configArg and modelsArg (and
builds urlScheme) but the physical-device branch never passes them to the
ProcessBuilder so the app runs with defaults; update the physical-device
ProcessBuilder call in launchBenchmarkAuto to include the benchmark arguments
(or pass the constructed urlScheme) so configArg and modelsArg reach the app —
e.g. add the appropriate "--argument", "-benchmark_config", "--argument",
configArg, "--argument", "-benchmark_models", "--argument", modelsArg (or a
single "--argument", urlScheme) to the existing devicectl invocation so the
physical-device path mirrors the simulator behavior.
In
`@examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkLaunchHandler.kt`:
- Around line 66-83: In parseConfig, remove the brittle
configStr.contains("warmupIterations") guard and instead attempt to deserialize
JSON first using json.decodeFromString<BenchmarkConfig>(configStr) inside a
try/catch (return the parsed BenchmarkConfig on success), log the exception with
Log.w(TAG, ...) on failure, and only then fall back to matching preset names
("quick", "comprehensive") or returning BenchmarkConfig.DEFAULT; reference
parseConfig, BenchmarkConfig, and json.decodeFromString in your change and keep
the existing null check at the top.
In
`@examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkService.kt`:
- Around line 107-110: The cancel() method in BenchmarkService currently only
flips _state and logs but does not stop the running coroutine; modify the
benchmark launch logic to store the Coroutine Job (e.g., benchmarkJob or
runningJob) when starting the benchmark and have cancel() call job.cancel() (and
job = null) before setting _state and logging; also add cooperative cancellation
checks inside long-running work (use coroutineContext.isActive or
ensureActive/yield) in the benchmark worker functions/loops so the coroutine
stops promptly and cannot overwrite state/results.
- Around line 135-190: The model unloading is only on the success path; wrap the
Load/Warmup/Benchmark block that calls RunAnywhere.loadLLMModel(modelId), the
warmup loop calling runSingleInference(...), and the benchmark loops that
collect runResults in a try/finally so RunAnywhere.unloadLLMModel() is always
executed (also handle cancellation/exception cases), e.g., call
RunAnywhere.unloadLLMModel() from the finally block and ensure
peakMemoryBytes/state updates remain correct after errors; update references in
BenchmarkService to keep load timing and progress updates intact while
guaranteeing unload in finally.
- Around line 332-337: The BenchmarkResult currently exports raw ANDROID_ID via
getDeviceId(), exposing PII; change this to either require explicit user consent
before including deviceId in exported results or replace the raw value with a
non-reversible hash (e.g., SHA-256 of ANDROID_ID + app-specific salt) or omit it
entirely. Locate getDeviceId() and the places that populate
BenchmarkResult.deviceId and implement one of: (1) a consent check flag that
prevents setting/exporting deviceId unless consent granted, or (2) compute and
store a hashedDeviceId instead of the raw ANDROID_ID (add a stable app-specific
salt stored securely), or (3) remove deviceId from exported JSON; ensure tests
and JSON serialization use the new field or omit it accordingly.
In
`@examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkViewModel.kt`:
- Around line 94-101: The ViewModel's loadAvailableModels() is only called once
and won't react if the SDK becomes ready later; in BenchmarkViewModel subscribe
to app.initializationState (the Flow/observable) from init (or start a
coroutine) and call loadAvailableModels() when the state becomes
SDKInitializationState.Ready, ensuring you debounce/cancel previous jobs if
needed; after loadAvailableModels() completes still update _uiState via
_uiState.update(...) as you already do so the UI reflects newly availableModels.
In `@examples/ios/RunAnywhereAI/RunAnywhereAI/App/RunAnywhereAIApp.swift`:
- Around line 124-134: The deep-link logging in handleDeepLink currently logs
url.absoluteString and may leak sensitive query parameters; change the logger
call to redact sensitive data using the app's existing privacy redaction pattern
(i.e., log a redacted form of the URL rather than url.absoluteString) and keep
the call to benchmarkLaunchHandler.handleURL(url) intact; update the logger.info
invocation in handleDeepLink to use the established privacy/redaction helper or
logger API so model URLs/tokens in query strings are not written to logs.
In
`@examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkView.swift`:
- Around line 331-389: Wrap the iOS-only UI and behavior in conditional
compilation so macOS builds don't reference unavailable APIs: gate the Copy and
Share Buttons (the Button instances that call copyResultsToClipboard and set
showShareSheet) and the .sheet(isPresented: $showShareSheet) that presents
ShareSheet with `#if` os(iOS) / `#endif`. Also adjust copyResultsToClipboard so the
UIPasteboard call and the success toast (showCopiedToast toggling and
DispatchQueue hide) are executed only under `#if` os(iOS) to avoid showing a toast
on macOS; reference the functions/vars getResultsJSON(),
copyResultsToClipboard(), showShareSheet, showCopiedToast, and the ShareSheet
usage when applying the changes.
🟡 Minor comments (8)
cli-swift/Sources/RunAnywhereCLI.swift-916-927 (1)
916-927: Hardcoded developer-specific path should be removed.Line 920 contains a hardcoded path
/Users/shubhammalhotra/Desktop/RunanywhereAI/master/runanywhere-sdkswhich is specific to one developer's machine and will not work for other contributors.Remove developer-specific path
// Method 3: Try common development paths let commonPaths = [ - NSHomeDirectory() + "/Desktop/RunanywhereAI/master/runanywhere-sdks", NSHomeDirectory() + "/Developer/runanywhere-sdks", NSHomeDirectory() + "/Projects/runanywhere-sdks", - "/Users/shubhammalhotra/Desktop/RunanywhereAI/master/runanywhere-sdks" ]examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkService.swift-123-127 (1)
123-127: Force unwraps can crash if config arrays are empty.
config.prompts.first!andconfig.maxTokensList.first!will crash with a runtime exception if either array is empty. While presets define non-empty arrays, custom configurations could be passed in.Suggested fix with guard
// Warmup phase + guard let warmupPrompt = config.prompts.first, + let warmupMaxTokens = config.maxTokensList.first else { + throw BenchmarkError.inferenceError("Config must have at least one prompt and maxTokens value") + } + for i in 0..<config.warmupIterations { state = .warmingUp(model: modelInfo.name, iteration: i + 1, total: config.warmupIterations) updateProgress(modelIndex: modelIndex, totalModels: totalModels) _ = try await runSingleInference( - prompt: config.prompts.first!, - maxTokens: config.maxTokensList.first! + prompt: warmupPrompt, + maxTokens: warmupMaxTokens ) }examples/benchmark-config.json-33-83 (1)
33-83: Update benchmark-config.json categories to match supported enum valuesThe JSON file documents 8 categories (code, translation, summarization, qa, plus short, medium, long, reasoning), but the enums support only 4 (SHORT/MEDIUM/LONG/REASONING in Android, short/medium/long/reasoning in iOS). Although the JSON is not currently parsed at runtime, this schema mismatch will confuse users attempting to use unsupported categories. Either remove the extra categories from the JSON or extend the enums to support them.
examples/ios/RunAnywhereAI/RunAnywhereAI/App/ContentView.swift-10-12 (1)
10-12: AddBenchmarkLaunchHandlerto the Preview to avoid runtime crashes.
ContentViewrequires aBenchmarkLaunchHandlerenvironment object that the Preview does not currently provide, causing a runtime crash.Suggested preview wiring
`#Preview` { ContentView() + .environmentObject(BenchmarkLaunchHandler.shared) }cli/src/main/kotlin/ai/runanywhere/cli/device/IOSDevice.kt-174-230 (1)
174-230: WrapFiles.list()stream with.use()and addREPLACE_EXISTINGoption for idempotent pulls.At line 192–198,
Files.list()opens a resource that is never closed, causing a resource leak. Additionally, line 196 will fail on repeated runs becauseFiles.copy()throwsFileAlreadyExistsExceptionwhen the destination file exists. Add theStandardCopyOptionimport and wrap the stream with.use()while includingREPLACE_EXISTINGto make pulls idempotent.🔧 Suggested fix
import java.io.BufferedReader import java.nio.file.Files import java.nio.file.Path +import java.nio.file.StandardCopyOption import kotlin.io.path.name @@ - Files.list(documentsPath) - .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } - .forEach { sourceFile -> - val destFile = outputDir.resolve("ios_sim_${sourceFile.name}") - Files.copy(sourceFile, destFile) - pulledFiles.add(destFile) - } + Files.list(documentsPath).use { stream -> + stream + .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } + .forEach { sourceFile -> + val destFile = outputDir.resolve("ios_sim_${sourceFile.name}") + Files.copy(sourceFile, destFile, StandardCopyOption.REPLACE_EXISTING) + pulledFiles.add(destFile) + } + }cli/src/main/kotlin/ai/runanywhere/cli/commands/BenchmarkCommand.kt-176-220 (1)
176-220: Specify locale for consistent numeric formatting in CLI output.
String.formatwithout a Locale argument uses the default JVM locale, which varies by system. In some regions, this produces comma decimals (e.g., "123,4" in France) instead of periods, causing inconsistent CLI output. SpecifyLocale.USto ensure stable formatting.Add the import and update both String.format calls on lines 215–216:
🔧 Suggested fix
import com.github.ajalt.mordant.rendering.TextColors.* import com.github.ajalt.mordant.terminal.Terminal import java.io.File +import java.util.Locale @@ - "${String.format("%.1f", result.avgTokensPerSecond).padEnd(10)}" + - "${String.format("%.0fms", result.avgTtftMs).padEnd(10)}" + "${String.format(Locale.US, "%.1f", result.avgTokensPerSecond).padEnd(10)}" + + "${String.format(Locale.US, "%.0fms", result.avgTtftMs).padEnd(10)}"cli/src/main/kotlin/ai/runanywhere/cli/device/AndroidDevice.kt-109-187 (1)
109-187: Close the walk stream, handle file overwrites, and log pull failures.
Files.walkleaks resources without.use {},Files.copywill fail on repeated pulls withoutREPLACE_EXISTING, and the catch block silently swallows errors that would help diagnoserun-asissues.🔧 Suggested fix
import java.io.BufferedReader import java.nio.file.Files import java.nio.file.Path +import java.nio.file.StandardCopyOption import kotlin.io.path.name @@ - Files.walk(tempDir) - .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } - .forEach { sourceFile -> - val destFile = outputDir.resolve("android_${serial}_${sourceFile.name}") - Files.copy(sourceFile, destFile) - pulledFiles.add(destFile) - } + Files.walk(tempDir).use { stream -> + stream + .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } + .forEach { sourceFile -> + val destFile = outputDir.resolve("android_${serial}_${sourceFile.name}") + Files.copy(sourceFile, destFile, StandardCopyOption.REPLACE_EXISTING) + pulledFiles.add(destFile) + } + } @@ - } catch (e: Exception) { - // run-as may not work on release builds - } + } catch (e: Exception) { + System.err.println("run-as pull failed for $serial: ${e.message}") + }examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkViewModel.kt-42-55 (1)
42-55: Surface benchmark failures instead of silently swallowing them.
The empty catch makes failures invisible to logs/UI; at least log or forward the exception so it can be surfaced.🛠️ Suggested change
+import android.util.Log ... viewModelScope.launch { try { benchmarkService.runLLMBenchmark( modelIds = selectedIds, config = _uiState.value.selectedConfig.config ) } catch (e: Exception) { - // Error is handled by benchmarkService + Log.e("BenchmarkViewModel", "Benchmark failed", e) } }
🧹 Nitpick comments (13)
cli/src/main/kotlin/ai/runanywhere/cli/device/DeviceManager.kt (2)
147-156: Consider adding timeout and exit code validation.The current implementation can hang indefinitely if a device command stalls, and silently returns error output when commands fail (exit code unchecked).
🛡️ Suggested hardening
- private fun runCommand(vararg args: String): String { + private fun runCommand(vararg args: String, timeoutSeconds: Long = 30): String { val process = ProcessBuilder(*args) .redirectErrorStream(true) .start() val output = process.inputStream.bufferedReader().use(BufferedReader::readText) - process.waitFor() + val completed = process.waitFor(timeoutSeconds, java.util.concurrent.TimeUnit.SECONDS) + if (!completed) { + process.destroyForcibly() + throw RuntimeException("Command timed out: ${args.joinToString(" ")}") + } + if (process.exitValue() != 0) { + throw RuntimeException("Command failed with exit code ${process.exitValue()}: ${args.joinToString(" ")}") + } return output }
17-22: Consider logging swallowed exceptions for debuggability.Multiple catch blocks (lines 20-22, 41-48, 85-87, 132-134) silently swallow exceptions. While graceful degradation is appropriate here, consider logging at debug/trace level to aid troubleshooting when device tools are misconfigured.
Based on static analysis hint (detekt.exceptions.SwallowedException).
cli-swift/README.md (1)
157-163: Consider updating Swift version requirement for Swift 6 compatibility.The requirements list Swift 5.9+ but the coding guidelines indicate Swift 6 APIs should be used. Consider updating to Swift 6.0+ if the CLI implementation uses Swift 6 concurrency features, or document any Swift 6-specific requirements separately.
cli/README.md (1)
58-100: Add language specifiers to fenced code blocks.Per static analysis hints, the fenced code blocks at lines 71 and 128 are missing language specifiers. Consider adding a language identifier (e.g.,
textorconsole) for better rendering.-``` +```text ╔═══════════════════════════════════════════════════════════╗examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkTypes.swift (1)
26-31: Potential crash ifstandardPromptsis empty.Line 30 accesses
BenchmarkPrompt.standardPrompts[0]without bounds checking. WhilestandardPromptsis currently hardcoded with 4 items, this creates a fragile dependency.Suggested safer approach
static let quick = BenchmarkConfig( warmupIterations: 1, testIterations: 3, maxTokensList: [50], - prompts: [BenchmarkPrompt.standardPrompts[0]] + prompts: BenchmarkPrompt.standardPrompts.prefix(1).map { $0 } )Or use
Array(BenchmarkPrompt.standardPrompts.prefix(1))to safely take up to 1 element.cli/src/main/kotlin/ai/runanywhere/cli/commands/ModelsCommand.kt (1)
11-12: Unused imports.
kotlinx.serialization.json.Jsonis imported but never used in this file. The@Serializableannotation onModelEntryworks without explicitly usingJson.Remove unused import
import kotlinx.serialization.Serializable -import kotlinx.serialization.json.Jsonexamples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkLaunchHandler.swift (1)
252-262: Use Swift 6 concurrency instead ofDispatchQueue.main.asyncAfter.Per coding guidelines, prefer Swift 6 concurrency primitives.
DispatchQueue.main.asyncAfteris an older pattern.Refactor to use Task with sleep
struct AutoBenchmarkModifier: ViewModifier { `@ObservedObject` var handler = BenchmarkLaunchHandler.shared `@Binding` var selectedTab: Int func body(content: Content) -> some View { content - .onAppear { + .task { if handler.shouldAutoStart { - // Navigate to benchmark tab (index 4) - DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) { + try? await Task.sleep(for: .seconds(1)) + await MainActor.run { selectedTab = 4 } } } } }Based on coding guidelines requiring Swift 6 APIs.
cli/src/main/kotlin/ai/runanywhere/cli/benchmark/ResultsAggregator.kt (1)
36-64: Swallowed exceptions lose diagnostic information.The nested try-catch blocks silently discard exceptions without logging. While trying multiple formats is reasonable, completely swallowing exceptions makes debugging difficult when valid-looking files fail to parse.
Add debug logging for parse failures
+ private val logger = java.util.logging.Logger.getLogger(ResultsAggregator::class.java.name) + private fun parseResults(content: String, filename: String): List<AggregatedResult> { val results = mutableListOf<AggregatedResult>() try { // Try parsing as BenchmarkExport (array wrapper) val export = json.decodeFromString<BenchmarkExportDto>(content) export.results.forEach { result -> results.add(mapToAggregatedResult(result, filename)) } - } catch (e: Exception) { + } catch (e: Exception) { + logger.fine("$filename: Not BenchmarkExportDto format: ${e.message}") try { // Try parsing as single BenchmarkResult val result = json.decodeFromString<BenchmarkResultDto>(content) results.add(mapToAggregatedResult(result, filename)) - } catch (e2: Exception) { + } catch (e2: Exception) { + logger.fine("$filename: Not single BenchmarkResultDto: ${e2.message}") try { // Try parsing as array of BenchmarkResult val resultList = json.decodeFromString<List<BenchmarkResultDto>>(content) resultList.forEach { result -> results.add(mapToAggregatedResult(result, filename)) } - } catch (e3: Exception) { - // Could not parse + } catch (e3: Exception) { + logger.warning("$filename: Could not parse as any known format: ${e3.message}") } } } return results }As flagged by static analysis (detekt.exceptions.SwallowedException).
cli-swift/Sources/RunAnywhereCLI.swift (2)
323-331: Unused variablelastCheckTime.
lastCheckTimeis declared at line 326 but never read. It appears to be a remnant from a previous implementation.Remove unused variable
private func waitForCompletion(device: IOSDevice) -> Bool { let startTime = Date() let timeoutInterval = TimeInterval(timeout) - var lastCheckTime = Date() var dotCount = 0 let pollInterval: TimeInterval = 5 // Check every 5 seconds let statusInterval: TimeInterval = 30 // Show status every 30 seconds var lastStatusTime = Date()
230-235: Hardcoded OS version "17.0" for simulators.The simulator OS version is hardcoded to
"17.0"in bothdetectDevicesWithXcode()(line 233) andlistDevices()(line 784). This should be parsed from the runtime information or the device output.Consider extracting the version from the simulator runtime section or xctrace output. The version could potentially be parsed from the
-- iOS x.x --section headers in the simctl output.Also applies to: 781-787
examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkTypes.kt (1)
156-164: Prefer enum forpromptCategoryLine 156-164 stores the category as
Stringeven though aPromptCategoryenum exists. This weakens type safety and makes serialization/validation brittle.♻️ Proposed refactor
data class PromptAggregatedResult( val promptId: String, - val promptCategory: String, + val promptCategory: BenchmarkPrompt.PromptCategory, val avgTokensPerSecond: Double, val avgLatencyMs: Double, val avgTtftMs: Double, val runCount: Int, )cli/src/main/kotlin/ai/runanywhere/cli/device/IOSDevice.kt (2)
66-92: Close theFiles.liststream to avoid descriptor leaks.♻️ Suggested fix
- val newResults = Files.list(documentsPath) - .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } - .filter { Files.getLastModifiedTime(it).toMillis() > benchmarkStartTime } - .count() + val newResults = Files.list(documentsPath).use { stream -> + stream + .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } + .filter { Files.getLastModifiedTime(it).toMillis() > benchmarkStartTime } + .count() + }
97-116: Close theFiles.liststream when deleting old results.♻️ Suggested fix
- Files.list(documentsPath) - .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } - .forEach { Files.deleteIfExists(it) } + Files.list(documentsPath).use { stream -> + stream + .filter { it.name.startsWith("benchmark_") && it.name.endsWith(".json") } + .forEach { Files.deleteIfExists(it) } + }
| Class.forName("org.xerial.JDBC") | ||
| connection = DriverManager.getConnection("jdbc:sqlite:${dbPath.absolutePath}") |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Find build configuration files to identify sqlite-jdbc version
find . -type f \( -name "build.gradle*" -o -name "pom.xml" -o -name "gradle.properties" \) | head -20Repository: RunanywhereAI/runanywhere-sdks
Length of output: 204
🏁 Script executed:
# Look for SQLite-related dependencies
rg "sqlite" -i --type gradle --type xml --type kotlinRepository: RunanywhereAI/runanywhere-sdks
Length of output: 570
🏁 Script executed:
# Check the HistoryStore.kt file for context around line 25-26
fd "HistoryStore.kt" --type fRepository: RunanywhereAI/runanywhere-sdks
Length of output: 138
🏁 Script executed:
# Read the HistoryStore.kt file around lines 25-26
cat -n cli/src/main/kotlin/ai/runanywhere/cli/benchmark/HistoryStore.kt | sed -n '20,30p'Repository: RunanywhereAI/runanywhere-sdks
Length of output: 475
🌐 Web query:
SQLite JDBC xerial driver class name org.sqlite.JDBC vs org.xerial.JDBC
💡 Result:
Use org.sqlite.JDBC. The official xerial sqlite-jdbc documentation and examples show the driver class name as org.sqlite.JDBC (and examples call Class.forName("org.sqlite.JDBC") or just use DriverManager.getConnection("jdbc:sqlite:...")). [1][2]
Sources:
- xerial sqlite-jdbc README. [1]
- Common troubleshooting / usage examples (Stack Overflow). [2]
SQLite driver class name is invalid
Line 25 uses org.xerial.JDBC, which is not a valid driver class. The correct driver class for xerial sqlite-jdbc is org.sqlite.JDBC. This will throw ClassNotFoundException at runtime. Use org.sqlite.JDBC or remove the explicit load if relying on JDBC 4 auto-registration.
🛠️ Proposed fix
- Class.forName("org.xerial.JDBC")
+ Class.forName("org.sqlite.JDBC")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Class.forName("org.xerial.JDBC") | |
| connection = DriverManager.getConnection("jdbc:sqlite:${dbPath.absolutePath}") | |
| Class.forName("org.sqlite.JDBC") | |
| connection = DriverManager.getConnection("jdbc:sqlite:${dbPath.absolutePath}") |
🤖 Prompt for AI Agents
In `@cli/src/main/kotlin/ai/runanywhere/cli/benchmark/HistoryStore.kt` around
lines 25 - 26, The Class.forName call is loading the wrong SQLite driver class
("org.xerial.JDBC") which will cause ClassNotFoundException; update the
Class.forName(...) invocation in HistoryStore (the call that precedes creating
the JDBC connection) to use "org.sqlite.JDBC", or remove the Class.forName line
entirely and rely on JDBC4 auto-registration, then leave the
DriverManager.getConnection("jdbc:sqlite:${dbPath.absolutePath}") as-is so the
connection opens correctly.
| private fun createTables() { | ||
| connection.createStatement().execute(""" | ||
| CREATE TABLE IF NOT EXISTS benchmark_results ( | ||
| id INTEGER PRIMARY KEY AUTOINCREMENT, | ||
| model_id TEXT NOT NULL, | ||
| model_name TEXT NOT NULL, | ||
| framework TEXT NOT NULL, | ||
| device_model TEXT NOT NULL, | ||
| platform TEXT NOT NULL, | ||
| os_version TEXT, | ||
| tokens_per_second REAL NOT NULL, | ||
| ttft_ms REAL, | ||
| latency_ms REAL, | ||
| peak_memory_mb REAL, | ||
| model_load_time_ms REAL, | ||
| git_commit TEXT, | ||
| timestamp INTEGER NOT NULL, | ||
| created_at INTEGER DEFAULT (strftime('%s', 'now')) | ||
| ) | ||
| """) | ||
|
|
||
| connection.createStatement().execute(""" | ||
| CREATE INDEX IF NOT EXISTS idx_model_timestamp | ||
| ON benchmark_results(model_id, timestamp) | ||
| """) | ||
|
|
||
| connection.createStatement().execute(""" | ||
| CREATE INDEX IF NOT EXISTS idx_platform_timestamp | ||
| ON benchmark_results(platform, timestamp) | ||
| """) | ||
| } | ||
|
|
||
| /** | ||
| * Save a benchmark result | ||
| */ | ||
| fun save(result: ResultsAggregator.AggregatedResult) { | ||
| val stmt = connection.prepareStatement(""" | ||
| INSERT INTO benchmark_results | ||
| (model_id, model_name, framework, device_model, platform, os_version, | ||
| tokens_per_second, ttft_ms, latency_ms, peak_memory_mb, model_load_time_ms, | ||
| git_commit, timestamp) | ||
| VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) | ||
| """) | ||
|
|
||
| stmt.setString(1, result.modelId) | ||
| stmt.setString(2, result.modelName) | ||
| stmt.setString(3, result.framework) | ||
| stmt.setString(4, result.deviceModel) | ||
| stmt.setString(5, result.platform) | ||
| stmt.setString(6, result.osVersion) | ||
| stmt.setDouble(7, result.avgTokensPerSecond) | ||
| stmt.setDouble(8, result.avgTtftMs) | ||
| stmt.setDouble(9, result.avgLatencyMs) | ||
| stmt.setDouble(10, result.peakMemoryMB) | ||
| stmt.setDouble(11, result.modelLoadTimeMs) | ||
| stmt.setString(12, result.gitCommit) | ||
| stmt.setLong(13, result.timestamp) | ||
|
|
||
| stmt.executeUpdate() | ||
| } | ||
|
|
||
| /** | ||
| * Query historical results | ||
| */ | ||
| fun query( | ||
| modelId: String? = null, | ||
| platform: String? = null, | ||
| lastDays: Int = 30, | ||
| ): List<HistoryEntry> { | ||
| val cutoffTime = Instant.now().minusSeconds(lastDays.toLong() * 24 * 60 * 60).epochSecond * 1000 | ||
|
|
||
| val sql = buildString { | ||
| append("SELECT * FROM benchmark_results WHERE timestamp > ?") | ||
| if (modelId != null) append(" AND model_id = ?") | ||
| if (platform != null) append(" AND platform = ?") | ||
| append(" ORDER BY timestamp DESC") | ||
| } | ||
|
|
||
| val stmt = connection.prepareStatement(sql) | ||
| var paramIndex = 1 | ||
| stmt.setLong(paramIndex++, cutoffTime) | ||
| if (modelId != null) stmt.setString(paramIndex++, modelId) | ||
| if (platform != null) stmt.setString(paramIndex++, platform) | ||
|
|
||
| val results = mutableListOf<HistoryEntry>() | ||
| val rs = stmt.executeQuery() | ||
|
|
||
| var previousTps: Double? = null | ||
|
|
||
| while (rs.next()) { | ||
| val currentTps = rs.getDouble("tokens_per_second") | ||
| val delta = previousTps?.let { (currentTps - it) / it * 100 } ?: 0.0 | ||
|
|
||
| results.add(HistoryEntry( | ||
| modelId = rs.getString("model_id"), | ||
| modelName = rs.getString("model_name"), | ||
| platform = rs.getString("platform"), | ||
| deviceModel = rs.getString("device_model"), | ||
| tokensPerSecond = currentTps, | ||
| tokensPerSecondDelta = delta, | ||
| ttftMs = rs.getDouble("ttft_ms"), | ||
| latencyMs = rs.getDouble("latency_ms"), | ||
| peakMemoryMb = rs.getDouble("peak_memory_mb"), | ||
| gitCommit = rs.getString("git_commit"), | ||
| timestamp = rs.getLong("timestamp"), | ||
| dateString = formatDate(rs.getLong("timestamp")) | ||
| )) | ||
|
|
||
| previousTps = currentTps | ||
| } | ||
|
|
||
| return results | ||
| } | ||
|
|
||
| /** | ||
| * Detect performance regression | ||
| */ | ||
| fun detectRegression( | ||
| modelId: String, | ||
| currentResult: ResultsAggregator.AggregatedResult, | ||
| thresholdPercent: Double = 10.0, | ||
| ): RegressionReport? { | ||
| // Get baseline (average of last 5 results for this model) | ||
| val stmt = connection.prepareStatement(""" | ||
| SELECT AVG(tokens_per_second) as avg_tps, | ||
| AVG(ttft_ms) as avg_ttft, | ||
| AVG(latency_ms) as avg_latency | ||
| FROM ( | ||
| SELECT tokens_per_second, ttft_ms, latency_ms | ||
| FROM benchmark_results | ||
| WHERE model_id = ? | ||
| ORDER BY timestamp DESC | ||
| LIMIT 5 | ||
| ) | ||
| """) | ||
| stmt.setString(1, modelId) | ||
|
|
||
| val rs = stmt.executeQuery() | ||
| if (!rs.next()) return null |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -name "HistoryStore.kt" -type fRepository: RunanywhereAI/runanywhere-sdks
Length of output: 140
🏁 Script executed:
cat -n ./cli/src/main/kotlin/ai/runanywhere/cli/benchmark/HistoryStore.ktRepository: RunanywhereAI/runanywhere-sdks
Length of output: 10781
Close JDBC statements and result sets to prevent resource leaks
Lines 32–170 create Statements, PreparedStatements, and ResultSets without closing them. In long benchmark runs this exhausts DB resources:
createTables()(lines 33, 53, 58): Three createStatement() calls with no closesave()(line 68): PreparedStatement never closed after executeUpdate()query()(lines 110, 117): Both PreparedStatement and ResultSet leak; ResourceSet is never closed before returndetectRegression()(lines 155, 169): Both PreparedStatement and ResultSet leak, including via early return path (line 170)
Wrap all statement and query calls with .use {} blocks to auto-close resources.
Example fix pattern
- val stmt = connection.prepareStatement(sql)
- var paramIndex = 1
- stmt.setLong(paramIndex++, cutoffTime)
- if (modelId != null) stmt.setString(paramIndex++, modelId)
- if (platform != null) stmt.setString(paramIndex++, platform)
-
- val results = mutableListOf<HistoryEntry>()
- val rs = stmt.executeQuery()
+ val results = mutableListOf<HistoryEntry>()
+ connection.prepareStatement(sql).use { stmt ->
+ var paramIndex = 1
+ stmt.setLong(paramIndex++, cutoffTime)
+ if (modelId != null) stmt.setString(paramIndex++, modelId)
+ if (platform != null) stmt.setString(paramIndex++, platform)
+ stmt.executeQuery().use { rs ->
+ while (rs.next()) {
+ // process results
+ }
+ }
+ }🤖 Prompt for AI Agents
In `@cli/src/main/kotlin/ai/runanywhere/cli/benchmark/HistoryStore.kt` around
lines 32 - 170, Several JDBC Statements/PreparedStatements and ResultSets are
not closed, causing resource leaks; update createTables(), save(), query(), and
detectRegression() to wrap each connection.createStatement(),
connection.prepareStatement(), and the ResultSet returned by executeQuery() in
Kotlin's .use { } blocks so they are auto-closed (including the early-return
path in detectRegression()); specifically, change the createTables() calls that
call createStatement() for the three DDL/INDEX executes to use .use, wrap the
PreparedStatement in save() around stmt.use { it.executeUpdate() }, wrap the
PreparedStatement and the ResultSet in query() so both stmt.use { rs =
it.executeQuery(); rs.use { ... } } (ensuring rs is closed before returning),
and do the same for detectRegression() ensuring the PreparedStatement and its
ResultSet are closed even when returning null after rs.next() check.
| private fun buildAll() { | ||
| val terminal = Terminal() | ||
| terminal.println(cyan("Building all SDKs...")) | ||
|
|
||
| listOf("swift", "kotlin", "flutter", "react-native").forEach { sdk -> | ||
| terminal.println(gray(" Building $sdk...")) | ||
| } | ||
|
|
||
| terminal.println(green("✓ All SDKs built successfully")) |
There was a problem hiding this comment.
--all currently performs no builds
Line 33-41 only prints progress and always reports success; no scripts are executed, so runanywhere build --all is effectively a no‑op and can mask missing scripts.
🛠️ Proposed fix
private fun buildAll() {
val terminal = Terminal()
terminal.println(cyan("Building all SDKs..."))
-
- listOf("swift", "kotlin", "flutter", "react-native").forEach { sdk ->
- terminal.println(gray(" Building $sdk..."))
- }
-
- terminal.println(green("✓ All SDKs built successfully"))
+ val scripts = listOf(
+ "sdk/runanywhere-swift/scripts/build-swift.sh",
+ "sdk/runanywhere-kotlin/scripts/build-kotlin.sh",
+ "sdk/runanywhere-flutter/scripts/build-flutter.sh",
+ "sdk/runanywhere-react-native/scripts/build-react-native.sh",
+ )
+ for (path in scripts) {
+ terminal.println(gray(" Building ${File(path).nameWithoutExtension}..."))
+ val script = findScript(path) ?: error("Build script not found: $path")
+ runScript(script)
+ }
+ terminal.println(green("✓ All SDKs built successfully"))
}🤖 Prompt for AI Agents
In `@cli/src/main/kotlin/ai/runanywhere/cli/commands/BuildCommand.kt` around lines
33 - 41, The buildAll function currently only prints progress and never runs any
build scripts; update buildAll to actually invoke each SDK's build command
(e.g., via ProcessBuilder or Runtime.exec) for the SDKs listed in
listOf("swift","kotlin","flutter","react-native") instead of only printing
gray(" Building $sdk..."), capture each process exit code and stdout/stderr,
print failures to Terminal and stop or aggregate errors, and only print green("✓
All SDKs built successfully") if all commands return successful exit codes;
reference the buildAll function and the printed messages (cyan/gray/green) to
locate where to run and handle the external processes.
| // Build | ||
| val buildResult = ProcessBuilder("./gradlew", "assembleDebug") | ||
| .directory(appDir) | ||
| .inheritIO() | ||
| .start() | ||
| .waitFor() | ||
|
|
||
| if (buildResult != 0) { | ||
| terminal.println(red("✗ Build failed")) | ||
| return | ||
| } | ||
|
|
||
| terminal.println(green("✓ Build successful")) | ||
|
|
||
| if (run) { | ||
| terminal.println(cyan("Installing and launching...")) | ||
|
|
||
| ProcessBuilder("./gradlew", "installDebug") | ||
| .directory(appDir) | ||
| .inheritIO() | ||
| .start() | ||
| .waitFor() | ||
|
|
||
| ProcessBuilder( | ||
| "adb", "shell", "am", "start", | ||
| "-n", "com.runanywhere.runanywhereai.debug/.MainActivity" | ||
| ) | ||
| .inheritIO() | ||
| .start() | ||
| .waitFor() | ||
|
|
||
| terminal.println(green("✓ App launched")) |
There was a problem hiding this comment.
Handle install/launch failures and return non‑zero
Line 155-186 ignores exit codes for installDebug and adb, and errors only print then return (exit code stays 0). This can report success even when install/launch failed.
🛠️ Proposed fix
val buildResult = ProcessBuilder("./gradlew", "assembleDebug")
.directory(appDir)
.inheritIO()
.start()
.waitFor()
if (buildResult != 0) {
terminal.println(red("✗ Build failed"))
- return
+ error("Android build failed with exit code $buildResult")
}
terminal.println(green("✓ Build successful"))
if (run) {
terminal.println(cyan("Installing and launching..."))
- ProcessBuilder("./gradlew", "installDebug")
+ val installResult = ProcessBuilder("./gradlew", "installDebug")
.directory(appDir)
.inheritIO()
.start()
.waitFor()
+ if (installResult != 0) {
+ terminal.println(red("✗ Install failed"))
+ error("Install failed with exit code $installResult")
+ }
- ProcessBuilder(
+ val launchResult = ProcessBuilder(
"adb", "shell", "am", "start",
"-n", "com.runanywhere.runanywhereai.debug/.MainActivity"
)
.inheritIO()
.start()
.waitFor()
+ if (launchResult != 0) {
+ terminal.println(red("✗ Launch failed"))
+ error("Launch failed with exit code $launchResult")
+ }
terminal.println(green("✓ App launched"))
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Build | |
| val buildResult = ProcessBuilder("./gradlew", "assembleDebug") | |
| .directory(appDir) | |
| .inheritIO() | |
| .start() | |
| .waitFor() | |
| if (buildResult != 0) { | |
| terminal.println(red("✗ Build failed")) | |
| return | |
| } | |
| terminal.println(green("✓ Build successful")) | |
| if (run) { | |
| terminal.println(cyan("Installing and launching...")) | |
| ProcessBuilder("./gradlew", "installDebug") | |
| .directory(appDir) | |
| .inheritIO() | |
| .start() | |
| .waitFor() | |
| ProcessBuilder( | |
| "adb", "shell", "am", "start", | |
| "-n", "com.runanywhere.runanywhereai.debug/.MainActivity" | |
| ) | |
| .inheritIO() | |
| .start() | |
| .waitFor() | |
| terminal.println(green("✓ App launched")) | |
| // Build | |
| val buildResult = ProcessBuilder("./gradlew", "assembleDebug") | |
| .directory(appDir) | |
| .inheritIO() | |
| .start() | |
| .waitFor() | |
| if (buildResult != 0) { | |
| terminal.println(red("✗ Build failed")) | |
| error("Android build failed with exit code $buildResult") | |
| } | |
| terminal.println(green("✓ Build successful")) | |
| if (run) { | |
| terminal.println(cyan("Installing and launching...")) | |
| val installResult = ProcessBuilder("./gradlew", "installDebug") | |
| .directory(appDir) | |
| .inheritIO() | |
| .start() | |
| .waitFor() | |
| if (installResult != 0) { | |
| terminal.println(red("✗ Install failed")) | |
| error("Install failed with exit code $installResult") | |
| } | |
| val launchResult = ProcessBuilder( | |
| "adb", "shell", "am", "start", | |
| "-n", "com.runanywhere.runanywhereai.debug/.MainActivity" | |
| ) | |
| .inheritIO() | |
| .start() | |
| .waitFor() | |
| if (launchResult != 0) { | |
| terminal.println(red("✗ Launch failed")) | |
| error("Launch failed with exit code $launchResult") | |
| } | |
| terminal.println(green("✓ App launched")) | |
| } |
🤖 Prompt for AI Agents
In `@cli/src/main/kotlin/ai/runanywhere/cli/commands/BuildCommand.kt` around lines
155 - 186, The install/launch steps spawned with ProcessBuilder("./gradlew",
"installDebug") and ProcessBuilder("adb", "shell", "am", ...) currently ignore
exit codes so the CLI can report success despite failures; update the code to
capture each ProcessBuilder.start().waitFor() result into variables (e.g.,
installResult and adbResult), check if they are non‑zero, print an error via
terminal.println(red(...)) with context (install or adb) and then return (or
exit with a non‑zero status) to propagate failure; ensure the green("✓ App
launched") message is only printed when both installResult and adbResult are
zero.
| fun runScript(script: File, args: List<String> = emptyList()) { | ||
| val terminal = Terminal() | ||
| val command = listOf(script.absolutePath) + args | ||
|
|
||
| val result = ProcessBuilder(command) | ||
| .directory(script.parentFile) | ||
| .inheritIO() | ||
| .start() | ||
| .waitFor() | ||
|
|
||
| if (result == 0) { | ||
| terminal.println(green("✓ Script completed successfully")) | ||
| } else { | ||
| terminal.println(red("✗ Script failed with exit code $result")) | ||
| } |
There was a problem hiding this comment.
Propagate script failures to callers
Line 221-224 logs failures but never fails the CLI process; CI/automation will treat failures as success. Consider throwing or returning a non‑zero result so callers can fail fast.
🛠️ Proposed fix
if (result == 0) {
terminal.println(green("✓ Script completed successfully"))
} else {
terminal.println(red("✗ Script failed with exit code $result"))
+ error("Script failed with exit code $result")
}🤖 Prompt for AI Agents
In `@cli/src/main/kotlin/ai/runanywhere/cli/commands/BuildCommand.kt` around lines
211 - 225, The runScript function currently logs non-zero exit codes but does
not propagate failures; update runScript to surface failures to callers by
either throwing an exception or returning the exit code. Specifically, modify
runScript (and callers if needed) so that when
ProcessBuilder(...).start().waitFor() yields a non-zero result, the function
throws an IOException/RuntimeException (including the exit code and script path)
or returns the non-zero integer instead of only printing via Terminal; ensure
callers of runScript (if expecting Unit) are updated to handle the thrown
exception or check the returned code so the CLI process can exit with non‑zero
on script failures.
| // Load model and measure load time | ||
| val loadStart = System.nanoTime() | ||
| RunAnywhere.loadLLMModel(modelId) | ||
| val loadTimeMs = (System.nanoTime() - loadStart) / 1_000_000.0 | ||
|
|
||
| Log.i(TAG, "Model loaded in ${loadTimeMs}ms") | ||
|
|
||
| // Reset peak memory tracking | ||
| peakMemoryBytes = 0 | ||
|
|
||
| // Warmup phase | ||
| for (i in 0 until config.warmupIterations) { | ||
| _state.value = BenchmarkState.WarmingUp( | ||
| model = modelInfo.name, | ||
| iteration = i + 1, | ||
| total = config.warmupIterations | ||
| ) | ||
| updateProgress(modelIndex, totalModels) | ||
|
|
||
| runSingleInference( | ||
| prompt = config.prompts.first(), | ||
| maxTokens = config.maxTokensList.first() | ||
| ) | ||
| } | ||
|
|
||
| // Benchmark runs | ||
| val runResults = mutableListOf<SingleRunResult>() | ||
| val totalRuns = config.prompts.size * config.maxTokensList.size * config.testIterations | ||
| var currentRun = 0 | ||
|
|
||
| for (prompt in config.prompts) { | ||
| for (maxTokens in config.maxTokensList) { | ||
| repeat(config.testIterations) { | ||
| currentRun++ | ||
| _state.value = BenchmarkState.Running( | ||
| model = modelInfo.name, | ||
| prompt = prompt.id, | ||
| iteration = currentRun, | ||
| total = totalRuns | ||
| ) | ||
| updateProgress(modelIndex, totalModels) | ||
|
|
||
| val result = runSingleInference(prompt, maxTokens) | ||
| runResults.add(result) | ||
|
|
||
| // Track peak memory | ||
| val currentMemory = getMemoryUsage() | ||
| if (currentMemory > peakMemoryBytes) { | ||
| peakMemoryBytes = currentMemory | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Unload model | ||
| RunAnywhere.unloadLLMModel() |
There was a problem hiding this comment.
Ensure models are unloaded even on failure/cancel.
RunAnywhere.unloadLLMModel() is only called on the success path; any exception during warmup or inference can leave the model loaded and memory elevated. Use try/finally.
🛠️ Suggested change
- // Load model and measure load time
- val loadStart = System.nanoTime()
- RunAnywhere.loadLLMModel(modelId)
- val loadTimeMs = (System.nanoTime() - loadStart) / 1_000_000.0
+ var modelLoaded = false
+ val loadStart = System.nanoTime()
+ try {
+ RunAnywhere.loadLLMModel(modelId)
+ modelLoaded = true
+ } finally {
+ // no-op: actual unload happens below in finally
+ }
+ val loadTimeMs = (System.nanoTime() - loadStart) / 1_000_000.0
...
- // Unload model
- RunAnywhere.unloadLLMModel()
+ // Unload model (always)
+ try {
+ // existing warmup + runs + aggregation
+ } finally {
+ if (modelLoaded) {
+ RunAnywhere.unloadLLMModel()
+ }
+ }🤖 Prompt for AI Agents
In
`@examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkService.kt`
around lines 135 - 190, The model unloading is only on the success path; wrap
the Load/Warmup/Benchmark block that calls RunAnywhere.loadLLMModel(modelId),
the warmup loop calling runSingleInference(...), and the benchmark loops that
collect runResults in a try/finally so RunAnywhere.unloadLLMModel() is always
executed (also handle cancellation/exception cases), e.g., call
RunAnywhere.unloadLLMModel() from the finally block and ensure
peakMemoryBytes/state updates remain correct after errors; update references in
BenchmarkService to keep load timing and progress updates intact while
guaranteeing unload in finally.
| private fun getDeviceId(): String { | ||
| return android.provider.Settings.Secure.getString( | ||
| context.contentResolver, | ||
| android.provider.Settings.Secure.ANDROID_ID | ||
| ) ?: "unknown" | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, verify the file exists and check its basic structure
find . -name "BenchmarkService.kt" -type fRepository: RunanywhereAI/runanywhere-sdks
Length of output: 204
🏁 Script executed:
# Look at the file context around the getDeviceId function
cat -n examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkService.kt | sed -n '300,370p'Repository: RunanywhereAI/runanywhere-sdks
Length of output: 2848
🏁 Script executed:
# Search for all usages of getDeviceId in the file
rg "getDeviceId" examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkService.kt -A 3 -B 3Repository: RunanywhereAI/runanywhere-sdks
Length of output: 607
🏁 Script executed:
# Search for any export or result sharing mechanisms
rg "export|share|result|benchmark.*data|deviceId" examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/ -i --type kotlinRepository: RunanywhereAI/runanywhere-sdks
Length of output: 14998
🏁 Script executed:
# Look for privacy policy or consent mechanisms
find examples/android/RunAnywhereAI -type f \( -name "*.kt" -o -name "*.xml" -o -name "*.md" \) | xargs rg -l "privacy|consent|pii" -iRepository: RunanywhereAI/runanywhere-sdks
Length of output: 235
🏁 Script executed:
# Check BenchmarkTypes.kt to see the full structure of BenchmarkResult
rg "data class BenchmarkResult|data class BenchmarkExport" examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkTypes.kt -A 15Repository: RunanywhereAI/runanywhere-sdks
Length of output: 691
🏁 Script executed:
# Check README for privacy policy
head -100 examples/android/RunAnywhereAI/README.mdRepository: RunanywhereAI/runanywhere-sdks
Length of output: 3688
🏁 Script executed:
# Look for any privacy policy or consent-related code in the whole project
find examples/android/RunAnywhereAI -type f \( -name "*.kt" -o -name "*.md" \) | xargs rg -i "privacy|consent|pii|data.*sharing|user.*permission" -B 2 -A 2 | head -80Repository: RunanywhereAI/runanywhere-sdks
Length of output: 1662
Include user consent or hash ANDROID_ID in exported benchmark results.
ANDROID_ID is confirmed to be exported unencrypted in BenchmarkResult.deviceId within the JSON benchmark file. As a stable device identifier, this qualifies as PII under privacy regulations (GDPR, CCPA, etc.). No privacy policy or user consent mechanism for this export was found in the codebase. Either obtain explicit user consent before exporting, hash the device ID, or omit it from exported results.
🤖 Prompt for AI Agents
In
`@examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkService.kt`
around lines 332 - 337, The BenchmarkResult currently exports raw ANDROID_ID via
getDeviceId(), exposing PII; change this to either require explicit user consent
before including deviceId in exported results or replace the raw value with a
non-reversible hash (e.g., SHA-256 of ANDROID_ID + app-specific salt) or omit it
entirely. Locate getDeviceId() and the places that populate
BenchmarkResult.deviceId and implement one of: (1) a consent check flag that
prevents setting/exporting deviceId unless consent granted, or (2) compute and
store a hashedDeviceId instead of the raw ANDROID_ID (add a stable app-specific
salt stored securely), or (3) remove deviceId from exported JSON; ensure tests
and JSON serialization use the new field or omit it accordingly.
| private suspend fun loadAvailableModels() { | ||
| if (!app.isSDKReady()) return | ||
|
|
||
| val models = RunAnywhere.availableModels().filter { model -> | ||
| model.category == ModelCategory.LANGUAGE && model.isDownloaded | ||
| } | ||
|
|
||
| _uiState.update { it.copy(availableModels = models) } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, search for isSDKReady usage context
rg -n -C3 '\bisSDKReady\b'Repository: RunanywhereAI/runanywhere-sdks
Length of output: 3143
🏁 Script executed:
#!/bin/bash
# Search for where loadAvailableModels is called
rg -n 'loadAvailableModels' examples/android/Repository: RunanywhereAI/runanywhere-sdks
Length of output: 383
🏁 Script executed:
#!/bin/bash
# Inspect BenchmarkViewModel structure
cat -n examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkViewModel.kt | head -100Repository: RunanywhereAI/runanywhere-sdks
Length of output: 3812
🏁 Script executed:
#!/bin/bash
# Check RunAnywhereApplication for SDK initialization
cat -n examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/RunAnywhereApplication.kt | head -250Repository: RunanywhereAI/runanywhere-sdks
Length of output: 12816
🏁 Script executed:
#!/bin/bash
# Check if BenchmarkViewModel observes initializationState or retries anywhere
rg -n 'initializationState|retryInitialization|observ' examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkViewModel.ktRepository: RunanywhereAI/runanywhere-sdks
Length of output: 56
🏁 Script executed:
#!/bin/bash
# Search for BenchmarkViewModel instantiation to understand timing
rg -n 'BenchmarkViewModel' examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/ -t kotlin | grep -v 'BenchmarkViewModel.kt'Repository: RunanywhereAI/runanywhere-sdks
Length of output: 527
🏁 Script executed:
#!/bin/bash
# Check BenchmarkScreen to understand when ViewModel is created
cat -n examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkScreen.kt | head -100Repository: RunanywhereAI/runanywhere-sdks
Length of output: 5367
🏁 Script executed:
#!/bin/bash
# Check navigation to understand if benchmark screen is accessible early
rg -n 'BenchmarkScreen\|Benchmark' examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/ -t kotlin --max-count=20 | grep -v '.kt:' | head -30Repository: RunanywhereAI/runanywhere-sdks
Length of output: 56
🏁 Script executed:
#!/bin/bash
# Search for navigation/routing setup
fd -t f "Navigation\|NavHost" examples/android/RunAnywhereAI/app/src/main/java/ --exec grep -l "BenchmarkScreen" {} \;Repository: RunanywhereAI/runanywhere-sdks
Length of output: 56
🏁 Script executed:
#!/bin/bash
# Search for where BenchmarkScreen is composed/navigated to
rg -n 'BenchmarkScreen' examples/android/RunAnywhereAI/app/src/main/java/ -t kotlin -B5 -A2Repository: RunanywhereAI/runanywhere-sdks
Length of output: 3862
🏁 Script executed:
#!/bin/bash
# Check the main app composition to see if there's SDK readiness checking before showing navigation
cat -n examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/presentation/navigation/AppNavigation.kt | head -100Repository: RunanywhereAI/runanywhere-sdks
Length of output: 4245
🏁 Script executed:
#!/bin/bash
# Search for MainActivity or the main app composition
fd -t f "MainActivity\|App.kt" examples/android/RunAnywhereAI/app/src/main/java/ --exec grep -l "AppNavigation\|setContent" {} \;Repository: RunanywhereAI/runanywhere-sdks
Length of output: 56
🏁 Script executed:
#!/bin/bash
# Find main activity
fd -t f "MainActivity" examples/android/RunAnywhereAI/app/src/main/java/Repository: RunanywhereAI/runanywhere-sdks
Length of output: 168
🏁 Script executed:
#!/bin/bash
# Check MainActivity to see how navigation is set up
cat -n examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/MainActivity.ktRepository: RunanywhereAI/runanywhere-sdks
Length of output: 4073
Add observer pattern for SDK readiness or implement retry logic.
While the MainActivity gates AppNavigation behind SDK readiness, the ViewModel lacks defensive programming. If the SDK state changes after ViewModel initialization (edge case) or if you want to make the code more robust, add an observer of app.initializationState to reload models when SDK becomes ready, or use a LaunchedEffect to retry if the initial load fails.
Consider:
LaunchedEffect(Unit) {
app.initializationState.collect { state ->
if (state is SDKInitializationState.Ready) {
loadAvailableModels()
}
}
}🤖 Prompt for AI Agents
In
`@examples/android/RunAnywhereAI/app/src/main/java/com/runanywhere/runanywhereai/benchmark/BenchmarkViewModel.kt`
around lines 94 - 101, The ViewModel's loadAvailableModels() is only called once
and won't react if the SDK becomes ready later; in BenchmarkViewModel subscribe
to app.initializationState (the Flow/observable) from init (or start a
coroutine) and call loadAvailableModels() when the state becomes
SDKInitializationState.Ready, ensuring you debounce/cancel previous jobs if
needed; after loadAvailableModels() completes still update _uiState via
_uiState.update(...) as you already do so the UI reflects newly availableModels.
| /// Handle deep links for CLI automation | ||
| /// URL format: runanywhere://benchmark?config=quick&model_url=https://... | ||
| private func handleDeepLink(_ url: URL) { | ||
| logger.info("📲 Received deep link: \(url.absoluteString)") | ||
|
|
||
| guard url.scheme == "runanywhere" else { return } | ||
|
|
||
| // Delegate to BenchmarkLaunchHandler | ||
| Task { @MainActor in | ||
| benchmarkLaunchHandler.handleURL(url) | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Check if the file exists and read lines 124-134
if [ -f "examples/ios/RunAnywhereAI/RunAnywhereAI/App/RunAnywhereAIApp.swift" ]; then
echo "=== File found. Reading lines 124-134 ==="
sed -n '120,140p' "examples/ios/RunAnywhereAI/RunAnywhereAI/App/RunAnywhereAIApp.swift" | cat -n
else
echo "File not found at specified path"
find . -name "RunAnywhereAIApp.swift" -type f 2>/dev/null
fiRepository: RunanywhereAI/runanywhere-sdks
Length of output: 204
🏁 Script executed:
# Check the Swift version and logging setup in the project
if [ -f "examples/ios/RunAnywhereAI/RunAnywhereAI.xcodeproj/project.pbxproj" ]; then
echo "=== Checking Swift version in project config ==="
grep -i "swift" "examples/ios/RunAnywhereAI/RunAnywhereAI.xcodeproj/project.pbxproj" | head -20
fi
# Check for any iOS deployment target and Swift settings
if [ -f "examples/ios/RunAnywhereAI/RunAnywhereAI/RunAnywhereAI.entitlements" ] || [ -d "examples/ios/RunAnywhereAI" ]; then
echo "=== Checking iOS project structure ==="
ls -la examples/ios/RunAnywhereAI/ 2>/dev/null | head -20
fiRepository: RunanywhereAI/runanywhere-sdks
Length of output: 204
🏁 Script executed:
# Search for logger declarations and logging framework usage
echo "=== Searching for logger declarations and imports ==="
rg -A 2 -B 2 "(import|private var logger|let logger)" "examples/ios/RunAnywhereAI/RunAnywhereAI/App/RunAnywhereAIApp.swift" 2>/dev/null || echo "Using ast-grep to find logger declarations"Repository: RunanywhereAI/runanywhere-sdks
Length of output: 585
🏁 Script executed:
# Read the specific lines 124-134 to verify the code snippet
echo "=== Code at lines 124-134 ==="
sed -n '124,134p' "examples/ios/RunAnywhereAI/RunAnywhereAI/App/RunAnywhereAIApp.swift" | cat -nRepository: RunanywhereAI/runanywhere-sdks
Length of output: 629
🏁 Script executed:
# Search for any existing privacy redaction usage in the codebase
echo "=== Searching for existing privacy redaction usage ==="
rg "privacy:" "examples/ios/RunAnywhereAI/" 2>/dev/null | head -20Repository: RunanywhereAI/runanywhere-sdks
Length of output: 461
🌐 Web query:
Swift 6 os.Logger privacy redaction .private mask.hash documentation
💡 Result:
Short answer — Swift’s Logger (Unified Logging) still supports privacy redaction in Swift 6 using the privacy parameter on interpolations. To produce a stable, non-reversible identifier instead of the real value use:
- privacy: .private(mask: .hash) — logs a hashed/redacted representation so you can correlate items without revealing the original value.
Example:
let logger = Logger()
logger.info("User id: (userID, privacy: .private(mask: .hash))")
Key points
- By default dynamic strings/complex values are redacted in system logs; mark values .public to show them. [1][2]
- .private(mask: .hash) yields a consistent hash-like redaction so you can group or correlate records without leaking the real data. [2][3]
- When running under the debugger you may see actual values; on devices/macOS the Console normally shows unless you enable developer/advanced logging or install Apple’s logging configuration/profile. Use those only for debugging — do not ship them. [3][4]
Sources
- Apple Developer — Unified logging / os_log / Logger (docs on logging & privacy). [Apple Developer]. [1]
- Practical guides explaining privacy interpolation and mask: .hash usage (examples). [2]
- Articles about redaction, hash masking and how Console/debugger behavior differs. [3][4]
Would you like direct links to the Apple docs and one or two short examples/tutorials?
Redact sensitive data in deep-link URL logging.
The url.absoluteString can contain query parameters with model URLs, configuration tokens, or other sensitive data. Use privacy redaction:
- logger.info("📲 Received deep link: \(url.absoluteString)")
+ logger.info("📲 Received deep link: \(url.absoluteString, privacy: .private(mask: .hash))")This matches the privacy annotation pattern already used elsewhere in the codebase and prevents sensitive information from appearing in production logs.
🤖 Prompt for AI Agents
In `@examples/ios/RunAnywhereAI/RunAnywhereAI/App/RunAnywhereAIApp.swift` around
lines 124 - 134, The deep-link logging in handleDeepLink currently logs
url.absoluteString and may leak sensitive query parameters; change the logger
call to redact sensitive data using the app's existing privacy redaction pattern
(i.e., log a redacted form of the URL rather than url.absoluteString) and keep
the call to benchmarkLaunchHandler.handleURL(url) intact; update the logger.info
invocation in handleDeepLink to use the established privacy/redaction helper or
logger API so model URLs/tokens in query strings are not written to logs.
| // Export buttons | ||
| Button(action: copyResultsToClipboard) { | ||
| Label("Copy", systemImage: "doc.on.doc") | ||
| .font(.caption) | ||
| } | ||
| .buttonStyle(.bordered) | ||
| .controlSize(.small) | ||
|
|
||
| Button(action: { showShareSheet = true }) { | ||
| Label("Share", systemImage: "square.and.arrow.up") | ||
| .font(.caption) | ||
| } | ||
| .buttonStyle(.bordered) | ||
| .controlSize(.small) | ||
| } | ||
|
|
||
| // Copied toast | ||
| if showCopiedToast { | ||
| HStack { | ||
| Image(systemName: "checkmark.circle.fill") | ||
| .foregroundStyle(.green) | ||
| Text("Results copied to clipboard!") | ||
| .font(.caption) | ||
| } | ||
| .padding(8) | ||
| .background(Color.green.opacity(0.1)) | ||
| .clipShape(RoundedRectangle(cornerRadius: 8)) | ||
| .transition(.opacity) | ||
| } | ||
|
|
||
| ForEach(viewModel.results) { result in | ||
| BenchmarkResultCard(result: result, viewModel: viewModel) | ||
| } | ||
| } | ||
| .padding() | ||
| .background(Color(.secondarySystemBackground)) | ||
| .clipShape(RoundedRectangle(cornerRadius: 12)) | ||
| .sheet(isPresented: $showShareSheet) { | ||
| ShareSheet(items: [getResultsJSON()]) | ||
| } | ||
| } | ||
|
|
||
| private func copyResultsToClipboard() { | ||
| let json = getResultsJSON() | ||
| #if os(iOS) | ||
| UIPasteboard.general.string = json | ||
| #endif | ||
|
|
||
| withAnimation { | ||
| showCopiedToast = true | ||
| } | ||
|
|
||
| // Hide toast after 2 seconds | ||
| DispatchQueue.main.asyncAfter(deadline: .now() + 2) { | ||
| withAnimation { | ||
| showCopiedToast = false | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -name "BenchmarkView.swift" -type fRepository: RunanywhereAI/runanywhere-sdks
Length of output: 155
🏁 Script executed:
git ls-files | grep -i benchmarkviewRepository: RunanywhereAI/runanywhere-sdks
Length of output: 349
🏁 Script executed:
fd -t f "BenchmarkView.swift"Repository: RunanywhereAI/runanywhere-sdks
Length of output: 153
🏁 Script executed:
wc -l examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkView.swiftRepository: RunanywhereAI/runanywhere-sdks
Length of output: 157
🏁 Script executed:
cat -n examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkView.swiftRepository: RunanywhereAI/runanywhere-sdks
Length of output: 25517
🏁 Script executed:
find examples/ios/RunAnywhereAI -name "*.pbxproj" -o -name "Package.swift" | head -5Repository: RunanywhereAI/runanywhere-sdks
Length of output: 181
🏁 Script executed:
fd -t f "project.pbxproj" examples/ios/RunAnywhereAIRepository: RunanywhereAI/runanywhere-sdks
Length of output: 140
🏁 Script executed:
find examples/ios -type f -name "Info.plist" | head -3Repository: RunanywhereAI/runanywhere-sdks
Length of output: 56
🏁 Script executed:
cat examples/ios/RunAnywhereAI/RunAnywhereAI.xcodeproj/project.pbxproj | head -100Repository: RunanywhereAI/runanywhere-sdks
Length of output: 4760
🏁 Script executed:
rg "SUPPORTED_PLATFORMS|MACOSX_DEPLOYMENT_TARGET|IPHONEOS_DEPLOYMENT_TARGET" examples/ios/RunAnywhereAI/RunAnywhereAI.xcodeproj/project.pbxprojRepository: RunanywhereAI/runanywhere-sdks
Length of output: 997
Guard iOS-only share/copy UI to prevent macOS build failures.
The project supports macOS builds (SUPPORTED_PLATFORMS includes macOS), but ShareSheet is referenced unconditionally at line 369 and will cause compilation to fail. The Copy and Share buttons (lines 331–344) are also unconditionally rendered, making them appear on macOS even though they cannot function properly. Additionally, copyResultsToClipboard() shows a success toast on macOS despite UIPasteboard being guarded and unavailable. Wrap the buttons and sheet modifier in #if os(iOS) blocks.
🛠️ Platform gating fix
- // Export buttons
- Button(action: copyResultsToClipboard) {
- Label("Copy", systemImage: "doc.on.doc")
- .font(.caption)
- }
- .buttonStyle(.bordered)
- .controlSize(.small)
-
- Button(action: { showShareSheet = true }) {
- Label("Share", systemImage: "square.and.arrow.up")
- .font(.caption)
- }
- .buttonStyle(.bordered)
- .controlSize(.small)
+ // Export buttons
+ `#if` os(iOS)
+ Button(action: copyResultsToClipboard) {
+ Label("Copy", systemImage: "doc.on.doc")
+ .font(.caption)
+ }
+ .buttonStyle(.bordered)
+ .controlSize(.small)
+
+ Button(action: { showShareSheet = true }) {
+ Label("Share", systemImage: "square.and.arrow.up")
+ .font(.caption)
+ }
+ .buttonStyle(.bordered)
+ .controlSize(.small)
+ `#endif`- .sheet(isPresented: $showShareSheet) {
- ShareSheet(items: [getResultsJSON()])
- }
+ `#if` os(iOS)
+ .sheet(isPresented: $showShareSheet) {
+ ShareSheet(items: [getResultsJSON()])
+ }
+ `#endif`🤖 Prompt for AI Agents
In
`@examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkView.swift`
around lines 331 - 389, Wrap the iOS-only UI and behavior in conditional
compilation so macOS builds don't reference unavailable APIs: gate the Copy and
Share Buttons (the Button instances that call copyResultsToClipboard and set
showShareSheet) and the .sheet(isPresented: $showShareSheet) that presents
ShareSheet with `#if` os(iOS) / `#endif`. Also adjust copyResultsToClipboard so the
UIPasteboard call and the success toast (showCopiedToast toggling and
DispatchQueue hide) are executed only under `#if` os(iOS) to avoid showing a toast
on macOS; reference the functions/vars getResultsJSON(),
copyResultsToClipboard(), showShareSheet, showCopiedToast, and the ShareSheet
usage when applying the changes.
1 participant
Description
Brief description of the changes made.
Type of Change
Testing
Platform-Specific Testing (check all that apply)
Swift SDK / iOS Sample:
Kotlin SDK / Android Sample:
Flutter SDK / Flutter Sample:
React Native SDK / React Native Sample:
Labels
Please add the appropriate label(s):
SDKs:
Swift SDK- Changes to Swift SDK (sdk/runanywhere-swift)Kotlin SDK- Changes to Kotlin SDK (sdk/runanywhere-kotlin)Flutter SDK- Changes to Flutter SDK (sdk/runanywhere-flutter)React Native SDK- Changes to React Native SDK (sdk/runanywhere-react-native)Commons- Changes to shared native code (sdk/runanywhere-commons)Sample Apps:
iOS Sample- Changes to iOS example app (examples/ios)Android Sample- Changes to Android example app (examples/android)Flutter Sample- Changes to Flutter example app (examples/flutter)React Native Sample- Changes to React Native example app (examples/react-native)Checklist
Screenshots
Attach relevant UI screenshots for changes (if applicable):
Summary by CodeRabbit
Release Notes
New Features
Documentation
✏️ Tip: You can customize this high-level summary in your review settings.
Greptile Summary
cli-swift/) and Kotlin (cli/) implementations for iOS and Android device management, automated testing, and performance evaluationImportant Files Changed
cli-swift/Sources/RunAnywhereCLI.swiftcli/src/main/kotlin/ai/runanywhere/cli/commands/LintCommand.ktexamples/ios/RunAnywhereAI/RunAnywhereAI/Features/Benchmark/BenchmarkService.swiftcli/src/main/kotlin/ai/runanywhere/cli/benchmark/BenchmarkConfig.ktcli/src/main/kotlin/ai/runanywhere/cli/device/DeviceManager.ktConfidence score: 3/5
Sequence Diagram
sequenceDiagram participant User participant CLI as "CLI Tool" participant Device as "Mobile Device" participant App as "RunAnywhere App" participant SDK as "RunAnywhere SDK" participant BenchmarkService as "BenchmarkService" participant Storage as "Device Storage" User->>CLI: runanywhere benchmark auto CLI->>Device: Detect connected devices Device-->>CLI: Device info (iOS/Android) CLI->>App: Launch with benchmark args App->>BenchmarkService: Initialize benchmark BenchmarkService->>SDK: Get available models SDK-->>BenchmarkService: Model list BenchmarkService->>SDK: Load model SDK-->>BenchmarkService: Model loaded loop For each prompt/config BenchmarkService->>SDK: Generate text SDK-->>BenchmarkService: Generation result + metrics BenchmarkService->>BenchmarkService: Collect metrics end BenchmarkService->>SDK: Unload model BenchmarkService->>Storage: Export results to JSON Storage-->>BenchmarkService: Results saved CLI->>Device: Pull benchmark results Device-->>CLI: JSON result files CLI->>User: Display benchmark reportContext used:
dashboard- CLAUDE.md (source)Important
Adds comprehensive benchmarking feature for iOS and Android with UI components, model management, and CLI integration.
BenchmarkService,BenchmarkViewModel, andBenchmarkViewfor managing and displaying benchmarks on iOS and Android.BenchmarkLaunchHandlerhandling URL schemes and CLI arguments.BenchmarkConfigandBenchmarkPromptfor configuring benchmark runs.BenchmarkScreenandBenchmarkResultCardfor displaying benchmark results in a user-friendly manner.BenchmarkViewinto iOSContentViewand AndroidAppNavigation.RunAnywhere.registerModel.BenchmarkViewModel.build.gradle.ktsfor building and running benchmarks.This description was created by
for d3bdb1b. You can customize this summary. It will automatically update as commits are pushed.