Validation
How accurate is Migratorisk?
We test the scanner against what actually happens when real firmware is upgraded: real builds, public open-source projects, and the SDK vendors' own ports of their example code. These are the results, how each was measured, and what the scanner cannot do.
Results
| Test | Result | How it was measured |
|---|---|---|
| Real-build benchmark (nRF Connect SDK and Zephyr) | 20 of 20 build errors and deprecation warnings predicted; 5 of 15 exact-symbol findings confirmed by the build log | 11 public projects (Nordic and Zephyr example applications and samples, hello.nrfcloud.com firmware) were built unchanged against nRF Connect SDK 3.4.0 or Zephyr 4.5.0. Each error in the build log was compared with the scan of the same path. The scanner was improved against this same set (10 of 20 before), so this is an in-sample result; expect lower recall on new projects. Unconfirmed findings may still be real, for example behaviour or default changes that do not fail a build. |
| Silicon Labs: the SDK's own examples, Gecko SDK 4.4.0 to Simplicity SDK 2024.6.0 | 454 of 454 build-breaking name changes and 10 of 10 dropped components predicted; 466 of 505 changed names predicted overall; 0 of 1,558 flagged removals still used | 58 Zigbee, Bluetooth and Bluetooth Mesh examples present in both releases were scanned on the old release and compared with Silicon Labs' own updated versions. Ground truth comes from the SDK headers, not from the scanner's knowledge. The 39 unpredicted names were optional edits to APIs that still exist. One generator fix (Bluetooth event ids) was made after a similar comparison on another repository. |
| Silicon Labs: Zigbee rename suggestions | 25 of 25 suggested EmberZNet 8 names match the names Silicon Labs used | The Z3Switch example's application code on Gecko SDK 4.4.0 was compared with Silicon Labs' version for Simplicity SDK 2024.6.0. |
| Open-source project corpus | 1,615 applications from 59 public projects scanned with no invariant failures | Nordic, Zephyr, Espressif and Silicon Labs repositories and community firmware are scanned to the latest releases. Each scan is checked for crashes, valid findings, evidence that stays inside the project, report rendering, sanitized sharing and runtime. This checks robustness, not prediction accuracy. |
| Reading real builds | Build configuration, board, SDK version, partition layout and image size read correctly | nRF Connect SDK 2.9 (with MCUboot), Zephyr 4.0 and ESP-IDF 4.4 and 5.1 projects were built with the official toolchains and scanned with their build folder. An ESP-IDF 4.4 project built with 5.1.6, and 3.3-era code built with 4.4.8 and 5.1.6, had every reported build error predicted by the scan. |
Last updated with Migratorisk 0.3.0. The test scripts are part of the scanner's repository (scripts/benchmark.py, scripts/oss_corpus.py, scripts/check_build_scan.py, scripts/check_silabs_ports.py), so every number can be rerun.
What it cannot tell you
- It reads files; it does not build your firmware or run it on hardware. A clean report does not prove the upgrade is safe.
- Behaviour changes, timing, radio performance and power consumption are flagged for review, not measured.
- Findings are matched against documented changes. Undocumented SDK changes and bugs in a new release are not known in advance.
- Silicon Labs releases from 2025.12 are covered from release notes, not source, and are marked lower confidence.
- Accuracy numbers come from public projects. Your project may use SDK features those projects do not.
Try it on your own firmware
Migratorisk scans your project on your own computer and reports only the changes your code, configuration and build actually touch. Your source code is not uploaded.
python -m pip install migratorisk migratorisk scan --project ./your-firmware --from ncs-2.9.0 --to ncs-3.0.0 --out report.html
Requires Python 3.12 or newer. Results are free and complete without sign-up; sharing a report with us is optional.
Set up the local scanOr answer a few questions insteadVelocityCreate
Want us to handle the upgrade?
VelocityCreate can review your assessment, agree on the migration scope, and carry out the firmware changes and agreed verification. Discuss your deadline, hardware, and requirements with an embedded engineer.
Discuss my migrationBook a technical callThe initial discussion is free. Implementation is quoted separately.