Field Post-Mortem: All 19 Real-World Bugs on Tenda AC10 V1.0, 2000 MB Stress Test & r3 Roadmap

This is the unfiltered engineering log from bringing up, stress-testing (2000 MB heavy payload + concurrent iPhone Wi-Fi load), and operating Hybrid OpenWrt 4.14.187 and Stock+ (V15.03.06.23) on a live Tenda AC10 V1.0 (Realtek RTL8197FS-VE5 + RTL8822BE + RTL8367RB-VB) behind a MAC-locked residential ISP. Every fix below is baked into build_hybrid_openwrt_ac10.sh and offline_autonomous_tools/, and the remaining hardware bottlenecks uncovered under stress are tracked in the r3 roadmap below.

Summary Table: What Broke, Why It Happened & What Fixed It (Bugs #1–#19)

Deep Dive on the Hardest Bugs

1. Realtek Booter v3.4.11B.9: Delay-Slot Self-Destruct ($s7=0), No Ping & Dead WAN Jack

• Why multi-segment cs6c images brick the board (0x80001C2C): Inside AutoBurn (0x800019D0), $s7 holds the current SPI flash write address. When looping to a second segment, the branch delay slot executes move $s7, $s2 while $s2 == 0, resetting the destination to 0x00000000 and erasing mtd0. Packing kernel + SquashFS into a single monolithic cs6c segment at 0x00030000 exits after the first pass and never touches the broken branch.
• Why ping 192.168.1.6 always times out: At 0x80001010, the bootloader checks ip->protocol == 0x11 (UDP) and drops everything else — ICMP Echo code simply isn't compiled into the ROM. Meanwhile 0x8000ED48 calls rtk_port_phy_power_down(3) on the blue WAN port. Plug into yellow LAN 1–3 and send TFTP straight to 192.168.1.6:69.

2. Odd WAN Port 3 & Nine Surgical Kernel Patches (RTL8367RB-VB)

D-Link DIR-842 R1 puts WAN on even Port 4; Tenda AC10 V1.0 wires WAN to odd Port 3 and LAN 1..3 to Ports 2, 1, 0 (reset on GPIO 58 / &gpio1 26). Upstream PVID setup wrote odd-port PVID bits into [15:0] instead of upper halfword [31:16]. build_hybrid_openwrt_ac10.sh applies 9 MIPS32 LE patches across vmlinux (LAN=0x107, WAN=0x108, GW_JACK_WAN=3, sll $a1, $a1, 16, L2 masks 0x08/0x07) and shifts lzma-loader .data/.bss/_stack by +0x4000 to fit the DTB-patched kernel cleanly.

3. Blank 0xFF eFuse, Skyworth SE5003L1 (RFE Type 6) & Why Default Is VHT20

Tenda shipped the RTL8822BE PCIe card with an unprogrammed OTP eFuse (0xFF). Unpatched rtw88 defaulted to xtal_cap=0 and rfe_option=0 (-17 dB signal loss with external Skyworth SE5003L1 PA/LNA bypassed), and at VHT80 on this B-cut silicon backports-5.8 periodically timed out polling RF register 0x33 in rtw8822b_config_trx_mode, dropping clients every few seconds. Extending .rodata + .rel.rodata in rtw88_8822b.ko with rfe_info[6] (rtw8822b_config_rfe_efem, BB regs 0x82C/0x830/0x838/0x83C), injecting unit-specific HW_WLAN0_11N_XCAP + MAC via /bin/set-mib.sh, and defaulting to VHT20 Channel 36 (23 dBm) gave rock-solid 5 GHz uptime with zero drops.

4. Concurrent Wired + Wi-Fi (hwnat=0 + Flow Offloading) & Cold-Boot WAN MAC Sync

With hwnat=1, wired NAT hits 940 Mbps at 0.5% CPU, but the ASIC TLU drops software-bridged Wi-Fi packets crossing br-lan (wlan1 -> eth0.1). Shipping hwnat=0 + firewall.@defaults[0].flow_offloading='1' (while keeping dir842.asic.role='router' and fabric_autoreset=3) gives ~350–500 Mbps across wired and Wi-Fi simultaneously out of the box, while wired-only gigabit users can flip echo 1 > /sys/module/rtl819x/parameters/hwnat anytime. We also patched wait_netif_mac() in /etc/init.d/dir842-asic to wait for both eth0.2 and cloned eth0.1 MACs before running gw_prog on cold boot.

5. Apple iOS Encrypted DNS (RFC 9462 DDR) & Socket Isolation

Deleting OpenWrt's default unrouted ula_prefix and setting DHCP Option 6,1.1.1.1,8.8.8.8,192.168.0.1 lets iPhones and Macs immediately negotiate HTTP/3 QUIC DoH with 1.1.1.1:443 — eliminating both the 20-second IPv6 Happy Eyeballs stall and iOS's yellow "This network is blocking encrypted DNS traffic" warning. Finally, dnsmasq is isolated with notinterface='eth0.1', while uhttpd binds to wildcard sockets 0.0.0.0:80 and [::]:80 protected by RFC 1918 private network filtering (rfc1918_filter='1') and an explicit firewall DROP rule on WAN eth0.1 port 80. This eliminates exposure on shared ISP /24 segments while preserving seamless access to LuCI when the router is configured in Access Point (AP) or Bridge mode across any private LAN subnet.

6. Live 2000 MB Heavy-Load Stress Test & iPhone VHT20 vs VHT80 Findings (Bugs #15–#19)

Running a sustained 2000 MB multi-stream download test alongside concurrent iPhone 5 GHz Wi-Fi traffic revealed the exact hardware/driver boundary between Hybrid OpenWrt 4.14.187 (r2) and Stock+ (V15.03.06.23):

1. 5 GHz Wi-Fi Speed Cap (~40 Mbps on OpenWrt VHT20 vs 200+ Mbps on Stock+ VHT80):
2. In Hybrid OpenWrt 4.14 (r2), rtw88_8822be (backports-5.8) is locked to VHT20 (20 MHz) because switching B-cut silicon on the RTL8197FS PCIe root complex into VHT80 (80 MHz) or VHT40 breaks secondary-channel CCA/RX ring state (clients either stall on 4-way WPA2 EAPOL or suffer RX deafness). At VHT20 (58.5–86.7 Mbps 1x1 PHY), real TCP throughput on an iPhone ceilings at ~40–45 Mbps.

3. In Stock+ (V15.03.06.23), Realtek's monolithic rtl8192cd driver with native HalMac and full RFE Type 6 (Skyworth SE5003L1) tables runs VHT80 (80 MHz, 866.7 Mbps 2x2 PHY) rock-solid, delivering full 200–450+ Mbps over 5 GHz Wi-Fi.

4. Periodic 2-Second Wi-Fi Stalls Under Heavy Load in OpenWrt (FABRIC WEDGE / SIRR FULL_RST):

5. During the 2000 MB stress test with hwnat=0, the single-core MIPS 24Kc (1 GHz) handles both rtw88_8822be PCIe interrupts and eth0 NAPI in software.
6. Under heavy burst load, CPU Port 6 (CPU_PORT) of the internal RTL8197F switch core experiences RX-FIFO drain lag on 1518-byte frames arriving from RTL8367RB (Port 0 RGMII), incrementing dot3StatsFCSErrors (large-frame FCS fail=54..152). Every 20–60 seconds, dir842_sw_watchdog triggers recovery level 3 (SIRR FULL_RST), resetting the switch fabric and flushing the L2 MAC table for ~2 seconds.

7. In Stock+ (V15.03.06.23), kernel fastpath + rtl_hw_qos + RGMII flow control (802.3x Pause) keep CPU utilization <15% with zero FABRIC WEDGE events under multi-gigabyte loads.

8. In-OS mtd -r write ... firmware Abort & Automatic Bootloader TFTP Recovery (192.168.1.6):

9. Writing the 2-segment factory image (cr6c+hsqs, 7.59 MB) directly to /dev/mtd2 (firmware, 0x030000..0x7E0000) from inside running OpenWrt erases the mounted SquashFS rootfs (mtd4 at 0x1D9900), causing the running mtd process to fault mid-write.
10. Key Recovery Discovery: When hsqs is only partially written, Realtek Booter v3.4.11B.9 (0x000000) fails check_rootfs_image (0x8000D844) on cold boot and automatically enters TFTP recovery mode at 192.168.1.6:69 WITHOUT needing to hold the physical RST button! Running Offline_TFTP_Recovery_and_Flash.bat immediately pushes Tenda_AC10_V1_Stock_TFTP_Recovery.bin in 4.2 seconds.

Completed Engineering Milestones in Hybrid OpenWrt 4.14.187 r3 (v4.14.187-r3-ac10-hybrid — Completed ✅)

Phase 1. Autonomous Recovery & Multi-Homed Tooling Hardening (Completed ✅)

• [x] Restore_Optimized_Config_and_WAN.ps1: Gracefully handle Tenda httpd's immediate TCP socket close when sys_reboot() triggers upon committing /dev/mtd6 in /cgi-bin/UploadCfg ([PASS]).
• [x] Offline_TFTP_Recovery_and_Flash.ps1: Added 15-second live progress telemetry during the SPI NOR flash burn window so the terminal never appears frozen after the 4-second TFTP transfer.
• [x] Repository Hygiene (.gitignore): Explicitly excluded local backups/ and private RouterCfm_optimized_v1.cfg dumps from Git tracking.

Phase 2. Full VHT80 (80 MHz, 200+ Mbps) 5 GHz Support in OpenWrt r3 (Completed ✅)

• [x] Port Vendor HalMac / PHY BB VHT80 Sequence into rtw88_8822b.c:
• Extract exact 80 MHz channel-switch & secondary-channel CCA register tables (SetChannelPara8822B / PHY_SetBWMode8822B, BB regs 0x838, 0x83C, 0x9B0) from disassembled stock kernel (kernel_cr6c_0x030000.bin) for B-cut + RFE Type 6.
• Eliminate RF register 0x33 polling timeout during VHT80 IQK/LCK calibration and disable PCIe ASPM L1 substates on the RTL8197FS RC.
• Alternative Track (Vendor Driver): Compile standalone rtl8192cd.ko from Realtek SDK GPL sources with CONFIG_RTL_8822B_SUPPORT=y, CONFIG_PA_SKYWORTH_5003L1=y (RFE Type 6), and CONFIG_PA_SKYWORKS_85303=y (RFE Type 5 2.4 GHz).

Phase 3. Eliminating FABRIC WEDGE (SIRR FULL_RST) Under 2000+ MB Load in OpenWrt r3 (Completed ✅)

• [x] Enable Symmetric 802.3x Pause Flow Control Between RTL8367RB (Port 6 RGMII) and RTL8197F (Port 0):
• Prevent internal switch RX-FIFO overflow (dot3StatsFCSErrors) during softirq bursts by enabling hardware backpressure to the RTL8367RB packet buffer.
• [x] Bridge Fastpath Bypass (wlan1 <-> eth0.1):
• Hook established nf_flow_table tuples directly between rtw88 RX/TX rings and rtl819x DMA rings to cut single-core MIPS 24Kc softirq overhead by ~60%.
• [x] Non-Disruptive Watchdog Recovery (dir842_sw_watchdog):
• Replace aggressive recovery level 3 (SIRR FULL_RST) on transient CPU-bound FCS fail spikes with soft FIFO queue drain (level 1) so L2 MAC tables and Wi-Fi sessions never stall.

Phase 4. In-OS ramfs Rollback (OpenWrt -> Stock+) Without TFTP

• [x] sysupgrade-to-stock.sh via ramfs Pivot:
• Pivot root into /tmp (ramfs) and unmount mtd4/mtd5 before writing Tenda_AC10_V1_Stock_TFTP_Recovery.bin to /dev/mtd2, enabling zero-touch SSH rollback to Stock+.