Cognitora Edge AI

Bandwidth-first boards and clusters for local AI inference

Technical whitepaper, version 0.3, October 2026. EVT stage: the first carrier boards and the Edge Tower dock are designed, with schematics, layouts, BOMs priced from distributors and order files for JLCPCB and PCBWay (section 13). Production specifications are design targets. SoC, DRAM and RK1828 volume prices are estimates. Performance figures are marked as measured or estimated.

Abstract

Cognitora Edge AI is a family of small AI computers for robotics, home inference and low-energy clusters. The design rests on one observation: on small hardware, generating tokens from a language model is limited by memory bandwidth, not by compute. The family has three products.

• Edge Nano, 68.85 x 53.34 mm, under $100 BOM. An RK3588S with a 64-bit LPDDR5 bus, a real-time MCU and 2.54 mm expansion headers, for robots and vision.
• Edge Node, 160 x 100 mm, about $305 BOM. An RK3588 host plus an on-board Rockchip RK1828 coprocessor whose 5 GB of 3D-stacked DRAM acts as VRAM. It decodes an 8B model at 61 tokens/s (measured on the RK1828).
• Edge Tower, four Edge Nodes standing in a dock board around one finned thermal core, cooled by one 160 mm fan. The dock links the nodes in a ring over their two 2.5GbE ports and leaves one 12 V input, one LAN port and two USB-C ports at the rear. With a second RK1828 per node it holds 40 GB of VRAM and runs a 32B model at about 13 tokens/s (estimate).

On 8B-class models an Edge Node decodes about 3x faster than a Jetson Orin Nano Super and about 7x faster than a four-board Raspberry Pi 5 cluster, at a lower BOM than either one's retail price.

1. The problem: decode is a bandwidth equation

A language model generates one token at a time. For each token at batch size 1, the accelerator reads every weight once, plus the KV cache built so far. The arithmetic per byte is tiny: about 2 operations per weight. Speed therefore follows memory bandwidth:

tokens/s  ~  effective_bandwidth / (weight_bytes + kv_bytes_per_token x context)

Measured results on the RK3588 fit this closely. llama.cpp on four Cortex-A76 cores runs Qwen2.5-1.5B (Q4_K_M, about 1 GB of weights) at 22.57 tokens/s and Qwen2.5-7B (about 4.7 GB) at 5.44 tokens/s [1]. Both imply 22 to 25 GB/s of effective bandwidth. Adding TOPS does not move these numbers. Adding bandwidth does.

Memory is now the budget

DRAM prices rose sharply through 2026. A 12 GB LPDDR5X chip cost $77.1 in Q1 2026 and $145.9 in Q2 2026, about $12.2 per GB at the largest buyers' contract prices [2]. TrendForce expects mobile DRAM to rise another 8 to 13% in Q3 2026 and to hold at that level through Q4 [3]. Raspberry Pi reports LPDDR4 up sevenfold in a year [4]. This model uses about $13.4 per GB for Q3 2026.

Three consequences shape the design:

1. Capacity picks the model, bandwidth picks the speed. Every GB costs about $13, so a sub-$100 board carries 2 to 4 GB and targets 1.7B to 4B models.
2. A second memory pool only pays when it adds bandwidth. A separate VRAM chip at host-like bandwidth buys the same bytes twice. The RK1828's stacked DRAM earns its place because it is about 10x faster than the host LPDDR.
3. Bus width comes in packages. A 128-bit bus means two x64 packages and 8 GB or more. One x64 package is the widest bus a $100 board can afford.

2. Design principles

3. Product family

4. Architecture

Two silicon generations share one architecture.

4.1 Gen A: shipping silicon

Gen A uses Rockchip RK3588 and RK3588S. The CPU runs the agent: tokenizer, sampler, tool calls and sandboxes. The Mali GPU handles display and pre and post-processing. The 6 TOPS NPU in three cores runs vision and small language models. On the Edge Node the RK1828 runs the language model from its own stacked DRAM and talks to the host over PCIe 2.1 x1. Memory sharing on the host uses dma-buf buffers with zero copy.

4.2 Gen B: target SoC specification

Gen B is the SoC the family wants next. It is written as a requirements specification for a vendor SoC (the RK3668 class, announced with LPDDR5/5X/6 up to 100 GB/s and a 16 TOPS NPU [5]) or for licensed IP, not as a from-scratch tapeout.

5. Memory hierarchy

Zones in host memory. A V-zone holds weights and KV cache: a physically contiguous carve-out from a dma-buf heap, mapped with 2 MB pages. An S-zone holds the OS and agent sandboxes, capped per sandbox with cgroups v2 so a runaway tool cannot evict the model.

Memory budget, Edge Nano. Values in GB, 4-bit weights, INT8 KV at 4k context.

6. The VRAM tier: RK1828

The RK1828 is a Rockchip AI coprocessor with a 20 TOPS INT8 NPU and 5 GB of 3D-stacked DRAM inside the package, claimed at 1 TB/s. It supports INT4, INT8, INT16, FP8, FP16 and BF16, and connects to the host over PCIe 2.1 x1 [6].

Measured decode, Rockchip RKNN3 runtime, 128 input and 128 output tokens, W4A16 [7]:

Implied bandwidth rises with model size because fixed per-token overheads matter less. The model uses 250 GB/s for larger shards. Under a 7B load the module averages 13.6 W and peaks at 29.4 W [6], so the Edge Node power stage is sized for 30 W and the host power-gates the RK1828 at idle.

The PCIe 2.1 x1 link (0.5 GB/s) sets one hard rule: weights never cross it at run time. Each model slice stays resident in stacked DRAM. Only activations and token IDs cross.

7. Board-to-board link and clusters

Each Edge Node has two 2.5GbE ports, LINK A and LINK B. Standalone nodes chain A to B with short patch cables, no switch, and the head node bridges the chain to the home network.

In the Edge Tower there are no cables between nodes. Each node stands in a slot on a dock board in the base and brings its Ethernet out on its card edge, on the cable side of its own magnetics, so the same port works on the RJ45 or on the dock. The dock connects LINK B of each node to LINK A of the next as PCB traces, which closes a four-node ring. The nodes' management ports go to a 5-port gigabit switch on the dock (Microchip KSZ9567) whose fifth port is the tower's one LAN jack. The dock also carries 12 V to every slot, a USB hub that gives one rear USB-C port access to every node's console and flashing port, a second USB-C port for the head node, the CAN bus and the fan.

Pipeline parallelism. The model is split by layers. Node 1 holds the first quarter of the layers, node 4 the last. For each token, only the hidden state crosses a boundary: 2 bytes x hidden size, about 10 KB for a 32B model. At 2.5 Gbit/s that is about 33 microseconds on the wire.

Latency budget per hop (estimate). RK1828 to host over PCIe, host to host over 2.5GbE, host to RK1828: about 0.25 ms. A 32B model on 8 RK1828s streams 18.4 GB of weights per token at 250 GB/s, about 74 ms. Eight hops add about 2 ms, under 3%.

Pipeline parallelism scales capacity, not single-stream speed: one token still visits every stage in turn. With several users, each stage works on a different request, so aggregate throughput grows with node count.

8. Edge Tower thermal design

The Edge Tower turns four boards into one cooling system. Instead of a heatsink and a small fan on every chip, every hot chip presses onto one shared, finned core, and one slow fan moves all the air.

• Unified thermal core. A 100 mm square aluminium extrusion runs up the middle of the tower, with 28 radial fins inside.
• Boards chip side in. Each Edge Node stands in a vertical slot on the dock, 16 mm from one face of the core, components facing the core. The tallest part (RJ45, 13.5 mm) clears the core face.
• Dock in the base. A round 4-layer board under the core holds the four slots and all inter-node wiring. A 92 mm opening in its centre lets the intake air reach the core. Its rear tongue carries the only external connectors: 12 V (XT60), LAN (RJ45) and two USB-C.
• Copper pedestals. The RK3588, the on-board RK1828 and the M.2 RK1828 press onto copper pedestals machined on the core face, through 0.5 mm thermal pads. No per-chip heatsinks, no per-chip fans.
• One slow fan. A 160 mm fan in the top of the shell pulls air in through an 8 mm gap under the shell, through the dock's opening and up through the core fins.
• Fan-stop at idle (design target). At idle each node power-gates its RK1828s, the tower drops to about 15 W (estimate), and the core works as a convection chimney.

RK1828 figures are measured [6]; host and carrier figures are estimates. The enclosure is 182 mm across and 264 mm tall. The thermal design is a target: a four-node prototype with thermocouples on every pedestal is on the roadmap.

9. Software stack

1. OS layer. Linux with dma-buf heaps for the V-zone, cgroups v2 and cpusets for the S-zone, resctrl-style MPAM controls on Gen B.
2. One model format. GGUF on disk. Gen A runs llama.cpp on the CPU and Mali GPU (Vulkan) and RKLLM on the NPU. The RK1828 runs RKNN3 models. Gen B adds a GGML backend whose weight-decode DMA reads GGUF blocks directly.
3. Accelerator partitions. On the RK3588 NPU, one core for always-on vision and two for the LLM.
4. Prebuilt decode step. One recorded command list per decode step, replayed every token.
5. KV manager. Paged INT8 KV cache, a shared prefix cache for system prompts and tool schemas, spill of idle sessions to flash.
6. Speculative decoding. A small draft model proposes several tokens and the main model verifies them in one weight pass. The gain depends on acceptance rate and must be measured per workload.
7. Agent runtime on the big cores. Tokenizer, grammar-constrained sampling for tool calls, and WebAssembly sandboxes pinned to the Cortex-A76 cluster.
8. Pipeline runtime. A stage daemon per node, stage assignment by layer count and free VRAM, activations over LINK A/B.
9. Real-time MCU. Motor, IMU and CAN-FD loops on Zephyr, bridged to Linux over RPC. The MCU also wakes the host on voice, network or sensor events.
10. Bandwidth telemetry. DDR performance counters feed admission control.

10. Performance summary

Single board, 4-bit weights, short context:

Energy. An Edge Node under an 8B load draws about 18.6 W (13.6 W RK1828 average [6] plus about 5 W for the host, estimate), about 3.3 tokens per joule. A Jetson Orin Nano Super at its 25 W mode delivers 19.1 tok/s on Llama 3.1 8B [8], about 0.8 tokens per joule at that power.

11. Cost

BOM plus assembly at 10k units, LPDDR at about $13.4 per GB (Q3 2026). Volume SoC and RK1828 prices are estimates. The carrier lines (board controller, connectors, passives and ESD, the 12 V input stage) use the EVT BOM's distributor prices at qty 1000 (section 13), which added $24 to the Edge Node and $3.50 to the Edge Nano over the concept estimates. The Edge Tower is four Edge Nodes plus the dock, whose lines also come from its EVT BOM at qty 1000 (about $68 built); the thermal core, shell, fan and power supply are not included. Full line items are in docs/data/bom_edge_nano.csv, docs/data/bom_edge_node.csv and docs/data/bom_edge_tower_dock.csv.

The 4 GB Edge Nano falls under $100 once LPDDR drops below about $7.88 per GB, between the Q1 2026 level (about $6.4 per GB) and the Q2 2026 level. The same PCB takes both memory sizes.

12. Alternatives

8B-class dense models, 4-bit, batch 1. Cognitora figures are BOM, the others are list or retail prices, so prices are not like for like. At a 1.4x retail markup an Edge Node would sell for about $427, close to the $399 of a Jetson Orin Nano Super, with about 3x its decode speed.

A Raspberry Pi 5 with the AI HAT+ 2 (Hailo-10H, 8 GB, $130 for the HAT) reaches up to about 8 tok/s on 1 to 1.5B models and does not run 8B models [12].

13. From concept to EVT hardware

The first hardware is an engineering-validation (EVT) build of each board as a carrier for a bought-in compute module. The module brings the SoC, DRAM and accelerator that already work, so the first spin tests what the carrier adds: power, I/O, the board controller, the RK1828 power path, the tower mechanics and thermals. The carrier circuits carry over to the chip-down production boards.

The KiCad projects are generated by Python in hardware/evt/gen/, so every pin assignment can be reviewed as code. Pinouts and sub-circuits come from machine-readable open designs: Antmicro's Jetson Orin baseboard (Apache-2.0) supplies the SO-DIMM pin functions and the bucks, eFuses, M.2 slot, USB power switches and ESD, copied with their parts and values. Both schematics pass KiCad's ERC with no errors or warnings, and the exported netlist is checked back against the intended connections. Placement is DRC-clean on both boards. Routing is a Freerouting first pass.

Cost of the EVT build. From the real BOM, distributor prices taken on 8 October 2026, and JLCPCB's assembly price rules:

At qty 1000 the carrier parts come to $57 (Node) and $17 (Nano). Line by line against the concept estimates, connectors and passives were underestimated (Node connectors $19 against $8, passives and ESD $10 against $4) and Ethernet was overestimated. Section 11 now uses the EVT figures.

What the EVT pass changed. - The reference design's RJ45 jack is superseded, so the Node uses jacks from the KiCad library. - The RK1828 AUX outputs got a 3 A circuit breaker. The reference value would have allowed 6 A into a card cable. - The host kill switch got a gate pull-down, so the host stays on while the controller is in reset. - The expansion headers moved onto the standard 68.6 x 53.3 mm pattern. - Mounting holes now keep standoff screws clear of bottom-side parts.

Not yet ready to order. The PCIe, USB 3, CSI-2 and Ethernet pairs are autorouted without length matching or impedance-checked geometry. They need a hand-routing pass and a signal-integrity sign-off before Gerbers go out. At bring-up, the RK1's PCIe and USB 3 lane mapping is checked against its device tree. Details and the full bring-up list are in hardware/evt/README.md.

14. Risks and roadmap

Roadmap

1. Validate on existing hardware: an RK3588S board plus an RK1828 M.2 card. Gate: Qwen3-8B at 55 tok/s or more, two-card pipelining works.
2. EVT carriers for bought-in modules (this revision, section 13), then chip-down Edge Nano and Edge Node layouts. Gate: carriers bring up, BOM quotes at or under $100 and $300.
3. Runtime: V-zone daemon, NPU partitioning, KV manager, speculative decoding, pipeline over LINK A/B.
4. Edge Tower prototype. Gate: no thermal throttling at full load.
5. Gen B: evaluate RK3668-class samples. Gate: 45 GB/s or more effective and a landed SoC price at or under $30.

15. Reproduce

python3 model/perfmodel.py                 # tables, docs/data/*.csv, docs/figures/*.png
python3 hardware/specs.py                  # board JSON specs
pip install bpy==5.2.2                      # Blender as a Python module
python3 render/blender/render_boards.py     # board renders
python3 render/blender/render_cluster.py    # Edge Tower
python3 render/blender/render_diagrams.py   # 3D architecture diagrams
python3 web/viewer/build_viewer.py          # interactive viewer, web/viewer/index.html

References

1. Turing Pi, "LLM inference benchmarks on RK3588: GGUF quantization", https://turingpi.com/llm-inference-benchmarks-rk3588-gguf-quantization/
2. Wccftech, citing Sigmaintell, LPDDR5X 12 GB chip cost Q1 and Q2 2026 (June 23, 2026), https://wccftech.com/apple-is-in-for-a-sticker-shock-in-q3-with-lpddr5x-dram-costs-surging-by-68-8-in-a-single-quarter-as-operating-profit-margin-for-general-purpose-dram-to-hit-90-within-the-year/amp/
3. TrendForce, Mobile DRAM contract price 3Q26, https://www.trendforce.com/research/download/RP260803NC
4. BGR, "Not even the Raspberry Pi is safe from the RAM crisis" (April 23, 2026), https://www.bgr.com/2150106/raspberry-pi-cost-ram-crisis/
5. CNX Software, Rockchip RK3668 and RK182X announcement (July 2025), https://cnx-software.com/2025/07/18/rockchip-unveils-rk3668-10-core-arm-cortex-a730-cortex-a530-soc-with-16-tops-npu-rk182x-llm-vlm-co-processor
6. DFRobot, RK1828 AI accelerator M.2 module (DFR1263), https://www.dfrobot.com/product-3142.html
7. DFRobot wiki, RKNN3 supported models and performance, https://wiki.dfrobot.com/dfr1263/docs/24735
8. NVIDIA Technical Blog, Jetson Orin Nano Super, https://developer.nvidia.com/blog/?p=93942
9. Orange Pi 5 Pro 4 GB LPDDR5 price history, https://pricehistory.app/p/orange-pi-5-pro-4gb-lpddr5-rk3588s-zur6eOyp
10. Notebookcheck, "Nvidia quietly increases Jetson prices by up to 101%" (July 22, 2026), https://www.notebookcheck.net/Nvidia-quietly-increases-Jetson-prices-by-up-to-101.1348323.0.html
11. Zenodo, DeepSeek-R1-8B on four Raspberry Pi 5, https://zenodo.org/records/20411979
12. Hardware Corner, "Local LLMs on Raspberry Pi: what the AI HAT+ 2 can and cannot do", https://www.hardware-corner.net/local-llms-raspberry-pi-ai-hat-plus-2/
13. Qwen3 model configurations, https://huggingface.co/Qwen