When etcp_conn_reinit is called, it clears ETCP queues (input_wait_ack
etc.) which frees dgram objects to data_pool. If a router retransmit
timer fires concurrently, it sends through the same conn → normalizer
allocates from corrupted data_pool → SIGSEGV.
Fix: call etcp_router_pause_retrans_for_node() BEFORE etcp_conn_reset()
to cancel retrans timers and free inflight_q entries for all router
connections to the peer. No callbacks, no close notifications —
lightweight pause, router connections stay alive and resume when
ETCP connection comes back up.
- conn_mgr.c: 5 call sites used queue_entry_new(data_size) with data[] instead
of dgram — all etcp_route_send calls silently dropped. Fixed by switching to
dgram-based allocation (queue_entry_new(0)+u_malloc) matching all other callers.
- DIRCECT_REQ reuses already-allocated pkt as qe->dgram (no extra memcpy).
- etcp_router.c:1078 — 'empty entry' now shows dgram=%p len=%u dst=0x... force=%d
- etcp_connections.c:1139 — 'bad args' now shows all arg pointers to identify NULL
- etcp_connections.c:1145 — 'no socket' now shows addr_family=%d
- conn_mgr.c:633 — 'no candidates' now notes direct+reverse failed
- Renamed TOPO_BGP to TOPO_GROUP with topo_group_id (64bit)
- Added TOPO_GROUPS container (ll_queue group_list) in UTUN_INSTANCE
- Memory pools moved from TOPO_GROUP to TOPO_GROUPS (instance-level)
- Default group: TOPO_GROUP_UTUN = 0x8000000000000000
- topo_groups_get_default() searches by group_id, not just first entry
- All topo_bgp_* renamed to topo_group_*; function signatures updated
- Files: topo_bgp.h/c -> topo_group.h/c
- NODEINFO_MSG (protocol, packed): added flags byte + group_id field
- NODEINFO (memory): ref_count, linked list heads instead of inline arrays
- NODEINFO_ROUTES: separate struct with linked list subnet heads
- NODEINFO_Q: node*, routes*, tranzit_data*, hop_list* as separate mallocs
- 6 memory_pools in ROUTE_BGP for NI_* linked list items
- ni_list_count() universal counter via _ni_head cast
- nodeinfo_serialize/deserialize for protocol <-> memory conversion
- NODEINFO_FLAG_SEND_SUBNETS controls subnet data in protocol
- group_id defaults: utun=NODEINFO_GROUP_UTUN(1) with subnets
Fixes:
- deserialization: data+2 instead of data+sizeof(BGP_NODEINFO_PACKET)
- memset: start after ll_entry to preserve ll.size
- hop_src: subtract hop_count*8 to point at correct dynamic offset
- same-ver branch: remove_path(conn) instead of remove_path_by_hop(peer)
- double-free in route_bgp_remove_conn/process_withdraw
- conn_mgr_add_alien_node: rewritten for new structures
- all test files updated for new API
- TRANSIT_QUEUE: отдельная очередь на каждую пару src+dst узлов
- ll_queue с хеш-индексом для быстрого поиска transit-очередей
- backpressure через waiter на send_input_q (threshold=0, один пакет за вызов, round-robin)
- Очередь создаётся при первом пакете который не получается отправить напрямую, удаляется при опустошении
- Унифицирован etcp_send: единый путь queue_data_put(send_input_q) для UDP (normalizer->input) и STCP (tx_queue)
- Убран STCP-ветвления из router_forward_transit/drain_cb
- Исправлено перекрытие ll.data и tq->q в TRANSIT_QUEUE (добавлены явные поля src_node_id/dst_node_id)
- Фикс лика dgram в tx_queue_cb/client_tx_queue_cb (добавлен queue_dgram_free)
- transit_queues живут внутри ETCP_CONN, инициализируются лениво, очищаются в etcp_connection_close (до pn_deinit) и stcp_link_close
uasync: replace raw socket_node* handle with packed index — fixes UAF after socket_array realloc
tcp_io: defer u_free in tcp_conn_destroy via uasync_call_soon — prevents callback-chain UAF
tcp_io: guard read_cb/write_cb with NULL queue checks after deferred destroy
tcp_io: save read_queue to local before queue_data_put + NULL guard after
route_connectivity: linked-list probe_ctx — cancel all parallel probes before nq free
- Add ed25519_public_key[32] to NODEINFO struct for pubkey distribution
- Add ed25519_public_key[32] to ROUTE_BGP (derived from X25519 privkey at init)
- Add ROUTER_FLAG_SIGNED (0x08) to SVC_ROUTE_HDR flags byte
- router_send_one_flags: when is_signed, sign [hdr][payload] and append 64-byte sig
- etcp_router_recv_cb: verify Ed25519 signature when ROUTER_FLAG_SIGNED set
- New API: etcp_router_conn_send_signed()
- Drop signed packets if sender node not in routing table or no Ed25519 key
- 5 unit tests in test_etcp_router_unit.c covering OK/tampered/unknown/no-key/short
SVC_ROUTE_HDR: +uint8_t flags (23 bytes total).
bit7=START (0x80) — first data packet of new session
bit6=RST (0x40) — sequence violation detected
bit5-4=sess_id (0-3) — cyclically incremented on restart
router_send_one_flags: sets START+sess_id on first data packet.
router_send_ack: sets sess_id only (no START for ACK).
etcp_router_recv_cb: detects restart via sess_id change or START flag.
Old seq=0 heuristic replaced with explicit flags.
etcp_router_conn_get: init sess_id=0, peer_sess_id=0, start_sent=0.
etcp_router_conn_restart: increment sess_id before closing rconn.
etcp_router: new etcp_router_conn_restart(inst, node_id, svc_id) — closes
all rconns for the peer, notifies service via cb(NULL, entry) with
node_id, resets seq state.
seq=0 detection in etcp_router_recv_cb: when rx_seq >= 256 (MAX_INFLIGHT)
and a data packet arrives with seq=0, detect peer restart and reset rconn.
tcp_proxy_client: handle TCP_PROXY_SUBCMD_RESTART — clear all client conns
and server conns for the restarted peer.
Threshold prevents false positives on in-window seq=0 duplicates.
Six key log points at DEBUG level to trace the full ACK chain:
ETCP_SEND — server sends direction A data
ETCP_RECV — received any svc_route packet
SEND_Q_DRAIN — send_q drained after client ACK
SEND_Q_ACKED — server received client ACK (direction A)
CONSUMER_ACK — server sends consumer ACK (direction B)
ACK_SEND — low-level ACK packet sent
router_schedule_ack skips when rx_seq==last_sent_ack_seq (no-op),
so the first consumer_ack call after setting consumer_ack=1 was
silently ignored. Client never got ACK → send_q stuck at 256.
router_ack_do_send always sends the ACK unconditionally.
- consumer_ack flag: ACK sent only on consumption, not assembly
- etcp_router_consumer_ack() — called from tcp_proxy_client feed_from_transport
- last_ack_sent_tb: interval-based throttle — send immediately if >=10ms passed,
otherwise timer for remaining time
- timer rules: NULL handle after cancel/fire, NULL check before start
- ROUTER_ACK_INTERVAL_TB 1000→100 (100ms→10ms)
- test_etcp_router_unit: updated test 14 for new immediate-send behavior
rx_acked was used for two conflicting purposes:
1. remote ack of our sends (inflight = tx_seq - rx_acked)
2. our last sent ACK seq (dedup: rx_seq != rx_acked)
When the ack timer fired and set rx_acked = rx_seq, it overwrote
the inflight-tracking value. If rx_seq > tx_seq, the computation
tx_seq - rx_acked underflowed (e.g. 18 - 24 = 0xFFFFFFFA),
permanently blocking router_drain_send_q and causing send_q to
grow indefinitely.
Fix:
- Split rx_acked into tx_acked (remote ack, for inflight) and
last_sent_ack_seq (our ACK, for dedup)
- Incoming ACK handler only advances tx_acked forward (stale guard)
- Use int32_t cast on all inflight comparisons to handle stale states
- Add etcp_router architecture diagram (doc/etcp_router_arch.md)
- ROUTER_SEQ_CLOSE_FLAG (0x80000000): normal conn close
- ROUTER_SEQ_RST_FLAG (0x40000000): conn not found
- router_send_to(): SVC_ROUTE with flags, no conn needed
- router_send_close_to_service(): notify local service [svc_id],len=1
- router_close_and_notify(): CLOSE to remote + close to local + cleanup
- etcp_router_conn_close(): sends CLOSE to remote before cleanup
- etcp_router_conn_close_all_for_node(): close all conns for dead peer
- recv_cb: flags checked only when pl_len==0 (avoid seq collision)
- etcp_router_destroy: close notification to each service
- New SVC_ROUTE_HDR (packed struct, 22 bytes): cmd+dst+src+seq+svc_id
- ETCP_ROUTER_CONN: state per (remote_node_id, svc_id), hash-indexed in router_conns
- Reorder via recv_q (hash by seq), dedup with 32-bit circular compare
- Periodic ACK (100ms), idle ACK (500ms), inflight limit via tx_seq - rx_acked
- Legacy mode: no conn → direct delivery without reorder
- etcp_router_conn_get/send/close API for seq-managed connections
- Unit test test_etcp_router_unit: 18 tests, 3ms, no ETCP/sockets/BGP
etcp_router: routable service packets with dst/src node_id
- etcp_router_bind/unbind: per-instance service handler registry
- etcp_route_send: send to node by id via BGP routing
- multi-hop forwarding through intermediate nodes
- loopback optimization (dst==self dispatches directly)
remote_proxy: exit node creating OS sockets on demand
- handles CONNECT/DATA/CLOSE via etcp_router
- non-blocking socket connect with uasync callbacks
- echo back response data to requesting node
tcp_proxy: etcp_transport for remote proxy
- via_node_id in mapping selects between sock/etcp transport
- unified handler for both tcp_proxy and remote_proxy roles
- config: forward = port -> ip:port via NODE_HEX [remote_proxy] section
tests: test_etcp_router (20 pkts bidirectional), test_remote_proxy (echo)
Evgeny