Commit Graph
34 Commits
Author SHA1 Message Date
evgeny 25e2ffeea7 Stream member files with router backpressure and parallel transfer handles 2026-10-05 23:36:49 +02:00
evgeny e3caf15f32 Guard routing loops and reconcile blocked transit against current paths
Track visited nodes outside the signed body for all routed packet types. Recheck queued transit routes without losing FIFO priority, reject unknown-group transit, and suppress BGP announcements caused only by RTT jitter.

Document routing rules and wire format. Validate graph churn, REINIT, encrypted transit, malformed paths, loop drops and queue cleanup; full suite: 106 passed, 1 skipped, plus proxy integration and sanitizer checks.
2026-09-28 15:36:47 +03:00
evgeny 3131dcbb7c Fix ETCP session recovery and router lifecycle; confirm chat invites before publishing 2026-09-27 02:51:47 +03:00
evgeny 6163f34ce1 etcp_router: эпоха reset_id на wire + master/slave — фикс потери пакетов при рестарте пира 2026-09-06 22:20:08 +03:00
evgeny 53bce51201 etcp_router: корректный handshake/restart без таймаут-петли
- START на первом пакете сеанса (seq=0), гейт отправки до первого ACK
- дедуп рестарта по payload init-пакета (ретрансмит START не триггерит рестарт)
- RST переинициализирует только отправку (без CLOSE_ALL), дедуп повторных RST
2026-09-06 00:33:45 +03:00
evgeny 01ea6913ae etcp_router: sess_id как локальный счётчик переустановок (не в протоколе) 2026-09-05 20:38:17 +03:00
evgeny 2d04bf941b etcp_router: вынести подпись/шифрование в route_crypto, одна функция отправки с флагами
- route_crypto_encode/decode: автономный пред/пост-обработчик пакета
  (encrypt-then-sign, стандартный Ed25519 вместо sc_stream_sign)
- одна функция отправки etcp_route_send/etcp_router_conn_send с bitmask mode
- формат доставки: [svc_id][src][dst][rx_flags][payload] (статус подписи/шифрования)
- ROUTER_INFLIGHT хранит финальный wire-пакет (чинит потерю подписи при ретрансмите)
- дедуп инициализации: router_conn_reset/free_queues
- учёт inflight через inflight_q count (seq присваивается на enqueue)
2026-09-05 17:40:43 +03:00
evgeny 7817bb96aa etcp_router: START/RST handshake для переживания рестарта пира
Раньше START ставился один раз при первой отправке и терялся в ретрансмите,
а принимающая сторона молча авто-синкала rx_seq на любой первый пакет —
после рестарта пира/среднего узла seq рассинхронизировались (тихий data loss
или no-ACK close). Теперь:

- Клиент: флаг start_sent «новый» до первого ACK; пока он стоит, первый
  seq (0) шлётся со START, включая ретрансмит. Первый принятый ACK
  (tx_acked>0) сбрасывает флаг. При рестарте conn флаг снова «новый».
- Сервер: после рестарта peer_sync_done=0 («ждёт START»). Не-START пакет →
  отправка RST (через recv_conn). START → синк rx_seq + ACK → обычный режим.
- RST на принимающей стороне теперь сбрасывает локальное состояние
  (etcp_router_conn_restart), а не закрывает conn.

Тесты unit обновлены: первый data-пакет сессии инжектится со START.
2026-09-05 15:03:44 +03:00
evgeny 5b8c725c01 etcp_router: round-robin send + контроль заполненности send_q через waiter на send_input_q 2026-09-04 19:30:08 +03:00
evgeny f47ad7d87f tcp_proxy: fix throughput (tx_queue backpressure) + connect timeout + integration test
- Router: unified service delivery format [svc_id][src][dst][payload] (end-to-end src/dst from SVC_ROUTE header)
- tcp_proxy_client: replace single tx_buf drop-on-backpressure with bounded tx_queue + send_q waiter + retry timer (no data loss, bulk window reopen — removes 62.5ms delayed-ACK stall)
- tcp_io: configurable connect timeout (tcp_conn_set_connect_timeout)
- tcp_proxy_server: connect_timeout config option (ms, default 2000, 0=off)
- check.sh: run tcp_proxy_full integration test with sudo (skip if passwordless sudo unavailable)
- test_tcp_io: connect timeout unit test; docs (lwip_nuances.md)
2026-09-03 13:49:09 +03:00
evgeny bd6cd17bdf tests: fix close/restart detection after loopback_conn change
bb0866b changed close/restart notifications from NULL to loopback_conn().
Remove !conn check — use entry->len==9 alone to distinguish (9 bytes = svc_id+node_id, data always longer).
2026-08-11 12:21:10 +03:00
evgeny b2e4aadbf0 e2e: AES-128-CCM encryption via BGP router (ROUTER_FLAG_ENCRYPTED), ECDH key per peer-pair from NODEINFO pubkeys, cache 8 entries
- etcp_router.h/c: new flag ROUTER_FLAG_ENCRYPTED (0x04), etcp_route_send_encrypted(), inflight/retrans/send_q support
- utun_instance.h: e2e_ctx_cache[8] — per-peer sc_context_t cache with derived session key
- topo_node.c: topo_node_sign_self() — single point for Ed25519 self-signing of NODEINFO
- topo_node.c: fix stale x25519_self_sig after address update (topo_node_sign_self call)
- nat_detection.c: fix stale x25519_self_sig after in-place socket meta modification
- test_etcp_router_unit: 5 E2E tests (basic round-trip, no NODEINFO drop, tampered, cache reuse, ENCRYPTED+SIGNED combined)
- builds clean, 70/70 tests pass
2026-08-09 08:19:20 +03:00
evgeny 38c9f229bc stcp: ed25519 pubkey exchange in handshake, fix instance destroy segfault, fix chat_tcp_test
- u_async.c: process immediate_queue BEFORE epoll_wait in uasync_poll
- stcp_link.c: defensive NULL etcp_conn/etcp_link in stcp_link_close
- utun_instance.c: move stcp_server_list_destroy_all to Phase J (before ETCP),
  add deferred drain in both Phase J and L
- stcp.h/c: STCP_HS_ENC_CLIENT 6→38, STCP_HS_ENC_SERVER 7→39,
  exchange ed25519_pubkey in both handshake directions
- etcp_connections.c: copy peer_ed25519 from STCP to ETCP_CONN in
  etcp_link_enter_ready_tcp; guard link->etcp in tcp_link_close_cb
- chat_sync.c: fix lk->conn NULL dereference for TCP links
- test_stcp.c: update stcp_server_create/stcp_client_connect callsites
- tools/chat_tcp_test: change transport udp→tcp
2026-08-07 00:48:27 +03:00
evgeny d773f1724a refactor: STCP connections as ETCP links (is_tcp)
- Add is_tcp, tcp_link, tcp_reconnect_* fields to ETCP_LINK
- Remove transport_link from ETCP_CONN, add tcp_link_count
- stcp_link: replace embedded etcp_conn with pointer; add etcp_link backpointer
- stcp_link: link_rx_cb routes ETCP data to etcp_conn_input
- stcp_link: call on_close_cb in close
- New TCP functions: etcp_link_enter_ready_tcp, etcp_tcp_link_start_connect,
  etcp_tcp_link_start_reconnect, tcp_link_close_cb, etcp_tcp_send
- etcp_link_new: handle NULL conn for TCP links; skip enter_init for TCP
- etcp_link_close: close stcp_link for TCP
- etcp_encrypt_send: redirect TCP to etcp_tcp_send (skips AES-CCM)
- tcp_server_on_link: create ETCP_LINK(is_tcp=1) + enter_ready_tcp
- etcp_connect: TCP addrs -> etcp_tcp_link_start_connect
- Remove ctx->tcp_link/tcp_ready
- test_stcp_traffic: find stcp_link via links->is_tcp instead of transport_link
- 66/68 tests pass; 2 STCP tests expected to fail (integration not complete)
2026-08-06 22:14:32 +03:00
evgeny d310bccf08 fix: memory pool alloc mismatch in tests, db_sync SEND_DATA buffer overflow
- test_etcp_router_reconnect, test_etcp_connect, test_stcp_traffic:
  replace u_calloc with memory_pool_alloc for TOPO_SOCKMETA4/TOPO_ADDR4
- db_sync.c: fix req[14] -> req[15] — SEND_DATA request header is 15 bytes
  (type:1 + from:4 + count:2 + vp:4 + want_from:4)
2026-07-31 21:44:44 +03:00
Evgeny 852ce401e7 Добавлен group_id в etcp_router и topo_group, удалён msg_transport
- etcp_router: SVC_ROUTE_HDR (+8B), ETCP_ROUTER_CONN/TRANSIT_QUEUE хеши расширены под group_id
- Все caller'ы (proxy, routing, conn_mgr, nat, chatgui, tests) обновлены
- topo_group: глобальный реестр TOPO_NODE по node_id, ref_count=0 при создании
- BGP-обмен активируется для всех групп (а не только дефолтной)
- Протокол: group_id добавлен в WITHDRAW/TABLE_REQ/TABLE_COMPLETE/ERR_GROUP_MISMATCH
- Per-connection коллбэки заменены на instance-level conn_status
- msg_transport полностью удалён из проекта
57/57 тестов
2026-07-22 15:51:03 +03:00
Evgeny fcf3fb7f51 refactor: move transport (etcp/stcp/normalizer) to transport_layer/, routing (bgp/pings/topo*) to routing_layer/ 2026-07-21 20:25:41 +03:00
Evgeny 3f874a80ab TOPO_GROUP: embed struct ll_entry as first member (direct queue embedding, hash index) 2026-07-09 19:50:17 +03:00
Evgeny 28e5950d29 TOPO_BGP -> TOPO_GROUP: multi-group support
- Renamed TOPO_BGP to TOPO_GROUP with topo_group_id (64bit)
- Added TOPO_GROUPS container (ll_queue group_list) in UTUN_INSTANCE
- Memory pools moved from TOPO_GROUP to TOPO_GROUPS (instance-level)
- Default group: TOPO_GROUP_UTUN = 0x8000000000000000
- topo_groups_get_default() searches by group_id, not just first entry
- All topo_bgp_* renamed to topo_group_*; function signatures updated
- Files: topo_bgp.h/c -> topo_group.h/c
2026-07-09 18:08:49 +03:00
Evgeny 3c3d9a65d9 refactor: переименование route_node/route_bgp в topo_node/topo_bgp
Файлы: route_node.* route_node_lmdb.* route_bgp.* -> topo_node.* topo_node_lmdb.* topo_bgp.*

Структуры:
- ROUTE_BGP -> TOPO_BGP, ROUTE_BGP_CONN_ITEM -> TOPO_BGP_CONN_ITEM
- NODEINFO -> TOPO_NODE, NODEINFO_Q -> TOPO_NODEQ, NODEINFO_PATH -> TOPO_NODEPATH
- NODEINFO_ROUTES -> TOPO_NODESUBNETS, NODE_CONNECTIVITY -> TOPO_CONNECTIVITY
- wire: NODEINFO_MSG -> TOPOMSG_NODE, NODEINFO_IPV4_* -> TOPOMSG_*4 и т.д.
- memory: NI_IPV4_* -> TOPO_*4, NI_IPV6_* -> TOPO_*6
- ROUTE_SUBCMD_* -> TOPO_SUBCMD_*, ETCP_ID_ROUTE_ENTRY -> ETCP_ID_TOPO_ENTRY

Функции:
- route_bgp_* -> topo_bgp_*
- nodeinfo_* -> topo_node_*
- route_node_* -> topo_node_*
- route_node_lmdb_* -> topo_node_lmdb_*

Смысл: отделение механизма формирования топологии узлов от IP-роутинга.
2026-07-09 17:01:56 +03:00
Evgeny c7ac42eb5f route_node: refactor NODEINFO — separate protocol/memory structs, linked lists with memory_pool
- NODEINFO_MSG (protocol, packed): added flags byte + group_id field
- NODEINFO (memory): ref_count, linked list heads instead of inline arrays
- NODEINFO_ROUTES: separate struct with linked list subnet heads
- NODEINFO_Q: node*, routes*, tranzit_data*, hop_list* as separate mallocs
- 6 memory_pools in ROUTE_BGP for NI_* linked list items
- ni_list_count() universal counter via _ni_head cast
- nodeinfo_serialize/deserialize for protocol <-> memory conversion
- NODEINFO_FLAG_SEND_SUBNETS controls subnet data in protocol
- group_id defaults: utun=NODEINFO_GROUP_UTUN(1) with subnets

Fixes:
- deserialization: data+2 instead of data+sizeof(BGP_NODEINFO_PACKET)
- memset: start after ll_entry to preserve ll.size
- hop_src: subtract hop_count*8 to point at correct dynamic offset
- same-ver branch: remove_path(conn) instead of remove_path_by_hop(peer)
- double-free in route_bgp_remove_conn/process_withdraw
- conn_mgr_add_alien_node: rewritten for new structures
- all test files updated for new API
2026-07-09 16:20:48 +03:00
Evgeny 0b5a710fe6 etcp_connect + BGP_READY callbacks + router reconnect test
- etcp_connect.c: async connect API with EARLY/LATE/BGP_READY callbacks
- etcp_connect.h: declaration (via etcp_api.h)
- connect_deliver: fix to preserve unfired callback nodes
- etcp_set_routing_exchange_state: fires bgp_ready_cbk on TABLE_COMPLETE
- route_bgp: TABLE_COMPLETE handler + route_bgp_send_table_complete
- etcp.c: connections_count tracking, cleanup on close
- test_etcp_router_reconnect: a-b-c topology with b restart (etcp_connect)
- test_etcp_connect: unit tests for async connect API
- etcp_router: inflight/retrans/close improvements
- Various proxy/NAT/router cleanups
2026-07-04 18:51:01 +03:00
Evgeny 42d96f118d fix: swap queue_entry_free/queue_dgram_free order in 27 places — prevents UAF when dgram_free accesses freed entry
uasync: replace raw socket_node* handle with packed index — fixes UAF after socket_array realloc
tcp_io: defer u_free in tcp_conn_destroy via uasync_call_soon — prevents callback-chain UAF
tcp_io: guard read_cb/write_cb with NULL queue checks after deferred destroy
tcp_io: save read_queue to local before queue_data_put + NULL guard after
route_connectivity: linked-list probe_ctx — cancel all parallel probes before nq free
2026-07-02 17:14:38 +03:00
Evgeny f5b1c3e789 etcp_router: fix OOB drops after restart — add peer_sync_done to sync rx_seq with first incoming seq 2026-07-02 11:12:14 +03:00
Evgeny 7329fa65f1 etcp_router: add congestion counters, fix tx_seq rollback on etcp_send failure, add router metrics to etcpmon GUI
- etcp_router.h: add 9 per-conn counters (pkts_sent/err/rcvd, ack_sent/recv/stale, dup/oob/sign_fail)
- etcp_router.c: increment counters at 13 event points; fix tx_seq rollback on etcp_send failure (was causing phantom inflight=256 deadlock); add DEBUG_ERROR on all 5 ignored etcp_send returns; fix router_ack_do_send to update timestamps only on success
- etcpmon_protocol.h: add etcpmon_router_metrics (14 aggregated fields)
- control_server.c: iterate router_conns to fill aggregated router metrics
- etcpmon_gui: new 'Router Congestion' block with queue depths and all counters, Debug Levels shifted down, window 1300->1440
- test_etcp_router_unit: fix test_ack_timer for correct no-BGP-route behavior
2026-07-02 09:49:10 +03:00
Evgeny 6717c863e7 etcp_router: Ed25519 digital signature for service messages
- Add ed25519_public_key[32] to NODEINFO struct for pubkey distribution
- Add ed25519_public_key[32] to ROUTE_BGP (derived from X25519 privkey at init)
- Add ROUTER_FLAG_SIGNED (0x08) to SVC_ROUTE_HDR flags byte
- router_send_one_flags: when is_signed, sign [hdr][payload] and append 64-byte sig
- etcp_router_recv_cb: verify Ed25519 signature when ROUTER_FLAG_SIGNED set
- New API: etcp_router_conn_send_signed()
- Drop signed packets if sender node not in routing table or no Ed25519 key
- 5 unit tests in test_etcp_router_unit.c covering OK/tampered/unknown/no-key/short
2026-06-22 15:53:24 +03:00
Evgeny e765d62e43 etcp_route_send: add force parameter to bypass send_q limit for control messages
CLOSE/ERROR/CONNECT and routing messages use force=1 to never be
dropped by backpressure. Regular DATA uses force=0 (default).
2026-06-04 12:01:55 +03:00
Evgeny e0378ebf1d refactor: remove consumer_ack, isolate proxy from router internals
etcp_router:
  - removed consumer_ack field, consumer_ack function, consumer_ack logic
  - ACK always auto-sent via router_schedule_ack (10ms throttle)
  - conn_restart: local-only reset, no remote CLOSE, unified close notification (len=9)
  - START detection: only when rx_seq>0 (first conn doesn't trigger)
  - new API: etcp_router_on_send_ready / cancel_send_ready for send_q backpressure

tcp_proxy_server:
  - removed write_on_get_cb, queue_set_on_get, rconn->consumer_ack
  - removed etcp_router_conn_get, direct send_q access
  - uses etcp_router_on_send_ready for backpressure

tcp_proxy_client:
  - removed consumer_ack call
  - unified conn=NULL handler (len=9 = close/restart with node_id)

protocol: close/restart notification now len=9 (svc_id + node_id), no RESTART subcmd
2026-06-03 23:46:11 +03:00
Evgeny 0449a5fd9d test: 5 new router unit tests — RST, restart, sess_id
test_rst_flag: inject ROUTER_FLAG_RST → rconn destroyed, service notified
test_client_restart: pump random data → inject START → restart detected
test_sess_id_change: pump sess_id=0 → inject sess_id=1+START → detected
test_sess_id_wrap: pump sess_id=3, peer_sess_id=3 → inject sess_id=0+START → wrap detected
test_server_reinit: reinit router → new rconn start_sent=0, sess_id=0

inject() extended with uint8_t flags parameter.
pump_data helper for random data injection.
test_handler tracks g_notify_count for close/restart notifications.

fix: etcp_router_recv_cb — return after RST flag to avoid NULL rconn deref
2026-06-03 21:56:11 +03:00
Evgeny ac3fe79db8 etcp_router: consumer-driven ACK with interval throttle (10ms)
- consumer_ack flag: ACK sent only on consumption, not assembly
- etcp_router_consumer_ack() — called from tcp_proxy_client feed_from_transport
- last_ack_sent_tb: interval-based throttle — send immediately if >=10ms passed,
  otherwise timer for remaining time
- timer rules: NULL handle after cancel/fire, NULL check before start
- ROUTER_ACK_INTERVAL_TB 1000→100 (100ms→10ms)
- test_etcp_router_unit: updated test 14 for new immediate-send behavior
2026-06-03 11:50:50 +03:00
Evgeny 7f5325ac24 etcp_router: fix inflight underflow by splitting rx_acked into tx_acked + last_sent_ack_seq
rx_acked was used for two conflicting purposes:
1. remote ack of our sends (inflight = tx_seq - rx_acked)
2. our last sent ACK seq (dedup: rx_seq != rx_acked)

When the ack timer fired and set rx_acked = rx_seq, it overwrote
the inflight-tracking value. If rx_seq > tx_seq, the computation
tx_seq - rx_acked underflowed (e.g. 18 - 24 = 0xFFFFFFFA),
permanently blocking router_drain_send_q and causing send_q to
grow indefinitely.

Fix:
- Split rx_acked into tx_acked (remote ack, for inflight) and
  last_sent_ack_seq (our ACK, for dedup)
- Incoming ACK handler only advances tx_acked forward (stale guard)
- Use int32_t cast on all inflight comparisons to handle stale states
- Add etcp_router architecture diagram (doc/etcp_router_arch.md)
2026-06-02 22:38:20 +03:00
Evgeny a80ba30b06 test_etcp_router_unit: fix FAIL macro — bare return in non-void function 2026-06-01 10:39:33 +03:00
Evgeny 21cfe2d381 etcp_router: remove legacy, auto-conn for all services, send_q inflight control, remove tcp_proxy internal seq
- All SVC_ROUTE traffic now auto-creates ETCP_ROUTER_CONN, no legacy path
- etcp_route_send: auto-conn + seq + inflight check → send_q if full
- Inflight control via send_q: queue instead of drop, drain on ACK, retry timer 50ms
- router_send_one(): unified SVC_ROUTE wrapper with seq assignment
- router_deliver/try_assembly: pass real ETCP_CONN* to handlers (NAT needs it)
- tcp_proxy: remove internal 16-bit seq (send_seq/recv_last_seq/recv_init), shrink HDR 8→6
- remote_proxy: same seq removal, HDR/constants adjusted
- conn init/close/destroy: manage send_q, send_resume_timer lifecycle
- test_etcp_router_unit: −legacy, +send_q inflight test (18 tests)
2026-05-31 22:55:23 +03:00
Evgeny ac3cdd7739 etcp_router: simplified TCP — reorder recv_q, dedup, ACK timers, inflight control, per-(remote,svc) conn state
- New SVC_ROUTE_HDR (packed struct, 22 bytes): cmd+dst+src+seq+svc_id
- ETCP_ROUTER_CONN: state per (remote_node_id, svc_id), hash-indexed in router_conns
- Reorder via recv_q (hash by seq), dedup with 32-bit circular compare
- Periodic ACK (100ms), idle ACK (500ms), inflight limit via tx_seq - rx_acked
- Legacy mode: no conn → direct delivery without reorder
- etcp_router_conn_get/send/close API for seq-managed connections
- Unit test test_etcp_router_unit: 18 tests, 3ms, no ETCP/sockets/BGP
2026-05-31 21:59:46 +03:00