Commit Graph
1659 Commits
Author SHA1 Message Date
evgeny fb3e34080d media_delivery: implement full media delivery subsystem (stages 1-8)
- topo_group: add BGP node event callbacks (NEW/UPDATE/REMOVE)
- member_sync: add multi-subscriber node_props_changed callback (adm_tags)
- media_delivery: SQLite tables (block_availability + super_sync), full protocol handler
- media_delivery_proto.h: packed protocol structures (15 subcommands)
- media_download: download management, chunk streaming, block assembly, HAVE_BLOCK with retry
- Makefile.am: add media_download.c to builds
2026-07-29 17:14:32 +03:00
evgeny 7d9cbad8c5 media_delivery: add architecture description and implementation plan 2026-07-29 16:53:30 +03:00
evgeny 65dbfa6813 feat: media_async + media_index modules, media_files table, tests
- media_async: thread-per-task async engine with primitives (sha256, sign, copy, uuid, file_size)
- media_index: media_files DB table with register_async for indexing chunks
- chat_msg: async media submission via register_async, simplified chat_msg_submit fields
- voice_recorder/attachment_sender: write single file (no chunk splitting on caller side)
- inputbar/messagelist: updated to new simplified submit interface
- test_media_async: 4-thread stress test (200K ops / 2 sec, 0 errors)
- test_media_index: 9 cases covering commit edge cases + async flow
- build: Makefile.am, CMakeLists.txt, utun_sources.cmake updated
2026-07-29 14:15:10 +03:00
evgeny ce5a84da24 fix: uasync epoll event ordering + test_uasync_socket_race fix 2026-07-29 11:10:55 +03:00
evgeny 3c75c8f687 icons: replace emoji text with PNG resources + add media_dir debug logs 2026-07-29 01:27:23 +03:00
evgeny 39d6f19d97 fix: chat_msg media_dir path — strip filename from g_cc.db_path
g_cc.db_path is '/data/.../files/chats.db' (file path), but media blocks
are written to '/data/.../files/media/<channel>/'. chat_msg was concatenating
'chats.db/media/...' — now strips the database filename, using only the
directory portion for the media path.
2026-07-29 01:04:27 +03:00
evgeny e751e6ee69 android: voice messages + attachments architecture plan, build fixes 2026-07-28 23:07:23 +03:00
evgeny f8d7c21a79 Android: unify DB access via JNI/JSON, fix instance lifecycle, add restart + health check
- ChatRepository: remove SQLiteDatabase (READONLY 2nd handle), use JNI→shared g_cc.db (FULLMUTEX, like desktop)
- instance_lite: simplify start/stop lifecycle — start no longer waits for init, stop waits for thread exit with timeout, add internal restart (no thread kill)
- Restart trampoline: reinit UDP log target on config change
- Health check: ping/uasync_post round-trip every 3s, red dot on stall
- Settings: Restart instance button
- Fix: g_inst check instead of g_running for is_running()
- Fix: SharedFlow replay=1 + proactive DB_READY check in ViewModel
2026-07-28 21:13:11 +03:00
evgeny 03b34e9580 android: fix QR join flow + cleanup log levels + remove log artifacts
- MainActivity: call NativeLib.joinChannel directly (bypass ViewModel null repo)
- LogManager: add Log.d for Kotlin logs visibility in logcat
- ChatViewModel/NativeLib: add diagnostic logs for join path
- chat_sync: JNI-safe logs via __android_log_print, add error on !g_cs
- etcp_debug: DUMP INFO→DEBUG (remove 99.8% packet spam)
- secure_channel/etcp_connections/topo_node: keepalive PING/PONG spam INFO→DEBUG
- .gitignore: logreceiver output + build artifacts
- Remove tracked log files and build dir from repo
2026-07-28 18:11:11 +03:00
evgeny 0216b0bf50 android: utun shutdown diagnostics + non-blocking stop + logreceiver doc
- instance_lite: cleanup moved to uasync thread (self-detach), events on start/stop
- udp_log: cancel_timer before uasync_destroy, flush on stop, stopping flag
- jni_bridge: fixed UDP stop order (cancel_timer → destroy → flush)
- instance_lite: u_report_unfreed_blocks at all shutdown paths
- chat_event: CHAT_EVT_SERVICE_STARTED/STOPPED for GUI notifications
- AGENTS: logreceiver section
2026-07-28 17:06:09 +03:00
evgeny 064bed06f2 android: fix key regeneration, Clear DB, uasync eventfd, udp log from config
- jni_on_event: fix JNI_EDETACHED — AttachCurrentThread/DetachCurrentThread
- u_async.c: pipe→eventfd for reliable cross-thread wakeup
  add process_posted_tasks in poll() path, fix EINTR (saved_errno)
- bridge_debug_hook: keep udp_log_write, remove mutex from udp_log
- memory_pool.c: alloc/free logs INFO→DEBUG (reduce spam)
- instance_lite.c: chat_event_set_handler before ensure_keys,
  KEYS_GENERATED events, fchmod 0600 on keys.conf,
  UDP log from config [log_udp] section, regenerate_trampoline
- config_parser: parse [log_udp] ip/port
- NativeLib.kt: remove callbacksSet, simplify init/stop/destroy
- ChatViewModel.kt: unified event callback in init, pubKey StateFlow,
  requestRegenerateKeys, clearState fix
- SettingsScreen.kt: pubKey.collectAsState(), Clear DB via restart()
- HeadlessService.kt: remove setUdpLogTarget (now from config)
- android_udp_log.c: simplified — no mutex, buffer+send+timer in one thread
- utun_instance.c: NULL-check tcp_connections before destroy
- chat_event.h: +CHAT_EVT_KEYS_GENERATED 11, -NODE_CHANGED
- logreceiver: UDP log receiver tool
2026-07-28 16:29:33 +03:00
evgeny d0cca27edd invite: add proto field (UDP/TCP/BOTH) to invite link binary format
Format per v4 addr: family(1) + socketId(1) + proto(1) + addr(4) + port(2) = 9 bytes (was 8)
Proto values: 0x01=UDP, 0x02=TCP, 0x03=BOTH
Affects: invite_link_c.h/c (Android), invite_link.h/cpp (desktop), chat_sync.c, conn_mgr.c, joindialog.cpp, utun_node.cpp
2026-07-28 00:34:07 +03:00
Evgeny d1f6f6fca9 fix WELCOME deserialization: skip atl+adm_tags_sig fields before address data 2026-07-27 23:48:23 +03:00
Evgeny 2d53b3120f ffmpeg: add build script and CMake integration 2026-07-27 17:59:07 +03:00
Evgeny d4a396f888 media: Ed25519-подписи блоков, local_attrs, MediaBlockReader, файловые вложения 2026-07-27 17:09:31 +03:00
Evgeny ed6132fcda перенос chat-логики из chatgui/transport в src/chat/
- Создан src/chat/ с chat_event.h/c — система нотификаций
- Перенесены chat_core, chat_msg, chat_channel, chat_profile, chat_status
- Перенесены chat_sync, member_sync, merkle_sync, db_sync
- gui_bridge_post() заменён на chat_event_post() с возможностью регистрации обработчика
- chat_event_set_handler() — chatgui регистрирует мост в gui_bridge
- Без обработчика: события логируются через DEBUG_DEBUG + log_dump
- chat интегрирован в utun_instance.c (при db_path в конфиге)
- Обновлены include-пути, Makefile.am, CMakeLists.txt
- Из chatgui удалены старые transport/ файлы, обновлён utun_node.cpp
2026-07-27 12:58:33 +03:00
Evgeny 4fde0b0325 feat: integrate Opus codec into lib/libopus + voice messages in chatgui
- Add lib/libopus/ (opus v1.6.1 embedded source: CELT 18 .c, SILK 77 .c, SILK float 28 .c, OPUS core 14 .c)
- Add lib/opus_codec.h/.c (wrapper: encoder, decoder, raw PCM int16)
- Add tests/test_opus_codec.c (7 tests: silence, sine, compression, PLC, null, defaults, multi-frame)
- chatgui: AudioRecorder (miniaudio capture, 48kHz mono)
- chatgui: VoiceEncoder (PCM→Opus file with custom container)
- chatgui: VoicePlayer (Opus→PCM playback via SoundManager)
- chatgui: PTT button in InputBar, voice message send/receive in MessageList
- chatgui: AudioDeviceSettingsPage (device selection, persisted in ui_state)
- chatgui: MessageDelegate voice message rendering (play button + duration)
- Update autotools: configure.ac, Makefile.am, src/Makefile.am, tests/Makefile.am
- Update CMake: chatgui CMakeLists.txt, libutun CMakeLists.txt (link libopus_internal.a)
2026-07-26 20:34:38 +03:00
Evgeny a1ecab997e merkle_sync: fix 4 protocol bugs + add test suite (Stages 1-4)
- g_merkle singleton → per-instance msync (utun_instance.h)
- level_prefix: off-by-one shift (63→64)
- _handle_hashes: differs |= for shared buckets with different hashes
- _handle_batch: sub_pl double-level-offset (next_lvl→lvl)
- pending[] array replaces pending_requests counter
- member_sync: mask_shift 63→64 alignment
- test_merkle_sync.c: prefix arithmetic, tree building, 2-instance integration, randomized 50-iter sync
2026-07-26 17:42:27 +03:00
Evgeny 49da899964 chatgui: channel sorting by name or last message time with preview
- Add last_msg_at column to channels table
- Update last_msg_at on every message insert (send/receive)
- Parameterize getChannels() with ChannelSortMode enum (Alphabetical/Recent)
- Add getChannelLastMessage() for preview text in channel list
- Add sorting combo to Profile settings page (stored in ui_state)
- Reload channels on message receive when sort is recent
2026-07-26 11:08:59 +03:00
Evgeny 22db84816e chatgui: add local_nick TEXT to peers table DDL, remove ALTER TABLE hacks 2026-07-26 09:45:00 +03:00
Evgeny 11ab897fe5 chatgui: fix RTT display 10x overstatement (DB stores 0.1ms, displayed as ms); remove unused rtt_avg_100 2026-07-26 08:57:02 +03:00
Evgeny ecb03ca290 conn_mgr: detect duplicate conn in handshake; etcp: loud collision errors in link/peer_id 2026-07-25 19:43:26 +03:00
Evgeny 7a9e3bea5f db_sync: mc>pc guard with last_peer_count — detect stalled sync when pc doesn't change 2026-07-25 16:56:30 +03:00
Evgeny 3490789249 db_sync: skip SYNC_DONE in count>0 handler when received==0 — breaks bad_sig recv→send loop on client side 2026-07-25 16:49:46 +03:00
Evgeny c9436219a0 db_sync: fix synced_pos overwrite in SYNC_DONE — use max to preserve total_processed advance 2026-07-25 16:28:09 +03:00
Evgeny 34404403ef db_sync: verify sig before send + fix total_processed double-count 2026-07-25 16:10:20 +03:00
Evgeny e9fce43be5 db_sync: fix bad_sig infinite loop — delete+rechain, advance synced_pos past dup/bad_sig, accept on covered tail 2026-07-25 15:58:18 +03:00
Evgeny 27906e9f91 db_sync: односторонний sync, sparse-таблица, vp в SEND_DATA, PUSH-релей 2026-07-25 15:00:21 +03:00
Evgeny e071a9e2f8 db_sync: fix test sync protocol isolation (add auto_sync flag, destroy order, test refactor)
- utun_instance: fix destroy order (topo_groups before conn_mgr to avoid UAF in topo_recovery)
- topo_recovery: add NULL guard for instance/ua in try_next
- db_sync_instance_add: add auto_sync flag (1=send INIT_SYNC, 0=quiet)
  replaces separate _quiet variant, cleaner API
- db_sync_peer_set_state, db_sync_reinitiate: test-only peer control utilities
- test_db_sync: full rewrite — stable, isolated sync protocol testing
  all 3 branches (peer_empty, hash_MATCH, divergence) tested without PUSH contamination
  Phase 6 extended: triple merge + tail PUSH + mid PUSH cascade on star topology
  runs in 1.4s, consistently passes 10/10
- db_sync_doc: updated protocol architecture description
- chatgui: update db_sync_instance_add call with auto_sync=1
2026-07-24 21:27:23 +03:00
Evgeny ad56767646 db_sync: notify all peers when cascade_from changes chain (lock-free sync)
После cascade_from в db_handle_send_data оповещаем ВСЕХ пиров
с synced_pos >= fix_from о том что цепочка изменилась — откатываем
их synced_pos. Это устраняет неэффективность конкурентных sync:
раньше только пир-отправитель обновлялся, остальные видели устаревшие
хеши и обнаруживали ложное расхождение на следующем цикле.
2026-07-24 17:26:09 +03:00
Evgeny c46cb9a979 db_sync: fix si_send_data_batch column bug (author→node_id), add cascade_from to PUSH, add divergence+cascade test phases
- si_send_data_batch: неправильный SQL SELECT с несуществующей колонкой author
  вместо node_id → ломал всю передачу данных в REFINE/SEND_DATA
- Добавлена проверка si_prep и возврат int с кодом ошибки
- Все вызовы si_send_data_batch проверяют возврат, при ошибке сброс sync_state
- db_handle_push: добавлен db_cascade_from после вставки (раньше chain_hash
  не пересчитывался после PUSH)
- db_handle_refine: DEBUG_WARN при scnt==0 (fm за пределами диапазона)
- Добавлена публичная db_sync_chain_verify() для верификации цепочек
- test_db_sync: фаза 4 (divergence через разныe hashes + REFINE/SEND_DATA),
  фаза 5 (PUSH без хвоста + cascade verification)
- Убран мёртвый код db_sync_on_new_conn
- chatgui: адаптивная раскраска пузырей исходящих сообщений
2026-07-24 16:55:08 +03:00
Evgeny a7ec45c4af db_sync: replace all sync logs with unified narrative format sync [TBL:NODE]
- Unified format: 'sync [SHORT_TBL:PEER_HI] <-/->  action: key_data'
- Each log line explains what the algorithm is doing and why
- Added batch progress tracking in dump-all and tail SEND_DATA loops
- Track dropped records (duplicates, bad_sigs, parse_fails) in SEND_DATA recv
- Show synced_pos transitions (old->new) in SEND_DATA and SYNC_DONE
- Show all launched syncs in PEER CHECK log
- Convert all silent returns to ERROR logs with [TBL:NODE] context
- Convert old WARN/ERROR formats to new sync [TBL:NODE] format
- Add sync_state reset details on ERROR and timeout paths
2026-07-24 13:23:17 +03:00
Evgeny 23db012064 conn_mgr: fix duplicate connection and double-release bugs
- conn_mgr_connect_node: check instance->connections globally to reuse
  existing ETCP_CONN across groups instead of creating a new one (DUP addr)
- cm_start_phase_direct: close dead connection when etcp_link_new fails
  to prevent shadowing live connection in instance->connections hash
- cm_cleanup_db_node: remove redundant topo_node_registry_release
  (already done inside topo_node_free_lists)
2026-07-23 20:36:55 +03:00
Evgeny 58ad643516 Add debug logs to Ed25519 self-sig flow: computation, verification, skip due to no key 2026-07-23 20:00:11 +03:00
Evgeny 85269d1da3 Ed25519 self-signature in NODEINFO + fix sync sequence and zero-key bug
Add x25519_self_sig[64] to TOPOMSG_NODE/TOPO_NODE:
- topo_node.h: add field to both structs (wire format + memory)
- topo_node.c: compute Ed25519_sign(ed25519_privkey, x25519_pubkey) for own node
- topo_node.c: serialize/deserialize x25519_self_sig
- topo_group.c: verify Ed25519 self-sig on NODEINFO receive, reject forgery

Fix Ed25519 pubkey zero-key bug:
- topo_node_sqlite.c: get_ed25519_pubkey rejects all-zero key (MUST be non-zero)

Delay db_sync until Ed25519 key is available:
- db_sync.c: db_sync_on_conn_up checks db_get_ed25519_pubkey before starting
- db_sync.c: db_sync_peer_check_cb also checks Ed25519 key availability
- This prevents sync starting before NODEINFO with verified Ed25519 arrives
2026-07-23 19:55:15 +03:00
Evgeny 3890d2038d db_sync: add missing ERROR/WARN logs to all error branches
- db_record_insert: error logs for si_prep failures (SELECT, INSERT, cascade SELECT/UPDATE)
- si_parse_record: WARN logs for truncated header, data/sig overrun
- db_chain_hash_at: ERROR logs for missing row, corrupted blob
- db_prev_chain_hash: ERROR log for si_prep failure
- db_cascade_from: ERROR logs for si_prep failures
2026-07-23 18:16:53 +03:00
Evgeny acd4a189ae db_sync: add author_sig VERIFY FAIL diagnostics + SYNC_DONE retry limit
- db_verify_author_sig: show ed_pubkey and sig first bytes on failure
- SI_PEER: add sync_retry_count field
- db_handle_sync_done: limit re-initiate to 3 retries before giving up
- db_sync_initiate_sync / db_sync_on_conn_up: reset retry_count
2026-07-23 18:13:14 +03:00
Evgeny b48d3b4a77 db_sync: add diagnostic logs for message sync debugging during join
- db_handle_init_sync: TRACE→INFO with peer_cnt/my_cnt/tp_offset
- db_handle_send_data: detailed batch info (range, remaining count)
- db_sync_initiate_sync: add table name to log
- db_sync_on_conn_up: add first table name to summary log
2026-07-23 16:47:22 +03:00
Evgeny 3e95cf5ca6 fix: dialog hangs on join success + remove per-packet spam logs
- chat_sync.c: move pending_invite clear from INFO_RESP to WELCOME,
  post GUI_EVT_CONNECT_RESULT on invite success in cs_handle_welcome
- etcp_connections.c: NORM_DECRYPT_TRY/CRYPTO_DECRYPT_OK DEBUG_DEBUG->TRACE
- secure_channel.c: ENCRYPT/DECRYPT DEBUG_DEBUG->TRACE
2026-07-23 16:36:40 +03:00
Evgeny 4f831703f0 fix: etcp_conn_set_peer_node_id переиндексация при state=0 + conn_mgr ставит peer_id сразу 2026-07-23 16:19:03 +03:00
Evgeny 511a2dcd06 fix: авто-подключение Phase 2/3 — запрос всех нод из node_addresses, без привязки к peers_<ch> 2026-07-23 14:41:22 +03:00
Evgeny 01a0a568a6 debug: crypto_ctx session_key логгирование — CRYPTO_CONN_READY, CRYPTO_ON_UP, CRYPTO_DECRYPT_OK, DECRYPT_FAIL+seskey 2026-07-23 14:04:44 +03:00
Evgeny be258eea4d fix: remove stale peers_* CREATE TABLE from chat_core, drop ORDER BY node_RTT
chat_core_ensure_channel_ready() had a duplicate CREATE TABLE peers_*
with OLD schema (no node_RTT, node_type, connected). Since it was called
from 4 places WITHOUT topo_node_sqlite_channel_put() first, the old
schema was created first, and IF NOT EXISTS in the new DDL was a no-op.

Removed the entire old block. Only topo_node_sqlite_channel_put() now
creates peers_* tables with the full schema.

Also removed ORDER BY p.node_RTT from get_public_peers/get_local_peers —
ordering not needed for sequential connect attempts.
2026-07-23 00:24:16 +03:00
Evgeny 8facce46ac feat: auto-connect Phase 3 (local/netif addresses) + restart fix
Phase 1: connected peers (connected=1)
Phase 2: direct/EIM (addr_type=1,2), exclude self
Phase 3: local/strict NAT (addr_type=0,3), exclude self
Timeout: 5000→2000ms per attempt
Restart: DOWN triggers reconnect when phase=DONE & active=0

Debug logs: init, Phase1 per-peer, UP phase info, restart branches
2026-07-22 23:55:44 +03:00
Evgeny 7dee05c628 fix: node_registry crash — TOPO_NODE_REG_ENTRY node/lldata overlap
struct TOPO_NODE_REG_ENTRY имел node и ll.data[0] (гибкий массив) на одном смещении.
memcpy(re->ll.data, &node_id, 8) затирало re->node значением node_id,
из-за чего node_registry_find возвращал не указатель а битый node_id → SIGSEGV.

Убрал struct TOPO_NODE_REG_ENTRY, перешёл на явные смещения:
data[0..7]=ключ, data[8..15]=указатель, queue_entry_new(16).
2026-07-22 23:05:04 +03:00
Evgeny 6edd40d141 topo_group_connect: авто-подключение CHAT-групп + conn_mgr_connect_from_invite + замена chat_conn_mgr в chatgui
Добавлено:
- conn_mgr_connect_from_invite() — подключение к новому узлу по invite-данным
  (INIT → INVITE_INFO_REQ/RESP → проверка членства в группе → сохранение в БД)
- topo_group_connect.c — авто-подключение при старте CHAT-группы:
  Phase 1: одновременный connect всех connected=1 пиров, таймаут, отмена незавершённых
  Phase 2: последовательный перебор публичных адресов до первого успеха
- Счётчик active_conn_count с проверкой nodeinfo-путей (UP/DOWN)
- restart() при phase=DONE и count==0

БД:
- connected INTEGER NOT NULL DEFAULT 0 в peers_<ch> + ALTER TABLE для существующих
- topo_node_sqlite_set_connected / get_connected_peers / get_public_peers

Chatgui:
- Включён topo_group (убран utun_instance_set_topo_group_enabled(0))
- Создание TOPO_GROUP_TYPE_CHAT при channel_ready и при старте
- Invite connect через conn_mgr_connect_from_invite
- Удалён chat_conn_mgr (из сборки) + весь старый auto-connect код
2026-07-22 22:48:12 +03:00
Evgeny 4215349349 topo_recovery: каскадное восстановление узлов с группировкой по next_hop
При разрыве ETCP-соединения модуль собирает каскадно отвалившиеся узлы:
- add_node из hop_list вычисляет next_hop относительно failed_peer,
  группирует узлы в отдельные recovery-контексты (direct/indirect списки)
- start запускает асинхронный перебор для каждой группы параллельно
- is_next_hop-узел пробуется первым с приоритетом по мин. RTT
- таймаут 2с на попытку, conn_mgr_cancel_callback при отмене
- отмена: topo_group_new_conn (on_up) находит узел в любом ctx -> ctx целиком
- завершение: оба списка пусты -> освобождение ctx
2026-07-22 19:01:55 +03:00
Evgeny f6b37a857b conn_mgr: bind to topo_group, DB node loading, configurable DIRECT timeout, tests
- conn_mgr_init(TOPO_GROUP* group) — работает внутри заданной группы
- conn_mgr_connect_node: загрузка узла из SQLite если нет в BGP-таблице
- topo_node_sqlite_node_load(): загрузка TOPO_NODE (ref=0) из БД
- cm_start_phase_direct: db_loaded-ветка — перебор всех адресов без NAT-фильтра
- cm_cleanup_db_node: удаление TOPO_NODEQ из группы при неудаче, только если нет путей
- cm_start_phase_indirect: исправлено зависание при best_candidates==0 (cm_deliver_result)
- conn_mgr_set_direct_timeout_ms(): конфигурируемый таймаут фазы DIRECT
- conn_mgr_destroy до etcp_router_destroy (исправлен порядок)
- test_conn_mgr: +2 сценария (DB-успех, DB-недоступен), проверка topo_group
2026-07-22 17:33:58 +03:00
Evgeny 852ce401e7 Добавлен group_id в etcp_router и topo_group, удалён msg_transport
- etcp_router: SVC_ROUTE_HDR (+8B), ETCP_ROUTER_CONN/TRANSIT_QUEUE хеши расширены под group_id
- Все caller'ы (proxy, routing, conn_mgr, nat, chatgui, tests) обновлены
- topo_group: глобальный реестр TOPO_NODE по node_id, ref_count=0 при создании
- BGP-обмен активируется для всех групп (а не только дефолтной)
- Протокол: group_id добавлен в WITHDRAW/TABLE_REQ/TABLE_COMPLETE/ERR_GROUP_MISMATCH
- Per-connection коллбэки заменены на instance-level conn_status
- msg_transport полностью удалён из проекта
57/57 тестов
2026-07-22 15:51:03 +03:00
Evgeny e2d9879636 refactor: remove direct nodes table writes from chat_core/conn_mgr, use topo_node_sqlite API
- chat_core_init: replace INSERT INTO nodes with topo_node_sqlite_node_update_verified, remove redundant topo_node_sqlite_init call
- chat_core_update_my_name: replace UPDATE nodes with topo_node_sqlite_node_update_verified
- chat_conn_mgr_connect_from_invite: simplify invite node insert (x25519 only, no ed25519)
- topo_node_sqlite: add detailed column comments to peers table DDL (both server and chatgui)
- chat_core: add detailed column comments to peers table CREATE TABLE
2026-07-22 13:38:17 +03:00