proxy_client: give subscribe requests a deadline

A subscribe is a short request: the server answers immediately and the
device then relies on push notifications. Unlike a listen, nothing keeps
it under observation. The keepalive is only armed once the server has
answered, and no request in this client ever sets a timeout, so a
subscribe stalled in name resolution, connection or TLS handshake never
completes and never fails either.

The listener then stays registered locally while the server never learns
about it: the device stops being notified, without anything reporting an
error. This is most likely right after a wake-up, when the network stack
is still settling.

Arm a deadline on subscribe requests and fail the operation when it
expires, so restartListeners() sends the subscription again. Streaming
listens are left untouched: they are long lived by design.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

doc/dhtnode.1

@@ -1,102 +1,193 @@
-.TH DHTNODE 1 2019-06-08
+.TH DHTNODE 1 2026-04-18
 .SH NAME
 .B dhtnode
 - a simple OpenDHT command line node runner.
 .SH SYNOPSIS
 .B dhtnode
 [\fB\-h\fR]
+[\fB\-V\fR]
 [\fB\-v\fR [\fB\-l\fR \fIlogfile\fR] [\fB\-L\fR]]
-[\fB\-i\fR [\fB\-\-save\-identity\fR \fIfile\fR]]
+[\fB\-i\fR [\fB\-I\fR \fIfile\fR]]
+[\fB\-c\fR \fIfile\fR]
+[\fB\-k\fR \fIfile\fR]
+[\fB\-m\fR \fIpassword\fR]
 [\fB\-d\fR] [\fB\-s\fR]
+[\fB\-D\fR] [\fB\-U\fR] [\fB\-P\fR]
 [\fB\-n\fR \fInetwork_id\fR]
 [\fB\-p\fR \fIlocal_port\fR]
 [\fB\-b\fR \fIbootstrap_host\fR[:\fIport\fR]]
-[\fB\-\-certificate\fR \fIfile\fR]
-[\fB\-\-privkey\fR \fIfile\fR]
-[\fB\-\-privkey\-password\fR \fIpassword\fR]
-[\fB\-\-proxyserver\fR \fIport\fR]
-[\fB\-\-proxyclient\fR \fIserver\fR]
+[\fB\-f\fR \fIpath\fR]
+[\fB\-S\fR \fIport\fR]
+[\fB\-e\fR \fIport\fR]
+[\fB\-a\fR \fIaddress\fR]
+[\fB\-w\fR \fIfile\fR]
+[\fB\-K\fR \fIfile\fR]
+[\fB\-M\fR \fIpassword\fR]
+[\fB\-C\fR \fIserver\fR]
+[\fB\-y\fR \fIendpoint\fR]
+[\fB\-z\fR \fIkey\fR]
+[\fB\-u\fR \fIbundleid\fR]
 .SH DESCRIPTION
-Runs an OpenDHT node, with a CLI (default) or as a daemon (with \fB'-d'\fP or \fB'-s'\fP).
+Runs an OpenDHT node, with an interactive CLI (default) or as a daemon (with \fB'-d'\fP or \fB'-s'\fP).
 Commands available in the interactive shell are:
 .EX
     h, help    Print this help message.
-    q, quit    Quit the program.
+    x, quit    Quit the program.
     log        Start/stop printing DHT logs.
 
     Node information:
     ll         Print basic information and stats about the current node.
-    ls         Print basic information about current searches.
-    ld         Print basic information about currenty stored values on this node.
-    lr         Print the full current routing table of this node
+    ls [key]   Print basic information about current search(es).
+    ld [key]   Print basic information about currenty stored values on this
+               node (or for a specific key).
+    lr         Print the full current routing table of this node.
 
     Operations on the DHT:
-    b ip:port             Ping potential node at given IP address/port.
-    g [key]               Get values at [key].
-    l [key]               Listen for value changes at [key].
-    p [key] [str]         Put string value at [key].
-    s [key] [str]         Put string value at [key], signed with our generated
-                          private key.
-    e [key] [dest] [str]  Put string value at [key], encrypted for [dest] with
-                          its public key (if found).
+    b <ip:port>            Ping potential node at given IP address/port.
+    g <key>                Get values at <key>.
+    l <key>                Listen for value changes at <key>.
+    cl <key> <token>       Cancel listen for <token> and <key>.
+    p <key> <str>          Put string value at <key>.
+    pp <key> <str>         Put string value at <key> (persistent version).
+    cpp <key> <id>         Cancel persistent put operation for <key> and
+                           value <id>.
+    s <key> <str>          Put string value at <key>, signed with our
+                           generated private key.
+    e <key> <dest> <str>   Put string value at <key>, encrypted for <dest>
+                           with its public key (if found).
+    cc                     Trigger connectivity changed signal.
+
+    Operations with the proxy (if compiled with proxy server support):
+    pst [port] <pushServer>  Start the proxy interface on port.
+    psx [port]               Start the proxy SSL interface on port.
+    psp [port]               Stop the proxy interface on port.
+
+    Operations with the proxy (if compiled with proxy client support):
+    stt [server_address] <device_key>  Start the proxy client.
+    stp                                Stop the proxy client.
+    rs  [token]                        Resubscribe to opendht.
+    rp  [token]                        Inject a push notification.
 .EE
 .SH OPTIONS
 .TP
-\fB\-h\fP
-Prints some help.
+\fB\-h\fP, \fB\-\-help\fP
+Prints help and exits.
 .TP
-\fB\-v\fP
-Enable the verbose mode (log to stdout/stderr by default)
+\fB\-V\fP, \fB\-\-version\fP
+Prints version information and exits.
 .TP
-\fB\-l\fP \fIlog_file\fP
-Write log to file instead of stdout/stderr
+\fB\-v\fP, \fB\-\-verbose\fP
+Enable verbose mode (log to stdout/stderr by default).
 .TP
-\fB\-L\fP
-Write log to syslog instead of stdout/stderr
+\fB\-l\fP \fIlogfile\fP, \fB\-\-logfile\fP \fIlogfile\fP
+Write log to file instead of stdout/stderr.
 .TP
-\fB\-i\fP
-Generate cryptographic identity for the node.
+\fB\-L\fP, \fB\-\-syslog\fP
+Write log to syslog instead of stdout/stderr (enables verbose mode).
 .TP
-\fB\-\-save\-identity\fP \fIfile\fP
+\fB\-i\fP, \fB\-\-identity\fP
+Generate a cryptographic identity (RSA key pair and self-signed certificate)
+for the node. Required to perform cryptographic operations (encrypt/sign values)
+if no private key file is provided.
+.TP
+\fB\-I\fP \fIfile\fP, \fB\-\-save\-identity\fP \fIfile\fP
 Save generated identity (certificate and private key) to given file prefix.
+Files are saved as \fIfile\fP and \fIfile\fP_ca.
 .TP
-\fB\-\-certificate\fP \fIfile\fP
-Load identity certificate from given file.
+\fB\-c\fP \fIfile\fP, \fB\-\-certificate\fP \fIfile\fP
+Load identity certificate from given file. If specified, the certificate is
+published on the DHT.
 .TP
-\fB\-\-privkey\fP \fIfile\fP
-Load identity private key from given file.
+\fB\-k\fP \fIfile\fP, \fB\-\-privkey\fP \fIfile\fP
+Load identity private key from given file. Required to perform cryptographic
+operations if no identity is generated.
 .TP
-\fB\-\-privkey\-password\fP \fIpassword\fP
+\fB\-m\fP \fIpassword\fP, \fB\-\-privkey\-password\fP \fIpassword\fP
 Password to use for private key encryption or decryption (optional).
 .TP
-\fB\-d\fP
-Run the program in daemon mode (will fork in the background).
+\fB\-d\fP, \fB\-\-daemonize\fP
+Run the program in daemon mode (will fork in the background). When verbose mode
+is enabled, logs to syslog by default.
 .TP
-\fB\-s\fP
+\fB\-s\fP, \fB\-\-service\fP
 Run the program in service mode (non\-forking daemon).
 .TP
-\fB\-D\fP
+\fB\-D\fP, \fB\-\-peer\-discovery\fP
 Enables multicast automatic local peer discovery.
 .TP
-\fB\-f\fP \fIfile\fP
-Specify a file path to persist/load the node state.
+\fB\-U\fP, \fB\-\-no\-rate\-limit\fP
+Disable rate limiting (allows unlimited requests and storage).
 .TP
-\fB\-n\fP \fInetwork_id\fP
-Specify the network id. This let you connect to distinct networks and prevents
-the merge of two different networks (available since OpenDHT v0.6.1).
+\fB\-P\fP, \fB\-\-public\-stable\fP
+Enable public stable mode.
 .TP
-\fB\-p\fP \fIlocal_port\fP
-Use UDP port \fIlocal_port\fP for the program to bind to.
+\fB\-f\fP \fIpath\fP, \fB\-\-persist\fP \fIpath\fP
+Specify a file path to persist/load the node state (routing table and values).
 .TP
-\fB\-b\fP \fIbootstrap_host\fP[:\fIport\fP]
-The program needs to be given a node to connect to the network. You use this
-option to provide the ip address of that node.
+\fB\-n\fP \fInetwork_id\fP, \fB\-\-net\fP \fInetwork_id\fP
+Specify the network id. This lets you connect to distinct networks and prevents
+the merge of two different networks. Defaults to 0 (the main public network).
 .TP
-\fB\-\-proxyserver\fP \fIlocal_port\fP
-Run a proxy server bound to this DHT node on HTTP port \fIlocal_port\fP
+\fB\-p\fP \fIlocal_port\fP, \fB\-\-port\fP \fIlocal_port\fP
+Use UDP port \fIlocal_port\fP for the program to bind to. If not set, will use
+any available port. The default OpenDHT port for public nodes is 4222.
 .TP
-\fB\-\-proxyclient\fP \fIserver\fP
-Run this DHT node in proxy client mode, and connect to \fIserver\fP
+\fB\-b\fP \fIbootstrap_host\fP[:\fIport\fP], \fB\-\-bootstrap\fP \fIbootstrap_host\fP[:\fIport\fP]
+Specify a bootstrap node address to connect to the network.
+.TP
+\fB\-S\fP \fIlocal_port\fP, \fB\-\-proxyserver\fP \fIlocal_port\fP
+Run a REST API proxy server bound to this DHT node on HTTP port \fIlocal_port\fP.
+.TP
+\fB\-e\fP \fIlocal_port\fP, \fB\-\-proxyserverssl\fP \fIlocal_port\fP
+Run a REST API proxy server with SSL bound to this DHT node on HTTPS port \fIlocal_port\fP.
+.TP
+\fB\-a\fP \fIaddress\fP, \fB\-\-proxy\-addr\fP \fIaddress\fP
+Proxy server bind address.
+.TP
+\fB\-w\fP \fIfile\fP, \fB\-\-proxy\-certificate\fP \fIfile\fP
+Load proxy SSL certificate from given file.
+.TP
+\fB\-K\fP \fIfile\fP, \fB\-\-proxy\-privkey\fP \fIfile\fP
+Load proxy SSL private key from given file.
+.TP
+\fB\-M\fP \fIpassword\fP, \fB\-\-proxy\-privkey\-password\fP \fIpassword\fP
+Password for the proxy private key encryption or decryption (optional).
+.TP
+\fB\-C\fP \fIserver\fP, \fB\-\-proxyclient\fP \fIserver\fP
+Run this DHT node in proxy client mode, and connect to \fIserver\fP.
+.TP
+\fB\-y\fP \fIendpoint\fP, \fB\-\-pushserver\fP \fIendpoint\fP
+Push notification server endpoint.
+.TP
+\fB\-z\fP \fIkey\fP, \fB\-\-devicekey\fP \fIkey\fP
+Device key for push notifications.
+.TP
+\fB\-u\fP \fIbundleid\fP, \fB\-\-bundleid\fP \fIbundleid\fP
+Bundle ID for push notifications.
+.SH EXAMPLES
+Run a node using bootstrap.jami.net as a bootstrap:
+.PP
+.EX
+dhtnode -b bootstrap.jami.net:4222
+.EE
+.PP
+Run a node as a non-forking daemon on port 4222:
+.PP
+.EX
+dhtnode -b bootstrap.jami.net -p 4222 -s
+.EE
+.PP
+Run a proxy server on port 8000:
+.PP
+.EX
+dhtnode -b bootstrap.jami.net:4222 --proxyserver 8000
+.EE
+.PP
+Use a proxy client to connect to a proxy server:
+.PP
+.EX
+dhtnode --proxyclient dhtproxy.jami.net:80
+.EE
 .SH AUTHORS
 .TP
 Program written by

include/opendht/dhtrunner.h

@@ -299,13 +299,11 @@ public:
         putEncrypted(hash, to, value, bindDoneCb(cb), permanent);
     }
 
-    [[deprecated("Use the shared_ptr version instead")]]
     void putEncrypted(InfoHash hash,
                       const std::shared_ptr<crypto::PublicKey>& to,
                       Value&& value,
                       DoneCallback cb = {},
                       bool permanent = false);
-    [[deprecated("Use the shared_ptr version instead")]]
     void putEncrypted(InfoHash hash,
                       const std::shared_ptr<crypto::PublicKey>& to,
                       Value&& value,
@@ -322,6 +320,14 @@ public:
     {
         putEncrypted(hash, to, value, bindDoneCb(cb), permanent);
     }
+    void putEncrypted(InfoHash hash, const PkId& to, Value&& value, DoneCallback cb = {}, bool permanent = false)
+    {
+        putEncrypted(hash, to, std::make_shared<Value>(std::move(value)), std::move(cb), permanent);
+    }
+    void putEncrypted(InfoHash hash, const PkId& to, Value&& value, DoneCallbackSimple cb, bool permanent = false)
+    {
+        putEncrypted(hash, to, std::make_shared<Value>(std::move(value)), bindDoneCb(cb), permanent);
+    }
 
     /**
      * Insert known nodes to the routing table, without necessarly ping them.

include/opendht/proxy.h

@@ -10,6 +10,14 @@ namespace proxy {
 constexpr const std::chrono::hours OP_TIMEOUT {24}; // one day
 constexpr const std::chrono::hours OP_MARGIN {2};   // two hours
 
+/**
+ * Deadline for a subscribe request to complete. Unlike a listen, a subscribe is a short
+ * request: the server answers immediately and the device then relies on push notifications.
+ * Nothing else observes it, so without a deadline a subscribe stalled in name resolution,
+ * connection or TLS handshake would never complete nor fail.
+ */
+constexpr const std::chrono::seconds SUBSCRIBE_TIMEOUT {20};
+
 using ListenToken = uint64_t;
 
 } // namespace proxy

include/opendht/thread_pool.h

@@ -55,10 +55,12 @@ public:
 private:
     std::mutex lock_;
     std::condition_variable cv_ {};
+    std::condition_variable stopCv_ {};
     std::queue<std::function<void()>> tasks_ {};
     std::vector<std::unique_ptr<std::thread>> threads_;
     unsigned readyThreads_ {0};
     bool running_ {true};
+    bool stopping_ {false};
 
     unsigned minThreads_;
     const unsigned maxThreads_;

src/dht_proxy_client.cpp

@@ -1075,6 +1075,12 @@ DhtProxyClient::sendListen(const restinio::http_request_header_t& header,
         request->set_body(body);
 #endif
         auto rxBuf = std::make_shared<LineSplit>();
+        // Completion flag shared with the subscribe deadline armed below. The request
+        // callbacks replace each other rather than accumulating, so the flag is raised from
+        // the single on_done callback instead of a dedicated one.
+        std::shared_ptr<std::atomic_bool> subscribeDone;
+        if (method != ListenMethod::LISTEN)
+            subscribeDone = std::make_shared<std::atomic_bool>(false);
         request->add_on_body_callback([this, reqid, opstate, rxBuf, cb](const char* at, size_t length) {
             try {
                 auto& b = *rxBuf;
@@ -1113,7 +1119,9 @@ DhtProxyClient::sendListen(const restinio::http_request_header_t& header,
                 opstate->ok.store(false);
             }
         });
-        request->add_on_done_callback([this, opstate, reqid](const http::Response& response) {
+        request->add_on_done_callback([this, opstate, reqid, subscribeDone](const http::Response& response) {
+            if (subscribeDone)
+                subscribeDone->store(true);
             if (response.status_code != 200) {
                 if (logger_)
                     logger_->error("[proxy:client] [listen] send request #{} failed with code={}",
@@ -1138,6 +1146,29 @@ DhtProxyClient::sendListen(const restinio::http_request_header_t& header,
                 request->get_connection()->set_keepalive(seconds);
             }
         });
+        if (subscribeDone) {
+            // Give the subscribe a deadline. Nothing else would ever release it: the
+            // keepalive above is only armed once the server has answered, and no other
+            // request in this client sets a timeout. A subscribe that never completes
+            // leaves the listener registered locally while the server never learns about
+            // it, so the device stops being notified without anything reporting an error.
+            // Cancelling runs the on_done callback, which clears the request, and failing
+            // the operation lets restartListeners() send the subscription again.
+            std::weak_ptr<http::Request> wreq = request;
+            request->timeout(proxy::SUBSCRIBE_TIMEOUT,
+                             [this, reqid, opstate, subscribeDone, wreq](const asio::error_code& ec) {
+                                 if (ec == asio::error::operation_aborted or subscribeDone->load()
+                                     or isDestroying_)
+                                     return;
+                                 if (logger_)
+                                     logger_->error("[proxy:client] [listen] subscribe request #{} timed out",
+                                                    reqid);
+                                 opstate->ok.store(false);
+                                 if (auto req = wreq.lock())
+                                     req->cancel();
+                                 opFailed();
+                             });
+        }
         request->send();
     } catch (const std::exception& e) {
         if (logger_)

src/thread_pool.cpp

@@ -78,6 +78,8 @@ ThreadPool::run(std::function<void()>&& cb)
                             break;
                         task = std::move(tasks_.front());
                         tasks_.pop();
+                        if (tasks_.empty() && stopping_)
+                            stopCv_.notify_one();
                     }
 
                     // run task
@@ -135,7 +137,9 @@ ThreadPool::stop(bool wait)
 {
     std::unique_lock l(lock_);
     if (wait) {
-        cv_.wait(l, [&]() { return tasks_.empty(); });
+        stopping_ = true;
+        stopCv_.wait(l, [&]() { return tasks_.empty(); });
+        stopping_ = false;
     }
     running_ = false;
     tasks_ = {};

tests/test_threadpool.cpp

@@ -80,6 +80,46 @@ ThreadPoolTester::testContext()
     CPPUNIT_ASSERT_EQUAL(N, count.load());
 }
 
+void
+ThreadPoolTester::testJoinNonEmptyTaskQueue()
+{
+    dht::ThreadPool pool(2);
+
+    constexpr unsigned N = 128;
+    std::atomic_uint count {0};
+
+    // Schedule enough work so that the task queue is non-empty when calling
+    // pool.join() below.
+    for (unsigned i = 0; i < N; i++)
+        pool.run([&] {
+            std::this_thread::sleep_for(std::chrono::milliseconds(1));
+            count++;
+        });
+
+    std::atomic_bool joined {false};
+    std::condition_variable cv;
+    std::thread joinThread([&] {
+        CPPUNIT_ASSERT(count.load() < N);
+        pool.join();
+        joined = true;
+        cv.notify_one();
+    });
+
+    // After calling join(), all pending tasks should complete, and then all
+    // threads in the pool should join.
+    std::mutex mtx;
+    std::unique_lock lock(mtx);
+    bool poolJoined = cv.wait_for(lock, std::chrono::seconds(10), [&] { return joined.load(); });
+    if (!poolJoined) {
+        // Detach joinThread if we timed out, otherwise the test can crash.
+        joinThread.detach();
+    } else {
+        joinThread.join();
+    }
+    CPPUNIT_ASSERT_EQUAL_MESSAGE("Some tasks were not completed", N, count.load());
+    CPPUNIT_ASSERT_MESSAGE("Thread pool failed to join before timeout", poolJoined);
+}
+
 void
 ThreadPoolTester::tearDown()
 {}

tests/test_threadpool.h

@@ -14,6 +14,7 @@ class ThreadPoolTester : public CppUnit::TestFixture
     CPPUNIT_TEST(testThreadPool);
     CPPUNIT_TEST(testExecutor);
     CPPUNIT_TEST(testContext);
+    CPPUNIT_TEST(testJoinNonEmptyTaskQueue);
     CPPUNIT_TEST_SUITE_END();
 
 public:
@@ -29,6 +30,7 @@ public:
     void testThreadPool();
     void testExecutor();
     void testContext();
+    void testJoinNonEmptyTaskQueue();
 };
 
 } // namespace test

tools/dhtchat.cpp

@@ -123,7 +123,7 @@ main(int argc, char** argv)
                     iss >> idstr;
                     std::getline(iss, line);
                     dht.putEncrypted(room,
-                                     InfoHash(idstr),
+                                     PkId(idstr),
                                      dht::ImMessage(IdDist()(tools::rd), std::move(line), now),
                                      [](bool ok) {
                                          // dht.cancelPut(room, id);