tools/dhtnode: print listen sync status

Fail pending readiness when LISTEN retries are exhausted, reject search capacity exhaustion with a zero token rather than an exception, guard obsolete expiration callbacks, and cancel local and family listeners after partial registration fails.

CMakeLists.txt

@@ -9,8 +9,8 @@ if(POLICY CMP0167)
     cmake_policy(SET CMP0167 NEW)
 endif()
 
-set (opendht_VERSION_MAJOR 4)
-set (opendht_VERSION_MINOR 4.1)
+set (opendht_VERSION_MAJOR 5)
+set (opendht_VERSION_MINOR 0.0)
 set (opendht_VERSION ${opendht_VERSION_MAJOR}.${opendht_VERSION_MINOR})
 set (PACKAGE_VERSION ${opendht_VERSION})
 set (VERSION "${opendht_VERSION}")

README.md

@@ -23,6 +23,8 @@ Values are distributed over the network, with redundancy.
 ## Documentation
 See the wiki: <https://github.com/savoirfairelinux/opendht/wiki>
 
+Push subscription recovery protocol: [doc/proxy-sync.md](doc/proxy-sync.md).
+
 #### How to run a node
 
 You can help contribute to the public network by running a stable node with a public IP address.

doc/proxy-sync.md

@@ -0,0 +1,178 @@
+# Push Subscription Synchronization
+
+Revision-aware clients require an updated proxy. Old clients remain supported:
+requests without `sync` retain the original subscription response format.
+
+## Validation and Snapshots
+
+A `SUBSCRIBE /key/<hash>` request contains the existing push registration fields
+and a `sync` object. An empty object requests a full snapshot. A synchronized
+client instead supplies its last reconciled token:
+
+```json
+{"sync":{"epoch":"<listener-epoch>","revision":7}}
+```
+
+The epoch is unique to the server-side listener. New and restored listeners get
+fresh epochs. Each addition, edit, or expiration reported by the DHT listener
+increments its revision. Epoch/revision state is not restored from disk.
+
+When the token matches, the response contains only a completion record. Otherwise,
+the response contains newline-delimited value objects followed by this record:
+
+```json
+{"sync":{"epoch":"<listener-epoch>","revision":8,"snapshot":true,"complete":true}}
+```
+
+The listener retains its current values, including edits and expirations, under
+the same mutex as its revision. The server copies both atomically; it does not
+run a GET for a synchronization request. New listeners wait for initial readiness
+before responding. `snapshot:false` means validation found the client's cache
+unchanged. Successful responses always have `complete:true`; initial search
+failure returns HTTP 503. The deprecated `retry` field is not used. Every
+subscription request renews the lease, independently of whether values are sent.
+
+Clients buffer snapshot values and reconcile only after a valid completion
+record and successful HTTP framing. Reconciliation delivers new values and
+higher-sequence edits, suppresses duplicates and older versions, and expires
+cached IDs absent from the snapshot. Failed or invalidated responses never
+remove cached values. Snapshot deletion is authoritative and does not depend
+on wall-clock timestamps.
+
+## Push Ordering and Recovery
+
+Value pushes include string fields `epoch` and `revision`. A push no newer than
+the reconciled revision is ignored. Revisions must be unsigned decimal integers
+without signs, whitespace, trailing characters, or overflow. A contiguous
+expiration revision removes the listed IDs locally without a download. The
+`expired:"true"` field identifies batched expirations using `ids`; the original
+single-value `exp` field remains supported. Other same-epoch changes and revision
+gaps require a cached snapshot. Unknown epochs first validate against the proxy,
+so delayed pushes from old epochs do not force a value download. Revisions cover
+edits as well as additions and deletions.
+
+Clients allow one sync request per listener and coalesce overlapping triggers.
+Request generations reject callbacks from superseded requests. Completion state
+changes run under the listener's owning lock after checking the generation.
+Same-epoch push observations can invalidate an older snapshot. Confirmed retired
+epochs are remembered in a bounded, push-token-scoped set; other unknown epochs
+are validated without changing cached values. Returning to an earlier push token
+remains valid.
+Token rotation during a pending request is applied after that request completes.
+Clients require an updated proxy server. Value pushes without both epoch and
+revision are malformed and return `Error`; they do not mutate the listener cache
+or trigger synchronization requests. The proxy adds these fields independently
+of whether its upstream DHT peers support version-2 native LISTEN.
+Pushes for another client session return `IgnoredWrongSession` without changing
+subscriptions, cached values, or connectivity state. Their application-level
+purpose is to wake the mobile app; session creation and desired subscriptions
+are independent actions. A new session requests its own full initial snapshot
+and does not inherit synchronization state from the rejected push.
+
+Connectivity changes and existing lease refreshes validate the last reconciled
+token; unchanged data is not downloaded again. Successful lease renewal alone
+does not clear pending recovery. Failed recovery retries use the existing lease
+timer with exponential delays from one to sixty seconds; successful recovery
+restores the normal lease schedule. Stale and malformed responses also use this
+backoff, rather than retrying immediately. Each subscription request has a
+30-second steady-clock deadline covering DNS, connection, and response body
+receipt. Expired requests release the in-flight slot and enter recovery. No
+healthy-state polling timer is added.
+
+Without another event or polling, a silently dropped final push cannot be
+detected immediately. The next connectivity validation or existing lease refresh
+detects a revision mismatch. Guarantees concern reconciliation with the proxy's
+observed DHT state, not linearizable global DHT storage.
+
+## Listener Readiness
+
+The existing four-argument `listen` API remains available. The new overload adds
+a one-shot `DoneCallbackSimple` as its fifth argument:
+
+```cpp
+runner.listen(key, valuesCallback, {}, {}, readyCallback);
+```
+
+`readyCallback(true)` follows completed search/listener synchronization for each
+enabled address family. Version-2 native sources complete after initial delivery
+and contiguous updates through their observed high-water mark. Legacy sources
+complete when LISTEN registration is acknowledged; initial values can arrive
+after readiness. Isolated families with no peers are ready after delivering local
+values. Initial search or registration failure reports `false`; cancellation
+does not promise a callback.
+This is an initial completion notification, not a global consistency guarantee.
+It does not start a GET or add polling. Attaching to a synchronized cache delivers
+cached values before notifying readiness.
+Mixed-version and entirely legacy networks can initialize successfully without
+GET fallback or polling. A legacy acknowledgment confirms registration, not
+complete baseline delivery or absence of values. Late values remain ordinary
+listener updates: the proxy advances its revision and push clients reconcile
+through the existing subscription path. A client snapshot is authoritative for
+the proxy's observed cache, not proof of a complete upstream DHT baseline. A
+legacy update never received by the proxy cannot be detected by proxy revision
+validation. Proxy readiness callbacks run outside the client's search lock;
+cancellation suppresses queued notifications.
+
+Streaming proxy clients request `/key/<hash>/listen?sync=1`. Updated proxies send
+`{"sync":{"complete":true}}` after initial values, or `complete:false` on initial
+failure. Requests without this opt-in retain the original stream format.
+
+## Native Initial Delivery
+
+Native clients request LISTEN version `ve:2` with an `ls` tuple:
+
+```text
+[registration, epoch, sequence, snapshotEnd]
+```
+
+The registration is a fresh nonzero client nonce; a LISTEN request sends zeros
+for the other fields. The server explicitly acknowledges version 2, assigns a
+nonzero stream epoch, and declares the number of initial UPDATE batches in
+`snapshotEnd`. An empty baseline declares zero batches. Each UPDATE carries the
+same registration, epoch, and snapshotEnd with a consecutive sequence starting
+at one. The initial batches occupy sequences 1 through snapshotEnd; live changes
+follow them. The acknowledgment's sequence is the server's current high-water
+mark. All tuple fields are unsigned 64-bit integers.
+
+The server registers the listener and serializes the initial batches on its DHT
+thread. Existing UPDATE fragmentation and acknowledgments deliver each initial
+value once per source registration on a loss-free connection. There is no
+additional GET or second value download. Retries resend the same immutable
+request bytes. Repeated registrations with the same nonce reuse the baseline;
+healthy renewals return metadata only. Recreated sockets use a new nonce and
+cannot accept the previous registration's updates.
+
+Live updates, including edits, expirations, and refreshes, are sent immediately
+while the baseline is in flight. The receiver buffers them until every declared
+initial batch has been fully reassembled and validated. It then applies the
+baseline followed by contiguous live changes in sequence order. Duplicates are
+acknowledged without being applied again. Known sequence gaps prevent the cache
+from being considered synchronized; retransmission can fill them. Initialization
+or an unresolved live gap fails after 30 seconds and allows a fresh registration.
+
+Batches contain at most 32 values and target the existing 56 KiB serialized-value
+budget; individual larger values retain the existing fragment transport. The
+receiver bounds buffered batches to 4096 and payload bytes to 64 MiB, and rejects
+baselines exceeding its 4096-value cache capacity. Invalid version-2 values fail
+the batch instead of being silently skipped. This design adds neither pacing
+nor send windows. Per-peer baselines are coherent with their subsequent updates,
+but do not provide an atomic snapshot of the entire distributed DHT.
+
+Peers that do not explicitly acknowledge version 2 remain legacy listeners.
+Their values are delivered normally and successful LISTEN acknowledgment
+completes their best-effort initialization. No GET fallback or polling is added.
+Legacy LISTEN acknowledgments may omit the token; clients retain the token
+previously obtained during discovery and keep the listener socket open. A
+version-2 acknowledgment must include its token and synchronization metadata.
+The source records whether it uses legacy completion or a verified version-2
+stream. An established version-2 stream cannot downgrade on a metadata-free
+acknowledgment; invalid initialization and registration failures still report
+failure.
+
+## HTTP Transport Note
+
+HTTP requests now report status code zero for transport errors and cancellation,
+even if a 200 status was received before the failure. Cancellation also sets
+`aborted`. EOF succeeds only when the HTTP parser confirms complete framing;
+truncated content-length and chunked responses fail. Callers must not treat an
+earlier status callback as proof that a response body completed successfully.

include/opendht/dht.h

@@ -236,6 +236,7 @@ public:
      * @return a token to cancel the listener later.
      */
     size_t listen(const InfoHash&, ValueCallback, Value::Filter = {}, Where = {}) override;
+    size_t listen(const InfoHash&, ValueCallback, Value::Filter, Where, DoneCallbackSimple ready) override;
 
     size_t listen(const InfoHash& key, GetCallback cb, Value::Filter f = {}, Where w = {}) override
     {
@@ -460,7 +461,12 @@ private:
     void reportedAddr(const SockAddr&);
 
     // Storage
-    void storageAddListener(const InfoHash& id, const Sp<Node>& node, size_t tid, Query&& = {}, int version = 0);
+    std::optional<net::ListenSync> storageAddListener(const InfoHash& id,
+                                                      const Sp<Node>& node,
+                                                      size_t tid,
+                                                      Query&& = {},
+                                                      int version = 0,
+                                                      const std::optional<net::ListenSync>& sync = {});
     bool storageStore(const InfoHash& id,
                       const Sp<Value>& value,
                       time_point created,
@@ -535,7 +541,12 @@ private:
                   DoneCallback callback,
                   time_point created = time_point::max(),
                   bool permanent = false);
-    size_t listenTo(const InfoHash& id, sa_family_t af, ValueCallback cb, Value::Filter f = {}, const Sp<Query>& q = {});
+    size_t listenTo(const InfoHash& id,
+                    sa_family_t af,
+                    ValueCallback cb,
+                    Value::Filter f = {},
+                    const Sp<Query>& q = {},
+                    DoneCallbackSimple ready = {});
 
     /**
      * Refill the search with good nodes if possible.
@@ -609,6 +620,7 @@ private:
     void searchStep(std::weak_ptr<Search> ws);
 
     void searchSynchedNodeListen(const Sp<Search>&, SearchNode&);
+    void searchNodeListenDeadline(const Sp<Search>&, SearchNode&, const Sp<Query>&);
 
     void dumpSearch(const Search& sr, std::ostream& out) const;
 
@@ -626,8 +638,13 @@ private:
     net::RequestAnswer onGetValues(Sp<Node> node, const InfoHash& hash, want_t want, const Query& q);
     void onGetValuesDone(const Sp<Node>& status, net::RequestAnswer& a, Sp<Search>& sr, const Sp<Query>& orig_query);
     /* when we receive a listen request */
-    net::RequestAnswer onListen(
-        Sp<Node> node, const InfoHash& hash, const Blob& token, size_t socket_id, const Query& query, int version = 0);
+    net::RequestAnswer onListen(Sp<Node> node,
+                                const InfoHash& hash,
+                                const Blob& token,
+                                size_t socket_id,
+                                const Query& query,
+                                int version = 0,
+                                const std::optional<net::ListenSync>& sync = {});
     void onListenDone(const Sp<Node>& node, net::RequestAnswer& a, Sp<Search>& sr);
     /* when we receive an announce request */
     net::RequestAnswer onAnnounce(Sp<Node> node,

include/opendht/dht_interface.h

@@ -79,8 +79,7 @@ public:
 
     virtual time_point periodic(const uint8_t* buf, size_t buflen, SockAddr, const time_point& now) = 0;
     virtual time_point periodic(
-        const uint8_t* buf, size_t buflen, const sockaddr* from, socklen_t fromlen, const time_point& now)
-        = 0;
+        const uint8_t* buf, size_t buflen, const sockaddr* from, socklen_t fromlen, const time_point& now) = 0;
 
     /**
      * Get a value by searching on all available protocols (IPv4, IPv6),
@@ -93,17 +92,22 @@ public:
      * @param f a filter function used to prefilter values.
      */
     virtual void get(
-        const InfoHash& key, GetCallback cb, DoneCallback donecb = {}, Value::Filter&& f = {}, Where&& w = {})
-        = 0;
-    virtual void get(
-        const InfoHash& key, GetCallback cb, DoneCallbackSimple donecb = {}, Value::Filter&& f = {}, Where&& w = {})
-        = 0;
-    virtual void get(
-        const InfoHash& key, GetCallbackSimple cb, DoneCallback donecb = {}, Value::Filter&& f = {}, Where&& w = {})
-        = 0;
-    virtual void get(
-        const InfoHash& key, GetCallbackSimple cb, DoneCallbackSimple donecb, Value::Filter&& f = {}, Where&& w = {})
-        = 0;
+        const InfoHash& key, GetCallback cb, DoneCallback donecb = {}, Value::Filter&& f = {}, Where&& w = {}) = 0;
+    virtual void get(const InfoHash& key,
+                     GetCallback cb,
+                     DoneCallbackSimple donecb = {},
+                     Value::Filter&& f = {},
+                     Where&& w = {}) = 0;
+    virtual void get(const InfoHash& key,
+                     GetCallbackSimple cb,
+                     DoneCallback donecb = {},
+                     Value::Filter&& f = {},
+                     Where&& w = {}) = 0;
+    virtual void get(const InfoHash& key,
+                     GetCallbackSimple cb,
+                     DoneCallbackSimple donecb,
+                     Value::Filter&& f = {},
+                     Where&& w = {}) = 0;
 
     /**
       * Similar to Dht::get, but sends a Query to filter data remotely.
@@ -138,26 +142,22 @@ public:
                      Sp<Value>,
                      DoneCallback cb = nullptr,
                      time_point created = time_point::max(),
-                     bool permanent = false)
-        = 0;
+                     bool permanent = false) = 0;
     virtual void put(const InfoHash& key,
                      const Sp<Value>& v,
                      DoneCallbackSimple cb,
                      time_point created = time_point::max(),
-                     bool permanent = false)
-        = 0;
+                     bool permanent = false) = 0;
     virtual void put(const InfoHash& key,
                      Value&& v,
                      DoneCallback cb = nullptr,
                      time_point created = time_point::max(),
-                     bool permanent = false)
-        = 0;
+                     bool permanent = false) = 0;
     virtual void put(const InfoHash& key,
                      Value&& v,
                      DoneCallbackSimple cb,
                      time_point created = time_point::max(),
-                     bool permanent = false)
-        = 0;
+                     bool permanent = false) = 0;
 
     /**
      * Get data currently being put at the given hash.
@@ -261,6 +261,25 @@ public:
      */
     virtual PushNotificationResult pushNotificationReceived(const std::map<std::string, std::string>& data) = 0;
 
+    /**
+     * Listen with a one-shot notification after initial synchronization completes
+     * on all enabled address families. Version-2 native peers complete after
+     * initial delivery; legacy peers complete on LISTEN acknowledgment and may
+     * deliver initial values after the notification.
+     * @param ready called with false on initialization failure or shutdown.
+     *        Canceling the listener suppresses a pending readiness notification.
+     *        The callback may run before listen returns. Implementations that do
+     *        not support readiness report false while preserving the listener.
+     */
+    virtual size_t listen(
+        const InfoHash& key, ValueCallback cb, Value::Filter filter, Where where, DoneCallbackSimple ready)
+    {
+        auto token = listen(key, std::move(cb), std::move(filter), std::move(where));
+        if (ready)
+            ready(false);
+        return token;
+    }
+
 protected:
     std::shared_ptr<Logger> logger_ {};
     std::queue<std::function<void()>> onConnectCallbacks_ {};

include/opendht/dht_proxy_client.h

@@ -189,6 +189,7 @@ public:
      * @return a token to cancel the listener later.
      */
     virtual size_t listen(const InfoHash&, ValueCallback, Value::Filter = {}, Where = {}) override;
+    size_t listen(const InfoHash&, ValueCallback, Value::Filter, Where, DoneCallbackSimple ready) override;
 
     virtual size_t listen(const InfoHash& key, GetCallback cb, Value::Filter f = {}, Where w = {}) override
     {
@@ -291,9 +292,10 @@ public:
     virtual size_t getStorageLimit() const override { return 0; }
     void setLocalStorageLimit(size_t) override {}
     virtual size_t getLocalStorageLimit() const override { return STORAGE_LIMIT_UNLIMITED; }
-    void connectivityChanged(sa_family_t) override { getProxyInfos(); }
+    void connectivityChanged(sa_family_t) override { connectivityChanged(); }
     void connectivityChanged() override
     {
+        launchConnectedCbs_ = true;
         getProxyInfos();
         loopSignal_();
     }
@@ -329,7 +331,6 @@ private:
     enum class ListenMethod {
         LISTEN,
         SUBSCRIBE,
-        RESUBSCRIBE,
     };
     using CacheValueCallback
         = std::function<bool(const std::vector<std::shared_ptr<Value>>& values, bool expired, system_clock::time_point)>;
@@ -341,11 +342,16 @@ private:
                     const CacheValueCallback& cb,
                     const Sp<OperationState>& opstate,
                     Listener& listener,
-                    ListenMethod method = ListenMethod::LISTEN);
+                    ListenMethod method,
+                    const InfoHash& key,
+                    size_t token);
     void handleResubscribe(const asio::error_code& ec,
                            const InfoHash& key,
                            const size_t token,
                            std::shared_ptr<OperationState> opstate);
+    void scheduleListenRetry(const InfoHash& key, size_t token, Listener& listener);
+    bool reconcileListenerSnapshot(Listener& listener, const std::vector<Sp<Value>>& values, const Json::Value& marker);
+    std::chrono::seconds subscriptionRequestTimeout_ {30};
 
     void doPut(const InfoHash&, Sp<Value>, DoneCallbackSimple, time_point created, bool permanent);
     void handleRefreshPut(const asio::error_code& ec, InfoHash key, Value::Id id);
@@ -419,6 +425,7 @@ private:
      */
     std::mutex lockCallbacks_;
     std::vector<std::function<void()>> callbacks_;
+    std::vector<std::function<void()>> readinessCallbacks_;
 
     Sp<InfoState> infoState_;
 
@@ -437,6 +444,8 @@ private:
     /**
      * Refresh a listen via a token
      * @param token
+     * @param fetchValues ask the server to send the values it stores, for a listener
+     * that may have missed some (its previous subscription failed).
      */
     void resubscribe(const InfoHash& key, const size_t token, Listener& listener);
 
@@ -468,7 +477,6 @@ private:
     const std::function<void()> loopSignal_;
 
 #ifdef OPENDHT_PUSH_NOTIFICATIONS
-    std::string fillBody(bool resubscribe);
     void getPushRequest(Json::Value&) const;
 #endif // OPENDHT_PUSH_NOTIFICATIONS
 

include/opendht/dht_proxy_server.h

@@ -272,12 +272,19 @@ private:
     {
         std::mutex lock;
         std::string sessionId;
+        const std::string epoch {InfoHash::getRandom().toString()};
+        uint64_t revision {0};
+        std::map<Value::Id, Json::Value> values;
+        bool initialized {false};
+        bool initialOk {false};
+        std::vector<std::function<void()>> pendingSnapshots;
         PushSessionContext(const std::string& id)
             : sessionId(id)
         {}
     };
 
 #ifdef OPENDHT_PUSH_NOTIFICATIONS
+    void onPushListenReady(const std::shared_ptr<PushSessionContext>& sessionCtx, bool ok);
     PushType getTypeFromString(const std::string& type);
     std::string getDefaultTopic(PushType type);
 

include/opendht/dhtrunner.h

@@ -156,6 +156,7 @@ public:
     }
 
     std::future<size_t> listen(InfoHash key, ValueCallback vcb, Value::Filter f = {}, Where w = {});
+    std::future<size_t> listen(InfoHash key, ValueCallback vcb, Value::Filter f, Where w, DoneCallbackSimple ready);
 
     std::future<size_t> listen(InfoHash key, GetCallback cb, Value::Filter f = {}, Where w = {})
     {

include/opendht/network_engine.h

@@ -19,6 +19,7 @@
 #include <algorithm>
 #include <memory>
 #include <queue>
+#include <optional>
 
 namespace dht {
 namespace net {
@@ -78,6 +79,15 @@ private:
 
 struct ParsedMessage;
 
+struct ListenSync
+{
+    uint64_t registration {0};
+    uint64_t epoch {0};
+    uint64_t sequence {0};
+    uint64_t snapshotEnd {0};
+    MSGPACK_DEFINE_ARRAY(registration, epoch, sequence, snapshotEnd)
+};
+
 /**
  * Answer for a request.
  */
@@ -91,6 +101,7 @@ struct RequestAnswer
     std::vector<Sp<FieldValueIndex>> fields {};
     std::vector<Sp<Node>> nodes4 {};
     std::vector<Sp<Node>> nodes6 {};
+    std::optional<ListenSync> listenSync {};
     RequestAnswer() {}
     RequestAnswer(ParsedMessage&& msg);
 };
@@ -167,7 +178,9 @@ private:
      * @param token (type: Blob) security token.
      * @param rid (type: uint16_t) request id.
      */
-    std::function<RequestAnswer(Sp<Node>, const InfoHash&, const Blob&, Tid, const Query&, int)> onListen {};
+    std::function<
+        RequestAnswer(Sp<Node>, const InfoHash&, const Blob&, Tid, const Query&, int, const std::optional<ListenSync>&)>
+        onListen {};
     /**
      * Called on announce request.
      *
@@ -231,29 +244,32 @@ public:
      * @param values      The values to send.
      * @param version     If version = 1, a request will be used to answer to the listener
      */
-    void tellListener(const Sp<Node>& n,
-                      Tid socket_id,
-                      const InfoHash& hash,
-                      want_t want,
-                      const Blob& ntoken,
-                      std::vector<Sp<Node>>&& nodes,
-                      std::vector<Sp<Node>>&& nodes6,
-                      std::vector<Sp<Value>>&& values,
-                      const Query& q,
-                      int version);
+    std::vector<Sp<Request>> tellListener(const Sp<Node>& n,
+                                          Tid socket_id,
+                                          const InfoHash& hash,
+                                          want_t want,
+                                          const Blob& ntoken,
+                                          std::vector<Sp<Node>>&& nodes,
+                                          std::vector<Sp<Node>>&& nodes6,
+                                          std::vector<Sp<Value>>&& values,
+                                          const Query& q,
+                                          int version,
+                                          ListenSync* sync = nullptr);
 
     void tellListenerRefreshed(const Sp<Node>& n,
                                Tid socket_id,
                                const InfoHash& hash,
                                const Blob& ntoken,
                                const std::vector<Value::Id>& values,
-                               int version);
+                               int version,
+                               ListenSync* sync = nullptr);
     void tellListenerExpired(const Sp<Node>& n,
                              Tid socket_id,
                              const InfoHash& hash,
                              const Blob& ntoken,
                              const std::vector<Value::Id>& values,
-                             int version);
+                             int version,
+                             ListenSync* sync = nullptr);
 
     bool isRunning(sa_family_t af) const;
     inline want_t want() const { return dht_socket->hasIPv4() and dht_socket->hasIPv6() ? (WANT4 | WANT6) : -1; }
@@ -358,7 +374,8 @@ public:
                            const Blob& token,
                            Tid socketId,
                            RequestCb&& on_done,
-                           RequestExpiredCb&& on_expired);
+                           RequestExpiredCb&& on_expired,
+                           const std::optional<ListenSync>& sync = {});
     /**
      * Send a "announce" request to a given node.
      *
@@ -411,19 +428,22 @@ public:
      *
      * @return the request with information concerning its success.
      */
-    void sendUpdateValues(const Sp<Node>& n,
-                          const InfoHash& infohash,
-                          std::vector<Sp<Value>>&& values,
-                          time_point created,
-                          const Blob& token,
-                          size_t sid);
+    static size_t getUpdateBatchCount(const std::vector<Sp<Value>>& values);
+    std::vector<Sp<Request>> sendUpdateValues(const Sp<Node>& n,
+                                              const InfoHash& infohash,
+                                              std::vector<Sp<Value>>&& values,
+                                              time_point created,
+                                              const Blob& token,
+                                              size_t sid,
+                                              ListenSync* sync = nullptr);
     Sp<Request> sendUpdateValues(const Sp<Node>& n,
                                  const InfoHash& infohash,
                                  std::vector<Sp<Value>>::iterator begin,
                                  std::vector<Sp<Value>>::iterator end,
                                  time_point created,
                                  const Blob& token,
-                                 size_t sid);
+                                 size_t sid,
+                                 ListenSync* sync = nullptr);
 
     /**
      * Parses a message and calls appropriate callbacks.
@@ -552,7 +572,10 @@ private:
     std::pair<Blob, Blob> bufferNodes(
         sa_family_t af, const InfoHash& id, want_t want, std::vector<Sp<Node>>& nodes, std::vector<Sp<Node>>& nodes6);
     /* answer to a listen request */
-    void sendListenConfirmation(const SockAddr& addr, Tid tid, const Blob& token = {});
+    void sendListenConfirmation(const SockAddr& addr,
+                                Tid tid,
+                                const Blob& token = {},
+                                const std::optional<ListenSync>& sync = {});
     /* answer to put request */
     void sendValueAnnounced(const SockAddr& addr, Tid, Value::Id);
     /* answer in case of error */

include/opendht/securedht.h

@@ -272,6 +272,7 @@ public:
     bool cancelPut(const InfoHash& h, const Value::Id& vid) override { return dht_->cancelPut(h, vid); }
 
     size_t listen(const InfoHash& key, ValueCallback, Value::Filter = {}, Where = {}) override;
+    size_t listen(const InfoHash& key, ValueCallback, Value::Filter, Where, DoneCallbackSimple ready) override;
     size_t listen(const InfoHash& key, GetCallback cb, Value::Filter = {}, Where w = {}) override;
     size_t listen(const InfoHash& key, GetCallbackSimple cb, Value::Filter f = {}, Where w = {}) override
     {

meson.build

@@ -5,7 +5,7 @@ project(
     'cython',
     license: 'MIT',
     license_files: 'LICENSE',
-    version: '4.4.1',
+    version: '5.0.0',
     meson_version: '>=1.1.0',
     default_options: ['cpp_std=c++20', 'warning_level=3'],
 )
@@ -129,6 +129,8 @@ endif
 opendht = library(
     'opendht',
     opendht_src,
+    version: meson.project_version(),
+    soversion: '5',
     include_directories: opendht_inc,
     dependencies: deps,
     cpp_args: ['-DOPENDHT_BUILD', '-Dopendht_EXPORTS'],
@@ -178,6 +180,8 @@ if get_option('c').enabled()
     opendht_c = library(
         'opendht-c',
         ['c/opendht.cpp', 'c/opendht_c.h'],
+        version: meson.project_version(),
+        soversion: '5',
         dependencies: [opendht_dep, msgpack, jsoncpp, fmt],
         cpp_args: ['-DOPENDHT_C_BUILD', '-Dopendht_c_EXPORTS'],
         install: true,
@@ -344,6 +348,17 @@ if get_option('tests').enabled()
         test('Http', test_http)
     endif
 
+    if get_option('proxy_client').enabled()
+        test_proxy_client = executable(
+            'test_dhtproxy_client',
+            'tests/test_dhtproxy_client.cpp',
+            'tests/tests_runner.cpp',
+            include_directories: opendht_inc,
+            dependencies: [opendht_dep, cppunit, jsoncpp, fmt, openssl, msgpack, llhttp],
+        )
+        test('DhtProxyClient', test_proxy_client)
+    endif
+
     if get_option('peer_discovery').enabled()
         test_peerdiscovery = executable(
             'test_peerdiscovery',
@@ -361,7 +376,7 @@ if get_option('tests').enabled()
             'tests/tests_runner.cpp',
             dependencies: [opendht_dep, cppunit, jsoncpp, fmt, openssl, msgpack],
         )
-        test('DhtProxy', test_proxy)
+        test('DhtProxy', test_proxy, timeout: 120)
 
         if get_option('long_tests').enabled()
             test_proxystress = executable(

src/dht.cpp

@@ -83,6 +83,7 @@ Dht::shutdown(ShutdownCallback cb, bool stop)
                         cb(false, {});
                 }
                 sr.second->announce.clear();
+                sr.second->cache.onReady(false);
                 sr.second->listeners.clear();
             }
         }
@@ -648,6 +649,30 @@ Dht::searchSendAnnounceValue(const Sp<Search>& sr, unsigned syncLevel)
     }
 }
 
+void
+Dht::searchNodeListenDeadline(const Sp<Search>& search, SearchNode& source, const Sp<Query>& query)
+{
+    auto found = source.listenStatus.find(query);
+    if (found == source.listenStatus.end() || found->second.initialized || found->second.syncDeadline)
+        return;
+    std::weak_ptr<Search> pending = search;
+    found->second.syncDeadline
+        = scheduler.add(scheduler.time() + std::chrono::seconds(30),
+                        [this, pending, query, node = source.node, registration = found->second.registration] {
+                            if (auto search = pending.lock()) {
+                                if (auto source = search->getNode(node)) {
+                                    auto status = source->listenStatus.find(query);
+                                    if (status != source->listenStatus.end()
+                                        && status->second.registration == registration && !status->second.initialized) {
+                                        search->cache.onReady(query, false);
+                                        source->listenStatus.erase(query);
+                                        scheduler.edit(search->nextSearchStep, scheduler.time());
+                                    }
+                                }
+                            }
+                        });
+}
+
 void
 Dht::searchSynchedNodeListen(const Sp<Search>& sr, SearchNode& n)
 {
@@ -664,7 +689,11 @@ Dht::searchSynchedNodeListen(const Sp<Search>& sr, SearchNode& n)
                     if (auto sn = sr->getNode(node)) {
                         if (not answer.ntoken.empty())
                             sn->refreshToken(answer.ntoken);
-                        sn->onValues(query, std::move(answer), types, scheduler);
+                        if (!sn->onValues(query, std::move(answer), types, scheduler)) {
+                            sr->cache.onReady(query, false);
+                            sn->listenStatus.erase(query);
+                        } else
+                            searchNodeListenDeadline(sr, *sn, query);
                     }
                 }
             });
@@ -677,6 +706,10 @@ Dht::searchSynchedNodeListen(const Sp<Search>& sr, SearchNode& n)
         //     logger_->d(sr->id, n.node->id, "[search %s] [node %s] Sending 'listen'",
         //        sr->id.toString().c_str(), n.node->toString().c_str());
 
+        if (r != n.listenStatus.end() && !n.node->getSocket(r->second.socketId)) {
+            n.listenStatus.erase(r);
+            r = n.listenStatus.end();
+        }
         if (r == n.listenStatus.end()) {
             r = n.listenStatus
                     .emplace(std::piecewise_construct,
@@ -690,36 +723,57 @@ Dht::searchSynchedNodeListen(const Sp<Search>& sr, SearchNode& n)
                     }
                 }
             });
-        } else if (not n.node->getSocket(r->second.socketId)) {
-            r->second.socketId = openListenSocket();
         }
+        if (!r->second.registration)
+            r->second.registration = std::uniform_int_distribution<uint64_t>(1)(rd);
+        searchNodeListenDeadline(sr, n, query);
         auto new_req = network_engine.sendListen(
             n.node,
             sr->id,
             *query,
             n.token,
             r->second.socketId,
-            [this, ws, query](const net::Request& req, net::RequestAnswer&& answer) mutable {
+            [this, ws, query, socketId = r->second.socketId](const net::Request& req,
+                                                             net::RequestAnswer&& answer) mutable {
                 /* on done */
                 if (auto sr = ws.lock()) {
-                    scheduler.edit(sr->nextSearchStep, scheduler.time());
                     if (auto sn = sr->getNode(req.node)) {
+                        auto status = sn->listenStatus.find(query);
+                        if (status == sn->listenStatus.end() || status->second.socketId != socketId)
+                            return;
+                        scheduler.edit(sr->nextSearchStep, scheduler.time());
+                        onListenDone(req.node, answer, sr);
                         auto job = scheduler.add(sn->getListenTime(query, getListenExpiration()),
                                                  std::bind(&Dht::searchStep, this, ws));
-                        sn->onListenSynced(query, true, std::move(job));
+                        sn->onListenSynced(query, status->second.initialized, std::move(job));
+                        if ((answer.listenSync && answer.ntoken.empty()) || !answer.fields.empty()
+                            || !answer.values.empty()
+                            || !sn->confirmListen(query, answer.listenSync, types, scheduler)) {
+                            sr->cache.onReady(query, false);
+                            status = sn->listenStatus.find(query);
+                            if (status != sn->listenStatus.end())
+                                sn->listenStatus.erase(status);
+                        } else
+                            searchNodeListenDeadline(sr, *sn, query);
                     }
-                    onListenDone(req.node, answer, sr);
                 }
             },
-            [this, ws, query](const net::Request& req, bool over) mutable {
+            [this, ws, query, socketId = r->second.socketId](const net::Request& req, bool over) mutable {
                 /* on request expired */
                 if (auto sr = ws.lock()) {
-                    scheduler.edit(sr->nextSearchStep, scheduler.time());
-                    if (over)
-                        if (auto sn = sr->getNode(req.node))
+                    if (auto sn = sr->getNode(req.node)) {
+                        auto status = sn->listenStatus.find(query);
+                        if (status == sn->listenStatus.end() || status->second.socketId != socketId)
+                            return;
+                        scheduler.edit(sr->nextSearchStep, scheduler.time());
+                        if (over) {
+                            sr->cache.onReady(query, false);
                             sn->listenStatus.erase(query);
+                        }
+                    }
                 }
-            });
+            },
+            net::ListenSync {r->second.registration, 0, 0, 0});
         // Here the request may have failed and the CachedListenStatus removed
         r = n.listenStatus.find(query);
         if (r != n.listenStatus.end()) {
@@ -751,6 +805,9 @@ Dht::searchStep(std::weak_ptr<Search> ws)
     if (sr->refill_time + Node::NODE_EXPIRE_TIME < now and sr->nodes.size() - sr->getNumberOfBadNodes() < SEARCH_NODES)
         refill(*sr);
 
+    if (sr->nodes.empty())
+        sr->cache.onReady();
+
     /* Check if the first TARGET_NODES (8) live nodes have replied. */
     if (sr->isSynced(now)) {
         if (not(sr->callbacks.empty() and sr->announce.empty())) {
@@ -798,6 +855,7 @@ Dht::searchStep(std::weak_ptr<Search> ws)
 
         // Announce requests
         searchSendAnnounceValue(sr);
+        sr->cache.onSearchReady();
 
         if (sr->callbacks.empty() && sr->announce.empty() && sr->listeners.empty())
             sr->setDone();
@@ -954,30 +1012,47 @@ Dht::announce(
 }
 
 size_t
-Dht::listenTo(const InfoHash& id, sa_family_t af, ValueCallback cb, Value::Filter f, const Sp<Query>& q)
+Dht::listenTo(
+    const InfoHash& id, sa_family_t af, ValueCallback cb, Value::Filter f, const Sp<Query>& q, DoneCallbackSimple ready)
 {
-    if (!isRunning(af))
+    if (!isRunning(af)) {
+        if (ready)
+            ready(true);
         return 0;
+    }
     // logger__ERR("[search %s IPv%c] Search_time is now in %lfs", sr->id.toString().c_str(), (sr->af == AF_INET) ? '4'
     // : '6', print_duration(tm-clock::now()));
 
     // logger__WARN("listenTo %s", id.toString().c_str());
     auto& srs = searches(af);
     auto srp = srs.find(id);
-    Sp<Search> sr = (srp == srs.end()) ? search(id, af) : srp->second;
-    if (!sr)
-        throw DhtException("Unable to create search");
+    Sp<Search> sr = (srp == srs.end() || srp->second->expired) ? search(id, af) : srp->second;
+    if (!sr) {
+        if (ready)
+            ready(false);
+        return 0;
+    }
     if (logger_)
         logger_->warn("[search {} IPv{}] Listen", id.to_view(), (af == AF_INET) ? '4' : '6');
-    return sr->listen(cb, std::move(f), q, scheduler);
+    sr->done = false;
+    scheduler.edit(sr->nextSearchStep, scheduler.time());
+    return sr->listen(cb, std::move(f), q, scheduler, std::move(ready));
 }
 
 size_t
 Dht::listen(const InfoHash& id, ValueCallback cb, Value::Filter f, Where where)
+{
+    return listen(id, std::move(cb), std::move(f), std::move(where), {});
+}
+
+size_t
+Dht::listen(const InfoHash& id, ValueCallback cb, Value::Filter f, Where where, DoneCallbackSimple ready)
 {
     if (not id) {
         if (logger_)
             logger_->warn("Listen called with invalid key");
+        if (ready)
+            ready(false);
         return 0;
     }
     scheduler.syncTime();
@@ -994,18 +1069,60 @@ Dht::listen(const InfoHash& id, ValueCallback cb, Value::Filter f, Where where)
     size_t tokenlocal = 0;
     if (st != store.end()) {
         tokenlocal = st->second.listen(gcb, filter, query);
-        if (tokenlocal == 0)
+        if (tokenlocal == 0) {
+            if (ready)
+                ready(false);
             return 0;
+        }
     }
 
-    auto token4 = Dht::listenTo(id, AF_INET, gcb, filter, query);
-    auto token6 = token4 == 0 ? 0 : Dht::listenTo(id, AF_INET6, gcb, filter, query);
-    if (token6 == 0 && st != store.end()) {
-        st->second.cancelListen(tokenlocal);
+    struct InitialReadiness
+    {
+        DoneCallbackSimple callback;
+        unsigned remaining {2};
+        bool registered {false};
+        bool failed {false};
+        void notify(bool ok)
+        {
+            failed = failed || !ok;
+            if (remaining)
+                --remaining;
+            finish();
+        }
+        void finish()
+        {
+            if (registered && (failed || remaining == 0) && callback) {
+                auto ready = std::move(callback);
+                ready(!failed);
+            }
+        }
+    };
+    auto initial = std::make_shared<InitialReadiness>();
+    initial->callback = std::move(ready);
+    auto onReady = [initial](bool ok) {
+        initial->notify(ok);
+    };
+    auto token4 = Dht::listenTo(id, AF_INET, gcb, filter, query, onReady);
+    auto token6 = token4 == 0 && isRunning(AF_INET) ? 0 : Dht::listenTo(id, AF_INET6, gcb, filter, query, onReady);
+    initial->registered = true;
+    if ((isRunning(AF_INET) && token4 == 0) || (isRunning(AF_INET6) && token6 == 0)) {
+        if (st != store.end())
+            st->second.cancelListen(tokenlocal);
+        for (auto family : {AF_INET, AF_INET6}) {
+            auto listenerToken = family == AF_INET ? token4 : token6;
+            if (!listenerToken)
+                continue;
+            auto& familySearches = searches(family);
+            auto search = familySearches.find(id);
+            if (search != familySearches.end())
+                search->second->cancelListen(listenerToken, scheduler);
+        }
+        initial->notify(false);
         return 0;
     }
 
     listeners.emplace(token, std::make_tuple(tokenlocal, token4, token6));
+    initial->finish();
     return token;
 }
 
@@ -1414,7 +1531,8 @@ Dht::storageChanged(const InfoHash& id, Storage& st, const Sp<Value>& v, bool ne
                                             {},
                                             std::move(vals),
                                             l.second.query,
-                                            l.second.version);
+                                            l.second.version,
+                                            l.second.stream ? &l.second.stream->sync : nullptr);
             }
         }
     }
@@ -1487,35 +1605,56 @@ Dht::storageStore(const InfoHash& id,
     return std::get<0>(store);
 }
 
-void
-Dht::storageAddListener(const InfoHash& id, const Sp<Node>& node, size_t socket_id, Query&& query, int version)
+std::optional<net::ListenSync>
+Dht::storageAddListener(const InfoHash& id,
+                        const Sp<Node>& node,
+                        size_t socket_id,
+                        Query&& query,
+                        int version,
+                        const std::optional<net::ListenSync>& sync)
 {
     const auto& now = scheduler.time();
     auto st = store.find(id);
     if (st == store.end()) {
         if (store.size() >= max_store_keys)
-            return;
+            throw net::DhtProtocolException(net::DhtProtocolException::NON_AUTHORITATIVE_INFORMATION);
         st = store.emplace(id, now).first;
     }
     auto& node_listeners = st->second.listeners[node];
     auto l = node_listeners.find(socket_id);
+    if (l != node_listeners.end() && sync
+        && (!l->second.stream || l->second.stream->sync.registration != sync->registration)) {
+        node_listeners.erase(l);
+        l = node_listeners.end();
+    }
     if (l == node_listeners.end()) {
-        auto vals = st->second.get(query.where.getFilter());
+        l = node_listeners.emplace(socket_id, Listener {now, std::move(query), version}).first;
+        auto vals = st->second.get(l->second.query.where.getFilter());
+        if (version == 2 && sync) {
+            l->second.stream = std::make_shared<Listener::Stream>();
+            l->second.stream->sync = {sync->registration,
+                                      std::uniform_int_distribution<uint64_t>(1)(rd),
+                                      0,
+                                      network_engine.getUpdateBatchCount(vals)};
+        }
         if (not vals.empty()) {
-            network_engine.tellListener(node,
-                                        socket_id,
-                                        id,
-                                        WANT4 | WANT6,
-                                        makeToken(node->getAddr(), false),
-                                        dht4.buckets.findClosestNodes(id, now, TARGET_NODES),
-                                        dht6.buckets.findClosestNodes(id, now, TARGET_NODES),
-                                        std::move(vals),
-                                        query,
-                                        version);
+            auto requests = network_engine.tellListener(node,
+                                                        socket_id,
+                                                        id,
+                                                        WANT4 | WANT6,
+                                                        makeToken(node->getAddr(), false),
+                                                        dht4.buckets.findClosestNodes(id, now, TARGET_NODES),
+                                                        dht6.buckets.findClosestNodes(id, now, TARGET_NODES),
+                                                        std::move(vals),
+                                                        l->second.query,
+                                                        version,
+                                                        l->second.stream ? &l->second.stream->sync : nullptr);
+            if (l->second.stream)
+                l->second.stream->initialRequests = std::move(requests);
         }
-        node_listeners.emplace(socket_id, Listener {now, std::forward<Query>(query), version});
     } else
         l->second.refresh(now, std::forward<Query>(query));
+    return l->second.stream ? std::optional<net::ListenSync>(l->second.stream->sync) : std::nullopt;
 }
 
 void
@@ -1562,7 +1701,13 @@ Dht::storageRemoved(const InfoHash& id, Storage& st, const std::vector<Sp<Value>
                                   id.to_view(),
                                   node_listeners.first->toString());
                 Blob ntoken = makeToken(node_listeners.first->getAddr(), false);
-                network_engine.tellListenerExpired(node_listeners.first, l.first, id, ntoken, ids, l.second.version);
+                network_engine.tellListenerExpired(node_listeners.first,
+                                                   l.first,
+                                                   id,
+                                                   ntoken,
+                                                   ids,
+                                                   l.second.version,
+                                                   l.second.stream ? &l.second.stream->sync : nullptr);
             }
         }
     }
@@ -2015,7 +2160,7 @@ Dht::Dht(std::unique_ptr<net::DatagramSocket>&& sock,
                      std::bind(&Dht::onPing, this, _1),
                      std::bind(&Dht::onFindNode, this, _1, _2, _3),
                      std::bind(&Dht::onGetValues, this, _1, _2, _3, _4),
-                     std::bind(&Dht::onListen, this, _1, _2, _3, _4, _5, _6),
+                     std::bind(&Dht::onListen, this, _1, _2, _3, _4, _5, _6, _7),
                      std::bind(&Dht::onAnnounce, this, _1, _2, _3, _4, _5),
                      std::bind(&Dht::onRefresh, this, _1, _2, _3, _4))
     , persistPath(config.persist_path)
@@ -2714,7 +2859,13 @@ Dht::onGetValuesDone(const Sp<Node>& node, net::RequestAnswer& a, Sp<Search>& sr
 }
 
 net::RequestAnswer
-Dht::onListen(Sp<Node> node, const InfoHash& hash, const Blob& token, size_t socket_id, const Query& query, int version)
+Dht::onListen(Sp<Node> node,
+              const InfoHash& hash,
+              const Blob& token,
+              size_t socket_id,
+              const Query& query,
+              int version,
+              const std::optional<net::ListenSync>& sync)
 {
     if (not hash) {
         if (logger_)
@@ -2728,9 +2879,11 @@ Dht::onListen(Sp<Node> node, const InfoHash& hash, const Blob& token, size_t soc
         throw net::DhtProtocolException {net::DhtProtocolException::UNAUTHORIZED,
                                          net::DhtProtocolException::LISTEN_WRONG_TOKEN};
     }
+    if (version == 2 && (!sync || sync->epoch || sync->sequence || sync->snapshotEnd))
+        throw net::DhtProtocolException(net::DhtProtocolException::NON_AUTHORITATIVE_INFORMATION);
     Query q = query;
-    storageAddListener(hash, node, socket_id, std::move(q), version);
     net::RequestAnswer answer {};
+    answer.listenSync = storageAddListener(hash, node, socket_id, std::move(q), version, sync);
     answer.ntoken = makeToken(node->getAddr(), false);
     return answer;
 }
@@ -2872,8 +3025,13 @@ Dht::storageRefresh(const InfoHash& id, Value::Id vid)
                                       id.toString(),
                                       node_listeners.first->toString());
                     Blob ntoken = makeToken(node_listeners.first->getAddr(), false);
-                    network_engine
-                        .tellListenerRefreshed(node_listeners.first, l.first, id, ntoken, ids, l.second.version);
+                    network_engine.tellListenerRefreshed(node_listeners.first,
+                                                         l.first,
+                                                         id,
+                                                         ntoken,
+                                                         ids,
+                                                         l.second.version,
+                                                         l.second.stream ? &l.second.stream->sync : nullptr);
                 }
             }
         }

src/dht_proxy_client.cpp

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: MIT
 
 #include "dht_proxy_client.h"
+#include <charconv>
 #include "dhtrunner.h"
 #include "op_cache.h"
 #include "utils.h"
@@ -21,6 +22,8 @@ struct DhtProxyClient::OperationState
 {
     std::atomic_bool ok {true};
     std::atomic_bool stop {false};
+    std::atomic_uint64_t generation {0};
+    std::atomic_bool inFlight {false};
 };
 
 struct DhtProxyClient::Listener
@@ -35,6 +38,17 @@ struct DhtProxyClient::Listener
     CacheValueCallback cb;
     Sp<OperationState> opstate;
     std::shared_ptr<http::Request> request;
+    std::shared_ptr<asio::steady_timer> requestDeadline;
+    bool needsRecovery {true};
+    std::string syncEpoch;
+    uint64_t syncRevision {0};
+    std::string observedEpoch;
+    uint64_t observedRevision {0};
+    std::set<std::string> retiredEpochs;
+    std::string requestToken;
+    uint64_t pushGeneration {0};
+    std::chrono::seconds retryDelay {1};
+    DoneCallbackSimple ready;
 #ifdef OPENDHT_PUSH_NOTIFICATIONS
     std::unique_ptr<asio::system_timer> refreshSubscriberTimer;
 #endif
@@ -55,9 +69,17 @@ struct PermanentPut
 
 struct DhtProxyClient::ProxySearch
 {
+    struct Readiness
+    {
+        std::atomic_bool pending {true};
+        DoneCallbackSimple callback;
+        size_t token {0};
+    };
+
     SearchCache ops {false};
     std::unique_ptr<asio::steady_timer> opExpirationTimer;
     std::map<size_t, Listener> listeners {};
+    std::map<size_t, std::shared_ptr<Readiness>> readiness;
     std::map<Value::Id, PermanentPut> puts {};
     std::set<Sp<Value>> pendingPuts {};
 };
@@ -79,6 +101,7 @@ struct LineSplit
         return false;
     }
     const std::string& line() const { return line_; }
+    bool empty() const { return buf_.empty(); }
 
 private:
     std::deque<char> buf_ {};
@@ -202,18 +225,17 @@ DhtProxyClient::stop()
         }
         if (resolver)
             resolver->cancel();
-        cancelAllListeners();
         if (infoState_)
             infoState_->cancel = true;
-        {
-            std::lock_guard lock(requestLock_);
-            for (auto& request : requests_)
-                request.second->cancel();
-        }
         if (not httpContext_.stopped())
             httpContext_.stop();
         if (httpClientThread_.joinable())
             httpClientThread_.join();
+        cancelAllListeners();
+        for (auto& request : requests_) {
+            request.second->cancel();
+            request.second->terminate(asio::error::operation_aborted);
+        }
         requests_.clear();
     }
 }
@@ -241,19 +263,24 @@ DhtProxyClient::getLocalById(const InfoHash& k, Value::Id id) const
 void
 DhtProxyClient::cancelAllListeners()
 {
-    std::lock_guard lock(searchLock_);
+    std::map<InfoHash, ProxySearch> canceled;
+    {
+        std::lock_guard lock(searchLock_);
+        canceled.swap(searches_);
+    }
     if (logger_)
-        logger_->debug("[proxy:client] [listeners] [{} searches] cancel all", searches_.size());
-    for (auto& s : searches_) {
-        s.second.ops.cancelAll([&](size_t token) {
-            auto l = s.second.listeners.find(token);
-            if (l == s.second.listeners.end())
-                return;
-            l->second.opstate->stop.store(true);
-            l->second.request->cancel();
-            // implicit request.reset()
-            s.second.listeners.erase(token);
-        });
+        logger_->debug("[proxy:client] [listeners] [{} searches] cancel all", canceled.size());
+    for (auto& s : canceled) {
+        for (auto& listener : s.second.listeners) {
+            listener.second.opstate->stop.store(true);
+            if (listener.second.request)
+                listener.second.request->cancel();
+        }
+        s.second.ops.onReady(false);
+        s.second.ops.cancelAll([&](size_t token) { s.second.listeners.erase(token); });
+        for (const auto& readiness : s.second.readiness)
+            if (readiness.second->pending.exchange(false))
+                readiness.second->callback(false);
     }
 }
 
@@ -305,6 +332,12 @@ DhtProxyClient::periodic(const uint8_t*, size_t, SockAddr, const time_point& /*n
     for (auto& callback : callbacks)
         callback();
     callbacks.clear();
+    {
+        std::lock_guard lock(lockCallbacks_);
+        callbacks = std::move(readinessCallbacks_);
+    }
+    for (auto& callback : callbacks)
+        callback();
     return time_point::max();
 }
 
@@ -833,17 +866,52 @@ DhtProxyClient::getPublicAddress(sa_family_t family)
 
 size_t
 DhtProxyClient::listen(const InfoHash& key, ValueCallback cb, Value::Filter filter, Where where)
+{
+    return listen(key, std::move(cb), std::move(filter), std::move(where), {});
+}
+
+size_t
+DhtProxyClient::listen(const InfoHash& key, ValueCallback cb, Value::Filter filter, Where where, DoneCallbackSimple ready)
 {
     if (logger_)
         logger_->debug("[proxy:client] [listen] [search {}]", key.to_view());
-    if (isDestroying_)
+    if (isDestroying_) {
+        if (ready)
+            ready(false);
         return 0;
+    }
 
     std::lock_guard lock(searchLock_);
     auto& search = searches_[key];
     auto query = std::make_shared<Query>(Select {}, std::move(where));
-    return search.ops
-        .listen(cb, query, std::move(filter), [this, key](Sp<Query>, ValueCallback cb, SyncCallback) -> size_t {
+    std::shared_ptr<ProxySearch::Readiness> readiness;
+    DoneCallbackSimple notify;
+    if (ready) {
+        readiness = std::make_shared<ProxySearch::Readiness>();
+        readiness->callback = std::move(ready);
+        notify = [this, key, readiness](bool ok) {
+            {
+                std::lock_guard callbackLock(lockCallbacks_);
+                readinessCallbacks_.emplace_back([this, key, readiness, ok] {
+                    {
+                        std::lock_guard searchLock(searchLock_);
+                        if (!readiness->pending.exchange(false))
+                            return;
+                        auto search = searches_.find(key);
+                        if (search != searches_.end())
+                            search->second.readiness.erase(readiness->token);
+                    }
+                    readiness->callback(ok);
+                });
+            }
+            loopSignal_();
+        };
+    }
+    auto token = search.ops.listen(
+        cb,
+        query,
+        std::move(filter),
+        [this, key](Sp<Query> query, ValueCallback cb, SyncCallback synced) -> size_t {
             // Find search
             auto search = searches_.find(key);
             if (search == searches_.end()) {
@@ -871,6 +939,16 @@ DhtProxyClient::listen(const InfoHash& key, ValueCallback cb, Value::Filter filt
             // Add cache callback
             auto opstate = std::make_shared<OperationState>();
             l->second.opstate = opstate;
+            synced(ListenSyncStatus::ADDED);
+            l->second.ready = [this, key, query, synced, synchronized = std::make_shared<bool>(false)](bool ok) {
+                if (*synchronized != ok) {
+                    *synchronized = ok;
+                    synced(ok ? ListenSyncStatus::SYNCED : ListenSyncStatus::UNSYNCED);
+                }
+                auto search = searches_.find(key);
+                if (search != searches_.end())
+                    search->second.ops.onReady(query, ok);
+            };
             l->second.cb = [this, key, token, opstate](const std::vector<Sp<Value>>& values,
                                                        bool expired,
                                                        system_clock::time_point t) {
@@ -911,9 +989,17 @@ DhtProxyClient::listen(const InfoHash& key, ValueCallback cb, Value::Filter filt
                 header.method(restinio::http_method_subscribe());
                 header.request_target(fmt::format("/key/{}", key.to_view()));
             }
-            sendListen(header, l->second.cb, opstate, l->second, method);
+            sendListen(header, l->second.cb, opstate, l->second, method, key, token);
             return token;
-        });
+        },
+        notify);
+    if (readiness) {
+        readiness->token = token;
+        search.readiness.emplace(token, readiness);
+        if (!token)
+            notify(false);
+    }
+    return token;
 }
 
 void
@@ -957,6 +1043,11 @@ DhtProxyClient::cancelListen(const InfoHash& key, size_t gtoken)
         return false;
     auto& ops = it->second.ops;
     bool canceled = ops.cancelListen(gtoken, std::chrono::steady_clock::now());
+    auto readiness = it->second.readiness.find(gtoken);
+    if (readiness != it->second.readiness.end()) {
+        readiness->second->pending = false;
+        it->second.readiness.erase(readiness);
+    }
 
     // define real cancel listen only once
     if (not it->second.opExpirationTimer)
@@ -1053,77 +1144,313 @@ DhtProxyClient::handleExpireListener(const asio::error_code& ec, const InfoHash&
     }
 }
 
+bool
+DhtProxyClient::reconcileListenerSnapshot(Listener& listener,
+                                          const std::vector<Sp<Value>>& values,
+                                          const Json::Value& marker)
+{
+    if (!marker.isObject() || !marker["epoch"].isString() || marker["epoch"].asString().empty()
+        || !marker["revision"].isUInt64() || !marker["snapshot"].isBool() || !marker["complete"].isBool()
+        || !marker["complete"].asBool())
+        return false;
+    auto epoch = marker["epoch"].asString();
+    auto revision = marker["revision"].asUInt64();
+    if (listener.retiredEpochs.count(epoch) || (listener.syncEpoch == epoch && listener.syncRevision > revision)
+        || (listener.observedEpoch == epoch && listener.observedRevision > revision))
+        return false;
+    bool full = marker["snapshot"].asBool();
+    if (!full
+        && (listener.needsRecovery || epoch != listener.syncEpoch || revision != listener.syncRevision
+            || !values.empty()))
+        return false;
+    if (full) {
+        std::set<Value::Id> ids;
+        for (const auto& value : values)
+            ids.insert(value->id);
+        std::vector<Sp<Value>> expired;
+        for (const auto& value : listener.cache.getValues())
+            if (!ids.count(value->id))
+                expired.emplace_back(value);
+        if (!listener.cache.onValue(values, false, system_clock::time_point::min())
+            || (!expired.empty() && !listener.cache.onValue(expired, true, system_clock::time_point::max()))) {
+            listener.opstate->stop = true;
+            return false;
+        }
+    }
+    if (!listener.syncEpoch.empty() && listener.syncEpoch != epoch)
+        listener.retiredEpochs.insert(listener.syncEpoch);
+    while (listener.retiredEpochs.size() > 8)
+        listener.retiredEpochs.erase(listener.retiredEpochs.begin());
+    listener.syncEpoch = epoch;
+    listener.syncRevision = revision;
+    listener.observedEpoch = epoch;
+    listener.observedRevision = revision;
+    listener.needsRecovery = false;
+    listener.retryDelay = std::chrono::seconds(1);
+    return true;
+}
+
+void
+DhtProxyClient::scheduleListenRetry([[maybe_unused]] const InfoHash& key,
+                                    [[maybe_unused]] size_t token,
+                                    [[maybe_unused]] Listener& listener)
+{
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    if (!listener.refreshSubscriberTimer)
+        listener.refreshSubscriberTimer = std::make_unique<asio::system_timer>(httpContext_);
+    listener.refreshSubscriberTimer->expires_after(listener.retryDelay);
+    listener.retryDelay = std::min(listener.retryDelay * 2, std::chrono::seconds(60));
+    listener.refreshSubscriberTimer->async_wait(
+        std::bind(&DhtProxyClient::handleResubscribe, this, std::placeholders::_1, key, token, listener.opstate));
+#endif
+}
+
 void
 DhtProxyClient::sendListen(const restinio::http_request_header_t& header,
                            const CacheValueCallback& cb,
                            const Sp<OperationState>& opstate,
                            Listener& listener,
-                           ListenMethod method)
+                           ListenMethod method,
+                           const InfoHash& key,
+                           size_t token)
 {
     if (logger_)
         logger_->error("[proxy:client] [listen] sendListen: {}", (int) method);
     try {
+        auto generation = ++opstate->generation;
+        bool syncRequest = method != ListenMethod::LISTEN;
+        opstate->inFlight = syncRequest;
+        if (syncRequest)
+            opstate->ok = false;
         auto request = buildRequest();
         listener.request = request;
+        auto deadline = std::make_shared<asio::steady_timer>(httpContext_);
+        listener.requestDeadline = deadline;
+        if (syncRequest) {
+            deadline->expires_after(subscriptionRequestTimeout_);
+            deadline->async_wait([weak = std::weak_ptr<http::Request>(request)](const asio::error_code& ec) {
+                if (!ec)
+                    if (auto request = weak.lock()) {
+                        request->cancel();
+                        request->terminate(asio::error::timed_out);
+                    }
+            });
+        }
         auto reqid = request->id();
-        request->set_header(header);
-        setHeaderFields(*request);
+        auto requestHeader = header;
         if (method == ListenMethod::LISTEN)
+            requestHeader.request_target(std::string(header.request_target()) + "?sync=1");
+        request->set_header(requestHeader);
+        setHeaderFields(*request);
+        if (method == ListenMethod::LISTEN) {
             request->set_connection_type(restinio::http_connection_header_t::keep_alive);
+        }
 #ifdef OPENDHT_PUSH_NOTIFICATIONS
         std::string body;
-        if (method != ListenMethod::LISTEN)
-            body = fillBody(method == ListenMethod::RESUBSCRIBE);
-        request->set_body(body);
-#endif
-        auto rxBuf = std::make_shared<LineSplit>();
-        request->add_on_body_callback([this, reqid, opstate, rxBuf, cb](const char* at, size_t length) {
-            try {
-                auto& b = *rxBuf;
-                b.append(at, length);
-
-                // one value per body line
-                while (b.getLine('\n') and !opstate->stop) {
-                    std::string err;
-                    Json::Value json;
-                    const auto& line = b.line();
-                    if (!jsonReader_->parse(line.data(), line.data() + line.size(), &json, &err)) {
-                        opstate->ok.store(false);
-                        return;
-                    }
-                    if (json.size() == 0) { // it's the end
-                        break;
-                    }
-
-                    auto value = std::make_shared<Value>(json);
-                    if (cb) {
-                        auto expired = json.get("expired", Json::Value(false)).asBool();
-                        {
-                            std::lock_guard lock(lockCallbacks_);
-                            callbacks_.emplace_back([cb, value, opstate, expired]() {
-                                if (not opstate->stop.load()
-                                    and not cb({value}, expired, system_clock::time_point::min()))
-                                    opstate->stop.store(true);
-                            });
-                        }
-                        loopSignal_();
-                    }
-                }
-            } catch (const std::exception& e) {
-                if (logger_)
-                    logger_->error("[proxy:client] [listen] request #{} error in parsing: {}", reqid, e.what());
-                opstate->ok.store(false);
+        std::string requestToken;
+        if (syncRequest) {
+            Json::Value root;
+            getPushRequest(root);
+            requestToken = root["key"].asString();
+            if (listener.requestToken != requestToken) {
+                listener.requestToken = requestToken;
+                listener.observedEpoch.clear();
+                listener.observedRevision = 0;
+                listener.retiredEpochs.clear();
             }
-        });
-        request->add_on_done_callback([this, opstate, reqid](const http::Response& response) {
-            if (response.status_code != 200) {
-                if (logger_)
-                    logger_->error("[proxy:client] [listen] send request #{} failed with code={}",
-                                   reqid,
-                                   response.status_code);
-                opstate->ok.store(false);
-                if (not response.aborted and response.status_code == 0)
-                    opFailed();
+            root["sync"] = Json::Value(Json::objectValue);
+            if (!listener.needsRecovery && !listener.syncEpoch.empty()) {
+                root["sync"]["epoch"] = listener.syncEpoch;
+                root["sync"]["revision"] = Json::UInt64(listener.syncRevision);
+            }
+            body = Json::writeString(jsonBuilder_, root);
+        }
+        request->set_body(body);
+#else
+        std::string requestToken;
+#endif
+        auto pushGeneration = listener.pushGeneration;
+        auto rxBuf = std::make_shared<LineSplit>();
+        auto snapshot = std::make_shared<std::vector<Sp<Value>>>();
+        auto marker = std::make_shared<Json::Value>();
+        auto valid = std::make_shared<bool>(true);
+        request->add_on_body_callback(
+            [this, reqid, key, token, opstate, generation, rxBuf, cb, syncRequest, snapshot, marker, valid](
+                const char* at, size_t length) {
+                if (opstate->generation != generation || opstate->stop)
+                    return;
+                try {
+                    auto& b = *rxBuf;
+                    b.append(at, length);
+
+                    // one value per body line
+                    while (b.getLine('\n') and !opstate->stop) {
+                        std::string err;
+                        Json::Value json;
+                        const auto& line = b.line();
+                        if (!jsonReader_->parse(line.data(), line.data() + line.size(), &json, &err)) {
+                            *valid = false;
+                            return;
+                        }
+                        if (syncRequest) {
+                            if (!marker->isNull()) {
+                                *valid = false;
+                                continue;
+                            }
+                            if (json.isMember("sync")) {
+                                *marker = json["sync"];
+                            } else if (json.size() != 0) {
+                                snapshot->emplace_back(std::make_shared<Value>(json));
+                            } else {
+                                *valid = false;
+                            }
+                            continue;
+                        }
+                        if (json.isMember("sync")) {
+                            if (!json["sync"].isObject() || !json["sync"]["complete"].isBool()) {
+                                *valid = false;
+                                return;
+                            }
+                            std::lock_guard callbackLock(lockCallbacks_);
+                            callbacks_.emplace_back(
+                                [this, key, token, opstate, generation, ok = json["sync"]["complete"].asBool()] {
+                                    std::lock_guard searchLock(searchLock_);
+                                    if (opstate->generation != generation || opstate->stop)
+                                        return;
+                                    auto search = searches_.find(key);
+                                    if (search == searches_.end())
+                                        return;
+                                    auto found = search->second.listeners.find(token);
+                                    if (found != search->second.listeners.end() && found->second.ready) {
+                                        auto ready = found->second.ready;
+                                        ready(ok);
+                                    }
+                                });
+                            loopSignal_();
+                            continue;
+                        }
+                        if (json.size() == 0) { // it's the end
+                            break;
+                        }
+
+                        auto value = std::make_shared<Value>(json);
+                        if (cb) {
+                            auto expired = json.get("expired", Json::Value(false)).asBool();
+                            {
+                                std::lock_guard lock(lockCallbacks_);
+                                callbacks_.emplace_back([cb, value, opstate, generation, expired]() {
+                                    if (opstate->generation == generation && not opstate->stop.load()
+                                        and not cb({value}, expired, system_clock::time_point::min()))
+                                        opstate->stop.store(true);
+                                });
+                            }
+                            loopSignal_();
+                        }
+                    }
+                } catch (const std::exception& e) {
+                    if (logger_)
+                        logger_->error("[proxy:client] [listen] request #{} error in parsing: {}", reqid, e.what());
+                    *valid = false;
+                }
+            });
+        request->add_on_done_callback([this,
+                                       opstate,
+                                       generation,
+                                       reqid,
+                                       key,
+                                       token,
+                                       syncRequest,
+                                       rxBuf,
+                                       snapshot,
+                                       marker,
+                                       valid,
+                                       deadline,
+                                       requestToken,
+                                       pushGeneration](const http::Response& response) {
+            deadline->cancel();
+            bool current = opstate->generation == generation && !opstate->stop;
+            bool successful = response.status_code == 200 && !response.aborted && *valid
+                              && (!syncRequest || rxBuf->empty());
+            if (current && !isDestroying_) {
+                {
+                    std::lock_guard callbackLock(lockCallbacks_);
+                    callbacks_.emplace_back([this,
+                                             key,
+                                             token,
+                                             opstate,
+                                             generation,
+                                             successful,
+                                             syncRequest,
+                                             transportFailed = !response.aborted && response.status_code == 0,
+                                             snapshot,
+                                             marker,
+                                             requestToken,
+                                             pushGeneration]() {
+                        std::lock_guard searchLock(searchLock_);
+                        if (opstate->generation != generation || opstate->stop)
+                            return;
+                        auto search = searches_.find(key);
+                        if (search == searches_.end())
+                            return;
+                        auto found = search->second.listeners.find(token);
+                        if (found == search->second.listeners.end() || found->second.opstate != opstate)
+                            return;
+                        auto& listener = found->second;
+                        if (!successful) {
+                            opstate->ok = false;
+                            if (transportFailed)
+                                opFailed();
+                        }
+                        opstate->inFlight = false;
+                        if (!syncRequest) {
+                            if (!successful && listener.ready) {
+                                auto ready = listener.ready;
+                                ready(false);
+                            }
+                            return;
+                        }
+                        auto observedEpoch = listener.observedEpoch;
+                        if (successful && reconcileListenerSnapshot(listener, *snapshot, *marker)) {
+                            bool validateEpoch = listener.pushGeneration != pushGeneration && !observedEpoch.empty()
+                                                 && observedEpoch != listener.syncEpoch;
+                            opstate->ok = true;
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+                            if (listener.refreshSubscriberTimer) {
+                                listener.refreshSubscriberTimer->expires_at(system_clock::now() + proxy::OP_TIMEOUT
+                                                                            - proxy::OP_MARGIN);
+                                listener.refreshSubscriberTimer->async_wait(std::bind(&DhtProxyClient::handleResubscribe,
+                                                                                      this,
+                                                                                      std::placeholders::_1,
+                                                                                      key,
+                                                                                      token,
+                                                                                      opstate));
+                            }
+#endif
+                            if (listener.ready) {
+                                auto ready = listener.ready;
+                                ready(true);
+                            }
+                            bool changedToken;
+                            {
+                                std::lock_guard infoLock(lockCurrentProxyInfos_);
+                                changedToken = requestToken != deviceKey_;
+                            }
+                            if (changedToken || validateEpoch)
+                                resubscribe(key, token, listener);
+                            return;
+                        }
+                        if (opstate->stop)
+                            return;
+                        listener.needsRecovery = true;
+                        opstate->ok = false;
+                        if (listener.ready) {
+                            auto ready = listener.ready;
+                            ready(false);
+                        }
+                        scheduleListenRetry(key, token, listener);
+                    });
+                }
+                loopSignal_();
             }
             if (not isDestroying_) {
                 std::lock_guard l(requestLock_);
@@ -1134,14 +1461,21 @@ DhtProxyClient::sendListen(const restinio::http_request_header_t& header,
             std::lock_guard l(requestLock_);
             requests_[reqid] = request;
         }
-        request->add_on_status_callback([request, seconds = this->listenKeepIdle()](unsigned status_code) {
-            if (status_code == 200) {
-                // increase TCP_KEEPIDLE to save power
-                request->get_connection()->set_keepalive(seconds);
-            }
-        });
+        request->add_on_status_callback(
+            [weak = std::weak_ptr<http::Request>(request), seconds = this->listenKeepIdle()](unsigned status_code) {
+                if (status_code == 200) {
+                    // increase TCP_KEEPIDLE to save power
+                    if (auto request = weak.lock())
+                        if (auto connection = request->get_connection())
+                            connection->set_keepalive(seconds);
+                }
+            });
         request->send();
     } catch (const std::exception& e) {
+        opstate->ok = false;
+        opstate->inFlight = false;
+        listener.needsRecovery = true;
+        scheduleListenRetry(key, token, listener);
         if (logger_)
             logger_->error("[proxy:client] [listen] request failed: {}", e.what());
     }
@@ -1212,8 +1546,7 @@ DhtProxyClient::restartListeners(const asio::error_code& ec)
         // Connectivity changed, refresh all subscribe
         for (auto& search : searches_)
             for (auto& listener : search.second.listeners)
-                if (!listener.second.opstate->ok)
-                    resubscribe(search.first, listener.first, listener.second);
+                resubscribe(search.first, listener.first, listener.second);
         return;
     }
     if (logger_)
@@ -1223,7 +1556,8 @@ DhtProxyClient::restartListeners(const asio::error_code& ec)
             auto& listener = l.second;
             if (auto opstate = listener.opstate)
                 opstate->stop = true;
-            listener.request->cancel();
+            if (listener.request)
+                listener.request->cancel();
             listener.request.reset();
         }
     }
@@ -1239,16 +1573,34 @@ DhtProxyClient::restartListeners(const asio::error_code& ec)
             restinio::http_request_header_t header;
             header.method(restinio::http_method_get());
             header.request_target("/key/" + search.first.toString() + "/listen");
-            sendListen(header, cb, opstate, listener, ListenMethod::LISTEN);
+            sendListen(header, cb, opstate, listener, ListenMethod::LISTEN, search.first, l.first);
         }
     }
 }
 
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+static uint64_t
+parsePushNumber(const std::string& text)
+{
+    uint64_t number = 0;
+    auto result = std::from_chars(text.data(), text.data() + text.size(), number);
+    if (text.empty() || result.ec != std::errc {} || result.ptr != text.data() + text.size())
+        throw std::invalid_argument("Invalid push number");
+    return number;
+}
+#endif
+
 PushNotificationResult
 DhtProxyClient::pushNotificationReceived([[maybe_unused]] const std::map<std::string, std::string>& notification)
 {
     auto ret = PushNotificationResult::IgnoredNoOp;
 #ifdef OPENDHT_PUSH_NOTIFICATIONS
+    auto sessionId = notification.find("s");
+    if (sessionId != notification.end() and sessionId->second != pushSessionId_) {
+        if (logger_)
+            logger_->debug("[proxy:client] [push] ignoring push for other session");
+        return PushNotificationResult::IgnoredWrongSession;
+    }
     {
         // If a push notification is received, the proxy is up and running
         std::lock_guard l(lockCurrentProxyInfos_);
@@ -1260,12 +1612,6 @@ DhtProxyClient::pushNotificationReceived([[maybe_unused]] const std::map<std::st
     }
     auto launchLoop = false;
     try {
-        auto sessionId = notification.find("s");
-        if (sessionId != notification.end() and sessionId->second != pushSessionId_) {
-            if (logger_)
-                logger_->debug("[proxy:client] [push] ignoring push for other session");
-            return PushNotificationResult::IgnoredWrongSession;
-        }
         std::lock_guard lock(searchLock_);
         auto timeout = notification.find("timeout");
         if (timeout != notification.cend()) {
@@ -1292,14 +1638,25 @@ DhtProxyClient::pushNotificationReceived([[maybe_unused]] const std::map<std::st
                                                : PushNotificationResult::ListenRefresh;
             }
         } else {
-            system_clock::time_point sendTime = system_clock::time_point::min();
-            try {
-                sendTime = system_clock::time_point(std::chrono::milliseconds(std::stoull(notification.at("t"))));
-            } catch (...) {
-            }
             auto key = InfoHash(notification.at("key"));
             auto& search = searches_.at(key);
             auto expired = notification.find("exp");
+            auto expirationFlag = notification.find("expired");
+            bool isExpired = expired != notification.end()
+                             || (expirationFlag != notification.end() && expirationFlag->second == "true");
+            auto epoch = notification.find("epoch");
+            auto revision = notification.find("revision");
+            if (epoch == notification.end() || revision == notification.end() || epoch->second.empty())
+                throw std::invalid_argument("Incomplete push revision");
+            auto number = parsePushNumber(revision->second);
+            std::vector<Value::Id> ids;
+            if (isExpired) {
+                std::stringstream stream(expired != notification.end() ? expired->second : notification.at("ids"));
+                for (std::string id; std::getline(stream, id, ',');)
+                    ids.emplace_back(parsePushNumber(id));
+                if (ids.empty())
+                    throw std::invalid_argument("Empty expiration list");
+            }
             for (auto& list : search.listeners) {
                 if (list.second.opstate->stop)
                     continue;
@@ -1307,41 +1664,19 @@ DhtProxyClient::pushNotificationReceived([[maybe_unused]] const std::map<std::st
                     logger_->debug("[proxy:client] [push] [search {}] received", key);
                 auto token = list.first;
                 auto opstate = list.second.opstate;
-                if (expired == notification.end()) {
-                    auto cb = list.second.cb;
-                    auto oldValues = list.second.cache.getValues();
-                    auto receivedIds = std::make_shared<std::set<Value::Id>>();
-                    get(
-                        key,
-                        [cb, sendTime, receivedIds](const std::vector<Sp<Value>>& vals) {
-                            for (const auto& v : vals)
-                                receivedIds->insert(v->id);
-                            return cb(vals, false, sendTime);
-                        },
-                        [cb, oldValues, sendTime, receivedIds](bool ok) {
-                            if (!ok)
-                                return;
-                            // Decrement old values refcount to expire values not
-                            // present in the new list
-                            std::vector<Sp<Value>> toExpire;
-                            for (const auto& v : oldValues)
-                                if (receivedIds->find(v->id) == receivedIds->end())
-                                    toExpire.emplace_back(v);
-                            if (!toExpire.empty())
-                                cb(toExpire, true, sendTime);
-                        });
-                    ret = PushNotificationResult::Values;
-                } else {
-                    std::stringstream ss(expired->second);
-                    std::vector<Value::Id> ids;
-                    while (ss.good()) {
-                        std::string substr;
-                        getline(ss, substr, ',');
-                        ids.emplace_back(std::stoull(substr));
-                    }
+                auto& listener = list.second;
+                if (listener.retiredEpochs.count(epoch->second)
+                    || (epoch->second == listener.syncEpoch && number <= listener.syncRevision)
+                    || (epoch->second == listener.observedEpoch && number <= listener.observedRevision))
+                    continue;
+                ++listener.pushGeneration;
+                listener.observedEpoch = epoch->second;
+                listener.observedRevision = number;
+                if (isExpired && !listener.needsRecovery && epoch->second == listener.syncEpoch
+                    && number > listener.syncRevision && number - listener.syncRevision == 1) {
                     {
                         std::lock_guard lockCb(lockCallbacks_);
-                        callbacks_.emplace_back([this, key, token, opstate, ids, sendTime]() {
+                        callbacks_.emplace_back([this, key, token, opstate, ids, epoch = epoch->second, number]() {
                             if (opstate->stop)
                                 return;
                             std::lock_guard lock(searchLock_);
@@ -1349,15 +1684,27 @@ DhtProxyClient::pushNotificationReceived([[maybe_unused]] const std::map<std::st
                             if (s == searches_.end())
                                 return;
                             auto l = s->second.listeners.find(token);
-                            if (l == s->second.listeners.end())
+                            if (l == s->second.listeners.end() || l->second.opstate != opstate)
                                 return;
-                            if (not opstate->stop and not l->second.cache.onValuesExpired(ids, sendTime))
-                                opstate->stop = true;
+                            auto& listener = l->second;
+                            if (!listener.needsRecovery && listener.syncEpoch == epoch && number > listener.syncRevision
+                                && number - listener.syncRevision == 1) {
+                                if (!listener.cache.onValuesExpired(ids, system_clock::time_point::max()))
+                                    opstate->stop = true;
+                                listener.syncRevision = number;
+                            } else if (listener.syncEpoch != epoch || number > listener.syncRevision) {
+                                listener.needsRecovery = true;
+                                resubscribe(key, token, listener);
+                            }
                         });
                     }
                     launchLoop = true;
-                    ret = PushNotificationResult::ValuesExpired;
+                } else {
+                    if (epoch->second == listener.syncEpoch)
+                        listener.needsRecovery = true;
+                    resubscribe(key, token, listener);
                 }
+                ret = isExpired ? PushNotificationResult::ValuesExpired : PushNotificationResult::Values;
             }
         }
     } catch (const std::out_of_range& e) {
@@ -1385,8 +1732,19 @@ DhtProxyClient::resubscribe([[maybe_unused]] const InfoHash& key,
         logger_->debug("[proxy:client] [resubscribe] [search {}]", key);
 
     auto opstate = listener.opstate;
-    listener.request.reset(); // This will update ok to false
-    opstate->ok = true;
+    if (opstate->stop)
+        return;
+    listener.needsRecovery = listener.needsRecovery || (!opstate->ok && !opstate->inFlight);
+    if (opstate->inFlight && listener.request) {
+        if (listener.refreshSubscriberTimer && listener.refreshSubscriberTimer->expiry() <= system_clock::now())
+            scheduleListenRetry(key, token, listener);
+        return;
+    }
+    if (listener.request) {
+        ++opstate->generation;
+        listener.request->cancel();
+        listener.request.reset();
+    }
 
     restinio::http_request_header_t header;
     header.method(restinio::http_method_subscribe());
@@ -1398,7 +1756,7 @@ DhtProxyClient::resubscribe([[maybe_unused]] const InfoHash& key,
     listener.refreshSubscriberTimer->async_wait(
         std::bind(&DhtProxyClient::handleResubscribe, this, std::placeholders::_1, key, token, opstate));
     auto vcb = listener.cb;
-    sendListen(header, vcb, opstate, listener, ListenMethod::RESUBSCRIBE);
+    sendListen(header, vcb, opstate, listener, ListenMethod::SUBSCRIBE, key, token);
 #endif
 }
 
@@ -1406,6 +1764,7 @@ DhtProxyClient::resubscribe([[maybe_unused]] const InfoHash& key,
 void
 DhtProxyClient::getPushRequest(Json::Value& body) const
 {
+    std::lock_guard infoLock(lockCurrentProxyInfos_);
     body["key"] = deviceKey_;
     body["client_id"] = pushClientId_;
     body["session_id"] = pushSessionId_;
@@ -1413,23 +1772,6 @@ DhtProxyClient::getPushRequest(Json::Value& body) const
     body["platform"] = platform_;
 }
 
-std::string
-DhtProxyClient::fillBody(bool resubscribe)
-{
-    // Fill body with
-    // {
-    //   "key":"device_key",
-    // }
-    Json::Value body;
-    getPushRequest(body);
-    if (resubscribe) {
-        // This is the first listen, we want to retrieve previous values.
-        body["refresh"] = true;
-    }
-    auto content = Json::writeString(jsonBuilder_, body) + "\n";
-    std::replace(content.begin(), content.end(), '\n', ' ');
-    return content;
-}
 #endif // OPENDHT_PUSH_NOTIFICATIONS
 
 void
@@ -1446,6 +1788,10 @@ DhtProxyClient::setPushNotificationToken([[maybe_unused]] const std::string& tok
     }
     if (logger_)
         logger_->debug("[proxy:client] [push] token changed, resubscribing");
+    if (token.empty()) {
+        restartListeners({});
+        return;
+    }
     std::lock_guard lock(searchLock_);
     for (auto& search : searches_) {
         for (auto& listener : search.second.listeners) {

src/dht_proxy_server.cpp

@@ -467,25 +467,29 @@ DhtProxyServer::loadState(Is& is, size_t size)
                     for (auto& listeners : pushListener.second.listeners) {
                         for (auto& listener : listeners.second) {
                             // start listening
-                            listener.internalToken
-                                = dht_->listen(listeners.first,
-                                               [this,
-                                                infoHash = listeners.first,
-                                                pushToken = pushListener.first,
-                                                type = listener.type,
-                                                clientId = listener.clientId,
-                                                sessionCtx = listener.sessionCtx,
-                                                topic = listener.topic](const std::vector<Sp<Value>>& values,
-                                                                        bool expired) {
-                                                   return this->handlePushListen(infoHash,
-                                                                                 pushToken,
-                                                                                 type,
-                                                                                 clientId,
-                                                                                 sessionCtx,
-                                                                                 topic,
-                                                                                 values,
-                                                                                 expired);
-                                               });
+                            listener.internalToken = dht_->listen(
+                                listeners.first,
+                                [this,
+                                 infoHash = listeners.first,
+                                 pushToken = pushListener.first,
+                                 type = listener.type,
+                                 clientId = listener.clientId,
+                                 sessionCtx = listener.sessionCtx,
+                                 topic = listener.topic](const std::vector<Sp<Value>>& values, bool expired) {
+                                    return this->handlePushListen(infoHash,
+                                                                  pushToken,
+                                                                  type,
+                                                                  clientId,
+                                                                  sessionCtx,
+                                                                  topic,
+                                                                  values,
+                                                                  expired);
+                                },
+                                {},
+                                {},
+                                [this, sessionCtx = listener.sessionCtx](bool ok) {
+                                    onPushListenReady(sessionCtx, ok);
+                                });
                             // expire notify
                             listener.expireNotifyTimer = std::make_unique<asio::system_timer>(io_context(),
                                                                                               listener.expiration
@@ -560,6 +564,7 @@ DhtProxyServer::~DhtProxyServer()
                         l.expireNotifyTimer->cancel();
                     if (l.expireTimer)
                         l.expireTimer->cancel();
+                    onPushListenReady(l.sessionCtx, false);
                     dht_->cancelListen(ls.first, std::move(l.internalToken));
                 }
         }
@@ -834,16 +839,29 @@ DhtProxyServer::listen(restinio::request_handle_t request, restinio::router::rou
         auto& session = listeners_[request->connection_id()];
         session.hash = infoHash;
         session.response = response;
-        session.token = dht_->listen(infoHash, [this, response](const std::vector<Sp<Value>>& values, bool expired) {
-            for (const auto& value : values) {
-                auto jsonVal = value->toJson();
-                if (expired)
-                    jsonVal["expired"] = true;
-                response->append_chunk(Json::writeString(jsonBuilder_, jsonVal) + "\n");
-            }
-            response->flush();
-            return true;
-        });
+        bool notifyReady = request->header().query() == "sync=1";
+        session.token = dht_->listen(
+            infoHash,
+            [this, response](const std::vector<Sp<Value>>& values, bool expired) {
+                for (const auto& value : values) {
+                    auto jsonVal = value->toJson();
+                    if (expired)
+                        jsonVal["expired"] = true;
+                    response->append_chunk(Json::writeString(jsonBuilder_, jsonVal) + "\n");
+                }
+                response->flush();
+                return true;
+            },
+            {},
+            {},
+            [this, response, notifyReady](bool ok) {
+                if (notifyReady) {
+                    Json::Value marker;
+                    marker["sync"]["complete"] = ok;
+                    response->append_chunk(Json::writeString(jsonBuilder_, marker) + "\n");
+                    response->flush();
+                }
+            });
         return restinio::request_handling_status_t::accepted;
     } catch (const std::exception& e) {
         return serverError(*request);
@@ -958,8 +976,18 @@ DhtProxyServer::subscribe(restinio::request_handle_t request, restinio::router::
             listIt->clientId = clientId;
             listIt->sessionCtx = std::make_shared<PushSessionContext>(sessionId);
         } else {
-            std::lock_guard l(listIt->sessionCtx->lock);
-            listIt->sessionCtx->sessionId = sessionId;
+            bool failed;
+            {
+                std::lock_guard l(listIt->sessionCtx->lock);
+                listIt->sessionCtx->sessionId = sessionId;
+                failed = listIt->sessionCtx->initialized && !listIt->sessionCtx->initialOk;
+            }
+            if (failed) {
+                onPushListenReady(listIt->sessionCtx, false);
+                dht_->cancelListen(infoHash, std::move(listIt->internalToken));
+                listIt->sessionCtx = std::make_shared<PushSessionContext>(sessionId);
+                newListener = true;
+            }
         }
         auto& listener = *listIt;
 
@@ -998,56 +1026,84 @@ DhtProxyServer::subscribe(restinio::request_handle_t request, restinio::router::
                                                    infoHash,
                                                    clientId));
 
-        // Send response
-        if (not newListener) {
-            if (logger_)
-                logger_->debug("[proxy:server] [subscribe {}] found [client {}]", infoHash, listener.clientId);
-            // Send response header
-            auto response = std::make_shared<ResponseByPartsBuilder>(
-                initHttpResponse(request->create_response<ResponseByParts>()));
-            response->flush();
-            if (!root["refresh"].asBool()) {
-                // No Refresh
-                dht_->get(
-                    infoHash,
-                    [this, response](const Sp<Value>& value) {
-                        auto output = Json::writeString(jsonBuilder_, value->toJson()) + "\n";
-                        response->append_chunk(output);
-                        response->flush();
-                        return true;
-                    },
-                    [response](bool) { response->done(); });
-            } else {
-                // Refresh
-                response->append_chunk("{}\n");
-                return response->done();
-            }
-        } else {
-            // =========== No existing listener for an infoHash ============
-            // Add listen on dht
+        if (newListener) {
             if (logger_)
                 logger_->debug("[proxy:server] [subscribe {}] new", infoHash);
-            listener.internalToken = dht_->listen(infoHash,
-                                                  [this,
-                                                   infoHash,
-                                                   pushToken,
-                                                   type,
-                                                   clientId,
-                                                   sessionCtx = listener.sessionCtx,
-                                                   topic](const std::vector<Sp<Value>>& values, bool expired) {
-                                                      return this->handlePushListen(infoHash,
-                                                                                    pushToken,
-                                                                                    type,
-                                                                                    clientId,
-                                                                                    sessionCtx,
-                                                                                    topic,
-                                                                                    values,
-                                                                                    expired);
-                                                  });
-            // Send response header
+            listener.internalToken = dht_->listen(
+                infoHash,
+                [this, infoHash, pushToken, type, clientId, sessionCtx = listener.sessionCtx, topic](
+                    const std::vector<Sp<Value>>& values, bool expired) {
+                    return this
+                        ->handlePushListen(infoHash, pushToken, type, clientId, sessionCtx, topic, values, expired);
+                },
+                {},
+                {},
+                [this, sessionCtx = listener.sessionCtx](bool ok) { onPushListenReady(sessionCtx, ok); });
+        }
+        if (root["sync"].isObject()) {
+            auto sessionCtx = listener.sessionCtx;
+            auto finish = [this,
+                           request,
+                           context = std::weak_ptr<PushSessionContext>(sessionCtx),
+                           sync = root["sync"]]() {
+                auto sessionCtx = context.lock();
+                if (!sessionCtx) {
+                    auto response = initHttpResponse(request->create_response(restinio::status_service_unavailable()));
+                    response.done();
+                    return;
+                }
+                std::vector<Json::Value> snapshotValues;
+                Json::Value marker;
+                bool ok;
+                {
+                    std::lock_guard sessionLock(sessionCtx->lock);
+                    ok = sessionCtx->initialOk;
+                    bool snapshot = !sync["epoch"].isString() || sync["epoch"].asString() != sessionCtx->epoch
+                                    || !sync["revision"].isUInt64()
+                                    || sync["revision"].asUInt64() != sessionCtx->revision;
+                    marker["sync"]["epoch"] = sessionCtx->epoch;
+                    marker["sync"]["revision"] = Json::UInt64(sessionCtx->revision);
+                    marker["sync"]["snapshot"] = snapshot;
+                    marker["sync"]["complete"] = true;
+                    if (snapshot)
+                        for (const auto& value : sessionCtx->values)
+                            snapshotValues.emplace_back(value.second);
+                }
+                if (!ok) {
+                    auto response = initHttpResponse(request->create_response(restinio::status_service_unavailable()));
+                    response.done();
+                    return;
+                }
+                auto response = std::make_shared<ResponseByPartsBuilder>(
+                    initHttpResponse(request->create_response<ResponseByParts>()));
+                for (const auto& value : snapshotValues)
+                    response->append_chunk(Json::writeString(jsonBuilder_, value) + "\n");
+                response->append_chunk(Json::writeString(jsonBuilder_, marker) + "\n");
+                response->done();
+            };
+            {
+                std::lock_guard sessionLock(sessionCtx->lock);
+                if (sessionCtx->initialized)
+                    asio::post(io_context(), std::move(finish));
+                else
+                    sessionCtx->pendingSnapshots.emplace_back(std::move(finish));
+            }
+        } else if (newListener || root["refresh"].asBool()) {
             auto response = initHttpResponse(request->create_response());
             response.set_body("{}\n");
             return response.done();
+        } else {
+            auto response = std::make_shared<ResponseByPartsBuilder>(
+                initHttpResponse(request->create_response<ResponseByParts>()));
+            response->flush();
+            dht_->get(
+                infoHash,
+                [this, response](const Sp<Value>& value) {
+                    response->append_chunk(Json::writeString(jsonBuilder_, value->toJson()) + "\n");
+                    response->flush();
+                    return true;
+                },
+                [response](bool) { response->done(); });
         }
     } catch (...) {
         return serverError(*request);
@@ -1134,6 +1190,7 @@ DhtProxyServer::handleCancelPushListen(const asio::error_code& ec,
 
     for (auto listener = listeners->second.begin(); listener != listeners->second.end();) {
         if (listener->clientId == clientId) {
+            onPushListenReady(listener->sessionCtx, false);
             if (dht_)
                 dht_->cancelListen(key, std::move(listener->internalToken));
             listener = listeners->second.erase(listener);
@@ -1147,6 +1204,29 @@ DhtProxyServer::handleCancelPushListen(const asio::error_code& ec,
         pushListeners_.erase(pushListener);
 }
 
+void
+DhtProxyServer::onPushListenReady(const std::shared_ptr<PushSessionContext>& sessionCtx, bool ok)
+{
+    if (!sessionCtx)
+        return;
+    std::vector<std::function<void()>> pending;
+    {
+        std::lock_guard sessionLock(sessionCtx->lock);
+        if (sessionCtx->initialized)
+            return;
+        sessionCtx->initialized = true;
+        sessionCtx->initialOk = ok;
+        pending = std::move(sessionCtx->pendingSnapshots);
+        sessionCtx->pendingSnapshots.clear();
+    }
+    for (auto& response : pending) {
+        if (ok)
+            asio::post(io_context(), std::move(response));
+        else
+            response();
+    }
+}
+
 bool
 DhtProxyServer::handlePushListen(const InfoHash& infoHash,
                                  const std::string& pushToken,
@@ -1165,7 +1245,15 @@ DhtProxyServer::handlePushListen(const InfoHash& infoHash,
 
     {
         std::lock_guard l(sessionCtx->lock);
+        for (const auto& value : values) {
+            if (expired)
+                sessionCtx->values.erase(value->id);
+            else
+                sessionCtx->values[value->id] = value->toJson();
+        }
         json["s"] = sessionCtx->sessionId;
+        json["epoch"] = sessionCtx->epoch;
+        json["revision"] = std::to_string(++sessionCtx->revision);
     }
 
     // Build a comma-separated list of ids from the values. This is intended to be used when
@@ -1188,6 +1276,8 @@ DhtProxyServer::handlePushListen(const InfoHash& infoHash,
     if (expired && values.size() < 2) {
         json["exp"] = json["ids"];
     }
+    if (expired)
+        json["expired"] = "true";
 
     auto minPriority = 1000u;
     for (const auto& v : values)

src/dhtrunner.cpp

@@ -819,16 +819,24 @@ DhtRunner::query(const InfoHash& hash, QueryCallback cb, DoneCallback done_cb, Q
 
 std::future<size_t>
 DhtRunner::listen(InfoHash hash, ValueCallback vcb, Value::Filter f, Where w)
+{
+    return listen(hash, std::move(vcb), std::move(f), std::move(w), {});
+}
+
+std::future<size_t>
+DhtRunner::listen(InfoHash hash, ValueCallback vcb, Value::Filter f, Where w, DoneCallbackSimple ready)
 {
     auto ret_token = std::make_shared<std::promise<size_t>>();
     std::unique_lock lck(storage_mtx);
     if (running != State::Running) {
         lck.unlock();
         ret_token->set_value(0);
+        if (ready)
+            ready(false);
         return ret_token->get_future();
     }
     pending_ops.emplace([=](SecureDht& dht) mutable {
-        ret_token->set_value(dht.listen(hash, std::move(vcb), std::move(f), std::move(w)));
+        ret_token->set_value(dht.listen(hash, std::move(vcb), std::move(f), std::move(w), std::move(ready)));
     });
     cv.notify_all();
     return ret_token->get_future();

src/http.cpp

@@ -1527,9 +1527,8 @@ Request::terminate(const asio::error_code& ec)
         return;
 
     response_.aborted = ec == asio::error::operation_aborted;
-    if (ec == asio::error::basic_errors::broken_pipe)
-        response_.status_code
-            = 0U; // Avoid to give a successful answer (happen with a broken pipe, takes the last status)
+    if (ec and ec != asio::error::eof)
+        response_.status_code = 0U;
 
     if (logger_) {
         if (ec and ec != asio::error::eof and !response_.aborted)
@@ -1576,7 +1575,12 @@ Request::handle_response(const asio::error_code& ec, size_t /* n_bytes */)
         terminate(ec);
         return;
     }
-    auto request = (ec == asio::error::eof) ? std::string {} : conn_->read_bytes();
+    if (ec == asio::error::eof) {
+        if (!parser_ || llhttp_finish(parser_.get()) != HPE_OK || state_ != State::DONE)
+            terminate(asio::error::broken_pipe);
+        return;
+    }
+    auto request = conn_->read_bytes();
     enum llhttp_errno ret = llhttp_execute(parser_.get(), request.c_str(), request.size());
     if (ret != HPE_OK && ret != HPE_PAUSED) {
         if (logger_)

src/listener.h

@@ -5,6 +5,8 @@
 #include "value.h"
 #include "utils.h"
 #include "callbacks.h"
+#include "network_engine.h"
+#include "request.h"
 
 namespace dht {
 
@@ -13,9 +15,21 @@ namespace dht {
  */
 struct Listener
 {
+    struct Stream
+    {
+        net::ListenSync sync;
+        std::vector<Sp<net::Request>> initialRequests;
+        ~Stream()
+        {
+            for (const auto& request : initialRequests)
+                request->cancel();
+        }
+    };
+
     time_point time;
     Query query;
     int version;
+    std::shared_ptr<Stream> stream;
 
     Listener(time_point t, Query&& q, int version = 0)
         : time(t)

src/network_engine.cpp

@@ -103,6 +103,7 @@ RequestAnswer::RequestAnswer(ParsedMessage&& msg)
     , fields(std::move(msg.fields))
     , nodes4(std::move(msg.nodes4))
     , nodes6(std::move(msg.nodes6))
+    , listenSync(std::move(msg.listenSync))
 {}
 
 NetworkEngine::NetworkEngine(InfoHash& myid,
@@ -144,7 +145,7 @@ NetworkEngine::~NetworkEngine()
     clear();
 }
 
-void
+std::vector<Sp<Request>>
 NetworkEngine::tellListener(const Sp<Node>& node,
                             Tid socket_id,
                             const InfoHash& hash,
@@ -154,12 +155,13 @@ NetworkEngine::tellListener(const Sp<Node>& node,
                             std::vector<Sp<Node>>&& nodes6,
                             std::vector<Sp<Value>>&& values,
                             const Query& query,
-                            int version)
+                            int version,
+                            ListenSync* sync)
 {
     auto nnodes = bufferNodes(node->getFamily(), hash, want, nodes, nodes6);
     try {
         if (version >= 1) {
-            sendUpdateValues(node, hash, std::move(values), scheduler.time(), ntoken, socket_id);
+            return sendUpdateValues(node, hash, std::move(values), scheduler.time(), ntoken, socket_id, sync);
         } else {
             sendNodesValues(node->getAddr(), socket_id, nnodes.first, nnodes.second, std::move(values), query, ntoken);
         }
@@ -167,6 +169,7 @@ NetworkEngine::tellListener(const Sp<Node>& node,
         if (logger_)
             logger_->error("Unable to send value: buffer not large enough.");
     }
+    return {};
 }
 
 void
@@ -175,7 +178,8 @@ NetworkEngine::tellListenerRefreshed(const Sp<Node>& n,
                                      const InfoHash&,
                                      const Blob& token,
                                      const std::vector<Value::Id>& values,
-                                     int version)
+                                     int version,
+                                     ListenSync* sync)
 {
     msgpack::sbuffer buffer;
     msgpack::packer<msgpack::sbuffer> pk(&buffer);
@@ -183,7 +187,15 @@ NetworkEngine::tellListenerRefreshed(const Sp<Node>& n,
     pk.pack_map(4 + (version >= 1 ? 1 : 0) + (config.network ? 1 : 0) + (config.is_client ? 1 : 0));
 
     pk.pack(version >= 1 ? KEY_A : KEY_U);
-    pk.pack_map(1 + (version >= 1 ? 1 : 0) + (not values.empty() ? 1 : 0) + (not token.empty() ? 1 : 0));
+    pk.pack_map(1 + (version >= 1 ? 1 : 0) + (not values.empty() ? 1 : 0) + (not token.empty() ? 1 : 0)
+                + (sync ? 2 : 0));
+    if (sync) {
+        ++sync->sequence;
+        pk.pack(KEY_VERSION);
+        pk.pack(2);
+        pk.pack(KEY_LISTEN_SYNC);
+        pk.pack(*sync);
+    }
     pk.pack(KEY_REQ_ID);
     pk.pack(myid);
     if (version >= 1) {
@@ -246,7 +258,8 @@ NetworkEngine::tellListenerExpired(const Sp<Node>& n,
                                    const InfoHash&,
                                    const Blob& token,
                                    const std::vector<Value::Id>& values,
-                                   int version)
+                                   int version,
+                                   ListenSync* sync)
 {
     msgpack::sbuffer buffer;
     msgpack::packer<msgpack::sbuffer> pk(&buffer);
@@ -254,7 +267,15 @@ NetworkEngine::tellListenerExpired(const Sp<Node>& n,
     pk.pack_map(4 + (version >= 1 ? 1 : 0) + (config.network ? 1 : 0) + (config.is_client ? 1 : 0));
 
     pk.pack(version >= 1 ? KEY_A : KEY_U);
-    pk.pack_map(1 + (version >= 1 ? 1 : 0) + (not values.empty() ? 1 : 0) + (not token.empty() ? 1 : 0));
+    pk.pack_map(1 + (version >= 1 ? 1 : 0) + (not values.empty() ? 1 : 0) + (not token.empty() ? 1 : 0)
+                + (sync ? 2 : 0));
+    if (sync) {
+        ++sync->sequence;
+        pk.pack(KEY_VERSION);
+        pk.pack(2);
+        pk.pack(KEY_LISTEN_SYNC);
+        pk.pack(*sync);
+    }
     pk.pack(KEY_REQ_ID);
     pk.pack(myid);
     if (version >= 1) {
@@ -784,10 +805,15 @@ NetworkEngine::process(std::unique_ptr<ParsedMessage>&& msg, const SockAddr& fro
                 if (logIncoming_ and logger_)
                     logger_->debug("[node {}] got 'listen' request for {}", node->toString(), msg->info_hash.toString());
                 ++in_stats.listen;
-                RequestAnswer answer
-                    = onListen(node, msg->info_hash, msg->token, msg->socket_id, std::move(msg->query), msg->version);
+                RequestAnswer answer = onListen(node,
+                                                msg->info_hash,
+                                                msg->token,
+                                                msg->socket_id,
+                                                std::move(msg->query),
+                                                msg->version,
+                                                msg->listenSync);
                 auto nnodes = bufferNodes(from.getFamily(), msg->info_hash, msg->want, answer.nodes4, answer.nodes6);
-                sendListenConfirmation(from, msg->tid, answer.ntoken);
+                sendListenConfirmation(from, msg->tid, answer.ntoken, answer.listenSync);
                 break;
             }
             case MessageType::UpdateValue: {
@@ -1307,7 +1333,8 @@ NetworkEngine::sendListen(const Sp<Node>& n,
                           const Blob& token,
                           Tid socketId,
                           RequestCb&& on_done,
-                          RequestExpiredCb&& on_expired)
+                          RequestExpiredCb&& on_expired,
+                          const std::optional<ListenSync>& sync)
 {
     Tid tid(n->getNewTid());
     msgpack::sbuffer buffer;
@@ -1316,11 +1343,15 @@ NetworkEngine::sendListen(const Sp<Node>& n,
 
     auto has_query = not query.where.empty() or not query.select.empty();
     pk.pack(KEY_A);
-    pk.pack_map(5 + has_query);
+    pk.pack_map(5 + has_query + (sync ? 1 : 0));
     pk.pack(KEY_REQ_ID);
     pk.pack(myid);
     pk.pack(KEY_VERSION);
-    pk.pack(1);
+    pk.pack(sync ? 2 : 1);
+    if (sync) {
+        pk.pack(KEY_LISTEN_SYNC);
+        pk.pack(*sync);
+    }
     pk.pack(KEY_REQ_H);
     pk.pack(hash);
     pk.pack(KEY_REQ_TOKEN);
@@ -1366,14 +1397,23 @@ NetworkEngine::sendListen(const Sp<Node>& n,
 }
 
 void
-NetworkEngine::sendListenConfirmation(const SockAddr& addr, Tid tid, const Blob& token)
+NetworkEngine::sendListenConfirmation(const SockAddr& addr,
+                                      Tid tid,
+                                      const Blob& token,
+                                      const std::optional<ListenSync>& sync)
 {
     msgpack::sbuffer buffer;
     msgpack::packer<msgpack::sbuffer> pk(&buffer);
     pk.pack_map(4 + (config.network ? 1 : 0) + (config.is_client ? 1 : 0));
 
     pk.pack(KEY_R);
-    pk.pack_map(2 + (not token.empty() ? 1 : 0));
+    pk.pack_map(2 + (not token.empty() ? 1 : 0) + (sync ? 2 : 0));
+    if (sync) {
+        pk.pack(KEY_VERSION);
+        pk.pack(2);
+        pk.pack(KEY_LISTEN_SYNC);
+        pk.pack(*sync);
+    }
     pk.pack(KEY_REQ_ID);
     pk.pack(myid);
     insertAddr(pk, addr);
@@ -1471,29 +1511,50 @@ NetworkEngine::sendAnnounceValue(const Sp<Node>& n,
     return req;
 }
 
-void
+static std::vector<size_t>
+updateBatchEnds(const std::vector<Sp<Value>>& values)
+{
+    std::vector<size_t> ends;
+    size_t totalSize = 0;
+    size_t count = 0;
+    for (size_t index = 0; index < values.size(); ++index) {
+        auto size = packMsg(*values[index]).size();
+        if (count && (count == 32 || totalSize + size > 56 * 1024)) {
+            ends.emplace_back(index);
+            totalSize = 0;
+            count = 0;
+        }
+        totalSize += size;
+        ++count;
+    }
+    if (count)
+        ends.emplace_back(values.size());
+    return ends;
+}
+
+size_t
+NetworkEngine::getUpdateBatchCount(const std::vector<Sp<Value>>& values)
+{
+    return updateBatchEnds(values).size();
+}
+
+std::vector<Sp<Request>>
 NetworkEngine::sendUpdateValues(const Sp<Node>& n,
                                 const InfoHash& infohash,
                                 std::vector<Sp<Value>>&& values,
                                 time_point created,
                                 const Blob& token,
-                                size_t socket_id)
+                                size_t socket_id,
+                                ListenSync* sync)
 {
-    size_t total_size = 0;
-
-    auto b = values.begin(), e = values.begin();
-    while (e != values.end()) {
-        if (total_size >= MAX_MESSAGE_VALUE_SIZE) {
-            sendUpdateValues(n, infohash, b, e, created, token, socket_id);
-            b = e;
-            total_size = 0;
-        }
-        total_size += (*e)->size();
-        ++e;
-    }
-    if (b != e) {
-        sendUpdateValues(n, infohash, b, e, created, token, socket_id);
+    std::vector<Sp<Request>> requests;
+    size_t begin = 0;
+    for (auto end : updateBatchEnds(values)) {
+        requests.emplace_back(
+            sendUpdateValues(n, infohash, values.begin() + begin, values.begin() + end, created, token, socket_id, sync));
+        begin = end;
     }
+    return requests;
 }
 
 Sp<Request>
@@ -1503,7 +1564,8 @@ NetworkEngine::sendUpdateValues(const Sp<Node>& n,
                                 std::vector<Sp<Value>>::iterator end,
                                 time_point created,
                                 const Blob& token,
-                                size_t socket_id)
+                                size_t socket_id,
+                                ListenSync* sync)
 {
     Tid tid(n->getNewTid());
     Tid sid(socket_id);
@@ -1513,11 +1575,16 @@ NetworkEngine::sendUpdateValues(const Sp<Node>& n,
     pk.pack_map(5 + (config.network ? 1 : 0) + (config.is_client ? 1 : 0));
 
     pk.pack(KEY_A);
-    pk.pack_map((created < scheduler.time() ? 7 : 6));
+    pk.pack_map((created < scheduler.time() ? 7 : 6) + (sync ? 1 : 0));
     pk.pack(KEY_REQ_ID);
     pk.pack(myid);
     pk.pack(KEY_VERSION);
-    pk.pack(1);
+    pk.pack(sync ? 2 : 1);
+    if (sync) {
+        ++sync->sequence;
+        pk.pack(KEY_LISTEN_SYNC);
+        pk.pack(*sync);
+    }
     pk.pack(KEY_REQ_H);
     pk.pack(infohash);
     pk.pack(KEY_REQ_SID);

src/op_cache.cpp

@@ -174,13 +174,17 @@ SearchCache::getOp(const Sp<Query>& q) const
 }
 
 size_t
-SearchCache::listen(const ValueCallback& get_cb, const Sp<Query>& q, Value::Filter&& filter, const OnListen& onListen)
+SearchCache::listen(const ValueCallback& get_cb,
+                    const Sp<Query>& q,
+                    Value::Filter&& filter,
+                    const OnListen& onListen,
+                    DoneCallbackSimple ready)
 {
     // find exact match
     auto op = getOp(q);
     if (op == ops.end()) {
         // New query
-        op = ops.emplace(q, std::make_unique<OpCache>(countSources_)).first;
+        op = ops.emplace(q, std::make_unique<OpCache>(countSources_, waitForSearch_)).first;
         auto& cache = *op->second;
         cache.searchToken = onListen(
             q,
@@ -190,7 +194,7 @@ SearchCache::listen(const ValueCallback& get_cb, const Sp<Query>& q, Value::Filt
     auto token = nextToken_++;
     if (nextToken_ == 0)
         nextToken_++;
-    return op->second->addListener(token, get_cb, q, std::move(filter)) ? token : 0;
+    return op->second->addListener(token, get_cb, q, std::move(filter), std::move(ready)) ? token : 0;
 }
 
 bool

src/op_cache.h

@@ -105,7 +105,7 @@ private:
 class OpCache
 {
 public:
-    explicit OpCache(bool countSources = true)
+    explicit OpCache(bool countSources = true, bool waitForSearch = false)
         : cache(
               [this](const std::vector<Sp<Value>>& vals, bool expired) {
                   if (expired)
@@ -115,31 +115,79 @@ public:
                   return true;
               },
               countSources)
+        , searchReady(!waitForSearch)
     {}
 
     bool onValue(const std::vector<Sp<Value>>& vals, bool expired) { return cache.onValue(vals, expired); }
-    void onNodeChanged(ListenSyncStatus status) { cache.onNodeChanged(status); }
+    void onNodeChanged(ListenSyncStatus status)
+    {
+        cache.onNodeChanged(status);
+        if (searchReady && cache.isSynced())
+            onReady();
+        else
+            initialReady = false;
+    }
+
+    void onReady(bool ok = true)
+    {
+        initialReady = ok;
+        searchReady = ok;
+        std::vector<size_t> pending;
+        pending.reserve(readyCallbacks.size());
+        for (const auto& callback : readyCallbacks)
+            pending.emplace_back(callback.first);
+        for (auto token : pending) {
+            auto found = readyCallbacks.find(token);
+            if (found == readyCallbacks.end())
+                continue;
+            auto callback = std::move(found->second);
+            readyCallbacks.erase(found);
+            callback(ok);
+        }
+    }
+
+    void onSearchReady()
+    {
+        searchReady = true;
+        if (cache.isSynced())
+            onReady();
+    }
 
     void onValuesAdded(const std::vector<Sp<Value>>& vals);
     void onValuesExpired(const std::vector<Sp<Value>>& vals);
 
-    bool addListener(size_t token, const ValueCallback& cb, const Sp<Query>& q, Value::Filter&& filter)
+    bool addListener(size_t token,
+                     const ValueCallback& cb,
+                     const Sp<Query>& q,
+                     Value::Filter&& filter,
+                     DoneCallbackSimple ready = {})
     {
         auto cached = cache.get(filter);
         if (not cached.empty() and not cb(cached, false)) {
             return false;
         }
         listeners.emplace(token, LocalListener {q, std::move(filter), cb});
+        if (ready) {
+            if (initialReady || (searchReady && cache.isSynced()))
+                ready(true);
+            else
+                readyCallbacks.emplace(token, std::move(ready));
+        }
         return true;
     }
 
     bool removeListener(size_t token, const time_point& now)
     {
         lastRemoved = now;
+        readyCallbacks.erase(token);
         return listeners.erase(token) > 0;
     }
 
-    void removeAll() { listeners.clear(); }
+    void removeAll()
+    {
+        listeners.clear();
+        readyCallbacks.clear();
+    }
 
     bool isDone() { return listeners.empty(); }
 
@@ -163,19 +211,46 @@ private:
 
     OpValueCache cache;
     std::map<size_t, LocalListener> listeners;
+    std::map<size_t, DoneCallbackSimple> readyCallbacks;
+    bool initialReady {false};
+    bool searchReady {true};
     time_point lastRemoved {clock::now()};
 };
 
 class SearchCache
 {
 public:
-    explicit SearchCache(bool countSources = true)
+    explicit SearchCache(bool countSources = true, bool waitForSearch = false)
         : countSources_(countSources)
+        , waitForSearch_(waitForSearch)
     {}
     SearchCache(SearchCache&&) = default;
 
     using OnListen = std::function<size_t(Sp<Query>, ValueCallback, SyncCallback)>;
-    size_t listen(const ValueCallback& get_cb, const Sp<Query>& q, Value::Filter&& filter, const OnListen& onListen);
+    size_t listen(const ValueCallback& get_cb,
+                  const Sp<Query>& q,
+                  Value::Filter&& filter,
+                  const OnListen& onListen,
+                  DoneCallbackSimple ready = {});
+
+    void onReady(bool ok = true)
+    {
+        for (auto& operation : ops)
+            operation.second->onReady(ok);
+    }
+
+    void onReady(const Sp<Query>& query, bool ok)
+    {
+        auto operation = getOp(query);
+        if (operation != ops.end())
+            operation->second->onReady(ok);
+    }
+
+    void onSearchReady()
+    {
+        for (auto& operation : ops)
+            operation.second->onSearchReady();
+    }
 
     bool cancelListen(size_t gtoken, const time_point& now);
     void cancelAll(const std::function<void(size_t)>& onCancel);
@@ -208,6 +283,7 @@ private:
     size_t nextToken_ {1};
     time_point nextExpiration_ {time_point::max()};
     bool countSources_ {true};
+    bool waitForSearch_ {false};
 };
 
 } // namespace dht

src/parsed_message.h

@@ -5,6 +5,7 @@
 #include "infohash.h"
 #include "sockaddr.h"
 #include "net.h"
+#include "network_engine.h"
 
 #include <algorithm>
 #include <map>
@@ -44,6 +45,7 @@ static constexpr auto KEY_REQ_REFRESHED = "re"sv;
 static constexpr auto KEY_REQ_FIELDS = "fileds"sv;
 static constexpr auto KEY_REQ_WANT = "w"sv;
 static constexpr auto KEY_VERSION = "ve"sv;
+static constexpr auto KEY_LISTEN_SYNC = "ls"sv;
 
 static constexpr auto QUERY_PING = "ping"sv;
 static constexpr auto QUERY_FIND = "find"sv;
@@ -128,6 +130,7 @@ struct ParsedMessage
     /* reported address by the distant node */
     std::string ua;
     int version {0};
+    std::optional<ListenSync> listenSync {};
     SockAddr addr;
     void msgpack_unpack(const msgpack::object& o);
 
@@ -180,7 +183,10 @@ ParsedMessage::complete()
         auto assembled = pv.reassemble();
         msgpack::unpacked msg;
         msgpack::unpack(msg, (const char*) assembled.data(), assembled.size());
-        values.emplace_back(std::make_shared<Value>(msg.get()));
+        auto value = std::make_shared<Value>(msg.get());
+        if (listenSync && value->id == Value::INVALID_ID)
+            throw msgpack::type_error();
+        values.emplace_back(std::move(value));
     }
     return true;
 }
@@ -337,8 +343,22 @@ ParsedMessage::msgpack_unpack(const msgpack::object& msg)
             parsedReq.want = &o.val;
         else if (key == KEY_VERSION)
             version = o.val.as<int>();
+        else if (key == KEY_LISTEN_SYNC) {
+            if (o.val.type != msgpack::type::ARRAY || o.val.via.array.size != 4)
+                throw msgpack::type_error();
+            for (const auto& number :
+                 {o.val.via.array.ptr[0], o.val.via.array.ptr[1], o.val.via.array.ptr[2], o.val.via.array.ptr[3]})
+                if (number.type != msgpack::type::POSITIVE_INTEGER)
+                    throw msgpack::type_error();
+            listenSync = o.val.as<ListenSync>();
+            if (!listenSync->registration)
+                throw msgpack::type_error();
+        }
     }
 
+    if (listenSync && version != 2)
+        throw msgpack::type_error();
+
     if (parsedReq.sa) {
         if (parsedReq.sa->type != msgpack::type::BIN)
             throw msgpack::type_error();
@@ -360,18 +380,28 @@ ParsedMessage::msgpack_unpack(const msgpack::object& msg)
     if (parsedReq.values) {
         if (parsedReq.values->type != msgpack::type::ARRAY)
             throw msgpack::type_error();
+        if (listenSync && parsedReq.values->via.array.size > 32)
+            throw msgpack::type_error();
         for (size_t i = 0; i < parsedReq.values->via.array.size; i++) {
             auto& packed_v = parsedReq.values->via.array.ptr[i];
             if (packed_v.type == msgpack::type::POSITIVE_INTEGER) {
                 // Skip oversize values with a small margin for header overhead
-                if (packed_v.via.u64 > MAX_VALUE_SIZE + 32)
+                if (packed_v.via.u64 > MAX_VALUE_SIZE + 32 || (listenSync && !packed_v.via.u64)) {
+                    if (listenSync)
+                        throw msgpack::type_error();
                     continue;
+                }
                 value_parts[i].total_size = packed_v.via.u64;
             } else {
                 try {
-                    values.emplace_back(std::make_shared<Value>(parsedReq.values->via.array.ptr[i]));
+                    auto value = std::make_shared<Value>(parsedReq.values->via.array.ptr[i]);
+                    if (listenSync && value->id == Value::INVALID_ID)
+                        throw msgpack::type_error();
+                    values.emplace_back(std::move(value));
                 } catch (const std::exception& e) {
                     // DHT_LOG_WARN("Error reading value: %s", e.what());
+                    if (listenSync)
+                        throw msgpack::type_error();
                 }
             }
         }

src/search.h

@@ -65,6 +65,16 @@ struct Dht::SearchNode
         Sp<Scheduler::Job> refresh {};
         Sp<Scheduler::Job> cacheExpirationJob {};
         Sp<net::Request> req {};
+        Sp<Scheduler::Job> syncDeadline {};
+        uint64_t registration {0};
+        std::optional<net::ListenSync> manifest;
+        std::map<std::pair<uint64_t, uint64_t>, net::RequestAnswer> pendingUpdates;
+        uint64_t nextSequence {1};
+        uint64_t watermark {0};
+        size_t pendingBytes {0};
+        bool baselineComplete {false};
+        bool initialized {false};
+        bool legacy {false};
         Tid socketId {0};
         CachedListenStatus(ValueStateCallback&& cb, SyncCallback scb, Tid sid)
             : cache(std::forward<ValueStateCallback>(cb), std::forward<SyncCallback>(scb))
@@ -74,6 +84,12 @@ struct Dht::SearchNode
         CachedListenStatus(const CachedListenStatus&) = delete;
         ~CachedListenStatus()
         {
+            if (syncDeadline)
+                syncDeadline->cancel();
+            if (refresh)
+                refresh->cancel();
+            if (cacheExpirationJob)
+                cacheExpirationJob->cancel();
             if (socketId and req and req->node) {
                 req->node->closeSocket(socketId);
             }
@@ -253,17 +269,130 @@ struct Dht::SearchNode
         getStatus.clear();
     }
 
-    void onValues(const Sp<Query>& q, net::RequestAnswer&& answer, const TypeStore& types, Scheduler& scheduler)
+    bool confirmListen(const Sp<Query>& query,
+                       const std::optional<net::ListenSync>& manifest,
+                       const TypeStore& types,
+                       Scheduler& scheduler)
     {
-        auto l = listenStatus.find(q);
-        if (l != listenStatus.end()) {
-            auto next = l->second.cache.onValues(answer.values,
-                                                 answer.refreshed_values,
-                                                 answer.expired_values,
-                                                 types,
-                                                 scheduler.time());
-            scheduler.edit(l->second.cacheExpirationJob, next);
+        auto found = listenStatus.find(query);
+        if (found == listenStatus.end())
+            return false;
+        auto& status = found->second;
+        if (!manifest) {
+            if (status.manifest)
+                return false;
+            status.legacy = true;
+            status.initialized = true;
+            status.pendingUpdates.clear();
+            status.pendingBytes = 0;
+            scheduler.cancel(status.syncDeadline);
+            status.cache.onSynced(true);
+            return true;
         }
+        if (manifest->registration != status.registration || !manifest->epoch || manifest->snapshotEnd > 4096
+            || manifest->sequence < manifest->snapshotEnd
+            || (status.manifest
+                && (status.manifest->epoch != manifest->epoch || status.manifest->snapshotEnd != manifest->snapshotEnd)))
+            return false;
+        status.legacy = false;
+        status.manifest = manifest;
+        status.watermark = std::max(status.watermark, manifest->sequence);
+        return onValues(query, {}, types, scheduler);
+    }
+
+    bool onValues(const Sp<Query>& query, net::RequestAnswer&& answer, const TypeStore& types, Scheduler& scheduler)
+    {
+        auto found = listenStatus.find(query);
+        if (found == listenStatus.end())
+            return true;
+        auto& status = found->second;
+        auto apply = [&](net::RequestAnswer&& update) {
+            auto next = status.cache.onValues(update.values,
+                                              update.refreshed_values,
+                                              update.expired_values,
+                                              types,
+                                              scheduler.time());
+            scheduler.edit(status.cacheExpirationJob, next);
+        };
+        if (!answer.listenSync
+            && (!answer.values.empty() || !answer.refreshed_values.empty() || !answer.expired_values.empty())) {
+            if (status.manifest)
+                return true;
+            status.legacy = true;
+            apply(std::move(answer));
+            return true;
+        }
+        if (answer.listenSync) {
+            auto sync = *answer.listenSync;
+            if (sync.registration != status.registration || (status.manifest && sync.epoch != status.manifest->epoch))
+                return true;
+            if (!sync.epoch || !sync.sequence || sync.snapshotEnd > 4096
+                || (status.manifest && sync.snapshotEnd != status.manifest->snapshotEnd))
+                return false;
+            if (!answer.fields.empty()
+                || (sync.sequence <= sync.snapshotEnd
+                    && (answer.values.empty() || !answer.expired_values.empty() || !answer.refreshed_values.empty())))
+                return false;
+            if (sync.sequence < status.nextSequence)
+                return true;
+            auto key = std::make_pair(sync.epoch, sync.sequence);
+            if (!status.pendingUpdates.count(key)) {
+                size_t bytes = 0;
+                for (const auto& value : answer.values)
+                    bytes += value->size();
+                bytes += (answer.refreshed_values.size() + answer.expired_values.size()) * sizeof(Value::Id);
+                if (status.pendingUpdates.size() >= 4096 || bytes > 64 * 1024 * 1024 - status.pendingBytes)
+                    return false;
+                status.pendingBytes += bytes;
+                status.pendingUpdates.emplace(key, std::move(answer));
+            }
+        }
+        if (!status.manifest)
+            return true;
+        auto epoch = status.manifest->epoch;
+        for (auto pending = status.pendingUpdates.begin(); pending != status.pendingUpdates.end();) {
+            if (pending->first.first != epoch)
+                pending = status.pendingUpdates.erase(pending);
+            else {
+                if (pending->second.listenSync->snapshotEnd != status.manifest->snapshotEnd)
+                    return false;
+                status.watermark = std::max(status.watermark, pending->first.second);
+                ++pending;
+            }
+        }
+        if (!status.baselineComplete) {
+            size_t values = 0;
+            for (uint64_t sequence = 1; sequence <= status.manifest->snapshotEnd; ++sequence) {
+                auto batch = status.pendingUpdates.find({epoch, sequence});
+                if (batch == status.pendingUpdates.end())
+                    return true;
+                values += batch->second.values.size();
+            }
+            if (values > 4096)
+                return false;
+            status.baselineComplete = true;
+        }
+        while (true) {
+            auto pending = status.pendingUpdates.find({epoch, status.nextSequence});
+            if (pending == status.pendingUpdates.end())
+                break;
+            auto update = std::move(pending->second);
+            status.pendingUpdates.erase(pending);
+            ++status.nextSequence;
+            apply(std::move(update));
+        }
+        status.pendingBytes = 0;
+        for (const auto& pending : status.pendingUpdates) {
+            for (const auto& value : pending.second.values)
+                status.pendingBytes += value->size();
+            status.pendingBytes += (pending.second.refreshed_values.size() + pending.second.expired_values.size())
+                                   * sizeof(Value::Id);
+        }
+        status.initialized = status.nextSequence > status.watermark;
+        if (status.initialized)
+            scheduler.cancel(status.syncDeadline);
+        status.cache.onSynced(status.initialized);
+        return true;
     }
 
     void onListenSynced(const Sp<Query>& q, bool synced = true, Sp<Scheduler::Job> refreshJob = {})
@@ -467,7 +596,7 @@ struct Dht::Search
     size_t listener_token = 1;
 
     /* Cache */
-    SearchCache cache;
+    SearchCache cache {true, true};
     Sp<Scheduler::Job> opExpirationJob;
 
     ~Search()
@@ -596,16 +725,25 @@ struct Dht::Search
         }
     }
 
-    size_t listen(const ValueCallback& cb, Value::Filter&& f, const Sp<Query>& q, Scheduler& scheduler)
+    size_t listen(const ValueCallback& cb,
+                  Value::Filter&& f,
+                  const Sp<Query>& q,
+                  Scheduler& scheduler,
+                  DoneCallbackSimple ready = {})
     {
         // DHT_LOG.e(id, "[search %s IPv%c] listen", id.toString().c_str(), (af == AF_INET) ? '4' : '6');
-        return cache.listen(cb, q, std::move(f), [&](const Sp<Query>& q, ValueCallback vcb, SyncCallback scb) {
-            done = false;
-            auto token = ++listener_token;
-            listeners.emplace(token, SearchListener {q, vcb, scb});
-            scheduler.edit(nextSearchStep, scheduler.time());
-            return token;
-        });
+        return cache.listen(
+            cb,
+            q,
+            std::move(f),
+            [&](const Sp<Query>& q, ValueCallback vcb, SyncCallback scb) {
+                done = false;
+                auto token = ++listener_token;
+                listeners.emplace(token, SearchListener {q, vcb, scb});
+                scheduler.edit(nextSearchStep, scheduler.time());
+                return token;
+            },
+            std::move(ready));
     }
 
     void cancelListen(size_t token, Scheduler& scheduler)
@@ -769,6 +907,7 @@ struct Dht::Search
     {
         // no nodes or all expired nodes. This is most likely a connectivity change event.
         expired = true;
+        cache.onReady(false);
 
         nodes.clear();
         if (announce.empty() && listeners.empty())
@@ -841,6 +980,7 @@ struct Dht::Search
 
     void clear()
     {
+        cache.onReady(false);
         announce.clear();
         callbacks.clear();
         listeners.clear();

src/securedht.cpp

@@ -333,7 +333,17 @@ SecureDht::get(const InfoHash& id, GetCallback cb, DoneCallback donecb, Value::F
 size_t
 SecureDht::listen(const InfoHash& id, ValueCallback cb, Value::Filter f, Where w)
 {
-    return dht_->listen(id, getCallbackFilter(cb, std::forward<Value::Filter>(f)), {}, std::forward<Where>(w));
+    return listen(id, std::move(cb), std::move(f), std::move(w), {});
+}
+
+size_t
+SecureDht::listen(const InfoHash& id, ValueCallback cb, Value::Filter f, Where w, DoneCallbackSimple ready)
+{
+    return dht_->listen(id,
+                        getCallbackFilter(cb, std::forward<Value::Filter>(f)),
+                        {},
+                        std::forward<Where>(w),
+                        std::move(ready));
 }
 
 size_t

tests/test_dhtproxy.cpp

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: MIT
 
 #include "test_dhtproxy.h"
+#include <opendht/http.h>
 
 // std
 #include <iostream>
@@ -28,6 +29,81 @@ sendPush(dht::DhtRunner& node, const std::map<std::string, std::string>& data, s
     lk.lock();
     return ret;
 }
+
+void
+setValueRevision(dht::DhtProxyServer& server,
+                 const dht::InfoHash& key,
+                 const std::string& pushToken,
+                 const std::vector<std::pair<dht::Value::Id, uint16_t>>& expected,
+                 std::map<std::string, std::string>& data)
+{
+    std::shared_ptr<dht::DhtProxyServer::PushSessionContext> context;
+    {
+        std::lock_guard lock(server.lockListener_);
+        context = server.pushListeners_.at(pushToken).listeners.at(key).front().sessionCtx;
+    }
+    auto observed = std::make_shared<std::promise<void>>();
+    auto completed = std::make_shared<std::atomic_bool>(false);
+    auto future = observed->get_future();
+    auto token = server.dht_
+                     ->listen(key,
+                              [context, expected, observed, completed](const std::vector<std::shared_ptr<dht::Value>>&,
+                                                                       bool expired) {
+                                  if (expired)
+                                      return true;
+                                  std::lock_guard lock(context->lock);
+                                  for (const auto& version : expected) {
+                                      auto value = context->values.find(version.first);
+                                      if (value == context->values.end()
+                                          || dht::Value(value->second).seq != version.second)
+                                          return true;
+                                  }
+                                  if (!completed->exchange(true))
+                                      observed->set_value();
+                                  return true;
+                              })
+                     .get();
+    auto received = future.wait_for(10s) == std::future_status::ready;
+    server.dht_->cancelListen(key, token);
+    CPPUNIT_ASSERT(received);
+    {
+        std::lock_guard lock(context->lock);
+        data["epoch"] = context->epoch;
+        data["revision"] = std::to_string(context->revision);
+    }
+}
+
+void
+setExpirationRevision(dht::DhtProxyServer& server,
+                      const dht::InfoHash& key,
+                      const std::string& pushToken,
+                      dht::Value::Id id,
+                      std::map<std::string, std::string>& data)
+{
+    std::shared_ptr<dht::DhtProxyServer::PushSessionContext> context;
+    dht::PushType type;
+    std::string clientId;
+    std::string topic;
+    {
+        std::lock_guard lock(server.lockListener_);
+        auto& listener = server.pushListeners_.at(pushToken).listeners.at(key).front();
+        context = listener.sessionCtx;
+        type = listener.type;
+        clientId = listener.clientId;
+        topic = listener.topic;
+    }
+    std::shared_ptr<dht::Value> value;
+    {
+        std::lock_guard lock(context->lock);
+        value = std::make_shared<dht::Value>(context->values.at(id));
+    }
+    server.handlePushListen(key, pushToken, type, clientId, context, topic, {value}, true);
+    {
+        std::lock_guard lock(context->lock);
+        data["epoch"] = context->epoch;
+        data["revision"] = std::to_string(context->revision);
+    }
+}
 } // namespace
 #endif
 
@@ -347,6 +423,13 @@ DhtProxyTester::testPushListenValueEdit()
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {77, 1}
+    },
+                     push_data);
     sendPush(nodeClient, push_data, lk);
 
     // Wait for first add callback
@@ -372,6 +455,13 @@ DhtProxyTester::testPushListenValueEdit()
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {77, 2}
+    },
+                     push_data);
     sendPush(nodeClient, push_data, lk);
 
     // Wait for the second add
@@ -401,6 +491,7 @@ DhtProxyTester::testPushListenValueEdit()
 
     // Simulate expiration push notification
     push_data["exp"] = std::to_string((dht::Value::Id) 77);
+    setExpirationRevision(*serverProxy, key, clientConfig.push_token, 77, push_data);
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
@@ -484,6 +575,153 @@ DhtProxyTester::testResubscribeGetValues()
     CPPUNIT_ASSERT(firstVal_data == values.front()->data);
 }
 
+void
+DhtProxyTester::testInitializingSubscriptionCancellation()
+{
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    for (unsigned removal = 0; removal < 3; ++removal) {
+        auto key = dht::InfoHash::get("pending-snapshot-cancellation");
+        auto context = std::make_shared<dht::DhtProxyServer::PushSessionContext>("session");
+        std::weak_ptr<dht::DhtProxyServer::PushSessionContext> weakContext = context;
+        auto retained = std::make_shared<unsigned>(42);
+        std::weak_ptr<unsigned> weakRetained = retained;
+        auto completed = std::make_shared<std::promise<bool>>();
+        auto future = completed->get_future();
+        context->pendingSnapshots.emplace_back([context, retained, completed] {
+            std::lock_guard lock(context->lock);
+            completed->set_value(context->initialOk);
+        });
+        {
+            std::lock_guard lock(serverProxy->lockListener_);
+            auto& listeners = serverProxy->pushListeners_["push-token"].listeners[key];
+            listeners.emplace_back();
+            auto& listener = listeners.back();
+            listener.clientId = "client-id";
+            listener.sessionCtx = context;
+            std::promise<size_t> token;
+            token.set_value(0);
+            listener.internalToken = token.get_future();
+        }
+        context.reset();
+        retained.reset();
+        if (removal == 2) {
+            serverProxy.reset();
+        } else {
+            if (removal == 1) {
+                serverProxy->handleCancelPushListen(asio::error::operation_aborted, "push-token", key, "client-id");
+                CPPUNIT_ASSERT(future.wait_for(0s) == std::future_status::timeout);
+                CPPUNIT_ASSERT(!weakContext.expired());
+            }
+            serverProxy->handleCancelPushListen({}, "push-token", key, "client-id");
+        }
+        CPPUNIT_ASSERT(future.wait_for(5s) == std::future_status::ready);
+        CPPUNIT_ASSERT(!future.get());
+        CPPUNIT_ASSERT(weakContext.expired());
+        CPPUNIT_ASSERT(weakRetained.expired());
+    }
+#endif
+}
+
+void
+DhtProxyTester::testListenReadiness()
+{
+    auto key = dht::InfoHash::get("proxy-listen-readiness");
+    auto value = std::make_shared<dht::Value>("present-before-ready");
+    value->id = 42;
+    std::promise<bool> stored;
+    auto storedFuture = stored.get_future();
+    nodePeer.put(key, value, [&](bool ok) { stored.set_value(ok); });
+    CPPUNIT_ASSERT(storedFuture.wait_for(10s) == std::future_status::ready);
+    CPPUNIT_ASSERT(storedFuture.get());
+    nodeClient.run(0, clientConfig);
+    std::promise<bool> ready;
+    auto readyFuture = ready.get_future();
+    bool received = false;
+    auto token = nodeClient.listen(
+        key,
+        [&](const std::vector<dht::Sp<dht::Value>>& values, bool expired) {
+            if (!expired)
+                for (const auto& current : values)
+                    received = received || current->id == 42;
+            return true;
+        },
+        {},
+        {},
+        [&](bool ok) { ready.set_value(ok && received); });
+    CPPUNIT_ASSERT(readyFuture.wait_for(10s) == std::future_status::ready);
+    CPPUNIT_ASSERT(readyFuture.get());
+    std::promise<bool> attached;
+    auto attachedFuture = attached.get_future();
+    auto second = nodeClient.listen(
+        key,
+        [](const std::vector<dht::Sp<dht::Value>>&, bool) { return true; },
+        {},
+        {},
+        [&](bool ok) { attached.set_value(ok); });
+    CPPUNIT_ASSERT(attachedFuture.wait_for(10s) == std::future_status::ready);
+    CPPUNIT_ASSERT(attachedFuture.get());
+    nodeClient.cancelListen(key, token.get());
+    nodeClient.cancelListen(key, second.get());
+}
+
+void
+DhtProxyTester::testFullSubscribeReturnsValuesWithoutExistingListener()
+{
+    auto key = dht::InfoHash::get("full-subscribe-no-server-listener");
+    std::mutex mutex;
+    std::condition_variable cv;
+    bool stored = false;
+    nodePeer.put(key, dht::Value("stored-before-subscription"), [&](bool ok) {
+        std::lock_guard lock(mutex);
+        stored = ok;
+        cv.notify_all();
+    });
+    {
+        std::unique_lock lock(mutex);
+        CPPUNIT_ASSERT(cv.wait_for(lock, 10s, [&] { return stored; }));
+    }
+    auto subscribe = [&](const std::string& clientId, const Json::Value& sync = Json::Value(Json::objectValue)) {
+        asio::io_context context;
+        auto request = std::make_shared<dht::http::Request>(context,
+                                                            clientConfig.proxy_server + "/key/" + key.toString());
+        request->set_method(restinio::http_method_subscribe());
+        Json::Value root;
+        root["key"] = "review-token";
+        root["client_id"] = clientId;
+        root["session_id"] = "review-session";
+        root["platform"] = "android";
+        root["sync"] = sync;
+        request->set_body(Json::writeString(Json::StreamWriterBuilder {}, root));
+        unsigned statusCode = 0;
+        std::string body;
+        request->add_on_done_callback([&](const dht::http::Response& response) {
+            statusCode = response.status_code;
+            body = response.body;
+        });
+        request->send();
+        context.run();
+        CPPUNIT_ASSERT_EQUAL(200u, statusCode);
+        return body;
+    };
+    auto snapshot = subscribe("existing-client");
+    CPPUNIT_ASSERT(snapshot.find("\"data\"") != std::string::npos);
+    Json::Value marker;
+    std::istringstream lines(snapshot);
+    auto reader = std::unique_ptr<Json::CharReader>(Json::CharReaderBuilder {}.newCharReader());
+    for (std::string line; std::getline(lines, line);) {
+        std::string error;
+        CPPUNIT_ASSERT(reader->parse(line.data(), line.data() + line.size(), &marker, &error));
+    }
+    CPPUNIT_ASSERT(marker["sync"]["complete"].asBool());
+    auto validation = subscribe("existing-client", marker["sync"]);
+    CPPUNIT_ASSERT(validation.find("\"data\"") == std::string::npos);
+    std::string error;
+    CPPUNIT_ASSERT(reader->parse(validation.data(), validation.data() + validation.size(), &marker, &error));
+    CPPUNIT_ASSERT(marker["sync"]["complete"].asBool());
+    CPPUNIT_ASSERT(!marker["sync"]["snapshot"].asBool());
+    CPPUNIT_ASSERT(subscribe("new-client").find("\"data\"") != std::string::npos);
+}
+
 void
 DhtProxyTester::testPushNotification()
 {
@@ -556,6 +794,13 @@ DhtProxyTester::testPushNotification()
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {1337, 1}
+    },
+                     push_data);
     CPPUNIT_ASSERT(sendPush(nodeClient, push_data, lk) == dht::PushNotificationResult::Values);
 
     // Wait for client to receive the value
@@ -585,6 +830,13 @@ DhtProxyTester::testPushNotification()
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {1337, 2}
+    },
+                     push_data);
     CPPUNIT_ASSERT(sendPush(nodeClient, push_data, lk) == dht::PushNotificationResult::Values);
 
     // Wait for client to receive the edited value
@@ -599,6 +851,7 @@ DhtProxyTester::testPushNotification()
 
     // Simulate expiration push notification for the edited value
     push_data["exp"] = std::to_string((dht::Value::Id) 1337);
+    setExpirationRevision(*serverProxy, key, clientConfig.push_token, 1337, push_data);
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
@@ -1192,6 +1445,14 @@ DhtProxyTester::testPushMultiValueEditExpire()
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {100, 1},
+                         {200, 1}
+    },
+                     push_data);
     sendPush(nodeClient, push_data, lk);
 
     // Wait for both values to arrive
@@ -1225,6 +1486,14 @@ DhtProxyTester::testPushMultiValueEditExpire()
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
     push_data.erase("exp");
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {100, 2},
+                         {200, 1}
+    },
+                     push_data);
     sendPush(nodeClient, push_data, lk);
 
     // Wait for the edited value (id=100, seq=2) to arrive
@@ -1245,6 +1514,7 @@ DhtProxyTester::testPushMultiValueEditExpire()
 
     // Expire value B via push notification
     push_data["exp"] = std::to_string((dht::Value::Id) 200);
+    setExpirationRevision(*serverProxy, key, clientConfig.push_token, 200, push_data);
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
@@ -1343,6 +1613,15 @@ DhtProxyTester::testPushRefreshNoSpuriousExpire()
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {10, 1},
+                         {20, 1},
+                         {30, 1}
+    },
+                     push_data);
     sendPush(nodeClient, push_data, lk);
 
     // Wait for all 3 values to arrive
@@ -1369,6 +1648,15 @@ DhtProxyTester::testPushRefreshNoSpuriousExpire()
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
     push_data.erase("exp");
+    setValueRevision(*serverProxy,
+                     key,
+                     clientConfig.push_token,
+                     {
+                         {10, 1},
+                         {20, 1},
+                         {30, 1}
+    },
+                     push_data);
     sendPush(nodeClient, push_data, lk);
 
     // Wait a bit for any callbacks
@@ -1387,6 +1675,7 @@ DhtProxyTester::testPushRefreshNoSpuriousExpire()
     // then trigger another refresh. The removed value should be expired.
     // We'll use the "exp" field to explicitly expire value 20.
     push_data["exp"] = "20";
+    setExpirationRevision(*serverProxy, key, clientConfig.push_token, 20, push_data);
     push_data["t"] = std::to_string(
         std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
             .count());
@@ -1616,6 +1905,13 @@ DhtProxyTester::testPushRapidEditsNoPhantom()
         push_data["t"] = std::to_string(
             std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
                 .count());
+        setValueRevision(*serverProxy,
+                         key,
+                         clientConfig.push_token,
+                         {
+                             {500, uint16_t(seq)}
+        },
+                         push_data);
         sendPush(nodeClient, push_data, lk);
     }
 
@@ -1730,6 +2026,13 @@ DhtProxyTester::testPushRefreshAfterEditKeepsLatest()
         push_data["t"] = std::to_string(
             std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
                 .count());
+        setValueRevision(*serverProxy,
+                         key,
+                         clientConfig.push_token,
+                         {
+                             {600, 1}
+        },
+                         push_data);
         sendPush(nodeClient, push_data, lk);
     }
 
@@ -1764,6 +2067,13 @@ DhtProxyTester::testPushRefreshAfterEditKeepsLatest()
         push_data["t"] = std::to_string(
             std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
                 .count());
+        setValueRevision(*serverProxy,
+                         key,
+                         clientConfig.push_token,
+                         {
+                             {600, 3}
+        },
+                         push_data);
         sendPush(nodeClient, push_data, lk);
     }
 
@@ -1784,6 +2094,13 @@ DhtProxyTester::testPushRefreshAfterEditKeepsLatest()
         push_data["t"] = std::to_string(
             std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::system_clock::now().time_since_epoch())
                 .count());
+        setValueRevision(*serverProxy,
+                         key,
+                         clientConfig.push_token,
+                         {
+                             {600, 3}
+        },
+                         push_data);
         sendPush(nodeClient, push_data, lk);
     }
 

tests/test_dhtproxy.h

@@ -7,7 +7,26 @@
 #include <cppunit/extensions/HelperMacros.h>
 
 #include <opendht/dhtrunner.h>
+#include <opendht/http.h>
+#include <restinio/all.hpp>
+#include <restinio/tls.hpp>
+#include <json/json.h>
+
+#if defined(__clang__)
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wkeyword-macro"
+#elif defined(__GNUC__)
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wpedantic"
+#endif
+#define private public
 #include <opendht/dht_proxy_server.h>
+#undef private
+#if defined(__clang__)
+#pragma clang diagnostic pop
+#elif defined(__GNUC__)
+#pragma GCC diagnostic pop
+#endif
 
 namespace test {
 
@@ -16,9 +35,16 @@ class DhtProxyTester : public CppUnit::TestFixture
     CPPUNIT_TEST_SUITE(DhtProxyTester);
     CPPUNIT_TEST(testGetPut);
     CPPUNIT_TEST(testListen);
+    CPPUNIT_TEST(testListenReadiness);
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    CPPUNIT_TEST(testInitializingSubscriptionCancellation);
+#endif
     CPPUNIT_TEST(testListenValueEdit);
     CPPUNIT_TEST(testPushListenValueEdit);
     CPPUNIT_TEST(testResubscribeGetValues);
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    CPPUNIT_TEST(testFullSubscribeReturnsValuesWithoutExistingListener);
+#endif
     CPPUNIT_TEST(testPushNotification);
     CPPUNIT_TEST(testProxyServerClientFullChain);
     CPPUNIT_TEST(testPutGet40KChars);
@@ -51,6 +77,8 @@ public:
      * Test listen
      */
     void testListen();
+    void testListenReadiness();
+    void testInitializingSubscriptionCancellation();
     /**
      * Test that editing a value only triggers an add callback via proxy, no expire
      */
@@ -65,6 +93,7 @@ public:
      * it should retrieve existant values
      */
     void testResubscribeGetValues();
+    void testFullSubscribeReturnsValuesWithoutExistingListener();
     /**
      * Test push notification mechanism and OpValueCache
      */

tests/test_dhtproxy_client.cpp

@@ -31,6 +31,308 @@ CPPUNIT_TEST_SUITE_REGISTRATION(DhtProxyClientTester);
 
 using namespace dht;
 
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+namespace {
+void
+finishSync(DhtProxyClient& client,
+           const std::shared_ptr<dht::http::Request>& request,
+           const std::vector<Sp<Value>>& values,
+           uint64_t revision = 1,
+           bool complete = true,
+           bool snapshot = true,
+           const std::string& epoch = "test-epoch")
+{
+    Json::Value marker;
+    marker["sync"]["epoch"] = epoch;
+    marker["sync"]["revision"] = Json::UInt64(revision);
+    marker["sync"]["complete"] = complete;
+    marker["sync"]["snapshot"] = snapshot;
+    auto body = std::string {};
+    for (const auto& value : values)
+        body += Json::writeString(client.jsonBuilder_, value->toJson()) + "\n";
+    body += Json::writeString(client.jsonBuilder_, marker) + "\n";
+    request->cbs_.on_body(body.data(), body.size());
+    dht::http::Response response;
+    response.status_code = 200;
+    request->cbs_.on_state_change(dht::http::Request::State::DONE, response);
+    client.periodic(nullptr, 0, {}, clock::now());
+}
+} // namespace
+#endif
+
+void
+DhtProxyClientTester::testReadinessShutdown()
+{
+    for (bool push : {false, true}) {
+        DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", push ? "push-token" : "");
+        client.httpContext_.stop();
+        client.httpClientThread_.join();
+        client.proxyUrl_ = "http://127.0.0.1:1";
+        auto key = InfoHash::get("readiness-shutdown");
+        std::vector<bool> results;
+        auto values = [](const std::vector<Sp<Value>>&, bool) {
+            return true;
+        };
+        CPPUNIT_ASSERT(client.listen(key, values, {}, {}, [&](bool ok) {
+            results.push_back(ok);
+            CPPUNIT_ASSERT(client.getLocal(key, {}).empty());
+        }) != 0);
+        CPPUNIT_ASSERT(results.empty());
+        client.shutdown({}, true);
+        CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+        client.shutdown({}, true);
+        CPPUNIT_ASSERT_EQUAL(size_t(1), results.size());
+        CPPUNIT_ASSERT_EQUAL(size_t(0), client.listen(key, values, {}, {}, [&](bool ok) { results.push_back(ok); }));
+        CPPUNIT_ASSERT(results == std::vector<bool>({false, false}));
+    }
+}
+
+void
+DhtProxyClientTester::testReadinessReentrancy()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("ready-reentrancy");
+    std::vector<bool> results;
+    client.listen(
+        key,
+        [](const std::vector<Sp<Value>>&, bool) { return true; },
+        {},
+        {},
+        [&](bool ok) {
+            auto unlocked = client.searchLock_.try_lock();
+            if (unlocked)
+                client.searchLock_.unlock();
+            CPPUNIT_ASSERT(unlocked);
+            CPPUNIT_ASSERT(client.getLocal(key, {}).empty());
+            results.push_back(ok);
+        });
+    auto request = client.searches_.at(key).listeners.begin()->second.request;
+    std::string marker = "{\"sync\":{\"complete\":true}}\n";
+    request->cbs_.on_body(marker.data(), marker.size());
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(results == std::vector<bool>({true}));
+}
+
+void
+DhtProxyClientTester::testCachedReadinessDispatch()
+{
+    for (unsigned action = 0; action < 3; ++action) {
+        DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "");
+        client.httpContext_.stop();
+        client.httpClientThread_.join();
+        client.proxyUrl_ = "http://127.0.0.1:1";
+        auto key = InfoHash::get("cached-readiness-dispatch");
+        auto values = [](const std::vector<Sp<Value>>&, bool) {
+            return true;
+        };
+        auto first = client.listen(key, values, {}, {});
+        CPPUNIT_ASSERT(first != 0);
+        auto value = std::make_shared<Value>("cached");
+        value->id = 42;
+        auto request = client.searches_.at(key).listeners.begin()->second.request;
+        auto body = Json::writeString(client.jsonBuilder_, value->toJson()) + "\n{\"sync\":{\"complete\":true}}\n";
+        request->cbs_.on_body(body.data(), body.size());
+        client.periodic(nullptr, 0, {}, clock::now());
+        CPPUNIT_ASSERT(client.getLocalById(key, 42));
+
+        std::vector<bool> results;
+        size_t delivered = 0;
+        auto attached = client.listen(
+            key,
+            [&](const std::vector<Sp<Value>>& cached, bool expired) {
+                if (!expired)
+                    delivered += cached.size();
+                return true;
+            },
+            {},
+            {},
+            [&](bool ok) {
+                results.emplace_back(ok);
+                if (ok) {
+                    CPPUNIT_ASSERT(client.getLocalById(key, 42));
+                    auto nested = client.listen(key, values, {}, {});
+                    CPPUNIT_ASSERT(nested != 0);
+                    CPPUNIT_ASSERT(client.cancelListen(key, nested));
+                } else
+                    CPPUNIT_ASSERT(client.getLocal(key, {}).empty());
+            });
+        CPPUNIT_ASSERT(attached != 0);
+        CPPUNIT_ASSERT_EQUAL(size_t(1), delivered);
+        CPPUNIT_ASSERT(results.empty());
+        CPPUNIT_ASSERT_EQUAL(size_t(1), client.readinessCallbacks_.size());
+        if (action == 1)
+            CPPUNIT_ASSERT(client.cancelListen(key, attached));
+        else if (action == 2) {
+            client.shutdown({}, true);
+            CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+        }
+        client.periodic(nullptr, 0, {}, clock::now());
+        if (action == 1)
+            CPPUNIT_ASSERT(results.empty());
+        else
+            CPPUNIT_ASSERT(results == std::vector<bool>({action == 0}));
+        client.shutdown({}, true);
+        CPPUNIT_ASSERT_EQUAL(action == 1 ? size_t(0) : size_t(1), results.size());
+    }
+}
+
+void
+DhtProxyClientTester::testReadinessCancelsAnotherListener()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("ready-cancel-neighbor");
+    size_t second = 0;
+    unsigned firstCalls = 0;
+    unsigned secondCalls = 0;
+    auto values = [](const std::vector<Sp<Value>>&, bool) {
+        return true;
+    };
+    client.listen(key, values, {}, {}, [&](bool ok) {
+        auto unlocked = client.searchLock_.try_lock();
+        if (unlocked)
+            client.searchLock_.unlock();
+        CPPUNIT_ASSERT(unlocked);
+        CPPUNIT_ASSERT(ok);
+        ++firstCalls;
+        CPPUNIT_ASSERT(client.cancelListen(key, second));
+    });
+    second = client.listen(key, values, {}, {}, [&](bool) { ++secondCalls; });
+    auto request = client.searches_.at(key).listeners.begin()->second.request;
+    std::string marker = "{\"sync\":{\"complete\":true}}\n";
+    request->cbs_.on_body(marker.data(), marker.size());
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT_EQUAL(1u, firstCalls);
+    CPPUNIT_ASSERT_EQUAL(0u, secondCalls);
+}
+
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+void
+DhtProxyClientTester::testRejectsUnrevisionedPushes()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("unrevisioned-push-rejection");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("obsolete");
+    value->id = 42;
+    finishSync(client, listener.request, {value}, 5);
+    auto synchronized = listener.request;
+    auto requestCount = client.requests_.size();
+    for (const auto& notification : std::vector<std::map<std::string, std::string>> {
+             {{"key", key.toString()}, {"t", "10000"}},
+             {{"key", key.toString()}, {"expired", "true"}, {"ids", "42"}},
+             {{"key", key.toString()}, {"epoch", "test-epoch"}},
+             {{"key", key.toString()}, {"revision", "6"}}
+    }) {
+        CPPUNIT_ASSERT(client.pushNotificationReceived(notification) == PushNotificationResult::Error);
+        CPPUNIT_ASSERT(listener.request == synchronized);
+        CPPUNIT_ASSERT_EQUAL(requestCount, client.requests_.size());
+        CPPUNIT_ASSERT_EQUAL(uint64_t(5), listener.syncRevision);
+        CPPUNIT_ASSERT(listener.opstate->ok.load());
+        CPPUNIT_ASSERT(client.getLocalById(key, 42));
+    }
+    CPPUNIT_ASSERT(
+        client.pushNotificationReceived({
+            {"key",      key.toString()},
+            {"epoch",    "test-epoch"  },
+            {"revision", "6"           },
+            {"expired",  "true"        },
+            {"ids",      "42"          }
+    })
+        == PushNotificationResult::ValuesExpired);
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(!client.getLocalById(key, 42));
+    client.restartListeners({});
+    auto validation = listener.request;
+    CPPUNIT_ASSERT(client.pushNotificationReceived({
+                       {"key", key.toString()}
+    })
+                   == PushNotificationResult::Error);
+    finishSync(client, validation, {}, 6, true, false);
+    CPPUNIT_ASSERT(listener.request == validation);
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+    CPPUNIT_ASSERT(!client.getLocalById(key, 42));
+}
+
+void
+DhtProxyClientTester::testInitialReadinessFailures()
+{
+    for (unsigned failure = 0; failure < 4; ++failure) {
+        bool push = failure == 1 || failure == 3;
+        DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", push ? "push-token" : "");
+        client.httpContext_.stop();
+        client.httpClientThread_.join();
+        client.proxyUrl_ = "http://127.0.0.1:1";
+        auto key = InfoHash::get("initial-readiness-failure");
+        std::vector<bool> results;
+        CPPUNIT_ASSERT(client.listen(
+                           key,
+                           [](const std::vector<Sp<Value>>&, bool) { return true; },
+                           {},
+                           {},
+                           [&](bool ok) { results.push_back(ok); })
+                       != 0);
+        auto& listener = client.searches_.at(key).listeners.begin()->second;
+        auto request = listener.request;
+        if (failure == 0) {
+            std::string marker = "{\"sync\":{\"complete\":false}}\n";
+            request->cbs_.on_body(marker.data(), marker.size());
+        } else if (failure == 3) {
+            finishSync(client, request, {}, 1, false);
+        } else {
+            dht::http::Response response;
+            response.status_code = 503;
+            request->cbs_.on_state_change(dht::http::Request::State::DONE, response);
+        }
+        client.periodic(nullptr, 0, {}, clock::now());
+        CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+        client.shutdown({}, true);
+        CPPUNIT_ASSERT_EQUAL(size_t(1), results.size());
+    }
+}
+
+void
+DhtProxyClientTester::testReadinessRecoveryIsOneShot()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("readiness-recovery");
+    auto values = [](const std::vector<Sp<Value>>&, bool) {
+        return true;
+    };
+    std::vector<bool> first;
+    CPPUNIT_ASSERT(client.listen(key, values, {}, {}, [&](bool ok) { first.push_back(ok); }) != 0);
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto request = listener.request;
+    dht::http::Response response;
+    response.status_code = 503;
+    request->cbs_.on_state_change(dht::http::Request::State::DONE, response);
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(first == std::vector<bool>({false}));
+    std::vector<bool> second;
+    CPPUNIT_ASSERT(client.listen(key, values, {}, {}, [&](bool ok) { second.push_back(ok); }) != 0);
+    CPPUNIT_ASSERT(second.empty());
+    client.handleResubscribe({}, key, 1, listener.opstate);
+    finishSync(client, listener.request, {}, 1);
+    CPPUNIT_ASSERT(first == std::vector<bool>({false}));
+    CPPUNIT_ASSERT(second == std::vector<bool>({true}));
+    client.shutdown({}, true);
+    CPPUNIT_ASSERT_EQUAL(size_t(1), first.size());
+    CPPUNIT_ASSERT_EQUAL(size_t(1), second.size());
+}
+#endif
+
 void
 DhtProxyClientTester::testResubscribeUsesKeyRoute()
 {
@@ -39,6 +341,8 @@ DhtProxyClientTester::testResubscribeUsesKeyRoute()
 
     DhtProxyClient::Listener listener {ValueCallback([](const std::vector<Sp<Value>>&, bool) { return true; })};
     listener.opstate = std::make_shared<DhtProxyClient::OperationState>();
+    listener.needsRecovery = false;
+    listener.syncEpoch = "test-epoch";
     listener.cb = [](const std::vector<Sp<Value>>&, bool, system_clock::time_point) {
         return true;
     };
@@ -48,8 +352,7 @@ DhtProxyClientTester::testResubscribeUsesKeyRoute()
 
     CPPUNIT_ASSERT(listener.request);
     CPPUNIT_ASSERT_EQUAL(std::string("/key/") + key.toString(), std::string(listener.request->header_.request_target()));
-    CPPUNIT_ASSERT(listener.request->body_.find("\"refresh\"") != std::string::npos);
-    CPPUNIT_ASSERT(listener.request->body_.find("true") != std::string::npos);
+    CPPUNIT_ASSERT(listener.request->body_.find("test-epoch") != std::string::npos);
 #else
     CPPUNIT_ASSERT(true);
 #endif
@@ -73,6 +376,9 @@ DhtProxyClientTester::testSetPushNotificationTokenResubscribesWithNewToken()
 
     auto& listener = it->second;
     listener.opstate = std::make_shared<DhtProxyClient::OperationState>();
+    listener.needsRecovery = false;
+    listener.syncEpoch = "test-epoch";
+    listener.syncRevision = 4;
     listener.cb = [](const std::vector<Sp<Value>>&, bool, system_clock::time_point) {
         return true;
     };
@@ -83,11 +389,644 @@ DhtProxyClientTester::testSetPushNotificationTokenResubscribesWithNewToken()
     CPPUNIT_ASSERT_EQUAL(std::string("/key/") + key.toString(), std::string(listener.request->header_.request_target()));
     CPPUNIT_ASSERT(listener.request->body_.find("new-token") != std::string::npos);
     CPPUNIT_ASSERT(listener.request->body_.find("old-token") == std::string::npos);
-    CPPUNIT_ASSERT(listener.request->body_.find("\"refresh\"") != std::string::npos);
-    CPPUNIT_ASSERT(listener.request->body_.find("true") != std::string::npos);
+    CPPUNIT_ASSERT(listener.request->body_.find("test-epoch") != std::string::npos);
 #else
     CPPUNIT_ASSERT(true);
 #endif
 }
 
+void
+DhtProxyClientTester::testRestartListenersFetchesValuesAfterFailedSubscribe()
+{
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    DhtProxyClient client({}, {}, [] {}, "http://127.0.0.1:8080", "OpenDHT-Test", "client-id", "push-token");
+
+    auto addListener = [&](const InfoHash& key, bool ok) -> DhtProxyClient::Listener& {
+        auto& search = client.searches_[key];
+        auto [it, inserted] = search.listeners.emplace(std::piecewise_construct,
+                                                       std::forward_as_tuple(1),
+                                                       std::forward_as_tuple(ValueCallback(
+                                                           [](const std::vector<Sp<Value>>&, bool) { return true; })));
+        CPPUNIT_ASSERT(inserted);
+        auto& listener = it->second;
+        listener.opstate = std::make_shared<DhtProxyClient::OperationState>();
+        listener.opstate->ok = ok;
+        listener.cb = [](const std::vector<Sp<Value>>&, bool, system_clock::time_point) {
+            return true;
+        };
+        return listener;
+    };
+
+    // The subscription of this listener failed (e.g. no network): values stored on the
+    // key since then were never received.
+    auto failedKey = InfoHash::get("proxy-client-failed-subscribe");
+    auto& failed = addListener(failedKey, false);
+    auto healthyKey = InfoHash::get("proxy-client-healthy-subscribe");
+    auto& healthy = addListener(healthyKey, true);
+
+    client.restartListeners({});
+
+    CPPUNIT_ASSERT(failed.request);
+    CPPUNIT_ASSERT_EQUAL(std::string("/key/") + failedKey.toString(),
+                         std::string(failed.request->header_.request_target()));
+    CPPUNIT_ASSERT(failed.request->body_.find("push-token") != std::string::npos);
+    // Without "refresh", the server answers with the values it stores for the key.
+    CPPUNIT_ASSERT(failed.request->body_.find("\"refresh\"") == std::string::npos);
+
+    CPPUNIT_ASSERT(healthy.request);
+    CPPUNIT_ASSERT(healthy.request->body_.find("\"sync\"") != std::string::npos);
+#else
+    CPPUNIT_ASSERT(true);
+#endif
+}
+
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+void
+DhtProxyClientTester::testWrongSessionPushDoesNotTouchSubscriptions()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("wrong-session-isolation");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("current-session-value");
+    value->id = 42;
+    finishSync(client, listener.request, {value}, 5);
+    client.statusIpv4_ = NodeStatus::Disconnected;
+    client.statusIpv6_ = NodeStatus::Disconnected;
+    client.launchConnectedCbs_ = false;
+    auto request = listener.request;
+    auto generation = listener.opstate->generation.load();
+    auto expiration = listener.refreshSubscriberTimer->expiry();
+    std::vector<std::map<std::string, std::string>> notifications {
+        {{"s", "old-session"}, {"key", key.toString()}, {"epoch", "old-epoch"}, {"revision", "99"}},
+        {{"s", "old-session"}, {"key", key.toString()}, {"exp", "42"}},
+        {{"s", "old-session"}, {"timeout", key.toString()}},
+        {{"s", "old-session"}, {"timeout", key.toString()}, {"vid", "42"}}
+    };
+    for (const auto& notification : notifications) {
+        CPPUNIT_ASSERT(client.pushNotificationReceived(notification) == PushNotificationResult::IgnoredWrongSession);
+        CPPUNIT_ASSERT(listener.request == request);
+        CPPUNIT_ASSERT_EQUAL(generation, listener.opstate->generation.load());
+        CPPUNIT_ASSERT(!listener.needsRecovery);
+        CPPUNIT_ASSERT(listener.opstate->ok.load());
+        CPPUNIT_ASSERT(listener.refreshSubscriberTimer->expiry() == expiration);
+        CPPUNIT_ASSERT_EQUAL(std::string("test-epoch"), listener.syncEpoch);
+        CPPUNIT_ASSERT_EQUAL(uint64_t(5), listener.syncRevision);
+        CPPUNIT_ASSERT(client.getLocalById(key, 42));
+        CPPUNIT_ASSERT(client.statusIpv4_ == NodeStatus::Disconnected);
+        CPPUNIT_ASSERT(client.statusIpv6_ == NodeStatus::Disconnected);
+        CPPUNIT_ASSERT(!client.launchConnectedCbs_.load());
+        CPPUNIT_ASSERT(client.callbacks_.empty());
+    }
+}
+
+void
+DhtProxyClientTester::testNewSessionSynchronizesIndependently()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("new-session-independent-sync");
+    CPPUNIT_ASSERT(client.pushNotificationReceived({
+                       {"s",   "old-session" },
+                       {"key", key.toString()}
+    })
+                   == PushNotificationResult::IgnoredWrongSession);
+    CPPUNIT_ASSERT(client.searches_.empty());
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    Json::Value body;
+    std::string error;
+    auto reader = std::unique_ptr<Json::CharReader>(Json::CharReaderBuilder {}.newCharReader());
+    const auto& requestBody = listener.request->body_;
+    CPPUNIT_ASSERT(reader->parse(requestBody.data(), requestBody.data() + requestBody.size(), &body, &error));
+    CPPUNIT_ASSERT_EQUAL(client.pushSessionId_, body["session_id"].asString());
+    CPPUNIT_ASSERT(body["sync"].isObject() && body["sync"].empty());
+    auto value = std::make_shared<Value>("new-session-snapshot");
+    value->id = 43;
+    finishSync(client, listener.request, {value});
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+    CPPUNIT_ASSERT(!listener.needsRecovery);
+    CPPUNIT_ASSERT(client.getLocalById(key, 43)->data == value->data);
+}
+
+void
+DhtProxyClientTester::testFailedTimerRetryFetchesValues()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("failed-timer-retry");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    listener.opstate->ok = false;
+    client.handleResubscribe({}, key, 1, listener.opstate);
+    CPPUNIT_ASSERT(listener.request->body_.find("\"refresh\"") == std::string::npos);
+}
+
+void
+DhtProxyClientTester::testFullRetryRemovesAbsentValues()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("full-retry-absent-values");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("expired-while-offline");
+    value->id = 42;
+    listener.cb({value}, false, system_clock::time_point(std::chrono::seconds(10)));
+    listener.opstate->ok = false;
+    client.restartListeners({});
+    auto request = listener.request;
+    finishSync(client, request, {});
+    CPPUNIT_ASSERT(!client.getLocalById(key, value->id));
+}
+
+void
+DhtProxyClientTester::testStreamingFallbackExpiresSnapshot()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("full-retry-push-ordering");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("live-value");
+    value->id = 42;
+    listener.cb({value}, false, system_clock::time_point(std::chrono::seconds(10)));
+    listener.opstate->ok = false;
+    client.restartListeners({});
+    finishSync(client, listener.request, {value}, 10);
+    client.deviceKey_.clear();
+    client.restartListeners({});
+    auto expired = value->toJson();
+    expired["expired"] = true;
+    auto body = Json::writeString(client.jsonBuilder_, expired) + "\n";
+    listener.request->cbs_.on_body(body.data(), body.size());
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(!client.getLocalById(key, value->id));
+}
+
+void
+DhtProxyClientTester::testLateRetryResponseDoesNotResurrectExpiredValue()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("full-retry-late-response");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("already-expired-value");
+    value->id = 42;
+    finishSync(client, listener.request, {value}, 10);
+    listener.opstate->ok = false;
+    client.restartListeners({});
+    auto staleRequest = listener.request;
+    client.pushNotificationReceived({
+        {"key",      key.toString()},
+        {"exp",      "42"          },
+        {"epoch",    "test-epoch"  },
+        {"revision", "11"          }
+    });
+    finishSync(client, staleRequest, {value}, 10);
+    client.handleResubscribe({}, key, 1, listener.opstate);
+    finishSync(client, listener.request, {}, 11);
+    finishSync(client, staleRequest, {value}, 10);
+    CPPUNIT_ASSERT(!client.getLocalById(key, value->id));
+}
+
+void
+DhtProxyClientTester::testTruncatedRecoveryRemainsFailed()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("truncated-full-retry");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    listener.opstate->ok = false;
+    client.restartListeners({});
+    auto request = listener.request;
+    request->response_.status_code = 200;
+    request->cbs_.on_body("{\"id\":", 6);
+    request->terminate(asio::error::connection_reset);
+    CPPUNIT_ASSERT(!listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testTruncatedRecoveryAtEofFails()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("truncated-recovery-eof");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto request = listener.request;
+    request->init_parser();
+    std::string response = "HTTP/1.1 200 OK\r\nContent-Length: 10\r\n\r\n{";
+    CPPUNIT_ASSERT(llhttp_execute(request->parser_.get(), response.data(), response.size()) == HPE_OK);
+    request->handle_response(asio::error::eof, 0);
+    CPPUNIT_ASSERT(!listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testMissingCompletionDoesNotReconcile()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("missing-completion");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("keep-value");
+    value->id = 42;
+    finishSync(client, listener.request, {value});
+    client.restartListeners({});
+    auto request = listener.request;
+    request->cbs_.on_body("{}\n", 3);
+    dht::http::Response response;
+    response.status_code = 200;
+    request->cbs_.on_state_change(dht::http::Request::State::DONE, response);
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(!listener.opstate->ok.load());
+    CPPUNIT_ASSERT(client.getLocalById(key, 42));
+    CPPUNIT_ASSERT(listener.refreshSubscriberTimer->expiry() <= system_clock::now() + std::chrono::seconds(2));
+}
+
+void
+DhtProxyClientTester::testSnapshotExpirationIgnoresClockRollback()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("snapshot-clock-rollback");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("expired-value");
+    value->id = 42;
+    listener.cb({value}, false, system_clock::now() + std::chrono::hours(1));
+    finishSync(client, listener.request, {});
+    CPPUNIT_ASSERT(!client.getLocalById(key, 42));
+}
+
+void
+DhtProxyClientTester::testSupersededFailureDoesNotPoisonSuccessfulRetry()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("superseded-full-retry");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto oldRequest = listener.request;
+    oldRequest->response_.status_code = 503;
+    oldRequest->terminate(asio::error::eof);
+    client.periodic(nullptr, 0, {}, clock::now());
+    client.restartListeners({});
+    auto newRequest = listener.request;
+    CPPUNIT_ASSERT(newRequest != oldRequest);
+    finishSync(client, newRequest, {});
+    dht::http::Response response;
+    response.status_code = 503;
+    oldRequest->cbs_.on_state_change(dht::http::Request::State::DONE, response);
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testSyncCoalescesConnectivityChanges()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("coalesced-sync");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto request = listener.request;
+    client.restartListeners({});
+    client.restartListeners({});
+    CPPUNIT_ASSERT(listener.request == request);
+    finishSync(client, request, {}, 5);
+    CPPUNIT_ASSERT(listener.request == request);
+    client.restartListeners({});
+    CPPUNIT_ASSERT(listener.request != request);
+    auto body = listener.request->body_;
+    CPPUNIT_ASSERT(body.find("test-epoch") != std::string::npos);
+    CPPUNIT_ASSERT(body.find("\"revision\":5") != std::string::npos);
+    auto validation = listener.request;
+    client.restartListeners({});
+    client.restartListeners({});
+    CPPUNIT_ASSERT(listener.request == validation);
+    finishSync(client, listener.request, {}, 5, true, false);
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testTokenRotationDuringRecovery()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "old-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    client.statusIpv4_ = NodeStatus::Connected;
+    auto key = InfoHash::get("rotation-during-recovery");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto recovery = listener.request;
+    client.setPushNotificationToken("new-token");
+    CPPUNIT_ASSERT(listener.request == recovery);
+    finishSync(client, recovery, {}, 1);
+    CPPUNIT_ASSERT(listener.request != recovery);
+    CPPUNIT_ASSERT(listener.request->body_.find("new-token") != std::string::npos);
+    finishSync(client, listener.request, {}, 1, true, false);
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testTokenRotationBackToPreviousEpoch()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "first-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    client.statusIpv4_ = NodeStatus::Connected;
+    auto key = InfoHash::get("rotation-back-to-previous-epoch");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    finishSync(client, listener.request, {}, 1);
+    client.setPushNotificationToken("second-token");
+    finishSync(client, listener.request, {}, 1, true, true, "second-epoch");
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+    client.setPushNotificationToken("first-token");
+    finishSync(client, listener.request, {}, 2);
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+    CPPUNIT_ASSERT_EQUAL(std::string("test-epoch"), listener.syncEpoch);
+}
+
+void
+DhtProxyClientTester::testAuthoritativeEpochAfterStalePush()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("authoritative-epoch-after-stale-push");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    finishSync(client, listener.request, {}, 1);
+    client.restartListeners({});
+    auto stale = listener.request;
+    client.pushNotificationReceived({
+        {"key",      key.toString()     },
+        {"epoch",    "intervening-epoch"},
+        {"revision", "1"                }
+    });
+    finishSync(client, stale, {}, 1);
+    finishSync(client, listener.request, {}, 1, true, true, "current-epoch");
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+    CPPUNIT_ASSERT_EQUAL(std::string("current-epoch"), listener.syncEpoch);
+    auto current = listener.request;
+    client.pushNotificationReceived({
+        {"key",      key.toString()     },
+        {"epoch",    "intervening-epoch"},
+        {"revision", "2"                }
+    });
+    CPPUNIT_ASSERT(listener.request != current);
+    CPPUNIT_ASSERT(listener.request->body_.find("current-epoch") != std::string::npos);
+    finishSync(client, listener.request, {}, 1, true, false, "current-epoch");
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testSyncEditsAndDuplicateValues()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("snapshot-edits");
+    unsigned additions = 0;
+    unsigned expirations = 0;
+    client.listen(key,
+                  [&](const std::vector<Sp<Value>>& values, bool expired) {
+                      (expired ? expirations : additions) += values.size();
+                      return true;
+                  },
+                  {},
+                  {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto identity = dht::crypto::generateIdentity("SnapshotEditAuthor");
+    auto original = std::make_shared<Value>("original");
+    original->id = 42;
+    original->seq = 1;
+    original->sign(*identity.first);
+    finishSync(client, listener.request, {original}, 1);
+    client.restartListeners({});
+    finishSync(client, listener.request, {}, 1, true, false);
+    CPPUNIT_ASSERT_EQUAL(1u, additions);
+    auto edited = std::make_shared<Value>("edited");
+    edited->id = 42;
+    edited->seq = 2;
+    edited->sign(*identity.first);
+    client.pushNotificationReceived({
+        {"key",      key.toString()},
+        {"epoch",    "test-epoch"  },
+        {"revision", "2"           }
+    });
+    finishSync(client, listener.request, {edited}, 2);
+    CPPUNIT_ASSERT_EQUAL(2u, additions);
+    CPPUNIT_ASSERT_EQUAL(0u, expirations);
+    CPPUNIT_ASSERT(client.getLocalById(key, 42)->data == edited->data);
+    client.pushNotificationReceived({
+        {"key",      key.toString()},
+        {"epoch",    "test-epoch"  },
+        {"revision", "1"           }
+    });
+    CPPUNIT_ASSERT(client.getLocalById(key, 42)->data == edited->data);
+}
+
+void
+DhtProxyClientTester::testIncompleteSnapshotDoesNotReconcile()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("incomplete-snapshot");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto value = std::make_shared<Value>("keep-until-complete");
+    value->id = 42;
+    finishSync(client, listener.request, {value}, 1);
+    listener.opstate->ok = false;
+    client.restartListeners({});
+    finishSync(client, listener.request, {}, 2, false);
+    CPPUNIT_ASSERT(client.getLocalById(key, 42));
+    CPPUNIT_ASSERT(!listener.opstate->ok.load());
+    client.handleResubscribe({}, key, 1, listener.opstate);
+    auto recovery = listener.request;
+    finishSync(client, recovery, {}, 2);
+    CPPUNIT_ASSERT(!client.getLocalById(key, 42));
+}
+void
+DhtProxyClientTester::testSubscriptionDeadline()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    client.subscriptionRequestTimeout_ = std::chrono::seconds(0);
+    auto key = InfoHash::get("subscription-deadline");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto request = listener.request;
+    CPPUNIT_ASSERT(listener.opstate->inFlight.load());
+    client.httpContext_.restart();
+    client.httpContext_.poll();
+    client.httpContext_.stop();
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(!listener.opstate->inFlight.load());
+    CPPUNIT_ASSERT(!listener.opstate->ok.load());
+    CPPUNIT_ASSERT(request->finishing_.load());
+    CPPUNIT_ASSERT(listener.needsRecovery);
+    CPPUNIT_ASSERT(listener.refreshSubscriberTimer->expiry() <= system_clock::now() + std::chrono::seconds(2));
+    client.handleResubscribe({}, key, 1, listener.opstate);
+    CPPUNIT_ASSERT(listener.request != request);
+}
+
+void
+DhtProxyClientTester::testInvalidPushRevision()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("invalid-push-revision");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    finishSync(client, listener.request, {}, 5);
+    auto request = listener.request;
+    for (const auto& revision : {"-1", "+6", " 6", "6 ", "6x", "", "18446744073709551616"}) {
+        CPPUNIT_ASSERT(client.pushNotificationReceived({
+                           {"key",      key.toString()},
+                           {"epoch",    "test-epoch"  },
+                           {"revision", revision      }
+        })
+                       == PushNotificationResult::Error);
+        CPPUNIT_ASSERT_EQUAL(uint64_t(5), listener.observedRevision);
+        CPPUNIT_ASSERT_EQUAL(uint64_t(5), listener.syncRevision);
+        CPPUNIT_ASSERT(listener.request == request);
+        CPPUNIT_ASSERT(!listener.needsRecovery);
+    }
+    client.pushNotificationReceived({
+        {"key",      key.toString()},
+        {"epoch",    "test-epoch"  },
+        {"revision", "6"           }
+    });
+    CPPUNIT_ASSERT(listener.request != request);
+    finishSync(client, listener.request, {}, 6);
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+}
+
+void
+DhtProxyClientTester::testRecoveryBackoff()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("recovery-backoff");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    for (unsigned attempt = 0; attempt < 8; ++attempt) {
+        auto request = listener.request;
+        auto generation = listener.opstate->generation.load();
+        Json::Value marker;
+        marker["sync"]["epoch"] = "test-epoch";
+        marker["sync"]["revision"] = Json::UInt64(1);
+        marker["sync"]["snapshot"] = true;
+        marker["sync"]["complete"] = false;
+        marker["sync"]["retry"] = Json::Value(Json::objectValue);
+        auto body = Json::writeString(client.jsonBuilder_, marker) + "\n";
+        request->cbs_.on_body(body.data(), body.size());
+        dht::http::Response response;
+        response.status_code = 200;
+        request->cbs_.on_state_change(dht::http::Request::State::DONE, response);
+        client.periodic(nullptr, 0, {}, clock::now());
+        CPPUNIT_ASSERT(listener.request == request);
+        CPPUNIT_ASSERT_EQUAL(generation, listener.opstate->generation.load());
+        CPPUNIT_ASSERT(!listener.opstate->inFlight.load());
+        CPPUNIT_ASSERT(!listener.opstate->ok.load());
+        CPPUNIT_ASSERT(listener.retryDelay <= std::chrono::seconds(60));
+        client.handleResubscribe({}, key, 1, listener.opstate);
+        CPPUNIT_ASSERT(listener.request != request);
+    }
+    CPPUNIT_ASSERT(listener.retryDelay == std::chrono::seconds(60));
+    finishSync(client, listener.request, {}, 1);
+    CPPUNIT_ASSERT(listener.retryDelay == std::chrono::seconds(1));
+    CPPUNIT_ASSERT(listener.refreshSubscriberTimer->expiry() > system_clock::now() + std::chrono::minutes(1));
+}
+
+void
+DhtProxyClientTester::testContiguousExpiration()
+{
+    DhtProxyClient client({}, {}, [] {}, "", "OpenDHT-Test", "client-id", "push-token");
+    client.httpContext_.stop();
+    client.httpClientThread_.join();
+    client.proxyUrl_ = "http://127.0.0.1:1";
+    auto key = InfoHash::get("contiguous-expiration");
+    client.listen(key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {});
+    auto& listener = client.searches_.at(key).listeners.begin()->second;
+    auto first = std::make_shared<Value>("first");
+    first->id = 42;
+    auto second = std::make_shared<Value>("second");
+    second->id = 43;
+    finishSync(client, listener.request, {first, second}, 5);
+    CPPUNIT_ASSERT(listener.cache.onValuesAdded({first, second}, system_clock::now()));
+    auto request = listener.request;
+    client.pushNotificationReceived({
+        {"key",      key.toString()},
+        {"epoch",    "test-epoch"  },
+        {"revision", "6"           },
+        {"expired",  "true"        },
+        {"ids",      "42,43"       }
+    });
+    client.periodic(nullptr, 0, {}, clock::now());
+    CPPUNIT_ASSERT(listener.request == request);
+    CPPUNIT_ASSERT(!client.getLocalById(key, 42));
+    CPPUNIT_ASSERT(!client.getLocalById(key, 43));
+    CPPUNIT_ASSERT_EQUAL(uint64_t(6), listener.syncRevision);
+    client.resubscribe(key, 1, listener);
+    finishSync(client, listener.request, {}, 6, true, false);
+    CPPUNIT_ASSERT(!client.getLocalById(key, 42));
+    CPPUNIT_ASSERT(!client.getLocalById(key, 43));
+    CPPUNIT_ASSERT(listener.opstate->ok.load());
+    request = listener.request;
+    client.pushNotificationReceived({
+        {"key",      key.toString()},
+        {"epoch",    "test-epoch"  },
+        {"revision", "8"           },
+        {"exp",      "42"          }
+    });
+    CPPUNIT_ASSERT(listener.request != request);
+    CPPUNIT_ASSERT(listener.needsRecovery);
+}
+#endif
+
 } // namespace test

tests/test_dhtproxy_client.h

@@ -12,11 +12,73 @@ class DhtProxyClientTester : public CppUnit::TestFixture
     CPPUNIT_TEST_SUITE(DhtProxyClientTester);
     CPPUNIT_TEST(testResubscribeUsesKeyRoute);
     CPPUNIT_TEST(testSetPushNotificationTokenResubscribesWithNewToken);
+    CPPUNIT_TEST(testRestartListenersFetchesValuesAfterFailedSubscribe);
+    CPPUNIT_TEST(testReadinessShutdown);
+    CPPUNIT_TEST(testReadinessReentrancy);
+    CPPUNIT_TEST(testCachedReadinessDispatch);
+    CPPUNIT_TEST(testReadinessCancelsAnotherListener);
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    CPPUNIT_TEST(testInitialReadinessFailures);
+    CPPUNIT_TEST(testReadinessRecoveryIsOneShot);
+    CPPUNIT_TEST(testWrongSessionPushDoesNotTouchSubscriptions);
+    CPPUNIT_TEST(testNewSessionSynchronizesIndependently);
+    CPPUNIT_TEST(testFailedTimerRetryFetchesValues);
+    CPPUNIT_TEST(testFullRetryRemovesAbsentValues);
+    CPPUNIT_TEST(testStreamingFallbackExpiresSnapshot);
+    CPPUNIT_TEST(testLateRetryResponseDoesNotResurrectExpiredValue);
+    CPPUNIT_TEST(testTruncatedRecoveryRemainsFailed);
+    CPPUNIT_TEST(testTruncatedRecoveryAtEofFails);
+    CPPUNIT_TEST(testMissingCompletionDoesNotReconcile);
+    CPPUNIT_TEST(testSnapshotExpirationIgnoresClockRollback);
+    CPPUNIT_TEST(testSupersededFailureDoesNotPoisonSuccessfulRetry);
+    CPPUNIT_TEST(testSyncCoalescesConnectivityChanges);
+    CPPUNIT_TEST(testTokenRotationDuringRecovery);
+    CPPUNIT_TEST(testTokenRotationBackToPreviousEpoch);
+    CPPUNIT_TEST(testAuthoritativeEpochAfterStalePush);
+    CPPUNIT_TEST(testSyncEditsAndDuplicateValues);
+    CPPUNIT_TEST(testIncompleteSnapshotDoesNotReconcile);
+    CPPUNIT_TEST(testSubscriptionDeadline);
+    CPPUNIT_TEST(testInvalidPushRevision);
+    CPPUNIT_TEST(testRecoveryBackoff);
+    CPPUNIT_TEST(testContiguousExpiration);
+    CPPUNIT_TEST(testRejectsUnrevisionedPushes);
+#endif
     CPPUNIT_TEST_SUITE_END();
 
 public:
     void testResubscribeUsesKeyRoute();
     void testSetPushNotificationTokenResubscribesWithNewToken();
+    void testRestartListenersFetchesValuesAfterFailedSubscribe();
+    void testReadinessShutdown();
+    void testReadinessReentrancy();
+    void testCachedReadinessDispatch();
+    void testReadinessCancelsAnotherListener();
+#ifdef OPENDHT_PUSH_NOTIFICATIONS
+    void testInitialReadinessFailures();
+    void testReadinessRecoveryIsOneShot();
+    void testWrongSessionPushDoesNotTouchSubscriptions();
+    void testNewSessionSynchronizesIndependently();
+    void testFailedTimerRetryFetchesValues();
+    void testFullRetryRemovesAbsentValues();
+    void testStreamingFallbackExpiresSnapshot();
+    void testLateRetryResponseDoesNotResurrectExpiredValue();
+    void testTruncatedRecoveryRemainsFailed();
+    void testTruncatedRecoveryAtEofFails();
+    void testMissingCompletionDoesNotReconcile();
+    void testSnapshotExpirationIgnoresClockRollback();
+    void testSupersededFailureDoesNotPoisonSuccessfulRetry();
+    void testSyncCoalescesConnectivityChanges();
+    void testTokenRotationDuringRecovery();
+    void testTokenRotationBackToPreviousEpoch();
+    void testAuthoritativeEpochAfterStalePush();
+    void testSyncEditsAndDuplicateValues();
+    void testIncompleteSnapshotDoesNotReconcile();
+    void testSubscriptionDeadline();
+    void testInvalidPushRevision();
+    void testRecoveryBackoff();
+    void testContiguousExpiration();
+    void testRejectsUnrevisionedPushes();
+#endif
 };
 
 } // namespace test

tests/test_dhtrunner.cpp

@@ -24,7 +24,7 @@ constexpr auto stoppedPushResult = dht::PushNotificationResult::IgnoredStopped;
 #else
 constexpr auto stoppedPushResult = dht::PushNotificationResult::IgnoredDisabled;
 #endif
-}
+} // namespace
 
 void
 DhtRunnerPushTester::testPushBeforeRun()
@@ -77,6 +77,152 @@ getFutureValue(std::future<T> future, std::chrono::steady_clock::duration timeou
     return future.get();
 }
 
+void
+DhtRunnerTester::testListenReadiness()
+{
+    auto key = dht::InfoHash::get("native-listen-readiness");
+    auto value = std::make_shared<dht::Value>("initial-value");
+    value->id = 42;
+    std::promise<bool> stored;
+    auto storedFuture = stored.get_future();
+    node2.put(key, value, [&](bool ok) { stored.set_value(ok); });
+    CPPUNIT_ASSERT(storedFuture.wait_for(10s) == std::future_status::ready);
+    CPPUNIT_ASSERT(storedFuture.get());
+    bool received = false;
+    std::promise<bool> ready;
+    auto readyFuture = ready.get_future();
+    auto token = node1.listen(
+        key,
+        [&](const std::vector<dht::Sp<dht::Value>>& values, bool expired) {
+            if (!expired)
+                for (const auto& current : values)
+                    received = received || current->id == 42;
+            return true;
+        },
+        {},
+        {},
+        [&](bool ok) { ready.set_value(ok && received); });
+    CPPUNIT_ASSERT(readyFuture.wait_for(10s) == std::future_status::ready);
+    CPPUNIT_ASSERT(readyFuture.get());
+    node1.cancelListen(key, token.get());
+
+    dht::DhtRunner isolated;
+    isolated.run(0);
+    std::promise<bool> emptyReady;
+    auto emptyFuture = emptyReady.get_future();
+    auto emptyToken = isolated.listen(
+        key,
+        [](const std::vector<dht::Sp<dht::Value>>&, bool) { return true; },
+        {},
+        {},
+        [&](bool ok) { emptyReady.set_value(ok); });
+    CPPUNIT_ASSERT(emptyFuture.wait_for(5s) == std::future_status::ready);
+    CPPUNIT_ASSERT(emptyFuture.get());
+    isolated.cancelListen(key, emptyToken.get());
+    isolated.join();
+}
+
+void
+DhtRunnerTester::testListenReadinessStopped()
+{
+    dht::DhtRunner stopped;
+    auto key = dht::InfoHash::get("stopped-listen-readiness");
+    std::vector<bool> results;
+    unsigned deliveries = 0;
+    auto listen = [&] {
+        auto token = stopped.listen(
+            key,
+            [&](const std::vector<dht::Sp<dht::Value>>&, bool) {
+                ++deliveries;
+                return true;
+            },
+            {},
+            {},
+            [&](bool ok) { results.push_back(ok); });
+        CPPUNIT_ASSERT(token.wait_for(0s) == std::future_status::ready);
+        CPPUNIT_ASSERT_EQUAL(size_t(0), token.get());
+    };
+    listen();
+    CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+    stopped.run(0);
+    stopped.join();
+    listen();
+    CPPUNIT_ASSERT(results == std::vector<bool>({false, false}));
+    CPPUNIT_ASSERT_EQUAL(0u, deliveries);
+}
+
+void
+DhtRunnerTester::testListenReadinessInvalidKey()
+{
+    auto ready = std::make_shared<std::promise<bool>>();
+    auto future = ready->get_future();
+    auto deliveries = std::make_shared<std::atomic<unsigned>>(0);
+    auto token = node1.listen(
+        dht::InfoHash {},
+        [deliveries](const std::vector<dht::Sp<dht::Value>>&, bool) {
+            ++*deliveries;
+            return true;
+        },
+        {},
+        {},
+        [ready](bool ok) { ready->set_value(ok); });
+    CPPUNIT_ASSERT_EQUAL(size_t(0), getFutureValue(std::move(token), 5s));
+    CPPUNIT_ASSERT(!getFutureValue(std::move(future), 5s));
+    CPPUNIT_ASSERT_EQUAL(0u, deliveries->load());
+}
+
+void
+DhtRunnerTester::testListenReadinessFilteredSignedValues()
+{
+    dht::DhtRunner source;
+    dht::DhtRunner::Config config;
+    config.dht_config.id = dht::crypto::generateIdentity();
+    source.run(0, config);
+    auto bound = node1.getBound();
+    if (bound.isUnspecified())
+        bound.setLoopback();
+    source.bootstrap(bound);
+    auto key = dht::InfoHash::get("signed-listen-readiness");
+    auto initial = std::make_shared<dht::Value>("initial-signed-value");
+    initial->id = 42;
+    auto stored = std::make_shared<std::promise<bool>>();
+    auto storedFuture = stored->get_future();
+    source.putSigned(key, initial, [stored](bool ok) { stored->set_value(ok); });
+    CPPUNIT_ASSERT(getFutureValue(std::move(storedFuture), 10s));
+
+    auto ignored = std::make_shared<dht::Value>("filtered-value");
+    ignored->id = 43;
+    auto ignoredStored = std::make_shared<std::promise<bool>>();
+    auto ignoredFuture = ignoredStored->get_future();
+    source.put(key, ignored, [ignoredStored](bool ok) { ignoredStored->set_value(ok); });
+    CPPUNIT_ASSERT(getFutureValue(std::move(ignoredFuture), 10s));
+
+    struct InitialValues
+    {
+        unsigned count {0};
+        bool valid {true};
+        std::promise<bool> ready;
+    };
+    auto state = std::make_shared<InitialValues>();
+    auto future = state->ready.get_future();
+    auto token = source.listen(
+        key,
+        [state](const std::vector<dht::Sp<dht::Value>>& values, bool expired) {
+            if (!expired)
+                for (const auto& value : values) {
+                    ++state->count;
+                    state->valid = state->valid && value->id == 42 && value->isSigned() && value->checkSignature();
+                }
+            return true;
+        },
+        [](const dht::Value& value) { return value.id == 42; },
+        {},
+        [state](bool ok) { state->ready.set_value(ok && state->valid && state->count == 1); });
+    CPPUNIT_ASSERT(getFutureValue(std::move(future), 10s));
+    source.cancelListen(key, getFutureValue(std::move(token), 5s));
+    source.join();
+}
+
 void
 DhtRunnerTester::setUp()
 {

tests/test_dhtrunner.h

@@ -34,6 +34,10 @@ class DhtRunnerTester : public CppUnit::TestFixture
     CPPUNIT_TEST(testPutDuplicate);
     CPPUNIT_TEST(testPutOverride);
     CPPUNIT_TEST(testListen);
+    CPPUNIT_TEST(testListenReadiness);
+    CPPUNIT_TEST(testListenReadinessStopped);
+    CPPUNIT_TEST(testListenReadinessInvalidKey);
+    CPPUNIT_TEST(testListenReadinessFilteredSignedValues);
     CPPUNIT_TEST(testListenValueEdit);
     CPPUNIT_TEST(testListenLotOfBytes);
     CPPUNIT_TEST(testStoreEmptyValue);
@@ -86,6 +90,10 @@ public:
      * Test listen method
      */
     void testListen();
+    void testListenReadiness();
+    void testListenReadinessStopped();
+    void testListenReadinessInvalidKey();
+    void testListenReadinessFilteredSignedValues();
     /**
      * Test methods requiring a node identity
      */

tests/test_networkengine.cpp

@@ -94,8 +94,8 @@ public:
         return 0;
     }
 
-    bool hasIPv4() const override { return true; }
-    bool hasIPv6() const override { return false; }
+    bool hasIPv4() const override { return ipv4; }
+    bool hasIPv6() const override { return ipv6; }
     const SockAddr& getBoundRef(sa_family_t family = AF_UNSPEC) const override
     {
         (void) family;
@@ -112,6 +112,8 @@ public:
 
     SockAddr bound_;
     std::vector<SentPacket> sends {};
+    bool ipv4 {true};
+    bool ipv6 {false};
 };
 
 static Blob
@@ -230,7 +232,9 @@ makeEngine(std::unique_ptr<TestDatagramSocket>&& socket,
         [](Sp<Node>) { return net::RequestAnswer {}; },
         [](Sp<Node>, const InfoHash&, want_t) { return net::RequestAnswer {}; },
         [](Sp<Node>, const InfoHash&, want_t, const Query&) { return net::RequestAnswer {}; },
-        [](Sp<Node>, const InfoHash&, const Blob&, Tid, const Query&, int) { return net::RequestAnswer {}; },
+        [](Sp<Node>, const InfoHash&, const Blob&, Tid, const Query&, int, const std::optional<net::ListenSync>&) {
+            return net::RequestAnswer {};
+        },
         [](Sp<Node>, const InfoHash&, const Blob&, const std::vector<Sp<Value>>&, const time_point&) {
             return net::RequestAnswer {};
         },
@@ -252,6 +256,164 @@ addPendingGetRequest(net::NetworkEngine& engine, const InfoHash& id, const SockA
     return node;
 }
 
+struct ListenSnapshotTransport
+{
+    InfoHash key {InfoHash::get("initial-listen-snapshot")};
+    std::vector<Value::Id> received;
+    std::vector<bool> results;
+    std::vector<size_t> snapshots;
+    std::vector<TestDatagramSocket::SentPacket> delayedUpdates;
+    TestDatagramSocket* clientPackets;
+    TestDatagramSocket* serverPackets;
+    SockAddr clientAddr;
+    SockAddr serverAddr;
+    std::unique_ptr<Dht> client;
+    std::unique_ptr<Dht> server;
+    size_t token {0};
+
+    explicit ListenSnapshotTransport(uint64_t serverSeed = 31)
+    {
+        Config config {};
+        config.max_req_per_sec = -1;
+        config.max_peer_req_per_sec = -1;
+        auto clientSocket = std::make_unique<TestDatagramSocket>();
+        clientPackets = clientSocket.get();
+        clientAddr = clientPackets->bound_;
+        client = std::make_unique<Dht>(std::move(clientSocket),
+                                       config,
+                                       Sp<Logger> {},
+                                       std::make_unique<std::mt19937_64>(30));
+        auto serverSocket = std::make_unique<TestDatagramSocket>();
+        serverPackets = serverSocket.get();
+        serverPackets->bound_.setPort(5031);
+        serverAddr = serverPackets->bound_;
+        server = std::make_unique<Dht>(std::move(serverSocket),
+                                       config,
+                                       Sp<Logger> {},
+                                       std::make_unique<std::mt19937_64>(serverSeed));
+    }
+
+    ~ListenSnapshotTransport()
+    {
+        if (token)
+            client->cancelListen(key, token);
+    }
+
+    static ParsedMessage parse(const TestDatagramSocket::SentPacket& packet)
+    {
+        ParsedMessage message;
+        message.msgpack_unpack(msgpack::unpack((const char*) packet.data.data(), packet.data.size()).get());
+        return message;
+    }
+
+    void store(size_t count, size_t payloadSize = 20)
+    {
+        for (size_t index = 0; index < count; ++index) {
+            auto value = std::make_shared<Value>(std::string(payloadSize, 'x'));
+            value->id = index + 1;
+            CPPUNIT_ASSERT(server->storageStore(key, value, server->scheduler.time()));
+        }
+    }
+
+    void listen()
+    {
+        token = client->listen(
+            key,
+            [this](const std::vector<Sp<Value>>& values, bool expired) {
+                if (!expired)
+                    for (const auto& value : values)
+                        received.emplace_back(value->id);
+                return true;
+            },
+            {},
+            {},
+            [this](bool ok) {
+                results.push_back(ok);
+                snapshots.emplace_back(received.size());
+            });
+        CPPUNIT_ASSERT(token != 0);
+    }
+
+    void start(bool confirm = true)
+    {
+        listen();
+        auto search = client->searches(AF_INET).at(key);
+        auto node = client->network_engine.insertNode(server->getNodeId(), serverAddr);
+        auto* searchNode = search->getNode(node);
+        CPPUNIT_ASSERT(searchNode);
+        searchNode->token = server->makeToken(clientAddr, false);
+        searchNode->last_get_reply = client->scheduler.time();
+        client->searchStep(search);
+        if (confirm)
+            acknowledge();
+    }
+
+    void acknowledge(bool legacy = false, bool updatesFirst = false, bool includeToken = true)
+    {
+        auto requests = std::move(clientPackets->sends);
+        clientPackets->sends.clear();
+        unsigned registrations = 0;
+        Blob legacyToken;
+        for (const auto& packet : requests) {
+            if (parse(packet).type == MessageType::Listen) {
+                ++registrations;
+                if (legacy) {
+                    auto message = parse(packet);
+                    auto node = server->network_engine.insertNode(client->getNodeId(), clientAddr);
+                    auto answer = server->onListen(node, key, message.token, message.socket_id, message.query, 1, {});
+                    legacyToken = std::move(answer.ntoken);
+                } else
+                    server->network_engine.processMessage(packet.data.data(), packet.data.size(), clientAddr);
+            }
+        }
+        CPPUNIT_ASSERT_EQUAL(1u, registrations);
+        auto replies = std::move(serverPackets->sends);
+        serverPackets->sends.clear();
+        std::vector<TestDatagramSocket::SentPacket> confirmations;
+        for (const auto& packet : replies) {
+            if (parse(packet).type == MessageType::Reply)
+                confirmations.emplace_back(packet);
+            else
+                delayedUpdates.emplace_back(packet);
+        }
+        if (updatesFirst)
+            deliverUpdates();
+        if (legacy) {
+            ParsedMessage acknowledgment;
+            acknowledgment.type = MessageType::Reply;
+            if (includeToken)
+                acknowledgment.token = std::move(legacyToken);
+            initialRequest()->setDone(std::move(acknowledgment));
+        }
+        for (const auto& packet : confirmations)
+            client->network_engine.processMessage(packet.data.data(), packet.data.size(), serverAddr);
+    }
+
+    void getRound(bool reversed = false)
+    {
+        for (const auto& packet : clientPackets->sends)
+            CPPUNIT_ASSERT(parse(packet).type != MessageType::GetValues);
+        if (reversed)
+            std::reverse(delayedUpdates.begin(), delayedUpdates.end());
+        deliverUpdates();
+    }
+
+    void deliverUpdates()
+    {
+        for (const auto& packet : delayedUpdates)
+            client->network_engine.processMessage(packet.data.data(), packet.data.size(), serverAddr);
+        delayedUpdates.clear();
+    }
+
+    Sp<net::Request> initialRequest() const
+    {
+        auto search = client->searches(AF_INET).at(key);
+        auto& status = search->nodes.front()->listenStatus.begin()->second;
+        CPPUNIT_ASSERT(status.req);
+        return status.req;
+    }
+};
+
 void
 NetworkEngineTester::setUp()
 {}
@@ -479,7 +641,8 @@ NetworkEngineTester::testListenConfirmationCarriesToken()
         [](Sp<Node>) { return net::RequestAnswer {}; },
         [](Sp<Node>, const InfoHash&, want_t) { return net::RequestAnswer {}; },
         [](Sp<Node>, const InfoHash&, want_t, const Query&) { return net::RequestAnswer {}; },
-        [responseToken](Sp<Node>, const InfoHash&, const Blob&, Tid, const Query&, int) {
+        [responseToken](
+            Sp<Node>, const InfoHash&, const Blob&, Tid, const Query&, int, const std::optional<net::ListenSync>&) {
             net::RequestAnswer answer {};
             answer.ntoken = responseToken;
             return answer;
@@ -555,6 +718,682 @@ NetworkEngineTester::testListenConfirmationUpdatesSearchNodeToken()
     CPPUNIT_ASSERT(!searchNodePtr->isSynced(localDht.scheduler.time()));
 }
 
+void
+NetworkEngineTester::testListenReadinessWaitsForInitialUpdate()
+{
+    Config config {};
+    config.max_req_per_sec = -1;
+    config.max_peer_req_per_sec = -1;
+    auto clientSocket = std::make_unique<TestDatagramSocket>();
+    auto* clientPackets = clientSocket.get();
+    auto clientAddr = clientPackets->bound_;
+    Dht client(std::move(clientSocket), config, {}, std::make_unique<std::mt19937_64>(20));
+    auto serverSocket = std::make_unique<TestDatagramSocket>();
+    auto* serverPackets = serverSocket.get();
+    serverPackets->bound_.setPort(5020);
+    auto serverAddr = serverPackets->bound_;
+    Dht server(std::move(serverSocket), config, {}, std::make_unique<std::mt19937_64>(21));
+    auto key = InfoHash::get("readiness-reordered-update");
+    auto value = std::make_shared<Value>("initial-value");
+    value->id = 42;
+    CPPUNIT_ASSERT(server.storageStore(key, value, server.scheduler.time()));
+
+    std::vector<Value::Id> received;
+    std::vector<size_t> snapshots;
+    auto token = client.listen(
+        key,
+        [&](const std::vector<Sp<Value>>& values, bool expired) {
+            if (!expired)
+                for (const auto& current : values)
+                    received.emplace_back(current->id);
+            return true;
+        },
+        {},
+        {},
+        [&](bool ok) {
+            CPPUNIT_ASSERT(ok);
+            snapshots.emplace_back(received.size());
+        });
+    CPPUNIT_ASSERT(token != 0);
+    auto search = client.searches(AF_INET).at(key);
+    auto node = client.network_engine.insertNode(server.getNodeId(), serverAddr);
+    auto* searchNode = search->getNode(node);
+    CPPUNIT_ASSERT(searchNode);
+    searchNode->token = server.makeToken(clientAddr, false);
+    searchNode->last_get_reply = client.scheduler.time();
+    CPPUNIT_ASSERT(search->isSynced(client.scheduler.time()));
+    client.searchStep(search);
+    CPPUNIT_ASSERT(snapshots.empty());
+
+    auto listen = std::find_if(clientPackets->sends.begin(), clientPackets->sends.end(), [](const auto& packet) {
+        ParsedMessage message;
+        message.msgpack_unpack(msgpack::unpack((const char*) packet.data.data(), packet.data.size()).get());
+        return message.type == MessageType::Listen;
+    });
+    CPPUNIT_ASSERT(listen != clientPackets->sends.end());
+    server.network_engine.processMessage(listen->data.data(), listen->data.size(), clientAddr);
+    CPPUNIT_ASSERT(serverPackets->sends.size() >= 2);
+
+    std::vector<TestDatagramSocket::SentPacket> updates;
+    std::vector<TestDatagramSocket::SentPacket> confirmations;
+    for (const auto& packet : serverPackets->sends) {
+        ParsedMessage message;
+        message.msgpack_unpack(msgpack::unpack((const char*) packet.data.data(), packet.data.size()).get());
+        if (message.type == MessageType::Reply)
+            confirmations.emplace_back(packet);
+        else if (message.type == MessageType::UpdateValue)
+            updates.emplace_back(packet);
+    }
+    CPPUNIT_ASSERT_EQUAL(size_t(1), confirmations.size());
+    CPPUNIT_ASSERT(!updates.empty());
+    for (const auto& packet : confirmations)
+        client.network_engine.processMessage(packet.data.data(), packet.data.size(), serverAddr);
+    CPPUNIT_ASSERT(snapshots.empty());
+    serverPackets->sends.clear();
+    for (const auto& packet : clientPackets->sends)
+        CPPUNIT_ASSERT(ListenSnapshotTransport::parse(packet).type != MessageType::GetValues);
+    for (const auto& packet : updates)
+        client.network_engine.processMessage(packet.data.data(), packet.data.size(), serverAddr);
+    CPPUNIT_ASSERT(client.cancelListen(key, token));
+    CPPUNIT_ASSERT(received == std::vector<Value::Id>({42}));
+    CPPUNIT_ASSERT_EQUAL(size_t(1), snapshots.size());
+    CPPUNIT_ASSERT_EQUAL(size_t(1), snapshots.front());
+}
+
+void
+NetworkEngineTester::testListenReadinessSnapshotVariants()
+{
+    for (size_t count : {size_t(0), size_t(3)}) {
+        ListenSnapshotTransport transport;
+        transport.store(count, 6000);
+        transport.start();
+        if (count != 0)
+            CPPUNIT_ASSERT(transport.results.empty());
+        transport.getRound(true);
+        CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+        CPPUNIT_ASSERT(transport.snapshots == std::vector<size_t>({count}));
+        transport.deliverUpdates();
+        CPPUNIT_ASSERT_EQUAL(count, transport.received.size());
+        CPPUNIT_ASSERT_EQUAL(size_t(1), transport.results.size());
+    }
+}
+
+void
+NetworkEngineTester::testListenReadinessSnapshotLegacy()
+{
+    for (size_t count : {size_t(0), size_t(3)})
+        for (bool updatesFirst : {false, true})
+            for (bool includeToken : {false, true}) {
+                ListenSnapshotTransport transport;
+                transport.store(count, 6000);
+                transport.start(false);
+                auto request = transport.initialRequest();
+                auto search = transport.client->searches(AF_INET).at(transport.key);
+                auto query = search->nodes.front()->listenStatus.begin()->first;
+                auto socketId = search->nodes.front()->listenStatus.at(query).socketId;
+                transport.acknowledge(true, updatesFirst, includeToken);
+                CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+                CPPUNIT_ASSERT(transport.snapshots == std::vector<size_t>({updatesFirst ? count : 0}));
+                CPPUNIT_ASSERT(request->node->getSocket(socketId));
+                transport.getRound(true);
+                CPPUNIT_ASSERT_EQUAL(count, transport.received.size());
+                CPPUNIT_ASSERT_EQUAL(size_t(1), transport.results.size());
+
+                std::vector<bool> reusedResults;
+                size_t reusedValues = 0;
+                auto reused = transport.client->listen(
+                    transport.key,
+                    [&](const std::vector<Sp<Value>>& values, bool expired) {
+                        if (!expired)
+                            reusedValues += values.size();
+                        return true;
+                    },
+                    {},
+                    {},
+                    [&](bool ok) { reusedResults.emplace_back(ok); });
+                CPPUNIT_ASSERT(reused != 0);
+                CPPUNIT_ASSERT_EQUAL(count, reusedValues);
+                CPPUNIT_ASSERT(reusedResults == std::vector<bool>({true}));
+                CPPUNIT_ASSERT(transport.client->cancelListen(transport.key, reused));
+            }
+}
+
+void
+NetworkEngineTester::testListenLegacyLateValuesAndRenewal()
+{
+    ListenSnapshotTransport transport;
+    transport.start(false);
+    transport.acknowledge(true, false, false);
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    auto node = transport.initialRequest()->node;
+    auto* source = search->getNode(node);
+    auto query = source->listenStatus.begin()->first;
+    auto socketId = source->listenStatus.at(query).socketId;
+    auto token = source->token;
+    CPPUNIT_ASSERT(!token.empty());
+
+    transport.client->scheduler.syncTime(transport.client->scheduler.time() + std::chrono::seconds(35));
+    transport.client->scheduler.run();
+    CPPUNIT_ASSERT(node->getSocket(socketId));
+    source = search->getNode(node);
+    CPPUNIT_ASSERT(source);
+    CPPUNIT_ASSERT(source->token == token);
+    auto first = std::make_shared<Value>("late-legacy-value");
+    first->id = 42;
+    CPPUNIT_ASSERT(transport.server->storageStore(transport.key, first, transport.server->scheduler.time()));
+    transport.delayedUpdates = std::move(transport.serverPackets->sends);
+    transport.serverPackets->sends.clear();
+    transport.deliverUpdates();
+    CPPUNIT_ASSERT(transport.received == std::vector<Value::Id>({42}));
+
+    if (!source->listenStatus.at(query).req->pending()) {
+        transport.clientPackets->sends.clear();
+        source->listenStatus.at(query).req->reply_time = transport.client->scheduler.time()
+                                                         - transport.client->getListenExpiration();
+        transport.client->searchSynchedNodeListen(search, *source);
+    }
+    transport.acknowledge(true, false, false);
+    CPPUNIT_ASSERT(transport.delayedUpdates.empty());
+    CPPUNIT_ASSERT(node->getSocket(socketId));
+    CPPUNIT_ASSERT_EQUAL(socketId, source->listenStatus.at(query).socketId);
+    CPPUNIT_ASSERT(source->token == token);
+
+    auto second = std::make_shared<Value>("after-legacy-renewal");
+    second->id = 43;
+    CPPUNIT_ASSERT(transport.server->storageStore(transport.key, second, transport.server->scheduler.time()));
+    transport.delayedUpdates = std::move(transport.serverPackets->sends);
+    transport.serverPackets->sends.clear();
+    transport.getRound();
+    CPPUNIT_ASSERT(transport.received == std::vector<Value::Id>({42, 43}));
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+}
+
+void
+NetworkEngineTester::testListenReadinessMixedPeers()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start(false);
+    auto nativeRequests = std::move(transport.clientPackets->sends);
+    transport.clientPackets->sends.clear();
+
+    ListenSnapshotTransport legacy(32);
+    legacy.serverAddr.setPort(5032);
+    legacy.serverPackets->bound_ = legacy.serverAddr;
+    legacy.store(2);
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    auto node = transport.client->network_engine.insertNode(legacy.server->getNodeId(), legacy.serverAddr);
+    auto* source = search->getNode(node);
+    CPPUNIT_ASSERT(source);
+    source->token = legacy.server->makeToken(transport.clientAddr, false);
+    source->last_get_reply = transport.client->scheduler.time();
+    transport.client->searchSynchedNodeListen(search, *source);
+    CPPUNIT_ASSERT_EQUAL(size_t(1), source->listenStatus.size());
+    auto& status = source->listenStatus.begin()->second;
+    auto peer = legacy.server->network_engine.insertNode(transport.client->getNodeId(), transport.clientAddr);
+    auto query = *source->listenStatus.begin()->first;
+    legacy.server->storageAddListener(transport.key, peer, status.socketId, std::move(query), 1, {});
+    ParsedMessage acknowledgment;
+    acknowledgment.type = MessageType::Reply;
+    status.req->setDone(std::move(acknowledgment));
+    CPPUNIT_ASSERT(transport.results.empty());
+    for (const auto& packet : legacy.serverPackets->sends)
+        transport.client->network_engine.processMessage(packet.data.data(), packet.data.size(), legacy.serverAddr);
+    CPPUNIT_ASSERT_EQUAL(size_t(2), transport.received.size());
+    CPPUNIT_ASSERT(transport.results.empty());
+
+    transport.clientPackets->sends = std::move(nativeRequests);
+    transport.acknowledge();
+    CPPUNIT_ASSERT(transport.results.empty());
+    transport.getRound();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    CPPUNIT_ASSERT_EQUAL(size_t(2), transport.received.size());
+}
+
+void
+NetworkEngineTester::testListenReadinessSnapshotFailure()
+{
+    for (unsigned failure = 0; failure < 3; ++failure) {
+        ListenSnapshotTransport transport;
+        transport.store(1);
+        transport.start(false);
+        auto request = transport.initialRequest();
+        if (failure == 0) {
+            request->setExpired();
+        } else {
+            ParsedMessage answer;
+            answer.type = MessageType::Reply;
+            if (failure == 1) {
+                auto search = transport.client->searches(AF_INET).at(transport.key);
+                auto registration = search->nodes.front()->listenStatus.begin()->second.registration;
+                answer.listenSync = net::ListenSync {registration, 1, 0, 0};
+            }
+            if (failure == 2) {
+                answer.token = transport.server->makeToken(transport.clientAddr, false);
+                answer.fields.emplace_back(std::make_shared<FieldValueIndex>());
+            }
+            request->setDone(std::move(answer));
+        }
+        CPPUNIT_ASSERT(transport.results == std::vector<bool>({false}));
+        transport.deliverUpdates();
+        CPPUNIT_ASSERT_EQUAL(size_t(1), transport.results.size());
+    }
+}
+
+void
+NetworkEngineTester::testListenReadinessSnapshotCancellation()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start();
+    CPPUNIT_ASSERT(transport.client->cancelListen(transport.key, transport.token));
+    transport.token = 0;
+    transport.getRound();
+    transport.getRound();
+    CPPUNIT_ASSERT(transport.results.empty());
+    transport.listen();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    CPPUNIT_ASSERT(transport.snapshots == std::vector<size_t>({1}));
+}
+
+void
+NetworkEngineTester::testListenReadinessSnapshotReplacement()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start();
+    auto previous = transport.initialRequest();
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    auto* node = search->nodes.front().get();
+    auto oldUpdates = std::move(transport.delayedUpdates);
+    node->listenStatus.clear();
+    transport.clientPackets->sends.clear();
+    transport.client->searchSynchedNodeListen(search, *node);
+    transport.acknowledge();
+    CPPUNIT_ASSERT(transport.results.empty());
+    ParsedMessage answer;
+    answer.token = transport.server->makeToken(transport.clientAddr, false);
+    answer.values = transport.server->getLocal(transport.key);
+    previous->setDone(std::move(answer));
+    for (const auto& packet : oldUpdates)
+        transport.client->network_engine.processMessage(packet.data.data(), packet.data.size(), transport.serverAddr);
+    CPPUNIT_ASSERT(transport.results.empty());
+    transport.getRound();
+    transport.getRound();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    CPPUNIT_ASSERT(transport.snapshots == std::vector<size_t>({1}));
+}
+
+void
+NetworkEngineTester::testListenReadinessSnapshotRenewal()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start();
+    transport.getRound();
+    transport.getRound();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    transport.deliverUpdates();
+    transport.clientPackets->sends.clear();
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    auto* node = search->nodes.front().get();
+    node->listenStatus.begin()->second.req->reply_time = transport.client->scheduler.time()
+                                                         - transport.client->getListenExpiration();
+    transport.client->searchSynchedNodeListen(search, *node);
+    transport.acknowledge();
+    CPPUNIT_ASSERT(transport.delayedUpdates.empty());
+    for (const auto& packet : transport.clientPackets->sends)
+        CPPUNIT_ASSERT(ListenSnapshotTransport::parse(packet).type != MessageType::GetValues);
+    CPPUNIT_ASSERT_EQUAL(size_t(1), transport.results.size());
+}
+
+void
+NetworkEngineTester::testListenUpdatesBeforeConfirmation()
+{
+    ListenSnapshotTransport transport;
+    transport.store(40, 6000);
+    transport.start(false);
+    auto request = transport.initialRequest();
+    auto listen = std::find_if(transport.clientPackets->sends.begin(),
+                               transport.clientPackets->sends.end(),
+                               [](const auto& packet) {
+                                   return ListenSnapshotTransport::parse(packet).type == MessageType::Listen;
+                               });
+    CPPUNIT_ASSERT(listen != transport.clientPackets->sends.end());
+    transport.server->network_engine.processMessage(listen->data.data(), listen->data.size(), transport.clientAddr);
+    auto replies = std::move(transport.serverPackets->sends);
+    std::vector<TestDatagramSocket::SentPacket> confirmations;
+    std::reverse(replies.begin(), replies.end());
+    for (const auto& packet : replies) {
+        if (ListenSnapshotTransport::parse(packet).type == MessageType::Reply)
+            confirmations.emplace_back(packet);
+        else
+            transport.client->network_engine.processMessage(packet.data.data(),
+                                                            packet.data.size(),
+                                                            transport.serverAddr);
+    }
+    CPPUNIT_ASSERT(transport.results.empty());
+    CPPUNIT_ASSERT(transport.received.empty());
+    CPPUNIT_ASSERT_EQUAL(size_t(1), confirmations.size());
+    auto& confirmation = confirmations.front();
+    transport.client->network_engine.processMessage(confirmation.data.data(),
+                                                    confirmation.data.size(),
+                                                    transport.serverAddr);
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    CPPUNIT_ASSERT_EQUAL(size_t(40), transport.received.size());
+    for (const auto& packet : replies)
+        transport.client->network_engine.processMessage(packet.data.data(), packet.data.size(), transport.serverAddr);
+    CPPUNIT_ASSERT_EQUAL(size_t(40), transport.received.size());
+}
+
+void
+NetworkEngineTester::testListenLiveChangesBeforeSnapshot()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start();
+    auto& storage = transport.server->store.at(transport.key);
+    auto removed = storage.remove(transport.key, 1);
+    CPPUNIT_ASSERT(removed);
+    transport.server->storageRemoved(transport.key, storage, {removed}, removed->size());
+    auto value = std::make_shared<Value>("new-live-value");
+    value->id = 99;
+    CPPUNIT_ASSERT(transport.server->storageStore(transport.key, value, transport.server->scheduler.time()));
+    CPPUNIT_ASSERT_EQUAL(size_t(2), transport.serverPackets->sends.size());
+    for (const auto& packet : transport.serverPackets->sends)
+        transport.client->network_engine.processMessage(packet.data.data(), packet.data.size(), transport.serverAddr);
+    CPPUNIT_ASSERT(transport.received.empty());
+    CPPUNIT_ASSERT(transport.results.empty());
+    transport.deliverUpdates();
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    CPPUNIT_ASSERT(!search->cache.get(1));
+    CPPUNIT_ASSERT(search->cache.get(99));
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    for (const auto& packet : transport.serverPackets->sends)
+        transport.client->network_engine.processMessage(packet.data.data(), packet.data.size(), transport.serverAddr);
+    CPPUNIT_ASSERT(!search->cache.get(1));
+    CPPUNIT_ASSERT_EQUAL(size_t(2), transport.received.size());
+}
+
+void
+NetworkEngineTester::testListenDuplicateRegistration()
+{
+    ListenSnapshotTransport transport;
+    transport.store(40);
+    transport.start(false);
+    auto listen = std::find_if(transport.clientPackets->sends.begin(),
+                               transport.clientPackets->sends.end(),
+                               [](const auto& packet) {
+                                   return ListenSnapshotTransport::parse(packet).type == MessageType::Listen;
+                               });
+    CPPUNIT_ASSERT(listen != transport.clientPackets->sends.end());
+    auto packet = *listen;
+    transport.acknowledge();
+    CPPUNIT_ASSERT_EQUAL(size_t(2),
+                         size_t(std::count_if(transport.delayedUpdates.begin(),
+                                              transport.delayedUpdates.end(),
+                                              [](const auto& update) {
+                                                  return ListenSnapshotTransport::parse(update).type
+                                                         == MessageType::UpdateValue;
+                                              })));
+    transport.server->network_engine.processMessage(packet.data.data(), packet.data.size(), transport.clientAddr);
+    CPPUNIT_ASSERT_EQUAL(size_t(1), transport.serverPackets->sends.size());
+    auto confirmation = ListenSnapshotTransport::parse(transport.serverPackets->sends.front());
+    CPPUNIT_ASSERT(confirmation.type == MessageType::Reply);
+    CPPUNIT_ASSERT(confirmation.listenSync);
+    CPPUNIT_ASSERT_EQUAL(uint64_t(2), confirmation.listenSync->snapshotEnd);
+    transport.deliverUpdates();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    CPPUNIT_ASSERT_EQUAL(size_t(40), transport.received.size());
+}
+
+void
+NetworkEngineTester::testListenLargeSnapshotDefaultLimits()
+{
+    ListenSnapshotTransport transport;
+    auto socket = std::make_unique<TestDatagramSocket>();
+    socket->bound_.setPort(5031);
+    transport.serverPackets = socket.get();
+    transport.serverAddr = socket->bound_;
+    transport.server = std::make_unique<Dht>(std::move(socket),
+                                             Config {},
+                                             Sp<Logger> {},
+                                             std::make_unique<std::mt19937_64>(31));
+    transport.store(4096);
+    transport.start();
+    CPPUNIT_ASSERT(transport.results.empty());
+    size_t batches = 0;
+    size_t payloads = 0;
+    for (const auto& packet : transport.delayedUpdates) {
+        auto message = ListenSnapshotTransport::parse(packet);
+        if (message.type == MessageType::UpdateValue) {
+            ++batches;
+            payloads += message.values.size() + message.value_parts.size();
+        }
+    }
+    CPPUNIT_ASSERT_EQUAL(size_t(128), batches);
+    CPPUNIT_ASSERT_EQUAL(size_t(4096), payloads);
+    transport.getRound(true);
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+    CPPUNIT_ASSERT_EQUAL(size_t(4096), transport.received.size());
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    CPPUNIT_ASSERT_EQUAL(size_t(4096), search->cache.get(Value::Filter {}).size());
+    for (const auto& packet : transport.clientPackets->sends)
+        CPPUNIT_ASSERT(ListenSnapshotTransport::parse(packet).type != MessageType::GetValues);
+}
+
+void
+NetworkEngineTester::testListenLiveUpdateOrdering()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start();
+    transport.deliverUpdates();
+    for (Value::Id id : {2, 3}) {
+        auto value = std::make_shared<Value>("live");
+        value->id = id;
+        CPPUNIT_ASSERT(transport.server->storageStore(transport.key, value, transport.server->scheduler.time()));
+    }
+    CPPUNIT_ASSERT_EQUAL(size_t(2), transport.serverPackets->sends.size());
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    auto& later = transport.serverPackets->sends.back();
+    transport.client->network_engine.processMessage(later.data.data(), later.data.size(), transport.serverAddr);
+    CPPUNIT_ASSERT(!search->cache.get(3));
+    auto& earlier = transport.serverPackets->sends.front();
+    transport.client->network_engine.processMessage(earlier.data.data(), earlier.data.size(), transport.serverAddr);
+    CPPUNIT_ASSERT(search->cache.get(2));
+    CPPUNIT_ASSERT(search->cache.get(3));
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({true}));
+}
+
+void
+NetworkEngineTester::testListenInitializationTimeout()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start();
+    CPPUNIT_ASSERT(transport.results.empty());
+    transport.client->scheduler.syncTime(transport.client->scheduler.time() + std::chrono::seconds(31));
+    transport.client->scheduler.run();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({false}));
+    transport.deliverUpdates();
+    CPPUNIT_ASSERT_EQUAL(size_t(1), transport.results.size());
+}
+
+void
+NetworkEngineTester::testListenRegistrationFailure()
+{
+    ListenSnapshotTransport transport;
+    transport.store(1);
+    transport.start(false);
+    auto search = transport.client->searches(AF_INET).at(transport.key);
+    auto* node = search->nodes.front().get();
+    auto request = node->listenStatus.begin()->second.req;
+    CPPUNIT_ASSERT(request->pending());
+    CPPUNIT_ASSERT(!node->node->isExpired());
+    request->setExpired();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({false}));
+    CPPUNIT_ASSERT(!node->node->isExpired());
+    transport.clientPackets->sends.clear();
+    transport.client->searchStep(search);
+    transport.acknowledge();
+    transport.getRound();
+    transport.getRound();
+    CPPUNIT_ASSERT(transport.results == std::vector<bool>({false}));
+}
+
+void
+NetworkEngineTester::testListenRegistrationCapacity()
+{
+    for (bool dualStack : {false, true}) {
+        auto socket = std::make_unique<TestDatagramSocket>();
+        socket->ipv6 = dualStack;
+        Config config;
+        config.max_searches = 1;
+        Dht localDht(std::move(socket), config, {}, std::make_unique<std::mt19937_64>(40));
+        auto occupiedKey = InfoHash::get("occupied-search");
+        auto rejectedKey = InfoHash::get("rejected-search");
+        auto values = [](const std::vector<Sp<Value>>&, bool) {
+            return true;
+        };
+        if (dualStack)
+            CPPUNIT_ASSERT(localDht.search(occupiedKey, AF_INET6));
+        else
+            CPPUNIT_ASSERT(localDht.listen(occupiedKey, values) != 0);
+        std::vector<bool> results;
+        CPPUNIT_ASSERT_EQUAL(size_t(0),
+                             localDht.listen(rejectedKey, values, {}, {}, [&](bool ok) { results.push_back(ok); }));
+        CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+        CPPUNIT_ASSERT(localDht.store.at(rejectedKey).local_listeners.empty());
+        if (dualStack) {
+            auto partial = localDht.searches(AF_INET).at(rejectedKey);
+            CPPUNIT_ASSERT(!partial->cache.cancelListen(1, localDht.scheduler.time()));
+            partial->cache.onReady();
+            CPPUNIT_ASSERT_EQUAL(size_t(1), results.size());
+        }
+    }
+}
+
+void
+NetworkEngineTester::testListenReadinessAddressFamilies()
+{
+    auto key = InfoHash::get("readiness-address-families");
+    for (unsigned families = 1; families <= 3; ++families) {
+        auto socket = std::make_unique<TestDatagramSocket>();
+        socket->ipv4 = families & 1;
+        socket->ipv6 = families & 2;
+        Dht localDht(std::move(socket), Config {}, {}, std::make_unique<std::mt19937_64>(families));
+        std::vector<bool> results;
+        auto token = localDht.listen(
+            key,
+            [](const std::vector<Sp<Value>>&, bool) { return true; },
+            {},
+            {},
+            [&](bool ok) { results.push_back(ok); });
+        CPPUNIT_ASSERT(token != 0);
+        CPPUNIT_ASSERT(results.empty());
+        if (families & 1) {
+            localDht.searchStep(localDht.searches(AF_INET).at(key));
+            if (families & 2)
+                CPPUNIT_ASSERT(results.empty());
+        }
+        if (families & 2)
+            localDht.searchStep(localDht.searches(AF_INET6).at(key));
+        CPPUNIT_ASSERT(results == std::vector<bool>({true}));
+        CPPUNIT_ASSERT(localDht.cancelListen(key, token));
+
+        results.clear();
+        CPPUNIT_ASSERT_EQUAL(size_t(0),
+                             localDht.listen(
+                                 InfoHash {},
+                                 [](const std::vector<Sp<Value>>&, bool) { return true; },
+                                 {},
+                                 {},
+                                 [&](bool ok) { results.push_back(ok); }));
+        CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+    }
+}
+
+void
+NetworkEngineTester::testListenReadinessFailedSearch()
+{
+    Dht localDht(std::make_unique<TestDatagramSocket>(), Config {}, {}, std::make_unique<std::mt19937_64>(22));
+    auto key = InfoHash::get("readiness-failed-search");
+    std::vector<bool> results;
+    auto token = localDht.listen(
+        key, [](const std::vector<Sp<Value>>&, bool) { return true; }, {}, {}, [&](bool ok) { results.push_back(ok); });
+    auto search = localDht.searches(AF_INET).at(key);
+    auto node = std::make_shared<Node>(InfoHash::getRandom(localDht.rd),
+                                       makeIPv4("127.0.0.2", 5022),
+                                       localDht.rd,
+                                       false);
+    node->setExpired();
+    search->nodes.emplace_back(std::make_unique<Dht::SearchNode>(node));
+    localDht.searchStep(search);
+    CPPUNIT_ASSERT(search->expired);
+    CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+    search->cache.onReady();
+    CPPUNIT_ASSERT_EQUAL(size_t(1), results.size());
+    CPPUNIT_ASSERT(localDht.cancelListen(key, token));
+}
+
+void
+NetworkEngineTester::testListenReadinessReusesExpiredSearch()
+{
+    Dht localDht(std::make_unique<TestDatagramSocket>(), Config {}, {}, std::make_unique<std::mt19937_64>(23));
+    auto key = InfoHash::get("readiness-reused-search");
+    auto values = [](const std::vector<Sp<Value>>&, bool) {
+        return true;
+    };
+    std::vector<bool> results;
+    auto ready = [&](bool ok) {
+        results.push_back(ok);
+    };
+    auto first = localDht.listen(key, values, {}, {}, ready);
+    auto search = localDht.searches(AF_INET).at(key);
+    search->expire();
+    CPPUNIT_ASSERT(results == std::vector<bool>({false}));
+    CPPUNIT_ASSERT(localDht.cancelListen(key, first));
+    auto second = localDht.listen(key, values, {}, {}, ready);
+    CPPUNIT_ASSERT(second != 0);
+    CPPUNIT_ASSERT(search == localDht.searches(AF_INET).at(key));
+    CPPUNIT_ASSERT(!search->expired);
+    CPPUNIT_ASSERT(!search->done);
+    CPPUNIT_ASSERT(localDht.scheduler.time() == search->nextSearchStep->t_);
+    localDht.searchStep(search);
+    CPPUNIT_ASSERT(results == std::vector<bool>({false, true}));
+    CPPUNIT_ASSERT(localDht.cancelListen(key, second));
+}
+
+void
+NetworkEngineTester::testListenReadinessShutdown()
+{
+    for (bool stop : {false, true}) {
+        std::vector<bool> results;
+        auto localDht = std::make_unique<Dht>(std::make_unique<TestDatagramSocket>(),
+                                              Config {},
+                                              Sp<Logger> {},
+                                              std::make_unique<std::mt19937_64>(24));
+        CPPUNIT_ASSERT(localDht->listen(
+                           InfoHash::get("readiness-shutdown"),
+                           [](const std::vector<Sp<Value>>&, bool) { return true; },
+                           {},
+                           {},
+                           [&](bool ok) { results.push_back(ok); })
+                       != 0);
+        CPPUNIT_ASSERT(results.empty());
+        if (stop) {
+            localDht->shutdown({}, true);
+            CPPUNIT_ASSERT_EQUAL(size_t(1), results.size());
+            CPPUNIT_ASSERT(!results.front());
+        }
+        localDht.reset();
+        CPPUNIT_ASSERT_EQUAL(size_t(1), results.size());
+        CPPUNIT_ASSERT(!results.front());
+    }
+}
+
 void
 NetworkEngineTester::testListenReopensSocketAfterNodeExpiration()
 {

tests/test_networkengine.h

@@ -22,6 +22,27 @@ class NetworkEngineTester : public CppUnit::TestFixture
     CPPUNIT_TEST(testSeparatesPartialSessionsBySource);
     CPPUNIT_TEST(testListenConfirmationCarriesToken);
     CPPUNIT_TEST(testListenConfirmationUpdatesSearchNodeToken);
+    CPPUNIT_TEST(testListenReadinessWaitsForInitialUpdate);
+    CPPUNIT_TEST(testListenReadinessSnapshotVariants);
+    CPPUNIT_TEST(testListenReadinessSnapshotLegacy);
+    CPPUNIT_TEST(testListenLegacyLateValuesAndRenewal);
+    CPPUNIT_TEST(testListenReadinessMixedPeers);
+    CPPUNIT_TEST(testListenReadinessSnapshotFailure);
+    CPPUNIT_TEST(testListenReadinessSnapshotCancellation);
+    CPPUNIT_TEST(testListenReadinessSnapshotReplacement);
+    CPPUNIT_TEST(testListenReadinessSnapshotRenewal);
+    CPPUNIT_TEST(testListenUpdatesBeforeConfirmation);
+    CPPUNIT_TEST(testListenLiveChangesBeforeSnapshot);
+    CPPUNIT_TEST(testListenDuplicateRegistration);
+    CPPUNIT_TEST(testListenLargeSnapshotDefaultLimits);
+    CPPUNIT_TEST(testListenLiveUpdateOrdering);
+    CPPUNIT_TEST(testListenInitializationTimeout);
+    CPPUNIT_TEST(testListenRegistrationFailure);
+    CPPUNIT_TEST(testListenRegistrationCapacity);
+    CPPUNIT_TEST(testListenReadinessAddressFamilies);
+    CPPUNIT_TEST(testListenReadinessFailedSearch);
+    CPPUNIT_TEST(testListenReadinessReusesExpiredSearch);
+    CPPUNIT_TEST(testListenReadinessShutdown);
     CPPUNIT_TEST(testListenReopensSocketAfterNodeExpiration);
     CPPUNIT_TEST(testUnauthorizedListenFlushClearsListenState);
 #endif
@@ -43,6 +64,27 @@ public:
     void testSeparatesPartialSessionsBySource();
     void testListenConfirmationCarriesToken();
     void testListenConfirmationUpdatesSearchNodeToken();
+    void testListenReadinessWaitsForInitialUpdate();
+    void testListenReadinessSnapshotVariants();
+    void testListenReadinessSnapshotLegacy();
+    void testListenLegacyLateValuesAndRenewal();
+    void testListenReadinessMixedPeers();
+    void testListenReadinessSnapshotFailure();
+    void testListenReadinessSnapshotCancellation();
+    void testListenReadinessSnapshotReplacement();
+    void testListenReadinessSnapshotRenewal();
+    void testListenUpdatesBeforeConfirmation();
+    void testListenLiveChangesBeforeSnapshot();
+    void testListenDuplicateRegistration();
+    void testListenLargeSnapshotDefaultLimits();
+    void testListenLiveUpdateOrdering();
+    void testListenInitializationTimeout();
+    void testListenRegistrationFailure();
+    void testListenRegistrationCapacity();
+    void testListenReadinessAddressFamilies();
+    void testListenReadinessFailedSearch();
+    void testListenReadinessReusesExpiredSearch();
+    void testListenReadinessShutdown();
     void testListenReopensSocketAfterNodeExpiration();
     void testUnauthorizedListenFlushClearsListenState();
 #endif

tests/test_opcache.cpp

@@ -56,6 +56,234 @@ void
 OpCacheTester::setUp()
 {}
 
+void
+OpCacheTester::testInitialListenerReadiness()
+{
+    dht::OpCache cache;
+    auto query = std::make_shared<dht::Query>();
+    unsigned calls = 0;
+    auto callback = [&](bool ok) {
+        CPPUNIT_ASSERT(ok);
+        ++calls;
+    };
+    CPPUNIT_ASSERT(cache.addListener(1, [](const Values&, bool) { return true; }, query, {}, callback));
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+    cache.onNodeChanged(dht::ListenSyncStatus::ADDED);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(1u, calls);
+    cache.onNodeChanged(dht::ListenSyncStatus::UNSYNCED);
+    CPPUNIT_ASSERT(cache.addListener(4, [](const Values&, bool) { return true; }, query, {}, callback));
+    CPPUNIT_ASSERT_EQUAL(1u, calls);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(2u, calls);
+    CPPUNIT_ASSERT(cache.addListener(2, [](const Values&, bool) { return true; }, query, {}, callback));
+    CPPUNIT_ASSERT_EQUAL(3u, calls);
+    dht::OpCache isolated;
+    CPPUNIT_ASSERT(isolated.addListener(3, [](const Values&, bool) { return true; }, query, {}, callback));
+    isolated.onReady();
+    CPPUNIT_ASSERT_EQUAL(4u, calls);
+    dht::OpCache searching(true, true);
+    CPPUNIT_ASSERT(searching.addListener(5, [](const Values&, bool) { return true; }, query, {}, callback));
+    searching.onNodeChanged(dht::ListenSyncStatus::ADDED);
+    searching.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(4u, calls);
+    searching.onSearchReady();
+    CPPUNIT_ASSERT_EQUAL(5u, calls);
+}
+
+void
+OpCacheTester::testInitialReadinessCancellation()
+{
+    dht::OpCache cache(true, true);
+    auto query = std::make_shared<dht::Query>();
+    auto values = [](const Values&, bool) {
+        return true;
+    };
+    unsigned calls = 0;
+    auto ready = [&](bool) {
+        ++calls;
+    };
+    CPPUNIT_ASSERT(cache.addListener(1, values, query, {}, ready));
+    CPPUNIT_ASSERT(cache.removeListener(1, dht::clock::now()));
+    cache.onReady();
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+
+    dht::OpCache cleared(true, true);
+    CPPUNIT_ASSERT(cleared.addListener(1, values, query, {}, ready));
+    cleared.removeAll();
+    cleared.onReady(false);
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+
+    dht::OpCache reentrant(true, true);
+    CPPUNIT_ASSERT(reentrant.addListener(1, values, query, {}, [&](bool ok) {
+        CPPUNIT_ASSERT(ok);
+        CPPUNIT_ASSERT(reentrant.removeListener(2, dht::clock::now()));
+    }));
+    CPPUNIT_ASSERT(reentrant.addListener(2, values, query, {}, ready));
+    reentrant.onReady();
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+
+    dht::OpCache clearing(true, true);
+    CPPUNIT_ASSERT(clearing.addListener(1, values, query, {}, [&](bool) { clearing.removeAll(); }));
+    CPPUNIT_ASSERT(clearing.addListener(2, values, query, {}, ready));
+    clearing.onReady();
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+}
+
+void
+OpCacheTester::testInitialReadinessFailure()
+{
+    dht::OpCache cache(true, true);
+    auto query = std::make_shared<dht::Query>();
+    auto values = [](const Values&, bool) {
+        return true;
+    };
+    std::vector<bool> results;
+    auto ready = [&](bool ok) {
+        results.push_back(ok);
+    };
+    CPPUNIT_ASSERT(cache.addListener(1, values, query, {}, ready));
+    CPPUNIT_ASSERT(cache.addListener(2, values, query, {}, ready));
+    cache.onReady(false);
+    CPPUNIT_ASSERT(results == std::vector<bool>({false, false}));
+    cache.onReady();
+    CPPUNIT_ASSERT_EQUAL(size_t(2), results.size());
+
+    cache.onReady(false);
+    CPPUNIT_ASSERT(cache.addListener(3, values, query, {}, ready));
+    CPPUNIT_ASSERT_EQUAL(size_t(2), results.size());
+    cache.onNodeChanged(dht::ListenSyncStatus::ADDED);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(size_t(2), results.size());
+    cache.onSearchReady();
+    CPPUNIT_ASSERT(results == std::vector<bool>({false, false, true}));
+
+    dht::SearchCache searches(true, true);
+    auto first = std::make_shared<dht::Query>(dht::Select {}, dht::Where {}.id(42));
+    auto second = std::make_shared<dht::Query>(dht::Select {}, dht::Where {}.id(43));
+    dht::SyncCallback secondSource;
+    auto onListen = [&](dht::Sp<dht::Query> query, dht::ValueCallback, dht::SyncCallback source) {
+        source(dht::ListenSyncStatus::ADDED);
+        if (query == second)
+            secondSource = std::move(source);
+        return size_t(1);
+    };
+    std::vector<bool> firstResults;
+    std::vector<bool> secondResults;
+    CPPUNIT_ASSERT(searches.listen(values, first, {}, onListen, [&](bool ok) { firstResults.push_back(ok); }) != 0);
+    CPPUNIT_ASSERT(searches.listen(values, second, {}, onListen, [&](bool ok) { secondResults.push_back(ok); }) != 0);
+    searches.onReady(first, false);
+    CPPUNIT_ASSERT(firstResults == std::vector<bool>({false}));
+    CPPUNIT_ASSERT(secondResults.empty());
+    secondSource(dht::ListenSyncStatus::SYNCED);
+    searches.onSearchReady();
+    CPPUNIT_ASSERT(secondResults == std::vector<bool>({true}));
+}
+
+void
+OpCacheTester::testInitialReadinessMultipleSources()
+{
+    dht::OpCache cache(true, true);
+    auto query = std::make_shared<dht::Query>();
+    unsigned calls = 0;
+    CPPUNIT_ASSERT(cache.addListener(
+        1,
+        [](const Values&, bool) { return true; },
+        query,
+        {},
+        [&](bool ok) {
+            CPPUNIT_ASSERT(ok);
+            ++calls;
+        }));
+    cache.onSearchReady();
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+    cache.onNodeChanged(dht::ListenSyncStatus::ADDED);
+    cache.onNodeChanged(dht::ListenSyncStatus::ADDED);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(1u, calls);
+    cache.onSearchReady();
+    cache.onNodeChanged(dht::ListenSyncStatus::UNSYNCED);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(1u, calls);
+}
+
+void
+OpCacheTester::testInitialReadinessCachedValues()
+{
+    dht::OpCache cache(true, true);
+    auto query = std::make_shared<dht::Query>();
+    auto value = makeValue(42, 0, "initial");
+    cache.onNodeChanged(dht::ListenSyncStatus::ADDED);
+    cache.onValue({value, makeValue(43, 0, "filtered")}, false);
+    cache.onNodeChanged(dht::ListenSyncStatus::SYNCED);
+    cache.onSearchReady();
+    std::vector<std::string> events;
+    CPPUNIT_ASSERT(cache.addListener(
+        1,
+        [&](const Values& values, bool expired) {
+            CPPUNIT_ASSERT(!expired);
+            CPPUNIT_ASSERT_EQUAL(size_t(1), values.size());
+            CPPUNIT_ASSERT_EQUAL(value->id, values.front()->id);
+            events.emplace_back("value");
+            return true;
+        },
+        query,
+        [](const dht::Value& current) { return current.id == 42; },
+        [&](bool ok) {
+            CPPUNIT_ASSERT(ok);
+            events.emplace_back("ready");
+        }));
+    CPPUNIT_ASSERT(events == std::vector<std::string>({"value", "ready"}));
+
+    unsigned calls = 0;
+    CPPUNIT_ASSERT(!cache.addListener(2, [](const Values&, bool) { return false; }, query, {}, [&](bool) { ++calls; }));
+    cache.onReady();
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+}
+
+void
+OpCacheTester::testInitialReadinessReusedOperation()
+{
+    dht::SearchCache cache(true, true);
+    auto query = std::make_shared<dht::Query>();
+    dht::SyncCallback source;
+    unsigned operations = 0;
+    unsigned calls = 0;
+    auto onListen = [&](dht::Sp<dht::Query>, dht::ValueCallback values, dht::SyncCallback sync) {
+        ++operations;
+        source = std::move(sync);
+        source(dht::ListenSyncStatus::ADDED);
+        values({makeValue(42, 0, "cached")}, false);
+        source(dht::ListenSyncStatus::SYNCED);
+        return size_t(1);
+    };
+    auto values = [](const Values&, bool) {
+        return true;
+    };
+    auto ready = [&](bool ok) {
+        CPPUNIT_ASSERT(ok);
+        ++calls;
+    };
+    auto first = cache.listen(values, query, {}, onListen, ready);
+    CPPUNIT_ASSERT(first != 0);
+    CPPUNIT_ASSERT_EQUAL(0u, calls);
+    cache.onSearchReady();
+    CPPUNIT_ASSERT_EQUAL(1u, calls);
+    CPPUNIT_ASSERT(cache.cancelListen(first, dht::clock::now()));
+
+    auto second = cache.listen(values, std::make_shared<dht::Query>(), {}, onListen, ready);
+    CPPUNIT_ASSERT(second != 0);
+    CPPUNIT_ASSERT_EQUAL(1u, operations);
+    CPPUNIT_ASSERT_EQUAL(2u, calls);
+    source(dht::ListenSyncStatus::UNSYNCED);
+    CPPUNIT_ASSERT(cache.listen(values, query, {}, onListen, ready) != 0);
+    CPPUNIT_ASSERT_EQUAL(2u, calls);
+    source(dht::ListenSyncStatus::SYNCED);
+    CPPUNIT_ASSERT_EQUAL(3u, calls);
+}
+
 void
 OpCacheTester::tearDown()
 {}

tests/test_opcache.h

@@ -20,6 +20,12 @@ class OpCacheTester : public CppUnit::TestFixture
     CPPUNIT_TEST(testCallbacks);
     CPPUNIT_TEST(testFilters);
     CPPUNIT_TEST(testSyncStatus);
+    CPPUNIT_TEST(testInitialListenerReadiness);
+    CPPUNIT_TEST(testInitialReadinessCancellation);
+    CPPUNIT_TEST(testInitialReadinessFailure);
+    CPPUNIT_TEST(testInitialReadinessMultipleSources);
+    CPPUNIT_TEST(testInitialReadinessCachedValues);
+    CPPUNIT_TEST(testInitialReadinessReusedOperation);
     CPPUNIT_TEST(testGetWhileSynced);
     CPPUNIT_TEST(testGetWhileNotSynced);
     CPPUNIT_TEST(testGetEmptySynced);
@@ -49,6 +55,12 @@ public:
     void testCallbacks();
     void testFilters();
     void testSyncStatus();
+    void testInitialListenerReadiness();
+    void testInitialReadinessCancellation();
+    void testInitialReadinessFailure();
+    void testInitialReadinessMultipleSources();
+    void testInitialReadinessCachedValues();
+    void testInitialReadinessReusedOperation();
     void testGetWhileSynced();
     void testGetWhileNotSynced();
     void testGetEmptySynced();

tests/test_parsedmessage.cpp

@@ -105,6 +105,69 @@ void
 ParsedMessageTester::setUp()
 {}
 
+void
+ParsedMessageTester::testParseListenSync()
+{
+    for (const auto& fields : std::vector<std::vector<int64_t>> {
+             {1, 2, 3, 1},
+             {0, 2, 3, 1},
+             {1, 2, -1, 1},
+             {1, 2, 3},
+             {1, 2, 3, 1, 0}
+    }) {
+        msgpack::sbuffer buffer;
+        msgpack::packer<msgpack::sbuffer> packer(buffer);
+        packer.pack_map(3);
+        packer.pack(KEY_Y);
+        packer.pack(KEY_R);
+        packer.pack(KEY_TID);
+        packer.pack(1);
+        packer.pack(KEY_R);
+        packer.pack_map(2);
+        packer.pack(KEY_VERSION);
+        packer.pack(2);
+        packer.pack(KEY_LISTEN_SYNC);
+        packer.pack(fields);
+        if (fields == std::vector<int64_t>({1, 2, 3, 1})) {
+            auto message = parsePackedMessage(buffer);
+            CPPUNIT_ASSERT(message.listenSync);
+            CPPUNIT_ASSERT_EQUAL(uint64_t(1), message.listenSync->registration);
+            CPPUNIT_ASSERT_EQUAL(uint64_t(2), message.listenSync->epoch);
+            CPPUNIT_ASSERT_EQUAL(uint64_t(3), message.listenSync->sequence);
+            CPPUNIT_ASSERT_EQUAL(uint64_t(1), message.listenSync->snapshotEnd);
+        } else {
+            CPPUNIT_ASSERT_THROW(parsePackedMessage(buffer), msgpack::type_error);
+        }
+    }
+}
+
+void
+ParsedMessageTester::testParseRejectsMalformedSyncedValues()
+{
+    for (unsigned failure = 0; failure < 3; ++failure) {
+        msgpack::sbuffer buffer;
+        msgpack::packer<msgpack::sbuffer> packer(buffer);
+        packer.pack_map(3);
+        packer.pack(KEY_Y);
+        packer.pack(KEY_R);
+        packer.pack(KEY_TID);
+        packer.pack(1);
+        packer.pack(KEY_R);
+        packer.pack_map(3);
+        packer.pack(KEY_VERSION);
+        packer.pack(2);
+        packer.pack(KEY_LISTEN_SYNC);
+        packer.pack(ListenSync {1, 2, 1, 1});
+        packer.pack(KEY_REQ_VALUES);
+        packer.pack_array(1);
+        if (failure == 0)
+            packer.pack_nil();
+        else
+            packer.pack(failure == 1 ? MAX_VALUE_SIZE + 33 : 0);
+        CPPUNIT_ASSERT_THROW(parsePackedMessage(buffer), msgpack::type_error);
+    }
+}
+
 void
 ParsedMessageTester::tearDown()
 {}

tests/test_parsedmessage.h

@@ -27,6 +27,8 @@ class ParsedMessageTester : public CppUnit::TestFixture
     CPPUNIT_TEST(testParseRejectsEmptyFieldSelection);
     CPPUNIT_TEST(testParseRejectsEmptyErrorArray);
     CPPUNIT_TEST(testParseRejectsNonMapRequest);
+    CPPUNIT_TEST(testParseListenSync);
+    CPPUNIT_TEST(testParseRejectsMalformedSyncedValues);
     CPPUNIT_TEST(testParseErrorOnlyMessageKeepsRequestFieldsEmpty);
     CPPUNIT_TEST(testParseIgnoresIncompleteValueDataEntry);
     CPPUNIT_TEST(testCompleteEmpty);
@@ -60,6 +62,8 @@ public:
     void testParseRejectsEmptyFieldSelection();
     void testParseRejectsEmptyErrorArray();
     void testParseRejectsNonMapRequest();
+    void testParseListenSync();
+    void testParseRejectsMalformedSyncedValues();
     void testParseErrorOnlyMessageKeepsRequestFieldsEmpty();
     void testParseIgnoresIncompleteValueDataEntry();
     void testCompleteEmpty();

tools/dhtnode.cpp

@@ -405,7 +405,8 @@ cmd_loop(std::shared_ptr<DhtRunner>& node,
                     return true;
                 },
                 {},
-                dht::Where {rem});
+                dht::Where {rem},
+                [](bool synced) { std::cout << "Listen: " << (synced ? "synced" : "not synced") << std::endl; });
             auto t = token.get();
             std::cout << "Listening, token: " << t << std::endl;
         }