With this blog post, we† tried to clarify a few names. Feel free to point out things that are still unclear, the documentation is pretty much a WIP.
UnifiedPush aims at replacing the push notifications mechanism provided by Google services with something independent, that anyone can self-host or any OS can provide without compatibility issues. It does require small adjustments server-side and client-side for applications assuming Google services.
Doesn’t Apple and Google on the OS level only allow push notifications to come from its own services? Like, they look for the signature to be compatible with https certificate etc. Otherwise the phone could have a ton of push notifications incoming from anywhere on the Internet. The APN service for instance would only route notifications to a phone if it came from a registered website. How do you get around this?
As far as I know, even for VOIP permissions, iOS only wakes your code up if ITS APNS SERVICE sends the notification. How are you able to install a long-lived application in the background on iOS?
Similarly, the phone radio is listening for calls from the nearby cell towers in the network, not from absolutely any tower. I guess a Stingray can impersonate one, but they would probably need to fake the attestation, right?
>and there probably is no limit to those on Android.
Not true since Android Oreo. Long running background Service execution (without a foreground notification) has been dead for a while. Most of the useful system-wide config change updates (implicit Broadcasts) got killed as well. Basically, only Google (or GCM/Firebase Messaging) gets to do timely push notifications. You can try to roll your own MQTT server, but there’s a good chance the OS will kill your Service, in which case you would have to poll every 15 minutes with JobScheduler/WorkManager, with no guarantee that background task would even get run every 15 min.
They have special deals with the telcos that TCP connections to their servers can have longer TTLs. Plus of course, even if you do polling, if you have 5 different apps that connect every 10 minutes to their own respective services with different offsets, you have the device waking up every 2 minutes, with a potential latency of up to 10 minutes for each individual service. If it uses a central push service, you can set it to 5 minutes, the device would therefore wake up less than half of the time, and maximum latency would be 5 minutes, not 10.
Thus, it makes total sense to have one mechanism for push messages for the entire device. It's great to have degoogled alternatives here.
> Plus of course, even if you do polling, if you have 5 different apps that connect every 10 minutes to their own respective services with different offsets, you have the device waking up every 2 minutes,
This is now impossible with background execution restraints set in Oreo. The OS clusters requests for background execution and does them with the same wake. Apps can only wake the device every 15 minutes at most (and requests to wake are not guaranteed).
> This is now impossible with background execution restraints
Signature spoofing is similarly "impossible" there's no rule saying this stuff can't be changed by distributions that care about it. You already have to install a custom OS to get rid of gapps to begin with so aside from the maintenance burden for changes that will never get upstreamed this is really a nonissue.
Thanks for your great work creating this protocol. Hope more app developers add support for it, especially for privacy focused apps like Signal which by relying on Google servers are leaking metadata.
I think the consensus was that we wanted to wait for UnifiedPush to be more mature before approaching them, but it's probably about time. S1m made an interesting implementation for the (compatible) Signal fork Molly: https://github.com/mollyim/mollyim-android/pull/152
The implementation is quite interesting: it adds a linked device that does not receive encryption keys, but sends push notifications to the client.
In theory, if you were worried about this, I think you could get around that problem by just constantly send fixed sized encrypted messages large enough to hold any potential notifications. When a notification needs to get sent, you add it to the next outgoing encrypted message. Of course, now the delay for notifications is now coupled to the rate of these messages being sent.
The messages can still be traced to the client picking them up. Cut their networking and the client doesn’t receive the messages anymore… this shows up as a bounced message.
You can accumulate encrypted messages at random urls instead
If nobody in the chat room can know that you lost connectivity, that means you can't do real time chat and might as well use email instead.
Note that a similar thing happens with any two-way conversation; if an immediate reply is expected and there's no reply, it's not a conversation anymore. (For more possible reasons, though.)
Disappearances can't have good error messages if you want plausible deniability. This will make any UI much less user-friendly.
Simply use free Usenet Servers via Tor and post your anonymous* PGP
messages to the Usenet group alt.anonymous.messages.
*You send PGP encrypted messages via Mixmaster Remailers (via Tor) and
preferably use a hashed subject (hsub) so that the intended receiver(s)
can filter out their messages.
> A police unit parked outside a guy’s house and confirmed it was him in the real time chatroom, by cutting his internet and seeing him drop off.
I guess I'm going off of too few details probably, but this doesn't seem like a very sound strategy to me. It's certainly not impossible that the real criminal lost internet around the same time as some suspect's was cut by the police. I know I've shown up as online to other people for up to several minutes after losing internet in real-time situations (e.g. games). I suppose police just need "reasonably likely" to take action though, not definite reasons.
Congrats! It's always surprised me that Amazon or one of the other Android vendors didn't do this themselves to make porting to FireOS or other services-included Android distributions easier, but I hope you get funding from them for doing their job for them.
Everyone wants to be Google, and I bet most managers see lock-in to their platform as a good thing, and ease of porting to competitors a bad thing.
At least, if they feel big enough to gain initial traction...
Huawei is pushing their own thing, alternative to Firebase[1]; Amazon as well[2].
There's a dozen services that provide cloud-based services to send push notifications to various services. I think the incentive to gain subscription revenue was greater than the opportunity to make these partially redundant.
I'm not sure why it took so long for something like UnifiedPush to appear. It took a fair bit of debugging, but ultimately, it was mostly up to one dedicated individual (S1m) with Android experience.
Before UnifiedPush, there was OpenPush[3], but it never materialized.
Yes, they need to have a UnifiedPush distributor on their device, that acts like Google services.
OS makers could install a default UnifiedPush distributor though.
Lastly, the Embedded FCM UnifiedPush library (to be embeded into a client app) handles UnifiedPush over FCM (Google-provided push notifications), so that a server only has to handle the UnifiedPush API (with a rewrite proxy for UP->FCM).
We can't get users to understand that the ceo isn't sending them mail from giftcards563@gmail.com to go and buy giftcards, and not to ask any questions.
What's the actual likely hood that the average end user is going to install a random service?
Except, in the threads here, this is called out repeatedly as a replacement for push notifications on phones, replacing webpush, apple push notifications, and whatever google is calling it.
Which implies the use case people are seeing is phone apps. Unless people are building one off apps for themselves, you've got end users in the mix.
I don't see how this can be positions as much outside of phone apps, as txt messages or emails would be easier and more universal. Being that they're probably going to be supported here.
To note, i haven't see anyone talking about this being replacement for the part of webpush that handles registration for notifications or whatnot, but as a whole separate channel/technology. Not 'install this to avoid using google/apple for webpush' but 'use this in your app, and have your users install the service'.
Some apps just offer both, and even auto-detect. So if your phone is de-googled, the app will use UP, if it has the Google Services it will use that. For some apps it is available in the settings.
So yeah, the app has to support it, but the user can choose then.
Is there any plan to have a method available for patching proprietary apps to work with this? Notifications is the only reason I still have Goolag services on my phone's work profile, and having an alternative to that would be great for privacy.
As a sibling commenter said, those proprietary apps have a server part that sends notifications to Google's server.
If UnifiedPush were to gain enough traction (say it would be adopted on FireOS, HuaweiOS and whatever platform don't use Google services), those apps would likely start to support it directly.
Another possible avenue is to add UnifiedPush support to push libraries used by these apps both server and client-side. Multiple libraries (Google FCM included, I linked at least two others from Amazon and Huawei in another comment here) abstract push notifications away so the developer gets a single server to talk to, and a unified API across iOS, Android, Web, etc. If these libraries started supporting UnifiedPush, adoption could increase a lot, without involving any developer effort for these proprietary apps.
They should, which is why nobody noticed it was broken.
Someone told us IPv6 was broken after we submitted here, as they use an IPv6-only network, apparently. I just posted that up there in case that was the actual issue encountered by the commenter. We're not sure why it broke yet, but we'll fix it when there's less traffic.
Off-topic: Sometimes I work on ships that have Wi-Fi for talking to local services, but due to strict metering, Internet access is usually disabled. So most of the time there's no way to get notifications from monitoring systems and your fellow crew, let alone shoreside team members.
It would be nice to have a solution for delivering pushes internally, and also hold open a best-effort connection to a remote server that allows some notifications through.
The last solution I considered was running a Matrix homeserver on the ship with some local channels, and another instance on shore, and using federation to connect them when the uplink is available. Monitoring systems could send their alerts as DMs or to a shared channel.
Then for the notifications, Apple has its Local Push Connectivity API you might be able to use to send notifications within the ship network. I'm not sure if this UnifiedPush would solve the problem on Android. Either way, though, it seems like a lot of engineering effort just for a chatroom.
One of the first uses of UnifiedPush was to self-host the whole Matrix messaging stack. In fact, we have already had someone in the UnifiedPush chat set this up on their ship [1]. Matrix + UnifiedPush would be perfect for a disconnected situation on ships (or Mars colonies :)
I'm a huge fan of Matrix and the idea of UnifiedPush, but I don't quite grok what this means (and I'm reluctant to fire up my Matrix client because I'm avoiding some things). Is there any chance you can elaborate on what "self-host the whole Matrix stack" means?
I read the thread just now:
~" Self hosted matrix + UP. Nothing special there. Only issue is cert expiration for setups without connections that result in expired certs for https. That requires some work. "~
ntfy [1] would work for you entirely in the LAN, if you self-host the server and the phones are connected to the same Wifi. It'll only work for Android phones though, since iOS forces APNS for push notifications.
ntfy is also a distributor for UnifiedPush, so you're not entirely off topic here. Hehe.
Interesting thanks. Have you looked into Apple's Local Push Connectivity API? It's specifically designed for VoIP calls and text messages, but perhaps the latter is enough. Not sure if it requires you to bake in any keys at compile time, or if the iOS app could allow the end-user to configure the push server.
In addition to what the others wrote (a local Matrix server and UnifiedPush provider such as ntfy would work fine), I suggest you look into p2p messaging systems such as Briar. I think Tox is also p2p?
Lastly, Secure Scuttlebutt was actually designed on a boat for sporadic internet connection and local connections, as well as sneakernet, so that would work too, but I don't think it's designed for IM. Perfect for blogs or facebook-like stuff though.
Have a member of the crew go onshore up to a Wi-Fi AP, it will act as an async data transfer for everyone. You can also exhange data with other travellers, including passing boats.
Interestingly, there are a few Matrix p2p experiments. I wish apple opened their Airdrop feature as that would make proximity networking easier, though that should be achievable with Wi-Fi NAN+P2P.
Ah, thanks for the correct term. I am not sure the specification is open enough for other operating systems to support it, which is what I meant.
Among my acquaintances, I can count Apple users on a single hand (<1% of the people I know), which severly limits the usefulness of the network.
There's OWL [1] which looks promising on Linux. I hope Android devices can get something soon. I'm still a bit confused by how Linux drives Wi-Fi radio, and what is and isn't implementable in software: could older Android devices gain support for NAN and OWL through a software update? What about old computers with old Wi-Fi cards?
I first got aware of Wi-Fi NAN by chatting with neilalexander on #yggdrasil:matrix.org :)
It looks promising for advertising capabilities to nearby hosts, allowing to negotiate p2p connections.
You can self host a Gotify server (single file executable, web interface), install the app, set it to connect to the server on the local network. Sending a message is as easy as a curl to the server. It will show up in the apps.
ntfy is a Gotify alternative. It has an iOS app, but it still relies on Apple's push service, as it cannot open a persistent backgroung connection due to iOS restrictions.
> It would be nice to have a solution for delivering pushes internally, and also hold open a best-effort connection to a remote server that allows some notifications through.
I've been using NNTP as a Gossip-type protocol for eventually-consistent pub/sub telemetry, monitoring, and notifications between partially-connected nodes in an unstable mesh since about 1995.
Why use a new custom-built protocol instead of adding support for the Web Push protocol already supported by Firefox, Chrome, Safari and others? https://web.dev/push-notifications-web-push-protocol/. Now backend developers have Yet Another Push Notifications Protocol to build support for, instead of being able to just easily plug into their existing code. As a benefit, since all notifications through the Web Push protocol are end-to-end encrypted, there's no concern about "trusting" the distributing server like there seems to be in this protocol (based on https://unifiedpush.org/spec/server/)
UnifiedPush is compatible with WebPush: both just involve an HTTP POST to a specific URL. If your application server supports WebPush, it can send encrypted notifications to UnifiedPush servers (that is how we're planning to support Telegram, for example). You can then decrypt these notifications in the app [1]. However, some protocols only send a wake-up ping or random ID in the push notification, Trifa and Matrix respectively. There, encryption is unnecessary complexity.
However, we don't use the WebPush API between the push server and distributor since there is a lot of scope for innovation in that space (for example Google's FCM uses a custom XMPP based protocol).
I agree that mandatory encryption would massively help privacy, integrity and even compliance [1]. However, the main cost is developer-effort involved in keeping track of keys and implementing encryption.
As a small project, ease of adoption has been our most important goal. However, as we work towards better WebPush compatibility [2], I personally absolutely want to promote RFC8291 encryption and make it easy to implement for developers using UnifiedPush, hopefully making it the norm.
Making it the "norm" isn't good enough, because developers will always want to cut corners to ship, either based on pressure from management or (in open source projects) just not feeling comfortable with encryption or thinking that it's important enough to implement. Web push libraries are already widely available for major languages to handle implementing encryption, and keeping track of keys isn't any harder than keeping track of device IDs or endpoints would be without encryption—it's just one more row in your database.
WebPush doesn't really have keys to keep track of. Other than the VAPID signature (which is optional but may be enforced by the push service) all of the keys are ephemeral. They are used for just one subscription so storing them is just as easy as storing the rest of the notification URL.
I agree, though it of course it depends on the threat model. As a self hoster, encryption is pretty much irrelevant, as my push server is controlled by myself, and accessible with https. Not having encryption makes it easy to debug and experiment.
I agree encryption should be the default choice though. Maybe we can introduce it in a future protocol revision? Client side, I expect most people use libraries, making this easy. Server side? Not so much.
The main goal was to get it adopted as widely as possible though, and encryption certainly seemed like something that would slow the effort.
Even for self-hosted I'd rather trust the server as little as necessary. Just in case it is completed why let it see things that it doesn't need to see?
> However, some protocols only send a wake-up ping or random ID in the push notification, Trifa and Matrix respectively
While I understand that some developers may think of security as wasteful, having unencrypted data be the default relies on every developer to spend much more time considering the possible threat models and how . Case in point here is Matrix—the IDs they send aren't random at all, they actually send full room IDs and event IDs for each message, so you'd be able to build up someone's entire social graph just from reading their unencrypted push notification data and looking up room IDs on the Matrix server in question or even correlating room IDs across different Element installs. This also hurts users, since they're not in a position to inspect the source code and understand how or whether they need to trust their "push provider" with whatever data the app developers might have chosen to regard as "less sensitive"—WebPush's encryption-by-default design removes trust from the equation entirely and prevents of these sorts of "passive surveillance" attacks (except those based on timing or message length)
> You can then decrypt these notifications in the app
This pushes all of the complexity of supporting encryption onto the app developer, instead of having it be provided by the platform where it can be centrally audited and securely managed.
> However, we don't use the WebPush API between the push server and distributor since there is a lot of scope for innovation in that space (for example Google's FCM uses a custom XMPP based protocol).
Why not do this innovation in an open standards body like the IESG where other stakeholders can give feedback and review? The web push standard does allow for Push Servers to use other custom wire protocols besides the default RFC8030, but having open and standards-based discussion of the possible performance improvements would "lift all boats" in terms of being able to provide low-power devices with timely push notifications
Yes, the WebPush standard provides protocols for both. This is necessary because otherwise there wouldn't be an intercompatible way for backends to send notifications to different browsers. The protocol for the federation of notifications would be the most useful for other non-JS projects to reuse, so that backend developers can plug any native app in their existing webpush support without coding yet another system.
I think GP got it backwards, actually: Web Push seems to have a wire interface for submission and a JavaScript API for subscription, while the internals of how the notifications get from e.g. the FCM backend to the notification drawer on your Android phone remain unspecified (and partly secret). UnifiedPush has a wire interface for both submission and subscription+reception, plus an Android(?) intent API for interacting with the client for the latter that’s running on your phone.
It looks like the submission side of UnifiedPush could indeed have been Web Push-compatible but isn’t, though.
> while the internals of how the notifications get from e.g. the FCM backend to the notification drawer on your Android phone remain unspecified (and partly secret)
Actually, the web push standard strongly recommends that you use RFC 8030 for this, but it does allow browsers to substitute other protocols as long as their semantics are the same https://datatracker.ietf.org/doc/html/rfc8030. I believe Mozilla and Edge both use HTTP Push, I'm not sure what Google uses on Desktop Chrome but I could see it going either way. Safari probably uses APNS.
I'm using Google Firebase Cloud Messaging [0] which allows pushing notifications to devices even if they're asleep, unlike local notification services where the device must be awake and the app must be active.
Will this service allow the same thing FCM does, including waking up the device? I'm looking at both iOS and Android devices, I see there's Flutter available but not iOS necessarily in the docs [1]. Related question, will the Flutter version also work for iOS devices? I know FCM (and its Flutter SDK) works for both, as well as web.
Yes, this is an FCM replacement on Android (and it can easily fall back to FCM if the user doesn't have UnifiedPush). There is no native UnifiedPush on iOS because Apple doesn't allow background services. The server side to support both is simple, but on the client you would need code for both FCM (for iOS) and UP (for Android). It would be interesting to build an FCM wrapper for iOS into the UP Flutter library.
I don't know too much about this, but is this not APNs [0]? I see there is a FCM integration (which is how I assume FCM does it anyway) [1] as well as a standalone Flutter implementation [2].
How would iOS native only apps on Swift do push notifications, surely Apple has their own service and everyone doesn't use Firebase?
Interesting. This could have interest from China, since they have to use Android without the Google push service, causing battery drain issues. They are trying to come up with their own push notification service.
I have been working and running ntfy [1] on my phone for a year now (ntfy is a UnifiedPush distributor), and it is true that Android does kill the app every now and then. But it is instantly restarted. Usually ntfy delivers notifications much much much faster than Google's FCM, especially in doze mode. FCM seems to try and conserve battery much more.
Sadly, I had phones where I couldn't keep anything running. Not sure what the manufacturer did there. Even the normal notifications didn't arrive at times. (E.g. Signal)
AFAIK Modern Android (>= 8.0) reserves the right to kill your app when running in the background unless there's a foreground toast bound (even then, can still be killed in low memory situations). How does this solution work around that? Is there a need to opt into the legacy "battery saver disabled" mode for the app which more closely mirrors <= 7.0 execution model?
Most distributors both show a foreground notification at all times and require exempting them from battery optimization. Because of its efficient design, something like ntfy only uses a few percent of battery per day [1].
System-level integration into Android ROMs would eliminate the need for those things.
I haven't seen any complaints, but folks who set this up are usually used to having persistent notifications (I currently have 6) without Google services.
At least I replaced the one from my Matrix client with the UnifiedPush-enabled ntfy. Hopefully it'll replace Signal too at some point. KDE Connect? Why not. K-9 mail? I hope, but that would require JMAP or IMAP adjustments.
One option would be to install the distributor as a system app.
The headline got my attention. I'm looking for a standard for subscribeable notifications / callbacks to update users of data changes in an API. Bonus point for federated. But server to server, not server to client. Double bonus points for connecting to consumer services like Zapier or IFTTT.
I'm planning to use WebSub and/or RSS, but on the look out for others. Any pointers?
WebSub + RSS/Atom is a good option for relatively slowly changing data. (Maybe at most a couple of times an hour). It is nice because it is opt-in and an enhancement on the regular polling API.
Using WebSub or UnifiedPush would technically very similar but you need a way for the client to pass you the subscription information. WebSub has a standard discovery process to manage this.
The WebSub spec looks disarmingly simple but also complete for everything I can think of. I've not yet tried to implement it, but I'm waiting for the catch! Thanks for the confirmation.
We're running an API, we want to say to users e.g. "we finished processing your submission" or "this API resource changed". And offer pull (e.g. RSS) and push flavour.
> You say it would be server to server, but are both publicly reachable?
So yes, the user would have to run a public server. But only for practical reasons, otherwise how can you push.
Webmention isn't quite it, we're notifying about our own API Resources.
Websub (FKA PubSubHubub) does. It just doesn't seem to be widely adopted for some reason.
Please don’t get me wrong but I wonder what is the use case for this. First of all this tries to solve a problem that has already been solved by the mobile device OS layer within IOS and Android and furthermore is only working for Android. Why should anyone use this?
It looks like this aims to be a replacement for people want to use an Android-based OS but don't want to rely on Google services, normally for reasons of philosophical objection to big tech's de facto surveillance capabilities. So presumably users of GrapheneOS, CalyxOS, LineageOS, etc.
Also since Google is blocked in China, Firebase Cloud Messaging doesn't work there. Android has something like a 75% market share in that market. Some vendors have their own offerings (e.g. Xiaomi MiPush, Huawei PushKit) etc, I could buy that a popular centralised solution would be more attractive than doing N integrations.
This. It also works without internet connection if you want push notifications within a LAN.
The vision is that you only need to implement support for 1 integration: UnifiedPush, possibly by embedding one of the libraries that manage it and fallbacks. You automatically gain support for compatible alternatives.
There's a variety of reasons why people can't or don't want to rely on Google services. Before UnifiedPush, one would have to "hardcode" an alternative, or more commonly implement a persistent connection within the app.
This also has some potential for more efficient push implementation: for instance, if you have access to an SMS gateway, you could use that to push notification without enabling data connection.
The specification also makes it easier to support third-party operating systems and clients.
Apps draining your battery to check for notifications is the worst thing about degoogled Android. With notifications enabled my telegram client can drain my battery in a very low number of hours (got suggestions here? any client supporting UnifiedPush?)
So this is welcome, but it requires application support. And if apps on your phone use both GCM and UnifiedPush you still end up with multiple providers. But at least I think I can trust UnifiedPush to be light on my battery.
The reason I said it is that it's a problem for me. Notifications on/off makes a night and day difference. Did you link ntfy because that's what Telegram FOSS uses?
the main issue isn't with the foss apps though. Those usually support polling. The actual problem is with other proprietary spyware apps that you're forced to use but they are locking you in with google's FCM. Until there's an easy way to patch those apps we can't really solve this problem
Yeah, I wouldn't even think about a degoogled phone at this point, none of my downloadable wiretaps would work, and I use a lot of them.
But this at least gives FOSS a chance to compete on battery life without FCM fallback like some do.
Most likely FOSS will never get full parity, Android might be gone and replaced long before then, but the ability to make an app that's F-droid legal is still a big deal.
hmm, I think most spyware apps still work without google services (on GrapheneOS), but you'll have to go without notification, among a few other things. If you need a tracking devi.. cough.. googled phone for work only, you can consider using a separate phone for it, and a personal degoogled phone.
I use a googled for both home and work pretty extensively, including Google Calendar, Keep, and Assistant, plus Tile and YoLink(Which loses most of it's value without notifications).
Carrying two phones seems like a lot of stress, I generally like to minimize the number of objects in my life as much as I can.
That's an interesting thought, but currently the Server-Server specification is quite trivial, the more complicated part is the Android (and Linux) API, which I don't think the W3C would cover.
There's definitely room for adoption within the freedesktop.org umbrella for the Linux part, once this part is a bit more mature.
Each application gets a unique URL, that is supposedly kept secret from any other party. They are also very easy to rotate, as the specification advises to ask for a new one (that could be the same) on every app launch.
Moreover, when something is sent to that URL, it gets forwarded to the application, which then interprets it before displaying a notification (for most apps).
By spam, I think you meant someone sending messages to random people's notification area. No, this is highly unlikely.
>However, if each app actively maintains a server connection, the OS cannot suspend them.
This is a limitation with Android and not a fundamental network/OS limitation. As long as your protocol allows for long enough keep alive latency you can periodically wake up for a short period and allow apps to service their connections (which is what I'd imagine this does anyway just with a singular app.) This works on Linux if you don't have the extra Android crap getting in the way.
It's kind of a bummer to see bad OS architecture feed into over-complicated protocol/application design.
EDIT: I guess I wasn't clear here. In my experiments I had the background wake up service wake the device up once and applications had just that singular window for all of them to service their connections. The problem with Android is instead applications either keep the phone from suspending entirely or (apparently) have some API allowing them to register their own periodic wake up that isn't shared.
EDIT2: I also want to be clear that this isn't a criticism of NTFY. The developers behind it deserve congratulations for solving a serious problem with FOSS Android apps. My complaint is that the poor architecture and unescisary inflexibility of Android made this such a complex problem in the first place.
> you can periodically wake up for a short period and allow apps to service their connections
For this to work, you need apps to somehow conform to a standard timing and respond to a signal that says "hey, the OS is awake for you to do network things". Now each developer needs to implement some special code that understands the situation. After a while, all the devs start using libSleepyNetwork because getting those details right is hard. The libSleepyNetwork people realize that they can use a system daemon to further coalesce work for greater efficiency (one event loop is better than N event loops), so now all the consumer program C APIs are now backed by IPCs to the daemon.
This is exactly what's already happened on Android, except libSleepyNetwork is implemented by either Google, or these people. I don't think it's bad OS design at all. It's chasing efficiency via abstraction that encapsulates some complex coordination.
> This is a limitation with Android and not a fundamental network/OS limitation. As long as your protocol allows for long enough keep alive latency you can periodically wake up for a short period and allow apps to service their connections (which is what I'd imagine this does anyway just with a singular app.)
The protocol does not specify how the distributor and push server communicate or keep a connection going. It's up to them to define an energy efficient way to do that. For ntfy [1], a single WebSocket or JSON stream connection is used to deliver messages to the device. This is very energy efficient and consumes <1% of battery on most phones.
As for regular wakeups (for polling), Android does not allow periodic work to be done more frequently than every 15 minutes, which is obviously not frequent enough to be useful.
> It's kind of a bummer to see bad OS architecture feed into over-complicated protocol/application design.
This is somewhat true. The restrictive Android eco-system is what led to the creation of UnifiedPush, but it's far from complicated IMHO
Disclaimer: I am the maintainer of ntfy [1], one of the UnifiedPush distributors.
You can do that, but if keep alive are not synchronized (I don't think there is a mechanism to do so), you will wake more often if you keep more connections open.
I think the OS could synchronize them if keep alive packets are sent at the TCP level, but that would be hard to do at a higher level.
Letting a single program handle persistent connections leaves a lot more potential for energy efficiency optimizations.
Google famously tried allowing app keepalives to be synchronised... The scheduler allowed a wakeup in 'approximately' X minutes, and the OS would wake up and run all apps callbacks at once. If any app needed a wakeup at a precise time, then other callbacks at nearby approximate times would be run at the same time too.
Overall, the thought was that wakeups are expensive, wakeups that involve the network are particularly expensive, so might as well run them all at once (they tend not to be CPU bound - often much of the wakeup handler is spent waiting for the network to connect and send a packet or an ack to arrive back).
This turned out to be a very very bad idea. All it takes is some app to have a 'wake up exactly on the hour', and now all other app callbacks will also run exactly on the hour. End result: Mobile networks get 10 million phones all waking up within a few milliseconds of one another, and everyone wanting to connect, and the whole mobile network fails because it was never designed to have 10 million phones all trying to connect within a few milliseconds.
Unfortunately there are lots of bits of hardware in mobile networks that fake acks - ie. TCP level acks will be received very quickly to indicate the mobile network has accepted the data, and will deliver it at some future time out to the internet.
The problem is they still send acks even if the remote end is no longer responsive.
So you can't reliably use TCP acks to know if a connection is still alive. You need to send actual data and have the application respond.
I would think that in order to not have an obvious negative impact on battery life, it would be necessary for the OS to coalesce app notification service keepalives, performing them in scheduled batches instead of the moment an app requests one. That should be roughly equal to the current situation with APNS/GCN, where the OS can schedule network maintenance during periods where the device and antenna(s) are likely to be awake already.
this is NOT a limitation of Android. Android has supported wakelock based notification managers for a long time. And FCM works pretty much the same as was mentioned in the OP. The FCM manager maintains one long polled connection to Google notification servers and every other app subscribes to wakelocks from that manager.
Your app is only woken up by the OS if FCM receives a notification for you. For e.g. Xiaomi will not wake you up even if you get a notification, unless you're on a white list (the much glorified Xiaomi whitelist).
When Android is put in power saving mode, the FCM manager optimized this even further.
Even if this were all synchronized, you’d still need to have all the apps memory state available - either taking up loads of RAM or requiring a lot of expensive paging.
I just read the specification you linked. If I understand it right, it just specifies what we call a "Push gateway" in our last diagram in the "under the hood" section.
As in, it specifies how multiple servers can connect to a single (XMPP-speaking) push server. This is also evident from the excerpt I'm quoting:
>XMPP Push works between the user's XMPP server and two push notification services in tandem:
> The user's XMPP server publishes notifications to the XMPP Push Service of each of the user's client applications.
> The XMPP Push Service (as defined here) for a client application then delivers the notification to a third-party notification delivery service.
> The third-party (and potentially proprietary or platform-dependent) push service delivers the notification from the client application's backend service to the user's device.
UnifiedPush is the third-party push (or notification delivery) service here.
Please make an effort to come across as less antagonizing in the future.
You seem to not see the forest behind the trees. It doesn't do 100% of what your UnifiedPush does, sure, but it does 99%. All it needs to reach the final 1% is to send a message with an agreed payload format to a client. A day of work.
As I said, we did it as far back as 2016: a watered-down XMPP client (a "distributor", as you call it) was used to deliver push notifications to a few different apps on devices that didn't have GCM (as it was called then).
> Please make an effort to come across as less antagonizing in the future.
Someone has to antagonize people who ignore prior art.
I almost used my sentence myself. The overlap is almost zero.
> All it needs to reach the final 1% is to send a message with an agreed payload format to a client.
This is the part we're interested in. We standardize this part and make it modular. The core issue is that multiple "client"s that do not interact with each other traditionally each implement their own payload format and background connection. Which drains battery. We propose a mechanism to delegate that to a third-party that can then multiplex those messages. That's all.
It's not a lot of work per se, but coming up with a modular API is not that trivial either, and we have to spread the word so that it can be used by multiple projects.
Lastly, you seem pretty focused on XMPP clients. Unfortunately (or maybe fortunately), not every app uses XMPP. On my phone, apps that could theoretically use UnifiedPush are: NextCloud, Etar Calendar, DavX5 caldav synchronization, K-9 mail, Element & others Matrix clients, HomeAssistant, FindMy & other device locators, covid tracker, Carnet notes application, Fennec (Firefox), KDE Connect, Mobilizon, Peertube Client, UnCiv, and various others, including chat apps and weather apps. XMPP would bring very little to those, especially as they just need that "1%" functionality you quoted.
There is some overlap indeed, as there is a server-server API. However, 99% of the spec you linked is completely useless or overkill for our use-case. We just accept POST requests to an endpoint and forward them to the app running on a user's device.
> As I said, we did it as far back as 2016: a watered-down XMPP client (a "distributor", as you call it) was used to deliver push notifications to a few different apps on devices that didn't have GCM (as it was called then).
That's pretty good indeed; was the specification open so that other apps could join in? Do you mind sharing a link? You can just implement the UnifiedPush specification in that distributor, make sure the server-side part accepts requests in the format specified, and you have a UnifiedPush-compliant application that other apps can use.
I've been looking for such a specification for a while, there's been some talks around the topic for years, including OpenPush [1], but nothing concrete or public that I know of. The link you shared in your previous comment does not address the "distributor" part.
> Someone has to antagonize people who ignore prior art.
Well, please at least try to understand the thing you're criticizing before, that's the bare minimum.
> Unfortunately (or maybe fortunately), not every app uses XMPP.
App doesn't need to 'use' XMPP to use push notifications. For developers, it is essentially identical to using FCM. XMPP part here simply offers a very effective way to decentralize the service, as federation server-to-server part in XMPP is wonderful, and top XMPP servers like ejabberd have great performance.
> Well, please at least try to understand the thing you're criticizing before, that's the bare minimum.
My company has created this very thing for a customer years ago, so I think I pretty much do understand what you are creating.
It wasn't open, yet, the effort to make it open would be pretty small. We could probably release it, but I don't see the real need for it, as it is a likely dead end: making apps run in the background on Android was what we did for years, and the writing is on the wall. Background running becomes more and more difficult with each new version of Android, even on pure Android phones. Biggest manufacturers, however, like Samsung, Huawei, Xiaomi simply kill off background processes without mercy. So it is more than likely we won't be able to run 'dispatcher' apps on Android, just like we can't on iOS, and we'll have to rely solely on push notification service which is built into the phone OS. If there is a fight worth fighting, it is the one to force phone OS developers to open this part of an OS to allow using custom push notifications services.
I am not trying to denigrate XMPP in any way, I also use it, though I am not very familiar with the internals. Here, the decentralized aspect doesn't seem to bring a lot of added value (what would you federate? Push senders and push servers? The relationship is mostly 1:1 in every case).
> It wasn't open
Then, I'm sorry, but even if the design was miles ahead of what is being proposed here, it might as well not have existed.
> If there is a fight worth fighting, it is the one to force phone OS developers to open this part of an OS to allow using custom push notifications services.
Well, we're proposing an API for it, and we will certainly push alternative OSes such as LineageOS, /e/ and others to adopt it; we hope it gets picked up by bigger fish, but we can't exactly pressure them into doing it, besides demonstrating that it works, documenting it and raising awareness, which is what we're doing here.
Please be nice. You can convey your information without the snarky tone. You may be factually correct, but your comment comes across quite rude.
From the HN guidelines [1]:
> Be kind. Don't be snarky. Have curious conversation; don't cross-examine. Please don't fulminate. Please don't sneer, including at the rest of the community. Edit out swipes.
> One key feature of UnifiedPush is that the communication between the push server and the distributor is not specified. This means that a variety of technologies can be employed, such as WebSockets, Server-Sent Events, *XMPP*, raw TCP, or even SMS, whatever works best for the user.
Sigh. What you miss is that this protocol introduces nothing that already isn't done by XMPP push notifications. It is decentralized, it can deliver to different endpoints, etc. You only need to create a dispatcher app that will relay the notifications to third party apps. Again, this too was done years ago.
So what! Few use that. I host a prosody server with lots of plugins and it never occurred to me to use it, let alone use cases outside if the XMPP ecosystem. Can the XMPP implementation fall back to google's push system. Does it have a blog entry, lovely diagrams, comprehensive docs, a nice on-boarding experience for developers? Ideas aren't worth much, implementation--getting things done is what matters.
UnifiedPush has been a thing for a while, see the official website: https://unifiedpush.org/
With this blog post, we† tried to clarify a few names. Feel free to point out things that are still unclear, the documentation is pretty much a WIP.
UnifiedPush aims at replacing the push notifications mechanism provided by Google services with something independent, that anyone can self-host or any OS can provide without compatibility issues. It does require small adjustments server-side and client-side for applications assuming Google services.
There is also a Matrix room at #unifiedpush:matrix.org and a Mastodon account at https://fosstodon.org/@unifiedpush
Note: we'll be monitoring this topic for a while, you don't have to reply to this thread.
† karmanyaahm and S1m are main authors