Guys, really, that is an awesome idea, especially when it comes to several countries in the world that limit the access to data. This data doesn't necessarily require multimedia, sometimes its all about access to basic information like dates for demonstrations, or other political topics and things like that. This could be really hand for hundreds of thousands of people who are just not informed and kept silent by their governments. Think about that when implementing. There is a not-so-small audience out there.
Do not get discouraged by all those people who are telling you that its too expensive. Those guys are sitting in behind a high-speed access and don't seem to look beyond the horizon, that sometimes its not about getting 2GB across with SMS (really? ... I mean... REALLY?!), its about getting basic access to information. Not every country is as awesome as the US or Europe in this world. Not everyone is free. And not everyone uses the internet the way we seem to got used to it.
Keep up the great work, its an awesome idea. For sailors, for people without access to internet, for quite a bit of an audience we like so much to forget. Keeping it open and turning down VC firms is even better.
They're using Twilio. Each SMS they send costs them $0.0075.
However, simply stripping the junk out of web pages for mobile delivery has real promise. Just put all third-party content on "load on user request only", like mail attachments.
Wow, that gets expensive very quickly especially at 3 messages/sec.
Assuming the average user gets only one burst of three messages per query, uses it only twice a day for 200 days a year and the service gets only 10,000 users signed up, the costs just to send the texts for the next year alone would be $90,000.
Seems like it will be hard to scale but I agree that this is a really cool way to get connectivity! Best of luck.
I don't know, next to paying $35/mo for a cheapo cell with unlimited texts and a USB gateway to SMS, you're saying that for moderate users they would need to pay about $9/year to get their internet access on top of that.
Seems reasonable to think that you might pay $5-9/mo on a budget for a limited access to internet such as this; now it sounds like potentially economically viable to even have an employee or two after paying for all the texts.
That would be pretty cool for a normal web browser too.
EDIT: I'm also part of the team, that comment looks really strange out of context.
We don't have specific monetization plans yet, but we're probably going to do some sort of donation system, or maybe ads (through wifi). No idea though, it is pretty expensive.
The price goes down with volume. They should also look at alternative providers. Nexmo and Plivo both offer free inbound SMS and have APIs similar to Twilio's.
To the operator, not the provider, Twilio in this case. Twilio doesn't send them for free. Operators charge for access to them and you have to pay, if you want to send SMS.
They also aren't entirely free to the operators themselves - once you start counting them and charging for them, there are administrative costs; then there are costs related to interconnections with other operators, and costs related to maintaining their own SMSCs (SMS centers). The only thing in this whole mechanism that the operator gets for free (actually not free since they have to pay for the network and base stations) is the delivery channel which is not insignificant but it is also just one part of many.
Wow. I knew it was cheap - that makes sense in light of what you're saying. Though there is a minor sting: "mobile networks charge each other interconnect fees of at least US$0.04 when connecting between different phone networks" [1]
I had a browser (WAP) on my Nokia 7110 that supported browsing via a SMS bearer; 1999. I tried it once and got billed up the wazoo, SMS was expensive back in those days.
There was also WIG. I was writing WAP sites in '99, incurring pretty large bills while developing and testing. The company had a very close relationship with the mobile company so fortunately we just got them written off.
I don't see this could EVER be economical. Even wholesale SMS (not Twilio/Plivo/Nexmo/Etc) can still cost $0.001 / SMS. There is simply no way they can operate the proxy/gateway profitably.
Unlimited SMS is common in the US too: But that's not really "unlimited" like you'd want to run a service of this scale. 3 messages/second is way too small. You could only host a single client per relay that way. The right way to do this is to not involve an SMS plan, and connect to a proper SMS gateway, which won't be "unlimited" at all.
If you (on the server side) connect via a mobile modem, yes, and probably violate the TOS. Otherwise, typical prices for bulk SMS in the UK is in the 2p-4p range per message (with higher risk of delays and failure to deliver towards the lower end of that scale). I'm sure it's possible to get below that if you have really huge volumes, but those prices are fairly typical in volumes up to millions of messages per month.
A big problem to solve is that SMS is notoriously bad at delivery. You may have SMS's dropped for weird reasons, and delivery reports aren't exactly reliable.
Yes but delivery is guaranteed by TCP, which is the layer above IP. In this scenario, SMS is "above" TCP in the stack, so unless they add ACKs to their system (which would be pretty prohibitive when every text costs money), dropped SMSs would break their system.
Instead of an ACK text, maybe an MD5 hash could be sent as part of the message. If the MD5 mismatches with the concatenated text message stream then you know you have a dropped or corrupted packet.
Unfortunately that does introduce additional complications like knowing which packet failed (maybe there's a better method for checksuming for this type of situation?), but even in the worst case scenario, it only adds a small overhead to infrequently resend the entire stream than to double up each stream with ACK responses.
Good idea - You'd also have to add a "length" header at the beginning. Otherwise if the final SMS (containing the hash) was dropped, you'd be waiting forever.
MD5 was more a proof of concept than a design specification. However 32 characters is only 1/5 of an SMS so it's not beyond the realms of a practical (albeit lazy) fix.
Ideally though, you'd be right that something more concise would be preferable.
Are there any advantages of using SailMail versus WinLink over HF? I can't say I'm significantly familiar with either system, but at first glance they appear to offer similar functionality, with of course the latter requiring an Amateur Radio license.
SMS is the most expensive data service on the planet. It costs more per byte to send SMS then it does to send data to Jupiter (factoring in the space ship you had to launch there to receive it).
WhatsApp's entire business model is based on the fact that SMS is just stupidly expensive.
My cellphone plan gives me unlimited SMS but only 2 gbs of data. If unlimited really means unlimited, then my data comes to me for free (assuming I was going to keep my sms package regardless, which I am).
Unlimited never means unlimited. It means the seller of the service thinks there's an effective limit to what most customers can consume and he thinks he can provide more than that for a flat price.
This generally works out well until people figure out creati e ways to consume levels of service beyond the effective limit -- then the service provider is oversold, and it's either underdeliver or invest in living up to new demand (probably financed with a price increase).
You can send an amount of data equivalent to 13,421,772 SMS with 2GB. That's almost 6 SMS per second on average. I'm sure "unlimited" SMS has a limit much lower than 1M SMS.
That's only in some countries (eg: US). In other countries, SMS is way cheaper for example, for the cost of a dollar or so, you can get a plan that allows you "unlimited messaging" or some obscene number of messages per day (more than you can reasonably type). For example, I know this was the case in India as recently as a couple of years ago -- and then they capped it to try and stop telemarketing spam.
So there are definitely places where SMS is cheaper than data access. In the US it's inverted purely due to the business model choice of the carriers.
These plans are only "unlimited" because nobody is using them as high bandwidth data channels.
Try to push high volumes through it, and expect to see SMS's delayed - SMS can be delayed for hours at the whim of the operator - or simply dropped, at the whim of the operator. Or simply rate limited.
In some countries (like mine), WhatsApp is not popular at all because unlimited SMS is cheaper than unlimited internet. We've had unlimited SMS for a long time now.
For all of those countries where maybe the best speed for internet is EDGE or GPRS, but where SMS are cheap, if not unlimited, this service is huge. I don't think the target is rich western world with LTE.
The backend takes the url, gets the HTML source of the website, minifies it, gets rid of the css, javascript, and images, GZIP compresses it, encodes it in Base64, and sends the data as a series of SMS's.
This is yet another interesting idea that will get screwed up in practice because of massive overreliance on JavaScript in modern web development. This is why developers should do progressive enhancement. There are a lot of thing that are possible with declarative markup that are not possible or not safely feasible with imperative code, which generates things on the fly.
Running arbitrary JavaScript on your server will likely cause performance, security and compatibility problems. It's not practical at large scale, and it's not something that would "fix" all the use-cases anyway.
Also, very few companies and no individuals will have the same technical capabilities as Google with it's horde of developers, years of experience in the domain, billions of dollars and ability to control everyone's visitor stream. They shouldn't be the only company that can effectively innovate in this field.
Here in Brazil internet on mobile is expensive and limited, and the SMS is easily found free and unlimited. This project can potentially improve mobile Internet here.
Remember me the Opera Mobile case that was largely used in Africa where 3G was too much expensive. I remember to saw in some news that Opera brought digital inclusion to many African countries with their browser that basically shrink the requested pages at server side before sending to the phone.
I don`t know about the speed and reliability, but at least, it would be a option.
Right, but with the GSM 7-bit alphabet they could easily expand their index table to 127 entries rather than Base64's 64, netting them much greater efficiency.
Yes, Base64 encoding will expand a payload by 33%. However, SMS limits the character set you can send easily, with the most common mode of transit being the 7 bit SMS character set. With the SMS character set having an escape character that can get in the way of doing a direct encoding, I imagine they went with Base64 because there were already libraries for it, minimizing the work they had to do at this stage.
You can send anything in the GSM 7-bit alphabet, which has 128 characters rather than Base64's 64-entry index table.
They could gain a lot of efficiency by using a 127-entry index table (or 128 if the ESC character doesn't need to be avoided - I don't know how the Android SMS APIs handle it) rather than a 64-entry one.
On most plans offered in the US nowadays texting is included. All of the T-Mobile, AT&T, Verizon, Straight Talk, Republic, etc plans have unlimited texting included.
It appears the HTML of the requested page is returned to the requesting phone via SMS, which could be many many messages if you are loading say, a wikipedia page.
I know that in the past in US and Canada carriers billed incoming SMS, but I thought that alongside with the new age of messaging apps like WhatsApp and iMessage they dropped this nonsense.
In Europe even when in Roaming there is no cost for incoming SMS.
Is it a web browser? If so, then I don't see the point. Why not simply write some kind of driver that does IP over SMS; then any app that uses the net could connect that way.
(And if it isn't a web browser, then why "browser"?)
Well, I'm sure this was a nice achievement, and no offense intended, but that doesn't make sense. You were limited to 36 hours, and therefore, instead of simply writing a driver, you wrote a driver and and entire web browser sitting on top of it?
Surely they've reused a browser component rather than writing one from scratch. And they do more on the server than just pipe the full HTTP responses back, they strip CSS, images, JS etc. and minify the HTML. I think it's pretty cool!
But we might take it as a negative comment on the modularity or ease of configurability of the environment supported by Android (or, more generally, by Linux).
Interesting approach. I wonder if this HTTP-specific approach is better or worse than coming up with some form of generalized TCP-over-SMS stack (with accompanying server-side router software, of course).
Reminds me, though, of Georgia Weidman's Smartphone Pentest Framework[1] which provides tools to hack a target smartphone and from there create a data connection via SMS via which you can then access a firewalled corporate network using SMS over the cell network as your tunnel in. I've seen it demoed, and it's frankly incredible.
Coming up with new ways of compressing data to travel down legacy or fundamentally anti-functional data channels (i.e. low bandwidth, low quality of data, high latency, preprocessing that destroys the majority of the experiencial data) is always a fun hack, but is never a marketable strategy for a few reasons.
- It bets on technology going backwards rather than forwards (i.e. will have cheap android phones with cheap data plans before you can scale it)
- It doesn't provide much more than novelty self promotion. "Hey i made a webpage deliver via SMS"
- Its usually more expensive and time-consuming to produce what almost always becomes an inherently inferior product.
Now, let's take those 3 points and consider a 3rd world country with interwebz-censorship or very poor infrastructure and no forseeable future where it'll improve to anywhere near what most first-world countries have. Cellphone coverage is the easiest thing to setup in those kinds of environments.
Do you have any examples of such countries where at the same time SMS will still work and be cheap enough to be viable?
Even most countries in sub-Saharan Africa has at least GPRS - most of them have 3G. Some of them are deploying LTE. Certainly there will be areas where for some time still there will still be limited data access, but the question is how many of those areas actually has functioning cell coverage.
THANK YOU for this very lucid post!!! People who have access to these services almost certainly have access to a data plan and would view it as a novelty rather than a means of communication. I learned this the hard way by having done a very similar project a year ago, but then basically lost interest as soon as I got a smartphone.
It's open-sores and free to set up and use (if you have a text plan) It would be pretty easy to turn it into a sms terminal which might be negligibly more useful (using a PTY with the kbd is easier than using an X Server). I guess the OP's project and mine would be perfectly equivalent if anyone's written a text-only, line-based browser, but I stopped short of this. Maybe someone else will make one? (:
My biggest takeaway from the whole project was how easy it is to fall into the trap of doing something just to be able to pat yourself on the back. Real altruism isn't this cheap.
You could sell this in a prepaid kind of way ? 30 webpages for USD 1 or something. So I would first have to buy credit and then each requested page is deducted from my credit.
Assuming the user has to other internet access, you could offer access to the login / payment / info pages for free. However, you do need some secure sms payment option. I believe all current sms payment options are operator dependent. A huge business in Africa if I recall correctly. An independent version might be another opportunity ;-)
In germany unlimited sms has only been a thing since ip based messengers like WhatsApp took over messaging and nobody is using SMS anymore, before that SMS were quite expensive after you hit the budget of your contract (typically about 50-100 SMS). Now, unlimited SMS is the norm and things like this browser get interesting. But even if it would be a great working alternative, it will only work until carriers adapt. I really want unlimited data...currently i get 750MB with a contract that costs EUR 50.
I'm not sure how badly I want to find out just how unlimited "unlimited texting" is, especially at 0 extra cost a month. Of course it's included in the price somewhere, but I can't turn it off to pay less. Suddenly receiving thousands of text messages a minute after hardly using sms for years doesn't sound like a great idea...
Calling my number and using the good old dial-up method sounds much more efficient actually. Isn't the bitrate around 128kbps in normal calls?
In most parts of the world, POTS is 64 kbit/s, over which you can reliably run a 56 kbit/s modem. Cell phone audio uses less bandwidth, but I can imagine that you could get 8 kbit/s or more reliably, compared to the SMS data rate (7 bits/char * 160 chars/msg * 3 msg/sec = 3.4 kbit/s).
GSM data used to be 9600 bps back in the day when you'd connect a separate GSM modem. A full data rate raw GSM channel is less than 13kbps, and you'll lose some to signalling.
> The phone recieves this stream at a rate of 3 messages per second
So note that's 180 text messages a minute maximum throughput, not thousands.
I think you raise an interesting question of if it might be faster to implement a software modem over your voice connection, but I also have no idea if that's actually feasible or not.
(will various kinds of digital audio compression or filtering get in the way? I dunno. Can android apps easily get access to the voice audio stream in the way they'd need? I dunno.) (it'd certainly be a lot harder than letting the SMS network take care of the transmission layer, at any rate). (But yeah, if internet over SMS is a cool hack, then internet over software modem over audio would be even cooler!)
I wonder if they could do some intelligent parsing of the content and just return a plain text version of what the user requests. It wouldn't work for everything, but for things where there are articles with a defined "content" it shouldn't be too hard.
In fact, isn't that what the "read it later" type apps do? I'll bet there is a free js library that already does the detection or you.
Awesome app. I was thinking about something like this recently for a niche.
I was going to ask how you direct the SMS to the app, but taking a quick look at the src, it seems the app reads the sms inbox, looking for the specific phone number?
I guess this means that:
- the user sees alerts for the incoming sms?
- the user needs to delete (does the app/could the app do that?)
Great stuff, really interesting. Thanks for sharing it.
I'd do the jquery function .text() on a page, spaces and new lines with a max sequential occurence of 1 and would send that through in markdown (to get some layout).
The browser contains same basic stylesheets and the user can change this (h1,h2,p,...)
Keep the link functionality with # (for single page apps) somehow, for links {{the text}{the link}} could be used.
All modern cellphone systems are already digital, and the voice channel is heavily compressed using codecs that are typically geared for a human voice. It's pretty much impossible to get a high datarate with an acoustic modem over them, if you can even get a connection at all.
While there are higher bitrate codecs, a full-rate GSM channel is only about 13 kbps, and depending on radio conditions the voice data may be compressed down to half that or even less.
For comparisons, when landline connections are digitised, it is typically compressed to a 56kbps or 64kbps channel.
Agreed it normally would be more expensive. Although I'm on a plan that gives me a few hundred minutes of talk time per month that I don't come anywhere near to using.
In terms of saving on data, I think Opera Mini [0] is a good compromise. They compress the hell out of the page and images (there's even an option to strip images away), but they don't mess with the layout.
I think this is an interesting proof of concept; it seems like the exact opposite of WhatsApp, which is based on the concept that data is cheaper than SMS.
I know that SMS plans are often "unlimited" in the US, which is why (at least from what I've seen) WhatsApp is far more popular in Europe.
SMS isn't going to get good bandwidth, given the path it uses over the mobile network.
But I've thought about building HTTP-over-WhatsApp. Mobile providers here offer prepaid contracts where you don't pay for WhatsApp access, regardless of your balance or traffic remaining.
Would this allow users to circumvent internet censorship? It seems like that as long as the main server was outside of the censored zone, SMS traffic would be able to get information in under the radar.
Love it. A few q's: all traffic on your servers is plain open? "The phone recieves this stream at 3 messages per second", how about upstream? How much data (kb) fits on each message?
Could you achieve a more high throughput version over voice? I wonder, after compression what the data rate ends up being. I'm reading gzip can get 2:1-4:1 with text. So about a kb/s?
I may be wrong but aren't we going backwards here...
data over voice=dialup right? 56kb/s ideal? Still possibly useful but this just reminds me of the wep browser days on my old nokia
56kbps over landlines. That works because the landline network, when it is trunked, is compressed into 56kbps or 64kbps channels using methods the 56kbps modems were explicitly designed to take maximal advantage of.
Over a cell phone connection you will get much less. GSM data modems gets to bypass the voice codec, and are still limited to 9600 bps. An acoustic modem over a GSM connection will get less than that, if it will connect at all: A full data rate GSM code is compressing the audio down to about 12.2kbps using codecs geared towards reproducing voice as well as possible, which is going to be brutal to a regular modem and far more wasteful than GSM data.
It's unfortunately pretty slow. I mean, when you factor in the overhead for text messages, it adds up. We haven't tested it enough but it definitely won't match 4g. We also are planning to switch to a better compression algorithm.
I'm guesstimating you'll find an upper bound, assuming "friendly" telcos that don't start rate limiting, or dropping SMSs, at no more than about 6kbps. Quite likely less.
GSM data (which is normally possible anywhere where you can get a GSM voice connection) is 9600bps and uses the full, raw GSM data/voice channel. You'd not be able to exceed the channel bandwidth. Then you have the SMS overhead. And you're unlikely to manage to max out the channel.
Before this fancy GPRS thing, we used to have something called CSD (circuit-switched data), where your phone could use a raw GSM voice circuit to send data instead of digital audio.
I think we lost that feature in the transition to 3G.
that's what error correcting codes are for :)
If you design things to play nicely with the compression algorithm that's used by trying to stay within normal human vocal range and stuff, might be okay.
True, but if you were somewhere where data was unavailable or you didn't want to use a local data connection, it might be a nice way to force a map app to update to your current location, get an email, send your location to somebody, or answer a chat message.
Or you could text them directly. This scenario is only useful in the case where you have SMS but no internet, and your intended recipient has internet but no SMS.
EDIT: Also, as someone pointed out above, Twitter already has an SMS interface.
They're just a terrible example cause they're the one service you might already have access to that way. Twitter's only got 232 million users so anything that hinges on them is immediately not as useful as something that makes more of the internet - fb, email etc - available to more people, especially from someone else's device.
It's becoming common for Twitter to used for updates and coordination in disaster zones. And I guess if I wanted to let my friends know I was safe in such a situation, and didn't want to text them all, Facebook would be a good choice.
I've actually been working on a side project related to this since about 2006.
Currently I have a working email-based proxy server, with Browse (HTML to email), Translate, Base64 Download, and Base64 upload.
I send the URL in the email's subject line, and Google App Engine fetches the page and replies to the sender. Base64 uploading is sent via POST to a PHP script on an OpenShift web server.
Email to SMS is also possible via gmail2sms (their blogspot is in Russian, but it's legit, I promise). That uses Google App Engine to copy emails to Google Calendar, which sends SMS event notifications for free. You need to authenticate your number first though.
I'm still missing some features though. Originally my purpose was to use SIP (e.g. Jajah, Voipcheap: Betamax services) to dual-dial a payphone and another phone number for a 5 minute free trial call. I almost got that working in early 2013, but ended up being banned by most Betamax services.
The other is how to initiate communications. I found a hack that allows me to send email for free while on prepaid 3G roaming with zero balance. There was another hack before that allowed me to receive email as well, but the website's format changed and it's no longer working. Also, most people just want me to have a "normal" local phone number instead of using my Google Voice SMS-to-email.
I've learned a lot about web technology through this ongoing project, though. Now I can code PHP and Python, browse using Elinks, script using Expect, and more.
On a meta-level though: is it worth it?
Internet access is basically used for just a few purposes:
- Information recall (Wikipedia)
- Maps (Google Maps)
- Communication (Email, Facebook, Twitter, Skype, etc)
- News
The first two can be completely offline using Wiki2Touch (7.5GB text only Wikipedia) and Galileo (1 GB map of Taiwan). It's only communication and news that change often, so I need to access them.
I can get news through FM radio (does anyone still use that? I'm pleased to have it on my Sony MW600 Bluetooth remote).
Facebook and Twitter have email APIs. Skype doesn't. WeChat, WhatsApp, KakaoTalk, QQ, LINE, and all those other chat apps are terrible for their lack of standard APIs. Why can't we all just use instant messaging that's compatible with email or Jabber? For the same reason, these apps aren't available on my Raspberry Pi.
The Cosmos Browser's SMS gateway doesn't provide communication links, just HTTP GETs of static pages. Pages that I can save offline beforehand.
Congratulations to the developers for some impressive coding, but I can't yet find a killer app for this.
And we should all switch to simple email instead of those pesky chat apps.
Do not get discouraged by all those people who are telling you that its too expensive. Those guys are sitting in behind a high-speed access and don't seem to look beyond the horizon, that sometimes its not about getting 2GB across with SMS (really? ... I mean... REALLY?!), its about getting basic access to information. Not every country is as awesome as the US or Europe in this world. Not everyone is free. And not everyone uses the internet the way we seem to got used to it.
Keep up the great work, its an awesome idea. For sailors, for people without access to internet, for quite a bit of an audience we like so much to forget. Keeping it open and turning down VC firms is even better.
GREAT JOB!