>We shot off an email to Apple Technical Support — one of the two free emails Apple allows developers to send (after which they cost $99 for a 2-pack).
Wow, this statement is such drama.
There is no "allowance" for how many emails you can send to Apple. The Apple developer evangelist email is public and emails to it are not limited. The author is confusing emails and TSIs (Technical Support Incidents).
You can also call the developer support line, which picks up after a couple rings with a live human. And you can file bugs. And you can tweet to the developer evangelists, which may be the fastest, most effective way to get an urgent problem like this on their radar.
The two TSIs that are free with your developer account (and then cost after that, if you ever even use your first two) are for code-level support, meaning "how do I accomplish this with code." This is clearly spelled out (see https://developer.apple.com/support/technical/submit/).
They will get your question answered, if an answer is possible. Apple pulls out all the stops to get these resolved, and it's a huge misunderstanding of their purpose to use them for an outage report, and further to claim that you can only send two emails to Apple without paying.
Far from being indifferent, the Apple developer relations team is hitting it way out of the park. Yes, outages happen, and that's unfortunate. But there's no need to make something out of it that it's not.
A similar problem exists on the Android Platform. The Geocoding (forward and backward service) is non-reliable, sometimes requiring a device reboot. But, at least there is a public issue tracker!
I'm not a developer, but I think it's clear that as a developer you should expect your relationship with Apple to be similar to Google's relationship to users. They rely on both to be successful, but their bread gets buttered by someone else. In Apple's case that's consumers, so you're not going to get the love that consumers would get.
Actually, Apple's one of the main selling points is the apps(remember "there is an app for that" ads?). Apple definitely wants to make developers happy but I imagine this is very hard thing to do on scale.
I don't think that Apple would risk to lose it's developers to other platforms, they should have witnessed how Windows Mobile desperately tries to boost it's app numbers and how Android pushed hard to reach big numbers of apps for everything. Their Mac platform also suffered from "everybody supports Windows" situation.
Apple also tried to stop cross platform apps getting in the AppStore by banning apps made with 3rd party tools(I believe they rationalised it by saying that it's about the app quality but later they got less stricter on this).
So yes, Apple wants to make developers happy so that IOS platform gets the best apps but it's just hard thing to do.
However it would have been better if Apple had a competitor on the premium apps business, Google's money comes from advertisement which probably keeps them unmotivated to transform their store to more premium marketplace. If Android becomes serious money-making platform this probably would push Apple harder to solve developer problems too.
> "Apple definitely wants to make developers happy but I imagine this is very hard thing to do on scale."
The downtime isn't the issue, it's the lack of communication during the downtime. Things go down, it's inevitable, even Google suffers outages from time to time - but we've collectively been doing this web API thing for a while now, and there's are certain best practices that should be obvious by now.
Have a status page that's up to date. Post updates to the status page. Give ETAs when you have them, or state you don't have a ETA when you don't. I shouldn't have to waste customer service resources calling or emailing someone about this - if your service is down you should know it before I do, and should keep the status page up to date as needed.
I don't think this is really about Apple's care (or lack thereof) towards their developers. This is about Apple simply sucking really hard at anything related to the web. Despite this being the year 2014 they still have to take their store offline to make major product additions. iCloud was a veritable disaster for the first year of its existence. Their Maps data is still a cruel joke. And, evidently, their core OS web APIs can go down silently with no spokesperson or page to inform anybody.
This is, IMO, the weakest point of Apple, and where Google is most able to eat their lunch. The smartphone market is maturing, and IMO the innovation forwards will be less in the hardware side (where Apple is good) and more in the software services side (where Apple is atrocious).
>This is about Apple simply sucking really hard at anything related to the web
Like having the #1 app and music stores? I'd like to suck like that!
>Despite this being the year 2014 they still have to take their store offline to make major product additions.
I don't think that's any technical issue. That's how the want to present new products.
>Their Maps data is still a cruel joke
That are OpenStreetMap, not theirs. They just put a face on it. And its comparable with Google Maps, depending on where you live.
>And, evidently, their core OS web APIs can go down silently with no spokesperson or page to inform anybody.
Yeah, because if there was some spokeperson to tell the usual corporate BS we get from other web API vendors ("we are working to locate the problem" (well, duh), "seems to be an issue with our West Coast facilities" (well, duh), etc, it would really make a difference.
That's beside the point. It's all web services, just with a native front-end. You might not be able to access it from a web browser (though you can see app info in one), but it's still http all right.
Besides, OP also mentions Apple Maps as an example at "Apple sucking hard on anything web related" and you can't access that from the "web" either.
The web != just the ole HTML "world wide web". We know also have native clients. We still call services with native consumers "web services" after all.
And in fact iTunes is just a glorified browser view (albeit with a custom format now IIRC, used to use plain HTML at some point).
>This is about Apple simply sucking really hard at anything related to the web
I concur. My point was to challenge the statement about Apple not caring about it's developers. I don't defend Apple's way of handling this, there are industry standards to handle service outages, Apple should implement them.
"Have a status page that's up to date. Post updates to the status page. Give ETAs when you have them, or state you don't have a ETA when you don't. I shouldn't have to waste customer service resources calling or emailing someone about this - if your service is down you should know it before I do, and should keep the status page up to date as needed."
Those aren't industry standards. However with the exception of posting updates, I totally agree that Apple should do this with every service they operate, whether customer or developer facing.
That was him being imprecise in his terminology, which really only matters for those committed to nitpicking. Replace "industry standards" with "industry best practices" and the meaning of the comment is practically identical.
>The downtime isn't the issue, it's the lack of communication during the downtime.
What would that communication be?
If I'm a team of sysadmins working to find out what caused the issue and trying to fix it properly as soon as I can, the last thing I want to do is being asked by some manager type "well will it be ready".
It's not as if you can tell when a bug will be found and fixed, or if some idea you try will definitely work. The only answer is: "it will be ready when it's ready".
And in fact most other answers are BS. How many time have you heard from hosting providers for example: "We had a glitch in our network, but it's fixed now" -- and the issue persists or comes back with a vengeance a few hours later?
My perspective is that Apple really cares about their users, and only cares about developers inasmuch as it affects their users.
Unfortunately I'm a little jaded by some experiences I had with Apple developers at WWDC. Last year I tried working with a dev there to set up an XCode Bot to make Ad Hoc builds. He basically said we were doing it wrong and would have to use developer signed (not Ad Hoc) builds for QA. I asked him if they had ever talked anyone outside Apple about how they do CI or QA on apps before they came up with Bots... painful silence.
I'm pretty sure apple doesn't internally have to deal with app signing device limits and bullshit. The limit was 100-110 devices per account since the start, but now there are 5 generations of devices vs. 2 generations and 5 different types vs. the 2 they started with.
So why not develop your app for Android first (Dark Sky isn't available on Android)? I'm starting to see this too many times. Developers developing for Apple first, and then complaining about how badly Apple treats them. Maybe if iOS wasn't the default platform for all the hipster developers, Apple would start giving a damn about small developers.
I've never owned an Apple device (and I'm not a huge fan of the company).
That said if I had to develop a phone application for my company tomorrow I'd develop for iOS first and Android second.
There are good reasons for picking either platform but Apple has a single big chunk of the market which simplifies development and testing (Android has gotten better at this).
> Maybe if iOS wasn't the default platform for all the hipster developers, Apple would start giving a damn about small developers.
Nothing in this article mentioned anything about hipster developers no implied it, so apart from making you look a tad silly I'm not sure what the point of that bit was.
In addition not all small developers are hipster developers which seemed to be the implication.
Companies always provide better service to those customers who earn them the most revenue, this is not something new, try getting an engineer at Google to explain why they broke imap again if you have a dozen people using it then try if you have 10,000.
In fact prioritising such stuff by revenue is an entirely rational viewpoint when you have less than infinite resources.
Even with Apple-related headaches, developing for iOS is far simpler than developing for Android. You don't have to worry with making your application compatible with thirty different version/device combinations—more like eight. Additionally, iOS development is more profitable than Android development, despite having a lower userbase.
Your use of the word hipster is unclear. Does that word even have a meaningful definition anymore?
You don't have to support all the way back to 2.2 if it's too much trouble given your resources. Designing for 4.0+ will already give you more than 80% of the market, [1] and the 20% that you are excluding are the most likely to be underpowered and cause 80% of the device-specific glitches you would find in the wild.
I have developed for both recently. Developing for iOS is significantly more simple. I found trying to support many screen sizes on Android a nightmare. Also doing something simple like letting my app take a photo, save it, and display it was unnecessarily complicated.
I've worked on an applications on iOS and Android with over a million downloads each. I found that creating and maintaining applications in iOS was significiantly easier than on Android. While I have run into some Apple bugs, it's overshadowed by the dances I have to do to get Android working everywhere.
While I agree that parent used some words like hipster 'poorly', I completely disagree with your assessment of iOS and Android development. What you said is just pure FUD.
Both platforms have their share of problems and neither of them is 'simpler' or 'harder'. It just the way it is.
But neither really makes any money for the vast majority of developers. At least not from app sales. And if you were to divide it between games and non-games the difference would be even larger. The ship has long since sailed for making money by selling an app on an app store. Might as well play the lottery, same chances.
Let me be clear about where I'm coming from by stating my (extremely unpopular) opinion of Apple here: My view is that Apple is a hybrid marketing/hardware company that sells a kind of tech faux haut couture. I'm not a fan of Apple.
That being said, I absolutely will deploy to iOS before Android, because that is where the market and money is. Android markets to a broad swath of people, a huge of segment of which consists of lower socioeconomic classes, and the conversion rate for purchases is lower (my experience) for the same effort (I actually find developing for Android to be slightly easier than iOS, in spite of the unfortunate choice of Java as the language, but the fragmentation issues others might throw at you by way of countering your argument are largely irrelevant for the kinds of apps I'm interested in producing). Apple product users are generally in a higher socioeconomic class and generally are more willing to part with their dollar. It just makes business sense to target that first to get revenue, then target Android to extract value.
I don't have a product on the market so perhaps I'm being incredibly naive, but why didn't the OP throw Apple under the bus? It seems to me a better response than panicking, using up all your free support e-mails (to no avail), and etc.
There isn't really anything they could do. Even if they switched to a non-Apple geocoding service, it would still take over a week to get the new version of the app approved, and by then the Apple service was working again. Such a service might even cost money, hurting their ability to keep working on the app, and might even be less reliable than Apple. Then you'll say they need to write support for multiple services, but that may not be worth it to save 3 days location search down time a year vs. some new feature users will use every day.
Rereading it, it is ambiguous, but their wording ("I awoke to a deluge of user emails alerting us to this fact, and we quickly identified the culprit: Apple’s forward geocoding service was returning a 500 error code") gave me the impression that their app does not put up an informative error message, but instead "just didn't work", thus forcing the developer to spend time to figure out what was wrong.
If my original impression is correct, there is something they could have done: code more defensively, and give precise error messages ("the server is there, but..."). It is a little thing, but it can mean the difference between a deluge and 'just' a large flow of emails.
It's even worse because iOS users tend to have far greater love for Apple than for any app. Due to the degree to which Apple has managed to get a significant fraction of their customers to tie their self-image to ownership of Apple products, blaming Apple is just going to piss a lot of them off.
True it is not a solution but taking the fall for Apple is not good business practice either. In the spirit of full disclosure I would tell my users what was going on, point them to tweets, forum posts that showed that it was indeed Apple's issue and my brand should not take a hit for it.
Where are you guys getting the impression that Dark Skies didn't do this? I'm sure they did, but as I said it's not a solution, so it doesn't do anything to relieve the situation. Users won't be happy until the problem is resolved (and they shouldn't be.)
Ok agreed it is not a solution but since the functionality mentioned depended on the service Apple was providing, apart from telling the users that there is a problem with Apple or switching to an alternate service (not as easy as it sounds) during a crisis what other solution would you propose? Granted building a fallback into the design at the beginning would have been a better way to do things but I would forgive the OP not doing this since it is Apple they are dealing with and there is a certain expectation that your support tickets will be addressed.
> Granted building a fallback into the design at the beginning would have been a better way to do things
Notice that, that building the fallback is not building a fallback for the location API but for all the cloud based APIs that the app uses (or atleast the critical ones.) Then there is impedance mismatch between each of these APIs.
Definitely useful to do but sounds like a lot of work.
Agreed, If the fallback is using another service provider's API. I wouldn't recommend that because you then have to maintain version changes to each API. A fallback could also have been caching, slowly adding data to your own solution as users used the API so that the outage would affect some users whose queries were not cached or part of own solution. There could be more fallback strategies but I totally agree with you that using fallback APIs is not a good idea in terms of long term maintenance.
I used it for a personal project recently and it really easily allows you to switch between geocoders, which was useful for the edge case that Google didn't get.
> Using client-side Maps API services (JavaScript and Flash) in the browser is rate limited per map session. That means that requests are distributed across all your users and scale as the number of users grows. Therefore, client-side APIs should be always preferred and used whenever possible.
Google's ToS used to, and may still have, a clause that only allows the geocode API to be used in conjunction with maps, i.e. you can't use the geocode results for whatever you want, they have to be intended for display on a Google map.
I used MapQuest's free tier awhile back which didn't have the map usage restriction but was limited to 5000 queries a day, enough for what I needed but probably no where near enough for many applications.
>How does that happen? How does such an important service — the geolocation of addresses — just stop working for three days? And as of the writing of this post, five days after the fact, we have yet to get a response to our support inquiries.
It's not like there could be any other response than "It will be fixed as soon as we can". So, what exactly were they expecting to hear?
Regarding their customers, I'm not sure if this is an American vs Europe thing, but I never, ever, contact software companies for support. Nor do I know many (if any) people that do. It's 2014. You can read the manual or search for help online. And if a service is down, you wait it out and assume it will get back up. It's not like they don't know it's down, or telling them will make it go up faster. That's just like honking while on a traffic jam. Just adds noise.
Were those users really that dependent on a $1 dollar weather app that they had to "deluge" its developers in a 3-day service disruption?
I would either wait it out, or download another comparable app.
There are absolutely responses beyond "it will be fixed as soon as we can".
For clients of network services, it can be useful to know details about the scope of the outage (when did it start? who is affected? are there workarounds?). Yes, during the initial investigation there is often little to say, but even just opening up a communications channel signals to your clients that you take the outage seriously. Once a root cause has been identified, it is often possible to provide an ETA on the fix, which may take hours. Again, being able to say simply "it will be fixed in N hours" and delivering on that statement shows your clients that you know what you are doing.
Three days of outage of a externally-facing service with no communication shows that it's still amateur hour at Apple as far as network services are concerned. Full disclosure: I work for another tech company that specializes in network services, among other things.
This has happened to me before many times, we were working on a large app recently that used in app purchases and the sandbox was down for two whole weeks. Try explaining that someone who is paying you by the hour..
The only reply I ever really had was on an forum with "We're looking into it" and that was a few days before a fix was made
Rather than indifference, could also be a sign of how far Apple has to go to become great at networked services. The processes you need to maintain quality (including uptime) for a service with variable load, potentially frequent releases, actual racks of servers to keep humming, etc. is much different from the way you ensure quality in yearly hardware and OS upgrades. It's not counting Apple out or discounting the strides they've already made to note that Google and other Web-native companies still have a leg up on them here.
I've had a bug in Apple's bugtracker open since February 2013. I just rechecked the behavior of the bug and while it seems like it's no longer causing a crash, it is now messing up OpenGL's state.
A server being down is not a bug -- the app should expect 500 is a possibility and deal with it. In this case, why does Dark Sky need to geocode? My Lat/Long is enough to know my weather.
Even though I think this is kind of a snarky comment (and they said lat/long worked, the issue was with searches), I do agree with the underlying premise. As other folks in this thread have said, just letting people know Apple is having a problem is not an acceptable solution...Location search would still be broken, and they would still be sad.
Anyway, again, I agree with the underlying premise. It seems like overkill to need to have a fallback geocoding service, but I've use Geocod.io in the past and it seemed pretty cool. I have to say though, for a single application it seems like implementing 2 of every external service you use is a ridiculous thing to expect.
You bring up an excellent point against all the folks on here saying Dark Skies should just have had a fallback geocoding service in place. By that logic every application that uses a third party service should code in a primary and a fallback. That is unrealistic!!
Why is it unrealistic? For most common services, more than one alternative API provider exists. Why not think about "what would happen if that service I do not control goes down?" and have a plan B ready for that?
That's like saying deploying a service on more than one machine in different data centers in case one of them goes down is unrealistic. We've long past that and long recognized that it is not only realistic but necessary once you're on certain scale.
The argument is for a smaller business not for a business at a certain scale. You can argue that at a certain scale you can build out your own solutions and not rely on third party APIs, at a certain scale you can purchase the absolute premium platinum support package so you can be sure your email is answered within minutes. At a certain scale you can do a whole lot of things but this is about when you're not at that certain scale. so let's think about context here.
There's a big difference between using 2 existing geosearch APIs and building your own geosearch API. I'm saying the first one is feasible even for a smaller business.
yes if it was only geosearch API that one had integrate with. It would be manageable to maintain API changes for 2 providers for 1 service. As soon as you increase the number of services you use your solution soon becomes non-trivially complex. Consider a travel app that uses a geosearch API, a mapping API, a weather API and a enroute data points API (attractions, gas stations, hotels, etc.) Would you suggest the app developer to code 8 API integrations, a primary and secondary for each of the services and maintain them for API changes?
A service being unavailable for several days without any available status updates for affected customers is a pretty big failure.
I upvoted you because you make a good point about the fact that apps should handle network/service failures appropriately such that ideally it isn't a crisis but that doesn't make it an acceptable level of service.
100%. They don't make any money off of you using their geolocation. They provide it as a service to encourage development of iOS apps, not as a money maker.
Concluding that Apple doesn't care about developers has become a vogue statement to make partly because Marco has taken to saying it routinely.
It's not totally unreasonable to needle at Apple because there are lots of things they could do to improve developer relations, and openness about service status and reliability is clearly one of them.
On the other hand, taking it as true that they are actually indifferent takes a bizarre leap of logic, given how much effort they put into the tooling, the documentation, WWDC itself, making WWDC content available to those who can't attend, etc.
It's obvious that they are overloaded and are having problems scaling support for their developer community. It's far from obvious that they are indifferent.
[edit: downvotes - of course, because I'm not vehemently attacking Apple. Bear in mind that I don't think it's bad to needle at Apple, but I think it's only because people know that Apple does care about what developers think that it's worthwhile. Meanwhile let's not get things out of perspective.]
It feels indifferent because talking to apple is like talking to a wall. You never get a real response, ever. And if you do, it can be the height of laziness. All of those things also benefit apple internally for their own development work.
I don't deny that it feels bad. I have filed radars and had them trivially closed or marked as duplicates.
However, you are flat out wrong to say it's "Lazy", because that implies that they have the time and resources to devote to a more thorough handling of these cases. I'd prefer that they spend more time fixing bugs and improving the software than handholding developers.
Now this doesn't mean that I think they are doing it all perfectly. They certainly should have a service status dashboard for their backend, so that the forecast.io guys could have pointed at Apple as the culprit with authority.
Apple counts every duplicate radar as a vote for getting the issue fixed, so at least your filing of these wasn't wasted, assuming they were good issues.
See my experience is entirely different , the 3 bug reports I've filled got replied to eventually and dealt with in OS updates.
There is an argument for catching a server down response and alerting the user that the third party server is down.
But end of the day apple do suck when it comes to getting a fast reply , as a enterprise customer of Apple we often feel like its worthless talking to our reps. Since outside the US no apple rep has any pull in any way.
Probably a lot worse in terms of personal interaction, but then Microsoft scaled up as a platform decades ago and so have had much more time to build out this part of their operation. That is kind of my point.
Wow, this statement is such drama.
There is no "allowance" for how many emails you can send to Apple. The Apple developer evangelist email is public and emails to it are not limited. The author is confusing emails and TSIs (Technical Support Incidents).
You can also call the developer support line, which picks up after a couple rings with a live human. And you can file bugs. And you can tweet to the developer evangelists, which may be the fastest, most effective way to get an urgent problem like this on their radar.
The two TSIs that are free with your developer account (and then cost after that, if you ever even use your first two) are for code-level support, meaning "how do I accomplish this with code." This is clearly spelled out (see https://developer.apple.com/support/technical/submit/).
They will get your question answered, if an answer is possible. Apple pulls out all the stops to get these resolved, and it's a huge misunderstanding of their purpose to use them for an outage report, and further to claim that you can only send two emails to Apple without paying.
Far from being indifferent, the Apple developer relations team is hitting it way out of the park. Yes, outages happen, and that's unfortunate. But there's no need to make something out of it that it's not.