Thanks for a clear, sensible investigation. ISTM your auth complaint is well-taken. Either Lambda should be seeing the same role as the gateway automatically, or there ought to be a way to configure that.
However, I have to admit that my first reaction to that difficulty in getting the account ID from the path/query/etc would have been to just shove it in the request body JSON that was already coming through. Then it could probably be taken out of the path completely, which would be a win for url privacy at least. This doesn't seem like the showstopper that the auth issue does.
Last Sunday morning, I was so excited to move my APIs over to AWS’s API Gateway. That evening, I was ready to kill.
I spent the entire day (9+ hrs) getting merely two API routes working. A GET and a dreaded POST request.
I couldn’t tell from your post if you eventually figured it out, but it is possible to get the POST request body and url params in your Lambda event as JSON.
In my POST Method’s Integration Request, here’s how I get the query string parameter and the request body. I use this Input Mapping.
A “glass half full” way to think about all of this is that the AWS API Gateway is ridiculously strict about what data can be passed in to your Lambda functions, and you must explicitly state that data.
Here are other problems I ran into over the weekend:
- The web console is super buggy with broken links and really, really bad UX.
- The documentation is missing for many essential components.
- No support for API Gateway in the AWS SDKs
- I can't figure out how to copy resource methods, so I have to do tons of redundant manual input when creating each resource method, which includes explicitly stating every HTTP Error response code.
- Returning error objects from Lambda functions also needs mapping.
- Out-of-the-box slow response times. My Lambda function with two simple DynamoDB queries often takes 3 seconds to return a result when its route is hit.
Overall, I'm a huge AWS fanboy but the execution of this product is incomplete and out of touch.
C'mon AWS, you can pull through on this. Keep it simple. Build out from there. We all want you to succeed.
UPDATE: They just announced a nice fix for easily getting the request JSON Body into your Lambda functions and some documentation improvements. See jeffbar's post in this thread.
Thank you Jeff and the API Gateway team for the communication and prompt fixes.
I’ll update my original posts so that they do not misinform others and I look forward to future improvements.
One additional thing that would be incredibly useful is a simple blog post that describes steps/options one could take to increase API Gateway + Lambda response times. I'm fiddling with this now, but I wish I had some guidance as to where to focus my optimizations and how to make them, all in one place and straight from the source (AWS). For example, I'd love some examples on API Gateway caching, making my API accessible in multiple regions, how to make my Lambda functions faster, and why my first requests to my API Gateway + Lambda routes are so slow even after only a few minutes of inactivity. On a first request, my Lambda function will run for 1.2 seconds but the network request as a whole takes 2.8-3.2 seconds, yet every immediate subsequent request comes in much faster (~1.4 seconds)(Everything is in the same us-east region). If there is nothing I can do about this because the server my lambda function is on has been de-prioritized due to minutes of inactivity and is in an idle state, then the guide on optimizing elsewhere is even more necessary. This is the last piece of the story that is missing for me personally, and it causes me to hesitate before deploying more code. Also, if you have this written already and I've missed it, please let me know!
Overall, you've lowered the barriers to entry and increased efficiency so much I plan on deploying much more server side code than ever before.
Hi Jeff, thanks for responding to the feedback. Personally I’m just concerned about two things:
- Execution response time: the responses to API > Lambda > DynamoDB are too slow
- Reference Samples: we need docs and/or code samples with the most common patterns and workflows for mBaaS and IoT (i.e: user authentication with JWT and ACLs via the API > Lambda > Dynamo)
Many of us are stoked about this product. Hope you can iron out the issues that the community is raising and that are preventing wider adoption. The concept of server-less microservices via REST looks like the next big trend after containers. Hope you get it just right soon... I plan to make intensive use of this service in no time! =)
A good, useful, detailed blogpost that highlights a number of important blocker-level issues - and there's an unexpected, speedy response from the previously-anonymous devs announcing improvements as a result of the post and analysis itself.
I spend some time with API Gateway this weekend. Since I work at a company that has a product that helps customers with authentication and authorization for APIs, I am always curious where everyone else is skating.
Your section about authentication resounds with me loud and clear, I couldn't hack it out in 6 hours. This will be a pain point for anyone using API Gateway for anything other than a toy API.
Having the possibility to create an HTTP RESTful interface to many Lambda functions running on AWS is the biggest advantage of the AWS API Gateway.
Technically, stretching the system a bit, one could argue that we don't need backend software running on EC2 anymore, since everything can be a Lambda function with an associated HTTP endpoint to trigger it. I have been thinking about this a lot, and it could work if being locked into AWS is considered an acceptable risk.
For anything else more API-oriented (more authentication schemes, better security, more logging options, etc) I would suggest checking out Kong (https://github.com/Mashape/kong), which is pluggable with extra functionality and can work in front of the AWS API Gateway to enhance the REST-to-Lambda interface (and with any other API).
You won't be able to port it without any additional work, but if you are willing to do some refactoring (specifically removing the Lambda-specific objects, etc) then it can run anywhere. Of course the time this will take depends on the size of the project, but it can be definitely done without rewriting the whole application, but just rewriting the "invoker" part.
To make a future port easier, the functions could also be modularized by keeping the real business logic and the Lambda integration separate, but integrated.
My take after trying this out for a few hours mirrors yours: it looks like it was rushed out the door and not 100% ready. It's fun to play around with, but I wouldn't put anything production on it for at least another few months.
I agree - the docs are kinda hard to parse, and after launch it didn't work in us-west-2 for several hours. I've got great ideas for what I can build with it if they fix a few issues with it, but for now it's just a toy.
The other thing that is terrible is versioning. Lambda does not support versioning. And the resources/methods that you define for your various stages all share the same resources/methods, unless you want to change the signature of your calls. So if I want to change an existing Lambda function, then the minute I change it for my staging/test/qa environment, I have changed it for production as well. So, I can't roll back easily. I can script all this, but ouch. The alternative is to create a whole new API gateway, but that means that once I get my stuff tested, I essentially have to recreate my new environment in production, insteading of just clicking a button.
We have a client who is looking at sending a massive amount of data to a REST API. Looks like we are going back to Elastic Beanstalk, which is pretty good, it just has servers. But it works.
And, I'd say that using Dyanamo as a DB locks you much more securely into AWS than API Gateway and Lambda. Be pretty easy to take those Lambda functions and map them back into Express or Restify routes. Changing the database, arg.
I've started messing with API Gateway and Lambda there a couple errors.
If you create a lambda function and add a gateway inside the lambda function configuration screen, it will give you a bad API gateway URL. Going into the API gateway page and looking at your stages URL and trying that works.
This just proves that it was rushed out. I do like the idea though and want to use it, but I don't think it'll be ready until Q4.
> Out-of-the-box slow response times. My Lambda function with two simple DynamoDB queries often takes 3 seconds to return a result when its route is hit.
This is something I've also noticed. Even their official examples are really slow [0]
About auth: I've been researching API Gateway + Lambda and auth via Amazon Cognito. As of now the best example for auth I've found is LambdAuth [1]. I'm trying to add the API Gateway in the mix, it looks possible to integrate it.
I also agree that docs and more explicit examples are needed.
Apparently, there's no way to use API Gateway with Cognito, it looks (and should be possible) there are so many lacks in the documentation... I'm following this forum thread [0] regarding Cognito integration in JS.
[0] https://forums.aws.amazon.com/message.jspa?messageID=645618#...
Just a quick nitpick: the second link in the article (the one to AWS Lambda) should have an all-lowercase URL; the URL's case-sensitive and - in its current capitalized form - is resulting in a 404.
I've experienced the same frustrations with the API Gateway. Lack of knowing the IAM principal that authenticated, and no clear way to differentiate if a call came from the API gateway or not makes me feel that this needs some work. SNS, for example, will sign it's HTTP requests and that would be a good start IMHO. Ability to use security groups in proxy mode would make this even better.
My major use case is a read only, throttled, and cached front end for the mediawiki api at http://snpedia.com . The ability to monetize the api for higher throughput is an appealing plus. Gateway still seems pretty good for this. I'd welcome input or thoughts from others.
Isn't lambda suicide for a startup? Think of all the lock-in here. If you go big you could suddenly find yourself really stuck if you're overly dependent on it and the overhead of lambda/gateway is too much.
It's actually not bad at all - I can't speak for the java side of things, but transitioning a couple random nodejs services to lambda took under ten minutes each, with most of that time being testing (my only complaint is that the logging sucks). Lambda function are so modular that I can't imagine not being able to trivially slot them back in to a standard express endpoint. It might be tricky if you do a bunch of stuff with S3, but even then it shouldn't be terribly difficult to sub in the standard AWS modules.
If it enables a start up to grow without having to worry about scale, I don't think so. You could say the same of a lot of the technologies in a cloud provider. At the end of the day, if you want to take advantage of the speed to market and cost reductions these technologies enable, you're going to have to take a punt.
A startup is more likely to die for another reason first, such as a founder-breakup, or some form of "running out of money".
Google App Engine and Heroku and Azure also have lock-in too, to varying degrees -- but at the start it still makes sense to use these sorts of services rather than worry about lock-in. (And if someone later gives you $100k of compute credits on their platform, and it costs you less than that to rewrite the "lock-in" parts, then you still win.)
However, I have to admit that my first reaction to that difficulty in getting the account ID from the path/query/etc would have been to just shove it in the request body JSON that was already coming through. Then it could probably be taken out of the path completely, which would be a win for url privacy at least. This doesn't seem like the showstopper that the auth issue does.