Meeting with CloudFlare this week...
Added 2017-05-03 19:23:12 +0000 UTCI'm meeting with CloudFlare on Friday to discuss our options going forward using their services. Unfortunately the API is 100% dependent on their caching, even with the new server.
We are going to analyze the traffic with some simulated rate limiting rules applied and calculate the costs, they already warned me that the traffic I'm currently getting would result in a "high bill", and they want to discuss a discounted flat rate option. However, I already know this discounted flat rate option will still be in $1k-3k+ price range, which this Patreon is nowhere near covering those kind of costs.
I only see two viable options moving forward, and that's what this poll is about.
Option 1: Require registration/API Keys and enable rate limiting
Option 2: Go private with no rate limiting for ALL patrons
Comments
Hi Brian, I'd love to test. If you send me the URL I'll give it a run every day.
Johan Ramael
2017-08-10 18:49:30 +0000 UTCThat's an ongoing issue I'm currently investigating with two other users, it has NOTHING to do with my server or the rate limiting, in fact when the users bypass Cloudflare and talk directly to my server they are having zero timeouts etc. If you would like to test this also I would appreciate the extra data point as I'm currently gathering all of the information I can from users experiencing this before contacting Cloudflare. Thanks
Brian Fritz
2017-08-10 17:32:11 +0000 UTCEver since the API went paying, I cannot gat any batch processing done anymore and getting worse every day. After a few dozen requests I'm not getting any response anymore on my HTTP requests and I need to wait and continue. Sorry, but moving from free to paying, OK, but giving worse service, no can do. I this rate limiting / dropping requsts continues, I'm out.
Johan Ramael
2017-08-10 16:52:41 +0000 UTCSorry for the inconvenience, I basically had to disincentivize people from using it to quickly reduce traffic by 90%. Once the traffic gets under control, and I finish the rate limiting code I'm going to create a nice registration process to give free keys with limits.
Brian Fritz
2017-05-26 04:31:59 +0000 UTCYou should be rate limiting even with Patreon. It's just bad practice to have an API wide open, on both your end and the users end. I use this API on a website for a major movie theater on a VERY busy website and hit the API less than 50 API calls a month. Oh well, guess I'm paying now.
Arthur Guy
2017-05-26 03:41:20 +0000 UTCJust signed up for paid plan still it shows wrong api
sagar baranwal
2017-05-21 11:49:09 +0000 UTCAs the author of the metadata module that is used in Kodi to show the OMDB info I think a hybrid option would be great. Users want to have all bells and whistles, why not pay a few bucks if they want it updated. FYI: I have implemented a 2 weeks cache in my module a while ago. My suggestion would be that users get like 5 requests free a day (or something like that) and if they want the most updated information without any limits, they must be a patreon.
Marcel van der Veldt
2017-05-08 19:58:55 +0000 UTCInteresting ! Going to have to look into the cached response. I don't have even close to enough people using my current API that would require it yet, but I am working on V2 and I think that one will be much more popular.
Box Office Buz
2017-05-05 14:50:57 +0000 UTCWell, let Redis cache the responses of course. Redis can easily handle 10-20K requests per second on my laptop. And perhaps put Varnish in front. Varnish has modules that handle rate limiting via API keys & IP's. What I'm saying is that instead of just throwing bigger servers at it, I suggest looking at a different approach. (And I also think Rate limiting using should be avoided when possible. It requires people to update their scripting or roll out new versions of their Apps as well.)
Marco
2017-05-04 17:57:38 +0000 UTCWhat would the redis be used for ? Curious as I am building an API for my own website. Right now I just use redis to cache video view counts.
Box Office Buz
2017-05-04 13:26:17 +0000 UTCSounds really great!
Douglas Alves
2017-05-04 13:03:10 +0000 UTCGood suggestion!
Josar
2017-05-03 22:40:29 +0000 UTCI think that Option 1 is a good start, and see how the traffic gets from there. I know there are Kodi addons that use your API, and I can imagine the abuse and amounts of hits that is producing. Then if that still doesn't help, Option 2 could be implemented.
Josar
2017-05-03 22:36:29 +0000 UTCFor your kind of data (and especially when not using API keys), why not put Redis in front? It's easy to setup. And when properly done, a single instance can handle much more than you need right now.
Marco
2017-05-03 21:29:50 +0000 UTCFingers crossed for sure.
Box Office Buz
2017-05-03 20:33:22 +0000 UTCI'm looking into it now, I don't know if they play nice with CloudFlare and AWS is NOT an option (Amazon's lawyers don't like me very much).
Brian Fritz
2017-05-03 20:32:12 +0000 UTC@Matt That's what I'm hoping for also. However, I think Option 1 only buys me a limited amount of time with a lot of unknown costs due to the rate limiting.
Brian Fritz
2017-05-03 20:13:03 +0000 UTCHybrid sounds good for me too!
Peter Xander
2017-05-03 19:46:36 +0000 UTCThis might have the added benefit of encouraging more donations
Matt Sidor
2017-05-03 19:45:32 +0000 UTCOn a slight side note. Have you thought about something like "mashape" ? You would DNS threw them, they attach a header to ever request, and they qould actually take care of the amount of traffic one user can utilize. Than say (over 10,000 requets a day) you can charge people, and mashape actually takes care of that too.
Box Office Buz
2017-05-03 19:40:39 +0000 UTCThe only thing I could think of is combining layer 7 load balancing with cloudflare.
Box Office Buz
2017-05-03 19:36:24 +0000 UTCThat sounds good Matt
Hans Ecke
2017-05-03 19:36:09 +0000 UTCWhat about some hybrid of both? API keys with X number of free requests per day; after that you will need to donate to unlock unlimited requests.
Matt Sidor
2017-05-03 19:34:15 +0000 UTCThe traffic is out of control! Weekdays it's getting over 1.5 billion requests (24h) with peaks of 100 million requests per hour. I did some tests bypassing the CDN and the servers got crushed instantly. It's truly a testament to how powerful CloudFlares services actually are.
Brian Fritz
2017-05-03 19:32:13 +0000 UTCI wish there was a way I could help. I would say I could help with setting up load balancers for the API, but if you are usizing cloudfalre I am not 100% on what that would involve.
Box Office Buz
2017-05-03 19:26:19 +0000 UTC
