Even though I got over my initial desire to constantly twiddle the knobs and have moved on to creating content daily, I have been tweaking things here and there and learning as I do so. I am by no means a guru/wizard/ninja/whatever people want to call themselves when it comes to Web development, but as my dad says, I “know enough to be dangerous.” It’s impossible to know everything about anything, so deciding what to learn about has been tough, especially when you don’t know what you don’t know. The best way that I’ve found to discover my next topic to learn about is to run into a problem.

I’ve seen Twitter widgets on a multitude of websites and even some plugins written, but none of them quite met my needs. With my basic programming knowledge I decided to take a crack at implementing a custom Twitter sidebar. Starting with a script that was written by Twitter it was pretty easy to get a working list of my most recent tweets displaying. That had me feeling pretty good about myself. I hadn’t accomplished anything Earth shattering, but even learning a small skill is still an addition to my toolbox.

Then one day it stopped working.

That prompted me to look at the code I’d written to see what was causing the conflict. After all, Twitter was working and everything seemed OK on their end. After more research I determined that the method I’d used was based on a deprecated methodology and since it was no longer officially supported, that explained the flaky behavior. This caused me to learn about the new way that Twitter was handling requests for tweets and after a little while I was using the proper API calls and life was good.

Then one day it broke, again.

This was really frustrating. I was using the latest techniques, the fail whale was nowhere to be seen, so why wasn’t it working? I then learned that the API had a limit of 150 requests per hour and if you exceed that, no updates will be sent until the limit is reset. Twitter recommends caching your tweets so that on busy websites you don’t go over your rate limit. That made sense to me, since viewing tweets in the sidebar was meant more as a way to get a general sense of what I’ve been talking about rather than following a conversation minute by minute, after all, that’s what Twitter itself is for. OK, great, implement caching, how do I do that?

This was a whole new challenge for me, mainly because there wasn’t a lot of precedent I could find for how to go about solving the problem. It took a lot of trial and error, hours of testing, but by the time I was done I’d set up a process where the Twitter API was only being polled at most every ten minutes. That cut my requests down to six per hour, tops. There’s no way I would ever go over the limit of 150 requests per hour, right?

If you’ve read this far, you know that something else went wrong.

It turns out that requests made to the Twitter API are logged by IP address. Since every device that connects to the Internet has an IP address, it’s a good way to determine exactly who is making requests, and if they exceed their quota, to cut them off. I found out how I could determine my remaining number of requests and started to do some detective work. At first I thought my caching method was failing, that maybe instead of reading the local copy, Twitter was still being queried every time the page loaded. I tested locally and sure enough, everything looked good. The cache would update, then requests for my tweets would read from that cache for ten minutes and my API requests were not being used. After ten minutes an API request would be used and the cache would be updated. Then I moved on to my live website.

Using my basic script to see how many API requests I had remaining I took a baseline and then loaded my website, once. The remaining API requests went down by ten. Ten? How is that possible? I’ve enjoyed a modicum of success with my humble blog, but there’s no way that ten people all loaded my page within 5 seconds. Then I went back to just checking my remaining API calls, without reloading the web page. It kept dropping. Sometimes rapidly, other times by one or two calls. Then it hit me. I was on a shared web server.

For a website with moderate traffic, a shared web server is a good way to publish content on a budget. The downside is that you’re sharing resources with other websites. In most ways this means that you’ve got to share memory, processor cycles and disk access. It also means you’re sharing something else, an IP address. Do you see where I’m going? No? Well, the conclusion that I came to was that even though I’d implemented caching and ensured that my website was only talking to Twitter up to six times per hour, someone else (or multiple someone elses) were not. This wasn’t an issue for me unless someone went to my website and my cache file was more than ten minutes old and my server’s IP address had used its API quota. When my site tried to update the local cache Twitter would respond with “Sorry Charlie, you’ve used up your limit this hour, you’ll have to wait.”

This was frustrating because after all, it wasn’t me who was abusing the request limit to Twitter, it was someone else on my server. (A few of you may have thought, “Hey, you were doing the same thing when you first started, so don’t give them such a hard time. Maybe they haven’t figured out caching yet.” and you’d be right, but it doesn’t help correct my problem.) To work around this meant another iteration to my Twitter script, now taking my available API requests into consideration. Hopefully this will keep things working smoothly.

For those of you wondering, here’s the logic behind my Twitter script:

• Is there a cached copy of tweets?
  • Yes
    • Is it more than ten minutes old?
      • Yes
        • Run the function to retrieve new tweets
      • No
        • Use the cached copy
  • No
    • Run the function to retrieve new tweets
• Display the cached copy of tweets

The logic behind the function to retrieve new tweets:

• Can you connect to the Twitter API?
  • Yes
    • Are there any API requests left?
      • Yes
        • Request the most recent tweets and update the cached copy
      • No
        • Don’t attempt to update the cached copy
  • No
    • Don’t attempt to update the cached copy