What’s happening is that your Application and/or ApplicationSpawners are shutting down due to time-out. To process your new request, Passenger has to startup a new copy of your application, which can take several seconds, even on a fast machine. To fix the issue, there are a few Apache configuration options you can use to keep your Application alive.
Here’s specifically what I’ve done on my servers. The RailsSpawnMethod and RailsAppSpawnerIdleTime are the configuration options most important in your situation.

# Speeds up spawn time tremendously — if your app is compatible.
# RMagick seems to be incompatible with smart spawning
# Older versions of Passenger called this RailsSpawnMethod
PassengerSpawnMethod smart

# Keep the application instances alive longer. Default is 300 (seconds)
PassengerPoolIdleTime 1000

# Keep the spawners alive, which speeds up spawning a new Application
# listener after a period of inactivity at the expense of memory.
RailsAppSpawnerIdleTime 0

# Just in case you’re leaking memory, restart a listener
# after processing 5000 requests
PassengerMaxRequests 5000

 

By using “smart” spawning mode and turning off RailsAppSpawnerIdleTime, Passenger will keep 1 copy of your application in memory at all times (after the first request after starting Apache). IndividualApplication listeners will be forked from this copy, which is a super-cheap operation. It happens so quickly you can’t tell whether or not your application has had to spawn a listener.
If your app is incompatible with smart spawning, I’d recommend keeping a large PassengerPoolIdleTime and hitting your site periodically using curl and a cronjob or monit or something to ensure the listener stays alive.
The Passenger User Guide is an awesome reference for these and more configuration options.
Finally, but unrelated to your question, consider using Ruby Enterprise Edition if you go with Smart spawning. It can save you a lot of memory.
edit: If your app is incompatible with smart spawning, there are some new options that are very nice

 
# Automatically hit your site when apache starts, so that you don’t have to wait
# for the first request for passenger to “spin up” your application. This even
# helps when you have smart spawning enabled.
PassengerPreStart http://myexample.com/
PassengerPreStart http://myexample2.com:3500/

# the minimum number of application instances that must be kept around whenever
# the application is first accessed or after passenger cleans up idle instances
# With this option, 3 application instances will ALWAYS be available after the
# first request, even after passenger cleans up idle ones 

PassengerMinInstances 3

 
So, if you combine PassengerPreStart and PassengerMinInstances, Passenger will spin up 3 instances immediately after apache loads, and will always keep at least 3 instances up, so your users will rarely (if ever) see a delay.
Or, if you’re using smart spawning (recommended) with RailsAppSpawnerIdleTime 0 already, you can add PassengerPreStart to get the additional benefit of immediate startup.
Many thanks to the heroes at phusion.nl!

# Additionally keep a copy of the Rails framework in memory. If you’re
# using multiple apps on the same version of Rails, this will speed up
# the creation of new RailsAppSpawners. This isn’t necessary if you’re
# only running one or 2 applications, or if your applications use
# different versions of Rails.
RailsFrameworkSpawnerIdleTime 0
Just something to add and might be useful.

 
The default spawn method in the current release is “smart-lv2”, which skips the framework spawner, so setting the framework spawner timeout wouldn’t have effect anyway unless you explicitly set the spawn method to “smart”.

check the version of passenger. it was RailsSpawnMethod for old versions.

If so (if I remember correctly), replace Passenger with Rails in all the configuration directives or look for old passenger docs for more details