How to Build Persistent Background Jobs in FastAPI with APScheduler and PostgreSQL

Ever spent hours watching a script run, only to have it stop because your laptop went to sleep? I have. That moment, staring at a frozen terminal, taught me a clear lesson: background jobs are the unsung heroes of reliable software, and doing them well requires the right tools.

I’m not talking about simple scripts you run once. I mean the tasks that need to happen on a schedule, without fail, whether your app is busy or idle. Sending nightly reports, cleaning up old data, or syncing information with another service—these jobs are the quiet pulse of a production system. But what happens when you restart your app? Or when a job fails? Python’s time.sleep() or basic cron jobs often leave these questions unanswered.

That’s what led me to build a better system. Let’s talk about how to schedule jobs properly, ensuring they persist, survive restarts, and can be controlled on the fly.

First, we need a scheduler. For modern Python applications, especially those using async frameworks like FastAPI, APScheduler is a powerful choice. Its design is logical. Think of it as having four main parts: the brain (the scheduler), the memory (the job store), the muscle (the executor), and the clock (the trigger). For our async world, we use the AsyncIOScheduler, which plays nicely with the existing event loop.

Why is a separate job store so important? By default, schedules live in memory. If your application stops, your schedules vanish. This is where persistence comes in. We can use a PostgreSQL database as our job store. This means every job you define is saved. When you restart your application, APScheduler can look at the database and pick up right where it left off. No more lost schedules.

Setting this up starts with a solid foundation. We need our database connection and to tell APScheduler to use it. Here’s a basic look at how you might configure the scheduler to use SQLAlchemy with PostgreSQL.

from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from sqlalchemy.ext.asyncio import create_async_engine

# Define your database connection
engine = create_async_engine('postgresql+asyncpg://user:pass@localhost/dbname')

# Configure job stores to use PostgreSQL
jobstores = {
    'default': SQLAlchemyJobStore(engine=engine.sync_engine)  # Note: uses sync engine
}

# Create the scheduler
scheduler = AsyncIOScheduler(jobstores=jobstores)

Notice we use the synchronous engine for the job store. APScheduler’s SQLAlchemy integration currently works with synchronous sessions. Your actual job functions, however, can be fully async.

This brings us to a key point: defining the jobs themselves. What does a good background job look like? It should be a self-contained function that does one thing well. It should also handle its own errors gracefully. You don’t want a failing email job to stop your entire report generation pipeline. Here’s an example of a simple notification job.

import logging
logger = logging.getLogger(__name__)

async def send_daily_digest():
    """
    A job function to send a daily summary email.
    It catches and logs its own errors.
    """
    try:
        # ... your logic to compile data and send an email ...
        logger.info("Daily digest sent successfully.")
    except ConnectionError as e:
        logger.error(f"Failed to send digest: Network issue - {e}")
        # Perhaps re-raise the error or implement a retry logic here
    except Exception as e:
        logger.exception(f"An unexpected error occurred: {e}")

Now, how do we get this job onto the schedule? We can add it to our scheduler with a trigger. A cron-style trigger is familiar and powerful. Want it to run every weekday at 9 AM? That’s easy.

# Add the job to run on weekdays at 9 AM
scheduler.add_job(
    send_daily_digest,
    'cron',
    hour=9,
    minute=0,
    day_of_week='mon-fri',
    id='daily_digest',  # A unique ID so we can manage it later
    replace_existing=True  # Useful for updates
)

But what good is a schedule if you can’t see what’s happening? Or change it without stopping the server? This is where integrating with a web framework like FastAPI shines. We can create a control panel via API endpoints.

Imagine starting the scheduler when your FastAPI app starts and letting it run in the background. Then, you can add endpoints to list all jobs, pause a specific one, or even add a new job on the fly based on a user action. This dynamic control moves us from a static configuration file to a living, manageable system.

Have you considered what happens if a job is supposed to run while your app is down? This is a ‘misfire’. APScheduler can handle this. You can set a ‘misfire grace time’—a window during which a missed job can still be run after the app comes back online. It’s a small setting with a big impact on reliability.

Of course, running jobs is one thing. Understanding their history is another. While APScheduler manages the schedule, you might want to track the execution—when it started, if it succeeded, how long it took. For this, you can create a simple log table in your database. Your job function can write a record at the start and update it upon completion or failure. This log becomes invaluable for debugging and monitoring.

So, we have a persistent, reliable scheduler. But is it scalable? The AsyncIOExecutor can run many async functions efficiently within the event loop. For CPU-intensive tasks, you might offload them to a separate process pool. The key is to match the executor to the job’s nature. An async executor is perfect for I/O-bound tasks like calling APIs or writing to databases.

What about when things go wrong? A job might fail due to a temporary network glitch. A good pattern is to build retry logic directly into the job function. You could also use APScheduler’s built-in retry features, or for complex workflows, consider logging the failure and alerting a separate system to investigate.

Building this changed how I think about application design. It’s no longer about scripts that run sometimes. It’s about creating a dependable, observable backbone for all the work that happens behind the scenes. The jobs keep running, the data stays fresh, and you can sleep knowing the system has a memory of its own.

What’s the first scheduled task you would make truly robust? I’d love to hear about your use cases. If you found this walkthrough helpful, please share it with others who might be building the backbone of their applications. Feel free to leave a comment below with your thoughts or questions.

---

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

---

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

---

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

---

Our Creations

Be sure to check out our creations:

Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools

---

We are on Medium

Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva