Skip to Content
Infrastructure

Infrastructure

This section covers the infrastructure components of the Esteem Thrive API, including background job processing, queues, and scheduled tasks.

Background Jobs

The Esteem Thrive API uses Sidekiq for background job processing with Redis as the backing store. Jobs are organized into specialized queues for different types of operations.

Queue Configuration

The system uses 9 specialized queues configured in config/sidekiq.yml:

  • processors - File processing operations (claims, EDI)
  • eligibility - Insurance eligibility verification
  • notifications - Email and SMS notifications
  • uploads - File and video uploads
  • intakes - Intake questionnaire processing
  • states - Appointment state tracking
  • mailers - Email delivery
  • default - General background tasks
  • low - Low priority tasks

Sidekiq runs with 10 concurrent workers in all environments.

Job Classes

Clinic::Patients::EligibilityReportCheckJob

  • Queue: eligibility
  • Purpose: Verifies patient insurance eligibility using PVerify API
  • Scheduling: Triggered via rake task patients:check_eligibility_report
  • Frequency: On-demand for active eligibility information records
  • Key Features:
    • Prevents duplicate reports within the same month
    • Creates eligibility reports and appointment comments
    • Updates patient insurance status based on verification results
    • Integrates with PVerify Easy Eligibility Summary API

Clinic::Patients::AppointmentNotificationJob

  • Queue: notifications
  • Purpose: Sends appointment reminder notifications to patients
  • Scheduling: Scheduled via rake task notifications:coming_up_appointments
  • Frequency: Twice daily - morning (8:00 AM) and 2 hours before appointment
  • Key Features:
    • Sends both email and SMS notifications
    • Uses Pacific Time zone for scheduling
    • Prevents duplicate notifications with notification_sent flag

Clinic::Appointments::TrackStatesJob

  • Queue: states
  • Purpose: Tracks and updates appointment states throughout the appointment lifecycle
  • Scheduling: Scheduled via rake task appointments:update_state
  • Frequency: At appointment time and 60 minutes after (for missed appointments)
  • Key Features:
    • Monitors appointment state transitions
    • Handles missed appointment detection
    • Uses Pacific Time zone for scheduling

Clinic::Appointments::RescheduleNotificationJob

  • Queue: notifications
  • Purpose: Notifies patients when appointments are rescheduled
  • Scheduling: Triggered when appointments are rescheduled
  • Key Features:
    • Sends both SMS and email notifications
    • Integrates with appointment rescheduling workflow

Clinic::Appointments::DumpIntakeQuestionnairesJob

  • Queue: intakes
  • Purpose: Processes and stores intake questionnaire data for appointments
  • Scheduling: Triggered during appointment workflow
  • Key Features:
    • Prevents duplicate processing with existence check
    • Extracts and stores questionnaire responses
    • Creates IntakeQuestionnaireDatum records

Clinic::Claims::ProcessorJob

  • Queue: processors
  • Purpose: Processes insurance claims files from SFTP
  • Scheduling: Triggered via rake task claims:parser
  • Key Features:
    • Integrates with OfficeAlly EDI system
    • Processes claims files for submission

Clinic::Claims::ListFilesJob

  • Queue: processors
  • Purpose: Lists and processes files from SFTP for claims processing
  • Scheduling: Triggered via rake task claims:processor
  • Key Features:
    • SFTP file discovery and processing
    • Works with OfficeAlly integration

Clinic::Videos::UploadJob

  • Queue: uploads
  • Purpose: Processes video file uploads and attachments
  • Scheduling: Triggered when videos are uploaded
  • Key Features:
    • Handles video file processing and attachment
    • Updates video status (completed/failed)
    • Comprehensive error handling and logging
    • Sets processed_at timestamp on completion

UploadEdiJob

  • Queue: default
  • Purpose: Generates and uploads EDI 837 claims files to SFTP
  • Scheduling: Triggered when claims need to be submitted
  • Key Features:
    • Validates claims using Admin::Clinic::Edi837Validator
    • Generates separate files for medical and non-medical claims
    • Uses Stupidedi library for EDI generation
    • Uploads to OfficeAlly SFTP with timestamped filenames
    • Creates document records for audit trail
    • Updates claim status to “submitted”

RemoveUserJob

  • Queue: default
  • Purpose: Safely removes users from the system
  • Scheduling: Triggered when users need to be deleted
  • Key Features:
    • Handles foreign key constraints during deletion
    • Comprehensive error handling and logging
    • Uses DestroyUser service for safe deletion

ReportJob

  • Queue: default
  • Purpose: Generates various types of reports for patients
  • Scheduling: Triggered when reports are requested
  • Key Features:
    • Dynamic report generation using class name resolution
    • Supports multiple report types through polymorphic design

Scheduled Tasks (Cron Jobs)

The system uses the whenever gem for cron job scheduling, configured in config/schedule.rb:

Bed Time Buddy Notifications

  • Schedule: Every 1 minute
  • Task: bed_time_buddy:daily_bed_time_buddy_notifications
  • Purpose: Sends daily bedtime buddy notifications

Rake Tasks

notifications:coming_up_appointments

  • Purpose: Schedules appointment notification jobs for the current day
  • Frequency: Should be run daily
  • Behavior:
    • Finds appointments for the current day (Pacific Time)
    • Schedules morning notifications for 8:00 AM
    • Schedules reminder notifications 2 hours before appointment
    • Prevents duplicate scheduling with notification_sent flag

appointments:update_state

  • Purpose: Schedules appointment state tracking jobs
  • Frequency: Should be run daily
  • Behavior:
    • Finds appointments for the current day (Pacific Time)
    • Schedules state tracking at appointment time
    • Schedules missed appointment detection 60 minutes after appointment
    • Prevents duplicate scheduling with schedule_refresh_state_job flag

patients:check_eligibility_report

  • Purpose: Triggers eligibility verification for all active patients
  • Frequency: On-demand or scheduled as needed
  • Behavior:
    • Processes all active Clinic::EligibilityInformation records
    • Queues eligibility check jobs for each patient

claims:processor

  • Purpose: Triggers SFTP file listing and processing
  • Frequency: On-demand or scheduled as needed

claims:parser

  • Purpose: Triggers claims file processing
  • Frequency: On-demand or scheduled as needed

Infrastructure Notes

  • Time Zones: Most scheduling tasks use Pacific Time (America/Los_Angeles) for business logic but convert to UTC for job scheduling
  • Error Handling: Jobs include comprehensive error handling with detailed logging
  • Duplicate Prevention: Many jobs include mechanisms to prevent duplicate processing
  • External Integrations:
    • PVerify API for insurance eligibility
    • OfficeAlly SFTP for EDI claims processing
    • Twilio for SMS notifications
    • Email delivery through configured mailers
  • Monitoring: Sidekiq web interface available at /sidekiq with basic authentication
  • Queue Priority: Queues are processed in the order listed in config/sidekiq.yml

Development and Deployment

  • Local Development: Use bundle exec sidekiq to start the worker process
  • Production: Configured via Procfile with bundle exec sidekiq -C config/sidekiq.yml
  • Monitoring: Sidekiq web UI protected with environment variable authentication (SIDEKIQ_USERNAME, SIDEKIQ_PASSWORD)
CollaboratorsWebhooks