Product

Job Opportunity Monitor

A multi-source opportunity intelligence product that detects new vacancies, preserves complete listing records, evaluates them with a local model against a pinned policy revision and delivers approved opportunities through a recoverable Telegram pipeline.

Role
Product & Full-stack Engineering
Period
2024
Categories
FrontendBackendInfrastructure
Stack
TypeScriptNode.jsFastifyReactVitePrismaSQLiteOllamaDocker Compose
Sanitized Job Opportunity Monitor operations dashboard with verified aggregate metrics, source health and demonstration queue data.

Key metrics

Unique vacancies monitored
≈2.47M/year
Cross-source vacancy identities after deduplication in the private production aggregate.
Observed over the rolling 12-month window; not listing-card evaluations.
Vacancies filtered automatically
91.37%
Share automatically rejected in the private deduplicated production aggregate.
Measured against the same deduplicated 12-month aggregate.
Manual job-board checks avoided
2,000+/year
Checks no longer performed by hand across the documented monitoring workflow.
Measured against the documented pre-automation workflow; 2,000 is a conservative public floor.
Monitoring time saved
≈165 h/year
Annual time avoided across the documented manual monitoring workflow.
Derived from 2,000 conservative avoided checks and a measured average of about 5 minutes per check.

Overview

Job Opportunity Monitor continuously follows Duunitori and Jobly through separate source adapters and turns their changing search results into one normalized vacancy workspace. The product stores complete descriptions locally, supports source-specific settings and gives operators one place to inspect jobs, filters, scan history and notification state.

A private production aggregate deduplicates vacancy identities across sources before reporting annual monitoring volume; listing-card refresh activity remains a separate operational measure.

Challenge

New opportunities need to be discovered quickly, but repeatedly crawling every result page would waste requests and still leave gaps when a source changes. The first scan also has to establish a baseline without flooding a destination with old vacancies.

A local-model decision must remain tied to the exact policy revision used for that evaluation. The complete vacancy record also needs to remain inspectable whether the outcome is blocked or notified.

The local-model stage or Telegram delivery can fail temporarily. Those failures must remain visible and retryable instead of dropping an opportunity or silently skipping delivery.

Solution

I introduced a source-adapter boundary for Duunitori and Jobly. A lightweight watcher fingerprints the newest results on a short interval, triggers a full source scan when the fingerprint changes and is backed by scheduled reconciliation so deeper changes are still recovered.

The scanner performs a silent initial seed, fetches full vacancy details and saves the normalized record before evaluation. A delivery candidate pins the active policy revision and creates a persistent AI queue item before the local model records a blocked or notified outcome; notified items continue to Telegram.

The persistent AI queue retains processing and retry state in SQLite. Recoverable errors receive scheduled retries and can also be retried manually from the React/Vite admin, whose current workspaces are Dashboard, Jobs, AI Queue, AI Rules and Settings.

Responsibilities

  • Define the product model, monitoring workflow and operational safety rules.
  • Design independent Duunitori and Jobly adapters behind a shared source contract.
  • Implement change-aware polling, silent initial seeding and scheduled reconciliation.
  • Persist full vacancy records, pin the active policy revision and create persistent AI queue items before local-model evaluation.
  • Build the durable SQLite AI queue with retries and manual recovery.
  • Create the Fastify API, React/Vite operations admin and Docker Compose runtime.

Architecture

Interface

  • Configured destinations

  • React/Vite operations admin

Services

  • Fetch + normalize scanner

  • Fastify operations API

Data

  • Prisma + SQLite job store

  • Pinned policy revision

  • Persistent AI queue

Automation

  • Duunitori + Jobly adapters

  • Watcher + reconciliation

  • Local model decision

  • Telegram delivery adapter

Infrastructure

  • Docker Compose worker runtime

  1. Source-specific result snapshots

    Duunitori + Jobly adapters -> Watcher + reconciliation

  2. Change signal or scheduled reconciliation

    Watcher + reconciliation -> Fetch + normalize scanner

  3. Full normalized vacancy records

    Fetch + normalize scanner -> Prisma + SQLite job store

  4. Stored vacancies pin the active policy revision

    Prisma + SQLite job store -> Pinned policy revision

  5. Pinned policy context creates a persistent AI queue item

    Pinned policy revision -> Persistent AI queue

  6. Queued review enters the local model

    Persistent AI queue -> Local model decision

  7. Approved decisions enter Telegram delivery

    Local model decision -> Telegram delivery adapter

  8. Delivered to configured destinations

    Telegram delivery adapter -> Configured destinations

  9. Jobs, sources and scan history

    Prisma + SQLite job store -> Fastify operations API

  10. Queue state and manual retry controls

    Persistent AI queue -> Fastify operations API

  11. Typed operations requests

    Fastify operations API -> React/Vite operations admin

  12. Scanner production runtime

    Docker Compose worker runtime -> Fetch + normalize scanner

  13. API production runtime

    Docker Compose worker runtime -> Fastify operations API

  14. Admin production runtime

    Docker Compose worker runtime -> React/Vite operations admin

Technical decisions

Detect change before running a full scan

Problem
Full multi-page scans are too expensive to run every few seconds, but a long fixed interval delays genuinely new opportunities.
Decision
Fingerprint the newest result cards frequently, launch a full source scan when the fingerprint changes and keep a slower scheduled reconciliation as a safety net.
Reason
The fast path stays lightweight while periodic reconciliation catches deeper source changes and missed transitions.
Trade-off
The watcher is an optimization, not proof that the source is unchanged, so the reconciliation scan remains necessary.

Save every vacancy before evaluation

Problem
Discarding a vacancy during ingestion would remove the record needed to inspect a later blocked or notified outcome.
Decision
Normalize and persist the full vacancy record before creating its AI queue item.
Reason
The searchable job corpus remains independent from the evaluation policy and every outcome stays inspectable.
Trade-off
The local database stores more records than the active delivery set and needs deliberate retention and indexing.

Pin policy context before local-model evaluation

Problem
If a queued evaluation reads only the mutable current policy, a retry could lose the policy context used when the work was created.
Decision
Pin the active policy revision when creating the persistent AI queue item, before the local model evaluates the vacancy.
Reason
Each blocked or notified outcome remains tied to the policy context used for that decision.
Trade-off
Policy revision history becomes retained domain context that the queue and admin must preserve.

Persist AI queue state and expose recovery

Problem
A temporary local-model or Telegram failure must not turn into a lost evaluation or notification.
Decision
Store the queue item and pinned policy context in SQLite, retain retry state and expose manual recovery in the admin.
Reason
Processing survives restarts and operators can inspect blocked, notified or retryable work.
Trade-off
Queue lifecycle and retry policy add operational state that must be monitored and maintained.

Media

Gallery

Screenshot

01 / 06

Sanitized Job Opportunity Monitor operations dashboard with verified aggregate metrics, source health and queue status.

The real operations dashboard uses verified aggregate metrics and neutral demonstration rows. Vacancy identities, destinations, credentials and internal endpoints are not shown.

Outcomes

  • Two vacancy sources now feed one searchable record set through independent adapters and a shared normalization pipeline.
  • Fast change detection and scheduled reconciliation work together without sending the historical baseline as new opportunities.
  • Dashboard, Jobs, AI Queue, AI Rules and Settings expose vacancy records, policy revisions, queue outcomes, recovery controls and source configuration in one restricted operations interface.

Verified highlights

  • Duunitori and Jobly are implemented as separate source adapters with their own scan settings.
  • The system stores full vacancy details before evaluation and establishes the first source baseline silently.
  • Persistent AI queue items pin the active policy revision before local-model evaluation and retain retry state for manual recovery in the operations admin.
  • Telegram is the currently implemented delivery adapter; no additional delivery channels are claimed.

Learnings

  • A frequent lightweight fingerprint plus slower full reconciliation is more resilient than treating either rapid polling or a scheduled crawl as the only source of truth.
  • Separating vacancy records, pinned policy revisions and persistent queue items keeps each local-model outcome inspectable without making mutable policy part of the collected source data.