client: A Leading UK Medical Staff and Service Provider
VERTICAL: Healthcare
PRACTICE: Robotics
Scraper Design & Build

The project required the design and build of a scraper to 'scrape’ incoming structured and unstructured assignment data from a variety of sources (portals and emails).

The scraper is to be configurable to accommodate the varying data sources and formats and integrates with an established ‘parser’ to load the scraped data directly into DB (in this case, id-M's CRM Eclipse4) in order to automatically create an unassigned temporary contract for Nurses and Doctors.

Maantic's role:

  • Creating a configurable SMTP mail client
  • Creating a configurable rule engine for scraping the email header/body/footer
  • Providing an audit for the process by instrumenting all inputs and outputs to key components.
  • Using multi-threading to process read emails in parallel (to a configured max limit) that helps for faster response and concurrent operations.
  • Designing and implementing security model
  • Building out reusable, standardized Enterprise layer class structure for data and integration
  • Analyzing performance issues with JVM to fine tune GC