Senior Web Crawler & Data Extraction Engineer

To enhance user profiling and risk assessment, we are building web crawlers to collect relevant user data from third-party sources, forums, and the dark web. We are seeking a Senior Web Crawler & Data Extraction Engineer to design and implement these data collection solutions.

Key Responsibilities:

• Design, develop, and maintain web crawlers and scrapers to extract data from open web sources, forums, marketplaces, and the dark web.
• Implement data extraction pipelines that aggregate, clean, and structure data for fraud detection and risk profiling.
• Use Tor, VPNs, and other anonymisation techniques to safely crawl the dark web while avoiding detection.
• Develop real-time monitoring solutions for tracking fraudulent activities, data breaches, and cybercrime discussions.
• Optimise crawling speed and ensure compliance with website terms of service, ethical standards, and legal frameworks.
• Integrate extracted data with fraud detection models, risk scoring algorithms, and cybersecurity intelligence tools.
• Work with data scientists and security analysts to develop threat intelligence dashboards from collected data.
• Implement anti-bot detection evasion techniques and handle CAPTCHAs using AI-driven solvers where necessary.