Datics AI - Software Engineer – Python

Software Engineer – Python

Experience: 2+ years

Education: 4 years bachelor’s in Computer Science degree. Candidates with relevant experience and a high IQ are highly encouraged to apply.

We are looking for a Python Sofware Engineer specializing in Web Scraping to join our team and help us develop and maintain various scraping products which fuel our Big Data Market Analytics Platform. You will be responsible for designing, developing, and implementing various scrapers to extract data from simple to complex websites and maintaining Data Pipelines.

Requirements:

– 2+ years of work experience in large-scale scraping projects
– Good knowledge of Various Databases from RDBMS to NoSQL ( MSSQL, MongoDB)
– Good understanding of Data structures and Algorithms
– Good Experience in Web Scraping framework Scrapy, Beautifulsoup, Splash or other frameworks and related libraries
– Deploy the python scripts/crawlers on Linux-based servers
– Ability to inspect and replicate the API calls in the scrapy spiders with required payloads and session variables.
– Knowledge to parse HTML by using complex paths and JSON outputs with JSON modules.
– Capable of quickly learning and implementing new tools/technologies

Responsibilities:

– Analyze and design different data models/tables to store crawlers’ output.
– Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL.
– Create data normalization, data enrichment, and QA pipelines by using different modules in python to clean and normalize raw data scraped from different sources.
– Write complex SQL queries or procedures to manipulate/merge the data.
– Deploy the python scripts to Linux-based AWS servers
– Manage the performance of MySQL databases by performance tuning, query optimization & index tuning.

Apply here: talent@datics.ai

Department
Engineering
Position
In House - Full-Time
Experience
Junior

Get in Touch