Senior Data Engineer Big Data & Cloud Integration L3
Select Minds
Full-time
On-site
Dallas, Texas, United States
$60 - $65 USD hourly
Big Data
Replies within 24 hours
Benefits:
- HYBRID
- Competitive salary
- Opportunity for advancement
Senior Data Engineer – Big Data & Cloud Integration
DALLAS, TX - HYRBID
LONG TERM
IN-PERSON INTERVIEW
Data Engineer - L3 ROLE
- Translate complex cross-functional business requirements and functional specifications into logical program designs and data solutions.
- Partner with the product team to understand business needs and specifications.
- Solve complex architecture, design and business problems.
- Coordinate, Execute and participate in component integration (CIT) scenarios, system integration testing (SIT), and user acceptance testing (UAT) to identify application errors and to ensure quality software deployment.
- Continuously work with cross-functional development teams (Data Analysts and Software Engineers) for creating PySpark jobs using Spark SQL and help them build reports on top of data pipelines.
- Build, test and enhance data curation pipelines, integrate data from a wide variety of sources like DBMS, File systems and APIs for various OKRs and metrics development with high data quality and integrity.
- Execute the development, maintenance, and enhancements of data ingestion solutions of varying complexity levels across various data sources like DBMS, File systems (structured and unstructured), APIs and Streaming on on-prem and cloud infrastructure.
- Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.
- Work with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas.
- Develop spark applications in PySpark on distributed environment to load huge number CSV files with different schema in to Hive ORC tables.
- Perform ETL transformations on the data loaded into Spark Data Frames and do the in-memory computation.
- Develop and implement data pipelines using AWS services such as Kinesis, S3 to process data in real-time.
- Work with monitoring, logging and cost management tools that integrate with AWS.
- Schedule the spark jobs using Airflow scheduler to monitor their performance.
....
- Partner with the product team to understand business needs and specifications.
- Solve complex architecture, design and business problems.
- Coordinate, Execute and participate in component integration (CIT) scenarios, system integration testing (SIT), and user acceptance testing (UAT) to identify application errors and to ensure quality software deployment.
- Continuously work with cross-functional development teams (Data Analysts and Software Engineers) for creating PySpark jobs using Spark SQL and help them build reports on top of data pipelines.
- Build, test and enhance data curation pipelines, integrate data from a wide variety of sources like DBMS, File systems and APIs for various OKRs and metrics development with high data quality and integrity.
- Execute the development, maintenance, and enhancements of data ingestion solutions of varying complexity levels across various data sources like DBMS, File systems (structured and unstructured), APIs and Streaming on on-prem and cloud infrastructure.
- Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.
- Work with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas.
- Develop spark applications in PySpark on distributed environment to load huge number CSV files with different schema in to Hive ORC tables.
- Perform ETL transformations on the data loaded into Spark Data Frames and do the in-memory computation.
- Develop and implement data pipelines using AWS services such as Kinesis, S3 to process data in real-time.
- Work with monitoring, logging and cost management tools that integrate with AWS.
- Schedule the spark jobs using Airflow scheduler to monitor their performance.
....
Flexible work from home options available.
Compensation: $60.00 - $65.00 per hour
About Us
We work to deliver profitability in your business – with effective communication, consulting, and interactive solutions. Following an Agile Work Approach, we make sure you get the ideal solutions at minimum expenses.
Work Approach
Our Philosophy
Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Work Policy
We promote a collaborative work environment. We involve everyone working in the organization in community decisions and encourage them to think from a broader perspective. Our work process promotes flexibility and we maintain a high level of discipline at different levels of execution.
The Future
SelectMinds have years of experience in the domain helps us understand the need-of-the-hour better. This understanding drives us to a better future with every minute ticking. We believe we will be taking off major businesses from their flagship positions, with the products we are eyeing today.