Senior Data Engineer
3-6 month contract
Need to confirm resource within the next week or 2
Start date by Mid Augsut
This is an ongoing project we have been working on for awhile now and they need an additional Data Engineer to assist
 
Generally they are taking in data from IoT sensors that go around shipments high value shipments have sensors and processing that through spark pipeline and loading into datamart for app to consume
They will need to take that over and as data use cases come in, modify the pipeline to do that
We will probably also be converting it to a real data timeline
 
Must Have Skills:
Senior Data Engineering
Spark & Spark notes- need to have experience with sparknotes in python
SQL
Microsoft Fabric -They need to have fabric experience and develop spark notes and jobs in fabric
Overview:
We are looking for a talented Data Engineer with a strong focus on Microsoft Fabric and expertise in writing Spark notebooks and jobs. The ideal candidate will have extensive experience in building and optimizing data pipelines, with a deep understanding of Microsoft Fabric's capabilities. Experience with IoT data or geospatial data is a significant plus. This role involves designing and implementing efficient data engineering solutions, including real-time data processing using Spark Streaming, to support our analytics initiatives.
 
Key Responsibilities:

• • Data Pipeline Development: Design, develop, and optimize data pipelines using Microsoft Fabric and Spark to process and analyze large datasets.
  • Spark Notebooks/Jobs: Write and manage Spark notebooks and jobs, ensuring they are efficient, scalable, and maintainable.
  • Spark Streaming: Implement real-time data processing solutions using Spark Streaming, enabling timely analytics and insights from continuously flowing data.
  • Data Integration: Integrate data from various sources, including IoT devices and geospatial data, to support comprehensive analytics and reporting.
  • Data Transformation: Implement data transformation and cleansing processes to ensure data quality and consistency.
  • Performance Tuning: Optimize data processing workflows for performance, ensuring minimal latency and efficient resource usage.

Qualifications:

• • Experience: Proven experience as a Data Engineer with a strong focus on Microsoft Fabric and Spark.
  • Technical Skills:
    • Expertise in writing and optimizing Spark notebooks and jobs.
    • Proficiency in Microsoft Fabric, including data integration and pipeline development.
    • Experience with Spark Streaming for real-time data processing.
    • Experience with IoT data or geospatial data is highly desirable.
  • Analytical Skills: Strong analytical and problem-solving skills, with the ability to work with large and complex datasets.
  • Communication: Excellent communication skills, with the ability to collaborate effectively with technical and non-technical team members.
  • Education: Bachelor’s degree in Computer Science, Data Engineering, or a related field (or equivalent experience).

Preferred Qualifications:

• Certifications: Microsoft Azure certifications, particularly DP-600 - Microsoft Certified: Fabric Analytics Engineer Associate.
• Industry Knowledge: Experience working with IoT data, geospatial data, or other specialized data types is an advantage.

This is a remote, work from home position. This role is to be filled outside the states of California and Colorado.

**We are an equal opportunity employer and value diversity at our company. We do not discriminate based on race, religion, color, ethnic origin, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.**

#LI-KH1