Lab Engineer - Lab Operations (Broomfield)
In a world increasingly driven by software innovation, Wind River is pioneering the technologies to accelerate the digital transformations of our customers with a new generation of Mission Critical AI Systems, with the most exacting standards for safety, security, performance, and reliability. Success will be determined by our ability to innovate with velocity and deliver at the solutions level.
Wind River’s impact spans critical infrastructure domains such as telecommunications, including 5G; industrial (automation, sustainable energy, robotics, mining), connected healthcare and medical devices, automotive (connected and self-driving vehicles), and aerospace & defense. We were recognized by VDC Research in July 2020 as #1 in Edge Compute OS Platforms, overtaking Microsoft as the overall commercial leader. Wind River regularly wins industry recognitions for excellence in IoT security, cloud and edge computing, as well as 8 consecutive years as a “Top Work Place”. If you’re passionate about amplifying your impact on the world, in a caring, respectful culture with a growth mindset, come join us and help lead the way into the future of the intelligent edge!
The Lab Operations team supports the engineering teams such as VxWorks, Wind River Linux, Wind River Cloud Platform, and Wind River Studio inside Wind River. We provide reliable computer labs and tools for engineers to develop, test, and validate their software and solutions. Our new Colorado site will host a wide range of devices including embedded devices, single board computers, desktop platforms, servers, and cloud solutions. Our team currently supports 5 engineering lab sites globally.
We are looking for someone who will take an active hands-on approach to managing, configuring, and improving our test automation targets. In addition, you will be supporting, configuring, repairing and upgrading our lab equipment, networks and services worldwide.
You will be responsible for configuring and maintaining systems to allow remote Linux, VxWorks, and cloud engineering teams to use the environment. This includes:
- Installing targets in our new environment and configuring them for remote access and network booting
- Configuring firmware, BIOS setup
- Flashing new firmware images/BIOS/uboot/VxWorks boot loaders on systems, NICs and devices; clone USB and SD images
- Power configuration, PDU management and balancing power usage.
- Building clusters of machines into clouds to host our WRCP (Wind River Cloud Platform) software.
- System access control
- Networking and specialty server configuration
- Linux boot server configuration
- Linux administration on various distributions (Ubuntu, CentOS, Debian and others)
- IPMI/iLO/iDRAC/BMC configuration
- Serial and USB serial console cabling and provisioning and terminal server configuration
- Engineering team lab user support
- KVM configuration
Special note: During the Covid-19 pandemic, you will have to work on-site daily, with only a minimal number of other support staff. We do our best to keep personnel far apart from each other and follow strict masking and cleaning protocols. Personal safety is something we take seriously. The requirements of the lab hardware demands that someone be on site to install, repair, effect changes and provide support to the engineering teams. You will be working among a global team that can help provide assistance and assist in debugging issues. Initially, you will be working from home to help prepare the new lab configurations and help at our other global sites.
You must be organized and like helping solve our customers’ issues. You must love learning and working with technology and hardware. You document any new procedures and findings with the team. We work on a wide variety of technologies, so a willingness to learn new skills is a must. Your research skills will be tested as you find solutions to a wide range of different hardware, firmware, booting methods, operating systems, networking, and various other technologies. You must be innovative and motivated. You will have a cell phone available to help you provide business hours support. You understand the various types of physical interfaces for disks, USB, serial ports, and networking (100Base-T to QSFP28). You will help set up monitoring of critical resources as well as power and air conditioning. Resourcefulness is a must, as is the desire to find innovative solutions.
You must have experience with:
- Hands-on experience and mechanical aptitude in a lab environment
- Experience with embedded systems with multiple architectures Intel/PPC/MIPS/ARM from many vendors as NXP, ARM, TI, Renesas, Marvell, Dell , Intel, Cavium, and others.
- Intel based servers and systems. Understanding BMC, iDRAC, iLO, ipmitool, Serial-Over-LAN, SSH as well as server hardware, CPU, RAM, SATA, SAS, NVMe, and BIOS configuration for remote access.
- Various computer hardware. Understand network booting of devices, PXE, gPXE, iPXE, DHCP, and EFI network booting.
- Understand Linux network configuration, VLANs, untagged and tagged interfaces, L3 routing, OSPF, LAG, and network switch configuration for L2 and L3 (eg: Dell CLI, Cisco CLI)
- Understand serial ports (physical and USB), RS232, configuring terminal servers and KVM connectivity using VGA/DVI/DP/HDMI.
- Work safely with ESD-sensitive boards and devices.
- You must have strong Linux, CLI, editor, and networking skills. Must be knowledgeable in IPv4 and IPv6 and be able to resolve network issues. Must be able to extract detailed system data as well as configure and debug services.
- Update and validate inventory, tools, and provisioning info as you work with equipment.
- Create, update and improve documentation to aid the lab team and our lab users. Use screenshots to aid in explaining concepts. You document resolutions and/or provide links to solutions to issues as you solve them.
- Clearly communicate with our customers to keep them informed about the status of issues.
- Understand SSH and key-based authentication.
- Git version control
- Must be a self-starter and have research skills to find solutions to obscure problems.
- Be able to visually draw a lab/network/cluster of equipment using Visio to describe labs.
- Researching solutions to new problems.
- Manage over one thousand lab targets locally and assist remote labs with their targets and clouds.
- Manage servers that host cloud test and engineering environments.
- Move and migrate hardware. Upgrade BIOS, uboot, VxWorks Boot loaders, and firmware of NICs, disks, SSDs and other controllers. Set environment variables to enable booting and remote access of targets.
- Configure server hardware to enable remote access and provide the best environment for our cloud systems.
- Support customers with a variety of issues on hardware ranging from embedded boards to server class hardware.
- Work with the team to solve lab issues. These include firmware issues, software configuration, network booting of devices, console access, KVM access, debugging hardware, and upgrading firmware and BIOS of devices.
- Be able to work remotely on systems using a combination of serial, Serial-Over-LAN, IPMI, remote console, KVM, SSH or other methods. Assist engineers with remote access issues.
- Help engineers and developers find documentation and information to use the labs effectively.
- Use scripting and monitoring tools to identify lab issues and proactively fix issues.
- Document changes and solutions, to assist others. Create new procedures if one does not exist.
- Computer Science, Computer Networking, Electrical Eng. or IT degree/diploma
- Passion for technology, learning, and researching solutions to technical problems
- Mechanical aptitude is a must since you will be working with servers, boards, and computers with many architectures. A hands-on approach to be able to migrate, install and reconfigure boards, drives, networking gear, and servers. Must be able to rack heavy systems safely.
- Detail-oriented. Make sure all aspects of issues are completed.
- Positive attitude; be a great communicator; be eager to share findings and work well with others
- Provide customer support to our internal engineering teams
KNOWLEDGE AND SKILL REQUIREMENT:
- Computer and networking experience including Linux, networking skills (Cisco/Dell/HP/Quanta), and basic Bash/Python sysadmin scripting skills.
- Electronics and embedded board experience
- Ability to solder specialized connectors on occasion
Wind River is an Equal Opportunity Employer with a commitment to diversity. We prohibit discrimination based on race, color, religion, gender, national origin, age, disability, veteran status, marital status, pregnancy, gender expression or identity, sexual orientation or any other legally protected status.