Site Reliability Engineer
-
Lieu :RTP, North Carolina, US
-
Centre d'intérêtIngénieur - logiciels
-
Type de posteExpérimenté
-
Intérêt pour la technologieDéveloppement de logiciel, tests
-
ID de poste1430221
Who We Are
At Cisco, we are a global leader in networking and IT, driving innovation and redefining how people connect, communicate, and collaborate. Our mission is to shape the future of the internet by creating unprecedented value and opportunity for our customers, employees, investors, and ecosystem partners. We are committed to encouraging a diverse and partnership environment where everyone can thrive and encourage our collective success.
Who You Are
We are seeking a highly skilled and experienced Senior Engineer to join our team, focusing on the design and development of AI services and capabilities tailored to IT’s GPU-based AI Clusters observability. This role involves reshaping how we lead alerts, metrics, and logs by introducing deep learning and GenAI to enhance reliability services. The ideal candidate will have a strong background in artificial intelligence, machine learning, and GPU-based AI infrastructure, with a consistent record of delivering innovative solutions that enhance system monitoring, performance, and reliability.
Key Responsibilities:
- Design, build, and maintain observability systems for leading NVIDIA DGX clusters, ensuring flawless monitoring of AI workloads, hardware utilization (GPUs), and system health.
- Develop monitoring tools and dashboards that supervise key metrics such as GPU utilization, memory, temperature, latency, network bandwidth, model performance, and system availability.
- Build custom alerting systems for AI/ML workflows, enabling proactive issue detection (e.g., GPU failures, hardware bottlenecks, system crashes).
- Collaborate with IT and MLOps teams to design efficient, scalable solutions for deploying, monitoring, and leading machine learning models on DGX systems.
- Optimize DGX infrastructure by implementing standard processes for observability, ensuring high performance and reducing operational costs.
- Supervise system-level metrics such as hardware temperature, power consumption, and GPU/CPU health, preventing hardware degradation or failure.
- Develop solutions for supervising AI/ML model performance across DGX clusters, integrating logging and supervising for model training, inference, and deployment processes.
- Integrate observability tools (e.g., Prometheus, Grafana, Splunk) with NVIDIA-specific tools (e.g., DCGM, NVIDIA GPU Cloud) for real-time monitoring and alerting.
- Work closely with data scientists and machine learning engineers to ensure effective resource utilization and model observability, including the identification of performance bottlenecks and tuning for optimal GPU usage.
- Drive solving and root cause analysis for failures and anomalies in both the DGX hardware and AI/ML models running on the infrastructure.
- Ensure compliance with ethical AI standards by monitoring fairness, model drift, and performance consistency.
- Document standard methodologies and processes for managing, deploying, and monitoring AI workloads on DGX clusters.
Minimum Qualifications:
- Bachelor’s degree in Computer Science, Software Engineering, Data Science, or related fields.
- 7+ years of experience software engineering, systems engineering, or DevOps roles.
- 3+ years of experience in high-performance computing (HPC) or AI/ML environments.
Preferred Qualifications:
- Strong experience leading NVIDIA DGX systems or similar GPU-based computing clusters.
- Proficiency in GPU monitoring tools such as NVIDIA Data Center GPU Manager (DCGM) and related NVIDIA libraries/APIs.
- Experience with AI/ML model deployment and monitoring on large-scale infrastructure, including model performance metrics (latency, throughput, accuracy).
- Hands-on experience with observability tools such as Prometheus, Grafana, Splunk or similar, especially in high-performance computing environments.
- Proficiency in scripting/programming languages (e.g., Python, Bash, Go) for automating cluster management and monitoring tasks.
- Experience with container orchestration technologies (e.g., Docker, Kubernetes), including NVIDIA’s GPU operator for Kubernetes.
- Familiarity with AI/ML lifecycle management tools such as ML flow, Kubeflow, or similar.
- Strong understanding of HPC environments, including distributed computing, storage, and networking for AI/ML workloads.
- Experience with infrastructure monitoring and solving at both hardware (GPU, CPU, memory) and software (AI/ML models, applications) levels.
- Strong analytical and problem-solving skills, with the ability to interpret complex data and develop actionable insights.
- Excellent verbal and written communication skills, with the ability to convey technical concepts to non-technical partners.
- Ability to work effectively in a collaborative team environment and lead multiple projects simultaneously.
- Experience with NVIDIA NGC (NVIDIA GPU Cloud) and DGX OS software stack for large-scale AI workloads.
- Understanding of AI workload orchestration with frameworks such as Slurm or Kubernetes in GPU-based clusters.
- Knowledge of NVIDIA Deep Learning frameworks (TensorFlow, PyTorch) and their performance optimization on DGX infrastructure.
- Experience with AIOps tools for automated anomaly detection and solving of large-scale AI infrastructure.
- Certification or experience with cloud platforms that offer GPU instances (AWS, GCP, Azure).
- Familiarity with network performance tuning in HPC environments and large-scale AI workloads.
- Familiarity with DevOps practices and tools, including CI/CD pipelines and infrastructure as code. Knowledge of Graphs, Graph DB's and Graph Theory. Familiarity with Terraform, Helm Chart, Ansible, or similar tools.
Why Cisco
#WeAreCisco, where each person is unique, but we bring our talents to work as a team and make a difference powering an expansive future for all.
We adopt digital, and help our customers implement change in their digital businesses. Some may think we’re “old” (36 years strong) and only about hardware, but we’re also a software company. And a security company. We even invented an intuitive network that adapts, predicts, learns and protects. No other company can do what we do – you can’t put us in a box!
But “Digital Transformation” is an empty buzz phrase without a culture that allows for innovation, creativity, and yes, even failure (if you learn from it.)
Day to day, we focus on the give and take. We give our best, give our egos a break, and give of ourselves (because giving back is built into our DNA.) We take accountability, bold steps, and take difference to heart. Because without diversity of thought and a dedication to equality for all, there is no moving forward.
So, you have colorful hair? Don’t care. Tattoos? Show off your ink. Like polka dots? That’s cool. Pop culture geek? Many of us are. Passion for technology and world changing? Be you, with us!
Lorsqu’elle est disponible, la fourchette salariale affichée pour ce poste reflète l’échelle d’embauche prévue pour les salaires des nouveaux embauchés aux États-Unis et/ou Canada. Pour les postes non liés à la vente, les fourchettes d’embauche reflètent uniquement le salaire de base; les employés sont également admissibles à des primes annuelles. Les fourchettes d’embauche pour les postes de vente comprennent la rémunération de base et la rémunération incitative. La rémunération individuelle est déterminée par le lieu d’embauche du candidat et par d’autres facteurs, notamment les compétences, l’expérience et les études, certifications ou formations pertinentes. Les candidats peuvent ne pas être admissibles à la fourchette complète des salaires en fonction de leur lieu d’embauche aux États-Unis ou Canada. Le recruteur peut vous donner plus d'informations sur la rémunération du poste dans votre lieu au cours du processus de recrutement.
Les employés américains ont accès à une assurance médicale, dentaire et visuelle de qualité, à un régime 401(k) avec une contribution équivalente de Cisco, à une couverture d’invalidité à court et à long terme, à une assurance vie de base et à de nombreuses prestations de bien-être. Les employés reçoivent jusqu’à douze jours fériés payés par année civile, qui comprennent un jour férié flottant, plus un jour de congé pour leur anniversaire. Les employés peuvent cumuler jusqu'à 20 jours de congés payés (PTO) par an et ont accès à des absences payées pour faire face à des problèmes critiques ou urgents sans avoir à puiser dans leurs congés. Nous offrons du temps supplémentaire rémunéré pour faire du bénévolat et rendre service à la communauté. Les employés peuvent également acheter des actions de l’entreprise dans le cadre de notre Programme d’achat d’actions pour les employés.
Les employés participant à des plans de vente reçoivent, en plus de leur salaire de base, une rémunération incitative fondée sur les performances, qui est répartie entre les composantes sur quota et non. Pour la rémunération incitative basée sur des quotas, Cisco paie au taux standard de 1 % de la cible incitative pour chaque 1 % de chiffre d’affaires réalisé par rapport au quota jusqu’à 100 %. Une fois que la performance dépasse 100 % du quota, les taux incitatifs peuvent augmenter jusqu’à cinq fois le taux standard sans plafonnement de la rémunération incitative. Pour les éléments de performance des ventes non basés sur des quotas, tels que les objectifs de vente stratégiques, Cisco peut payer jusqu’à 125 % de l’objectif. Les plans de vente de Cisco ne prévoient pas de seuil minimum de performance pour le versement de la rémunération incitative des ventes.