Close Menu
CatchTheBullCatchTheBull
  • Home
  • Crypto News
  • Bitcoin
  • Altcoin
  • Blockchain
  • Airdrops News
  • NFT News
What's Hot

Ripple Expands Stablecoin Push With OpenPayd Integration

July 3, 2025

XRP Price Rebound Faces Historic Trendline Hurdle; Will it Breakout?

July 3, 2025

Do Banks Need Ripple XRP to Facilitate Money Transfers?

July 2, 2025
Facebook X (Twitter) Instagram
Facebook X (Twitter) Instagram
CatchTheBullCatchTheBull
  • Home
  • Crypto News
  • Bitcoin
  • Altcoin
  • Blockchain
  • Airdrops News
  • NFT News
CatchTheBullCatchTheBull
Blockchain

Enhancing Kubernetes with NVIDIA’s NIM Microservices Autoscaling

By WebDeskJanuary 24, 20252 Mins Read
Enhancing Kubernetes with NVIDIA’s NIM Microservices Autoscaling
Share
Facebook Twitter LinkedIn Pinterest Email


Terrill Dicki
Jan 24, 2025 14:36

Explore NVIDIA’s approach to horizontal autoscaling of NIM microservices on Kubernetes, utilizing custom metrics for efficient resource management.





NVIDIA has introduced a comprehensive approach to horizontally autoscale its NIM microservices on Kubernetes, as detailed by Juana Nakfour on the NVIDIA Developer Blog. This method leverages Kubernetes Horizontal Pod Autoscaling (HPA) to dynamically adjust resources based on custom metrics, optimizing compute and memory usage.

Understanding NVIDIA NIM Microservices

NVIDIA NIM microservices serve as model inference containers deployable on Kubernetes, crucial for managing large-scale machine learning models. These microservices necessitate a clear understanding of their compute and memory profiles in a production environment to ensure efficient autoscaling.

Setting Up Autoscaling

The process begins with setting up a Kubernetes cluster equipped with essential components such as the Kubernetes Metrics Server, Prometheus, Prometheus Adapter, and Grafana. These tools are integral for scraping and displaying metrics required for the HPA service.

The Kubernetes Metrics Server collects resource metrics from Kubelets and exposes them via the Kubernetes API Server. Prometheus and Grafana are employed to scrape metrics from pods and create dashboards, while the Prometheus Adapter allows HPA to utilize custom metrics for scaling strategies.

Deploying NIM Microservices

NVIDIA provides a detailed guide for deploying NIM microservices, specifically using the NIM for LLMs model. This involves setting up the necessary infrastructure and ensuring the NIM for LLMs microservice is ready for scaling based on GPU cache usage metrics.

Grafana dashboards visualize these custom metrics, facilitating the monitoring and adjustment of resource allocation based on traffic and workload demands. The deployment process includes generating traffic with tools like genai-perf, which helps in assessing the impact of varying concurrency levels on resource utilization.

Implementing Horizontal Pod Autoscaling

To implement HPA, NVIDIA demonstrates creating an HPA resource focused on the gpu_cache_usage_perc metric. By running load tests at different concurrency levels, the HPA automatically adjusts the number of pods to maintain optimal performance, demonstrating its effectiveness in handling fluctuating workloads.

Future Prospects

NVIDIA’s approach opens avenues for further exploration, such as scaling based on multiple metrics like request latency or GPU compute utilization. Additionally, leveraging Prometheus Query Language (PromQL) to create new metrics can enhance the autoscaling capabilities.

For more detailed insights, visit the NVIDIA Developer Blog.

Image source: Shutterstock


Credit: Source link

Previous Article$TRUMP Coin Faces Correction as Wall Street Pepe Presale Hits $58M Milestone
Next Article MicroStrategy Announces Full Redemption of Convertible Notes Due 2027

Related Posts

NVIDIA Omniverse Deprecates Launcher for Enhanced Developer Experience

July 2, 2025

Exploring Context Engineering in AI Agent Development

July 2, 2025

Solana (SOL) Breakout Hackathon: Winners Announced with Major Prizes

July 2, 2025
Add A Comment
Leave A Reply Cancel Reply

Top Posts

Ripple Expands Stablecoin Push With OpenPayd Integration

July 3, 2025

XRP Price Rebound Faces Historic Trendline Hurdle; Will it Breakout?

July 3, 2025

Do Banks Need Ripple XRP to Facilitate Money Transfers?

July 2, 2025

Subscribe to Updates

Get the latest Crypto, Blockchain and Airdrop News from us to Catch The Bull.

Advertisement Banner

Welcome to CatchTheBull, your trusted source for the latest Crypto News and Airdrops. We bring you real-time updates, expert insights, and opportunities to stay ahead in the crypto world. Discover trending projects, market analyses, and airdrop details all in one place.

Join us on this journey to navigate the ever-evolving blockchain universe!

Facebook X (Twitter) Instagram YouTube
Top Insights

Exploring Context Engineering in AI Agent Development

Solana (SOL) Breakout Hackathon: Winners Announced with Major Prizes

Monero eyes Rounded Bottom reversal towards $417 resistance

Get Informed

Subscribe to Updates

Get the latest Crypto, Blockchain and Airdrop News from us to Catch The Bull.

© 2025 CatchTheBull. All Rights Are Reserved.
  • Contact Us
  • Privacy Policy
  • Terms of Use
  • DMCA

Type above and press Enter to search. Press Esc to cancel.

  • bitcoinBitcoin(BTC)$108,681.002.84%
  • ethereumEthereum(ETH)$2,565.356.32%
  • tetherTether(USDT)$1.000.03%
  • rippleXRP(XRP)$2.242.94%
  • binancecoinBNB(BNB)$659.991.96%
  • solanaSolana(SOL)$152.463.02%
  • usd-coinUSDC(USDC)$1.000.00%
  • tronTRON(TRX)$0.2850801.79%
  • dogecoinDogecoin(DOGE)$0.1687686.45%
  • staked-etherLido Staked Ether(STETH)$2,565.566.35%
  • cardanoCardano(ADA)$0.597.58%
  • wrapped-bitcoinWrapped Bitcoin(WBTC)$108,648.002.94%
  • HyperliquidHyperliquid(HYPE)$39.887.76%
  • Wrapped stETHWrapped stETH(WSTETH)$3,103.746.92%
  • bitcoin-cashBitcoin Cash(BCH)$504.030.62%
  • suiSui(SUI)$2.897.75%
  • chainlinkChainlink(LINK)$13.484.92%
  • leo-tokenLEO Token(LEO)$8.990.57%
  • avalanche-2Avalanche(AVAX)$18.517.70%
  • stellarStellar(XLM)$0.2385335.12%
  • USDSUSDS(USDS)$1.000.00%
  • the-open-networkToncoin(TON)$2.873.08%
  • shiba-inuShiba Inu(SHIB)$0.0000125.34%
  • WETHWETH(WETH)$2,566.066.32%
  • Wrapped eETHWrapped eETH(WEETH)$2,750.246.41%
  • litecoinLitecoin(LTC)$87.154.16%
  • hedera-hashgraphHedera(HBAR)$0.1554607.66%
  • Binance Bridged USDT (BNB Smart Chain)Binance Bridged USDT (BNB Smart Chain)(BSC-USD)$1.00-0.01%
  • whitebitWhiteBIT Coin(WBT)$43.59-1.32%
  • moneroMonero(XMR)$320.772.77%
  • polkadotPolkadot(DOT)$3.547.78%
  • bitget-tokenBitget Token(BGB)$4.582.68%
  • Ethena USDeEthena USDe(USDE)$1.000.00%
  • Coinbase Wrapped BTCCoinbase Wrapped BTC(CBBTC)$108,694.002.71%
  • uniswapUniswap(UNI)$7.3211.81%
  • pepePepe(PEPE)$0.0000109.18%
  • aaveAave(AAVE)$274.735.53%
  • Pi NetworkPi Network(PI)$0.4940341.54%
  • daiDai(DAI)$1.000.04%
  • Ethena Staked USDeEthena Staked USDe(SUSDE)$1.180.07%
  • aptosAptos(APT)$4.746.44%
  • okbOKB(OKB)$49.872.25%
  • BittensorBittensor(TAO)$335.024.72%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • nearNEAR Protocol(NEAR)$2.259.37%
  • Jito Staked SOLJito Staked SOL(JITOSOL)$185.033.12%
  • internet-computerInternet Computer(ICP)$5.026.37%
  • ethereum-classicEthereum Classic(ETC)$16.966.20%
  • crypto-com-chainCronos(CRO)$0.0822793.32%
  • OndoOndo(ONDO)$0.795.76%
  • bitcoinBitcoin(BTC)$108,681.002.84%
  • ethereumEthereum(ETH)$2,565.356.32%
  • tetherTether(USDT)$1.000.03%
  • rippleXRP(XRP)$2.242.94%
  • binancecoinBNB(BNB)$659.991.96%
  • solanaSolana(SOL)$152.463.02%
  • usd-coinUSDC(USDC)$1.000.00%
  • tronTRON(TRX)$0.2850801.79%
  • dogecoinDogecoin(DOGE)$0.1687686.45%
  • staked-etherLido Staked Ether(STETH)$2,565.566.35%
  • cardanoCardano(ADA)$0.597.58%
  • wrapped-bitcoinWrapped Bitcoin(WBTC)$108,648.002.94%
  • HyperliquidHyperliquid(HYPE)$39.887.76%
  • Wrapped stETHWrapped stETH(WSTETH)$3,103.746.92%
  • bitcoin-cashBitcoin Cash(BCH)$504.030.62%
  • suiSui(SUI)$2.897.75%
  • chainlinkChainlink(LINK)$13.484.92%
  • leo-tokenLEO Token(LEO)$8.990.57%
  • avalanche-2Avalanche(AVAX)$18.517.70%
  • stellarStellar(XLM)$0.2385335.12%
  • USDSUSDS(USDS)$1.000.00%
  • the-open-networkToncoin(TON)$2.873.08%
  • shiba-inuShiba Inu(SHIB)$0.0000125.34%
  • WETHWETH(WETH)$2,566.066.32%
  • Wrapped eETHWrapped eETH(WEETH)$2,750.246.41%
  • litecoinLitecoin(LTC)$87.154.16%
  • hedera-hashgraphHedera(HBAR)$0.1554607.66%
  • Binance Bridged USDT (BNB Smart Chain)Binance Bridged USDT (BNB Smart Chain)(BSC-USD)$1.00-0.01%
  • whitebitWhiteBIT Coin(WBT)$43.59-1.32%
  • moneroMonero(XMR)$320.772.77%
  • polkadotPolkadot(DOT)$3.547.78%
  • bitget-tokenBitget Token(BGB)$4.582.68%
  • Ethena USDeEthena USDe(USDE)$1.000.00%
  • Coinbase Wrapped BTCCoinbase Wrapped BTC(CBBTC)$108,694.002.71%
  • uniswapUniswap(UNI)$7.3211.81%
  • pepePepe(PEPE)$0.0000109.18%
  • aaveAave(AAVE)$274.735.53%
  • Pi NetworkPi Network(PI)$0.4940341.54%
  • daiDai(DAI)$1.000.04%
  • Ethena Staked USDeEthena Staked USDe(SUSDE)$1.180.07%
  • aptosAptos(APT)$4.746.44%
  • okbOKB(OKB)$49.872.25%
  • BittensorBittensor(TAO)$335.024.72%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • nearNEAR Protocol(NEAR)$2.259.37%
  • Jito Staked SOLJito Staked SOL(JITOSOL)$185.033.12%
  • internet-computerInternet Computer(ICP)$5.026.37%
  • ethereum-classicEthereum Classic(ETC)$16.966.20%
  • crypto-com-chainCronos(CRO)$0.0822793.32%
  • OndoOndo(ONDO)$0.795.76%