Here's where the Big Four stand on hybrid work in 2025. The world's four largest consulting and accounting firms collectively employ 1.5 million staff and take up prime commercial real estate in hundreds of major cities worldwide. But all four firms are bucking the trend toward stricter return-to-office policies being followed by other corporate powerhouses. Companies including JPMorgan, Dell, AT&T, and Amazon have reversed their stances on remote ...

The Big Four are sticking with hybrid work. Here are the RTO policies of Deloitte, KPMG, EY, and PwC.

Employees can log into work from the comfort of their couch while raking it in as a nurse practitioner, sales associate or therapist.

Here are the top 10 side hustles that still allow you to work from home and earn up to $56 an hour—as firms like AT&T ban it | Fortune

In what spells more bad news for avid advocates of remote work, telecom giant AT&T has joined other high-profile RTO enforcers such as Amazon in mandating a five-days-a-week return to the office, starting January.

15 Companies Now Hiring Remote Jobs For 2025

Opening the roof at AT&T Stadium before Monday night's Texans-Cowboys game suddenly became dangerous.

Monday Night Football: Metal fragment falls from AT&T Stadium roof after opening before Texans-Cowboys game - Yahoo Sports

The Latest News and Updates in brought to you by the team at WTRF:

AT&T is providing extra safety support for families this Cybersecurity 
Awareness Month

IntelBroker claims the breach impacts Microsoft, SAP, AT&T, Verizon, T-Mobile US, and more

Cisco confirms ongoing probe into alleged data breach • The Register

According to Nvidia, AT&T is teaming with Quantiphi to create a conversational AI platform to assist employees with software development and financial services tasks.

Nvidia highlights AI software and services at D.C. AI Summit

AT&T

MONROVIA, Liberia, February 28th, 2025/Chainwire/--Chad Leads the Way: AI-Powered Memecoins Are Evolving. Memeagent has introduced Chad, an autonomous AI agent designed to bring intelligence and automation to the memecoin ecosystem. While many meme projects rely on fleeting trends, Memeagent is built differently – a fully AI agent-driven ecosystem designed to change the game for good. As AI reshapes the landscape, Memeagent is positioning itself at the forefront of this evolution. Chad: The AI Engine Driving Memeagent’s Evolution Chad is a key figure who represents Memeagent. He’s a fully automated AI agent that helps users to navigate Memeagent. Memeagent is an AI-powered ecosystem designed for sustained growth and real utility. At the core of this innovation is Chad, Memeagent’s flagship AI agent, showcasing the advanced capabilities that will soon be available to all users. Built on ElizaOS, a cutting-edge open-source framework for autonomous AI agents, Chad can think, learn, and interact independently. With persistent memory, adaptability, and live data integration, he stays ahead of trends, making real-time decisions in an ever-evolving market. ElizaOS is a pioneer in decentralized AI (DeAI), allowing Chad to operate seamlessly within blockchain ecosystems. With features like encrypted transactions, asset management, and real-time analytics, Chad isn’t just generating memes—he’s actively shaping market dynamics. Working 24/7, Chad monitors trends, predicts market shifts, and engages across major social platforms without hesitation. Unlike human traders, he never sleeps, never FOMOs, and never misses an opportunity – driving Memeagent’s AI-driven viral growth. He will automatically respond to all users needs and help them navigate through Memeagent itself. The Reasons Memeagent Is Gaining Attention Behind the scenes, big moves are happening. Memeagent combines AI-powered engagement, predictive analytics, and automated viral marketing into one platform, making it one of the most powerful ecosystem for meme adoption yet. Here’s why serious players are locking in: Always-On AI – Chad works around the clock, monitoring X/Twitter, Telegram, Instagram, and more to maximize exposure.
Predictive Trend Analysis – Spot what’s about to go viral before the rest of the market catches on.
Automated Growth – Hands-free content and engagement across major social platforms.
No Technical Barriers – AI handles the heavy lifting, so anyone can capitalize. $MA: The Fuel Behind the Memeagent Ecosystem Unlike other meme coins, $MA goes beyond speculation—it powers the entire Memeagent platform, unlocking a range of utilities that set it apart. AI agent Creation – Use $MA to create and deploy custom AI-powered Memeagents that generate viral content and interact across social platforms.
DeFi Utilities – Stake $MA for passive income and participate in governance decisions to shape the future of the Memeagent ecosystem.
Deflationary Impact – A portion of every $MA transaction is burned, reducing supply and increasing scarcity over time. Memeagent and Chad Gain Traction as AI-Powered Platforms Expand According to the team, Memeagent isn’t another speculative meme coin—it’s a structured, AI-powered ecosystem with long-term incentives. The Features That Set Memeagent Apart AI-Driven Growth – Automation supports long-term engagement beyond short-term trends.
$MA Token Utility – Powers AI tools, governance, staking, and a deflationary tokenomics model.
Security & Infrastructure – Built with SOC 2 Type 2-certified security for reliability and trust. Memeagent’s Roadmap is Packed Memeagent is gaining attention as its ecosystem expands. With a structured roadmap ahead and continuous development, the platform is set for steady growth. Upcoming milestones: Q2 2025: Beta AI automation tools roll out to early users.
Q3 2025: Advanced predictive analytics and expanded AI engagement.
Q4 2025: Full staking launch, multilingual support, and deeper integrations.
2026: Memeagent’s ecosystem expands to full-scale adoption. Market Movers Are Taking Notice Memeagent is already in motion, leveraging AI and automation to drive engagement. As the ecosystem expands, those recognizing its potential are positioning themselves early. The Next Evolution of Memecoins Is Built on Innovation, Not Just Hype. AI-Powered Innovation in Memecoins Memeagent is introducing AI-driven engagement to the memecoin ecosystem. As the platform grows, early users can explore $MA’s utility and experience Chad’s AI-powered automation in action. Users can visit www.memeagent.io to get started now. #CHADISCOMING About Memeagent Memeagent is developed by a team of experienced AI engineers and blockchain specialists with extensive backgrounds in DeFi, yield farming, token design, and AI-driven automation. Active in the crypto space since 2016, the team has contributed to various high-profile projects, focusing on scalability, security, and decentralized AI (DeAI) solutions. Contact Benjamin Peng info@memeagent.io This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here MONROVIA, Liberia, February 28th, 2025/Chainwire/--Chad Leads the Way: AI-Powered Memecoins Are Evolving. Memeagent has introduced Chad, an autonomous AI agent designed to bring intelligence and automation to the memecoin ecosystem. While many meme projects rely on fleeting trends, Memeagent is built differently – a fully AI agent-driven ecosystem designed to change the game for good. As AI reshapes the landscape, Memeagent is positioning itself at the forefront of this evolution. Chad: The AI Engine Driving Memeagent’s Evolution Chad is a key figure who represents Memeagent. He’s a fully automated AI agent that helps users to navigate Memeagent. Memeagent is an AI-powered ecosystem designed for sustained growth and real utility. At the core of this innovation is Chad, Memeagent’s flagship AI agent, showcasing the advanced capabilities that will soon be available to all users. Built on ElizaOS, a cutting-edge open-source framework for autonomous AI agents, Chad can think, learn, and interact independently. With persistent memory, adaptability, and live data integration, he stays ahead of trends, making real-time decisions in an ever-evolving market. ElizaOS is a pioneer in decentralized AI (DeAI), allowing Chad to operate seamlessly within blockchain ecosystems. With features like encrypted transactions, asset management, and real-time analytics, Chad isn’t just generating memes—he’s actively shaping market dynamics. Working 24/7, Chad monitors trends, predicts market shifts, and engages across major social platforms without hesitation. Unlike human traders, he never sleeps, never FOMOs, and never misses an opportunity – driving Memeagent’s AI-driven viral growth. He will automatically respond to all users needs and help them navigate through Memeagent itself. The Reasons Memeagent Is Gaining Attention Behind the scenes, big moves are happening. Memeagent combines AI-powered engagement, predictive analytics, and automated viral marketing into one platform, making it one of the most powerful ecosystem for meme adoption yet. Here’s why serious players are locking in: Always-On AI – Chad works around the clock, monitoring X/Twitter, Telegram, Instagram, and more to maximize exposure. Predictive Trend Analysis – Spot what’s about to go viral before the rest of the market catches on. Automated Growth – Hands-free content and engagement across major social platforms. No Technical Barriers – AI handles the heavy lifting, so anyone can capitalize. Always-On AI – Chad works around the clock, monitoring X/Twitter, Telegram, Instagram, and more to maximize exposure. Predictive Trend Analysis – Spot what’s about to go viral before the rest of the market catches on. Automated Growth – Hands-free content and engagement across major social platforms. No Technical Barriers – AI handles the heavy lifting, so anyone can capitalize. $MA: The Fuel Behind the Memeagent Ecosystem Unlike other meme coins, $MA goes beyond speculation—it powers the entire Memeagent platform, unlocking a range of utilities that set it apart. AI agent Creation – Use $MA to create and deploy custom AI-powered Memeagents that generate viral content and interact across social platforms. DeFi Utilities – Stake $MA for passive income and participate in governance decisions to shape the future of the Memeagent ecosystem. Deflationary Impact – A portion of every $MA transaction is burned, reducing supply and increasing scarcity over time. AI agent Creation – Use $MA to create and deploy custom AI-powered Memeagents that generate viral content and interact across social platforms. DeFi Utilities – Stake $MA for passive income and participate in governance decisions to shape the future of the Memeagent ecosystem. Deflationary Impact – A portion of every $MA transaction is burned, reducing supply and increasing scarcity over time. Memeagent and Chad Gain Traction as AI-Powered Platforms Expand According to the team, Memeagent isn’t another speculative meme coin—it’s a structured, AI-powered ecosystem with long-term incentives. The Features That Set Memeagent Apart AI-Driven Growth – Automation supports long-term engagement beyond short-term trends. $MA Token Utility – Powers AI tools, governance, staking, and a deflationary tokenomics model. Security & Infrastructure – Built with SOC 2 Type 2-certified security for reliability and trust. AI-Driven Growth – Automation supports long-term engagement beyond short-term trends. $MA Token Utility – Powers AI tools, governance, staking, and a deflationary tokenomics model. Security & Infrastructure – Built with SOC 2 Type 2-certified security for reliability and trust. Memeagent’s Roadmap is Packed Memeagent is gaining attention as its ecosystem expands. With a structured roadmap ahead and continuous development, the platform is set for steady growth. Upcoming milestones: Q2 2025: Beta AI automation tools roll out to early users. Q3 2025: Advanced predictive analytics and expanded AI engagement. Q4 2025: Full staking launch, multilingual support, and deeper integrations. 2026: Memeagent’s ecosystem expands to full-scale adoption. Q2 2025: Beta AI automation tools roll out to early users. Q3 2025: Advanced predictive analytics and expanded AI engagement. Q4 2025: Full staking launch, multilingual support, and deeper integrations. 2026: Memeagent’s ecosystem expands to full-scale adoption. Market Movers Are Taking Notice Memeagent is already in motion, leveraging AI and automation to drive engagement. As the ecosystem expands, those recognizing its potential are positioning themselves early. The Next Evolution of Memecoins Is Built on Innovation, Not Just Hype. AI-Powered Innovation in Memecoins Memeagent is introducing AI-driven engagement to the memecoin ecosystem. As the platform grows, early users can explore $MA’s utility and experience Chad’s AI-powered automation in action. Users can visit www.memeagent.io to get started now. www.memeagent.io www.memeagent.io #CHADISCOMING About Memeagent Memeagent is developed by a team of experienced AI engineers and blockchain specialists with extensive backgrounds in DeFi, yield farming, token design, and AI-driven automation. Active in the crypto space since 2016, the team has contributed to various high-profile projects, focusing on scalability, security, and decentralized AI (DeAI) solutions. Memeagent Memeagent Contact Benjamin Peng info@memeagent.io This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here here here

Memeagent Is Gaining Attention – Meeting Chad, The AI Agent That’s Changing The Game

Memeagent está ganando atención: conozca a Chad, el agente de inteligencia artificial que está cambiando las reglas del juego

Memeagent está a chamar a atención: coñece a Chad, o axente de IA que está a cambiar o xogo

Memeagent privlači pažnju – upoznajte Chada, AI agenta koji mijenja igru

Memeagent が注目を集める – ゲームを変える AI エージェント Chad との出会い

Memeagent добива внимание - средба со Чад, агентот за вештачка интелигенција што ја менува играта

Memeagent E Hwetša Tlhokomelo – Go Kopana le Chad, Moemedi wa AI Yo a Fetošago Papadi

Memeagent atrage atenția – Întâlnesc cu Chad, agentul AI care schimbă jocul

Memeagent таваҷҷӯҳро ба худ ҷалб мекунад - Вохӯрӣ бо Чад, агенти AI, ки бозиро тағир медиҳад

Memeagent Is Gaining Attention – Meeting Chad, Ang AI Agent That’s Changing The Game

Memeagent Dikkat Çekiyor – Oyunu Değiştiren Yapay Zeka Ajanı Chad ile Tanışma

Memeagent Yi Kuma Nyingiso – Ku Hlangana na Chad, Muyimeri wa AI Loyi A Cincaka Ntlangu

Memeagent đang thu hút sự chú ý – Gặp gỡ Chad, đặc vụ AI đang thay đổi trò chơi

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques D. Multi-Group Attention Family D.1. Detailed Analysis on Memory Access We show in Table 4 that the memory IO cost for ⟨q, K⟩ is dominated by the loading of K which costs bmhk in the case of multihead where g = h. This cost is particularly high due to the coupling of batch size b, context length m, and the entire hidden dimension d. Compared to the number of computations, which has complexity bmd, this attention module requires one memory IO per one tensor operation (memory-io bound). In contrast, other operations such as feedforw can be the main bottleneck for incremental decoding and our paper aims to tackle such problems.ard has much lower ratio of memory IO per compute (compute bound). These attention computation D.2. Model FLOPs The scaling laws by Kaplan et al. (2020) shows that the modelrelated FLOPs during the forward pass is 2N where N is the number of parameters (without the embeddings). We show that it holds for a general multi-group model as well. The only difference between the multi-group and the multi-head case is the projection PK and PV where they are of size dgk instead of dhk. Since this is a linear layer, the forward pass FLOPs for any input is still proportional such projection size. Therefore, it follows that for any multi-group attention, including multi-head, the forward FLOPs is 2N where N is the respective number of parameters. D.3. Comparing Capabilities-Equivalent Models This section outlines the analysis of latency change when we switch from an MH model to an MG model with F times the size. D.3.1. CONTEXT ENCODING D.3.2. INCREMENTAL DECODING This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques D. Multi-Group Attention Family D.1. Detailed Analysis on Memory Access We show in Table 4 that the memory IO cost for ⟨q, K⟩ is dominated by the loading of K which costs bmhk in the case of multihead where g = h. This cost is particularly high due to the coupling of batch size b, context length m, and the entire hidden dimension d. Compared to the number of computations, which has complexity bmd, this attention module requires one memory IO per one tensor operation (memory-io bound). In contrast, other operations such as feedforw can be the main bottleneck for incremental decoding and our paper aims to tackle such problems.ard has much lower ratio of memory IO per compute (compute bound). These attention computation D.2. Model FLOPs The scaling laws by Kaplan et al. (2020) shows that the modelrelated FLOPs during the forward pass is 2N where N is the number of parameters (without the embeddings). We show that it holds for a general multi-group model as well. The only difference between the multi-group and the multi-head case is the projection PK and PV where they are of size dgk instead of dhk. Since this is a linear layer, the forward pass FLOPs for any input is still proportional such projection size. Therefore, it follows that for any multi-group attention, including multi-head, the forward FLOPs is 2N where N is the respective number of parameters. D.3. Comparing Capabilities-Equivalent Models This section outlines the analysis of latency change when we switch from an MH model to an MG model with F times the size. D.3.1. CONTEXT ENCODING D.3.1. CONTEXT ENCODING D.3.2. INCREMENTAL DECODING D.3.2. INCREMENTAL DECODING This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Understanding Multi-Group Attention in AI Models

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques A. FAQs Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary?
A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts.

Q: How applicable is multi-query for single-batch inference without high batch sampling?
A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1.

Q: Is bifurcated attention applicable for the case where we process different inputs in a batch?
A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper.

Q: Any caveats to using bifurcated attention?
A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency.

Q: How does model quantization (or lower precision arithmetic) affect the findings?
A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same.

Q: Does the conclusion depend on the inference implementation or different hardware?
A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis.

Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques A. FAQs Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary?
A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts. Q: How applicable is multi-query for single-batch inference without high batch sampling?
A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1. Q: Is bifurcated attention applicable for the case where we process different inputs in a batch?
A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper. Q: Any caveats to using bifurcated attention?
A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency. Q: How does model quantization (or lower precision arithmetic) affect the findings?
A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same. Q: Does the conclusion depend on the inference implementation or different hardware?
A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis. Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary? A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts. Q: If we already have an MQ model that seems to be quite efficient at large batch sampling, is bifurcated attention necessary? Q: A: The proposed context-aware bifurcated attention is an exact computation that provides a different way to perform attention, so one can use it "for free" without a performance tradeoff. Due to the reduced memory I/O, it enables more extreme cases of batch sampling, such as a larger batch, even for long contexts. A: Q: How applicable is multi-query for single-batch inference without high batch sampling? A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1. Q: How applicable is multi-query for single-batch inference without high batch sampling? Q: A: If the context is long and the number of generated tokens is high, then the benefits of multi-query are clear. Please see Section 5.2.1. A: Q: Is bifurcated attention applicable for the case where we process different inputs in a batch? A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper. Q: Is bifurcated attention applicable for the case where we process different inputs in a batch? Q: A: No. In that case, if we need a solution to reduce memory I/O during incremental decoding, then multi-query attention can be appealing, especially in scenarios with a high number of generated tokens where the incremental decoding phase dominates the overall latency. This is because there is an overhead to multi-query due to the context encoding phase, as outlined in the main paper. A: Q: Any caveats to using bifurcated attention? A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency. Q : Any caveats to using bifurcated attention? Q A: For small workloads (low context length and batch size), due to the fact that we split the attention into two parts, there can be less parallelization of the GEMM kernels, which could lead to higher latency, especially for MQ models. However, one can get the best of both worlds given any model by triggering bifurcated attention under high workload scenarios and using normal attention otherwise. With such a workload-based switch, bifurcated attention is guaranteed to provide better latency and efficiency. A: Q: How does model quantization (or lower precision arithmetic) affect the findings? A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same. Q: How does model quantization (or lower precision arithmetic) affect the findings? Q: A: There are two regimes for quantization: model weight quantization and attention quantization. To date, most quantization only focuses on the weight since the attention computation is precision-sensitive and quantization has not proved to be viable. Model quantization can make incremental decoding faster due to lower memory I/O of the model itself, since the effective model size in memory is smaller. This shifts the latency curve downward for all context lengths or batch sizes. The overall conclusion for the bifurcated and multi-query attention remains the same, however, since the improvement proposed in the paper is on the attention component, which is orthogonal to the model weight. If attention quantization is viable in the future, the lower memory on the attention tensor will effectively reduce the memory I/O for KV cache by a factor of 2 in the case of int8 quantization (compared to fp16 or bf16) or a factor of 4 in the case of int4. Overall, this will flatten the latency growth with respect to batch size or context length. The overall comparative complexity (a) with or without bifurcated attention or (b) multi-head vs. multi-query remains the same. A: Q: Does the conclusion depend on the inference implementation or different hardware? A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis. Q: Does the conclusion depend on the inference implementation or different hardware? A: Different inference platforms, such as FasterTransformers (GPUs) or PaLM inference (TPUs), can yield different latency numbers. However, the relative I/O complexity among different attention mechanisms does not change, resulting in similar relative trends among different attention mechanisms. That being said, it is possible that more efficient implementations or more performant chip/system configurations, including different tensor parallelism degrees, can result in different slopes for the latency growth with respect to context length and batch size. In that case, the trade-off points in terms of context length or batch size can be different. The comparative complexity remains the same based on the analysis. Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? Q: How does bifurcated attention differ from using attention mask for sampling as in done in SpecInfer (Miao et al., 2023)? Q: This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

FAQs: How Bifurcated Attention Improves AI Model Efficiency

Faster AI Code Generation with Bifurcated Attention

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques 5. Experiments We first conduct experiments to see how capabilities scale with respect to model size for each attention type in Section 5.1. We find that attention types with higher compression (lower number of attention groups g) require model size compensation, ≈ 10% for multi-query (g = 1). We use such findings to compare the latency between the multi-head and the larger multi-query models of equal capabilities in Section 5.2. In Section 5.2.2, we focus on the single-context batch sampling scenario where we demonstrate the significant latency reduction of bifurcated attention and revisit the comparison between multi-head and multi-query in light of bifurcated attention. We outline inference details in Appendix C.5. 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention To compare the capabilities of different attention mechanisms, one can either scale other model configurations such as the number of layers ℓ, the number of heads h in order to make match the total model sizes between different attentions. However, it is often difficult to match the number of parameters exactly. In this work, we compare different attention mechanisms via the loss-vs-size scaling laws. For the setup, we use the model hyperparameters similar to that of GPT-3, where the size ranges from 125M to 13B, with hyperparameters such as ℓ, h, k increasing in tandem. Then, we consider three cases where g = 1 (multi-query), g = h (multi-head) and 1 < g < h (multi-group) where Appendix C.1 and C.2 shows the training and model configuration details. We train all three attention models of each size and plot the validation loss versus model size, shown in Figure 3. Our findings are summarized below. Higher number of attention groups g leads to higher expressiveness The results in Figure 3 shows the validation loss versus model size (log scale). The results indicate that, for the same model size (vertical slice across the plot), multi-head attention g = h achieves the lowest validation loss compared to 1 < g < h (multi-group) and g = 1 (multiquery). This trend holds consistently over three orders of magnitude of model sizes, where the curves corresponding to multi-head, multi-group and multi-query do not cross, implying that the rank of model expressiveness, or relative capabilities per number of parameters, is quite stable. An intuitive explanation is that the lower g corresponds to a lower rank representation of the key and value tensors, which encodes lower representation power of the past context and therefore yields lower capabilities than higher g, given the same model size. Scaling laws via downstream performance We use the average scores from two code generation benchmarks, multilingual HumanEval and MBXP (Athiwaratkun et al., 2022), as a proxy for model capabilities in addition to the validation loss. This approach is similar to that of the GPT-4 technical report (OpenAI, 2023) where HumanEval (Python) (Chen et al., 2021) is used to track the performance across multiple magnitudes of compute. In our case, we average across all 13 evaluation languages and two benchmarks to obtain a more stable proxy for capabilities. The result in Figure 3 demonstrates similar trend compared to the validation loss where the pass rate curves indicate the same relative expressiveness for multi-head, multi-group and multi-query attention. Matching capabilities by model size compensation Given the same capabilities (horizontal slice of the plot in Figure 3), the distance between two curves indicates the model size difference that the lower-rank attention needs to compensate in order to match the multi-head model performance. Empirically, we average the distance along the interpolated lines (log scale) and find this to correspond to 1.104 times; that is, a multi-query model can have the same capabilities as the multi-head model if the size is increased by ≈ 10% of the multi-head model size. Similarly, the gap is < 10% for multi-group attention. Alternatively, one can argue that a multi-query model of the same size could match a multi-head if the multi-query model is given more compute. However, in the regime where we train language models until or close to convergence and the performance saturates with respect to compute, the difference in capabilities will likely remain. Therefore, the size compensation is likely the most fair approach for comparison. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques 5. Experiments We first conduct experiments to see how capabilities scale with respect to model size for each attention type in Section 5.1. We find that attention types with higher compression (lower number of attention groups g) require model size compensation, ≈ 10% for multi-query (g = 1). We use such findings to compare the latency between the multi-head and the larger multi-query models of equal capabilities in Section 5.2. In Section 5.2.2, we focus on the single-context batch sampling scenario where we demonstrate the significant latency reduction of bifurcated attention and revisit the comparison between multi-head and multi-query in light of bifurcated attention. We outline inference details in Appendix C.5. 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention To compare the capabilities of different attention mechanisms, one can either scale other model configurations such as the number of layers ℓ, the number of heads h in order to make match the total model sizes between different attentions. However, it is often difficult to match the number of parameters exactly. In this work, we compare different attention mechanisms via the loss-vs-size scaling laws. For the setup, we use the model hyperparameters similar to that of GPT-3, where the size ranges from 125M to 13B, with hyperparameters such as ℓ, h, k increasing in tandem. Then, we consider three cases where g = 1 (multi-query), g = h (multi-head) and 1 < g < h (multi-group) where Appendix C.1 and C.2 shows the training and model configuration details. We train all three attention models of each size and plot the validation loss versus model size, shown in Figure 3. Our findings are summarized below. Higher number of attention groups g leads to higher expressiveness The results in Figure 3 shows the validation loss versus model size (log scale). The results indicate that, for the same model size (vertical slice across the plot), multi-head attention g = h achieves the lowest validation loss compared to 1 < g < h (multi-group) and g = 1 (multiquery). This trend holds consistently over three orders of magnitude of model sizes, where the curves corresponding to multi-head, multi-group and multi-query do not cross, implying that the rank of model expressiveness, or relative capabilities per number of parameters, is quite stable. An intuitive explanation is that the lower g corresponds to a lower rank representation of the key and value tensors, which encodes lower representation power of the past context and therefore yields lower capabilities than higher g, given the same model size. Higher number of attention groups leads to higher expressiveness Scaling laws via downstream performance We use the average scores from two code generation benchmarks, multilingual HumanEval and MBXP (Athiwaratkun et al., 2022), as a proxy for model capabilities in addition to the validation loss. This approach is similar to that of the GPT-4 technical report (OpenAI, 2023) where HumanEval (Python) (Chen et al., 2021) is used to track the performance across multiple magnitudes of compute. In our case, we average across all 13 evaluation languages and two benchmarks to obtain a more stable proxy for capabilities. The result in Figure 3 demonstrates similar trend compared to the validation loss where the pass rate curves indicate the same relative expressiveness for multi-head, multi-group and multi-query attention. Scaling laws via downstream performance Matching capabilities by model size compensation Given the same capabilities (horizontal slice of the plot in Figure 3), the distance between two curves indicates the model size difference that the lower-rank attention needs to compensate in order to match the multi-head model performance. Empirically, we average the distance along the interpolated lines (log scale) and find this to correspond to 1.104 times; that is, a multi-query model can have the same capabilities as the multi-head model if the size is increased Matching capabilities by model size compensation by ≈ 10% of the multi-head model size. Similarly, the gap is < 10% for multi-group attention. Alternatively, one can argue that a multi-query model of the same size could match a multi-head if the multi-query model is given more compute. However, in the regime where we train language models until or close to convergence and the performance saturates with respect to compute, the difference in capabilities will likely remain. Therefore, the size compensation is likely the most fair approach for comparison. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

A Little Less Memory, A Lot More Speed: The Science of Bifurcated Attention

Authors:
(1) Ben Athiwaratkun, AWS AI Labs;
(2) Sujan Kumar Gonugondla, AWS AI Labs;
(3) Sanjay Krishna Gouda, AWS AI Labs;
(4) Haifeng Qian, AWS AI Labs;
(5) Sanjay Krishna Gouda, AWS AI Labs;
(6) Hantian Ding, AWS AI Labs;
(7) Qing Sun, AWS AI Labs;
(8) Jun Wang, AWS AI Labs;
(9) Jiacheng Guo, AWS AI Labs;
(10 Liangfu Chen, AWS AI Labs;
(11) Parminder Bhatia, GE HealthCare (work done at AWS);
(12) Ramesh Nallapati, Amazon AGI (work done at AWS);
(13) Sudipta Sengupta, AWS AI Labs;
(14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction 2. Related Work 3. Background 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 6. Conclusion and References A. FAQs B. Related Work C. Setup D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention Multi-query attention, proposed by Shazeer (2019), is an attention mechanism for transformers models that uses a single head for the key and value tensors, compared to h heads in the traditional multi-head attention (Vaswani et al., 2017). This technique effectively reduces the KV memory IO by h times, which leads to higher inference efficiency during incremental decoding. In effect, the single-head key or value tensor is shared and used to attend to all the multi-head query, hence the name multi-query. This corresponds to a compression in representation power of the key and value tensor, which we will see in the scaling laws study (Section 5.1) that it results in a reduced expressiveness in terms of model parameter efficiency. Such reduced expressiveness can be compensated by scaling the model bigger than the multi-head counterpart to match the representation power. We can also extrapolate these insights to a generalized multiquery attention mechanism (Ainslie et al., 2023), which provides a framework to understand both multi-query and multi-head attention, and everything in between. Here, the degree of KV compression is dictated by the number of attention groups g, where we alternatively refer to the generalized multi-query as multi-group. Each attention group can be interpreted as the broadcasted attention between a single head of key or value tensor, and multiple heads of query. In this paradigm, multi-query attention is a special case where the number of groups g = 1; that is, there is exactly one such group. Conversely, multi-head attention is another special case where the number of attention groups matches the number of heads (g = h), in which case each head in the key or value tensor attends to one head in the query. More generally, the number of groups g can lie anywhere between 1 and h, indicating various degrees of compression. For practical purposes, it is most convenient when g divides h. The attention mechanism in this setting can be expressed in terms of Einstein summation as: logits = ⟨q, K⟩ : einsum(bgpnk, bgmk) → bgpnm (1) o = ⟨w, V ⟩ : einsum(bgpmn, bgmv) → bgpnv (2) This generalized multi-group attention mechanism thus provides a unified perspective on the design space of attention architectures. By adjusting the number of attention groups g, one can flexibly tune these trade-offs, potentially yielding new regimes of performance for transformer models. In Section 5.1, we will look into such capability vs latency trade-off. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Authors: Authors: (1) Ben Athiwaratkun, AWS AI Labs; (2) Sujan Kumar Gonugondla, AWS AI Labs; (3) Sanjay Krishna Gouda, AWS AI Labs; (4) Haifeng Qian, AWS AI Labs; (5) Sanjay Krishna Gouda, AWS AI Labs; (6) Hantian Ding, AWS AI Labs; (7) Qing Sun, AWS AI Labs; (8) Jun Wang, AWS AI Labs; (9) Jiacheng Guo, AWS AI Labs; (10 Liangfu Chen, AWS AI Labs; (11) Parminder Bhatia, GE HealthCare (work done at AWS); (12) Ramesh Nallapati, Amazon AGI (work done at AWS); (13) Sudipta Sengupta, AWS AI Labs; (14) Bing Xiang, Goldman Sachs (work done at AWS). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Background 3. Background 3.1. Notation and 3.2. Language Model Inference 3.1. Notation and 3.2. Language Model Inference 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4. Context-Aware Bifurcated Attention and 4.1. Motivation 4.2. Formulation and 4.3. Memory IO Complexity 4.2. Formulation and 4.3. Memory IO Complexity 5. Experiments 5. Experiments 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.1. Comparing Capabilities of Multi-Head, Multi-Query, and Multi-Group Attention 5.2. Latencies of Capabilities-Equivalent Models 5.2. Latencies of Capabilities-Equivalent Models 5.3. Applications 5.3. Applications 6. Conclusion and References 6. Conclusion and References A. FAQs A. FAQs B. Related Work B. Related Work C. Setup C. Setup D. Multi-Group Attention Family D. Multi-Group Attention Family E. Context-Aware Bifurcated Attention E. Context-Aware Bifurcated Attention F. Applications: Additional Results F. Applications: Additional Results G. Compatibility with Speculative Decoding and Fast Decoding techniques G. Compatibility with Speculative Decoding and Fast Decoding techniques 3.3. Multi-Query, Multi-Head and the Generalized Multi-Query Attention Multi-query attention, proposed by Shazeer (2019), is an attention mechanism for transformers models that uses a single head for the key and value tensors, compared to h heads in the traditional multi-head attention (Vaswani et al., 2017). This technique effectively reduces the KV memory IO by h times, which leads to higher inference efficiency during incremental decoding. In effect, the single-head key or value tensor is shared and used to attend to all the multi-head query, hence the name multi-query. This corresponds to a compression in representation power of the key and value tensor, which we will see in the scaling laws study (Section 5.1) that it results in a reduced expressiveness in terms of model parameter efficiency. Such reduced expressiveness can be compensated by scaling the model bigger than the multi-head counterpart to match the representation power. We can also extrapolate these insights to a generalized multiquery attention mechanism (Ainslie et al., 2023), which provides a framework to understand both multi-query and multi-head attention, and everything in between. Here, the degree of KV compression is dictated by the number of attention groups g, where we alternatively refer to the generalized multi-query as multi-group. Each attention group can be interpreted as the broadcasted attention between a single head of key or value tensor, and multiple heads of query. In this paradigm, multi-query attention is a special case where the number of groups g = 1; that is, there is exactly one such group. Conversely, multi-head attention is another special case where the number of attention groups matches the number of heads (g = h), in which case each head in the key or value tensor attends to one head in the query. More generally, the number of groups g can lie anywhere between 1 and h, indicating various degrees of compression. For practical purposes, it is most convenient when g divides h. The attention mechanism in this setting can be expressed in terms of Einstein summation as: logits = ⟨q, K⟩ : einsum(bgpnk, bgmk) → bgpnm (1) o = ⟨w, V ⟩ : einsum(bgpmn, bgmv) → bgpnv (2) This generalized multi-group attention mechanism thus provides a unified perspective on the design space of attention architectures. By adjusting the number of attention groups g, one can flexibly tune these trade-offs, potentially yielding new regimes of performance for transformer models. In Section 5.1, we will look into such capability vs latency trade-off. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Why Multi-Query Attention Matters for Large Language Models

**LUXEMBOURG, Luxembourg, February 11th, 2025/CyberNewsWire/--**Gcore, the global edge AI, cloud, network, and security solutions provider, today announced the findings of its Q3-Q4 2024 Radar report into DDoS attack trends. DDoS attacks have reached unprecedented scale and disruption in 2024, and businesses need to act fast to protect themselves from this evolving threat. The report reveals a significant escalation in the total number of DDoS attacks and their magnitude, measured in terabits per second (Tbps). Key highlights from Q3-Q4 2024 ●    Compared to Q3–Q4 2023, the number of DDoS attacks have risen by 56%, which highlights a steep long-term growth trend. ●    The gaming industry continues to be the most targeted by DDoS attacks, accounting for 34% of all attacks. ●    In Q3-Q4 2024, the financial services sector experienced a significant increase, accounting for 26% of all DDoS attacks, up from 12% in the previous period. ●    There was a 17% increase in the total number of attacks compared with Q1-Q2 2024. ●    The largest attack peaked at 2Tbps in Q3-Q4 2024, which is an 18% increase from Q1-Q2 2024. ●    DDoS attacks are becoming shorter in duration but more powerful. Attackers Are Shifting Their Focus The sectors that were targeted in Q3-Q4 2024 reflect a changing focus among DDoS attackers. The technology industry has seen a steady increase in its share of DDoS attacks, increasing from 7% to 19% since Q3-Q4 2023. This is because DDoS attackers recognise the wide-reaching disruption potential of attacking technology services. A single successful attack can take out a service that countless organizations depend on - causing significant harm to people and businesses. Another reason that technology platforms have seen an increase in DDoS attacks is due to their vast computational power, which malicious actors can exploit to intensify their attacks. The gaming industry continues to be the most-attacked industry, although there were 31% fewer attacks compared with Q1-Q2 2024. The decline in attacks may be attributed to several factors. For instance, gaming companies are strengthening their DDoS defenses in response to ongoing attacks, which may result in fewer successful attacks. Another explanation is that attackers may be shifting their focus towards other high-value sectors, such as financial services, which saw a 117% increase in the number of attacks. The sector’s critical online services and susceptibility to ransom-based attacks make it a prime target. Andrey Slastenov, Head of Security at Gcore, commented: “The latest Gcore Radar should be a wake-up call to businesses across all industries. Not only is the number and intensity of attacks increasing, but attackers are expanding the scope of their attacks to reach an increasingly wide range of sectors. Businesses must invest in robust DDoS detection, mitigation, and protection to prevent the financial and reputational impact of an attack.” The Geographical Distribution of DDoS attacks With a presence that spans six continents, Gcore can accurately track the geographical sources of DDoS attacks. Gcore derives these insights from the attackers’ IP addresses and the geographic locations of the data centers where malicious traffic is targeted. Gcore’s findings have highlighted the Netherlands as a key source of attacks; leading application-layer attacks with 21% and ranking second for network-layer attacks at 18%. The U.S. ranked highly across both layers, reflecting its vast internet infrastructure for hackers to exploit. Brazil featured prominently in network-layer attacks at 14%. Brazil’s growing digital economy and connectivity make it an emerging source of attacks. China and Indonesia also featured prominently, with Indonesia showing a growth in application-layer attacks at 8%, which reflects a broader trend of increased attack activity in Southeast Asia. Short But Potent Attacks Continue To Take Hold DDoS attacks are becoming shorter in duration, but no less disruptive. The longest DDoS attack duration during Q3-Q4 2024 was five hours, which is a significant decrease from 16 hours in the first half of the year. This is reflective of an increasing trend towards shorter but more intense attacks. These ‘burst attacks’ can be more difficult to detect as they may blend in with normal traffic spikes. The delay in detection gives attackers a window of opportunity to disrupt services before cyber defenses can kick in. The trend of shorter DDoS attack durations can in part be attributed to improvements in cybersecurity. As security tightens, attackers have learned to adapt with short burst attacks designed to bypass defenses. A short DDoS attack can also double as a smokescreen to conceal a secondary attack, such as ransomware deployment. To access the full report, users can visit https://gcore.com/library/gcore-radar-ddos-attack-trends-q3-q4-2024 About Gcore Gcore is a global edge AI, cloud, network, and security solutions provider. Headquartered in Luxembourg, with a team of 600 operating from ten offices worldwide, Gcore provides solutions to global leaders in numerous industries. Gcore manages its global IT infrastructure across six continents, with one of the best network performances in Europe, Africa, and LATAM due to the average response time of 30 ms worldwide. Gcore’s network consists of 180 points of presence worldwide in reliable Tier IV and Tier III data centers, with a total network capacity exceeding 200 Tbps. Users can learn more at gcore.com or follow them on LinkedIn, Twitter, and Facebook. Contact Gcore press contact pr@gcore.com This story was distributed as a release by Cybernewswire under HackerNoon’s Business Blogging Program. Learn more about the program here **LUXEMBOURG, Luxembourg, February 11th, 2025/CyberNewsWire/--**Gcore, the global edge AI, cloud, network, and security solutions provider, today announced the findings of its Q3-Q4 2024 Radar report into DDoS attack trends. DDoS attacks have reached unprecedented scale and disruption in 2024, and businesses need to act fast to protect themselves from this evolving threat. The report reveals a significant escalation in the total number of DDoS attacks and their magnitude, measured in terabits per second (Tbps). Key highlights from Q3-Q4 2024 ●    Compared to Q3–Q4 2023, the number of DDoS attacks have risen by 56%, which highlights a steep long-term growth trend. ●    The gaming industry continues to be the most targeted by DDoS attacks, accounting for 34% of all attacks. ●    In Q3-Q4 2024, the financial services sector experienced a significant increase, accounting for 26% of all DDoS attacks, up from 12% in the previous period. ●    There was a 17% increase in the total number of attacks compared with Q1-Q2 2024. ●    The largest attack peaked at 2Tbps in Q3-Q4 2024, which is an 18% increase from Q1-Q2 2024. ●    DDoS attacks are becoming shorter in duration but more powerful. Attackers Are Shifting Their Focus The sectors that were targeted in Q3-Q4 2024 reflect a changing focus among DDoS attackers. The technology industry has seen a steady increase in its share of DDoS attacks, increasing from 7% to 19% since Q3-Q4 2023. This is because DDoS attackers recognise the wide-reaching disruption potential of attacking technology services. A single successful attack can take out a service that countless organizations depend on - causing significant harm to people and businesses. Another reason that technology platforms have seen an increase in DDoS attacks is due to their vast computational power, which malicious actors can exploit to intensify their attacks. The gaming industry continues to be the most-attacked industry, although there were 31% fewer attacks compared with Q1-Q2 2024. The decline in attacks may be attributed to several factors. For instance, gaming companies are strengthening their DDoS defenses in response to ongoing attacks, which may result in fewer successful attacks. Another explanation is that attackers may be shifting their focus towards other high-value sectors, such as financial services, which saw a 117% increase in the number of attacks. The sector’s critical online services and susceptibility to ransom-based attacks make it a prime target. Andrey Slastenov, Head of Security at Gcore, commented: “The latest Gcore Radar should be a wake-up call to businesses across all industries. Not only is the number and intensity of attacks increasing, but attackers are expanding the scope of their attacks to reach an increasingly wide range of sectors. Businesses must invest in robust DDoS detection, mitigation, and protection to prevent the financial and reputational impact of an attack.” Andrey Slastenov, Head of Security at Gcore, commented: “The latest Gcore Radar should be a wake-up call to businesses across all industries. Not only is the number and intensity of attacks increasing, but attackers are expanding the scope of their attacks to reach an increasingly wide range of sectors. Businesses must invest in robust DDoS detection, mitigation, and protection to prevent the financial and reputational impact of an attack.” The Geographical Distribution of DDoS attacks With a presence that spans six continents, Gcore can accurately track the geographical sources of DDoS attacks. Gcore derives these insights from the attackers’ IP addresses and the geographic locations of the data centers where malicious traffic is targeted. Gcore’s findings have highlighted the Netherlands as a key source of attacks; leading application-layer attacks with 21% and ranking second for network-layer attacks at 18%. The U.S. ranked highly across both layers, reflecting its vast internet infrastructure for hackers to exploit. Brazil featured prominently in network-layer attacks at 14%. Brazil’s growing digital economy and connectivity make it an emerging source of attacks. China and Indonesia also featured prominently, with Indonesia showing a growth in application-layer attacks at 8%, which reflects a broader trend of increased attack activity in Southeast Asia. Short But Potent Attacks Continue To Take Hold DDoS attacks are becoming shorter in duration, but no less disruptive. The longest DDoS attack duration during Q3-Q4 2024 was five hours, which is a significant decrease from 16 hours in the first half of the year. This is reflective of an increasing trend towards shorter but more intense attacks. These ‘burst attacks’ can be more difficult to detect as they may blend in with normal traffic spikes. The delay in detection gives attackers a window of opportunity to disrupt services before cyber defenses can kick in. The trend of shorter DDoS attack durations can in part be attributed to improvements in cybersecurity. As security tightens, attackers have learned to adapt with short burst attacks designed to bypass defenses. A short DDoS attack can also double as a smokescreen to conceal a secondary attack, such as ransomware deployment. To access the full report, users can visit https://gcore.com/library/gcore-radar-ddos-attack-trends-q3-q4-2024 https://gcore.com/library/gcore-radar-ddos-attack-trends-q3-q4-2024 https://gcore.com/library/gcore-radar-ddos-attack-trends-q3-q4-2024 About Gcore Gcore is a global edge AI, cloud, network, and security solutions provider. Headquartered in Luxembourg, with a team of 600 operating from ten offices worldwide, Gcore provides solutions to global leaders in numerous industries. Gcore manages its global IT infrastructure across six continents, with one of the best network performances in Europe, Africa, and LATAM due to the average response time of 30 ms worldwide. Gcore Gcore Gcore’s network consists of 180 points of presence worldwide in reliable Tier IV and Tier III data centers, with a total network capacity exceeding 200 Tbps. Users can learn more at gcore.com or follow them on LinkedIn , Twitter , and Facebook . gcore.com gcore.com LinkedIn LinkedIn Twitter Twitter Facebook Facebook Contact Gcore press contact pr@gcore.com This story was distributed as a release by Cybernewswire under HackerNoon’s Business Blogging Program. Learn more about the program here This story was distributed as a release by Cybernewswire under HackerNoon’s Business Blogging Program. Learn more about the program here here here

Gcore Radar Report Reveals 56% Year-on-Year Increase In DDoS Attacks

El informe de Gcore Radar revela un aumento interanual del 56 % en los ataques DDoS

דוח Gcore Radar חושף עלייה של 56% בהתקפות DDoS משנה לשנה

Gcore レーダーレポート、DDoS 攻撃が前年比 56% 増加

Gcore Radar есебі DDoS шабуылдарының 56% өскенін көрсетеді

Извештајот за радар на Gcore открива 56% зголемување на DDoS нападите од година во година

Gcore радарын тайланд DDoS халдлага жилээс жилд 56%-иар өссөн байна.

Gcore Radar-rapport onthult 56% jaarlijkse toename van DDoS-aanvallen

د Gcore رادار راپور د DDoS بریدونو کې د کال په پرتله ۵۶٪ زیاتوالی ښیي

Relatório do Gcore Radar revela aumento de 56% em ataques DDoS ano a ano

Raportul Gcore Radar dezvăluie o creștere de 56% de la an la an a atacurilor DDoS

Gcore Radar Chirevo Chinoratidza 56% Gore-pa-Gore Kuwedzera muDDoS Kurwiswa

Гцоре Радар извештај открива 56% пораст ДДоС напада у односу на претходну годину

Table of Links Part 1: Abstract & Introduction Part 2: Background Part 3: Attacks & Countermeasures Part 4: Experimental Setup Part 5: Datasets & Evaluation Part 6: Attack, Countermeasure Parameters, & Baseline: Random Perturbations Part 7: Results & Discussion Part 8: Transfer Attacks & Countermeasures Part 9: Conclusion, Limitations, & Ethics Statement Part 10: Appendix: Audio Encoder Pre-training & Evaluation Part 11: Appendix: Cross-prompt attacks, Training Data Ablations, & Impact of random noise on helpfulness Part 12: Appendix: Adaptive attacks & Qualitative Examples A.6 Adaptive attacks In this section, we report the results using adaptive attacks, where the attacker has knowledge of any defense mechanism employed in the system. We employ α=0.0001 (Eq. 1) as we found that the attacker needs a larger step size in the presence of a defense to produce successful attacks. From Table 10, we see that the attacks become less successful in the presence of a defense. Also, the adaptive attacker needs to add much more perceptible perturbations (lower average SPR) in the presence of a defense. This clearly shows that a simple pre-processing defense can provide some degree of robustness against adversarial attacks. Also, from Figure 5, we observe that the presence of a defense in the system makes the attacks less effective under limited attack budgets. For a given attack budget of say T=50 iterations, only 60% of the attacks are successful on the system with TDNF defense, compared to ∼80% for a system without defense. However, note that these attacks were performed with a limited attack budget of T=100 iterations. A malicious actor with a larger attack budget can potentially produce a higher jailbreak rate. A.7 Qualitative Examples Table 11 compares an in-house and a public SLM model responses on harmful examples. We showcase scenarios where models produce safe content although irrelevant, as well as safe content with relevant understanding of the input audio. Overall, the in-house SLM model demonstrate better speech comprehension ability. Table 12 compares models on various helpfulness questions across different aspects of usefulness. We notice that the in-house SLM model SMistral-FT sometimes errs on the side of caution, indicating a healthy tension between harmlessness and helpfulness. We leave further explorations of such properties of SLM models to future work. On the other hand, we notice the importance of a strong audio understanding ability in an SLM, as failing to do so can impact the usefulness of an SLM model by mistaking entity names in the input audio. Table 13 showcases examples of jailbroken responses and the corresponding SPRs. We clearly see that the model produces safe responses adhering to its safety training without attack, but even minimal perturbations can cause it to produce unsafe responses. In some cases (last 2 examples), the model begins its response with a safety aligned response, but generates harmful content subsequently. This further demonstrates the need for thorough studies on model safety, and a cursory analysis may be insufficient. Authors:
(1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com);
(2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions;
(3) Srikanth Ronanki, AWS AI Labs, Amazon;
(4) Anshu Bhatia, AWS AI Labs, Amazon;
(5) Karel Mundnich, AWS AI Labs, Amazon;
(6) Saket Dingliwal, AWS AI Labs, Amazon;
(7) Nilaksh Das, AWS AI Labs, Amazon;
(8) Zejiang Hou, AWS AI Labs, Amazon;
(9) Goeric Huybrechts, AWS AI Labs, Amazon;
(10) Srikanth Vishnubhotla, AWS AI Labs, Amazon;
(11) Daniel Garcia-Romero, AWS AI Labs, Amazon;
(12) Sundararajan Srinivasan, AWS AI Labs, Amazon;
(13) Kyu J Han, AWS AI Labs, Amazon;
(14) Katrin Kirchhoff, AWS AI Labs, Amazon. This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Part 1 : Abstract & Introduction Part 1 Part 2 : Background Part 2 Part 3 : Attacks & Countermeasures Part 3 Part 4 : Experimental Setup Part 4 Part 5 : Datasets & Evaluation Part 5 Part 6 : Attack, Countermeasure Parameters, & Baseline: Random Perturbations Part 6 Part 7 : Results & Discussion Part 7 Part 8 : Transfer Attacks & Countermeasures Part 8 Part 9 : Conclusion, Limitations, & Ethics Statement Part 9 Part 10 : Appendix: Audio Encoder Pre-training & Evaluation Part 10 Part 11 : Appendix: Cross-prompt attacks, Training Data Ablations, & Impact of random noise on helpfulness Part 11 Part 12 : Appendix: Adaptive attacks & Qualitative Examples Part 12 A.6 Adaptive attacks In this section, we report the results using adaptive attacks, where the attacker has knowledge of any defense mechanism employed in the system. We employ α=0.0001 (Eq. 1) as we found that the attacker needs a larger step size in the presence of a defense to produce successful attacks. From Table 10, we see that the attacks become less successful in the presence of a defense. Also, the adaptive attacker needs to add much more perceptible perturbations (lower average SPR) in the presence of a defense. This clearly shows that a simple pre-processing defense can provide some degree of robustness against adversarial attacks. Also, from Figure 5, we observe that the presence of a defense in the system makes the attacks less effective under limited attack budgets. For a given attack budget of say T=50 iterations, only 60% of the attacks are successful on the system with TDNF defense, compared to ∼80% for a system without defense. However, note that these attacks were performed with a limited attack budget of T=100 iterations. A malicious actor with a larger attack budget can potentially produce a higher jailbreak rate. A.7 Qualitative Examples Table 11 compares an in-house and a public SLM model responses on harmful examples. We showcase scenarios where models produce safe content although irrelevant, as well as safe content with relevant understanding of the input audio. Overall, the in-house SLM model demonstrate better speech comprehension ability. Table 12 compares models on various helpfulness questions across different aspects of usefulness. We notice that the in-house SLM model SMistral-FT sometimes errs on the side of caution, indicating a healthy tension between harmlessness and helpfulness. We leave further explorations of such properties of SLM models to future work. On the other hand, we notice the importance of a strong audio understanding ability in an SLM, as failing to do so can impact the usefulness of an SLM model by mistaking entity names in the input audio. Table 13 showcases examples of jailbroken responses and the corresponding SPRs. We clearly see that the model produces safe responses adhering to its safety training without attack, but even minimal perturbations can cause it to produce unsafe responses. In some cases (last 2 examples), the model begins its response with a safety aligned response, but generates harmful content subsequently. This further demonstrates the need for thorough studies on model safety, and a cursory analysis may be insufficient. Authors: (1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com); (2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions; (3) Srikanth Ronanki, AWS AI Labs, Amazon; (4) Anshu Bhatia, AWS AI Labs, Amazon; (5) Karel Mundnich, AWS AI Labs, Amazon; (6) Saket Dingliwal, AWS AI Labs, Amazon; (7) Nilaksh Das, AWS AI Labs, Amazon; (8) Zejiang Hou, AWS AI Labs, Amazon; (9) Goeric Huybrechts, AWS AI Labs, Amazon; (10) Srikanth Vishnubhotla, AWS AI Labs, Amazon; (11) Daniel Garcia-Romero, AWS AI Labs, Amazon; (12) Sundararajan Srinivasan, AWS AI Labs, Amazon; (13) Kyu J Han, AWS AI Labs, Amazon; (14) Katrin Kirchhoff, AWS AI Labs, Amazon. Authors: Authors: (1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com); (2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions; (3) Srikanth Ronanki, AWS AI Labs, Amazon; (4) Anshu Bhatia, AWS AI Labs, Amazon; (5) Karel Mundnich, AWS AI Labs, Amazon; (6) Saket Dingliwal, AWS AI Labs, Amazon; (7) Nilaksh Das, AWS AI Labs, Amazon; (8) Zejiang Hou, AWS AI Labs, Amazon; (9) Goeric Huybrechts, AWS AI Labs, Amazon; (10) Srikanth Vishnubhotla, AWS AI Labs, Amazon; (11) Daniel Garcia-Romero, AWS AI Labs, Amazon; (12) Sundararajan Srinivasan, AWS AI Labs, Amazon; (13) Kyu J Han, AWS AI Labs, Amazon; (14) Katrin Kirchhoff, AWS AI Labs, Amazon. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Adaptive Attacks Expose SLM Vulnerabilities and Qualitative Insights

Table of Links Part 1: Abstract & Introduction Part 2: Background Part 3: Attacks & Countermeasures Part 4: Experimental Setup Part 5: Datasets & Evaluation Part 6: Attack, Countermeasure Parameters, & Baseline: Random Perturbations Part 7: Results & Discussion Part 8: Transfer Attacks & Countermeasures Part 9: Conclusion, Limitations, & Ethics Statement Part 10: Appendix: Audio Encoder Pre-training & Evaluation Part 11: Appendix: Cross-prompt attacks, Training Data Ablations, & Impact of random noise on helpfulness Part 12: Appendix: Adaptive attacks & Qualitative Examples 5.3 Transfer attacks In this section, we report the results of crossmodel attacks, where perturbations generated by white-box attacks on a surrogate model are applied directly on the victim models. Similar to Section 5.2, we consider only the set of originally safe audios for the jailbreak analysis. From Table 4, we see that the models exhibit varying amount of vulnerability to cross-model perturbations. In general, the FlanT5-based models are more robust to such perturbations even when the surrogate model uses similar architecture. This finding aligns with the sample-specific results we showed earlier in Section 5.2. Note that these experiments were conducted on internally developed models that use similar training strategies and data. Therefore, this is a “grey-box” scenario, where attacker has access to a limited knowledge about the victim models. To evaluate the effectiveness of the generated perturbations in a true black-box setting, we also apply the perturbations on the publicly available SpeechGPT model. The number of audio samples evaluated is much fewer for this model because of the relatively smaller number of originally safe audios. From the results in Table 5, first we see that random perturbations cause significant jailbreaks. Further, we observe that adversarial perturbations are not as effective in this setting. This demonstrates the need for characterizing the models’ safety from several angles including random perturbations. We also evaluated cross-prompt attacks, and observed that it yielded successful attacks only on par with random noise (see results in Appendix A.3). 5.4 Countermeasures In Table 6, we report the results of applying the proposed TDNF defense that injects random noise (at 4 different SNRs) to the audios. We observe that in all the cases, the TDNF defense reduces the %JSR by a substantial margin compared to no defense (shown in column “None”). Such noise addition does not significantly impact the helpfulness of our systems (see Appendix A.5 for more details). Combined with the results on random noise presented in Table 3, we can conclude that a simple pre-processing defense is a viable solution to thwart adversarial jailbreaking threats, with minimal impact on the utility. We also performed adaptive attacks, where the attacker has knowledge of the defense. We found that an adaptive attacker can evade the defense, albeit with reduced success and at a much lower SPR, rendering the perturbations more perceptible.[14] Authors:
(1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com);
(2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions;
(3) Srikanth Ronanki, AWS AI Labs, Amazon;
(4) Anshu Bhatia, AWS AI Labs, Amazon;
(5) Karel Mundnich, AWS AI Labs, Amazon;
(6) Saket Dingliwal, AWS AI Labs, Amazon;
(7) Nilaksh Das, AWS AI Labs, Amazon;
(8) Zejiang Hou, AWS AI Labs, Amazon;
(9) Goeric Huybrechts, AWS AI Labs, Amazon;
(10) Srikanth Vishnubhotla, AWS AI Labs, Amazon;
(11) Daniel Garcia-Romero, AWS AI Labs, Amazon;
(12) Sundararajan Srinivasan, AWS AI Labs, Amazon;
(13) Kyu J Han, AWS AI Labs, Amazon;
(14) Katrin Kirchhoff, AWS AI Labs, Amazon. This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Part 1 : Abstract & Introduction Part 1 Part 2 : Background Part 2 Part 3 : Attacks & Countermeasures Part 3 Part 4 : Experimental Setup Part 4 Part 5 : Datasets & Evaluation Part 5 Part 6 : Attack, Countermeasure Parameters, & Baseline: Random Perturbations Part 6 Part 7 : Results & Discussion Part 7 Part 8 : Transfer Attacks & Countermeasures Part 8 Part 9 : Conclusion, Limitations, & Ethics Statement Part 9 Part 10 : Appendix: Audio Encoder Pre-training & Evaluation Part 10 Part 11 : Appendix: Cross-prompt attacks, Training Data Ablations, & Impact of random noise on helpfulness Part 11 Part 12 : Appendix: Adaptive attacks & Qualitative Examples Part 12 5.3 Transfer attacks In this section, we report the results of crossmodel attacks, where perturbations generated by white-box attacks on a surrogate model are applied directly on the victim models. Similar to Section 5.2, we consider only the set of originally safe audios for the jailbreak analysis. From Table 4, we see that the models exhibit varying amount of vulnerability to cross-model perturbations. In general, the FlanT5-based models are more robust to such perturbations even when the surrogate model uses similar architecture. This finding aligns with the sample-specific results we showed earlier in Section 5.2. Note that these experiments were conducted on internally developed models that use similar training strategies and data. Therefore, this is a “grey-box” scenario, where attacker has access to a limited knowledge about the victim models. To evaluate the effectiveness of the generated perturbations in a true black-box setting, we also apply the perturbations on the publicly available SpeechGPT model. The number of audio samples evaluated is much fewer for this model because of the relatively smaller number of originally safe audios. From the results in Table 5, first we see that random perturbations cause significant jailbreaks. Further, we observe that adversarial perturbations are not as effective in this setting. This demonstrates the need for characterizing the models’ safety from several angles including random perturbations. We also evaluated cross-prompt attacks, and observed that it yielded successful attacks only on par with random noise (see results in Appendix A.3). 5.4 Countermeasures In Table 6, we report the results of applying the proposed TDNF defense that injects random noise (at 4 different SNRs) to the audios. We observe that in all the cases, the TDNF defense reduces the %JSR by a substantial margin compared to no defense (shown in column “None”). Such noise addition does not significantly impact the helpfulness of our systems (see Appendix A.5 for more details). Combined with the results on random noise presented in Table 3, we can conclude that a simple pre-processing defense is a viable solution to thwart adversarial jailbreaking threats, with minimal impact on the utility. We also performed adaptive attacks, where the attacker has knowledge of the defense. We found that an adaptive attacker can evade the defense, albeit with reduced success and at a much lower SPR, rendering the perturbations more perceptible.[14] Authors: (1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com); (2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions; (3) Srikanth Ronanki, AWS AI Labs, Amazon; (4) Anshu Bhatia, AWS AI Labs, Amazon; (5) Karel Mundnich, AWS AI Labs, Amazon; (6) Saket Dingliwal, AWS AI Labs, Amazon; (7) Nilaksh Das, AWS AI Labs, Amazon; (8) Zejiang Hou, AWS AI Labs, Amazon; (9) Goeric Huybrechts, AWS AI Labs, Amazon; (10) Srikanth Vishnubhotla, AWS AI Labs, Amazon; (11) Daniel Garcia-Romero, AWS AI Labs, Amazon; (12) Sundararajan Srinivasan, AWS AI Labs, Amazon; (13) Kyu J Han, AWS AI Labs, Amazon; (14) Katrin Kirchhoff, AWS AI Labs, Amazon. Authors: Authors: (1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com); (2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions; (3) Srikanth Ronanki, AWS AI Labs, Amazon; (4) Anshu Bhatia, AWS AI Labs, Amazon; (5) Karel Mundnich, AWS AI Labs, Amazon; (6) Saket Dingliwal, AWS AI Labs, Amazon; (7) Nilaksh Das, AWS AI Labs, Amazon; (8) Zejiang Hou, AWS AI Labs, Amazon; (9) Goeric Huybrechts, AWS AI Labs, Amazon; (10) Srikanth Vishnubhotla, AWS AI Labs, Amazon; (11) Daniel Garcia-Romero, AWS AI Labs, Amazon; (12) Sundararajan Srinivasan, AWS AI Labs, Amazon; (13) Kyu J Han, AWS AI Labs, Amazon; (14) Katrin Kirchhoff, AWS AI Labs, Amazon. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Transfer Attacks Reveal SLM Vulnerabilities and Effective Noise Defenses

Table of Links Part 1: Abstract & Introduction Part 2: Background Part 3: Attacks & Countermeasures Part 4: Experimental Setup Part 5: Datasets & Evaluation Part 6: Attack, Countermeasure Parameters, & Baseline: Random Perturbations Part 7: Results & Discussion Part 8: Transfer Attacks & Countermeasures Part 9: Conclusion, Limitations, & Ethics Statement Part 10: Appendix: Audio Encoder Pre-training & Evaluation Part 11: Appendix: Cross-prompt attacks, Training Data Ablations, & Impact of random noise on helpfulness Part 12: Appendix: Adaptive attacks & Qualitative Examples A.3 Cross-prompt attacks In Table 7, we report the results of jailbreaking the models using cross-prompt attack strategies. In particular, for each target question, we use 10 randomly selected perturbations (from successful attacks on the model). We report an attack successful if atleast one of the 10 perturbations is able to jailbreak the system. We observe that cross-prompt attacks are less effective than samplespecific attacks. However, they do show slightly more success in attacking than random perturbations. The reason for this could be the mismatched length between the perturbation and the target audio, which required truncation or replication of the perturbation. Further study is required to assess the possibility of more sophisticated cross-prompt attacks that leverage information about the audio length to tailor the perturbation accordingly. A.4 Training Data Ablations In Table 8, we study the usefulness of incorporating general instruction tuning data during cross-modal instruction fine-tuning stage for SLM models. We identify three of the best performing ASR pretrained and safety-aligned SLM models from Table 2 to conduct this study. As discussed in Section 4.1, we observe that incorporating TTS data of Alpaca improves the helpfulness of SLMs on general questions. We further notice that such models have capabilities to outperform their counterparts in safety and relevance as well, especially when the backbone LLMs are taken out-of-the-box without any safety alignments (Flan-T5 (3B) and Mistral-7B). However, when backbone LLMs are tuned for harmlessness (Mistral-7B-FT), we observe that there is a healthy tension between helpfulness and harmlessness, indicating merits of using general instruction data as well as a further scope for improving safety alignment of SLMs. A.5 Impact of random noise on helpfulness We study the effect of random noise perturbations on helpfulness questions against three in-house SLM models, trained with ASR modality preadaptation. We demonstrate the results in Table 9 and compare the results with the original spoken questions without any perturbations. We observe that for models with strong helpfulness capabilities, random noising can effect at most 15% of their usefulness. We believe that the robustness of SLMs for helpfulness can be improved by adding more general instruction tuning data and by noisy data augmentations during training. We leave this exploration to future work. Authors:
(1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com);
(2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions;
(3) Srikanth Ronanki, AWS AI Labs, Amazon;
(4) Anshu Bhatia, AWS AI Labs, Amazon;
(5) Karel Mundnich, AWS AI Labs, Amazon;
(6) Saket Dingliwal, AWS AI Labs, Amazon;
(7) Nilaksh Das, AWS AI Labs, Amazon;
(8) Zejiang Hou, AWS AI Labs, Amazon;
(9) Goeric Huybrechts, AWS AI Labs, Amazon;
(10) Srikanth Vishnubhotla, AWS AI Labs, Amazon;
(11) Daniel Garcia-Romero, AWS AI Labs, Amazon;
(12) Sundararajan Srinivasan, AWS AI Labs, Amazon;
(13) Kyu J Han, AWS AI Labs, Amazon;
(14) Katrin Kirchhoff, AWS AI Labs, Amazon. This paper is available on arxiv under CC BY 4.0 DEED license. Table of Links Part 1 : Abstract & Introduction Part 1 Part 2 : Background Part 2 Part 3 : Attacks & Countermeasures Part 3 Part 4 : Experimental Setup Part 4 Part 5 : Datasets & Evaluation Part 5 Part 6 : Attack, Countermeasure Parameters, & Baseline: Random Perturbations Part 6 Part 7 : Results & Discussion Part 7 Part 8 : Transfer Attacks & Countermeasures Part 8 Part 9 : Conclusion, Limitations, & Ethics Statement Part 9 Part 10 : Appendix: Audio Encoder Pre-training & Evaluation Part 10 Part 11 : Appendix: Cross-prompt attacks, Training Data Ablations, & Impact of random noise on helpfulness Part 11 Part 12 : Appendix: Adaptive attacks & Qualitative Examples Part 12 A.3 Cross-prompt attacks In Table 7, we report the results of jailbreaking the models using cross-prompt attack strategies. In particular, for each target question, we use 10 randomly selected perturbations (from successful attacks on the model). We report an attack successful if atleast one of the 10 perturbations is able to jailbreak the system. We observe that cross-prompt attacks are less effective than samplespecific attacks. However, they do show slightly more success in attacking than random perturbations. The reason for this could be the mismatched length between the perturbation and the target audio, which required truncation or replication of the perturbation. Further study is required to assess the possibility of more sophisticated cross-prompt attacks that leverage information about the audio length to tailor the perturbation accordingly. A.4 Training Data Ablations In Table 8, we study the usefulness of incorporating general instruction tuning data during cross-modal instruction fine-tuning stage for SLM models. We identify three of the best performing ASR pretrained and safety-aligned SLM models from Table 2 to conduct this study. As discussed in Section 4.1, we observe that incorporating TTS data of Alpaca improves the helpfulness of SLMs on general questions. We further notice that such models have capabilities to outperform their counterparts in safety and relevance as well, especially when the backbone LLMs are taken out-of-the-box without any safety alignments (Flan-T5 (3B) and Mistral-7B). However, when backbone LLMs are tuned for harmlessness (Mistral-7B-FT), we observe that there is a healthy tension between helpfulness and harmlessness, indicating merits of using general instruction data as well as a further scope for improving safety alignment of SLMs. A.5 Impact of random noise on helpfulness We study the effect of random noise perturbations on helpfulness questions against three in-house SLM models, trained with ASR modality preadaptation. We demonstrate the results in Table 9 and compare the results with the original spoken questions without any perturbations. We observe that for models with strong helpfulness capabilities, random noising can effect at most 15% of their usefulness. We believe that the robustness of SLMs for helpfulness can be improved by adding more general instruction tuning data and by noisy data augmentations during training. We leave this exploration to future work. Authors: (1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com); (2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions; (3) Srikanth Ronanki, AWS AI Labs, Amazon; (4) Anshu Bhatia, AWS AI Labs, Amazon; (5) Karel Mundnich, AWS AI Labs, Amazon; (6) Saket Dingliwal, AWS AI Labs, Amazon; (7) Nilaksh Das, AWS AI Labs, Amazon; (8) Zejiang Hou, AWS AI Labs, Amazon; (9) Goeric Huybrechts, AWS AI Labs, Amazon; (10) Srikanth Vishnubhotla, AWS AI Labs, Amazon; (11) Daniel Garcia-Romero, AWS AI Labs, Amazon; (12) Sundararajan Srinivasan, AWS AI Labs, Amazon; (13) Kyu J Han, AWS AI Labs, Amazon; (14) Katrin Kirchhoff, AWS AI Labs, Amazon. Authors: Authors: (1) Raghuveer Peri, AWS AI Labs, Amazon and with Equal Contributions (raghperi@amazon.com); (2) Sai Muralidhar Jayanthi, AWS AI Labs, Amazon and with Equal Contributions; (3) Srikanth Ronanki, AWS AI Labs, Amazon; (4) Anshu Bhatia, AWS AI Labs, Amazon; (5) Karel Mundnich, AWS AI Labs, Amazon; (6) Saket Dingliwal, AWS AI Labs, Amazon; (7) Nilaksh Das, AWS AI Labs, Amazon; (8) Zejiang Hou, AWS AI Labs, Amazon; (9) Goeric Huybrechts, AWS AI Labs, Amazon; (10) Srikanth Vishnubhotla, AWS AI Labs, Amazon; (11) Daniel Garcia-Romero, AWS AI Labs, Amazon; (12) Sundararajan Srinivasan, AWS AI Labs, Amazon; (13) Kyu J Han, AWS AI Labs, Amazon; (14) Katrin Kirchhoff, AWS AI Labs, Amazon. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv

Stories, News & Mentions about ATT

Att’s Stories on HackerNoon

Att's News & Mentions