Close Menu
    Main Menu
    • Home
    • News
    • Tech
    • Robotics
    • ML & Research
    • AI
    • Digital Transformation
    • AI Ethics & Regulation
    • Thought Leadership in AI

    Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    ‘Protected’ Photographs Are Simpler, Not Extra Tough, to Steal With AI

    June 9, 2025

    ⚡ Weekly Recap: Chrome 0-Day, Information Wipers, Misused Instruments and Zero-Click on iPhone Assaults

    June 9, 2025

    Google Gemini will allow you to schedule recurring duties now, like ChatGPT – this is how

    June 9, 2025
    Facebook X (Twitter) Instagram
    UK Tech Insider
    Facebook X (Twitter) Instagram Pinterest Vimeo
    UK Tech Insider
    Home»Machine Learning & Research»Scaling Diffusion Language Fashions by way of Adaptation from Autoregressive Fashions
    Machine Learning & Research

    Scaling Diffusion Language Fashions by way of Adaptation from Autoregressive Fashions

    Arjun PatelBy Arjun PatelApril 19, 2025Updated:April 29, 2025No Comments1 Min Read
    Facebook Twitter Pinterest Telegram LinkedIn Tumblr Email Reddit
    Scaling Diffusion Language Fashions by way of Adaptation from Autoregressive Fashions
    Share
    Facebook Twitter LinkedIn Pinterest Email Copy Link


    Diffusion Language Fashions (DLMs) have emerged as a promising new paradigm for textual content generative modeling, doubtlessly addressing limitations of autoregressive (AR) fashions. Nonetheless, present DLMs have been studied at a smaller scale in comparison with their AR counterparts and lack truthful comparability on language modeling benchmarks. Moreover, coaching diffusion fashions from scratch at scale stays difficult. Given the prevalence of open-source AR language fashions, we suggest adapting these fashions to construct textual content diffusion fashions. We display connections between AR and diffusion modeling aims and introduce a easy continuous pre-training strategy for coaching diffusion fashions. Via systematic analysis on language modeling, reasoning, and commonsense benchmarks, we present that we will convert AR fashions starting from 127M to 7B parameters (GPT2 and LLaMA) into diffusion fashions DiffuGPT and DiffuLLaMA, utilizing lower than 200B tokens for coaching. Our experimental outcomes reveal that these fashions outperform earlier DLMs and are aggressive with their AR counterparts. We launch a collection of DLMs (127M-355M-7B) able to producing fluent textual content, performing in-context studying, filling within the center with out immediate re-ordering, and following directions.

    † The College of Hong Kong
    ‡ College of Illinois at Urbana-Champaign
    § Tencent AI Lab

    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email
    Arjun Patel
    • Website

    Related Posts

    7 Cool Python Initiatives to Automate the Boring Stuff

    June 9, 2025

    ML Mannequin Serving with FastAPI and Redis for sooner predictions

    June 9, 2025

    Construct a Textual content-to-SQL resolution for information consistency in generative AI utilizing Amazon Nova

    June 7, 2025
    Leave A Reply Cancel Reply

    Top Posts

    ‘Protected’ Photographs Are Simpler, Not Extra Tough, to Steal With AI

    June 9, 2025

    How AI is Redrawing the World’s Electrical energy Maps: Insights from the IEA Report

    April 18, 2025

    Evaluating the Finest AI Video Mills for Social Media

    April 18, 2025

    Utilizing AI To Repair The Innovation Drawback: The Three Step Resolution

    April 18, 2025
    Don't Miss

    ‘Protected’ Photographs Are Simpler, Not Extra Tough, to Steal With AI

    By Amelia Harper JonesJune 9, 2025

    New analysis means that watermarking instruments meant to dam AI picture edits might backfire. As…

    ⚡ Weekly Recap: Chrome 0-Day, Information Wipers, Misused Instruments and Zero-Click on iPhone Assaults

    June 9, 2025

    Google Gemini will allow you to schedule recurring duties now, like ChatGPT – this is how

    June 9, 2025

    7 Cool Python Initiatives to Automate the Boring Stuff

    June 9, 2025
    Stay In Touch
    • Facebook
    • Twitter
    • Pinterest
    • Instagram
    • YouTube
    • Vimeo

    Subscribe to Updates

    Get the latest creative news from SmartMag about art & design.

    UK Tech Insider
    Facebook X (Twitter) Instagram Pinterest
    • About Us
    • Contact Us
    • Privacy Policy
    • Terms Of Service
    • Our Authors
    © 2025 UK Tech Insider. All rights reserved by UK Tech Insider.

    Type above and press Enter to search. Press Esc to cancel.