Buchumschlag

Gemma 3 Technical Report

Fuente: arXiv

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Gemma Team, Kamath, Aishwarya, Ferret, Johan, Pathak, Shreya, Vieillard, Nino, Merhej, Ramona, Perrin, Sarah, Matejovicova, Tatiana, Ramé, Alexandre, Rivière, Morgane, Rouillard, Louis, Mesnard, Thomas, Cideron, Geoffrey, Grill, Jean-bastien, Ramos, Sabela, Yvinec, Edouard, Casbon, Michelle, Pot, Etienne, Penchev, Ivo, Liu, Gaël, Visin, Francesco, Kenealy, Kathleen, Beyer, Lucas, Zhai, Xiaohai, Tsitsulin, Anton, Busa-Fekete, Robert, Feng, Alex, Sachdeva, Noveen, Coleman, Benjamin, Gao, Yi, Mustafa, Basil, Barr, Iain, Parisotto, Emilio, Tian, David, Eyal, Matan, Cherry, Colin, Peter, Jan-Thorsten, Sinopalnikov, Danila, Bhupatiraju, Surya, Agarwal, Rishabh, Kazemi, Mehran, Malkin, Dan, Kumar, Ravin, Vilar, David, Brusilovsky, Idan, Luo, Jiaming, Steiner, Andreas, Friesen, Abe, Sharma, Abhanshu, Sharma, Abheesht, Gilady, Adi Mayrav, Goedeckemeyer, Adrian, Saade, Alaa, Kolesnikov, Alexander, Bendebury, Alexei, Abdagic, Alvin, Vadi, Amit, György, András, Pinto, André Susano, Das, Anil, Bapna, Ankur, Miech, Antoine, Yang, Antoine, Paterson, Antonia, Shenoy, Ashish, Chakrabarti, Ayan, Piot, Bilal, Wu, Bo, Shahriari, Bobak, Petrini, Bryce, Chen, Charlie, Lan, Charline Le, Choquette-Choo, Christopher A., Carey, CJ, Brick, Cormac, Deutsch, Daniel, Eisenbud, Danielle, Cattle, Dee, Cheng, Derek, Paparas, Dimitris, Sreepathihalli, Divyashree Shivakumar, Reid, Doug, Tran, Dustin, Zelle, Dustin, Noland, Eric, Huizenga, Erwin, Kharitonov, Eugene, Liu, Frederick, Amirkhanyan, Gagik, Cameron, Glenn, Hashemi, Hadi, Klimczak-Plucińska, Hanna, Singh, Harman, Mehta, Harsh, Lehri, Harshal Tushar, Hazimeh, Hussein, Ballantyne, Ian, Szpektor, Idan, Nardini, Ivan, Pouget-Abadie, Jean, Chan, Jetha, Stanton, Joe, Wieting, John, Lai, Jonathan, Orbay, Jordi, Fernandez, Joseph, Newlan, Josh, Ji, Ju-yeong, Singh, Jyotinder, Black, Kat, Yu, Kathy, Hui, Kevin, Vodrahalli, Kiran, Greff, Klaus, Qiu, Linhai, Valentine, Marcella, Coelho, Marina, Ritter, Marvin, Hoffman, Matt, Watson, Matthew, Chaturvedi, Mayank, Moynihan, Michael, Ma, Min, Babar, Nabila, Noy, Natasha, Byrd, Nathan, Roy, Nick, Momchev, Nikola, Chauhan, Nilay, Bunyan, Oskar, Botarda, Pankil, Caron, Paul, Rubenstein, Paul Kishan, Culliton, Phil, Schmid, Philipp, Sessa, Pier Giuseppe, Xu, Pingmei, Stanczyk, Piotr, Tafti, Pouya, Shivanna, Rakesh, Wu, Renjie, Pan, Renke, Rokni, Reza, Willoughby, Rob, Vallu, Rohith, Mullins, Ryan, Jerome, Sammy, Smoot, Sara, Girgin, Sertan, Iqbal, Shariq, Reddy, Shashir, Sheth, Shruti, Põder, Siim, Bhatnagar, Sijal, Panyam, Sindhu Raghuram, Eiger, Sivan, Zhang, Susan, Liu, Tianqi, Yacovone, Trevor, Liechty, Tyler, Kalra, Uday, Evci, Utku, Misra, Vedant, Roseberry, Vincent, Feinberg, Vlad, Kolesnikov, Vlad, Han, Woohyun, Kwon, Woosuk, Chen, Xi, Chow, Yinlam, Zhu, Yuvein, Wei, Zichuan, Egyed, Zoltan, Cotruta, Victor, Giang, Minh, Kirk, Phoebe, Rao, Anand, Lo, Jessica, Moreira, Erica, Martins, Luiz Gustavo, Sanseviero, Omar, Gonzalez, Lucas, Gleicher, Zach, Warkentin, Tris, Mirrokni, Vahab, Senter, Evan, Collins, Eli, Barral, Joelle, Ghahramani, Zoubin, Hadsell, Raia, Matias, Yossi, Sculley, D., Petrov, Slav, Fiedel, Noah, Shazeer, Noam, Vinyals, Oriol, Dean, Jeff, Hassabis, Demis, Kavukcuoglu, Koray, Farabet, Clement, Buchatskaya, Elena, Alayrac, Jean-Baptiste, Anil, Rohan, Dmitry, Lepikhin, Borgeaud, Sebastian, Bachem, Olivier, Joulin, Armand, Andreev, Alek, Hardin, Cassidy, Dadashi, Robert, Hussenot, Léonard
Format:	Preprint
Veröffentlicht:	2025
Schlagworte:	Computation and Language Artificial Intelligence
Online-Zugang:	Acceder al recurso
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

_version_	1866913757567909888
author	Gemma Team Kamath, Aishwarya Ferret, Johan Pathak, Shreya Vieillard, Nino Merhej, Ramona Perrin, Sarah Matejovicova, Tatiana Ramé, Alexandre Rivière, Morgane Rouillard, Louis Mesnard, Thomas Cideron, Geoffrey Grill, Jean-bastien Ramos, Sabela Yvinec, Edouard Casbon, Michelle Pot, Etienne Penchev, Ivo Liu, Gaël Visin, Francesco Kenealy, Kathleen Beyer, Lucas Zhai, Xiaohai Tsitsulin, Anton Busa-Fekete, Robert Feng, Alex Sachdeva, Noveen Coleman, Benjamin Gao, Yi Mustafa, Basil Barr, Iain Parisotto, Emilio Tian, David Eyal, Matan Cherry, Colin Peter, Jan-Thorsten Sinopalnikov, Danila Bhupatiraju, Surya Agarwal, Rishabh Kazemi, Mehran Malkin, Dan Kumar, Ravin Vilar, David Brusilovsky, Idan Luo, Jiaming Steiner, Andreas Friesen, Abe Sharma, Abhanshu Sharma, Abheesht Gilady, Adi Mayrav Goedeckemeyer, Adrian Saade, Alaa Feng, Alex Kolesnikov, Alexander Bendebury, Alexei Abdagic, Alvin Vadi, Amit György, András Pinto, André Susano Das, Anil Bapna, Ankur Miech, Antoine Yang, Antoine Paterson, Antonia Shenoy, Ashish Chakrabarti, Ayan Piot, Bilal Wu, Bo Shahriari, Bobak Petrini, Bryce Chen, Charlie Lan, Charline Le Choquette-Choo, Christopher A. Carey, CJ Brick, Cormac Deutsch, Daniel Eisenbud, Danielle Cattle, Dee Cheng, Derek Paparas, Dimitris Sreepathihalli, Divyashree Shivakumar Reid, Doug Tran, Dustin Zelle, Dustin Noland, Eric Huizenga, Erwin Kharitonov, Eugene Liu, Frederick Amirkhanyan, Gagik Cameron, Glenn Hashemi, Hadi Klimczak-Plucińska, Hanna Singh, Harman Mehta, Harsh Lehri, Harshal Tushar Hazimeh, Hussein Ballantyne, Ian Szpektor, Idan Nardini, Ivan Pouget-Abadie, Jean Chan, Jetha Stanton, Joe Wieting, John Lai, Jonathan Orbay, Jordi Fernandez, Joseph Newlan, Josh Ji, Ju-yeong Singh, Jyotinder Black, Kat Yu, Kathy Hui, Kevin Vodrahalli, Kiran Greff, Klaus Qiu, Linhai Valentine, Marcella Coelho, Marina Ritter, Marvin Hoffman, Matt Watson, Matthew Chaturvedi, Mayank Moynihan, Michael Ma, Min Babar, Nabila Noy, Natasha Byrd, Nathan Roy, Nick Momchev, Nikola Chauhan, Nilay Sachdeva, Noveen Bunyan, Oskar Botarda, Pankil Caron, Paul Rubenstein, Paul Kishan Culliton, Phil Schmid, Philipp Sessa, Pier Giuseppe Xu, Pingmei Stanczyk, Piotr Tafti, Pouya Shivanna, Rakesh Wu, Renjie Pan, Renke Rokni, Reza Willoughby, Rob Vallu, Rohith Mullins, Ryan Jerome, Sammy Smoot, Sara Girgin, Sertan Iqbal, Shariq Reddy, Shashir Sheth, Shruti Põder, Siim Bhatnagar, Sijal Panyam, Sindhu Raghuram Eiger, Sivan Zhang, Susan Liu, Tianqi Yacovone, Trevor Liechty, Tyler Kalra, Uday Evci, Utku Misra, Vedant Roseberry, Vincent Feinberg, Vlad Kolesnikov, Vlad Han, Woohyun Kwon, Woosuk Chen, Xi Chow, Yinlam Zhu, Yuvein Wei, Zichuan Egyed, Zoltan Cotruta, Victor Giang, Minh Kirk, Phoebe Rao, Anand Black, Kat Babar, Nabila Lo, Jessica Moreira, Erica Martins, Luiz Gustavo Sanseviero, Omar Gonzalez, Lucas Gleicher, Zach Warkentin, Tris Mirrokni, Vahab Senter, Evan Collins, Eli Barral, Joelle Ghahramani, Zoubin Hadsell, Raia Matias, Yossi Sculley, D. Petrov, Slav Fiedel, Noah Shazeer, Noam Vinyals, Oriol Dean, Jeff Hassabis, Demis Kavukcuoglu, Koray Farabet, Clement Buchatskaya, Elena Alayrac, Jean-Baptiste Anil, Rohan Dmitry Lepikhin Borgeaud, Sebastian Bachem, Olivier Joulin, Armand Andreev, Alek Hardin, Cassidy Dadashi, Robert Hussenot, Léonard
author_facet	Gemma Team Kamath, Aishwarya Ferret, Johan Pathak, Shreya Vieillard, Nino Merhej, Ramona Perrin, Sarah Matejovicova, Tatiana Ramé, Alexandre Rivière, Morgane Rouillard, Louis Mesnard, Thomas Cideron, Geoffrey Grill, Jean-bastien Ramos, Sabela Yvinec, Edouard Casbon, Michelle Pot, Etienne Penchev, Ivo Liu, Gaël Visin, Francesco Kenealy, Kathleen Beyer, Lucas Zhai, Xiaohai Tsitsulin, Anton Busa-Fekete, Robert Feng, Alex Sachdeva, Noveen Coleman, Benjamin Gao, Yi Mustafa, Basil Barr, Iain Parisotto, Emilio Tian, David Eyal, Matan Cherry, Colin Peter, Jan-Thorsten Sinopalnikov, Danila Bhupatiraju, Surya Agarwal, Rishabh Kazemi, Mehran Malkin, Dan Kumar, Ravin Vilar, David Brusilovsky, Idan Luo, Jiaming Steiner, Andreas Friesen, Abe Sharma, Abhanshu Sharma, Abheesht Gilady, Adi Mayrav Goedeckemeyer, Adrian Saade, Alaa Feng, Alex Kolesnikov, Alexander Bendebury, Alexei Abdagic, Alvin Vadi, Amit György, András Pinto, André Susano Das, Anil Bapna, Ankur Miech, Antoine Yang, Antoine Paterson, Antonia Shenoy, Ashish Chakrabarti, Ayan Piot, Bilal Wu, Bo Shahriari, Bobak Petrini, Bryce Chen, Charlie Lan, Charline Le Choquette-Choo, Christopher A. Carey, CJ Brick, Cormac Deutsch, Daniel Eisenbud, Danielle Cattle, Dee Cheng, Derek Paparas, Dimitris Sreepathihalli, Divyashree Shivakumar Reid, Doug Tran, Dustin Zelle, Dustin Noland, Eric Huizenga, Erwin Kharitonov, Eugene Liu, Frederick Amirkhanyan, Gagik Cameron, Glenn Hashemi, Hadi Klimczak-Plucińska, Hanna Singh, Harman Mehta, Harsh Lehri, Harshal Tushar Hazimeh, Hussein Ballantyne, Ian Szpektor, Idan Nardini, Ivan Pouget-Abadie, Jean Chan, Jetha Stanton, Joe Wieting, John Lai, Jonathan Orbay, Jordi Fernandez, Joseph Newlan, Josh Ji, Ju-yeong Singh, Jyotinder Black, Kat Yu, Kathy Hui, Kevin Vodrahalli, Kiran Greff, Klaus Qiu, Linhai Valentine, Marcella Coelho, Marina Ritter, Marvin Hoffman, Matt Watson, Matthew Chaturvedi, Mayank Moynihan, Michael Ma, Min Babar, Nabila Noy, Natasha Byrd, Nathan Roy, Nick Momchev, Nikola Chauhan, Nilay Sachdeva, Noveen Bunyan, Oskar Botarda, Pankil Caron, Paul Rubenstein, Paul Kishan Culliton, Phil Schmid, Philipp Sessa, Pier Giuseppe Xu, Pingmei Stanczyk, Piotr Tafti, Pouya Shivanna, Rakesh Wu, Renjie Pan, Renke Rokni, Reza Willoughby, Rob Vallu, Rohith Mullins, Ryan Jerome, Sammy Smoot, Sara Girgin, Sertan Iqbal, Shariq Reddy, Shashir Sheth, Shruti Põder, Siim Bhatnagar, Sijal Panyam, Sindhu Raghuram Eiger, Sivan Zhang, Susan Liu, Tianqi Yacovone, Trevor Liechty, Tyler Kalra, Uday Evci, Utku Misra, Vedant Roseberry, Vincent Feinberg, Vlad Kolesnikov, Vlad Han, Woohyun Kwon, Woosuk Chen, Xi Chow, Yinlam Zhu, Yuvein Wei, Zichuan Egyed, Zoltan Cotruta, Victor Giang, Minh Kirk, Phoebe Rao, Anand Black, Kat Babar, Nabila Lo, Jessica Moreira, Erica Martins, Luiz Gustavo Sanseviero, Omar Gonzalez, Lucas Gleicher, Zach Warkentin, Tris Mirrokni, Vahab Senter, Evan Collins, Eli Barral, Joelle Ghahramani, Zoubin Hadsell, Raia Matias, Yossi Sculley, D. Petrov, Slav Fiedel, Noah Shazeer, Noam Vinyals, Oriol Dean, Jeff Hassabis, Demis Kavukcuoglu, Koray Farabet, Clement Buchatskaya, Elena Alayrac, Jean-Baptiste Anil, Rohan Dmitry Lepikhin Borgeaud, Sebastian Bachem, Olivier Joulin, Armand Andreev, Alek Hardin, Cassidy Dadashi, Robert Hussenot, Léonard
contents	We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.
format	Preprint
id	arxiv_https___arxiv_org_abs_2503_19786
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Gemma 3 Technical Report Gemma Team Kamath, Aishwarya Ferret, Johan Pathak, Shreya Vieillard, Nino Merhej, Ramona Perrin, Sarah Matejovicova, Tatiana Ramé, Alexandre Rivière, Morgane Rouillard, Louis Mesnard, Thomas Cideron, Geoffrey Grill, Jean-bastien Ramos, Sabela Yvinec, Edouard Casbon, Michelle Pot, Etienne Penchev, Ivo Liu, Gaël Visin, Francesco Kenealy, Kathleen Beyer, Lucas Zhai, Xiaohai Tsitsulin, Anton Busa-Fekete, Robert Feng, Alex Sachdeva, Noveen Coleman, Benjamin Gao, Yi Mustafa, Basil Barr, Iain Parisotto, Emilio Tian, David Eyal, Matan Cherry, Colin Peter, Jan-Thorsten Sinopalnikov, Danila Bhupatiraju, Surya Agarwal, Rishabh Kazemi, Mehran Malkin, Dan Kumar, Ravin Vilar, David Brusilovsky, Idan Luo, Jiaming Steiner, Andreas Friesen, Abe Sharma, Abhanshu Sharma, Abheesht Gilady, Adi Mayrav Goedeckemeyer, Adrian Saade, Alaa Feng, Alex Kolesnikov, Alexander Bendebury, Alexei Abdagic, Alvin Vadi, Amit György, András Pinto, André Susano Das, Anil Bapna, Ankur Miech, Antoine Yang, Antoine Paterson, Antonia Shenoy, Ashish Chakrabarti, Ayan Piot, Bilal Wu, Bo Shahriari, Bobak Petrini, Bryce Chen, Charlie Lan, Charline Le Choquette-Choo, Christopher A. Carey, CJ Brick, Cormac Deutsch, Daniel Eisenbud, Danielle Cattle, Dee Cheng, Derek Paparas, Dimitris Sreepathihalli, Divyashree Shivakumar Reid, Doug Tran, Dustin Zelle, Dustin Noland, Eric Huizenga, Erwin Kharitonov, Eugene Liu, Frederick Amirkhanyan, Gagik Cameron, Glenn Hashemi, Hadi Klimczak-Plucińska, Hanna Singh, Harman Mehta, Harsh Lehri, Harshal Tushar Hazimeh, Hussein Ballantyne, Ian Szpektor, Idan Nardini, Ivan Pouget-Abadie, Jean Chan, Jetha Stanton, Joe Wieting, John Lai, Jonathan Orbay, Jordi Fernandez, Joseph Newlan, Josh Ji, Ju-yeong Singh, Jyotinder Black, Kat Yu, Kathy Hui, Kevin Vodrahalli, Kiran Greff, Klaus Qiu, Linhai Valentine, Marcella Coelho, Marina Ritter, Marvin Hoffman, Matt Watson, Matthew Chaturvedi, Mayank Moynihan, Michael Ma, Min Babar, Nabila Noy, Natasha Byrd, Nathan Roy, Nick Momchev, Nikola Chauhan, Nilay Sachdeva, Noveen Bunyan, Oskar Botarda, Pankil Caron, Paul Rubenstein, Paul Kishan Culliton, Phil Schmid, Philipp Sessa, Pier Giuseppe Xu, Pingmei Stanczyk, Piotr Tafti, Pouya Shivanna, Rakesh Wu, Renjie Pan, Renke Rokni, Reza Willoughby, Rob Vallu, Rohith Mullins, Ryan Jerome, Sammy Smoot, Sara Girgin, Sertan Iqbal, Shariq Reddy, Shashir Sheth, Shruti Põder, Siim Bhatnagar, Sijal Panyam, Sindhu Raghuram Eiger, Sivan Zhang, Susan Liu, Tianqi Yacovone, Trevor Liechty, Tyler Kalra, Uday Evci, Utku Misra, Vedant Roseberry, Vincent Feinberg, Vlad Kolesnikov, Vlad Han, Woohyun Kwon, Woosuk Chen, Xi Chow, Yinlam Zhu, Yuvein Wei, Zichuan Egyed, Zoltan Cotruta, Victor Giang, Minh Kirk, Phoebe Rao, Anand Black, Kat Babar, Nabila Lo, Jessica Moreira, Erica Martins, Luiz Gustavo Sanseviero, Omar Gonzalez, Lucas Gleicher, Zach Warkentin, Tris Mirrokni, Vahab Senter, Evan Collins, Eli Barral, Joelle Ghahramani, Zoubin Hadsell, Raia Matias, Yossi Sculley, D. Petrov, Slav Fiedel, Noah Shazeer, Noam Vinyals, Oriol Dean, Jeff Hassabis, Demis Kavukcuoglu, Koray Farabet, Clement Buchatskaya, Elena Alayrac, Jean-Baptiste Anil, Rohan Dmitry Lepikhin Borgeaud, Sebastian Bachem, Olivier Joulin, Armand Andreev, Alek Hardin, Cassidy Dadashi, Robert Hussenot, Léonard Computation and Language Artificial Intelligence We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.
title	Gemma 3 Technical Report
topic	Computation and Language Artificial Intelligence
url	https://arxiv.org/abs/2503.19786