Wednesday, May 22, 2024
HomeNewsPubMed GPT: A GPT Model Trained on PubMed Biomedical Papers at Stanford

PubMed GPT: A GPT Model Trained on PubMed Biomedical Papers at Stanford

Researchers at the Stanford Center for Research on Foundation Models (CRFM) have recently worked on investigating industry-specific LLMs (large language models). They introduce PubMed GPT as a part of their research, specifically focusing on biomedicine. 

Using the MosaicML cloud platform, CRFM researchers trained a GPT model on PubMed biomedical papers. The resultant model is highly accurate in several NLP tasks. PubMed GPT is based on a HuggingFace foundation and uses a biomedical tokenizer trained on the Pile dataset abstracts and PubMed central sections. It uses the PyTorch framework and the composer from MosaicML for training LLMs.

After training the model, researchers evaluated it on several popular benchmarks, a critical measure being the MedQA-USMLE question-answer challenge. In addition, the researchers manually assessed its generations for a task that involved summarising questions. The researchers employed several previous CRFM and biological models, including GPT-Neo, Galactica, and PubMedBERT.

Read More: Meta Turned Down the Galactica Demo After Being Criticized as “Dangerous”

The researchers concluded that LLMs are versatile and have much to offer when trained on domain-specific datasets. But the versatility comes at a cost due to many parameters. Model complexity, cost, specialized architectures, and domain knowledge are all trade-offs with the performance of PubMedGPT.

The researchers plan to concentrate future work on enhancing the scope of the model and assessing it against a more extensive collection of NLP tasks. PubMed GPT is intended solely for research as it is yet to be developed for production.

Subscribe to our newsletter

Subscribe and never miss out on such trending AI-related articles.

We will never sell your data

Join our WhatsApp Channel and Discord Server to be a part of an engaging community.

Disha Chopra
Disha Chopra
Disha Chopra is a content enthusiast! She is an Economics graduate pursuing her PG in the same field along with Data Sciences. Disha enjoys the ever-demanding world of content and the flexibility that comes with it. She can be found listening to music or simply asleep when not working!


Please enter your comment!
Please enter your name here

Most Popular