Sébastien Bubeck - Exclusive Interview
Added 2023-12-17 19:42:06 +0000 UTCAn exclusive interview with Microsoft's Sébastien Bubeck, co-author of Sparks of AGI and the Phi series of models. From practical tips to AGI timelines, AI on a phone to whether models can reason, it was a great chat and it even ended on a powerful note regarding the future of scaling language models.
Exclusive Interview +
Bubeck: https://en.m.wikipedia.org/wiki/S%C3%A9bastien_Bubeck
Phi-2: https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
Textbooks are All You Need: https://www.microsoft.com/en-us/research/publication/textbooks-are-all-you-need/
Microsoft Prompting: https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/
MedPrompt: https://www.microsoft.com/en-us/research/blog/the-power-of-prompting/
H100s and B100s: https://www.tomshardware.com/tech-industry/nvidia-ai-and-hpc-gpu-sales-reportedly-approached-half-a-million-units-in-q3-thanks-to-meta-facebook
https://www.digitimes.com/news/a20231215PD209/microsoft-nvidia-amd-h100-b100-ai-it+ce-it-components-peripherals-llm.html
AI Explained Phi-2: https://youtu.be/nPgs8THgbuI?si=MgiUS1HikYSR0IJi
TinyGSM (paper he had just published): https://arxiv.org/abs/2312.09241
Comments
Great points Shawn
Philip
2024-01-28 14:06:21 +0000 UTCAlso, while easier said than done, a perspective shift can help a little. I’ve been doing meditation for a while now and between Buddhism and other sources (there’s a recent interesting book called 4000 Weeks which presents as a time management book but really more about philosophy), a big source of stress is trying to control the future. AI makes it more obvious, but don’t need that for the future to be uncertain. Could have a car accident tomorrow and be paralyzed. Could meet the person of your dreams and move with them to another state. Doesn’t mean we shouldn’t plan at all, but often we need to let go a little too and be ready to pivot as opportunities come up.
Shawn Fumo
2024-01-27 14:39:34 +0000 UTCMaybe one thing to take solace in is that in a lot of ways we’re all in the same boat. As a senior dev, I have an advantage in the very near term in that maybe there will be less interns and juniors. But at the same time, not all older people will do that, and it also empowers people who are starting out to learn and do more. And if my job is totally automated, tons of others will be too by then. Maybe we do UBI, though I have to think there is a very rough transition. I don’t know your major but if it is dev focused, I think I’d try to learn as much ML as you can (Karpathy’s zero to hero vids are a great dev intro) vs making websites. But yeah, I feel like a crazy person sometimes. If you aren’t following all of what is going on, it hard to have any idea of the scope. Not just LLMs but everything else in images, video, voice, robotics.
Shawn Fumo
2024-01-27 13:54:20 +0000 UTCHave you ever done a deep dive on the concept of toxicity and bias itself? Whenever it’s discussed it always occurs to me that it’s not at all trivial to be objective of what is bias and what is just data. Is the only way to meet one observers notion of non-bias alignment with all of their own biases?
Andrew Walker
2023-12-22 14:42:30 +0000 UTCChristo, I created prompt based on MedPrompt methodology. I tested it on the most challanging diagnostic cases from NEJM magazine. The performence was amazing - more accurate that most physicians that participated into challanges. In Microsoft"s publications you do not find the examples of used prompts. Below this is my version: Task: Diagnose the disease based on symptoms from provided [patient history] . Recall and select examples of similar cases you were trained on. Now, based on the selected examples, describe your reasoning process. Step 1: Identify commonalities in symptoms among the selected cases. Step 2: Consider possible diagnoses for these symptoms. Step 3: Draw a conclusion about the most likely diagnosis. Potential Diagnoses: D1, D2, D3. Shuffle these diagnoses in different orders and create a reasoning path for each order. Finally, determine the most consistent diagnosis across these paths. [patient history] = '''...'''
Michal Babula
2023-12-21 11:56:26 +0000 UTCgreat and informative video! thx. :)
Christopher Pollin
2023-12-20 09:53:32 +0000 UTCI love the direction in which he is working! The performance seems quite interesting. Are there any good performance tests of Phi-2 for code generation that someone could share? I'm fascinated by the idea of using SLMs to quickly generate less than 150 lines of code from text with less than 2K tokens, plus technologies like etched.ai (mentioned in the latest AI Explained YT vid).
Daniel Schönbohm
2023-12-18 20:55:00 +0000 UTCAll of this is incredibly fascinating, but quite terrifying. I just finished my first semester at college, but with all the rampant progress, I am unsure what to do. How does someone make a decision about the future in today's age? Also I absolutely love your videos, as it is so far the clearest and most objective un-clickbait information I can find all wrapped up into a digestible format👍
Jonathan Kirk
2023-12-18 20:48:01 +0000 UTCThanks so much Christian, much more to come! :)
Philip
2023-12-18 16:17:23 +0000 UTCLoved this one. Really incredible how you’re pushing the conversations forward Phillip
Christian Nofziger
2023-12-18 15:05:31 +0000 UTCGreat interview - thank you :) Based on my experience, I can achieve significant improvement in LMM's output by providing high-quality data for retrieval (which I prepare myself; instead of a full 200-page book, I create a 20-page draft containing the most important information). I totally agree with what Sebastian said.
Michal Babula
2023-12-18 09:56:59 +0000 UTCThank you!
Christo Wilken
2023-12-18 08:52:32 +0000 UTCFascinating work. Thank you for this.
Shaun McDonogh
2023-12-18 04:15:42 +0000 UTCI find it interesting that the evidence hints that synthetic data could assist in runaway progress, at least in the short term. Yet he was reluctant to speculate on "AGI" dates. 🤔
GGuy
2023-12-18 01:10:44 +0000 UTCThanks for the interview! In case others were wondering, the SLM + verifier paper he mentioned: https://arxiv.org/abs/2312.09241
Younesse Kaddar
2023-12-18 00:54:51 +0000 UTCAdded links!
Philip
2023-12-17 21:32:14 +0000 UTCThanks Christo! It's MedPrompt: https://www.microsoft.com/en-us/research/blog/the-power-of-prompting/, plus this: https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/
Philip
2023-12-17 21:27:58 +0000 UTCThank you, good interview! Add around 18:43 you say something about a “metprompt thing that Microsoft did”. I couldn’t quite make it out. Could you clarify what your meant by that?
Christo Wilken
2023-12-17 21:12:35 +0000 UTC
