Jarvis with weather, the internet, and Spotify!
Added 2024-06-30 03:12:10 +0000 UTCI’m thrilled to share the next step in our journey to create your own Jarvis. In my latest tutorial, we’re not just building a smart assistant; we’re empowering Jarvis to interact with various tools and APIs to make it even more useful.
What's New?
1. Access to Time and Weather:
Jarvis can now tell you the current time and weather conditions. This integration will help you stay updated with real-time information without lifting a finger.
2. Internet Searching:
Want to find an image or search for information? Jarvis can now browse the web and fetch the data for you. This feature brings a whole new level of convenience to your assistant.
3. Spotify Controls:
Imagine having Jarvis control your Spotify account. Play your favorite tracks, skip songs, or tell you about the song playing. This turns Jarvis into a full-fledged media assistant.
How to Get Started
Step 1: Follow the Basics
Ensure you’ve completed the initial setup of Jarvis as described in my previous post:
Speech Recognition with Whisper V3 using RealTimeSTT
Intelligent Interactions with GPT-4o Turbo
Speaking Back with OpenAI’s TTS
Skip the tutorial and run the code: https://github.com/Concept-Bytes/jarvis
Join the Journey!
This is just the beginning of what we can achieve with AI in personal assistants. Whether you’re a tech enthusiast, a developer, or someone interested in the future of AI interactions, your support can make a significant difference.
Comments
I've been really sturggling to get jarvis to respond to any of the commands properly. Spotify does nothing apart from ay I can't control music and weather is telling me 20c when it is actually 8c. This is with the latest code in github both using OpenAI and Ollama. Not sure if I am doing something wrong though. The speech to text is also not working very well so need to look into that more as I think I am speaking clearly enough aha
Ben Hussenet
2025-01-28 12:10:47 +0000 UTCYes! You can use stuff like PyTTSx3 or gTTS. I believe gTTS is free, and they have amazing voices too, and is cloud-based. You'd receive a million characters monthly for free to use with gTTS, and you can use PyTTSx3 infinitely as it runs locally on your device. Keep in mind there are other alternatives, these are just popular options
Berle
2024-12-03 23:02:46 +0000 UTCHi, Is there any free and local alternative instead of openai tts?
Francesco
2024-10-30 07:44:19 +0000 UTCI haven’t made the Holomat, I only coded the Jarvis part. I was on the wrong video when I posted that comment. Sorry.
Oliver
2024-10-14 19:30:50 +0000 UTCdo you have a list of materials used in making the holomat itself. Having very hard time finding it. Also is there any guide on the hardware assembly or design guide.?
kshitiz
2024-10-14 19:29:27 +0000 UTCCan anyone give me a hand. I am trying to get a detailed list of what i need to build the holomat and am having a heck of a time. Maybe im looking in the wrong place? What is being used as the projector and tracking camera?
Irish Justin
2024-09-26 23:11:19 +0000 UTCOk so what I did to fix this is I put in a variable called update_knowledge = “” right above current_text = recorder.text() on the Jarvis.py. Inside of the quotes you can write a string of anything you want to tell it. Then scroll down on the Jarvis.py page when you see the last current_text variable in use. And replace it with (update_knowlege) then run your code and just say the wake word and it will answer all that text you put in so you don’t have it cut you off when you talk.
Oliver
2024-09-18 02:02:00 +0000 UTCHey guys, mine works absolutely amazing, but I’m not sure if I’m missing something. How do I get it to play specific songs? I want to be able to say play this song and have it play but it will only play, pause, skip, go back and so forth. Thanks for your help.
Oliver
2024-09-18 01:57:14 +0000 UTCHe's human and life happens. I know I have to work things like this around work and other day to day things. I would assume it's the same for Concept_Bytes.
Alex
2024-09-18 00:52:45 +0000 UTCWhy don't you use a .env file to hold your API keys, client ID, and client secret? Thanks for the walkthrough of the Spotify code. I couldn't get the copy from GitHub to work, but after following along with this video I have a working copy I can apply to my virtual assistant. Thanks again!
Alex
2024-09-18 00:49:01 +0000 UTCI'm curious what microphone you're using? i don't think I've seen that piece of hardware documented. I'm currently using the mic on my Logitech webcam, but I seem to have intermittent trouble with it. Thinking about a better solution. any tips?
Brian Turntine
2024-08-29 02:19:01 +0000 UTCcan I get some help on this too
Kingoctopus888
2024-08-20 01:57:19 +0000 UTCYeah mine too why does this happen?
Kingoctopus888
2024-08-19 12:36:43 +0000 UTCSo I used the basis of your code to create my own AI Assistant and everything works great! Except...for whatever reason, it refuses to pull images and use other external apps or features. It keeps telling me it can't do it. I have the assistant set up in OpenAI using the gpt-4 turbo, and I have set one as a text format and another as a json object, but both refuse to utilize external functions. Do I need to use a different gpt model or is there an issue with the code I am not seeing? Any help would be greatly appreciated! Love your work by the way!
Trav
2024-08-18 23:37:19 +0000 UTCFr
Hi
2024-08-17 00:46:13 +0000 UTCIt's been a month and a half.
Hi
2024-08-17 00:45:51 +0000 UTCUhm that should be stated before you charge patrons and advertise a weekly tutorial promise
Sheetyman Sheety
2024-08-17 00:42:54 +0000 UTCEveryone needs holiday sometimes...
Sascha Peth
2024-08-14 13:27:13 +0000 UTCthank you, i followed that but it just says: \openai\_client.py", line 105, in __init__ raise OpenAIError( openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
bb
2024-08-06 21:27:37 +0000 UTCin the .env file declare OPENAI_API_KEY = "" and enter your api key......also add it to .gitignore if you're planning to upload on git so no one knows your API key
Sounic Akkaraju
2024-08-06 10:07:33 +0000 UTCcreate a .env file in the root directory.....pip install python-dotenv
Sounic Akkaraju
2024-08-06 10:06:36 +0000 UTCheya, i've been stuck with the error: openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable. and dont know how to fix it, could anyone help?
bb
2024-08-06 08:44:08 +0000 UTCYes, you can do it on a raspberry pi too
Matteo
2024-08-05 04:54:57 +0000 UTCYeah like the 3D printing and stuff that was crazy!!! Please integrate JARVIS
Designer123
2024-07-31 16:01:13 +0000 UTCHi we've paid for a whole month with no additions or communication ?
Hi
2024-07-30 05:35:54 +0000 UTCWhy there has been no other videos lately? It would be nice to see the continuation.
Sardor Ibrokhimjonov
2024-07-29 05:41:02 +0000 UTCis this possible to run in a raspberry pi? or must be on windows?
Verom3D
2024-07-28 12:21:28 +0000 UTCOkay, thanks i'll try that 👍
Brian Lund Kofoed
2024-07-28 07:13:37 +0000 UTCnot sure if you fixed this yet or not but this is what worked for me. Install the 'Desktop Development with C++' components using Visual Studio Build tool. Locate the directory where MSBuild is installed (typically "C:\Program Files (x86)\Microsoft Visual Studio\20xx\BuildTools\MSBuild\Current\Bin") and add it to the PATH variable. Install cmake (pip install cmake) if not already installed. RESTART you computer. The error should be fixed now.
Jesse Meese
2024-07-28 00:39:49 +0000 UTCanytime i try to ask for the weather it instatanly breaks my jarvis with like 20 different code error that i cant fix an example of one is " in _request raise self._make_status_error_from_response(err.response) from None openai.BadRequestError: Error code: 400 - {'error': {'message': "[{'type': 'string_too_short', 'loc': ('body', 'input'), 'msg': 'String should have at least 1 character', 'input': '', 'ctx': {'min_length': 1}}]", 'type': 'invalid_request_error', 'param': None, 'code': None}}"
Tazer Grazer
2024-07-25 22:35:33 +0000 UTCWould you in future be posting a tutorial on smart mirror?
Krishna
2024-07-21 08:12:30 +0000 UTCJust joined...will you be posting a tutorial on the smart mirror?
Jonathan Israel Arana
2024-07-17 07:17:07 +0000 UTCHey Concept Bytes for some reason my AI can’t tell the time and I did exactly like you did
Elias Mendez-Mejia
2024-07-10 04:27:02 +0000 UTCDoes anyone know how I might change where Jarvis will keep listening and answer within a certain amount of time, instead of only if he asks me a question? Like the skip hot word part of the code. Like if I wanted him to keep listening after like 10 seconds of us interacting, so I don't have to say the hot word again so soon
Doug Reed
2024-07-09 23:37:20 +0000 UTCCan you show me how you did it? We've already written on Discord.
Kevko
2024-07-09 20:22:13 +0000 UTCYea, there is a way to control your smart home devices via the Home Assistant API. I'm currently building a script that can do this. The script itself can do it, I can already control devices with it, but only manually. I need to connect the script to Jarvis soon.
Kevko
2024-07-09 20:20:11 +0000 UTCjust play your music by your self for the first time and let your ai pause the music. after that he can control it.
Cebo
2024-07-07 14:47:57 +0000 UTCHey concept bytes, I’m having trouble with the Spotify connecting. I followed everything it all works however, the Spotify says: RealtimeSTT: spotipy.client - error- HTTP error for PUT to https://api.Spotify.com/v1/me/player/play with params: {} returned 404 due to Player command failed: No active device found. Not sure what I need to do to fix this. I’ve tried a few things but it’s just not playing not even my trusty ChatGPT knows😭😭
Oliver
2024-07-07 05:33:50 +0000 UTCHello! When can we expect updates for Holomat with Jarvis or the integrated apps that you frequently shared on Instagram?
LF
2024-07-05 18:03:45 +0000 UTCplease help (Python 3.10)
Brian Lund Kofoed
2024-07-05 07:47:28 +0000 UTCI cant pip install webrtcvad... in VS code.. Building wheel for webrtcvad (pyproject.toml) ... error error: subprocess-exited-with-error × Building wheel for webrtcvad (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [19 lines of output] running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-cpython-310 copying webrtcvad.py -> build\lib.win-amd64-cpython-310 running build_ext building '_webrtcvad' extension creating build\temp.win-amd64-cpython-310 creating build\temp.win-amd64-cpython-310\Release creating build\temp.win-amd64-cpython-310\Release\cbits creating build\temp.win-amd64-cpython-310\Release\cbits\webrtc creating build\temp.win-amd64-cpython-310\Release\cbits\webrtc\common_audio creating build\temp.win-amd64-cpython-310\Release\cbits\webrtc\common_audio\signal_processing creating build\temp.win-amd64-cpython-310\Release\cbits\webrtc\common_audio\vad "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -D_WIN32 -Icbits -IC:\Users\brian\AppData\Local\Programs\Python\Python310\include -IC:\Users\brian\AppData\Local\Programs\Python\Python310\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0\shared" /Tccbits\pywebrtcvad.c /Fobuild\temp.win-amd64-cpython-310\Release\cbits\pywebrtcvad.obj pywebrtcvad.c C:\Users\brian\AppData\Local\Programs\Python\Python310\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.40.33807\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for webrtcvad Failed to build webrtcvad ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (webrtcvad) PS C:\Users\brian>
Brian Lund Kofoed
2024-07-05 07:46:53 +0000 UTCyou can also try and upload your files to the assistant, then tell it to analyze them. the ai itself could easily learn how to answer
Ricardo Lesmes
2024-07-04 21:15:51 +0000 UTCyou should look up if the home assistant has an api. in that same sense. check how the assistant sends and receives commands, and try and simulate it
Ricardo Lesmes
2024-07-04 21:14:01 +0000 UTCeverything is perfect but still there was response on weather it cannot provide realtime data can somebody help me with that
WhiteBoss Gamer
2024-07-03 07:12:34 +0000 UTCassist.ask_question_memory('I have some commands for you, when I ask for the weather use #weather in your response to fetch weather data. If I ask you to play a song use #play in your response to play music. If I ask you to pause or stop a song use #pause in your response to stop the music.')
Klynt
2024-07-03 02:41:22 +0000 UTCI ended up adding the below "ask_question_memory" before the while loop to initialize some commands. The weather works a little but it says it cant play music, maybe I need to reword it?
Klynt
2024-07-03 02:40:57 +0000 UTCSome of the # commands aren't working the best, is there a way to hard code it into the software? Instead of using playground or with TTS. In playground it said it understood the commands but when I run the script it doesn't understand and says it can't retrieve that info. When I ask for weather it will say it cant get it, then adds #weather then retrieves it. Seems like the thread ID might be changing or not using the specified one.
Klynt
2024-07-03 02:36:16 +0000 UTCYea it’s the Jarvis.py, now my original one I did with your video works, only this new one with 20 lines
Spyder Flag Check
2024-07-02 21:46:06 +0000 UTCHmmm I'm familiar with eleven labs but I was unaware it was used in this project. Do you see what script is throwing that error?
Kevin Coda
2024-07-02 21:45:22 +0000 UTCI'm glad you think so. This functionality shocked me too. I've got more pieces to this puzzle I'm working on. Let me know if you got questions or need help!
Kevin Coda
2024-07-02 21:44:23 +0000 UTCThis is incredible! Thank you for this. I am implementing each piece and it amazes me what it can do and how easy it is.
TheK-B1000
2024-07-02 14:00:32 +0000 UTCHey team! I’m building the new Jarvis but running into a “ImportError: cannot import name generate from elevenlabs” I don’t see in the script or requirement for this, how can I add that import for this to work? I found some stuff online but nothing is sticking
Spyder Flag Check
2024-07-01 14:20:15 +0000 UTCand i am using gpt 3.5 turbo too and yeah i named him aiden
RafaCrackPL
2024-07-01 11:58:43 +0000 UTCso i actually went on the playground and pretty much this is what shows up User Yo Jarvis, so like you used to search the term query, you can do like just like you did it before, hashtag search, and then minus the query. Instead of the query, go and search for a Raspberry Pi, Arduino, or a Flipper 0, pretty much whatever you want. But you can go and show me for example Raspberry Pi 5. 2024-m-01 13-56-04 aiden Got it, Marcus! I'll use the format #search -Raspberry Pi 5 to conduct a search for an image of a Raspberry Pi 5. Let me start the search now. aiden It seems I encountered a technical issue while trying to perform the search. Let me attempt the search again for an image of a Raspberry Pi 5.
RafaCrackPL
2024-07-01 11:58:20 +0000 UTCHey, that didnt work either, now he sometimes does gympie and he speaks super like weirdly he talks about the lushious lands and stable like a golden retreiver. is it possible in anyway to fix this or do i need to keep trying to fine-tune him???\
Jacob Mrgroth
2024-07-01 02:31:06 +0000 UTCTry going to your open AI assistant dashboard and try typing the commands. It might take him seeing the #search to actually use it. Also add an example in his setup prompt telling him that he is an assistant that can use commands.
Kevin Coda
2024-07-01 02:04:05 +0000 UTCFor the second issue of it not understanding you, you may want to try adjusting the RealTimeSTT sensitivity. That might help. There are a bunch of other settings too. Maybe also tried other mics. It should work really well on a normal mic.
Kevin Coda
2024-07-01 02:01:57 +0000 UTCHey, so for your first issue, I'm not too sure what the problem might be, but here is the documentation I used to get started. It's super helpful for setting up a Spotify app: https://developer.spotify.com/documentation/web-api
Kevin Coda
2024-07-01 02:00:44 +0000 UTCthanks!
Jacob Mrgroth
2024-07-01 01:58:30 +0000 UTCHey, try going into your open AI assistants account. And type in the playground. This way you can type the # instead of verbally telling him. Then use the same thread ID
Kevin Coda
2024-07-01 01:57:24 +0000 UTCI cannot get the weather to work he refuses to use hashtags.
Jacob Mrgroth
2024-07-01 01:48:31 +0000 UTCHey Concept Bytes, I checked all the code I put in my clientID secret key and username port 8888 was open for tiny proxy on my MacBook so I set the redirect URI to port 8000 but everytime I start it and tell it to play a specific song or anything to do with Spotify it says it can't and redirects my screen to the uri it also barely makes it what I say correctly
Rithvik
2024-06-30 20:50:37 +0000 UTCwhen i ask jarvis to search the internet for a picture or just in general, he says he is not able to, and it doesn't work?
RafaCrackPL
2024-06-30 18:59:19 +0000 UTCYou updates the script?
Venell James
2024-06-30 17:42:30 +0000 UTCI've already set mine to match the projector and the webcam which are both 1920x1080 but this has not resolved. I did this as a troubleshooting step before reaching out here.
Vincent
2024-06-30 16:25:40 +0000 UTCHey I think they just changed their API. For assistants like within the last couple weeks. Check my latest code. Maybe upgrade openAi with a pip install.
Kevin Coda
2024-06-30 16:23:58 +0000 UTCI got my master's in computer science, I add another real time voice changer model on top of the TTS. I'll put that out soon
Kevin Coda
2024-06-30 16:22:14 +0000 UTCHi, yes it's very important to set your screen resolution. Mine was 1920x1200 but yours might not be.
Kevin Coda
2024-06-30 16:20:55 +0000 UTCYes. Even reinstalled OS to have a totaly clean pc to work with. Happend again.
Dutchfather
2024-06-30 16:20:40 +0000 UTCThat's strange are you using real time STT library
Kevin Coda
2024-06-30 16:19:42 +0000 UTCIt's crazy how fast he can learn to use them! It's super fun to make them and teach them. Let me know if you make any other tools!
Kevin Coda
2024-06-30 16:18:09 +0000 UTCThat's strange, it shouldn't have any access to weather. Maybe that's a new plugin with open AI assistants
Kevin Coda
2024-06-30 16:16:54 +0000 UTCIs there a way to get this to work with homeassistant?
Hi
2024-06-30 15:18:50 +0000 UTCHaven't seen the video yet.. but oddly after setting up whisper my assistant was able to retrieve current weather updates without any additional coding. No idea how it obtain that data. Low key had me spooked
Jonathan Luna
2024-06-30 14:36:14 +0000 UTCThe approach of creating the necessary commands and having Jarvis learn them is very shocking to me. Thank you for the lesson.
Enchic Gong
2024-06-30 13:24:42 +0000 UTCIm stil having issues with the initial setup with Jarvis recognizing speech without me saying anything. Have tried the setup on 3 different computers with 3 different hardware setups. Still wont work. It tells me im saying very,very,very,very,very,very,very all the time..
Dutchfather
2024-06-30 07:16:17 +0000 UTCThank you for the response! I have set the resolution to match my projector, and my webcam. The distance is *roughly* around 4 feet so far and it's well lit. I suspect it has to do with the shadow of my hand, but that doesn't round because I would have caught that during the run script (to verify it's mapped to my hand), as well as the hand calibration
Vincent
2024-06-30 05:48:00 +0000 UTCtry to set the screen resolution and also try to fix the cam at perfect distance, I attached the cam away from the table 5fit. HOPE IT WORK FOR YOU
WhiteBoss Gamer
2024-06-30 05:07:08 +0000 UTCHey there Concept Bytes. I've been following along but have been consistently having issues with the Holomat setup. Everything looks fine when when I run homescreen calibration - I touch the points, press enter. when I run the run script, I can clearly see the points are on my hands. However, when I run the homescreen app, the menu is not centered (but other apps are, the apps seem to be perfect). Additionally, my cursor isn't exactly where my index finger is, but to the bottom left (I'd estimate roughly several inches or so) Any advice? I've been troubleshooting on my own for several days but I'm fresh out of ideas.
Vincent
2024-06-30 04:06:41 +0000 UTCwhere did you learn to code? also how do you change his voice?
daniel zambrano
2024-06-30 03:29:49 +0000 UTCJust different stuff every time I try and run it. One major one is it will tell me “client.beta.” No attribute or something like that. Even though I have all my keys in place
Venell James
2024-06-30 03:27:52 +0000 UTCHey it should be able to run on a mac! What problem are you running into? I would love to help you!
Kevin Coda
2024-06-30 03:24:36 +0000 UTC😂 I’m still having trouble build Jarvis to begin with. I’m just chalking it up to using a Mac mini m1😂 I’m sure that’s not why but it makes me feel better
Venell James
2024-06-30 03:16:20 +0000 UTC
