And
Marlies Gillis: okay,
Marlies Gillis: I want to share this link again.
Marlies Gillis: I'm trying to copy the stitch.
Okay.
Marlies Gillis: but I hope everyone can access it normally, if you open it, you should see. And a window like this.
Marlies Gillis: And you need to log in with your own Google IP,
Marlies Gillis: so I'll
no.
Marlies Gillis: and then you should see the folder again.
Marlies Gillis: Can
Marlies Gillis: like. I don't know.
Marlies Gillis: There! There's an option
Marlies Gillis: to give some feedback like a thumbs up or so, whether it works whether I can go further.
Marlies Gillis: So
CNSP: I've got it just fine. I'm following along so far.
Marlies Gillis: Might it be like? I see people are still joining us guests or still joining the zoom. Would you be able to share the the link again.
Marlies Gillis: just to make sure they are
Marlies Gillis: being able. I'll do that. Okay, cool. Thanks. So
Marlies Gillis: whenever you're logged in, you can.
Marlies Gillis: they don't know how to. There's like.
Marlies Gillis: yeah, this little folder, and then you can make shortcuts to your own drive.
Marlies Gillis: So
CNSP: I just thought it. Maybe if people are having problems, or if they're a little slow or like, if they're taking a little bit longer, that you can just raise your hand
CNSP: that way, you can be more confident in in like
CNSP: knowing that people are following along or knowing that people are having problems. So maybe if you have an issue, just raise your hand and we'll come to you.
Marlies Gillis: Yes, because it's for me rather tricky to do this.
Marlies Gillis: without any feedback. So give me the shortcuts to your own drive.
Marlies Gillis: and then, if you go to your drive. Then you will see the workshop.
Marlies Gillis: and you can open it as Google Colab.
Marlies Gillis: And this should run. So I run. Anyway.
Marlies Gillis: this should run, and I suggest just to clicking. We're running until you're here.
Marlies Gillis: and then, if this is in progress, then I jump back to
Marlies Gillis: to the slides.
Marlies Gillis: So that's for you. I'm also going to open my version.
Marlies Gillis: Okay.
Marlies Gillis: okay.
Marlies Gillis: it's can I give some feedback on
Marlies Gillis: like, I give a thumbs up whenever it worked out for you.
Marlies Gillis: Like, you have normally in the Zoom bar like reactions, and then you can give a thumbs up just to make sure
Marlies Gillis: that I'm
Marlies Gillis: that are that people are following.
Marlies Gillis: Sometimes this happened. You see here the call app check. Just run it another time, and then it will.
Marlies Gillis: The error is resolved. I don't know. Sometimes errors happen. Unfortunately. Okay. So in my case, Gilbrain is installing. I hope.
Marlies Gillis: because I don't see lots of thumbs up. I
Marlies Gillis: whoops. Everyone is okay.
CNSP: Mine is still installing as well.
Marlies Gillis: Okay, if it's
Marlies Gillis: still installing is fine, because then II would like to have the opportunity to quickly give some more details that I didn't have time to cover during the during the workshop.
Marlies Gillis: So using this trfs, or neural tracking, you can actually, or the call is, you can actually go
Marlies Gillis: and describe the whole auditory pathway. and depending on the features
Marlies Gillis: or which features you select for your analysis, which is your yeah.
Marlies Gillis: the the thing you want to investigate
Marlies Gillis: depending on your choice. You can investigate different characteristics up to the auditory pathway. And actually, the Trfs gives you some additional insights. For example, fundamental frequency. Which is still to to cover support. You have very short latencies, envelope.
Marlies Gillis: and the Trf. Shows later latencies. And then for linguistic processes or linguistic features, you see even longer responses.
Marlies Gillis: So the goal of this workshop is first to create those features. That I've that I use during my teaching
Marlies Gillis: ranging from acoustics so suplexable and linguistic features. And then also this abstraction approach.
Marlies Gillis: So we will start first with creating acoustic features. So the spectrogram and acoustic onset.
Marlies Gillis: I
Marlies Gillis: So creating these features, takes quite some time.
Marlies Gillis: or also not only like your time, but also the computer. Sometimes it can be really heavy.
Marlies Gillis: and so
Marlies Gillis: I will quickly open every repository which will I share at the end of this workshop with all the quotes to create these features.
Marlies Gillis: So because, for for example, the spectrogram I have created and the feature a priori, so we can just open it and look at it.
Marlies Gillis: But I want to pretty show you the vote on how it works
Marlies Gillis: open repository that I that I will also share with you.
Marlies Gillis: And
Marlies Gillis: okay.
Marlies Gillis: so hopefully, you can successfully see this coach.
Marlies Gillis: can you?
Marlies Gillis: I see a question from, Yeah, okay, the Google call app is, is it worked for everyone to if you go to the
Marlies Gillis: you can normally go here and then open with. And if you're you don't have this Google collaboratory, you just say connect more apps. And you need to install the Plugin for a Google. Co-ab.
Marlies Gillis: Okay? I hope everyone is like on the same page with me. II worry bit that some people are licking behind and cannot follow the workshop.
Marlies Gillis: Matt.
Marlies Gillis: Okay, anyway. Jumping from here to Derekan.
Marlies Gillis: So in the repository that I've shared, there's a create acoustic features erez agmoni. So basically, in order to create feed to create an acidic feature, what you need?
Marlies Gillis: It's actually just the audio file file. Just the what file of the
Marlies Gillis: of the thing you presented. So
Marlies Gillis: the
Marlies Gillis: This is the link to the web file.
Marlies Gillis: I saw my slides that I forgot to introduce the datasets. So this is an open source. Datasets that we will use.
Marlies Gillis: It is our datasets of 85 participants listening to different stories or podcasts.
Marlies Gillis: And it's yeah, super interesting, because you have many different speech
Marlies Gillis: stimuli, and it's a very long protocol. So if you, you have multiple trials, per participants so definitely, if you have any full hypothesis, you want to check out, then definitely check out these datasets.
Marlies Gillis: So that's a dataset that that I will use, or that I used in the repository that I will share with you at the end.
Marlies Gillis: The
Marlies Gillis: yeah, way, more details on which subjects were included and so forth. Can be found so that you can really exactly re replicate. Whatever I show here in the
Marlies Gillis: in the
Marlies Gillis: yeah, in the workshop. Okay? So we basically just need the audio file. It is not directly shared as a profile in the repository. It's saved as an
Marlies Gillis: and like an empire, like a search kind of data format from numpy
Marlies Gillis: but I just it's like exactly the Waff, so you can convert you Ca, you just need to write it out as a waff to get it again to the waff, or just download
Marlies Gillis: this downloaded fire to slams the eggs at the scene.
Marlies Gillis: Anyway. So how how does this do? How do you extract the acoustic features to do so I use the trf, tool trf, tools, toolbox.
Marlies Gillis: It allows you to first load a web file. then then I filter the data because our insert phones. Have a filtering function.
Marlies Gillis: Then I resample it just to yeah. Make the computation time less.
Marlies Gillis: And then I apply gamma to a filter bank. So the gamma turn filter bank. It's like a human inspired
Marlies Gillis: filter bank, and you can say between which
Marlies Gillis: low and high frequency cut off. You want this. And how many channels you use? I typically, just use 256 channels.
Marlies Gillis: But this is yeah, this is based upon previous literature, but
Marlies Gillis: now I just divided it by 2, because otherwise my laptop couldn't run this
Marlies Gillis: for the spectrogram. So now you have the gamuto inspectogram, which has a lot of filter bands.
Marlies Gillis: And then you group the filter bends key into, for example, 8 pins, or whatever you're interested in. And then you apply the power law.
Marlies Gillis: and then we resample it through the to the
Marlies Gillis: original or or to the same sampling frequency as you.
Marlies Gillis: Okay? So now let's just visualize the
Marlies Gillis: visualize the or go further in the work or in a collapse folder and check
Marlies Gillis: how this crf looks like how the spectrum looks like sorry.
Marlies Gillis: in between. Also looking at the chats.
Marlies Gillis: is, is everyone looking like? Is it everyone for everyone working out to.
Marlies Gillis: Most people are on track. Okay?
Marlies Gillis: So okay, then just execute the the rest of the cells. So here it will ask you to mount the drive.
Marlies Gillis: And this is actually because within and
Marlies Gillis: within the folder that I shared shared, there's the data folder which will call where certain files will be called from in the.
Marlies Gillis: in the, from the co-lab file. So you need to allow it, and you need to
Marlies Gillis: login and supports.
Marlies Gillis: And if you did that.
Marlies Gillis: and normally it should be mounted.
Marlies Gillis: Erez agmoni, you should get this message. And this is important, because here's some cells. Later we will
Marlies Gillis: call the data. Oh, I have observed that I still have the wrong name here. But anyway,
Marlies Gillis: you first import the packages, and then
Marlies Gillis: you know
Marlies Gillis: you need to copy. I'll also paste it in the chat
Marlies Gillis: and change that woods
Marlies Gillis: foolish.
Marlies Gillis: That's work for everyone today
Marlies Gillis: is is the mounting successfully.
Marlies Gillis: Okay? Great
Marlies Gillis: so then we will visualize the acoustic speech feature.
Marlies Gillis: and if you run it you should see something like this. So you clearly see, like some sentences of
Marlies Gillis: of the the spectrogram, it added the different center frequencies which looks nice.
Marlies Gillis: Another interesting speech feature to look at is the acoustic onsets.
Marlies Gillis: Yes, acoustic onsets are
Marlies Gillis: the half wave rectified
Marlies Gillis: derivatives of the spectrogram.
Marlies Gillis: This actually accentuates all the edges of the spectrogram. And it's an important feature, because apparently the auditory brain can respond
Marlies Gillis: like, or has a more prominent response, studious edges than the spectrogram. So I always consider the acoustic onset or acoustic edges as well.
Marlies Gillis: so like as it's a handsome tutorial. I also included some exercises.
Marlies Gillis: So the first exercise is to make the acoustic homesets feature. So it's basically you take the derivative of the feature and then click all negative parts to 0
Marlies Gillis: and to help you with this
Marlies Gillis: check out the the ill bring documentation because the whole
Marlies Gillis: workshop will rely on the toolbox. Ill brain. And this spectrogram is already an nd variable so, and
Marlies Gillis: and ill brain object, so to speak, which has methods. To take the derivative and to keep the negative parts to 0.
Marlies Gillis: I will give you some time. Meanwhile, if there is any any question, or if you want to give any feedback on where you're at in the workshop, then, please.
Marlies Gillis: and let me know just either unmute or put it in the chats.
Marlies Gillis: And then, you know, pop like, in a yeah few minutes, then we we can.
Marlies Gillis: But
Marlies Gillis: ethnic existence.
Marlies Gillis: Yes, installing eel brain takes quite some time. That's also like.
Marlies Gillis: if you ever want to install, you'll bring use. Mumba Mumbai is really a time saver?
Marlies Gillis: because taking using one left search or whatever it takes, quite a
Marlies Gillis: long time to install the brain.
CNSP: Do
CNSP: these methods work in a similar way to Matlab, and that if you have something with, I'm assuming the spectrogram. Channels are organized along columns. Does it do the thing along columns.
Marlies Gillis: Well, you if you go to maybe it's in there. And I didn't see
Marlies Gillis: and you, for example, want to take the derivative.
Marlies Gillis: Then you can really say the dimension and the dimension is really super straightforward time frequency. And it's
Marlies Gillis: yeah. In this case you want to use time
Marlies Gillis: as a spoiler. But
CNSP: you want to use what
Marlies Gillis: you you need to take the derivative overtime.
Marlies Gillis: And you will have. Yeah. So
Marlies Gillis: okay, yeah, let's I'll I'll
Marlies Gillis: I'll make this first exercise. So you
Marlies Gillis: oh, I see there's an yes, I sorry. For the mounting stuff I last minute. Yesterday we named the Cnsb underscore workshop. So in the
Marlies Gillis: this one data boots, so you should rename it to this.
Marlies Gillis: if you replace it, the Csp workshop. It's
Marlies Gillis: tutorial check one, then it should solve.
Marlies Gillis: Brian, can you let me know if it worked out and then
Marlies Gillis: so back to the code? So if you want to.
Marlies Gillis: take the so good goal.
It's a
Marlies Gillis: we make a new variable. We use a spectrogram, and then we
Marlies Gillis: take the derivative over time.
Marlies Gillis: Next. You can also use the the clip function. And it's you. You basically tell it's like to keep all to clip at 0. So all negative parts will be clipped to 0.
Marlies Gillis: You can run this and then add another so, and I can, for example, plot it.
Marlies Gillis: and
Marlies Gillis: I'm going to take the scene.
Marlies Gillis: So the sub function is, it's basically and
Marlies Gillis: telling them to take. Yeah, only like a very small part of the data, because otherwise, yeah, it's completely ready to.
Marlies Gillis: which is a bit more.
Marlies Gillis: Now, you see, it's it's way, more sparser than the spectrogram, which is normal, because, like the
Marlies Gillis: if, if the
Marlies Gillis: spectrogram would be like this, the edges would be more like this.
Marlies Gillis: So yeah, in a in a next exercise, I quickly wanted to show you how the
Marlies Gillis: which apparently I already spoke the answer so, and to show you the relationship between the spectrogram and the edges. So if you just
Marlies Gillis: Yeah, uncommon the first line. So either you do, you can take the the main
Marlies Gillis: of over across the frequency like this. So I mean
2, 4,
Marlies Gillis: okay? And so if you run this, then normally, you can see the relationship between 2.
Marlies Gillis: So
Marlies Gillis: Brian, sorry.
Marlies Gillis: So you changed the data route paths
Marlies Gillis: like this.
Marlies Gillis: and and from then on it should
Marlies Gillis: it should run normally and does it work out for the other participants.
Marlies Gillis: I hope it's working out for you guys.
Andre Palacios Duran: you might need to respeify other parts of the Route Directory name depending on where you saved it in your drive. So like I didn't save it in my drive. I saved it in a sub folder of my drive called Google Colabs Cause. Then it. When I added shortfit, I needed to put in another folder.
Andre Palacios Duran: if you use that.
Marlies Gillis: Yeah, thanks. That makes sense
Marlies Gillis: good.
Marlies Gillis: And does that fix the trick.
Marlies Gillis: anyway? If it's not working out, II will share
Marlies Gillis: like this will be available. The Google drive will still be available until Caleb kicks me out.
Marlies Gillis: And
Marlies Gillis: all the quotes is on my kid lab. So you will have the resources to replicate this.
Marlies Gillis: Okay? So
Marlies Gillis: then, this is about the audio feature. So basically, we have the spectrogram acoustic on sets. That's sufficient
Marlies Gillis: acoustic information. We went already to the quote. So we can move on. And next up are the suplexical features.
Marlies Gillis: So basically, we want to you to make this sparse area. Of features. Where there's a spike at every onset of a phoneme or
Marlies Gillis: or word on 6.
Marlies Gillis: So basically, what do we need to do? So you need to have a transcript. So you need to know the words which were spoken during the audio fragment
Marlies Gillis: and also you need to have a text script, file or like it shouldn't be necessarily a text. Good file. It can also be another file. But basically it tells you when exactly a phone name was spoken.
Marlies Gillis: So first, maybe a bit more on this vertex. So
peanuts. this link
Marlies Gillis: whisper X, and basically what you just should do. But we don't have time to go through it right now. But I give you all the tools
Marlies Gillis: just install this repository. It's very, very nice. And then
Marlies Gillis: The install installation instructions are quite straightforward and well explained. So normally you should be able to manage.
Marlies Gillis: And then, basically, what I do is just running this line.
Marlies Gillis: in in command line, where, in my case, this is nl, because I'm I'm using Netherlands Dutch.
Marlies Gillis: and the name of my wife, and then it will out, put you like this. And in the end really a text file with all the the lines which has have been spoken.
Marlies Gillis: which is really really nice. I noticed tool only for the last
Marlies Gillis: 3 months or so. So. The majority of my Phd. Have been transcribing a lot of stimuli. So this will save you crazy amount of time time. So that's really, really cool.
Marlies Gillis: The thing is, it's super nice, but there are still stakes in it. So just make sure you also do manual correction afterwards.
Marlies Gillis: Then, secondly, is to get that phone segmentation. And that's actually also not super difficult. There's a
Marlies Gillis: like in my series. We only we always use our own calendar stuff. But
Marlies Gillis: this bus web service is also super super cool.
Marlies Gillis: So basically, what you do is and I think it's another.
Marlies Gillis: You include the the text.
Marlies Gillis: the output of the whisper X, and you also include the waf file. You open them, you can upload them. And then basically
Marlies Gillis: after after it's uploaded, and you change
Marlies Gillis: the the language to touch, which is the language of the waffle fragments. You can run this, and it will output you a
Marlies Gillis: it will output you the the text read, file
Marlies Gillis: this like, yeah, I. I'm not going to do this right now, because this again takes a couple of minutes. But now we have the tools to make your own text grids, files, and you see it's not rocket science. It's
Marlies Gillis: quite nice and easy.
Marlies Gillis: Then again.
Marlies Gillis: if you want to go for it, then I also suggest you use paths.
Marlies Gillis: I'm quite convinced.
Marlies Gillis: A lot of lots of people.
Marlies Gillis: No, Pratt already. And but to make sure that opens.
Marlies Gillis: We know, like this, where you can really few open the text scripts and the
Marlies Gillis: what file, which I'll do. So the textbook and the web file
Marlies Gillis: very much.
Marlies Gillis: You can really open them together, and you can really play around with all of this, and you see in the beginning, maybe I'm quickly going to check whether audio sponsors shared
Marlies Gillis: and look shorter. Can you hear us?
Marlies Gillis: Nope. anyways.
Marlies Gillis: here. What I'm hearing is music. So here the the segmentation is not done right later. It's it becomes it becomes good.
Marlies Gillis: And so just
Marlies Gillis: make sure you check your segmentations, your transcript, make sure to check them afterwards for music. I would suggest to, for example, with audacity. Cut out to music to make sure your volume segmentation is done right?
Marlies Gillis: So as you see. Making okay. making a transcript nor a text switch file, it's not rockets science, so you can do it, too.
Marlies Gillis: Okay, through the code, then.
giorgia cantisani: hey, Melissa, before you go on, there is a question in the chat.
giorgia cantisani: So it's asking about acoustic on sets. Is there a recommended paper investigating the relation of the acoustic concepts with actual linguistic elements
Marlies Gillis: and
Marlies Gillis: the the the paper that I would recommend is from broderick.
Marlies Gillis: Sorry, Broadbeck.
Marlies Gillis: It's not necessarily related to linguistics, because linguistics is still bit of a new field. But paper is super interesting, because there they compared
Marlies Gillis: How how do you listen to an attendance and an ignored speaker? And what they see is that whenever an attendant speaker is masked by another like an an ignored speaker, or distracted by a distractor, then you can actually see that the ma, the the acoustic
Marlies Gillis: masked on sets get unmasked in case they are attended to, but not in case there they are. Ignore to. I hope this answers a bit of the question. I think
Marlies Gillis: there's like, I just want to highlight the importance, I think, including acoustic on sets, is really important because of this feature, but I don't know the exact
Marlies Gillis: relation with linguistics, because
Marlies Gillis: in my or on the top of my head, I don't know a paper that's directly investigated. This.
Maya מיה مايا: Yeah, okay, great thanks. yeah, I would. I know the paper. And
Maya מיה مايا: like, I know that it is commonly used feature. But I'm just actually curious, like
Maya מיה مايا: as with the many other features that we often use zoom correspondence.
Maya מיה مايا: who like linguistic elements, but like the
Maya מיה مايا: like, yeah, I would like to know? Better.
Check that.
Marlies Gillis: Thanks for your question. Thanks.
CNSP: Marlene. Do you mind scrolling up for just a second to get the end of that.
Marlies Gillis: I am. Yeah, sorry. This is basically the uncommon like, it wasn't a common. So you should just remove
CNSP: silly sorry. Keep going.
Marlies Gillis: So yeah, then there's a really nice toolbox which basically allows you to
Marlies Gillis: by reading this textbook. So although I already made it like as an ill brain.
Marlies Gillis: or I made already all features for you. But maybe it might be nice to make and and d variable ourselves. So basically, we're first going to loads and the text scripts.
Marlies Gillis: so you you have the. Here's where the textbook is safe, the paths and the sampling frequency of the Eeg.
Marlies Gillis: Then what you see, whenever you load the text grids using the yeah, the text grids toolbox. then you will see that you basically get an output of
Marlies Gillis: yeah, a dictionary with 3 keys. So you have the automobile. You have the camera and you have the Mao and
Marlies Gillis: In automobile. You. So you have the words.
Marlies Gillis: any. So, for example, if you do automatically, you have a lot of intervals.
Marlies Gillis: And each interval still tells you which words has been set
Marlies Gillis: and when the onset of that word was, and when the ending of it was
Marlies Gillis: so sorry.
Marlies Gillis: So to extract the word onset time. Basically, you you loop over all these intervals in sort, mouse, mao, and elements.
Marlies Gillis: And you just take X-men.
Marlies Gillis: Sometimes there are empty words. And to really empty. So those you discard because those are those are silences and those you don't want to get an onset phone.
Marlies Gillis: Then how it'll bring works is you need to couple a lot of or you first need to create a dimension before you can create a feature. So in this case, we're creating our time
Marlies Gillis: dimension.
Marlies Gillis: which is from 0 to the amount of samples that we require, and then the sampling rate that the feature should be.
Marlies Gillis: And then next up, we create the instance.
Marlies Gillis: and from all zeros. So we make a feature with all zeros, and then at the timing of the word on sets. We just put a one
Marlies Gillis: erez agmoni. And then you can, of course, visualize this. So basically, what I visualize here is the the spectrogram which is averaged across frequency bands. Hence it's only one band.
Marlies Gillis: and then I also visualize the word on set features or hearing who. So whenever there's a small peak here. There's
Marlies Gillis: okay. So
Marlies Gillis: next up as a next hand-on hands-on exercise. I challenge you to make phonium onsets.
Marlies Gillis: It's very, very similar to word on sets but you should take the Mao
Marlies Gillis: instance
Marlies Gillis: and II give gave you already an hint that unknown utterances are given by these elements in the
Marlies Gillis: yeah, by by the textbook.
Marlies Gillis: So I'll give you some time to to make this exercise. Meanwhile, if there are any questions or issues still. Then
Marlies Gillis: please. Let me know also, Brian, can you give me an update on whether
Marlies Gillis: whether it worked out
danna pinto: when you I have a question about the phoneme segmentation.
danna pinto: The actual segmentation through mouse
danna pinto: and the manual changes. How
danna pinto: how exact would you say?
danna pinto: Do you do this segmentation because we found that when we were trying to manually correct it, sometimes like auditorially, it's hard
danna pinto: to find the exact boundary of the phonemes, and we do it. We we separate between words, syllables, and phonemes. So we try to use the other boundaries to help us. But.
danna pinto: do you have
danna pinto: suggestions about how maybe to improve on that? Or is that just something that sort of rely on?
danna pinto: more like, let's exact.
Marlies Gillis: Yes, that's indeed very tricky, because some phoning say they start rather silence
Marlies Gillis: like for it, or II don't know whether this is a correct example, but, like.
Marlies Gillis: like, you know. When you say, Hello, like.
Marlies Gillis: there's no super hard edge at the beginning that's indeed, tricky, I,
Marlies Gillis: i usually do this? Together with colleagues who are audiologists and which are who are really trained. In in doing stuff like this.
Marlies Gillis: yeah, it's it's also something that you learn through practice. It might be.
Marlies Gillis: It might be nice, maybe, to have
Marlies Gillis: a good reference system to make it more uniform. Because, yeah, II rely on how colleagues taught me to and to do it a lot with colleagues who have way, more knowledge. And then I have, because I'm not really trained to use.
Marlies Gillis: II know it's it doesn't 100. Answer your question.
danna pinto: No, it's good to know that everyone is in the same. Isn't the same realm like everyone's tackling the same challenges.
Marlies Gillis: Yeah, hoping that you have a good phoning segmentation tool like the bus services. Or so.
danna pinto: so we don't. We can't use the mouse because we are. Hebrew isn't on the as one is not one of the languages that it offers.
danna pinto: So we needed to find different different methods.
Marlies Gillis: Yeah, previously, we also used in house a thing a guy created a couple of years ago. But
Marlies Gillis: yeah, it's it's
Marlies Gillis: It's always tricky when you're not English speaking. II feel hopefully. hopefully, the bus services have they? They do suggest something.
Marlies Gillis: They do have some suggestion on, on how to deal with language languages they don't recognize, but
Marlies Gillis: I never checked that out, so I don't know whether it's working to to spend time to check that out.
Marlies Gillis: Okay, let's make this exercise. So basically, I'm
Marlies Gillis: just going to use
bye.
Marlies Gillis: So basically
16
Marlies Gillis: this one this time. I only want them. Now, instance,
Marlies Gillis: we don't want it to be like one of these.
Marlies Gillis: Isn't some of what we
CNSP: was also having problems with copy and paste.
Marlies Gillis: Yeah, weird.
Marlies Gillis: Normally, it's not such a pain in the
Marlies Gillis: and lights
Marlies Gillis: for the bigger
Marlies Gillis: syntax. because I shouldn't do the books.
Marlies Gillis: Okay? And then we can create.
Marlies Gillis: And
let me post. Yeah, I can.
Marlies Gillis: Sorry.
Marlies Gillis: And
Marlies Gillis: there is.
Marlies Gillis: And
Marlies Gillis: spectrogram of which.
Marlies Gillis: Then we leave
Marlies Gillis: the one on 6
Marlies Gillis: the
Marlies Gillis: but for this first
26,
Marlies Gillis: you know, and maybe I'll immediately multiply them by 2, so we can see them.
Marlies Gillis: Course new
cool.
Marlies Gillis: If you put another brackets in there on the same.
Marlies Gillis: Hey? There? So
Marlies Gillis: then, you see phoning onsets which are slightly higher than uses.
Marlies Gillis: Okay, cool.
Marlies Gillis: nice. I hope that worked out for everyone.
Marlies Gillis: So yeah, we created now just the acoustic features already this is flexible features.
Marlies Gillis: so let's go quickly over the linguistic features. So basically, we want now impulses with the amount of information
Marlies Gillis: for each volume or words.
Marlies Gillis: So first, a very conceptual
Marlies Gillis: description of what these phoneme feature features are before we're going to go really into the.
Marlies Gillis: And
Marlies Gillis: yeah, we're not really going to delve into the mathematics. So that's maybe good. So. But imagine whenever you hear the sound bar there are a lot of words starting as well. With a bar, and the idea is that our brain activates all the active words
Marlies Gillis: and based upon this active words. You can actually calculate, for example, the the flowing surprises. And so how likely the B is in this whole overall of words.
Marlies Gillis: or you can calculate the cohort entropy.
Marlies Gillis: And this is actually the Shannon entropy. So basically, what this formula is
Marlies Gillis: of these, of the compatible.
Marlies Gillis: yeah, words in cohorts. It's basically the degree of competition. You should see this. If there are a lot of
Marlies Gillis: likely words
Marlies Gillis: if there is. If if in the chords of activated words are lots of words which have a high word probability.
Marlies Gillis: then the degree of competition of which word it's actually is going to be is high. But if the court, if there is only one item which has a high probability, and all the other items are fairly low.
Marlies Gillis: Then, of course, you have a low degree of competition, and your cohort entropy is low.
Marlies Gillis: The general trend of these phoning features is in the beginning, and the first phonemes, in a word, have high phoning, surpriseal, and covert entropy.
Marlies Gillis: and the more towards the end of the word, the smaller the value becomes. That's the general trend of how it works, because then you have a better prediction on what the word is actually going to to be. For example, if I tell lf, then you are probably aware that the word is going to be elephant, and the last volumes
Marlies Gillis: don't really matter that much.
Marlies Gillis: I in my studies I always computed it, and using just the word probabilities of the standalone word.
Marlies Gillis: So independent of the context of the previous words. So the code that I share with you is also using the word probabilities of the independent words
Marlies Gillis: so in order to create these linguistic features, we, of course, need a pronunciation.
Marlies Gillis: Add dictionary
Marlies Gillis: a pronunciation dictionary is a mapping of a word, and how the word would be pronounced
Marlies Gillis: So to to create this, to create create a a pronunciation dictionary, we, you again have have this
Marlies Gillis: bus to go.
Marlies Gillis: There is the P, and basically you just input
Marlies Gillis: input a file with all your words that you that where you want a phonem segmentation from and in included here. So based upon my previous studies, I always use the word frequencies of scripts.
Marlies Gillis: I think I also have a slide on this.
Marlies Gillis: I am here, so I use the word probabilities on Sotalex. and this is also
Marlies Gillis: available in lots of languages, not sure whether it's available in in Hebrew. But anyway, any
Marlies Gillis: any word, frequency measure that you have will work for this. You just
Marlies Gillis: yeah, need to extract it another way. So basically, what I've done in in my code.
Marlies Gillis: and
Marlies Gillis: is first to say the words from subtext. This is a very like
Marlies Gillis: stupid script, but it's written sutilix. And it just takes all the words. And yeah, remove some which have like are very infrequent, because for often they are very weak words that I've never heard of, or have very weird symbols, and it's just to PIN it up a bit.
Marlies Gillis: And then there is
Marlies Gillis: a maximum amount of words per file that you can insert in the bus web application. So then I just write multiple files. And then basically I have.
Marlies Gillis: And I get
Marlies Gillis: this subtext words, and that's then the file which I insert in this in this past tool. And then, after a couple of minutes.
Marlies Gillis: it's it gives me this
Marlies Gillis: this file. Where you have a words. And then the volume segmentation.
Marlies Gillis: So, yeah, so based upon this, you can basically go ahead and compute word surpriseal and
Marlies Gillis: Gordon entropy.
Marlies Gillis: If you don't want to do all the minimics or I have
Marlies Gillis: II wrote these functions which you can find in the Github Repository.
Marlies Gillis: and that you can use to
Marlies Gillis: calculates your own phoneme linguistics. But for now yeah, I don't know whether it should.
Marlies Gillis: But from now for this workshop I just provide you the outcomes of these models, because also
Marlies Gillis: creating these
Marlies Gillis: like phoneme calculations, it, for example, it takes me
Marlies Gillis: like it takes almost a full night before this 15 min long. Audio book is fully that I have the the phone in linguistics for this audio book. So it takes a while. So it's not possible to do this during this workshop.
Marlies Gillis: Now you have to vote.
Marlies Gillis: and that at word level. So this is the example from previous. So
Marlies Gillis: at word level, I use word surprises. So basically, it's a probability of a word given the book for previous words.
Marlies Gillis: to do so. To do so, I use a simple engram model. Unfortunately, I haven't found a way to share it with you yet, because it's 2 GB, and it's it's
Marlies Gillis: and too big to share. But if you're interested in the Dutch version, then just send me a message.
Marlies Gillis: The word probability is just. Yeah, the probability of the words.
Marlies Gillis: So as I said, I use an enron model.
Marlies Gillis: this is yeah, a fairly simple model. And to create this, I use the NLLM. And Toolkits
Marlies Gillis: where you can write. Yeah. The the Github page is is, it's pretty
Marlies Gillis: yeah, extensive on how you should train a language model, and it
Marlies Gillis: results or gives us an output and output model which you and which you will use to estimate the word probabilities.
Marlies Gillis: So in the in the file.
Marlies Gillis: And
Marlies Gillis: nice here, I think so. You loads and the appa model, and then you can really using this Appa toolbox. you can just
Marlies Gillis: gets the word probability with this very simple line of quotes, or the probability of a word given the previous words.
Marlies Gillis: But those are already created because it takes a while to complete these features. But we can look at them and load them
Marlies Gillis: standards.
Marlies Gillis: So here I
Marlies Gillis: show you
Marlies Gillis: the footing surpriseal
Marlies Gillis: and derived from receptilecks.
Marlies Gillis: and then I challenge you to just loading all the other features.
Marlies Gillis: so you can see how they they are names. If you open stimuli and you have all the features
Marlies Gillis: basically just
Marlies Gillis: writing, I will
contributing
Marlies Gillis: book. You can rename it to. So you can. Damn, I'm not going to be able to paste
6.
Life is hard.
Marlies Gillis: everyone having copy-based issues.
Marlies Gillis: because that's pretty weird.
CNSP: I'm not able to right click, copy and paste. But I can use the control C and V on the PC keyboard just fine.
Marlies Gillis: because that doesn't work either. For me, that's a bit
Marlies Gillis: the centers in
Marlies Gillis: I think it's word surpriseal and graham
Marlies Gillis: sorry if I'm keeping you guys up.
Okay.
Marlies Gillis: let's put them for the shooter.
Let's see.
danna pinto: you chose the same thing for the oh.
danna pinto: the word frequency word surprising
Marlies Gillis: now.
Marlies Gillis: So I'm gonna add an offsets to each.
Marlies Gillis: Hmm. 4. So we can see
Marlies Gillis: this is up. If I have spelled stuff correctly, of course.
Marlies Gillis: And then you see what it's like.
Yeah.
Marlies Gillis: which.
Marlies Gillis: So now we have
Marlies Gillis: features. then, I want to just quickly show the slides like before. So basically, now to assess the edit value.
Marlies Gillis: we just need to create 2 models like a complete model and a baseline model, where one
Marlies Gillis: has all features. And one where we have all the features except for linked sequence, and then we take the difference between the 2
Marlies Gillis: as a quick note. The elbowing toolbox uses boosting. So this is not the same as we saw in ole stock.
Marlies Gillis: So it doesn't use which regression
Marlies Gillis: to quickly explain how it works. So basically, it fits it estimates the Trf, first by flat line, and then it's basically at each sample of the Trf dependent of like, at least sample this trf, it starts predicting the Eeg, using a small increment
Marlies Gillis: size, and it says, sees how well the model performs, then it does it with the next sample, it puts a small increment on it. It sees how good the model performs, and that that it does for all possible samples.
Marlies Gillis: positive increments as negative increments, and it then it detects and which increments on which sample and the model perform best.
Marlies Gillis: Then it puts the small increments on that spots, and basically, then the same process happens with this as a baseline. So basically, then a new increment is added and fitted on top. On top of all,
Marlies Gillis: all samples, positive and negative, to see where? Yeah, this increment is best placed on top of this one, or maybe here a negative one.
Marlies Gillis: So it's a different procedure. It's an iterative but procedure so just to
Marlies Gillis: keep in mind that it's slightly different than what Audi explains.
Marlies Gillis: So here's first I am.
Marlies Gillis: I just take one subject, the eeg of one subject, and I make an nd variable
Marlies Gillis: from yeah.
Marlies Gillis: from the eg.
Marlies Gillis: and to do so, you need also a sensors dimension which I created here, so that it's that ill-brain knows that it's a sensor dimension, that it has a sensor dimension and a time dimension.
Marlies Gillis: Then, you estimate the model
Marlies Gillis: by calling boosting
Marlies Gillis: here, I limited the Eeg to, or I just use 3 min of data.
Marlies Gillis: Because, yeah, feeding a model takes some time.
Marlies Gillis: I start from 0 ms. And I stop at 500 ms. So 0 point 5 s. This is a the basis function. The basis function is basically the kernel that you convolve.
Marlies Gillis: the trfids. So basically, you should see this, as these are all very small spikes.
Marlies Gillis: And the basis is like a kernel that you convolve with this window. So all those spikes become a bit becomes a little bit broader.
Marlies Gillis: Yeah. So let's run this. This should take not super long. It takes a bit.
Marlies Gillis: I hope that's working for everyone.
CNSP: What does that basis value represent.
Marlies Gillis: It's it's also a time, that of this. Yeah, of this kernel which which which you control.
Marlies Gillis: There's a small exercise here where we will
Marlies Gillis: play with the the width of the basis, and then you will clearly see how it affects the yeah. Yeah.
Marlies Gillis: the the basis, think function like. It's
Marlies Gillis: a definition of the kernel that you convolve is pi, t rf, with
Marlies Gillis: and if you want to go into more ill range stuff, you can also say which kind of window this should be, whether this should be a hamming window, which is default, or a Gaussian, or or whatever. So
yeah.
Marlies Gillis: does that completely? Answer your question?
CNSP: Yeah, it does. Thanks.
danna pinto: Is there something that you do when you input the different features to account for the the fact that the acoustic envelope is
danna pinto: at a different scale.
danna pinto: Like, do you scale the features, the different features to the yeah pushing those is.
Marlies Gillis: we're seeing normalizes the features outside the toolbox does it automatically.
Marlies Gillis: Yeah, in the posting function.
Marlies Gillis: You will see. Yeah, later on. Or if you're playing with with the results 100, and you have an H and an H scales and counter it to intuitive the H is the scaled one. So when everyone, everything is normalized and H. Scale is, if I'm not mistaken, the not scaled one.
Marlies Gillis: So if you want to assess the effect of
Marlies Gillis: so I hope the boosting models ran for everyone. And so in you can visualize the prediction, accuracy.
Marlies Gillis: yeah, by running the next cell. And the Trf, so those 2 important outcomes of the forward model.
Marlies Gillis: Like this. So this is just one subject
Marlies Gillis: fairly high. and prediction accuracies. and
Marlies Gillis: which is probably just due to luggage shops.
Marlies Gillis: And then the visualization of the Trf. Where you see this peak ish around 200 and like a peak around 200 ms.
Marlies Gillis: So then we're coming through the exercise
Marlies Gillis: to
Marlies Gillis: play around with the basis fashion.
Marlies Gillis: and
danna pinto: sorry. I don't know if I missed it. But what did we include in this model.
Marlies Gillis: And this is really just running one form of model. So we have here
Marlies Gillis: the spectrogram. So this is the Eeg and spectrogram
Marlies Gillis: and we start at 0 until 500 ms. That's the window of Rtr F from 0 to 500 ms.
Marlies Gillis: This is partition 5 partition. So meaning that we divide our data in 5 equal parts.
Marlies Gillis: Where you use here one partition for testing the model and 4 for estimating the the Trf.
danna pinto: And your data set you said is just a 45 min
Marlies Gillis: yes, but we but we here like here we just load one subject, and we just stay the first 3 min.
danna pinto: Okay, so those 3 min are divided into 5.
Marlies Gillis: Yes.
Marlies Gillis: II will. I can hear. I plan to show a bit more details on the
Marlies Gillis: on how these partitioning works.
So.
Marlies Gillis: okay, so we have the prediction. Accuracy is in a trf, so let's play with this basis function.
Marlies Gillis: And so
Marlies Gillis: for this, I will just
Marlies Gillis: limits the
Marlies Gillis: the time for training now until 1 min, because taking like using only 3 min, takes already quite a long
Marlies Gillis: thanks. and
Marlies Gillis: so
can go. Stephen.
Marlies Gillis: Here.
Marlies Gillis: I I will just
Marlies Gillis: give it the the starts and stop any amount of. But I can can please.
Marlies Gillis: And
yeah.
Marlies Gillis: thanks
Marlies Gillis: to
Marlies Gillis: here, I will disable the testing partition. Just so. patience.
It's
Marlies Gillis: just so it's
Marlies Gillis: takes less time.
Marlies Gillis: And
so many people
Marlies Gillis: going to switch or not.
Nope. yeah.
Marlies Gillis: let me see.
Marlies Gillis: And
hmm
both programs.
Marlies Gillis: We also need to take a new book?
Marlies Gillis: something.
Marlies Gillis: because I didn't improve my basis. Sorry.
Marlies Gillis: this is a stupid reverse.
Marlies Gillis: Okay?
Marlies Gillis: Is everyone still a bit following ish.
Marlies Gillis: okay? Hopefully, when that works.
Marlies Gillis: Voila and now you see that for longer.
Marlies Gillis: Colonel, maybe I can. and
Marlies Gillis: zip the
Marlies Gillis: the basis names.
Marlies Gillis: yeah, let's not do this, because otherwise that
Marlies Gillis: I lose time again. But for the for the long basis we know, you see, that the Trf. Is very, very smooth and depending on just like the shorter your basis kernel is the yeah, the the more sparse becomes.
Marlies Gillis: I generally use 50 ms basis function. But this, of course, depends also on your sampling rates. I generally
Marlies Gillis: and use
Marlies Gillis: 50 ms for a hundred
Marlies Gillis: under 28 sampling rates, but for 60 4. It's probably better to check out at different places.
Marlies Gillis: yes. So the kernel simply yes,
Marlies Gillis: like like in in this. In these cases. It. It will be be just nearly an impulse. If you go broader than it's more like, yeah, in this case it's a hamming window. But think of it as a bit of a like
Marlies Gillis: Gaussian. But it's all the gauge. And then it's just smooth. So it's all all small music.
Marlies Gillis: Okay?
Marlies Gillis: Yeah. So basically, I would like to challenge you to make
Marlies Gillis: basically the same model. But then next year's spectrogram also includes phoneme on sets. And then we can investigate the added value of phone analysis.
Marlies Gillis: Interesting?
Marlies Gillis: so we can take 3 min
Marlies Gillis: and this time. So then you include it as an array. And the spectrogram spectro.
Marlies Gillis: If there are any questions meanwhile, then, please
Marlies Gillis: let me know.
Marlies Gillis: Hmm.
Marlies Gillis: yeah. I need to check out how I build it again.
Marlies Gillis: 6.
Marlies Gillis: Oh, there was like 10. Well,
giorgia cantisani: So there is a discussion in the chat about
giorgia cantisani: What is this bus is currently so
giorgia cantisani: and then
giorgia cantisani: with
giorgia cantisani: so Andre is asking. So the business parameter specifies the width of a hamming window, which is the current correct?
Marlies Gillis: Yes.
Marlies Gillis: yes. And then you do this convolution. So this multiplication over time to have your speaker trf into something way more smooth.
giorgia cantisani: Okay. thank you.
Marlies Gillis: and then we run it with the same specifications, which was
Marlies Gillis: I see. For what reason would trf be smooters?
Marlies Gillis: Sometimes it performs slightly better when it's suited. Because, for example, you will have due to training Peak like lots of peaks like this, and due to smoothing. Your peak will look like this.
Marlies Gillis: And that's
Marlies Gillis: might slightly improve the testing on the testing partition.
Marlies Gillis: Additionally. Why, you want to also smooth a bit is just to to allow it to be more
Marlies Gillis: generalizable across all participants.
Marlies Gillis: Because if you work, for example, with a very sparse trf for example, let's say that it's for subject one like this and for subject to, it's exactly the same with opposites than the average.
Marlies Gillis: Yeah, okay, maybe that example, maybe should subject. So it should be a small peak. Strike straight next to it, then add the average Drf, would look
Marlies Gillis: that color selection
Marlies Gillis: would look like like this. Well, in case you would have smoothed it, smoothed it both both peaks a bit broader than it would be. Maybe like this
Marlies Gillis: is that Kirch?
Marlies Gillis: I hope it's
Marlies Gillis: okay, so
Marlies Gillis: I think I use a business function.
Patience.
Marlies Gillis: please.
danna pinto: I have a small question.
danna pinto: Here in this line that you're
Marlies Gillis: yeah sorry I'm I'm quickly checking. Whether my arguments are correct. And then
danna pinto: it says time equals 0 to 180, and on the one before it says, none.
Marlies Gillis: the 1 80 is there? But yeah. I don't
Marlies Gillis: Google. It's in the one before it's
Marlies Gillis: where does it seem now?
danna pinto: Here in the one they were looking at before here. No, down more in the boosting model. Yeah, here.
Marlies Gillis: Here's because the first sample will be known. So then, because this sub takes basically a sub section from up point up to 180 seconds, and others it's known meaning same like, just take up to this time. For example, if I would have for example, if I had a trf from minus 100
Marlies Gillis: to 300 ms, and I would say time going from Num 2 0 point 3. Then I would get from minus 100 to 300 ms. If I would do 0, then I would only get to 0 to 300 ms. But now, yeah, Eeg and spectrogram both start at 0. So it will be the same.
Okay.
CNSP: it's more like the first sample of of whatever you're indexing there.
Marlies Gillis: And where's the error
Marlies Gillis: here, I guess.
Marlies Gillis: Excuse me. should have
Marlies Gillis: it might be the browser.
Marlies Gillis: or something about I don't know what browser you're using. But something says it's weird, because previously it worked out. But now it's acting weird, anyway. Doesn't know
Marlies Gillis: really matters.
Marlies Gillis: As long as you bear with me and me, struggling and typing type person, that's okay. they can see like a frustrate quota in front of
danna pinto: what is the end of that code. Say, the
Marlies Gillis: so basically, I against basis window, the amount of partition and selective stopping selective stop stopping is actually so when you have this algorithm.
Marlies Gillis: It will. It is iteratively so. It will
Marlies Gillis: basically never stops unless you tell it to stop. And this
Marlies Gillis: says, like, ditch
Marlies Gillis: the stopping criteria. L. One so it's that the
Marlies Gillis: should be based upon the oh, no, I'm sorry. I'm I'm
Marlies Gillis: missing up with a different arguments. This is saying that after that one increment e iterative step increases in in error, so the prediction becomes worse. Then you should stop with the fitting of the Trf.
make sense
Marlies Gillis: about the L. 2 and L. One stuff. You can also have an you also have an argument, Eros. So I was.
Marlies Gillis: I think. Yeah.
Marlies Gillis: caught up or mixed between. So you can block the
Marlies Gillis: prediction. Accuracies of this R. 2 model, and how the T. Rf. Of the spectrogram looks, and how the T. Rf. Of the phone sets looks
Marlies Gillis: by the way, and these effects here, as I said, these are edge effects.
Marlies Gillis: and these generally occur. Whenever your Tf is trying to explain something which is happening outside the boundaries of your integration window.
Marlies Gillis: and that can happen, I would suggest to increase the
Marlies Gillis: and the integration window here.
Marlies Gillis: Or sometimes with adding multiple features, these effects and disappears. I would really suggest to in in like increase your integration window. So the T start and T scop
Marlies Gillis: especially when these effects are prominent on group level subjects, if specifically, it might be still a bit noisy and stuff. That's
and yeah.
Marlies Gillis: So here you plot the top of butterfly, and you can set the timing of this topography using the tardins.
Marlies Gillis: So then let's together determine the added value. of
Marlies Gillis: off funny on sets in this case before we do the go on to the next step. So
Marlies Gillis: I have a question.
Marlies Gillis: Our plots are very similar, but slightly different. Is there like some randomness built into the boosting algorithm.
Marlies Gillis: Yeah. Well. normally, it should be a fairly similar.
CNSP: It's fairly similar. It's just there's, you know, some really subtle differences. And I'm just curious. If there's like any like optimization or randomness built into it.
Marlies Gillis: I just don't know how the boosting algorithm works. Yeah, not that I'm really aware of, unless there there should. There is change, or they something has changed.
Marlies Gillis: and let's see.
Marlies Gillis: That's a bit weird, too.
CNSP: I mean, if it was just choosing at random. Say, let's put a an impulse, or let's put a kernel here. Does that help? No? And then randomly.
Marlies Gillis: yeah. But it's still. Then decides which one is the best one, and that should be in my case, and in your case, exactly the same.
CNSP: What if but if if it starts off with, you know, maybe a first couple of steps that are slightly different, and they all like add some value, at least at the beginning, things can be adding some value. Maybe, like it goes up on slightly a different tree. I mean, it's very similar. I just like, for example, the the low V at the up far left
CNSP: has a few other like lines that are close by instead of the big gap.
Marlies Gillis: But I mean, like otherwise, it's like, almost exactly the same. Yeah, I'm I'm not completely completely sure. Because, for example, how large the this, the this
Marlies Gillis: the speak, is set. And so on. Those are all defaults. Normally, it's probably if it's if it's slightly different than probably their
Marlies Gillis: should be some small randomness in the code. But I'm not sure where it is introduced, like on the top of my head. III didn't think so, but maybe I missed something more recent versions or so I don't know
CNSP: no worries. Maybe we'll ask Christian.
Marlies Gillis: Yeah.
danna pinto: Did you send the last
danna pinto: code for the last model? There's
danna pinto: seem to have some kind of error.
Marlies Gillis: Yeah, I can try to copy and paste it. Thanks.
Marlies Gillis: So basically, you just take the prediction accuracy of the second model minus. I hope I call it R. One.
Marlies Gillis: I'm sorry.
Marlies Gillis: Okay. And I got this one. Rb, well, nice.
Marlies Gillis: and
Marlies Gillis: so that we can take the difference so hopefully, this works.
Marlies Gillis: And then we can visualize the topography using the yielding photo plots.
Marlies Gillis: the prompt extension
Marlies Gillis: and our
Marlies Gillis: yeah.
Marlies Gillis: and then, yeah, these are just some arguments
Marlies Gillis: to make it beautiful. You shouldn't.
Marlies Gillis: but I like to. Nice or
Marlies Gillis: message, voila
Marlies Gillis: we can also put the additional
Marlies Gillis: color bar, and that's color brush.
Marlies Gillis: and then you can see this one's in addition.
See?
Marlies Gillis: Yeah.
Marlies Gillis: Well, this really high for a difference in production accuracy. It's
Marlies Gillis: and let me check.
Marlies Gillis: And the
Marlies Gillis: it's probably it's just because the values here are so crazy
Marlies Gillis: that it because it's situated more here. So we
Marlies Gillis: you can set the
Marlies Gillis: works.
Marlies Gillis: And for example, this
Marlies Gillis: already added values.
Marlies Gillis: Okay.
Marlies Gillis: so that's basically how to calculate the added value. Here, I just wanted to give you additional like, just to to
Marlies Gillis: yeah, have a heavy.
Marlies Gillis: Have some feeling with this
Marlies Gillis: and segmentation stuff.
Marlies Gillis: So in this case we use 5 partitions, and if you don't use the test arguments. So if you do test equals 0, then you see
Marlies Gillis: that your brain, basically before it starts testing the model divides your data in 5 uses, 4 4 partitions
Marlies Gillis: to train the model on. And this is for validation. So to determine when the fitting needs to be stopped and to apply. And
Marlies Gillis: and yeah, validation of the model and to determine.
Marlies Gillis: yeah, when the fitting of this increment switch. Stop. If you use a testing partition like this.
Marlies Gillis: then you see that there are way, more splits, and then it also takes way. More time. So then you, you feed a model on on 3 partitions, use one for validation.
Marlies Gillis: and then apply the model on on this phones. If you plot the trfs, so the the R dot, h, then you
Marlies Gillis: obtain the average of all splits.
Marlies Gillis: Okay.
Marlies Gillis: is that okay, for everyone is everyone still on the same track.
danna pinto: Could you scroll up a little bit to where you did the difference in the prediction? Accuracy?
Marlies Gillis: And this part, yeah.
danna pinto: So so what we do here is we.
danna pinto: But R, 2 is the
Marlies Gillis: yeah. Our tool is the model we trained the last, which, including the phone license.
Marlies Gillis: So it was the combined one of the phonemes, and then we substructed the spectrogram. So we have the added value of the well. What do they do here?
Marlies Gillis: we have the added value of the
Hmm.
Marlies Gillis: okay, so will there be answers to the exercises?
Marlies Gillis: Yeah, sure. So I will save this file. And then, whenever you open the
Marlies Gillis: whenever you open the document again, you will have the file with all answered
Marlies Gillis: exercises.
Marlies Gillis: so no worries, and also II will give you a link to the Github Repository, so you can.
Marlies Gillis: So you have all the codes
Marlies Gillis: so. And now I want to show how your linguistic features actually look like. So basically, we have all the tools. We have, all the features. We know how to estimate a Trf, to compute the added value that we just did for phoning on sets. Now we use exactly the same pla pipeline, but use it for linguistic features.
Marlies Gillis: because this takes a crazy amount of time.
Marlies Gillis: because you do this for 15 min. Lots of subjects. And yeah, you have lots of features in the model. This takes quite some time, and so I already
Marlies Gillis: it's good to go to make sure you don't hear the back ones
boom.
Marlies Gillis: so I will show you I pre-trained all the trfs already for all subjects. So for every subject you have a complete model, and you also have a baseline model.
Marlies Gillis: So basically, we will just load in the the base, the complete and the baseline model for every subject. And we'll do the the difference between them
Marlies Gillis: notes? Yeah, this is exactly the same outcome as you get got in the our instance here. So
Marlies Gillis: okay, so basically this calls the Trf. Directory that I just show you with all the pre-trained T. Rfs.
Marlies Gillis: And then it's loads. It's and then puts it in a in a row. To make an ill brain data frame. So
Marlies Gillis: if you show this instance.
Marlies Gillis: then yeah, it's
Marlies Gillis: it. It has the subject name, the model type, whether it's complete or baseline, and then it also has a an an nd variable called prediction person.
Marlies Gillis: And
Marlies Gillis: so then, using top-level map, you can really easily use this data frame to show
Marlies Gillis: and the prediction accuracy for each model.
Marlies Gillis: So then, you see, on the first
Marlies Gillis: site, you don't see a clear difference between the 2 models
Marlies Gillis: then using statistics which is like, really handy to to use the toolbox for. So if you do the key test.
Marlies Gillis: related it will.
Marlies Gillis: It's not ugly, like. Actually, the it's a permutation test.
Marlies Gillis: Based on that is that also felt that will be applied to test whether a prediction, accuracy, is significantly different between the 2 models, and it, it says, like, use the meshing factor subjects.
Marlies Gillis: So if you
Marlies Gillis: name nacelle
danna pinto: nylon, you know that
danna pinto: a violin.
Marlies Gillis: Okay? So
Marlies Gillis: then you you can see the the difference between the 2 models. So here in in the like, we know, there's a significant difference, because there are these curves. But the the polar part is way, too. To rob to actually visualize the difference, because the added value is small from linguistics.
Marlies Gillis: So if you do res difference, you can actually plot it better. And then you see in blue, because it's baseline minus complete, and the complete model has the added value of the linguistics.
Marlies Gillis: So the complete model is hired as a baseline. So the the the difference should be negative. And then you see that there's a prominent in improvements here in central areas of the brain where we expected linguistic features.
Marlies Gillis: have an added value.
Marlies Gillis: So
Marlies Gillis: get it to. Maybe it's it's not.
Marlies Gillis: And
Marlies Gillis: yeah, super interesting, because we know that that there is a significant difference between the 2. But just for sake of
Marlies Gillis: exercise we will make a new data frame.
Marlies Gillis: with the difference.
Marlies Gillis: and then.
Marlies Gillis: Now we can plot it right? So that's nice, I guess.
Marlies Gillis: So we loop over all the subjects.
Marlies Gillis: point 3.
Marlies Gillis: Well,
Marlies Gillis: working listed. I'm sorry
Marlies Gillis: if you're typing exactly, you say as I'm doing.
Marlies Gillis: So we want the subject to be the same, and then the prediction accuracy should be.
Marlies Gillis: and of the
Marlies Gillis: he's going to move.
Marlies Gillis: and then
Marlies Gillis: Oh.
Marlies Gillis: and then I like to use a lot of research functions to make sure it's it's the right
Marlies Gillis: that's how I expect to to be so. I expect only one correct entry corresponding to this line of code, and
should be.
Marlies Gillis: and that's
Marlies Gillis: let me see
Marlies Gillis: this time we want to.
Marlies Gillis: No, it's worth
Marlies Gillis: face. My little
Marlies Gillis: I know what I'm missing here.
Marlies Gillis: The data errors make their votes on mute.
Marlies Gillis: Let me check out the viewers
Marlies Gillis: and
Marlies Gillis: family. We are.
yeah.
Marlies Gillis: okay. And let's
Marlies Gillis: and show them
Marlies Gillis: to pull up.
Marlies Gillis: Okay.
Marlies Gillis: so this is what you expect. It's very similar to what I've shown above.
Marlies Gillis: So if you want to make it more beautiful, and you can
Marlies Gillis: these items
Marlies Gillis: which I like. voila. okay, beautiful. Yeah.
Marlies Gillis: last minutes. I hope I didn't include an exercise, anyway. Yeah, I did. Still, I'm sorry. Maybe I made it too intensive.
Marlies Gillis: Anyway, this is to load the trfs. We can also just quickly go over it
Marlies Gillis: basically loads again. The trf, I only look at the complete model. Because that's like what we're interested in.
Marlies Gillis: In. If the so
Marlies Gillis: your brain requires in a data frame, in that the nd variables saved in the same. Yeah.
Marlies Gillis: column, so to say, to have the same dimensions. And because the spectrogram have the frequency dimensions and the, for example, the tr for word surpriseal. Don't I take the average across frequency to make a nice data frame?
Marlies Gillis: And so
Marlies Gillis: yeah, here again, the trf of the spectrogram.
Marlies Gillis: Then, yeah, how to use this soup channel selection to take the mean of
Marlies Gillis: a certain channel selection. So this Channel selection we defined earlier. It's this here.
Marlies Gillis: And so huh!
Marlies Gillis: Centuries. So these channels. And then you see that if you have
Marlies Gillis: like lots of participants that you have really a very beautiful trf, it's a n one.
Nice.
Marlies Gillis: This is also a nice function. If you want to. Do a cluster-based permutation test
Marlies Gillis: user base permutations, as you see where the Trf is significantly different from 0. So this is currently running
Marlies Gillis: meanwhile, I'll dive
Marlies Gillis: the solution of the of the last exercise so that we can still have a quick chat. Maybe if there are additional follow up questions. So basically here, you see, whenever it's covered. The tidf is significantly different from 0, and especially if you have
Marlies Gillis: lots of participants. This is quite permanent to see the create that for some channels. Sometimes when it's very close to 0, it's not significant.
Marlies Gillis: Okay?
Marlies Gillis: And
Marlies Gillis: maybe I should just do 2 features
Marlies Gillis: 4, 6, 5.
Marlies Gillis: Okay, yeah. Could have included all of them.
Marlies Gillis: Mmm.
let's see.
Marlies Gillis: maybe I spell something
Marlies Gillis: and
Marlies Gillis: subjects
handle here in the middle.
Marlies Gillis: And
Marlies Gillis: so
it's just using.
Marlies Gillis: So
Marlies Gillis: maybe also set the title to the future.
Marlies Gillis: Here we see both the feature of chord, entropy, and word surpriseal. This is rather a subtle effect, and here.
Marlies Gillis: This is this for me, because this is for me, the first time that I try to send this data set. Normally, we expect this effect slightly earlier. But
Marlies Gillis: now, I created created linguistic features, using a completely different corpus. So maybe that does a trick.
Marlies Gillis: Maybe this also
Marlies Gillis: slightly. And of course.
Marlies Gillis: meeting.
Marlies Gillis: But it will be a bit of being around we did. But again, as expected linguistic features they have. This characteristic topography.
Marlies Gillis: yeah. As expected.
Marlies Gillis: II think for sake of time I'll stop here. I'll add the I'll add some more codes later to show this in one Channel selection, and to do as
Marlies Gillis: statistical testing, and to see whenever the Trf. Is significant.
Marlies Gillis: But I'll add that so you so you will have that
Marlies Gillis: as well.
Marlies Gillis: Just to end this workshop with. I'm sorry that it went super fast pace. I expected that it would have that it was more chill. But anyway, somehow.
Marlies Gillis: II slightly prepared too much if this overwhelmed you or so then, and you have further questions. Really don't hesitate to contact me. Here you have my email address. You can also reach out to me on twitter.
Marlies Gillis: I have put all my codes on my giglap.
Marlies Gillis: I think you should be able to find this. So basically, it's this repository, so you can every generate everything from scratch. Also, I try to document and my steps as
Marlies Gillis: as good as possible, but if you still have questions, then don't hesitate to ask them now, or to reach out to me later. And
Marlies Gillis: it's a bit sad that it's online. It's also really cool that we have, that you can all participate. But I wish we could have a chat together. I could see your faces, and maybe grab a peer or so so if we if you ever at the conference, and I'm there too. Then please come and reach out, and then we can have a beer together.
CNSP: How to have a beer?
giorgia cantisani: Me, too. Thank you very much.
giorgia cantisani: So there is any questions
danna pinto: I have a few
danna pinto: it sort of has to do with what you were saying about channel selection.
danna pinto: In the
danna pinto: in the data that you presented earlier today? Do you usually do some kind of
danna pinto: cluster like? Do you look at a specific cluster of channels? Do you just look at what is significant above chance like, how do you usually make that selection? And how do you prevent
danna pinto: sort of the multiple like, when you do the statistics, the multiple test.
Marlies Gillis: yeah. And so
Marlies Gillis: I am.
Marlies Gillis: I think there are 2 proper ways to set a channel selections.
Marlies Gillis: and either it's a pretty like, even before you start processing your code. By looking at what previous studies looked at so
Marlies Gillis: because, like, you should set your Channel selection in an unbiased way, so to speak. If you look at the prediction accuracy, see where it's significant. So on.
Marlies Gillis: Then you're actually already biasing your your Tidf analysis on where it will be significant
Marlies Gillis: if you want to bypass this, I think you can also. But then it's super nested cross validation wise just to
Marlies Gillis: really test test the prediction accuracy where is significantly higher than 0 in one partition, where you where the data is unseen, and then test that
Marlies Gillis: on on the crf, but that's that's very, very nested. So I just like to stick with a priori Channel selections.
danna pinto: and that brings me to another.
danna pinto: It brings me to another sort of topic that wasn't really discussed a lot in the workshop in general. But our lab started using the Rca method.
danna pinto: to sort of filter
danna pinto: the data before into this sort of components. And then we look, we sort of run our trfs on those components?
danna pinto: do you think like, what have you had any experience with sort of those methodologies?
Marlies Gillis: Do you have any thoughts about that.
Marlies Gillis: so I only use this using a principled component analysis first, and then applying trfs, or first estimating T rfs, and then applying Pca.
Marlies Gillis: because then, in a way, you do your Channel selection already.
Marlies Gillis: which is nice, right? Because you only have one component depending on which, component like how you go to the components. Which method you use.
Marlies Gillis: And
Marlies Gillis: II use Pca that I set or notes for any projects that I present now, but previously. That, I said subject specifically, and then especially for acoustics, the first component will be super similar across all participants. And then I think it's safe to use
Marlies Gillis: in case of Pca. The app polarity. The sign is arbitrary, so you just need to make sure that all the signs are flipped. But except from that, I think that's a very valid approach to do as well.
Hmm.
danna pinto: thank you.