Screen. You also see I tracking by the way those 2 circles in the middle is eye tracking. So we will then correlate the brain activity to how strong these red arrows are.
So kind of like the volume if you've will of the of the visuals. How much movement is there?
And what we find is that when people are playing this game, so this is the stimulus response correlation now.
So this is yoke to the stimulus, if you will, specifically optic flow. So if you look at people actively playing this game, you get pretty high correlation.
So these are not small numbers, by the way. With it eg activity and this optic flow.
Now this is the total optic flow anywhere on this screen. And then this is when they're passively viewing.
Now, we also did another thing, which is we told them that they're controlling the video game with their brain activity.
Of course that was not true. They were just told that that's what they're doing. So this is brain computer interface.
They thought, oh my, this works really well. It actually we're playing back a previous game and they had no idea that they're not actually controlling this the screen so we call this sham play and sure enough people that thought they're controlling the brain the video game.
Had brain activity those just as correlated if not more to the optic flow in the screen as the people that were actively playing.
My point being that this metric of correlation really captures how engaged the audience is with the game, even though they're not really controlling the input.
So, just like with the temporal response, approach that you guys are familiar, we extract components.
So rather than doing this one electrode at a time, we do it again with components. So this is the first, second and third component and this are the corresponding filters.
So, so how's this whole thing work actually? And here's a cartoon and this cartoon is already too complicated.
So I'm actually going to, stop screen share. And give people the opportunity to ask and instead I'm gonna show you something on the board to explain what we just did.
You guys see the board?
Yep.
We're still here. Any questions so far by the way?
No? All right. So the approach that maybe some of you are familiar with is that there's some that some stimulus feature.
That's coming in. Let's call it. So this could be if you've been working with sound, this might be the spectrogram over time or it might be the speech volume over time or something like that.
And, that goes into A. We call this a linear system, right? This is the so-called temporal.
Responds. Okay.
More conventionally, by the way, this is called the impulse response. So I, I resent at Lala for calling it what it should not calling it what it should be but at any rate temporal response are signed by me is impulse response and then the output It's another thing, maybe the brain activity.
Okay, and so this is this is essentially the model of the temporary response function and people call this model so why Is age convolve with x of t.
They call this encoding.
This is an encoding model. How does the brain? Signal encode the feature of the stimulus.
So that's that some of you are familiar with. Another approach is very popular in. Also in EG, but also in the FMRI is to decode the brain activity.
So a lot of what the news articles pick up about brain reading works kinda like what I'm just about to draw, which is you have many brain signals.
Maybe like so. And so yeah, at every moment in time you have a whole vector. Let's call this y vector y of t.
Okay. And that vector y of t. Let's multiply with a matrix.
And outcomes the stimulus feature. So the equation here is something like X is, W times.
. Okay. And of course these things can also depend on time This could also be multiple features and this can also depend on time.
And so this is typically called decoding. Brain decoding.
This is a brain reading, right? So all of the news articles you see about, oh my god, we can code speech or music or whatever else.
From brain activity or some flavor of this. Decoding. So what we did is what you just saw.
But we did with that video game and other applications is sort of a hybrid thing. It's a bit above.
So we take this stimulus. And pass it through an encoder with a convolution like so.
So this is again X of T that generates the signal C of T and on the other hand We're going to decode.
So we have the brain activity.
Make sure it here. That's why Okay.
It's another signal, the C of COT and the goal is that there as correlated as possible.
So this measures correlation and we call this stimulus Then. Response. Correlation.
Okay, so that's the approach and and this is sort of a hybrid between encoding and decoding.
All right, so that's that's That's the approach that we have taken with the results that you just saw on video games.
We've done this also with speech and a whole bunch of other things. So let me go back to screen share.
So I'll show you some more results. Any questions so far?
Yeah, there was one in the chat and then another that might want to ask you in person. In the chat, can one move from sensor space to source space with the metric just presented to quantify interpersonal correlation?
Not quite sure.
Yeah, Yeah, so. To go back to this. To this picture.
So this picture here, this would be the, this would be the source.
Right? And this would be the sensor.
And yes, you can go back and forth between the 2. So they. Going backwards is what's called the forward model.
So this is a forward model and this is the backward model. So the algorithm really extracts the backward model, but you can always reconstruct the forward model from it.
And in fact, what I was showing you before was the forward model. So going from source space to sensor space, where you can also go that way.
In fact, that's what the algorithm really does. By the way, the algorithm to find these things is called canonical correlation analysis was developed by Hartling in the fiftys.
So it's a really old algorithm. It's not something we invented. What we invented is this sort of idea of doing both decoding and encoding at the same time.
Alright, other questions?
Yeah.
Yeah, I heard a quick question about the type of stimuli that you can use. So I'm more familiar with auditory evoked responses thinking of like bond die or something repeated over and over.
Can you do this kind of analysis on that using that type of stimuli or does it need to be continuous stimuli?
No, anything works. Now if you have events like the ones you have, the conventional approaches, event related potentials, it turns out that doing something like this or even the temporal response function without by the way this this thing here what I drew here if you get rid of this you basically it back here and this is what
If you get rid of this sort of complication of extracting components, but you do an electrode at a time.
One electrode at a time, you get the conventional temporal response function. So there's a temporal response function or this one work for events like the ones you have and the answer is yes.
In the and the impulse response you find here here the temporary response function you will call an evoke response.
Or an event related potential. So absolutely you can do with either.
Other questions?
Okay. And then my other my other question I had is you're just displaying 3 components is that just because of those are the 3 largest or is it like ICA where you know if you have a hundred 28 channels you get a hundred 28 components out of this.
You get a hundred 28 components here too, but they they're nicely sorted by relevance.
Okay.
The first one captures most of the correlation. The last one captures the least. And mostly you're in the noise at the end.
Okay.
So unlike ICA that doesn't have a preference for one component or another, this decomposition technique gives you a sorting for the one that explains most of the stimulus and then the second and the third and so on and between each other they are actually uncorrelated.
So you can think of this as doing temporary response function and some sort of blanket composition at the same time.
Are they?
Perfect.
In fact, let's see ICA in front because you're doing it here.
Sorry I had a question, but I just totally forgot it when you were saying that last part.
I'll ask it again if I think of it.
Okay. We're good. I'm happy to chat. Not show any more slides.
And Lala in the house. Very nice, hi, Dad.
Yeah.
I have a question, Lucas. Do you think in terms of number of subjects, would there be?
An upper bound what you can get like is it the amount the case that if you keep adding subjects it will keep getting better.
Or which are the 2 techniques, the inter subject correlation, you know, thing?
I suppose, sorry, I'm talking about the CCA style, the inter subject correlation.
Yeah.
The CTA. So, you know, the more subjects you have, the more robust it becomes in a way.
But also it gets blurry. So you might argue that eventually too many is too many. Usually we get nice results.
With this starting at 15 and upwards. You can, by the way, Alan de Javigne, he has done it.
With extracting components for each individual subject used on similar things. So you can mix and match whatever you want, either single set of components for all subjects or each one gets your own.
It's a usual sort of per number of parameters issue. Yeah.
Would you ever mix different types of recordings?
What do you mean by different types?
Even, you know, starting from simple to, you know, data recorded from here in my lab and recorded in your lab.
Still, EG, or even EG, MEG.
If they are, if they are in the same sensor space, so if they have the same location of electrode, I think that's quite all right.
If you are combining activity from different sensors all together, I would have to think a bit about it, but I think you could potentially play games and combine.
Okay.
But I have to think about it, certainly not. Off the shelf code. So let me show you some code for this.
So, into this is stimulus response correlation. Does the same website and the resources you click here it brings you to Yasik's GitHub.
You download the code here, download. I did this already, so let me change to MATLAB to show you.
What that looks like. So.
How you doing that? I remembered my question. Are all the components that come out, are they all nonlinear or do you have like all the linear parts of the relationship in the first component likes.
CCA, I think it is.
So no, this, this is now back to the linear world. So here you're extracting linear relationship between the stimulus feature and the brain activity.
So this is very much like the temporal response function. You're limited to linear response. There's no nothing nonlinear here.
Oh.
So let me demo the code that I hope I answered it. This is the code that I have from Yasik that says demo let's see let's run this thing Okay, demo.
Okay.
That's not the demo, give me some sort of math, that's not the one I want to run.
I have to change here now again.
So these are the correlations in the first component.
So these are the 3 components that they extracted and they correspond corresponding impulse responses. This is on date of a single subject, so these are a little noisy.
If we had run this on more, say, 10 or 15 would have come out a lot smoother, but what do you see here?
The first one at least resembles the figures in in the paper. Which is this figure here where we see that that the the activity so this was again this was the shamplay versus passive viewing we see that the activity around 400 ms was weaker in the past viewing and that was predominantly in the electrodes in the back of the head.
As indicated by this first component.
Can we do this with other stimuli? So, we have done it with a lot with video.
We have done it also with the usual things like speech envelope and even the semantic dissimilarity signal and a whole bunch of things.
So you can use this pretty much where you would usually use it in speech. One example that I thought was kind of cool, we did this with the optic flow in different locations of the screen.
So we looked at how much correlations or stimulus response, so how correlated is optic flow between a location and the screen and the brain.
And so this gives us essentially a map in this video. Where our subjects were responding to. I would argue tending to.
And interestingly enough during active play they're paying attention to the horizon, the upcoming buildings.
Presumably because they're making an attempt not to crash into them. And we don't see that as clearly in the passive viewing.
By the way, the eyes are always peeled to the center of the screen. So that's not related to eye movements.
So let me just summarize what I've told you so far. Is sort of a partial summary of this talk.
Natural stimuli drive the brain, but only when the mind is engaged. So basically when you're not attending, we had seen that with inter subject correlation, the inter subject correlation drops protect precipitously or in this video game we saw that when the stimulus when the brain is no longer correlated to the stimulus then we know that people aren't paying attention and aren't
really engaged and these are basically 2 metrics that we have used. To measure this. So, so this sort of concludes that part.
The main the main title of my talk though is not just about the brain but about the body. So I'm gonna switch topic on us.
Unless there are questions. So far.
So now we're gonna start working with not just the brain signals, but we started looking at the heart, specifically the heart rate.
We capture now respiration. We are going to measure gaze position. The pupil size in addition to the EG and we do this with audio only or with audio visual stimuli.
So videos or continuous speech, audio books, that kind of thing. So this is work that, you know, in the last several years, I would say.
We had some really interesting results. So here I'm gonna show you some. Basic signals, this was collected while people were watching educational videos.
You'll get to see one of them in a minute. So this is EG. The 2 colors are 2 participants.
This is the heart rate. So you see how the heart rate is going up and down here.
This is over several seconds. So this little bar here is 1 s. Our gates position you see these 2 subjects are pretty much I don't know if this vertical horizontal case position pretty much looking at the same place on the screen while watching movie.
Distant pupil sized the pupil size and these 2 people is also very correlated.
And this is respiration. So as we inhale, it goes up and when we excel it goes down.
You see that they seem to go up and down but it's actually they're not very correlated.
So we published this in PNS Nexus. I'll show you the gaze. And here I have audience participation so I need you to all pay attention.
So you're gonna see the same video. And on the left, you will see the red dots that you see here are the case position of people watching a video and on the right the same thing but it's different people in one of the group of people either left or right, were paying attention.
And on the other side, people are weren't paying attention and the task for you. Will be to tell us or guess after you watch this.
Who is paying attention? Is it the people on the left or these red dots on the left or is it the people on the right?
So I'm going to show you the video and then we're gonna take a poll, okay, an informal poll.
Ready?
So which one of the 2 groups are paying attention? When you ask someone to draw something like this or or this, even though you don't worry, you know, this, this, this, this, this, this, this, this, All right, so now comes my poll.
Raise your hand if you think that the people on the left, these people or paying more attention. So everybody that thinks it's the people on the left, please raise your hand.
We'll take an informal count.
Okay, we got about. 32 people raised their hand. Okay, everybody raised a hand down. And down, hit down.
Now we're gonna see who thinks the people on the right. We're paying more attention.
No 1. One.
Oh wow, that's a biased sample. No, it's a totally biased sample.
Alright, well, good. You guys were right. I mean, the majority has the right answer. It's the people on the left that were more attentive.
Essentially, when people move their eyes together, are following the same thing on the screen, it's because they're paying attention.
If they're not following what other people are looking at, then they're probably spaced out.
And this is here, this is what we report here for all this signals for EG, heart rate, gaze, position, and pupil.
We're measuring interceptor correlation, but now we actually also resolved it by frequency.
So this is essentially a coherence measure. So we wanted to know in which frequencies this happen.
And before I was just showing you overall, I'll correlate our people. Here you see that the EG correlation is actually dominant at pretty low frequencies.
Most of you or most of us when we analyze EG, we'll learn activity of 1 s or faster.
Most of the activity that seems to be synchronizing, yes, there's something above one herds, even around 10 hertz.
But most of it is pretty slow. Here's for heart rate. And by the way, how do we measure our heart rate into such a correlation simply was Pierce and Correlation, nothing fancy, except here we first, our bandpass filter the signals.
So you see that there's a nice peak here around sort of 10 s point one hertz is 10 s timescale where the heart rate goes up and down.
So basically when we're watching a movie and this is also true when we're listening to a story.
Our hearts synchronize. So this is kind of a cute message here when you're driving in your car next time and you're listening to a part well not a podcast a radio program that's playing a life then your heart rate if you're really paying attention to the story.
Will be synchronized with out of all other listeners. So we do share something in common.
When we really go to the movies or we listen to a story at the same time. Our hearts are in sync.
The same is for the gaze position is a very strong effect with video. And the same is true for pupil.
So. When we look at this metric of inter subject correlation, it turns out it's very predictive of whether people get content of these stories they saw correct.
So these are educational videos we asked afterwards quiz questions about the content of the video and you see the pretty much all metrics except respiration are predictive of how many questions people got correct.
So these are questions that we're asked right after the video. So whether it's heart rate or EG or G, so beautiful, they all predict.
The level of synchrony in the in how similar you are to a group will tell me if you're paying attention therefore if you remember what was said during the video.
Again, if there are any questions for free to interrupt me.
It's okay, you've got, you've got people here answering questions for you now.
Kens is here.
Okay. And in fact, these, yeah, in fact, these metrics seem to be correlated with one another.
So subjects each.is your person. People that have a I intercept a correlation they I see in the in the sorry in the in the EG will also have a high inter subject correlation the heart rate that is if you if your heart is speeding similarly to somebody else then so is your brain.
And similarly for gays and pupil. So this seems to all capture a common underlying phenomenon.
So the theory we have behind it is that it's really the cognitive processing of the stimulus.
Is what entrains body in mind. So essentially say the video that in this similar thing happens with audio only.
So subject one may be attending to the video and subject 2 and and that will synchronize eyes hard.
It may be other signals and the level of synchrony in one modality will be similar to the level of synchron in the other because body in mind are really all in trained.
Oh, to the stimulus but only again if they pay attention if they're not paying attention none of this is true and this is how this the signals are correlated I mean co-modulated
Now, you see that I've drawn these errors in 2 directions. Generally, we think that the brain controls the eye, maybe the brain controls the heart, but there is also this notion that the body controls the brain to some regard.
So, so perhaps it's a two-way street and so we're not the first to suggest that the the model that we have really is a sort of a simplified model.
We know that when we're listening to a story, somehow that seems to drive also not just the brain, sort of the cortical brain that we pick up with EG, but when we listen to a story, it appears to drive the autonomic system within drives the pupil, the gaze and the heart rate.
But perhaps the opposite is also true. Perhaps the heart rate or the gaze or the pupil also drive the autonomic system and this is sort of has a bottom-up effect.
Does that make sense? So here we ran a couple of experiments where we intentionally control luminance.
Case and breathing I won't be able to tell you all of this we just don't have the time for it today.
But I'll tell you a little bit of the results and it's already complicated enough so here's some results of People just listening to a story.
So this is short stories, maybe 10 min, or 5 to 10 min long. They're kind of fun and interesting to listen to.
And what we first see is that the pupil synchronizes, but again only people are paying attention and the synchronization is pretty stark.
So. Attentive again at this very slow rate the pupil of individual synchronizes but not when they're distracted.
This is a very strong effect. We had seen this with video, but with video there's always the concern that maybe it's just the fluctuations of light that cause the pupil to dilate and contract.
The same happens apparently when we just listening to the story. By the way, people are watching looking at a blank screen we asked them to focus on the screen and There's a fixation cross, but regardless of the fixation cross whether it's there or not.
The pupil and trains. Apparently to the story. So there are moments in the story where everybody's pupils dilated in moments in the story when people's pupils are constricted, but only if they're really attending to the story.
We correlated the pupil with the EG. So this is conventional temporary response functions. So 0 is 0 delay.
This side means that the EG is ahead. Of the pupil and this side means the pupil is ahead of the EG.
So it's not clear which way the castle relationship goes. It seems to be that that a portion of the EG seems to respond the pupils seems to flex to it before they EG does and vice versa.
So we don't have a clear causal story. But the the important thing I want you to notice is that if we look at where this activity is coming from it seems to come from this sort of Ontario.
Central. Portion of the brain which I've seen in many studies that look at the Ontario cingulate so we haven't done source reconstruction here but we highly suspect that that's what might be going on.
So that's the pupil while people listen to narratives. How about the hard?
So the hard, we look at the heart rate, not the Ever beat, but the heart rate.
And we did again this temporal response function business so we take the pupil size and try to predict the heart rate from it.
And we see that there is a significant risk. Just a little bit on this temporal site that corresponds to the heart being ahead of the pupil.
Similarly, if we try to predict the pupil from the heart, we get a more clear story here.
Which again is the site where the heart is ahead of the pupil. So it seems to be that the fluctuations of heart rate are ahead of the pupils fluctuations.
And we do the same thing with the EEG signal, so can the heart rate predict the EG signal.
And by and large, we see positive time lags here, which means that the heart is ahead of the EG.
Now this is quite surprising. This is telling us. That the heart beat is slowing down or speeding up.
Almost half a second to a second ahead of a corresponding what we would call. evoke potential or potential fluctuation and once again same culprit same location on the scalp and this by the way is true with a person attending this track or resting so inherently it seems that the heart rate fluctuations proceed fluctuations of the EG of this sort of slower
timescale. And this is once again activity that to us looks a lot like Ontario Singular.
So the story that emerges here that perhaps the heart and by the way we drive it we can also drive with breeding that confirms this observation that I'm drawing here.
So breathing is known to drive the heart rate up and down. Seems to be driving down the autonomic system, which then affects the pupil size and also the electroencephalography.
So there seems to be evidence here for a cause, for a bottom-up effect of autonomic function of the heart onto a pupil and brain.
So, one last thing that I found cool that we found out is about gays. So it turns out that when we are listening to a story attentive, our eyes are moving around more.
So, the power of case position is basically how much are our eyes balancing around on the screen. So when we're attentive or actually moving our eyes more than when we're distracted.
I don't know if you noticed it, but somebody that's not listening to you, they tend to sort of stare blankly at you.
They're not really following anything. So they attentive listener will actually move their eyes. The disattentive, distracted listener would sort of stare blankly in the middle.
And it turns out that this amount of fluctuations of eye movements Also correlates between subjects.
So the amount of how much you're bouncing your eyes around changes in a story. So moments of the story.
Well, I don't know which ones, but some moments of this story are people move their eyes more than other moments of this story.
And if we look at that case variation signal, that case for issue signal 2 seems to correlate with this sort of anterior activity, though here it's possible that this has to do with the frontal eye field.
So I at any rate, this thought is a kind of interesting observation that I don't think is in the literature in any kind of way.
We move our eyes depending on whether we're attending or not. What we're listening.
So to summarize it all narratives drive the brain and the body at a very slow time scale, but only when the mind is really engaged.
Otherwise the body is not seen to in train at all. This sort of brain body connection may be a two-way street and we can measure these things either with inter subject correlation or stimulus response correlation.
We like to do one result in time because it gives us these sort of Casal interpretations and we like to resolve the other in frequency because it gives us an idea of what time scale this is operating at.
And with this, I would like to thank the people. Jacques C is the guy in Paris, by the way, who works on pages, disorders of consciousness who brought us to pay attention to this heart signal.
And, and these are the students I've been involved in all this work. Code is here.
You can just go to resources and download it on this page. And with that, I'm going to stop and give people opportunity to ask questions.
Well, thanks Lucas. It's really cool to see that. Pupil, you, you were come to fruition after.
Yeah.
The cog here meeting last year. That's really cool.
So if you want to ask questions, now is a great time. Raise your hand and I'll mute you.
Or you can ask a quote here's a question in the chat. If they distractor is a stimulus that is correlated with the stimulus.
How would you think it may affect the results? Would that in turn increase the ISC and is it a linear relationship with distractor strength?
So this business of how to distract people is really subtle problem, right?
The way we have done it and this was a suggestion for by Simon Kelly by the way, who maybe some of you know, Simon Kelly's idea was do not distract people by putting stuff on the screen because then you're not really sure are you analyzing activity associated with the distractor or as you analyzing activity.
That's this distraction related itself. So, so he suggested instead to do a very classic thing which is a mental task.
So you just tell people do something in your head. What's a good mental task? You can do any kind of thing.
You can ask them to think about what you were doing, what you have for breakfast, or you can ask them, well, the task we use is to do math.
So do complicated math in your head, count in steps of 7 backwards starting at say 1,000. So it's 1,000 is 993, 986 and 900 and.
7, 9, so you're really see I'm staring blankly into space And so that's the approach we take.
And so this is all silent. So there's no stimulus to confound your task, but yes, this whole business of distractors and measuring attention with a probe stimulus is a whole complicated story that I don't really have any good answers.
Hey, Lucas, I had a question as well. Sorry I was late for your talk. I was teaching this morning.
I'm still in my classroom here. Actually, I just joined as soon as I could. The question I had was the cool new findings with the I movements and stuff like that as well.
And maybe you explain this and I didn't get it at the time but the I was interested to know the kind of speed of those the size of those are these what we would consider just regular saccades are they microseconds do you need to measure and we're using some super fancy eye tracker or is it coer than that?
Is coarse. So it's literally You know, when I'm looking around, I can be looking around in a ferry, in a fairly narrow sort of area or I can be looking around a fairly big area.
And that's it. We're basically measuring the power. Resolved in time. Think of it as the saccade envelope.
Cool, okay.
Or not psychotic but gaze envelope. That's it. Variation. So there's a paper that we site that's also looked at this is case variation is a term that maybe is going to catch on.
It's not saccades, it's not smooth pursuit. We don't really care.
We're completely agnostic to it. We just like over a period of time how wide was the range that you covered.
Cool. Cool.
That's it. And so that range is what seems to be in moments of this story be bigger and other moments of the story be smaller and why.
I think it's wonderful area of, you know, research now to find out because The the thing that I find that interesting about it is it's it's incidental movement So it's more, it's totally unrelated to the task.
Just like you right now, are you touching your face, right? That's incidental movement, totally unrelated to your listening to me.
Hopefully you're listening. Looks like it. But so we do all of these incidental behaviors are seemingly totally a task unrelated and yet And yet they dominate brain activity.
Perhaps you follow that literature, but also we find that they are linked to what's what we're doing in our head, right?
We're listening to the story and apparently that's driving these, we think it has to do with arousal.
Different moments of the story increase arousal and that's when we move our eyes perhaps more.
That's that's our current hypothesis. We have to still figure it out.
Cool. Thanks.
Here's one from the chat. Do you think it would be interesting or informative to measure inner subject correlation without a shared stimulus.
Lot of research is done to study interpersonal neural synchronization where people are actually trying to exclude the synchronization due to shared sensory input.
So we use, we record people. Sequentially like you know now we're recording you and tomorrow will record another software and so on right so there's no way that be synchronized any kind of way other than through the stimulus.
Now there's a very big literature in in in what's called physiological synchrony PS.
And physiological signal where the interest is sort of when people are sitting together side by side or looking at each other interacting in some kind of way.
That somehow they are in synchronized, the physiology synchronized. Classic example.
Mother and child. Or therapist and and patient or or a performance and audience. So there's a whole sequence of a series of papers along those lines.
Where it's never really quite well controlled. Why are they synchronized? Is it because they're sharing the moment and there's some common stimulus or this is some sort of empathic you know, correlation, but I would content that if you put a screen between the 2 people and they don't see each other, then don't hear each other and they don't smell each other there's nothing
that can possibly synchronize unless there's unless you believe in some sort of you know esoteric kind of thing.
And so I think that the only way that people will synchronize is because they are somehow sharing something that they're perceiving.
And so, and so I think that understanding what it is that synchronizes people in this interpersonal situations is kind of an interesting topic and there's a big literature but let's not pretend that this is some sort of Budu synchronization.
It really is, I believe. The the the sensory person of the other person that will synchronize people even when they are sitting together.
Okay.
Hi, Augustus, Nick Cross here. I have a question. Great talk. I found your work really interesting.
Looking at all the relationships between these different biometrics. And I'm wondering, looking at this correlation you showed between the heart and the EG or heart rate.
Do you think that there are any interesting or useful, markers of cognitive function that could be from like a smart watch heart rate measure, you know, given that we're seeing a lot of companies trying to develop.
Brain computer interfaces now and And also to different NOR technologies.
Absolutely. Absolutely. I mean, the watch has a problem that's very noisy, but if you get a high-end arm band, Absolutely.
I do think that you could do all kinds of in the wild experiments and we want to do it and we're thinking of how to do them and we're doing a lot of online experiments with eye movements already.
Yes. Go for it. I think absolutely is a very rich signal. I mean, I don't know if you caught the scale of these correlations that I was displaying on the screen.
They're much bigger than EG signals. So the correlation of the pupil that we find at the heart rate.
I mean, it's a noisy signal. I give you that. You need something for the heart in particular you need much longer record because you know it's a very slow signal so you need a lot longer records.
Yeah, and presumably for
But the level of correlation is definitely higher than D. For instance. So I think it's a very it's It we're only scratching the surface.
I think there's there's so many experiments you can think of looking at pupil hard.
Yeah.
Now, I should say it's not the heartbeat. It's not like people speed synchronizes.
It's the heart rate. So your heart might be going, it's that speed up and slow down.
That's what synchronize. It's not the actual B 2 B. It's not the actual B 2 B. It's that speed up and slow down. That's what synchronize.
It's not like our hearts are in absolute But that's not the case.
Thanks.
Okay, so for the sake of time, we'll move on. Thank you, Lucas.
Yeah.
Everyone join me in thanking him with our. See there there click that button. Thanks, Lucas. That was a great talk.
I really enjoyed it. We'll move on to Giovanni.
Yeah.
Let me, so Giovanni will give us some updates, about CNSB, what he's been doing.
Over the last year or so. What we've been doing and then he'll introduce a his visiting student, Giorgio, who will give us an update on the project.
So Giovanni, please take over.
I'm stealing the screen now.
Please.
Is this the right? Can you see my slides?
Not yet.
Oh, okay. What can you see?
Sorry, 2 pictures in your group.
Oh, great. So these are the slides. And Lucas, thanks a million. That was great.
And also the format. I like this interactive. 4 months for online presentations. I'll have to switch back to the old format.
And But it's very brief. So before before moving on with the next invite the speaker who's Giorgio Piazza, I'll just give you this brief update.
On the work we have been doing but also at the end I'll give a brief mention on the travel grant supported by sponsored by a and train.
Let me just prepare a slide. One of the embarrassing slides, a lot of faces and nobody looks at, but it's it's a necessary to thank all the people that helped in the last 3 years, in, in all the work we did with the workshop and also with the resources.
So Lucas has shown this picture in different flavors on the board on the slides. But the idea is yes, we have in the different type of experiments that we deal with that the CSP.
Initiative and work for care about. We talk about, you know, some neural recordings and some Sounds, they're continuous or some videos or everything that Lucas has mentioned.
And ultimately we have an X. And the why in machine learning sense. So and next where we have if you want some stimulus features and the why where we have the neural, signal.
And want to relate them somehow. So when we have, and again, the way that Lucas described when we have multiple features, imagine and this is something that happened to be anyway we had recorded the data set thinking about phonology and then that same data set has been reanalyzed in I don't know how many different ways.
So so many different papers from different labs. Can come out from the same data. And so this this is an opportunity.
It's an opportunity to reuse data from, you know, rather than having to record a thousand times.
The same kind of experiment. And then the type of task that we're talking about, like listening to speech and music is just more pleasant.
So, it gives you better access to particular cohorts that wouldn't be as willing to take long experiments where they have to look at, you know, flashing lights or individual syllables.
So just more pleasant. Now the challenges are that we want to tackle. One challenge is that You know the correct use and interpretation of the methodologies we are talking about Right, it's very important that if we start sharing resources, we start sharing code or toolboxes.
That we also you know make sure that there is no misuse of these tools. And so that's why we try to publish papers and and and also organize this workshop where we try to provide this.
Resources for for guiding towards the interpretation and how not to use these methods.
Then the one thing that is very practical and even within the same lab. What happens is that is that we just end up using different versions of the same code.
For doing the same thing, which is problematic. It causes it increased the risk for errors and also makes everything slower.
So code reusability.
So with here CNSP at the center of this slide, what I'm trying to say is what's CNSP is is about.
So one element is Trying to share some standardized analysis code. For running different types of analysis. And of course we started from a kind of analysis we were doing already and by we I mean Aaron and I and Nick and so on of the organizers.
To share standardized code, we need to agree on some standardized format for the data. That's also another activity we have been working on.
And then trying to take some existing data sets and putting them in the same format so that basically that same standardized code can be run instantly on, all of these different data sets without wasting time.
Then of course we have the workshops where we try to to share new research and new tutorials, question and answers and so on.
And finally, we see this as an opportunity for. You know, creating a community for networking, for connecting people that are more on the methodological side and instead other scientists that are more in you know they have maybe less confidence on the methods and they want to connect with someone that has that expertise.
On doing more applied research. And for students, of course. And that's this is the kind of resources I use also in my own lab.
So every student PhD student or master student that joins, I just tell them, okay, open the CSP website and download and such and such data sets, download these scripts and try to, to, learn from these.
We have these new sections that we didn't have last year on lectures and tutorials from the from the various editions of the workshop so you you find the 2,023 here as well.
And it is not just a repetition. Every year, right? So for example, last year we covered CCA, we won't cover it this year, so we don't need to repeat things that we have seen already.
So that we leave space for new things. So check out those things. So what's new?
So the first thing that is new this year is the CND format. So the continuous event in your data format.
It's a data structure that that we proposed for storing neural data. And the is specific to continuous sensory tasks like listening to speech.
So it was as I said it was a necessity because if you want to share standardized code we had to find a way to agree on some standardized data format.
And the idea is that if you save your data in this way, then you can run these scripts that we're sharing directly.
And then it also enables the design of graphical user interfaces and that's sort of stuff.
So C and D, this data format. Is somewhat after the raw data, so it's not your raw data.
It's minimally preprocessed. In a way that your stimulus features and your EEG are aligned and you're with the same sampling rate.
And so you can run TRFs and CCA or you know, Into subject correlation.
More rapidly and so you can you have access in this case you've seen a space scripts But also we connected.
This year we connected this format with the the work from and Christian brought back and Jonathan Simon and Nima's lab these toolboxes that they design so you can load CNSP, sorry, CND.
Data files from their toolboxes now and you can easily transition between bits and C and D. They are they're connected and we included all the functions for doing so
But in reality, this is not something complicated. It's just basically a guidance in terms of what is.
The folder structure you should use. Where if you do put the code, where should you put the tutorials?
Sorry, the code, for example. Where should you put the data sets? How should you call the file name?
So if you follow these instructions then basically it's it's just much easier to run the code we share.
And we share this on our GitHub. See if you just go on our GitHub and the links are all on the on the website.
Show them in a second. You have this folder structure with the example scripts. We have more tutorials throughout here.
The libraries are not ours. Some of these are ours, but others are just dependencies that we use.
And then the data sets are empty. You just need to go to the website, download the data set and put it into folder structure, and then you can run the script.
We are setting up a new documentation page. It's a working progress, but you can find you can already find some instructions on again how to save your data in this format and so on.
So on the website you have this. On the resource page, you have all of these links, the GitHub, the documentation.
And these other things I show you in a second.
The data browser. So we one thing that we built in the last year is a graphical user.
So just an interface like this. Which is built in Mathub in this case. And wait for the next slide because that's much cooler.
So the idea apart from the ugly looking interface which is matter of style This gives you an idea of the parameters that you need to decide when you when you fit these models.
And it gives you an easy access to, to do so. So basically what you can do is And I'll see if I can.
Take it up here. Yeah, so it on top. So this does the the GUI in my my computer and you can open here and you have access to whatever data sets you have on your computer and they're all in the same format so you can switch and
A loads and then it's going to give you the stimulus feature for that particular data set.
And you can say, oh, I want to fit the model on this number of participants on this amount of data per participant, for our backer models.
And just for sake of time, I don't have time to run it now, but basically. You can run the user TRFs, get prediction correlations, get the RF weights.
But you can also compare different data sets. You can take your speech data set, your music data set, you can plug them on top of each other.
It's still work to do, but it's a it's a starting point to make things a bit a bit faster a bit easier when we want to look at data that we have.
You know available without having to talk with the person that that actually share that data every time and this is a proof of concept.
We, it's a work in progress. It is a web-based data browser. So while the other one is a standalone application built in Matlab, you can just, you don't need to have MATLAB, you just install it.
But you still need to install it. This thing is that is in your browser. So you just go on your browser, you type this link, which I'll share on the chat in a second.
And you can fit. Forward to your apps and so on. So encoding decoding models.
Why do I think this is useful? And again, lots of this stuff is done selfishly because because we use it.
So it happened to me, for example, to talk with scientists that wanted to run a new experiment and they were asking me Can I run this speech listening experiment or this music listening experiment?
In with 5 min of data. Because my particular population can't deal with experiments that are longer than 5 min.
And usually my answer is, I don't know. I need to open my Matlab and run some reanalize some of my code.
So with interface, basically you can go to some loads some existing data and select how much data you want to.
You want to use and see basically, for yourself, if that is a suitable amount of data for your experiment.
And much more.
So again, for sake of time, I'll. I'll cut you short. We describe lots of these new components in the paper that we just submitted to archive.
So the C and D data structure, the analysis pipeline resources. We also include the simulation toolkit.
Where you can simulate EEG data in different ways. I encouraged using that because simulating data can be really, really useful.
Especially when your hypothesis are not that Clear, it's not clear what you would get from a TRF analysis.
So maybe before actually running an experiment, it can be quick and useful to to use these these functions.
Okay. So just to conclude the call for project proposal, so, and bring train, and which is one of the most interesting producers of portable EG systems.
They have been sporting us with this scholarship last year and also this year. It's 1,500 Europe.
And so what we encourage you to do is to propose a project. And to this proposal in 500 words.
You can decide to say, okay, I want to visit Giovanni's lab in Dublin or I want to visit Aaron's love in Rochester or Sekosia or Nottingham Trent or a Commonwealth computer can visit any of these institutions in the list.
And that are related the organizers or you can propose some other visit to someone else as long as it makes sense.
With CNSP. So as long as it has something to do with, you know, the resources that we share or something related to that.
Or anyway something that is has an open science mindset. And so whatever data you deal with, we would encourage you to share it.
The call for proposal is on the website. You can find it here in the workshops, page.
Okay, that's all from me. So Right on time. I'm a bit late.
Sorry. So I will I would like to I'm happy to introduce now the recipient of the travel grant from last year, who's a PhD student in the infant language and cognition group.
In BCBL the bask center on cognition brain language so, G, he, he was awarded the project, the, the travel grant last year with a project title does not native directive speech Enhanced non-native listeners critical tracking.
So Georgia, we're not talking just about that. Jordan will be talking about his PhD project and also a little bit about what he did here.
And one cool thing that he did and was to complement the scholarship that he won through the workshop with some other funding.
And so we managed to spend 3 full months in Dublin. And so he managed to collect data and also to, you know, his close to publication with that.
And so that's the kind of idea that we like to promote, but. Even if you wanted to come for a week, that's still okay.
Are you online?
Is George online?
Okay, yeah, yeah, I, I couldn't, I muted myself. Yes, sorry.
That's fine. Welcome to the workshop again, Giorgio. And the floor is yours.
Thank you. Thank you. Okay, I think, yeah, I'm trying to share the screen.
. Not Right, here we are. Alright, so well, thank you for all, Joandy, everybody from be here.
I guess you can hear me well. If not, please let me know now. And so yeah, I'm gonna talk about, well, mostly actually the project that I carry out during my Christian stay at Telanus Lab.
That well the title I gave slightly different so non-native direct speech supports and to learn this corporate tracking speech but it's basically the same thing.
So it's also part of the bigger project of my PST station. It's all about non-native director speech, which is a speech register.
And Sorry, okay, it works and it also has other names for some foreign director speech. Usually, a non-native director speech is the prefer name because it's more neutral term.
But for this presentation, I'm going to use the acronym from the old name of the state.
So FDS, just because it's easier because it will be compared to native director speech, which is, well, the acronym is NDS, of course.
So, just to make clear, it's not foreigner, Sanders speech would be the guy here in the white speaking in English for instance.
It's about the way native speakers like the English girl left, but it could be any language of course.
That's the way how she speaks. She addresses, Mel T, so someone's learning, English in that case, a second language.
Fds as compared to native language speech, it increases speech clarity. And this is reflected in some of the features.
For instance, North speed train, so for, it will be a low number of syllables per minutes or hours per minutes and also an expanding vocal space is known as vowel hyper articulation.
Just to give you like a graphical representation, there's nothing the meant to understand that this presentation.
But this is how we calculate our population. They'll be taking the first 2 4 months of the vowel production, plotting on a criticism plane.
And then we calculate the area, the triangles. The red triangle represents how at the GERL left will address the guy on the right.
So, a hyper articulating power. So producing a vowels far away from each other. And the blue triangle represent the way the vowels of native vector space.
So how you speak to another native listener. And this is interpreted as being promoter of language learning and also processing because it supports detection vocal consciousness because they're easier to detect, so they're also overlapping.
Apart from that, apart from assumptions, we also found in our latest study that's now under review.
There and L to learners when they're exposed to FDS, they actually learn better, faster and no worse as compared to participant were exposed to an And so it seems to be a tool for teaching a second language.
And we use single representation, a short presentation, sorry. So it is still unclear whether FDS, but most of the perception of continuous speech would be like more in that, yes, but most of the perception, continue speech, it would be like more in that realistic environment.
So, and also, beaver studies were mostly most studies on FDS or behavioral.
And that do not allow us to investigate the process of ultra processing while this is going on. But when it's already completed and of course you know that EG allows that.
And And so anyway, so far there's just a single study on foreign director speech with EG.
It was doing single presentation, single representation. There's nothing on continuous speech and ecological body stimulate.
And so one way of course to investigate whether FDS supports through processing is by using EG and measuring cortical tracking of continued speech.
Well, there's no need to say this, but we'll say anyway, so are you tracking a synchronization of cortical, cortical, sensor activity to fluctuation of sudden stimuli.
Of course I write it. And of course, one feature you can study is speech. The most typical feature.
So this is embedded evidence showing that already tracking speech, especially in low focus events, usually that's the more or less.
Is a marker of underlines process. And And that's also some evidence showing that it's fundamental for other speech processing.
In some cases, a quarter track has been linked to speech clarity, possibility, and so on so forth.
And also as has been studied in infant language development. So really in relation to infant right to speech there is another speech register in this case directed to infants and which shares a lot of features with foreign direct speech.
So it is possible to explore to investigate coordinate tracking using temp response function. Of course, you're here because you know what is.
That is, RIGALIZED linear regression. That maps it to the continuous stimulus to an error response.
And I think Joanne is familiar with this figure here. You know that. And, and then it's possible to using the giraffe to predict UG response and correlates it to actual EG activity.
So this is a one possible outcome, one possible and variable. And using this presentation. Okay.
Another interesting feature of TRF that's possible to see by your sets of features, not just like a speech handle for other features that are known acoustic too as well.
For instance, like Samantha, say. And of course you already know why would actually going. So we aim to measure what you're talking in foreign director speech and then yes, so native after speech.
But in literature, it has been shown that Korgo tracking is also affected by speech rate. And so we for this reason we had the third condition that we call slow NDS.
Which had the same features of ideas. So actually we we use and yes, and then we slow it down to measure the speech rate or FDS and we did it by means of Well, sorry, I will say this is the next slide.
Then I'm, sorry. So and slow in yes, and so slow and yes doesn't have any feature that is supposed that is expected to support and to learn as perception.
And also, well, it has never been addressed, but of FDS and Portuguese, you know, second language speech.
And of course, it's important and supportive to shut light at this point because it will tell us more about how a second language is process also acquired.
Because of the features of foreign director speech that I told you about before, and they're adapted to the Learner and FDS is a certainty remote as I said I do processing but not at yes.
And this should be reflected on more efficient target tracking life. Yes, then and yes. And this should be true just for alcohol.
So in the study in the British study, we aim to answer the question here. So the first question does FTS promote L to learners cardic tracking speech?
And if you have time, I'll just give you a hint about the next. You're running.
Just the second question. So, and we're into experience. So they were collecting 2 different experiments.
Why to experiments because we were 2 different labs like Spain and one in Ireland and also with 2 different systems.
One is by a seminal one, it's a great vision. So in the first experiment, we collected a from 25 at to Burns of English.
Who was just, who's as one L one as Spanish. And then the experiment, so far, 1880, native English listeners, we are still in the process of collective data.
And all the participants listen to continuous speech inform us covertelling in the 3 store in the 3 conditions.
And each story was about 50 min each and well the stories were called where I recorded in the British English accent just to make it make it easier for English for Spanish participants.
So while we were listening to the story of scores, we record their participants, the EG activity, and we also ask them to ask some questions about the content in the story in total with the after each break as a 5 question for about 25 questions the story and also we where we went for TF model.
So the low focus events want to get herts and for the features of speech and look so far. And here we are the results.
And for the comprehension results. And for the potential score as expected, we found main effects be register and participant respond with higher accuracy.
It was stories when the story was pronounced and foreign direct to speech as compared to the other 2 conditions.
Here I like to involve the main comparison of interest, which is foreign vector speech versus NDS, Native American speech.
And this is in line what we were expecting. And so they seem to be more accurate, so to understand better, say, and the stories in F.
Yes, of course we know there are other funds, so we cannot conclude much. So because there's tension, engagement, but this is still the first results interesting.
So for this reason we also had more of course check you measure and we use prediction correlations of the mode node for TF model.
And we found that participants showed higher petition correlation when they listen to stories in foreign right to speech as compared to native direct speech as compared to slow and yes and again here the interesting comparison between FDS and then yes and so it seems that they were encoding better speech handle aware with this name to FDS.
Okay, so then the results of the English Lisa group. And I suspect that we didn't find any difference across the precondition for the compression score because the the native so they don't need any any acoustically featured development, understand process.
And the stories. And we also, well, like was to stress out that there's no sending effect.
We made sure that it didn't go all the same wrong answers. There's some variability which is important as well.
And for the traditional relations, we had something that was kind of unexpected and because we got a higher pollution correlation and yes as compared to FDS and slowness.
So we're expecting that because of the slow speed trade or because of the who to feature somehow we will observe hibernation correlations in FDS and slow india.
But instead it seems that it's more important the fact that NDS was sweated so it was like a man to address native listeners.
As it was the same for F, yes. So just to then now I will just sum up all the results.
That'd be clear. So. And we asked ourselves whether FDS supports processing of continuous peace, especially for learners.
We found that it was the case, at least for the, results. With the limitation I said before and we saw we found the same well Congrent like assistant results for the TRF.
And in fact, we found that the Spanish Lisa had high British relation FDS whereas English Lisa had higher prediction, prediction correlation in MDS.
So it seems to be the case that is not just FBS having some sort of some special acoustic features that make the, you know, make space to be encoded better is the fact that they are the men they intended atrocities of those speech registers and also it seems to suggest the results seem to suggest that this is not an effect of speech rate.
Because of the way we find the same fact in both groups. So there was low speech rate, would drive the fact and this was not the case.
And also we will find the same fact for flow ideas since it was as lower as low as FDS.
And one might ask whether slow india was like weird or natural and we don't think so also because we would expect an effect here so we don't find that we didn't find any difference between FDS as low NDS in the in this different group.
So probably must probably was just a matter of fact that they are not exposed both groups both experiments, the participant nox usually exposed to slowing yes because they are they are either exposed to FDS, is there are no native and to end yes and yes if they are usually you know interact with other native speakers.
And so these are results for the presentation today and just a quick lean on the ongoing analysis. So we are trying to answer that question more than about companion should be how FBS supports El Dulinus, symbolic and pointing, let's say.
And so you use GPT to destruct some of the surprises values and of course we align them to the opposite of the worse and so we're using again to ref model so for a model to predict a response.
But so I have no time to show. SIR, but I have some time for some questions if you have any.
Thank you very much. Hope wasn't too fast.
Thank you.
Great talk, Giorgio. Again, feel free to raise your hand. We'll,
On mute you were allow you to unmute if. Yeah, I have a question or you can put a question in the chat.
Have one for Georgia.
I did have a question. Actually, Ed has a question. This would probably be way better than mine.
Hi, Georgia. Thanks for the nice talk. And hope you can hear me okay. I'm in a different room now.
Thank you.
I'm I just had a quick question about how these results might fit with them. I know Giovanni has published paper before looking at using a lot of the EEG measures to determine someone's.
Abilities in English for example as they go from a novice to an expert or a native speaker.
Is there is there a relationship there with what you're finding in terms of this foreign? I know it's a different question, but I'm just wondering if if there's still a
Yeah, I mean, we are thinking about using the coding and to, to call the comprehensive score they got.
So in that case, I think will be something more similar than from here. And yeah, so in I think you're referring to the ones paper, point 21, 2 speakers.
And in that case, of course, there's no variation between speech registers. So it's just English as it is and most probably I think they use like native direct speech because like stories.
So, that's allowing common, of course, and by that case, would be, well, we didn't have any variability between, well, we have some variability, but not that much between the for the Spanish listener group.
So they are all mid low proficient. I guess, well, I was back to find senior results.
But it would be interesting to see the recording if we sign find something different for like NDS and and FDS and that piece.
So I have that we I will work on next future. Thank you.
Any other questions?
There is a raise hand.
Accidentally muted you Marley. So you can unmute again.
Hi. Yes, I'm, mated now, I guess. Can you hear me?
Yep.
Yeah.
Thanks.
Thanks for a talk. It was, really cool, cool project. And maybe I'm missed it, but How is the acoustics of the speakers?
Are they those 2? Are they from the same speaker? Or not?
Yeah, yeah, sorry. I didn't say just for my time mostly. But yeah, it was the same speaker that recorded the same story twice.
So the same British speaker, female. And then we transform from NDS. We just going to slow NDS.
But yeah, we compile between FDS and DS and ways the same speaker is.
Okay, cool. Thanks.
Okay, so if there's no other questions. Leave it there. Thanks again, Giorgio.
Thank you. Thank you, everybody.
It's a great job. You did a lot of work in, Short amount of time actually not not quite so short, but you did a good job and your project so Thanks for setting a good example for whoever follows next year.
So we'll. Ended it there for this session. That's the last.