David Poeppel - What Language Processing in the Brain Tells Us About the Structure of the Mind

and before we talk work let's talk about some other issues for a minute and for better or for worse we live in interesting times and I guess it's for worse so just as a little reminder before we talk about the fun and intellectually stimulating and more geeky things I think we should remind ourselves that were the people who have access to information we're trained to evaluate information so and don't forget to actually think about something and do something right so to remind you of an stein had a good quote about this which is and the

world is a dangerous place and not because of those who do evil but because of those who look on and do nothing so if you're doing nothing that's not enough and we were ostensibly the intellectuals who have access to evaluate things so and it's the responsibility of intellectuals to UM speak the truth and expose lies it's actually the title of a book by Noam Chomsky so that's my school marm ich and reminder before we start and on a lighter note although I don't know if it's possible to have a later no no no um the

I'm gonna talk about three things that are very much related to what John talked about so John and I had to do rock-paper-scissors who got to go first and got to talk about our paper and so he won so I have to talk about something else so thanks a lot for that and so I will talk about and the problem in a kind of related way and structured in three sort of little sub chunks one I call the Maps problem which is related to the kind of implementation level of analysis so localizing what what do

we do when we do cognitive neuroscience one I call the you know cleverly and confusingly the mapping problem which is how do we map establish linking hypotheses between different domains of experience but and that'll be the biggest chunk and I'm going to talk about some at least one experiment in perverse detail to give you the this kind of work but hopefully in a way that's clear to everyone and and then I'm going to talk about what we ended on and John's talk namely the experimental granularity problem that is what are we actually trying to do

are we doing more naturalistic experimentation should we have theories hypotheses and so it's the road map so let's begin with this question how naive are you willing to be when you do so I work on speech perception language processing as Barbara said how naive are you willing to be and apparently in our field really naive so if you go to a vision talk this is the picture you'll see right so this is the an old from an old paper by fellow man and Venice and this is the visual system the anatomy and all its links

okay and nobody bats and Isis it's a really complicated system makes good sense here's the auditory system really complicated right so by the way this must have been done by somebody who works on the brain stem because auditory cortex is just this little thing up here so this person really enjoys the brain stem and then you language talk even now even like let's say last week and then you get this like really so there's a Broca's region and the Venegas region and then there's a cable so I mean that seems um wildly optimistic and in

the unforgettable words of the physicist Pauli since we're quoting physicists today it's not even wrong so if you remember anything from my presentation and you should remember only one thing then it's this that's this is a it's this did a useful service it's very important but it's incorrect in many ways okay so and why what's wrong with this very simple model well for starters let's just look at it kind of pure anatomic fact so imaging has been useful in some ways and it's moved this ahead so for instance take the famous Broca's area which we

should think of as Broca's region well if you look at it and all the little colored blobs in a really tour-de-force experiment by cutting omelets and colleagues from a few years ago it's really like ten regions and that's each region obviously has laminar organization so do you really want to make a kind of monolithic functional claim about this much tissue I mean that's idiotic I mean nobody would do that in vision or in any other field so that's just a bad idea similarly the wire you know the nice wire going from the back to the

front yeah that's not really true there's actually a huge spaghetti wiring diagram that's extremely complicated which people are trying to work out so that's not a good and if you really look through the whole map of things that are going on if you look at recent review articles the parts of the brain that are underpinning language processing various ways look more like Europe in 1648 so extremely complicated and basically the entire brain is implicated and from my own work I also know that it's the other side of the brain so you can't just be a

left hemisphere imperialist that's also wrong okay so these are kind of maps right we make maps and of course it's extremely useful because we have to figure out where we're looking I mean where are you going and how are you trying to you know parse the problem and I've done some of that work myself so for instance when you process speech very you know just you want to recognize the word or - I want to say reg thank you sir red chair and there's a lot of machinery involved I mean even something as elementary as

recognizing a word and putting it together with another word requires a lot of different sub routines so computational subroutines and a lot of different brain areas in both hemispheres so even something that's putative Lee but you know functionally banal has it has lots of different moving parts which we try to understand and some other things hold for and that meaning corpse now if you really want to go down the you know let's throw a big data everything routes you end up with something like this from an influential paper by and Jack gallon slab a couple

of years ago and this was actually the cover of nature so this is where all the words are in the brain so apparently everywhere so this is if you have a kind of orgy of data and do very sophisticated data analysis you end up with this now since you just heard John you might ask yourself who cares I mean what if I now I mean so this is an interesting kind of Maps problem I made a very complicated map but have you understood anything and if you explain anything when and clearly if you reflect on

this for let's say 30 milliseconds no right I mean it's a beautiful I mean I recommend the movies they're a magnificent piece of art I'm an empirical aesthetics professor after all and they're aesthetically valuable but they don't actually teach you anything so what's the problem when we do this map stuff the problem is that and it's just an intermediate step it's kind of part of our homework that is to say we do need to do this kind of cartography as we do need to know because it's in fact true that things are spatially organized but

it doesn't mean that that's actually that's sort of like the beginning of the inquiry not the end it's not an explanation so first message other than the classical model that you saw in a textbook for language is not correct right remember that that's on the quiz and the second point is don't confuse localization with explanation I mean I'm as much of a toy obsessed scientist as the next guy I like all my big tools and but I do I too and mislead by you know making it's fantastic graphs and thinking oh I found this blob

and therefore it's involved reflected and so lift the filler terms that I've all used and continue to use so you know you have to back off of that so first message and be very careful about and what an explanation is and what understanding is so this is riff you know we started this earlier on so that's the Maps problem so simply taking the cool techniques we have in making maps is a reasonable intermediate part of the research program but it surely can't be the end and it's not an explanation of any kind um so let's

get to this issue what I called a mapping problem that's a little so that's actually a difficult problem and that's the issue that that John laid out earlier which is how do we negotiate between levels of study and what are different levels of explanation so suppose I do the experiments that I go to the and Society for Neuroscience or the door check is there chef Jun Fossum or any of these kind of things and I say look what do you think are the primitives of your field of study so if you had to write down

the parts list literally that you know what are the parts that you need to account for any of the phenomena you study you're gonna get concepts like dendrites or oscillation or synapse or you know you name it so the class a sort of architecture of neuroscience and there's very little debate about that I mean there's some debate that people might say now we have a good sense of what the implementational level is in terms of its primitives now if you do the same so and there's you know we we love that that's all good yes

do the same exercise for let's say the linguistic Society of America and say well what are they up to the elements that you work with to account for any phenomenon and language processing it's going to be really different right so it's going to be constant syllable morpheme small clause you know concatenation and so on so now you have these lists you have a parts list right right so you have the parts list from am I allowed to walk up to here because my mouth was connected to my feet so okay okay so you have a

parts lists from the neuroscientists which you know you can look up in any neuroscience textbook you have a parts list thank you a linguistics textbook and there's you know of course there's interesting debate but this is the right way to carve it up but it's you know it's productive and this is sort of the level of analysis and now what I mean how are these ostensibly related well we haven't the vaguest idea right we don't know anything at all so basically we're done here and we should stop and go to lunch and that is actually

a reasonable answer so a reasonable way to proceed is simply to give up now and say I want to study linguistic phenomena so I'm gonna stay at this level of analysis and I want to study neuroscience I'm gonna stay with us nothing wrong with that that's the question is do we have you know are we going to pursue the research program that these things are related and so then how are we going to make these linking hypotheses and what is their flavor so the flavor of the current climate is what John discussed earlier which is

a kind of relentless naive reductionism as we assume everything we do in the on the behavior or cognition sorry Brenda aside we can we can debate the merits of the terminal the strategic use of terminology and so everything on the left side should be reduced to the terminology on the right side right so there should be just a mapping from the left to the right that's the easiest because those are the more basic smaller units and but that really doesn't work and one example that we discuss you know you didn't have a chance to throw

in so I'll throw it an example that we discussed in our paper that is an example of why it doesn't work is the famous C elegans right so there you have a case where we really understand the the structure and of the worm extremely well so that C elegans has you know 302 neurons 7000 connections we know the genome we know the entire wiring diagram but we don't understand anything of how the damn worm does anything so that's shameful that's embarrassing and if you and you can actually have an you know let's say vigorous lunches

with Cori Bargmann debating why don't we don't we understand whether queer moves left or right and because there's no interest in that level of analysis to actually formulate the questions in a more granular way so that doesn't work so how do we proceed well we just have to if we want to not keep these separate if we want to try to find some linking hypothesis we just have to sort of figure out how to do it and I'm gonna give you now one kind of experimental example of how I'm proceeding just to show you how

you know it's not you know it's not successful but it sort of tells you something and what I want to conclude with in this section of the talk is to tell you here's some neuroscience data that has implications for how we interpret the structure of the minds which is what my title is about so this was the kind of bloviating mansplaining part now comes regular experimentation so here's the ready for that we good so far so good all right so now let's talk about some research so the experiments are a bunch of people but the

two people who've played the principal role in this work are and knighting who was a postdoc of mine and is now a professor at je Jiang University and we'll see a Maloney who was at NYU is now at max Duncan so the problem is a you know is a simple problem that Brenda mentioned is that as I'm talking all you're getting is a physical vibration of the air in some kind of structured way and you have to abstract from that information that's very complicated and weird and derive meaning from it and the question is how

do you do that and so I've worked for a long time on different aspects of this and I want to give you one example of experiments that I think give us an I think make a little bit of progress or hope they're wrong in an interesting way so here's what happens when you this is a sentence cats and crocodiles don't play together which is true and and this is what you learn when you take an intro class and you know linguistics or phonetics so this is called a spectrogram it has time on the x-axis and

frequency on the y-axis and then we learn to go through and say oh look at this little splotch here that's you know a burst of energy and then there's some transitions here and this is what automatic speech recognition does it takes a signal like this it goes through sliced by slice and tries to identify the information here and it does an amazing job actually automatic speech recognition the last few years has gotten really impressive it's it's quite something has nothing go to the brain by the way it's not never brain works but it works well

now there's another part of the signal that you see if you this is what actually arrives at the ear this is waveform it has these big bumps and valleys these are sort of Peaks and energy and then goes down up and down up and down and it turns out so it's called the envelope because it's like the energy envelope across the signal so it turns out and very usefully and this has now been done Mina for about ten years or so that your brain actually is incredibly sensitive to this particular variation in the signal and

actually locks on to it it phase locks really precisely to the entities energy fluctuations so here is just one example so one line the red line is the this envelope of the speech so this up and down the other line is in this case from M eg data you see how closely it tracks it so it's like the brain waves are surfing on the speech waves now um so that's just an observation right that's just data that we see and we can do it many times and it's been done in many different experiments and different

techniques you have to ask yourself who cares always a good question about one's experiment who cares so if you're tracking if you're just bumping up and down on this envelope does this do any work for you so let's actually look at the envelope for a second cross-linguistically is there anything interesting here that we want to know about so as it turns out it's extremely regular so this is what's called a modulation spectrum so this is basically this is this envelope this thing this line going up and down now characterized quantitatively in a cross language is

anything actually regular in this thing like should your brain care is it something useful and it turns out and that this and if you look at the frequency of change of this envelope it's extremely regular and it's very peaky across a whole bunch of languages so yours whatever nine languages and different speaking style so with audio books and sentences and huge corpora and conversations so whatever it is across languages it's between four and five Hertz so the modulation of your thing is between four and five Hertz no matter what obviously there's variation within a sentence

in between a little bit between speakers but it's extremely regular if that's not a peak I don't know what it is right so this is a peak so that means that the rates that you hear is roughly modulated between four and five Hertz so anything hmm have I seen this number before as it turns out that rate is exactly the mean so the period of that is the mean duration of syllables across languages so that's very useful so it turns out if you actually go and surf these waves of the speech and your brain locks

on to them what you're getting for free is the typical modulation of speech as its modulated by syllabic information but that's cool because you need that to actually segment the thing you need some kind of chunks or primitives so there's a kind of more fancy physiological way to talk about this I'm not but let me just point to one little thing and speech comes in as this continuous waveform right this is the thing you see up here it elicits activity in the afferent auditory system so this is a spike train so these are action potentials

can build up above right and you see this is not structured in any way then it in this input interacts with brain oscillations which are just part of the architecture that's the breathing of the brain at different rates and turns out this interaction inserts the little white spaces that are not there so why is speech that more difficult than reading in some sense because in reading I tell you where the white spaces are and you know where to look but speech doesn't have that luxury you have to do that work by yourself so the functional

interpretation of you know getting onto these waves is the following it's parsing the world into little acoustic chunks so here's the sentence she had your dark suit in greasy wash water all year there was a standard sentences from a database and this is what it looks like in a spectrogram right so it's time on the x-axis and frequency on the y-axis and you can't you know obviously there's no markers for words you don't know where the words are the phrases are anything and what this locking to the envelope does for you is it actually segments

the information into chunks so it inserts the white spaces so it's a algorithmic solution to actually segment into usable intermediate things so that's quite nifty because you have to solve that problem you know whether you're a newborn or an adult or listening to a different language so that's kind of cool and now comes the of course day well okay so the guy told me about I can listen to syllables but I'm not just interested in listening to Bob Bob Lee blah yeah I'm actually trying to extract words phrases and meaning from this information so is

this trick somehow scale-up abble to something that I'm more interested in so this is now where the experiments that gets more fun so back to the signal itself this is what you're sitting here this is what you're getting to your ears so the gray line is the waveform hitting your ear the red line is the speech envelope but of course are different levels of analysis that is I've been telling you about just a very surfacey level the phonetic level and as a syllabic interpretation and there's some correlation between these bumps and valleys and the the

segmentation into things like they went on vacation right so that's nice that's a nice intermediate result but what you're trying to understand are words like they went on vacation and none of that information is actually in the signal therein lies the problem right so when you hear something it doesn't come pre-labeled like now comes the now now comes the verb now comes the preposition you have to do all that work by yourself and then of course there's structural information and as I've think information like they went on vacation has a certain structure this is how

you interpret it it's its meaning ok so far so good you get the general problem so now can we scale up our little oscillations and train to the signal thing and so this is how we're thinking about it ultimately I'm interesting in parsing linguistic structures right so continuous speech and that has different levels of representation from sentences phrases words syllables these are sort of non technical ways to think about this and what I just talked about right now is that the neural code for each linguistic unit must change at the rate of that unit right

so if the syllables happen at a rate between four and five Hertz across all languages then the code dealing with that must be happening at that rate that's a reasonable assumption if that's and we have neural oscillations that are sort of the breathing of the brain due to its excitability cycles so that's that you get for free that's off the shelf and now the question is does that scale up to these higher levels of analysis right so this is the question we know the bottom that's very well-established for many years does it work for the

top okay here's how we try to test that in this very peculiar experiment so we make materials with different levels of abstraction so remember I just told you that the first thing we do from a brains eye view is we create we can we create discrete units right so we do we segment something into discrete units now the question is are they abstract so here's how we do the experiment we this week Disick's parent we've done and I'm gonna show you results from Mandarin in English but we've done it now in Hebrew and in German

and French Italian so it's a you know works across the across language so suppose I take a bunch of monosyllabic words so this is you know the Mandarin version and here's an English is impossible the way so it's an example like dry fur rubbed skin new plans gave hope each one each syllable is carefully crafted to be exactly the same duration 250 milliseconds and I concatenate them so the thing happens at a fixed rate right so they just happened one two three four one two three four one two three four there are no breaks so

there's a rate of the stimulus that's four Hertz completely regular but then I fiddle around with them I record each individual item so I can create higher order phrase right so to go together so they're going to be at a different frequency and if I put them four of them together I generate a sentence so the stimulus itself will change at four Hertz the phrases will change it to Hertz and the sentence at 1 Hertz but none of this information is in the stimulus you have to build that all by your lonesome right so let's

see what happens and let's see first of all to you how it sounds because it's so weird so this is what it sounds like in Mandarin for example now so you're allow you're lying in one of these machines in this case it's an Emmy Jima Sheen's is like a giant hair dryer which is superconducting quantum interference devices all around your head and you measure the magnetic fields and your mat this is now we're taking since it's Mandarin materials let's start with Mandarin speakers so we take native Mandarin speakers we stick them in and we record

their brain activity to see how they respond to this in the frequency domain since I just told you the syllables are happening at four Hertz the phrases at two Hertz and the sentences at 1 Hertz let's see what happens and here's what happens this is the part where you go wow and so unsurprised so and here's our little quasi linguistic tree right so each of these grapes is one of the syllables and they go together into you know two are paired for a phrase and the whole thing is a sentence and it happens continuously so

you get a big peak and brain activity at four Hertz it's exactly the syllable rate that's not surprising that's just a replication of what we did all along that just shows that your brain tracks the envelope faithfully okay that's a reality check but then you see two big bumps one at the rate of the phrase and one at the rate of the sentence but now you have to ask yourself where did they come from they didn't come from the stimulus from the signal because it wasn't there nobody said I'm the phrase I'm afraid I'm afraid

so you had to actually add that in so that's a very nifty result and from based on neurophysiology and now you have to ask yourself other questions of reviewers so let's go through reviewer number one reviewer number two and so on and so reviewer number one well maybe this is just some kind of interesting dynamical systems problem and you're seeing subharmonics I want to see a list what happens if you just mix them up and you can't put them into structured representations anymore good question you present a list right so this is like eggs butter

sugar toilet paper you know so there's no phrasal information you still present it at the same rate and you get a response at the correct rate but you don't see a response for phrases or sentences so it's not just some kind of physical property of the system so okay reviewer number one go away easy and and while we're at it we yeah you know reviewer number one be can you do it in just smaller chunks so can you show that you follow smaller chunks you can so if I just put them together in little phrases

like Angry Birds dirty laundry boring talk then you get basically you always get the frequency that goes with the stimulus that serve your internal reality check and then you can show that you actually track phrasal information but not the sentential information so that's that's a good sign and can you then you can replicate the original experiment now let's go the other way around let's take English listeners and from blue states and but give them the Chinese materials the red cherries now they are obviously going to hear the stimulus which is just a you know four

Hertz modulated and they're not going to get any of this because they don't understand Chinese as far as we can tell and it's true so the English listeners get a big peak so they follow the physical signal but they don't internally construct the representations necessary for decoding okay good now let's replicate the thing for English make sure that it works in here here's what sounds like in English so you're lying here in a machine for about an hour and a half listening to this just to give you a feeling for the pain poor friends Bill's

our boss wrote notes drunk dudes sang plums fun games waste time kind words warm hearts tall guys flayed camp rude cat's claw dog quiet lamb ate grass would shelf holds cans iced beer cost cents new plans give hope large hands built nest imagine doing that for a while and you know it's psychiatrically challenging yeah but it works quite nicely okay so it works for Chinese Mandarin it works for English so we're home free so interim conclusion cortical activity is the breathing of the brain these low frequency neural oscillations are entrained to sentential and phrasal information

in the speech but that wasn't there so you had to bring that right so your brain had to construct that on the fly and because we did a lot of you know this is a long painful paper don't read it and you it's not confounded by acoustics because these were actually individually made flattened and so on and so this is something really about you bringing it to the task so your brain is constructing these grammatical structures now comes a real reviewer number three kind of questions these were a little tougher so as you know the

current flavor in psychology computer science and neuroscience is everything is about statistics its statistics all the way down that is its corpus analysis of everything and then you can predict and fit almost anything so the question is is what we're seeing in this kind of pattern of brain activity really structure building operations or abstraction or is it simply following a statistical model of the corpus statistics in your own head that's a fair and irritating question to say the least so we built another experiment this is where we this is even more painful we brought people

into the lab and taught them a fake little language for several hours where the transition probability is held constant right so if you're just tracking transition probability which is what the argument is suppose I over load you with a particular transition probability like in this case let me play it for you this is now no really annoying John speaks Dutch her dad wrote a book her dad speaks French Jess wrote a poem her dad didn't answer a girl speaks Russian the boy wrote a story the boy didn't move John didn't come her dad speaks English

so a few days of that really you know well and why you undergrads whatever you know expensive tuition the so if you're just now you if you're just tracking transition probability you should be tracking you know whatever is the transition probability of this which is one over five should now show up in the response overriding or at least influencing the structure building operation right so let's see if so it's a different prediction the prediction is your brain response will look like something on the lower right you're just tracking the transition probability and no longer the

structure building operation we did one more that's even funnier so because you know many reviewers this one just I like playing cuz it's fun these are completely predictable sentences so once you know the first word you know the rest of the sentence if it works dogs can be smart my cat is so lovely amy owns and farm home work needs to be done nancy works at home rumors are not true he will take the train mat marry a carol earning money is hard new york never sleeps and so on and so forth right so for

these kind of things you participants come into the lab for several hours are exposed to this kind of nightmare then you come in the second day that again so numerous hours of learning this kind of stuff and then you put them in the machine and you measure their neuro physiological responses to this and the question is are they now tracking the transition probability which we've artificially ramped up or are they tracking the structural information which is part of the kind of syntactic information and the answer is of course other weather wouldn't be here they're tracking

the syntactic information okay so you can't override so the response that you get from these looks just like the response from sentences you're not that are you know novel to you each time so of course you always track the stimulus rate that makes sense but you also build these intermediate phrasal representations and sentential representations even though you have the entire stored so that's a nice and interesting challenge to a pure statistical explanation okay now reviewer number four we're almost done said it has five reviewers and reviewer number four says but nobody speaks like that which

is true except an uncle but um but but nobody really speaks like that we have we use you know short sentences long sentences so naturalistic speech is very different so we had to do the experiment again then you have to use very different analytic techniques to make sure the scales up to a naturalistically structured speech and it does I'm not going to explain this to you because this is a bit more you know this requires even a glance into the supplementary materials and but if you play these are sentences of different lengths many different lengths

short ones long ones and if you align them all let's say at the big break here where the vertical line is is the end of one sentence in the beginning of the next one and you of course see right away that there's a kind of dynamics going back at least five or six syllables towards the end and then a big onset to the beginning of the sentence but you have to ask yourself how did you know um how do you know this is the beginning of a phrase well you have to you don't right because

it's not in the signal you have to construct it and in fact if I give you naturalistic materials and I change the size of the Constituent so let's say three syllable noun phrases and four syllable noun phrases I can manipulate these responses in systematic ways so you must be tracking structural information or what in linguistics is called the constituent structure so that's cool and weird because it's abstract very abstract okay now last reviewer the last reviewer said this was even more annoying I want to see you localize these responses in the brain because this is

not interesting this is just behavior and something weird about linguistics where show me the brain so we to do this conventional electrophysiological data aren't so straightforward to use for localization so we had to use and epilepsy patients it's called electro quartic or graphic data so this is and we're a few patients in this case five we now I think a seven have each is one of the electrodes that is part of a grid implanted in the patients for pre surgical evaluation prior to epilepsy surgery and we now record the same experiments like this and here's

the cool so you know we follow the rules of and the reviewers relentlessly but something interesting comes out here so now what would you expect so is there really a structure building operation that just builds these constituents and then ships them off for the next computing step or are there many places where this might happen and it turns out this is what you find this is how it was pretty neat so here's the this is now done in English by the way right so these are English patients English speaking patients in the NYU epilepsy unit

so take a look at these down here the red ones these these are M electrodes where you get a response only to the phrasal information and to this sentential information so the one but not the stimulus itself that is those are electrodes and are particularly interested in what I'm you know what's happening at the surface at the input rate but they're really interested in these and they're distributed in a bunch of different places in the posterior part of the temporal lobe temporoparietal Junction some parts of the frontal cortex so it's not a little low it's

not a sink you know one off location that does this kind of structure building but it's actually in a but it's a kind of computation that's lives in a bunch of different areas which i think is a pretty interesting you can do it in many different places okay conclusion of this one experiment this is one experiment there's obviously many experiments and the so what you're you know the takeaway is that the brain breathes in these low-frequency rhythms neural oscillations and these slow rhythms can match the timescales of larger units of linguistic units even when they're

not present in the input signal so you can use those to actually build abstract structure and that's what you're trying to do and these tracking of these linguistic units is not based on statistical information transitional probabilistic information and it's not based on acoustic information so what's the most likely explanation using Occam's broum and it's that you actually have internal rule or grammar based cognitive based system or copy a set of computations that actually drive these things which is so what are we where are we at the moment you build discrete units and these discrete units

are abstract and interact with each other rule-based that is the conclusion of the brain data so far okay last point so this is unpopular and you can like this conclusion but you would be wrong because it's the it's the right view and it's not popular with neuro scientists and engineers that's for sure but I have tenure so okay last point so these little is so what's an oh let me finish what's the linking hypothesis so what's the you know what's sort of where does it leave me well the linking hypothesis is and between a particular

kind of neural oscillation and the physics of the signal right so the modulation spectrum so that speech is modulated in surprisingly regular ways across languages that's physics the neurophysiology directly linked to that is the neuroscience right so these oscillations you know building onto that and the fact that this links directly to syllabic duration and segmentation that's the linguistic part so I have a nice link between what you need to decode in linguistics what the mechanism is neurophysiological II and what the physical signal is that conditions it so that's a very you know kind of rich

research program to pursue okay last point now about the experimental granularity problem so again this goes back to John's talking and it goes without saying or I thinks it goes without saying that what we're all interested in is sort of you know how we work in real life I mean how do we work sitting at a kind of div' bar and Baltimore and walking in San Francisco or something like that amendment and that requires you know complicated aspects of multi so the world is multi-sensory it's highly contextual II dependent it's interactive and of course that's

what we're trying to do right so we want to have a kind of understanding of how we work in a naturalistic way and the kind of stuff that moves us that we really get excited about or let's say you know things that make you immediately reflexively emotional or that are super cute I have cat videos too here socially interactive and so these actually the most frequent pictures and Google - well the most frequent is not rated PG I can't show you that bit and or awesome these are the kind of things we want to understand

things that are rated you know in a social context exciting and I'm on board with that I would like to understand how emotion works and why things you know I think my dog is cute and but the fact of the matter is our understanding hasn't come from that our understanding has come from a very different kind of experiments such as this kind of thing right so this is just to learn something about contrast so that's neither cute nor socially interactive nor awesome it's just a carefully constructed stimulus to understand something about the decomposition of the

problem or color right so this is a kind of Mondrian inspired picture by Edwin lands in experiments on color vision the retinic theory which he developed or the famous gob or patch I mean another experiment is a subject using gob or patches of Troup myself but many many things have been learned using gob or patches which are also not particularly cool or visual motion is learned using you know the random dot patterns that move around so the models we've constructed have been pretty successful in terms of what we're trying to distinguish between causal mechanism and

understanding and build on these kind of careful decompositions and rich psychophysical research programs have been going on for a hundred years and so we had they make reference to concepts like you know similarity space or gob or filtering and things like that they don't make reference to emotional or awesomeness and cute so the issue is how do we balance between our desire to get to the rich socially interactive things that we care about but this kind of granularity that's theoretically well motivated to compute the explicit and also realistic in terms of neuroscience and that's quite

difficult and the right paper to read about it is this one this tells you this is the you know run don't walk to the library to download this or do it right now because I mean at least it's FEMA ties is the challenge of doing this in a meaningful way okay so what's the conclusion making maps is not an explanation it's an intermediate part of the homework problem it's a really good and important problem but having but localization is not explanation I fall into this trap every time I read the New York Times because there's

always a picture with the brain and it says love or freewill or you know God or whatever but it's just not it's in seconds the mapping problem is the really serious problem here which is how across levels of study do we find linking hypotheses that ultimately allow what you know what understanding is and what do you think an explanation is very difficult and the world is a complicated place but we still need to be what I all radical decompositional lists to generate explanatory understanding so we want context but we don't we won't get around but

what did we call it a carefully designed you know behavioral experiments that really parse apart the nature of the problem and give you a new thing so my resolution for this new year is that we have good spatial resolution and we have appropriate temporal resolution but we really need better conceptual resolution and that's all I have to say today thank you for staying [Applause] yes hi John in part one I felt like you were preaching to the choir as far as I'm concerned and then in part two I felt like either you meant to tell

the choir to go home and you're gonna replace it with like a fancy sound system or there was like a wink that I missed where it was gonna be hey this is how we convince neuroscientists but you know don't worry relax so you'll tell me which one it is but in part two I guess there's a number of ways one could ask the question but in what sense if the data is all really cool but in what sense is what are what we saw I think you use the term structure building operations or structure building

as opposed to just structure reflecting another way to ask the question is yeah I mean that's just as just on that particular point you're absolutely right I mean it could be so the we just measure some output of a series of underlying computations so that so when you see a peak like that that's not you know that's an index of something that's happened in building what's I take to be a pretty abstract structure so you can call the structure reflecting that's fine right so so in that sense at the end you also said something along

the lines of this is the right answer but the neuro scientists and engineers don't like it but didn't we know that was the right answer since at least like Chomsky's review of verbal behavior and you know even before that in the sense of you're saying look there's the signal doesn't in its raw form have a lot of information but the information has to be there and there have to be computations that are able to break it down but I I'm just we must not hang out with the same people because in my own lab or

I hire the wrong people in my own lab I have postdocs who come you know large let's say from engineering particularly who are deeply unhappy with the conclusion that's not based on let's say a mapping to the surface or the episodic feature based information of the surface stimulus those who say look this is why we do the experiments with for instance statistically controlling the materials or manipulating them right because the standard view and the most popular view is you don't need abstract structure right i I agree I think these are this is old news to

me I don't particularly need to be convinced that we have constituent structure but I assure you that let's say my colleague Tony Martin does need to be convinced that there's constituent structure even though there's you know a hundred years or you know actually two thousand years of evidence for that kind of concept and the idea is why should I care about that kind of concept because probably I have a reductive thing to do away with it and the fact that we need it to actually build internal representation is just dismissed as unnecessary so I I

think I I do think it's the right conclusion but I don't think that everybody believes it I think it very few people do I think if we polled people here people who come from engineering or computer science they would rather see an explanation based on statistical contingencies and surface features and like let's say acoustic variation in the signal because we can measure it and we can manipulate it I just think it's wrong yeah I feel you David so it was a wink to the plier I'm with you yeah yeah thanks yeah there's hi Joe Ganim

let me be reviewer number six here I do those lower frequency syntactic signals attenuate if the subject loses mental focus yeah they do yeah you can actually I mean Ilana's Ian Gollum a former postdoc of mine has done a bunch of work experiments on this in doing it at different levels of vigilance seeing if we could they go away and sleep and that kind of stuff so yeah say you can they require not specific attention to something like structure building but they you have but you have to be awake and able to process language in

a canonical way so you could actually measure mental you can you can yeah it's you can do more than that so the experiments at Ilana has recently completed our using for instance the cocktail party thing so suppose you have two speakers concurrently and you slightly offset the frequencies right so that their rates are a little bit different you can say okay look please listen to speaker a then you will track a and listen to the speaker B then you will track the frequencies will shift to track speaker B so you can actually use this I

mean for us this is largely a kind of methodological trick to use something like a but yes you can track it quite carefully that's right Paul Paul Smolenski Johns Hopkins cog sorry so I'm wondering whether the problem that you are up against really is about abstraction as opposed to intuition so suppose we repeat your talk with vision instead of language we show there are evidence of neural responses two objects two scenes two types of scenes these are very abstract things but they are also part of our intuitive understanding of the world and so do you

get the same kind of resistance to those abstractions and if not is it really about abstractness I think you do I actually get the same in the computer I don't and not a computational vision person but I think that they degree to which people agree on levels of abstraction and links between surface based features the visual objects and abstract interpretations that are independent of features let's say something as simple as viewer independent recognition that's a very high degree of abstraction where you need to be entertaining something at the servers and yet dealing with an interpretation

that's very abstract which allows you variability at the surface I think it is about abstraction and abstraction related across levels I just didn't think that that was controversial in the same way I knew you are not the same reviewer you're not dealing with the same I think people in neuroscience and in psychology are really resistant to this notion and surprisingly so even though I think it's hardly any counter evidence it's I mean it's the sort of there was one more question a little bit a few a little bit yes right there I'm Diana Grissom with

Johns Hopkins University and I'm working against that third objective of the Institute that Barbara spoke about and connecting the science to practice and on a very beginning stage of it and in finding your research fascinating I work on educational equity gaps and with literacy if you're studying if your subjects were NYU undergrads are they coming in with a better understanding of the linguistic of the grammar syntax and all this does that affect it your research and what you're finding opposed to if you are the subjects came from a lower educational yeah it's a good question

I mean the the the and the answer's no this is this is actually tacit knowledge I mean what you're tapping here is tacit knowledge that you get for free for everyone that's gonna be independent of that I mean it's sort of a very basic I mean every barring gross pathology every speaker from four years old or even younger has access to this kind of cognitive machinery and set of computations that do this so it's sort of that that is actually largely independent of that there may be other things so for instance the person xeo is

asking about attention and things like that they may be differentially regulated as a function of that but the basic building blocks you know building constituent structures and you know using them to decode information is sort of independent of that it's really a architectural feature of the operating system in terms of using this kind of machinery to put it into a school I mean I have done experiments where we measure an entire school class at the same time or semester and I can share that those papers with you but there you can see how you can

use some of the tools of your recording brain activity concurrently for instance two different teaching styles and try to figure out what we're you know what's what it means to be synchronous engaged you know how this kind of brain activity reflects what a kid is doing they're more that's more you know applied and that's it it just says that last mapping that you did where you say let's look at the regions of the brain yeah can we effectively use that working with educators on how do we stimulate that area for trying to reduce gaps I

think that that's a question may be left for the general discussion I think that's a very controversial that but it's a good one it's a good lead and maybe did the general discussion because I think people will have different different feelings about that it's very thank you very much [Applause]