Large scale, open neuroscience is a work-in-progress

The International Brain Lab (IBL) project has an outstanding team of PIs, covering both theory and experiments. The measurements themselves are interesting, and can lead to important insights. However, what is most distinctive is the organization of the project. The experimental labs will all train mice to perform the same task, and each lab will record from different brain areas. They will match conditions agreed to ahead of time. Then the labs will pool their data for analysis.

A project like this hasn’t been done before in neuroscience. Consortiums have pooled data before (genetic data, fMRI, etc.), but the individual labs conceived the projects and collected the data however they saw fit. The pooling process can be laborious, involving quality control, metadata conversion, etc. In the end, the data collection methods (e.g., task, modality, acquisition parameters) might simply be so heterogeneous that there is no gain in the pooling. Whatever small effects that could have been revealed by the increased N, or distributed measurements, are obscured by the heterogeneity. To address that weakness, the IBL will front-load the conforming process, agreeing ahead of time on all of the experimental specifications, including the data formatting for sharing (more on that below).

That’s the idea, anyway. How will this work in practice? Anne Churchland sketched out the management plan at the recent SCGB meeting. It’s a flat structure, rather than a hierarchy. Lab heads are not in the habit of taking orders, and the management plan appears to be very sensitive to this issue. In the limo back to the airport, one of our esteemed colleagues remarked that the slide describing the “propose – discuss – revise” decision-making process did not include a step where an actual decision was made. There’s also the issue of how to get lab personnel to spend time performing experiments for middle authorship on a paper with a very long author list. Presumably many postdocs and grad students would rather work on a project over which they have more control, rather than function like a research technician. The former could be better training as well, depending on the circumstances.

So those are a couple of reasons why the IBL project might not work smoothly. However, I want this to work. I want this team to figure out what works and what doesn’t, so that this can be a new modality for organizing neuroscience research. Independent labs are where the most creative work is done, and that isn’t likely to change anytime soon. However, this IBL model could be an efficient way to organize effort in the field, applied when and where it is appropriate.

Some might view this model as a way for already well-funded labs to band together and win 8-figure sums of additional funding with little competition. Funding organizations aren’t perfect, but they aren’t stupid either. They care about return-on-investment, and it’s very hard to beat the bang-for-the-buck one gets from smaller research grants to individual lab heads or small-scale collaborations. With that in mind, the pressure on the IBL is significant. If they can’t deliver results as staggering as their resources, then there will be less enthusiasm for further funding for such projects. So the IBL PIs not only have to get a new consortium model to work, they have to get results that would have been impossible with smaller individual grants.

That’s not always easy to do. Just ask the Allen Institute. They have excellent leadership, a stellar group of researchers, and vast resources. They have created some tremendously important data set resources. However, they are not as nimble as individual labs, and so they have to be very strategic in their pursuit of neuroscience projects. They don’t want to be competing with, say, an individual HHMI lab. They need to be certain that their work is in a whole different league. Some projects are a great fit for their structure, but others are not. In the years of the Mind Scope program, now in year 6 of 10, they have learned a lot about what they can excel at, and how to make their structure work efficiently. They knew it would be a challenge to figure out how to do neuroscience at scale, as Christof Koch and Clay Reid acknowledged:

There is a risk that this project will not work out as we anticipate, and that the various brain observatories — looking at anatomy, physiology and modelling, for example — will not synergize to form a sophisticated understanding of the mouse visual cortex. There is no guarantee that neuroscience is ready to become big science — but the only way to find out is to try.

It is in that same spirit that I hope the IBL succeeds. There is value not only in the science itself, but in figuring out how to make this consortium model work well. I have some of my own ideas about how to scale up neuroscience projects, and maybe some day I’ll get to try them. In the mean time, I want efforts like the Allen Institute and the IBL to succeed, and figure out ways to make large-scale neuroscience work well.

Footnote on open data formats

The data format that IBL will use is Neurodata Without Borders (NWB). On the topic of NWB, if you follow Labrigger on Twitter, you might have seen this re-tweet.

saving calcium traces in NWB is like “if you give a mouse a cookie”…

— Justin Kiggins (@neuromusic) September 25, 2017

Click through to read the thread.

I laughed at this, though I do support the work that the NWB team is doing. Just like the discussion above, about how large-scale neuroscience project models are important AND still a work-in-progress, the same is true of open data formats.

Justin’s NWB tweet thread reminds me a famous email from Bill Gates in which he gives an excruciating narrative account of his frustrating experience with Microsoft software (PDF, newspaper article). Most telling is his comment on that email becoming public, and how he’s not embarrassed by it at all:

There’s not a day that I don’t send a piece of e-mail … like that piece of e-mail. That’s my job.”